CN113221881A - Multi-level smart phone screen defect detection method - Google Patents

Multi-level smart phone screen defect detection method Download PDF

Info

Publication number
CN113221881A
CN113221881A CN202110489131.0A CN202110489131A CN113221881A CN 113221881 A CN113221881 A CN 113221881A CN 202110489131 A CN202110489131 A CN 202110489131A CN 113221881 A CN113221881 A CN 113221881A
Authority
CN
China
Prior art keywords
image
defect
area
foreground
detection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110489131.0A
Other languages
Chinese (zh)
Other versions
CN113221881B (en
Inventor
陈垣毅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Zhenwei Food Collection Technology Co ltd
Original Assignee
Hangzhou City University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou City University filed Critical Hangzhou City University
Priority to CN202110489131.0A priority Critical patent/CN113221881B/en
Publication of CN113221881A publication Critical patent/CN113221881A/en
Application granted granted Critical
Publication of CN113221881B publication Critical patent/CN113221881B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to a multi-level smart phone screen defect detection method, which comprises the following steps: manually marking defects in the image, and then separating the foreground defect image from the background image; performing multi-scale defect region feature extraction on the data set by using a depth residual error network and a feature pyramid network to obtain a multi-scale feature image; and extracting a target interest detection area. The invention has the beneficial effects that: through the extraction of the interest detection area, the efficiency of image preprocessing is effectively improved, the problem that the existing general target detection technology is greatly interfered by background factors in a small target detection task of mobile phone screen defect detection is solved, a targeted enhancement mode is provided for the problem of data set shortage in actual floor application, the early-stage manpower marking investment is greatly reduced, and the efficiency of the whole scheme is further improved.

Description

Multi-level smart phone screen defect detection method
Technical Field
The invention belongs to the field of mobile phone screen defect detection, and particularly relates to a multi-level smart phone screen defect detection method.
Background
The screen is used as a core component of the smart phone and is a key point of human-computer interaction, and the quality of the screen seriously affects the experience of a user on the smart phone. Therefore, the production process of the mobile phone screen by each large mobile phone manufacturer is more and more demanding. However, the mobile phone screen is very susceptible to the production environment and the production process during the production process. In order to prevent the mobile phone with the defective screen from flowing into the market to damage the benefits of consumers and influence the credit of mobile phone screen manufacturers, the mobile phone screen manufacturers adopt some necessary means to detect the quality of the mobile phone screen. The traditional detection means is that workers are arranged on a production line for watching, and the workers detect screens on the production line one by one with naked eyes. However, the method has the defects of low detection efficiency, high labor cost, lack of uniform judgment standard and the like. In addition, some mobile phone screen defect detection technologies for calculating the traditional computer vision technology exist, but most of the methods only carry out algorithm design aiming at one or more specific defect types, once a new defect is met, a new algorithm needs to be designed in a targeted mode, and the universality is poor. Specifically, in the task of appearance defect of the screen of the smart phone, the following main difficulties exist:
1) defects have a strong correlation with background. Whether the defects exist in the region of interest or not needs to be comprehensively judged by combining background information of the position of the region of interest, and under different backgrounds, whether the defects exist or not is uncertain. For example, the different color defects are determined under the condition that the foreground defects and the background image have significant differences in hue, lightness, and the like.
2) Background information is extremely disturbing. For an irrelevant background region between the two-dimensional rectangular image and the irregular target detection region, there is a high possibility that there is a region similar to the defect.
3) Defects are of a wide variety and define ambiguities. Due to the differences of different production environments, production standards, production technologies and the like, different types of defect types exist in mobile phone screens in different production scenes, a unified classification standard is lacked at present, and the problem that the defect types are different from person to person in calibration exists.
4) The calibration data sets are of varying quality. Unlike detection data sets such as VOC, COCO and the like widely accepted and used by the academic community, the data sets applied in the field of the industry need to be specifically and specifically customized collected according to different requirements of task scenes. Based on the factors that subjectively calibrated personnel lack of cognition deficiency on an artificial intelligence technology and objectively defect sample distinguishing degree is not high, the industrial defect data set of the screen defect of the smart phone has two main problems: firstly, the sample data quantity under different types has large difference and does not meet the independent same distribution condition; the second is that the number of whole samples is not sufficient.
The former three problems are specific difficult problems in the task of detecting the appearance defects of the mobile phone screen, the reason is that the definition of the defects is different according to scenes, and the 4 th problem is a common difficulty in landing application of various deep learning algorithms at present.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a multi-level smart phone screen defect detection method.
The method for detecting the defects of the multi-level smart phone screen comprises the following steps:
step 1, manually marking defects in an image, and then separating a foreground defect image from a background image; on the basis of the labeled foreground defect image, the color and the size of the foreground defect image are enhanced, and the sample number and the form diversity of the foreground defect are expanded; finally, combining the enhanced foreground defect image and the enhanced background image to generate a data set which is reliable in annotation, balanced in category and suitable for an actual production scene;
step 1.1, manually marking the position and the category information of a rectangular area where each foreground defect image in a small number of images is located, and then separating the foreground defect image from the background image;
step 1.2, performing image denoising treatment on the foreground defect image: processing image noise by median filtering, setting the gray value of the target point as the median of all points in a region around the target point, and enabling the pixel value in the field of the target point to be closer to the true value so as to eliminate independent noise signals;
after the image filtering process is carried out and the noise is reduced, the definition of the image contour and the edge can be reduced, the image after the noise is reduced is sharpened by adopting a Laplacian operator, and the difference between an object in the image and a background gray value is enhanced:
Figure BDA0003050158810000021
Figure BDA0003050158810000022
Figure BDA0003050158810000023
Figure BDA0003050158810000024
in the above formula, f (x, y) is an image after noise reduction;
Figure BDA0003050158810000025
the image is subjected to Laplace transform; c is an enhancement coefficient and is generally 1; g (x, y) is the enhanced image;
randomly adjusting the chromaticity, the sharpness and the brightness of the foreground defect image, and randomly rotating the foreground defect image by a random angle and randomly zooming in a fixed proportion range;
1.3, four defects such as black spots, dirt, scratches, hairs and the like are obvious in characteristics, and for heterochromatic defects, the defects are characterized by color difference with a region adjacent to a background, so that the enhanced defect foreground is difficult to be universally matched with a background image; in the randomly selected background image area, adjusting the pixel value:
Figure BDA0003050158810000026
Figure BDA0003050158810000031
in the above formula, valuenewIs the enhanced pixel value; valueoldThe value range is 0-255 for the original pixel value; brightness represents the brightness ratio and takes a value of 0.9 or 1.1, the brightness value represents 0.9 dark heterochrosis, and the brightness value 1.1 represents bright heterochrosis; the value range of exp is 1.5-3, and the method is suitable for different color defects of different forms; a and b represent half the width and height of the rectangular background image area, respectively; the geometric meaning of the factor represents the ratio of the distance from a certain point to the defect center to the distance from the defect area boundary to the defect center, the value is less than 1 and is 0.95, the defect area contains a part of background information, and the generalization is improved; centerxAnd centeryThe horizontal and vertical coordinates of the center of the rectangular background image region are expressed by (center)x，centery) Establishing a rectangular coordinate system for an origin, wherein x and y respectively represent horizontal and vertical coordinates of points in a rectangular background image area;
step 1.4, performing image multi-band fusion on the foreground defect image and the random position of the target interest detection area, constructing a Laplacian pyramid on the foreground defect image and the background image, and fusing each layer:
Figure BDA0003050158810000032
in the above formula, superscript i denotes the ith layer of the laplacian pyramid;
Figure BDA0003050158810000033
features of an i-th layer of the laplacian pyramid representing the output fused image;
Figure BDA0003050158810000034
laplaca representing foreground defect imagesFeatures of the ith layer of the pyramid;
Figure BDA0003050158810000035
features of the ith layer of the laplacian pyramid representing a background image; riA fusion area of an ith layer is represented, and the value of i is 2 generally; the multi-band fusion uses larger scale fusion at a low frequency position to avoid the cutting of a foreground defect and a background, and uses smaller scale fusion at a high frequency position to avoid the interference of background information on the foreground defect;
step 2, performing multi-scale defect region feature extraction on the data set by using a depth residual error network and a feature pyramid network to obtain a multi-scale feature image; the multi-scale feature images are fused by utilizing a feature pyramid network, so that the loss of small target defects in training is reduced, the defect detection generalization capability is improved, and a high-level feature map with high-level semantics is generated, wherein the high-level features have high-level semantic information and a larger receptive field and are suitable for detecting large objects; generating a shallow feature map with low-level semantics, wherein the shallow feature has low-level detail semantic information and a smaller receptive field and is suitable for detecting small objects; providing a series of possible defect candidate areas in the image by using an area detection network, and then adding the input and the output of the depth residual error network together by using a depth residual error network through jump layer connection to extract the image characteristics of the defect candidate areas; finally, classifying the feature map fused with all the information to complete the target classification and regression task; by means of convolution of a depth residual error network with the highest 152 layers, image features can be better learned, meanwhile, by means of a feature pyramid network, features of the image on different scales are extracted, large-scale position information and small-scale semantic information are combined, and the method is very suitable for a small-area defect detection task on a mobile phone screen;
and 3, extracting a target interest detection area based on the characteristic image in the step 2, aiming at reducing the detection area, improving the detection efficiency and avoiding background information interference: the background information is filtered, so that the accuracy of defect detection can be greatly improved; detecting a target interest detection area of an image to be detected shot at different stations through a large-scale anchored area detection network;
and 4, step 4: generating potential defect areas to be detected in the target interest detection area by using an area detection network, and judging whether defects exist in each potential defect area to be detected by combining the characteristic images in the step 2 to obtain the types of the defects; extracting feature vectors with equal length, judging the probability of each region as a foreground defect image or a background by a softmax function through setting anchoring with different scales by the region detection network, and regressing the positions of candidate regions of the foreground defect image;
and 5, outputting a defect detection result.
Preferably, the step 4 specifically comprises the following steps:
step 4.1, counting the sizes and proportions of defect samples in the data set, and setting a targeted anchoring scale; the anchored length-width proportion comprises five types of 0.2, 0.5, 1, 2 and 5, the pixel size comprises five types of 32 pixels, 64 pixels, 128 pixels, 256 pixels and 512 pixels, the defect area can be screened out better, and the regression process of the candidate frame can be accelerated;
step 4.2, combining the characteristic image, performing target interest detection area pooling operation on the defect candidate area, performing normalization processing on the image characteristic, and mapping the normalized image characteristic to probability distribution of defect categories through a softmax function, wherein the target function is as follows:
Figure BDA0003050158810000041
in the above formula, J (theta) is the objective function, N is the number of samples, y is the label class,
Figure BDA0003050158810000042
for prediction classes, θ is the model parameter, R (θ) is L to prevent overfitting2The regularization term, λ, is an empirically set parameter term.
Preferably, the defects in step 1 include hair, dirt, black spots, scratches and off-colors.
Preferably, in the step 2, the high-level semantics fuses the feature map with strong low-resolution semantic information or the high-level semantics fuses the feature map with weak high-resolution semantic information but rich spatial information.
Preferably, the target interest detection area in step 3 refers to an approximately rectangular mobile phone screen area in the image.
Preferably, the feature pyramid network in step 3 is composed of three parts, the first part is up-sampled by a convolutional network, the second part is down-sampled from the feature map, the third part is transversely connected, and the feature maps of the first two parts with the same size are fused to obtain the multi-scale feature map.
Preferably, in step 4.2, feature vectors with equal length are extracted by pooling the target interest detection region, and the maximum value of the defect classification probability is the classification category of the detection result.
The invention has the beneficial effects that:
aiming at the two points that the defect has stronger relevance with the background and the background information has great interference, the invention provides a method for extracting the interest area to be detected by the target by adding a layer of segmentation extraction process between the defect area detection networks, which can effectively avoid the interference of the background information; aiming at the two points of various defects, fuzzy definition and uneven quality of a calibration data set, the invention provides an image local area enhancement mechanism for the specific task requirement of the appearance defect detection of a mobile phone screen, can abandon the subjective difference of different annotators, has high expansibility and can be suitable for the continuous amplification of the task requirement; the defect detection method can effectively improve the defect detection capability and the defect classification accuracy, improve the defect detection efficiency and the real-time performance and meet the actual production requirement.
According to the invention, through the extraction of the interest detection area, the image preprocessing efficiency is effectively improved, the problem that the existing general target detection technology is greatly interfered by background factors in a small target detection task of mobile phone screen defect detection is solved, a targeted enhancement mode is provided for the problem of data set shortage in actual landing application, the early-stage manpower marking investment is greatly reduced, and the efficiency of the whole scheme is further improved.
Drawings
FIG. 1 is an overall flowchart of defect target detection according to the present invention;
FIG. 2 is a flow chart of defect sample enhancement according to the present invention;
FIG. 3 is a flow chart of an enhancement of the defect prospect of the present invention;
FIG. 4 is a flow chart of defect detection according to the present invention;
fig. 5 is a diagram of a feature pyramid network structure for extracting multi-scale image features according to the present invention.
Detailed Description
The present invention will be further described with reference to the following examples. The following examples are set forth merely to aid in the understanding of the invention. It should be noted that, for a person skilled in the art, several modifications can be made to the invention without departing from the principle of the invention, and these modifications and modifications also fall within the protection scope of the claims of the present invention.
Example 1:
the overall flow of a multi-level smart phone screen defect detection method is shown in fig. 1, and comprises the following steps:
step 1, as shown in FIG. 2, artificially marking defects in an image, and then separating a foreground defect image from a background image; on the basis of the labeled foreground defect image, the color and the size of the foreground defect image are enhanced (as shown in fig. 3), and the sample number and the form diversity of the foreground defect are expanded; finally, combining the enhanced foreground defect image and the enhanced background image to generate a data set which is reliable in annotation, balanced in category and suitable for an actual production scene; defects include hair, dirt, black spots, scratches, and off-colors;
step 1.1, manually marking the position and the category information of a rectangular area where each foreground defect image in a small number of images is located, and then separating the foreground defect image from the background image;
step 1.2, performing image denoising treatment on the foreground defect image: processing image noise by median filtering, setting the gray value of the target point as the median of all points in a region around the target point, and enabling the pixel value in the field of the target point to be closer to the true value so as to eliminate independent noise signals;
after the image filtering process is carried out and the noise is reduced, the definition of the image contour and the edge can be reduced, the image after the noise is reduced is sharpened by adopting a Laplacian operator, and the difference between an object in the image and a background gray value is enhanced:
Figure BDA0003050158810000061
Figure BDA0003050158810000062
Figure BDA0003050158810000063
Figure BDA0003050158810000064
in the above formula, f (x, y) is an image after noise reduction;
Figure BDA0003050158810000065
the image is subjected to Laplace transform; c is an enhancement coefficient and is generally 1; g (x, y) is the enhanced image;
randomly adjusting the chromaticity, the sharpness and the brightness of the foreground defect image, and randomly rotating the foreground defect image by a random angle and randomly zooming in a fixed proportion range;
1.3, four defects such as black spots, dirt, scratches, hairs and the like are obvious in characteristics, and for heterochromatic defects, the defects are characterized by color difference with a region adjacent to a background, so that the enhanced defect foreground is difficult to be universally matched with a background image; in the randomly selected background image area, adjusting the pixel value:
Figure BDA0003050158810000066
Figure BDA0003050158810000067
in the above formula, valuenewIs the enhanced pixel value; valueoldThe value range is 0-255 for the original pixel value; brightness represents the brightness ratio and takes a value of 0.9 or 1.1, the brightness value represents 0.9 dark heterochrosis, and the brightness value 1.1 represents bright heterochrosis; the value range of exp is 1.5-3, and the method is suitable for different color defects of different forms; a and b represent half the width and height of the rectangular background image area, respectively; the geometric meaning of the factor represents the ratio of the distance from a certain point to the defect center to the distance from the defect area boundary to the defect center, the value is less than 1 and is 0.95, the defect area contains a part of background information, and the generalization is improved; centerxAnd centeryThe horizontal and vertical coordinates of the center of the rectangular background image region are expressed by (center)x，centery) Establishing a rectangular coordinate system for an origin, wherein x and y respectively represent horizontal and vertical coordinates of points in a rectangular background image area;
step 1.4, performing image multi-band fusion on the foreground defect image and the random position of the target interest detection area, constructing a Laplacian pyramid on the foreground defect image and the background image, and fusing each layer:
Figure BDA0003050158810000071
in the above formula, superscript i denotes the ith layer of the laplacian pyramid;
Figure BDA0003050158810000072
features of an i-th layer of the laplacian pyramid representing the output fused image;
Figure BDA0003050158810000073
features of the ith layer of the laplacian pyramid representing the foreground defect image;
Figure BDA0003050158810000074
features of the ith layer of the laplacian pyramid representing a background image; riA fusion area of an ith layer is represented, and the value of i is 2 generally; the multi-band fusion uses larger scale fusion at a low frequency position to avoid the cutting of a foreground defect and a background, and uses smaller scale fusion at a high frequency position to avoid the interference of background information on the foreground defect;
step 2, as shown in fig. 4, performing multi-scale defect region feature extraction on the data set by using a depth residual error network and a feature pyramid network to obtain a multi-scale feature image; the multi-scale feature images are fused by utilizing a feature pyramid network, so that the loss of small target defects in training is reduced, the defect detection generalization capability is improved, and a high-level feature map with high-level semantics is generated, wherein the high-level features have high-level semantic information and a larger receptive field and are suitable for detecting large objects; generating a shallow feature map with low-level semantics, wherein the shallow feature has low-level detail semantic information and a smaller receptive field and is suitable for detecting small objects; providing a series of possible defect candidate areas in the image by using an area detection network, and then adding the input and the output of the depth residual error network together by using a depth residual error network through jump layer connection to extract the image characteristics of the defect candidate areas; finally, classifying the feature map fused with all the information to complete the target classification and regression task; by means of convolution of a depth residual error network with the highest 152 layers, image features can be better learned, meanwhile, by means of a feature pyramid network, features of the image on different scales are extracted, large-scale position information and small-scale semantic information are combined, and the method is very suitable for a small-area defect detection task on a mobile phone screen; in the high-level feature map, high-level semantics are fused with a feature map with strong low-resolution semantic information, or high-level semantics are fused with a feature map with weak high-resolution semantic information but rich spatial information;
step 3, as shown in fig. 4, extracting a target interest detection region based on the feature image in step 2, aiming at reducing the detection region, improving the detection efficiency and avoiding background information interference:
the method is different from the conventional target detection task which focuses on the whole image for target detection, the mobile phone screen appearance defect detection task only focuses on target detection of a specific area, images of different stations have different detection interest areas, and the proportion of the area of the detection area in the whole image is usually 50% -80%. The defects are defined relative to the screen appearance, and are not absolute standards, so that background factors outside the detection area have great interference, and the accuracy of defect detection can be greatly improved by filtering the background information.
The traditional edge contour detection algorithm can better extract the image area where the screen object is located, but has the following problems:
1) the time consumption is long, the time occupation ratio of image cutting in the whole model detection process is up to 50%, and the bottleneck problem greatly influencing the detection efficiency is solved.
2) The generalization is poor, the image needs to be binarized in the preposing step of the edge contour detection algorithm, the artificially set binarization threshold cannot adapt to different shooting stations and illumination conditions, the self-adaptive binarization algorithm, such as a watershed algorithm, still cannot achieve an ideal effect under certain illumination conditions, and false detection of interference factors in a background area is easily caused.
The interested detection area of the image to be detected shot at different stations is relatively fixed and is limited by the position deviation and the illumination deviation of different degrees under the condition of hardware. The image characteristics are relatively fixed, and the network is easy to detect through large-scale anchoring area detection. Compared with an extraction method based on edge contour detection, the method can greatly improve the detection time efficiency due to the fact that the characteristic images of the preamble steps are shared.
The background information is filtered, so that the accuracy of defect detection can be greatly improved; detecting a target interest detection area of an image to be detected shot at different stations through a large-scale anchored area detection network;
the target interest detection area refers to a mobile phone screen area which is approximately rectangular in the image; as shown in fig. 5, the feature pyramid network is composed of three parts, the first part is up-sampled by a convolutional network, the second part is down-sampled from the feature map, and the third part is transversely connected, and the feature maps of the first two parts with the same size are fused to obtain a multi-scale feature map;
and 4, step 4: as shown in fig. 4, generating potential defect regions to be detected in the target interest detection region by using a region detection network, and determining whether a defect exists in each potential defect region to be detected by combining the feature images in step 2 to obtain a defect type; extracting feature vectors with equal length, judging the probability of each region as a foreground defect image or a background by a softmax function through setting anchoring with different scales by the region detection network, and regressing the positions of candidate regions of the foreground defect image;
step 4.1, counting the sizes and proportions of defect samples in the data set, and setting a targeted anchoring scale; the anchored length-width proportion comprises five types of 0.2, 0.5, 1, 2 and 5, the pixel size comprises five types of 32 pixels, 64 pixels, 128 pixels, 256 pixels and 512 pixels, the defect area can be screened out better, and the regression process of the candidate frame can be accelerated;
step 4.2, combining the characteristic image, performing target interest detection area pooling operation on the defect candidate area, performing normalization processing on the image characteristic, and mapping the normalized image characteristic to probability distribution of defect categories through a softmax function, wherein the target function is as follows:
Figure BDA0003050158810000081
in the above formula, J (theta) is the objective function, N is the number of samples, y is the label class,
Figure BDA0003050158810000082
for prediction classes, θ is the model parameter, R (θ) is L to prevent overfitting2A regular term, wherein lambda is a parameter term set by experience;
extracting feature vectors with equal length in a target interest detection area pooling mode, wherein the maximum value of defect classification probability is a classification category of a detection result;
and 5, outputting a defect detection result.
Example 2:
on the basis of embodiment 1, multi-level convolutional neural network model training and testing are performed by using the data set generated in step 1, the experimental data includes 5422 pieces of real scene labeling data, the experimental evaluation indexes are defect detection accuracy, recall rate and F1 score, the accuracy (precision) indicates how many samples in the predicted result are correct, and the recall (recall) indicates how many positive samples in the predicted result are correctly detected. F1 is defined as follows:
Figure BDA0003050158810000091
the definition of correct detection is that IoU (overlap degree) between the detection area and the label area is 0.5 or more, and the detection type and the label type are consistent. IoU, the ratio of the areas of the intersection and union of the "predicted region" and the "true region" was calculated, and the results of the experiment are shown in Table 1 below.
TABLE 1 Defect Category test result Table
Type of defect Rate of accuracy Recall rate F1
Black spot 97.2％ 93.1％ 0.95
Hair, hair-care product and method for producing the same 96.1％ 97.0％ 0.96
Scratch mark 95.2％ 95.1％ 0.95
Smudge 94.9％ 93.6％ 0.94
Different colors 93.3％ 94.7％ 0.94

Claims (7)

1. A multi-level smart phone screen defect detection method is characterized by comprising the following steps:
step 1, manually marking defects in an image, and then separating a foreground defect image from a background image; on the basis of the labeled foreground defect image, the color and the size of the foreground defect image are enhanced; finally, combining the enhanced foreground defect image and the enhanced background image to generate a data set;
step 1.1, manually marking the position and the category information of a rectangular area where each foreground defect image in a small number of images is located, and then separating the foreground defect image from the background image;
step 1.2, performing image denoising treatment on the foreground defect image: processing image noise by adopting median filtering, setting the gray value of a target point as the median of all points in an area around the target point, and removing independent noise signals;
adopting a Laplace operator to sharpen the noise-reduced image, and enhancing the difference between the gray values of an object and a background in the image:
Figure FDA0003050158800000011
Figure FDA0003050158800000012
Figure FDA0003050158800000013
Figure FDA0003050158800000014
in the above formula, f (x, y) is an image after noise reduction;
Figure FDA0003050158800000015
the image is subjected to Laplace transform; c is an enhancement coefficient; g (x, y) is the enhanced image;
randomly adjusting the chromaticity, the sharpness and the brightness of the foreground defect image, and randomly rotating the foreground defect image by a random angle and randomly zooming in a fixed proportion range;
step 1.3, in the randomly selected background image area, adjusting the pixel value:
Figure FDA0003050158800000016
Figure FDA0003050158800000017
in the above formula, valuenewIs the enhanced pixel value; valueoldThe value range is 0-255 for the original pixel value; brightness represents the brightness ratio and takes a value of 0.9 or 1.1, the brightness value represents 0.9 dark heterochrosis, and the brightness value 1.1 represents bright heterochrosis; the value range of exp is 1.5-3; a and b represent half the width and height of the rectangular background image area, respectively; the factor represents the ratio of the distance from a certain point to the center of the defect and the distance from the boundary of the defect area to the center of the defect; centerxAnd centeryThe horizontal and vertical coordinates of the center of the rectangular background image region are expressed by (center)x，centery) Establishing a rectangular coordinate system for an origin, wherein x and y respectively represent horizontal and vertical coordinates of points in a rectangular background image area;
step 1.4, performing image multi-band fusion on the foreground defect image and the random position of the target interest detection area, constructing a Laplacian pyramid on the foreground defect image and the background image, and fusing each layer:
Figure FDA0003050158800000021
in the above formula, superscript i denotes the ith layer of the laplacian pyramid;
Figure FDA0003050158800000022
features of an i-th layer of the laplacian pyramid representing the output fused image;
Figure FDA0003050158800000023
features of the ith layer of the laplacian pyramid representing the foreground defect image;
Figure FDA0003050158800000024
features of the ith layer of the laplacian pyramid representing a background image; riA fusion region representing the ith layer;
step 2, performing multi-scale defect region feature extraction on the data set by using a depth residual error network and a feature pyramid network to obtain a multi-scale feature image; fusing the multi-scale feature images by using a feature pyramid network to generate a high-level feature map with high-level semantics and a shallow-level feature map with low-level semantics; providing a defect candidate area in the image by using an area detection network, and then adding the input and the output of a depth residual error network together by using the depth residual error network through jump layer connection to extract the image characteristics of the defect candidate area; finally, classifying the feature maps;
step 3, extracting a target interest detection area based on the characteristic image in the step 2: filtering background information; detecting a target interest detection area of an image to be detected shot at different stations through a large-scale anchored area detection network;
and 4, step 4: generating potential defect areas to be detected in the target interest detection area by using an area detection network, and judging whether defects exist in each potential defect area to be detected by combining the characteristic images in the step 2 to obtain the types of the defects; extracting feature vectors with equal length, judging the probability of each region as a foreground defect image or a background by a softmax function through setting anchoring with different scales by the region detection network, and regressing the positions of candidate regions of the foreground defect image;
and 5, outputting a defect detection result.
2. The method for detecting the defects of the multi-level smart phone screen according to claim 1, wherein the step 4 specifically comprises the following steps:
step 4.1, counting the sizes and proportions of defect samples in the data set, and setting an anchoring scale; the anchored aspect ratio includes 0.2, 0.5, 1, 2, and 5, and the pixel size includes 32 pixels, 64 pixels, 128 pixels, 256 pixels, and 512 pixels;
step 4.2, combining the characteristic image, performing target interest detection area pooling operation on the defect candidate area, performing normalization processing on the image characteristic, and mapping the normalized image characteristic to probability distribution of defect categories through a softmax function:
Figure FDA0003050158800000025
in the above formula, J (theta) is the objective function, N is the number of samples, y is the label class,
Figure FDA0003050158800000026
for prediction classes, θ is the model parameter, R (θ) is L to prevent overfitting2A regular term and lambda is a parameter term.
3. The multi-level smart phone screen defect detection method of claim 1, wherein: defects in step 1 include hair, dirt, black spots, scratches, and off-colors.
4. The multi-level smart phone screen defect detection method of claim 1, wherein: and 2, fusing a feature map with stronger low-resolution semantic information by using high-level semantics in the high-level feature map, or fusing a feature map with weaker high-resolution semantic information and rich spatial information by using high-level semantics.
5. The multi-level smart phone screen defect detection method of claim 1, wherein: the target interest detection area in the step 3 refers to a mobile phone screen area in the image.
6. The multi-level smart phone screen defect detection method of claim 1, wherein: and 3, the characteristic pyramid network consists of three parts, wherein the first part is subjected to up-sampling through a convolutional network, the second part is subjected to down-sampling from the characteristic diagram, the third part is transversely connected, and the two parts of characteristic diagrams with the same size are fused to obtain the multi-scale characteristic diagram.
7. The multi-level smart phone screen defect detection method of claim 2, wherein: and 4.2, extracting feature vectors with equal length in a target interest detection area pooling mode, wherein the maximum value of the defect classification probability is the classification category of the detection result.
CN202110489131.0A 2021-04-30 2021-04-30 Multi-level smart phone screen defect detection method Active CN113221881B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110489131.0A CN113221881B (en) 2021-04-30 2021-04-30 Multi-level smart phone screen defect detection method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110489131.0A CN113221881B (en) 2021-04-30 2021-04-30 Multi-level smart phone screen defect detection method

Publications (2)

Publication Number Publication Date
CN113221881A true CN113221881A (en) 2021-08-06
CN113221881B CN113221881B (en) 2022-03-22

Family

ID=77090932

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110489131.0A Active CN113221881B (en) 2021-04-30 2021-04-30 Multi-level smart phone screen defect detection method

Country Status (1)

Country Link
CN (1) CN113221881B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114022657A (en) * 2022-01-06 2022-02-08 高视科技(苏州)有限公司 Screen defect classification method, electronic equipment and storage medium
CN114445483A (en) * 2022-01-28 2022-05-06 泗阳三江橡塑有限公司 Injection molding part quality analysis method based on image pyramid
CN116542975A (en) * 2023-07-05 2023-08-04 成都数之联科技股份有限公司 Defect classification method, device, equipment and medium for glass panel

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030053673A1 (en) * 2001-09-18 2003-03-20 Piet Dewaele Radiographic scoring method
CN107016664A (en) * 2017-01-18 2017-08-04 华侨大学 A kind of bad pin flaw detection method of large circle machine
CN107678192A (en) * 2017-07-16 2018-02-09 中科院成都信息技术股份有限公司 A kind of Mura defects detection method and system based on machine vision
CN110514665A (en) * 2019-09-03 2019-11-29 博科视(苏州)技术有限公司 A kind of detection method of electronic product plastic shell scratch defects
US20200167907A1 (en) * 2017-08-29 2020-05-28 Ping An Technology (Shenzhen) Co., Ltd. Automatic screen state detection robot, method and computer-readable storage medium
CN111553929A (en) * 2020-05-12 2020-08-18 重庆邮电大学 Mobile phone screen defect segmentation method, device and equipment based on converged network
CN112184744A (en) * 2020-11-29 2021-01-05 惠州高视科技有限公司 Display screen edge defect detection method and device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030053673A1 (en) * 2001-09-18 2003-03-20 Piet Dewaele Radiographic scoring method
CN107016664A (en) * 2017-01-18 2017-08-04 华侨大学 A kind of bad pin flaw detection method of large circle machine
CN107678192A (en) * 2017-07-16 2018-02-09 中科院成都信息技术股份有限公司 A kind of Mura defects detection method and system based on machine vision
US20200167907A1 (en) * 2017-08-29 2020-05-28 Ping An Technology (Shenzhen) Co., Ltd. Automatic screen state detection robot, method and computer-readable storage medium
CN110514665A (en) * 2019-09-03 2019-11-29 博科视(苏州)技术有限公司 A kind of detection method of electronic product plastic shell scratch defects
CN111553929A (en) * 2020-05-12 2020-08-18 重庆邮电大学 Mobile phone screen defect segmentation method, device and equipment based on converged network
CN112184744A (en) * 2020-11-29 2021-01-05 惠州高视科技有限公司 Display screen edge defect detection method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
A.F. LIMAS SERAFIM: "Object detection in images of natural scenes represented by AR models using Laplacian pyramids: application to leather defects localization", 《PROCEEDINGS OF THE 1992 INTERNATIONAL CONFERENCE ON INDUSTRIAL ELECTRONICS, CONTROL, INSTRUMENTATION, AND AUTOMATION》 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114022657A (en) * 2022-01-06 2022-02-08 高视科技(苏州)有限公司 Screen defect classification method, electronic equipment and storage medium
CN114022657B (en) * 2022-01-06 2022-05-24 高视科技(苏州)有限公司 Screen defect classification method, electronic equipment and storage medium
CN114445483A (en) * 2022-01-28 2022-05-06 泗阳三江橡塑有限公司 Injection molding part quality analysis method based on image pyramid
CN116542975A (en) * 2023-07-05 2023-08-04 成都数之联科技股份有限公司 Defect classification method, device, equipment and medium for glass panel
CN116542975B (en) * 2023-07-05 2023-09-12 成都数之联科技股份有限公司 Defect classification method, device, equipment and medium for glass panel

Also Published As

Publication number Publication date
CN113221881B (en) 2022-03-22

Similar Documents

Publication Publication Date Title
CN113160192B (en) Visual sense-based snow pressing vehicle appearance defect detection method and device under complex background
CN111310558B (en) Intelligent pavement disease extraction method based on deep learning and image processing method
CN107833220B (en) Fabric defect detection method based on deep convolutional neural network and visual saliency
CN113221881B (en) Multi-level smart phone screen defect detection method
CN108562589B (en) Method for detecting surface defects of magnetic circuit material
CN107230202B (en) Automatic identification method and system for road surface disease image
CN107610114B (en) optical satellite remote sensing image cloud and snow fog detection method based on support vector machine
CN107437243B (en) Tire impurity detection method and device based on X-ray image
CN111915704A (en) Apple hierarchical identification method based on deep learning
CN113592845A (en) Defect detection method and device for battery coating and storage medium
CN113240626B (en) Glass cover plate concave-convex type flaw detection and classification method based on neural network
CN108648168A (en) IC wafer surface defects detection methods
CN108229458A (en) A kind of intelligent flame recognition methods based on motion detection and multi-feature extraction
CN104794502A (en) Image processing and mode recognition technology-based rice blast spore microscopic image recognition method
CN113240623B (en) Pavement disease detection method and device
CN114926407A (en) Steel surface defect detection system based on deep learning
CN106557740A (en) The recognition methods of oil depot target in a kind of remote sensing images
CN115797813B (en) Water environment pollution detection method based on aerial image
CN108509950A (en) Railway contact line pillar number plate based on probability characteristics Weighted Fusion detects method of identification
CN115082776A (en) Electric energy meter automatic detection system and method based on image recognition
CN113537211A (en) Deep learning license plate frame positioning method based on asymmetric IOU
CN114863493A (en) Detection method and detection device for low-quality fingerprint image and non-fingerprint image
CN117975175B (en) Plastic pipeline appearance defect detection method based on machine vision
CN106407975B (en) Multiple dimensioned layering object detection method based on space-optical spectrum structural constraint
Sui et al. ECGAN: An improved conditional generative adversarial network with edge detection to augment limited training data for the classification of remote sensing images with high spatial resolution

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20231127

Address after: Room 521, Unit 2, Building 4, Dexin Yunchuan Business Center, Xihu District, Hangzhou City, Zhejiang Province, China (self declared)

Patentee after: Hangzhou Zhenwei Food Collection Technology Co.,Ltd.

Address before: 310015 No. 51 Huzhou street, Hangzhou, Zhejiang, Gongshu District

Patentee before: HANGZHOU City University

TR01 Transfer of patent right