CN110619620B

CN110619620B - Method, device and system for positioning abnormity causing surface defects and electronic equipment

Info

Publication number: CN110619620B
Application number: CN201810564884.1A
Authority: CN
Inventors: 陈佳伟
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2018-06-04
Filing date: 2018-06-04
Publication date: 2022-04-05
Anticipated expiration: 2038-06-04
Also published as: CN110619620A

Abstract

The embodiment of the invention provides an abnormal positioning method, an abnormal positioning device, an abnormal positioning system and electronic equipment for causing surface defects, wherein the abnormal positioning method for causing the surface defects comprises the following steps: acquiring a defect detection result of an image to be detected, wherein the defect detection result comprises defect statistical information in the image to be detected, and the image to be detected is an image which is shot by image acquisition equipment and comprises a detection object; acquiring an image to be analyzed matched with the defect statistical information; and detecting and analyzing the image to be analyzed, and determining abnormal information associated with the defect statistical information, wherein the abnormal information comprises abnormal position information. By the scheme, the abnormity causing the surface defect can be accurately positioned.

Description

Method, device and system for positioning abnormity causing surface defects and electronic equipment

Technical Field

The invention relates to the technical field of target detection, in particular to an abnormity positioning method, device and system for surface defects and electronic equipment.

Background

The surface defects refer to the defects of spots, pits, chromatic aberration, scratches, defects and the like on the surface of a product, and have the characteristics of various types, variable forms, unfixed positions, diversified background textures and the like. In the industrial field, surface defects directly affect the aesthetic, performance, etc. attributes of the product, and therefore, the surface quality of the product is of great importance.

In order to ensure the surface quality of the product, the surface defects of the product need to be detected. The detection method of the surface defect mainly comprises an artificial detection method and a machine learning method based on deep learning, and the position, type and other information of the defect on the surface of the product can be determined through surface defect detection.

In the conventional surface defect processing method, after information such as the position and type of a surface defect is detected, it is necessary to analyze the cause of the abnormality causing the surface defect by a manual analysis method, estimate position information at which the abnormality may occur from the analyzed cause of the abnormality, and adjust the cause of the abnormality based on the position information to eliminate the abnormality. Because the types and positions of the defects caused by the abnormalities at different positions are probably similar, the defects with similar types or positions are probably identified as the defects at the same position by manual analysis, so that the positioning of the abnormalities is wrong.

Disclosure of Invention

An object of the embodiments of the present invention is to provide a method, an apparatus, a system and an electronic device for locating an abnormality causing a surface defect, so as to accurately locate the abnormality causing the surface defect. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides an anomaly locating method for causing surface defects, where the method includes:

acquiring a defect detection result of an image to be detected, wherein the defect detection result comprises defect statistical information in the image to be detected, and the image to be detected is an image which is shot by image acquisition equipment and comprises a detection object;

acquiring an image to be analyzed matched with the defect statistical information;

and detecting and analyzing the image to be analyzed, and determining abnormal information associated with the defect statistical information, wherein the abnormal information comprises abnormal position information.

Optionally, obtaining the defect detection result of the image to be detected includes:

acquiring an image to be detected;

inputting the image to be detected into a pre-trained deep learning network model to obtain a multi-value diagram corresponding to the image to be detected;

and acquiring defect statistical information of the image to be detected from the multi-value image.

Optionally, the defect statistic information includes: the type of defects, the number of each type of defects, the positions of each type of defects and the area ratio of each type of defects in the image to be detected;

the obtaining of the defect statistical information of the image to be detected from the multi-value image includes:

determining a connected region where each type of defect is located in the multi-value image according to a preset corresponding relation between the pixel value and each type of defect, wherein the pixel values in the connected regions where the same type of defect is located are the same;

for each type of defect, counting the number of connected regions where the type of defect is located, the position of the connected regions where the type of defect is located and the area ratio of the connected regions where the type of defect is located in the multi-value graph;

and determining the defect types, the number of the defects of each type, the positions of the defects of each type and the area ratio of the defects of each type according to the number of the connected regions where the defects of each type are located, the positions of the connected regions where the defects of each type are located and the area ratio of the connected regions where the defects of each type are located in the multi-value image.

Optionally, the acquiring the image to be analyzed matched with the defect statistical information includes:

inputting a synthesized image counted in a database into the deep learning network model to obtain defect statistical information of the synthesized image, wherein the synthesized image is a two-dimensional image obtained by extracting correlation information according to signal data collected by at least one signal collection device and converting the correlation information into the two-dimensional image, and the synthesized image reflects the state of each production device on a production line of the detection object in the production process;

and matching the defect statistical information of the image to be detected with the defect statistical information of the synthetic image, and determining the synthetic image as the image to be analyzed if the defect statistical information of the image to be detected is matched with the defect statistical information of the synthetic image.

Optionally, the detecting and analyzing the image to be analyzed to determine abnormal information associated with the defect statistical information includes:

detecting whether the states of all production equipment on a production line for producing the detection object are abnormal or not according to the image to be analyzed;

determining abnormal production equipment on the production line;

and determining abnormal position information of the abnormal production equipment according to the equipment state information of the abnormal production equipment.

Optionally, before the acquiring the image to be analyzed matched with the defect statistical information, the method further includes:

judging whether the defects in the image to be detected are newly generated defects or not or whether the accumulated occurrence frequency of the defects exceeds a preset threshold value or not according to the defect statistical information of the image to be detected;

and if the defect is a newly generated defect or the accumulated occurrence frequency of the defect exceeds a preset threshold value, executing the acquisition of the image to be analyzed matched with the defect statistical information.

Optionally, after the detecting and analyzing the image to be analyzed and determining the abnormal information associated with the defect statistical information, the method further includes:

determining a generation cause of the abnormal information based on the abnormal information;

and displaying the abnormal information and/or the reason for generating the abnormal information.

In a second aspect, an embodiment of the present invention provides an anomaly locating device for causing surface defects, the device including:

the detection result acquisition module is used for acquiring a defect detection result of an image to be detected, wherein the defect detection result comprises defect statistical information in the image to be detected, and the image to be detected is an image which is shot by image acquisition equipment and comprises a detection object;

the image acquisition module is used for acquiring an image to be analyzed matched with the defect statistical information;

and the analysis module is used for detecting and analyzing the image to be analyzed and determining abnormal information associated with the defect statistical information, wherein the abnormal information comprises abnormal position information.

Optionally, the detection result obtaining module is specifically configured to:

acquiring an image to be detected;

the detection result obtaining module is specifically configured to:

Optionally, the image obtaining module is specifically configured to:

Optionally, the analysis module is specifically configured to:

determining abnormal production equipment on the production line;

Optionally, the apparatus further comprises:

the judging module is used for judging whether the defects in the image to be detected are newly generated defects or not or whether the accumulated occurrence frequency of the defects exceeds a preset threshold value or not according to the defect statistical information of the image to be detected;

the image obtaining module is specifically configured to, if the determination result of the determining module is that the defect is a newly generated defect or the cumulative occurrence frequency of the defect exceeds a preset threshold, perform the obtaining of the image to be analyzed that is matched with the defect statistical information.

Optionally, the apparatus further comprises:

the determining module is used for determining the generation reason of the abnormal information based on the abnormal information;

and the display module is used for displaying the abnormal information and/or the reason for generating the abnormal information.

In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein,

the memory is used for storing a computer program;

the processor is configured to implement the method steps of the first aspect of the embodiment of the present invention when executing the computer program stored in the memory.

In a fourth aspect, an embodiment of the present invention provides an anomaly locating system for locating an anomaly causing a surface defect, the system including an image acquisition device, a plurality of signal acquisition devices, and an electronic device according to the third aspect of the embodiment of the present invention.

According to the method, the device and the system for positioning the surface defect abnormity and the electronic equipment, the defect detection result of the image to be detected is obtained, the image to be analyzed matched with the defect statistical information in the defect detection result is obtained, the image to be analyzed is detected and analyzed, the abnormal information related to the defect statistical information is determined, wherein the abnormal information comprises abnormal position information, and the image to be detected is an image which is shot by image acquisition equipment and comprises a detection object. Because the image to be analyzed can reflect the abnormal information causing the defect, after the defect detection result is obtained through detection, the defect statistical information in the defect detection result is matched with the image to be analyzed, so that the abnormal information related to the defect statistical information can be determined, and the abnormal information is related to the defect statistical information, thereby ensuring that the abnormality causing the surface defect can be accurately positioned.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart illustrating a method for locating anomalies that cause surface defects, in accordance with an embodiment of the present invention;

FIG. 2a is a schematic view of a cloth according to an embodiment of the present invention having a wrong yarn defect;

FIG. 2b is a schematic view of a cloth of an embodiment of the present invention showing a broken needle defect;

FIG. 2c is a schematic diagram of a cloth having a defect of a scutching line according to an embodiment of the present invention;

FIG. 2d is a schematic diagram illustrating a cloth having a hole defect according to an embodiment of the present invention;

FIG. 3 is a schematic flow chart of deep learning model training according to an embodiment of the present invention;

FIG. 4 is a schematic flow chart of a defect online detection method in an industrial plant example according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an anomaly locating device for detecting surface defects according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the invention;

fig. 7 is a schematic structural diagram of an anomaly locating system for causing surface defects according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In order to accurately locate an anomaly causing a surface defect, the embodiment of the invention provides an anomaly locating method, device and system causing a surface defect and electronic equipment.

The terms in the examples of the present invention are explained as follows:

deep learning: the motivation for deep learning is to let the computer perform learning analysis data in a way of simulating the human brain.

A database: the database is a warehouse for organizing, storing and managing data according to a data structure, and is a core part of various information systems such as a management information system, an office automation system, a decision support system and the like.

First, a method for locating an anomaly causing a surface defect according to an embodiment of the present invention will be described.

An execution subject of the method for locating an abnormality causing a surface defect provided by the embodiment of the present invention may be an electronic device, where the electronic device is used to implement functions such as image processing and target identification, and may be an image acquisition device, a remote processor, and the like having a logic processing capability, and the electronic device at least includes a chip capable of completing the logic processing. The method for positioning the abnormality causing the surface defect provided by the embodiment of the invention can be implemented by at least one of software, a hardware circuit and a logic circuit arranged in an execution main body.

As shown in fig. 1, an anomaly locating method for causing surface defects according to an embodiment of the present invention may include the following steps:

s101, obtaining a defect detection result of the image to be detected.

The defect detection result comprises defect statistical information in an image to be detected, and the image to be detected is image data which is shot by image acquisition equipment and comprises a detection object. The detection object mentioned here is an article that needs to be subjected to surface defect detection, for example, an article being produced on a production line, and in order to ensure the surface integrity of the article, the surface defect detection needs to be performed, and the abnormal occurrence position causing the defect on the production line is monitored in real time. Taking cloth as an example, detecting an image to be detected with at least one of defects shown in fig. 2a (wrong yarn), fig. 2b (broken needle), fig. 2c (open line) and fig. 2d (broken hole), and obtaining defect detection results such as the number, position, area ratio and the like of the defects in the image to be detected.

The defect detection result may be obtained by deep learning or by image feature comparison. The acquisition mode of the defect detection result is to judge whether the image to be detected has defects by detecting the characteristics in the image to be detected, and determine the defect detection results such as the number, the position, the type and the like of the defects. The traditional characteristic comparison mode needs to compare the characteristics in the image to be detected with the characteristics in the defect image collected in advance one by one, so that a defect detection result is obtained, and the process is complex to realize. Therefore, with the continuous development of deep learning, the defect detection result is often obtained by adopting a deep learning manner.

Optionally, S101 may specifically be: acquiring an image to be detected; inputting an image to be detected into a pre-trained deep learning network model to obtain a multi-value diagram corresponding to the image to be detected; and acquiring defect statistical information of the image to be detected from the multi-value image.

The image to be detected can be an image stored in a database, and the image acquisition equipment shoots a detection object and stores the detection object into the database; the image to be detected can also be an image obtained by shooting a detection object in real time by image acquisition equipment, the deep learning network model is obtained by pre-training based on the collected original sample image, the confidence coefficient of each pixel in the image to be detected as a defect can be obtained through the deep learning network model, a multi-value image corresponding to the image to be detected can be established based on the confidence coefficient of each pixel, and the mode of establishing the multi-value image can comprise image data filtering, data binarization and the like. The pixel values of the pixels in the multi-value image are a plurality of fixed values, and specifically, the pixel values of the pixels in the multi-value image may be obtained by performing semantic segmentation processing on the image to be detected.

Because surface defect detection is mostly used in the field of industrial production, the size of a shot image is often large, and an area in which surface defects are easy to exist is often a small area. The image to be detected acquired by the image acquisition equipment can be stored in a database and acquired from the database; the image acquisition device can also send the image to be detected acquired in real time to the execution main body of the embodiment. The extraction mode of the region of interest can be preset according to parameters such as the type of the detection object, the historical detection result and the like, and the region of interest is the region needing to be subjected to defect detection in the image to be detected.

The deep learning network model can be a convolutional neural network model, the convolutional neural network is a feed-forward neural network, artificial neurons in the convolutional neural network can respond to peripheral units in a part of coverage range, and the deep learning network model has excellent performance on large-scale image processing. The convolutional neural network generally comprises network layers such as a convolutional layer, a pooling layer, a nonlinear layer, and a full-link layer, and of course, the deep learning model in this embodiment may also be a full-convolutional network, or a combination of a plurality of full-convolutional networks connected in series and in parallel. In the process of detecting the surface defects by using the deep learning model, the detection precision or speed can be adjusted by setting detection parameters, wherein the detection parameters comprise global detection parameters and local detection parameters.

Optionally, the defect statistics may include: the defect type, the number of each type of defect, the position of each type of defect and the area ratio of each type of defect in the image to be detected.

The step of obtaining the defect statistical information of the image to be detected from the multi-value image can be specifically as follows: determining a connected region where each type of defect is located in the multi-value image according to a preset corresponding relation between the pixel value and each type of defect, wherein the pixel values in the connected regions where the same type of defect is located are the same; for each type of defect, counting the number of connected regions where the type of defect is located, the position of the connected regions where the type of defect is located and the area ratio of the connected regions where the type of defect is located in the multi-value graph; and determining the defect type, the number of the defects of each type, the position of the defects of each type and the area ratio of the defects of each type according to the number of the connected regions where the defects of each type are located, the position of the connected regions where the defects of each type are located and the area ratio of the connected regions where the defects of each type are located in the multi-value graph.

In the multi-value map, a region composed of a plurality of continuous pixels having the same pixel value is a connected region, the connected region having the same pixel value represents the same object, and a corresponding relationship between the pixel value and the object is established in advance, for example, for an image of a person riding a motorcycle, in the corresponding multi-value map, the person region is pink, the motorcycle region is dark green, and the background region is black. The method comprises the steps that the corresponding relation between pixel values and various defects is established in advance in a multi-value diagram corresponding to an image to be detected, for example, when the defects are wrong yarns aiming at cloth, the pixel values in the multi-value diagram are specified as the pixel values corresponding to red; when the defect is a broken needle, the pixel value in the multi-value image is specified as the pixel value corresponding to green; when the defect is an open line, the pixel values in the multi-value map are specified as the pixel values corresponding to black, and so on. Of course, in some scenes, there may be only one defect, and the multi-value map may be a binary map, where the defective area has a pixel value of 1, i.e. the color is white; in the defect-free region, the pixel value is 0, i.e., the color is black.

Through statistics of the connected regions in the multi-value image, due to the fact that the pixel values of the connected regions have corresponding relations with various types of defects, defect statistical information of the image to be detected can be obtained correspondingly; the deep learning network model may include classifiers such as an SVM (Support Vector Machine), a regression model softmax, an ANN (Artificial Neural Networks), and the like, and the confidence of which type of defect the defect detected in the image to be detected is can be obtained through the classifiers. Based on the analysis, the obtained defect statistical information at least comprises the defect type, the number of each type of defect, the position of each type of defect and the area ratio of each type of defect in the image to be detected through the deep learning network model.

As shown in fig. 3, a training process of the deep learning network model specifically includes the following steps:

the method comprises the following steps: raw sample images are collected from a field or database, the raw sample images including a defective sample image and a non-defective sample image.

After the image acquisition equipment obtains the original sample images through shooting, the original sample images are stored in the database, so that the original sample images can be obtained from the database; of course, the original sample image may also be collected in real time from the shooting site.

Step two: the area used as a training sample is defined for the collected original sample image, i.e. the pre-processing procedure.

In the field of industrial production, the size of a shot image is often large, the number of samples is limited, and in order to improve the efficiency of surface defect detection and obtain a more accurate deep learning model, an original sample image can be preprocessed, and the preprocessing process can be to extract an image in an area of interest in the original image and to perform enhancement transformation on the original image. The enhancement transformation may be a rotation operation, a flipping operation, a luminance transformation, and the like, where the rotation operation is to rotate each original sample image according to a plurality of preset angles, and the preset angles may be, for example, 15 degrees, 30 degrees, 45 degrees, 60 degrees, 90 degrees, and the like; the turning operation is to turn over each original sample image; the brightness conversion is to adjust the brightness of each original sample image according to a plurality of preset brightnesses. Sample data can be expanded through preprocessing, and the accuracy of a training result is guaranteed.

Step three: and marking information such as the position and the category of the defect on the defect sample image and marking the non-defective sample image as non-defective.

And storing the marked sample images into a database so as to be called in the deep learning model training process.

Step four: and sending the marked sample image into a deep learning network model for training.

The training is subjected to parameter setting according to the characteristics of defects required to be detected on site, and the detection precision can be improved. As described above, parameters such as data enhancement can be set before training: rotation, flipping, or brightness transformation; parameters such as times of multiple times of training can be set in the training process; the output parameters of the model may be set after training.

The training process may be online training using an existing deep learning network platform (e.g., a buffer (Convolutional neural network framework), a Tensorflow for implementing a complex data structure and transmitting the complex data structure to an artificial intelligent neural network for analysis and processing, and the like), or may be online training using a built program framework, which is not limited herein. In the training process, parameter setting can be carried out on training according to the defect characteristics detected as required, and then the detection precision is improved.

Step five: and storing the trained deep learning network model into a local database or transmitting the trained deep learning network model into a detection device for online detection in a network or other forms for online detection.

After the deep learning model is obtained through training, the deep learning model can be stored in a database, and can also be transmitted to a device for online detection in a form of network and the like. The on-line training improves the convenience of deep learning model customization, and compared with the traditional defect detection algorithm research and development, the method reduces the research and development cost and accelerates the research and development speed.

After the defect detection result of the image to be detected is obtained, trigger information for moving the detection object can be generated based on the defect detection result when the detection object has surface defects, and the trigger information is sent to mechanical equipment such as a mechanical arm and a robot, so that the mechanical equipment can move the detection object to a specific area for defect processing. The obtained defect detection result can also be stored in a database so as to carry out statistics, analysis and other processing on the occurrence reason of the defect and the generated defect. The defect detection result may also be presented to the user in an image display manner, for example, the position of the defect, the type of the defect in the defective area, the area ratio of the defect, and the like are marked in the image to be detected, and of course, information such as the severity of the defect, the yield of the production line in the past period, the occurrence ratio of various types of defects, and the like may also be displayed. The user can visually see the defects of the detection object, and the user can determine whether to process the defects.

And S102, acquiring an image to be analyzed matched with the defect statistical information.

The image to be analyzed can be image data stored in a database, the image to be analyzed can reflect abnormal information causing defects, and under an actual application scene, a plurality of signal acquisition devices (including a temperature sensor, an infrared sensor, an oil quantity sensor and the like) are often arranged on a production line and used for monitoring the device state of a generation device on the production line. The signal acquired by the signal acquisition equipment can effectively reflect the state of production equipment, for example, a temperature sensor arranged on an oil nozzle is used for acquiring the temperature of the oil nozzle, if the temperature of the oil nozzle is overhigh, the oil nozzle is abnormal, and the temperature of the ejected oil is overhigh to cause the scalding scar on a detection object; for another example, the infrared sensor mounted on the cutting device is used for acquiring the cutting speed of the cutting device, and if the cutting speed of the cutting device is too high, it indicates that the cutting device is abnormal, which may cause an excessively deep dent to appear on the detected object; for another example, an oil amount sensor mounted on the oil nozzle is used for collecting the oil injection amount of the oil nozzle, and if the oil injection amount of the oil nozzle is too high, it indicates that the oil nozzle is abnormal, which may cause large oil stains on the detection object. Therefore, the signal data acquired by the signal acquisition device can reflect the defect of the detected object in the form of an image.

Optionally, S102 may specifically be: inputting a synthesized image counted in a database into the deep learning network model to obtain defect statistical information of the synthesized image, wherein the synthesized image is a two-dimensional image obtained by extracting correlation information according to signal data acquired by at least one signal acquisition device and converting the correlation information into the two-dimensional image, and the synthesized image reflects the state of each production device on a production line of a detection object in the production process; and matching the defect statistical information of the image to be detected with the defect statistical information of the synthetic image, and determining the synthetic image as the image to be analyzed if the defect statistical information of the image to be detected is matched with the defect statistical information of the synthetic image.

The relationship between the temperature of the oil nozzle and the type of the scar appearing on the detection object, and the relationship between the oil injection quantity of the oil nozzle and the oil stain area are the correlation information. The database stores two-dimensional images converted according to the associated information between all the data acquired by at least one signal acquisition device and the corresponding defects respectively, and the two-dimensional images are composite images. The composite image can reflect the states of all production equipment on a production line of the detection object in the production process, and can directly reflect whether the production equipment has oil leakage, over-temperature, over-speed and other abnormalities.

Because the synthetic image is synthesized based on the acquired data, the difference exists between the synthetic image and the actual synthetic image, but the defect statistical information of the synthetic image can be detected through the deep learning network model. Therefore, the defect statistical information of the image to be detected and the defect statistical information of the synthetic image can be matched, if the defect statistical information is matched, the synthetic image can be determined as the image to be analyzed, the matching process can be to compare the information of the size, the shape, the pixel value and the like of the defect area, a threshold value can be set, and if the information exceeding a certain threshold value is the same, the matching can be considered.

Optionally, before S102, the method may further include: judging whether the defects in the image to be detected are newly generated defects or not or whether the accumulated occurrence frequency of the defects exceeds a preset threshold value or not according to the defect statistical information of the image to be detected; if the defect is a newly generated defect or the cumulative number of occurrences of the defect exceeds a preset threshold, S102 is performed.

If the number of times of occurrence of a certain defect is large or a certain defect is a newly occurring defect, data anomaly detection needs to be triggered, that is, when the above conditions are met, an image to be analyzed which is matched with defect statistical information needs to be obtained to analyze the anomaly causing the defect, and the specific way of obtaining the image to be analyzed is as described above, and is not described herein again.

S103, detecting and analyzing the image to be analyzed, and determining abnormal information related to the defect statistical information.

Wherein the abnormality information includes abnormality position information. The image to be analyzed is acquired, the image to be analyzed can reflect the abnormal equipment state causing the defect, so that the position of the equipment with the abnormality can be positioned, and certainly, the image to be analyzed is constructed based on the data acquired by the signal acquisition equipment, the acquired data of the signal acquisition equipment can be analyzed through the image to be analyzed, and further, the reason causing the defect can be effectively reflected based on the acquired data.

Optionally, S103 may specifically be: detecting whether the states of all production equipment on a production line for producing the detection object are abnormal or not according to the image to be analyzed; determining abnormal production equipment on the production line; and determining abnormal position information of the abnormal production equipment according to the equipment state information of the abnormal production equipment.

The image to be analyzed reflects the states of various production devices on a production line of the detection object in the production process, such as whether the devices are overspeed, over-temperature, oil leakage and the like, so that the abnormal production devices can be determined.

However, in an actual scene, there is a possibility that the defect is not caused by the abnormality of the device acquired by the signal acquisition device, and may occur in the moving process of the detection object, and the signal acquisition device may not be arranged at the positions, but there is a defect in the acquired image to be detected, so that it is necessary to prompt the information of the abnormality causing the defect not to be found by prompting, so as to prompt the troubleshooting by other methods, such as manual troubleshooting.

Optionally, after S103, the method may further include: determining a generation cause of the abnormal information based on the abnormal information; and displaying the abnormal information and/or the abnormal information generation reason.

After the abnormal position information is located, the generation reason of the abnormal information can be determined based on the abnormal information, for example, the 3 rd oil nozzle on the production line is located to be abnormal, the abnormal reason of the oil nozzle can be determined to be oil leakage through analysis, and the problem of oil leakage is oil nozzle breakage and the like. Then, the abnormality information and/or the cause of the abnormality information may be displayed to the user in a display manner, so that the user may quickly locate the abnormality occurrence point and handle the abnormality as soon as possible. When the abnormal position and the abnormal reason are displayed, an alarm can be triggered, and the alarm can be in the modes of buzzing, color warning, voice warning and the like.

By using the embodiment, the defect detection result of the image to be detected is obtained, the image to be analyzed matched with the defect statistical information in the defect detection result is obtained, the image to be analyzed is subjected to detection and analysis, and the abnormal information associated with the defect statistical information is determined, wherein the abnormal information comprises abnormal position information, and the image to be detected is an image shot by the image acquisition equipment and comprises a detection object. Because the image to be analyzed can reflect the abnormal information causing the defect, after the defect detection result is obtained through detection, the defect statistical information in the defect detection result is matched with the image to be analyzed, so that the abnormal information related to the defect statistical information can be determined, and the abnormal information is related to the defect statistical information, thereby ensuring that the abnormality causing the surface defect can be accurately positioned. The acquired data of the signal acquisition equipment is imaged, and the abnormal production equipment is accurately positioned through data matching positioning. The detection method provided by the embodiment is on-line detection, the research and development cost of the detection algorithm is reduced, and the universality of the anomaly detection is improved.

For convenience of understanding, an application scenario of an industrial factory building is taken as an example, and with reference to fig. 4, the defect online detection method according to the embodiment of the present invention is described, including the following steps.

The method comprises the following steps: and acquiring pictures to be detected from the field through an industrial camera and storing the pictures to a database.

Step two: and defining the area needing to be detected for the acquired picture, namely a preprocessing process.

Step three: and acquiring the trained deep learning network model from a local or remote database for detection. In the detection process, parameters such as global detection, local detection and the like are set to adjust the detection precision or speed.

Step four: the detected data is subjected to post-processing such as filtering and binarization of the image data.

Step five: according to the post-processing data, acquiring statistical information such as defect number, defect type, defect position and area ratio of the defect in the current detection picture, and taking the following measures: (1) triggering a mechanical arm and other devices, and moving the detection object to a specific area according to the defect type; (2) storing the post-processed data and the statistical information into a database; (3) presenting the statistical result to a user through modes such as a display screen, and the content which can be acquired by the user through parameter setting comprises the following steps: marking a picture of the defect position; the defective area displays information such as the type of defect, the area of defect, and the severity of defect; the yield of a certain production line and the proportion of various defect types in the past period.

Step six: and (4) digging out the reasons which are easy to cause specific defects by using a data mining module, and giving an alarm. Different types of sensors, such as temperature sensors, infrared sensors and the like, are arranged on different production devices of the whole plant. The data collected by these sensors is stored in a local database or transmitted to a core database where the data is integrated in a graph according to correlations. And when a new defect occurs or the occurrence rate of a certain defect exceeds a human set threshold value, triggering the database to perform data anomaly detection. The detection process is as follows: the method comprises the steps of mining data which are most matched with defects in the whole database by utilizing a deep learning network model, searching data collected by corresponding sensors by taking the mined data as a starting point according to a graph retrieval mode, detecting whether the obtained data are abnormal or not, displaying the positions and abnormal reasons of the corresponding sensors if the obtained data are abnormal, and triggering an alarm; otherwise, the reason is not found, and the alarm is prompted and manual troubleshooting is carried out.

By applying the embodiment, the defect detection result of the image to be detected is obtained, the image to be analyzed matched with the defect statistical information in the defect detection result is obtained, the image to be analyzed is detected and analyzed, and the abnormal information related to the defect statistical information is determined, wherein the abnormal information comprises abnormal position information, and the image to be detected is an image shot by the image acquisition equipment and comprises a detection object. Because the image to be analyzed can reflect the abnormal information causing the defect, after the defect detection result is obtained through detection, the defect statistical information in the defect detection result is matched with the image to be analyzed, so that the abnormal information related to the defect statistical information can be determined, and the abnormal information is related to the defect statistical information, thereby ensuring that the abnormality causing the surface defect can be accurately positioned.

Corresponding to the above method embodiment, an embodiment of the present invention provides an anomaly locating device for causing a surface defect, as shown in fig. 5, where the anomaly locating device for causing a surface defect may include:

the detection result obtaining module 510 is configured to obtain a defect detection result of an image to be detected, where the defect detection result includes defect statistics information in the image to be detected, and the image to be detected includes an image of a detection object captured by the image capturing device.

And an image obtaining module 520, configured to obtain an image to be analyzed that is matched with the defect statistical information.

An analyzing module 530, configured to perform detection analysis on the image to be analyzed, and determine abnormal information associated with the defect statistical information, where the abnormal information includes abnormal position information.

Optionally, the detection result obtaining module 510 may be specifically configured to: acquiring an image to be detected; inputting the image to be detected into a pre-trained deep learning network model to obtain a multi-value diagram corresponding to the image to be detected; and acquiring defect statistical information of the image to be detected from the multi-value image.

Optionally, the defect statistic information may include: the defect type, the number of each type of defect, the position of each type of defect and the area ratio of each type of defect in the image to be detected.

The detection result obtaining module 510 may be specifically configured to: determining a connected region where each type of defect is located in the multi-value image according to a preset corresponding relation between the pixel value and each type of defect, wherein the pixel values in the connected regions where the same type of defect is located are the same; for each type of defect, counting the number of connected regions where the type of defect is located, the position of the connected regions where the type of defect is located and the area ratio of the connected regions where the type of defect is located in the multi-value graph; and determining the defect types, the number of the defects of each type, the positions of the defects of each type and the area ratio of the defects of each type according to the number of the connected regions where the defects of each type are located, the positions of the connected regions where the defects of each type are located and the area ratio of the connected regions where the defects of each type are located in the multi-value image.

Optionally, the image obtaining module 520 may be specifically configured to: inputting a synthesized image counted in a database into the deep learning network model to obtain defect statistical information of the synthesized image, wherein the synthesized image is a two-dimensional image obtained by extracting correlation information according to signal data collected by at least one signal collection device and converting the correlation information into the two-dimensional image, and the synthesized image reflects the state of each production device on a production line of the detection object in the production process; and matching the defect statistical information of the image to be detected with the defect statistical information of the synthetic image, and determining the synthetic image as the image to be analyzed if the defect statistical information of the image to be detected is matched with the defect statistical information of the synthetic image.

Optionally, the analysis module 530 may be specifically configured to: detecting whether the states of all production equipment on a production line for producing the detection object are abnormal or not according to the image to be analyzed; determining abnormal production equipment on the production line; and determining abnormal position information of the abnormal production equipment according to the equipment state information of the abnormal production equipment.

Optionally, the apparatus may further include: and the judging module is used for judging whether the defects in the image to be detected are newly generated defects or not or whether the accumulated occurrence frequency of the defects exceeds a preset threshold value or not according to the defect statistical information of the image to be detected.

Optionally, the apparatus may further include: the determining module is used for determining the generation reason of the abnormal information based on the abnormal information; and the display module is used for displaying the abnormal information and/or the reason for generating the abnormal information.

By using the embodiment, the defect detection result of the image to be detected is obtained, the image to be analyzed matched with the defect statistical information in the defect detection result is obtained, the image to be analyzed is subjected to detection and analysis, and the abnormal information associated with the defect statistical information is determined, wherein the abnormal information comprises abnormal position information, and the image to be detected is an image shot by the image acquisition equipment and comprises a detection object. Because the image to be analyzed can reflect the abnormal information causing the defect, after the defect detection result is obtained through detection, the defect statistical information in the defect detection result is matched with the image to be analyzed, so that the abnormal information related to the defect statistical information can be determined, and the abnormal information is related to the defect statistical information, thereby ensuring that the abnormality causing the surface defect can be accurately positioned.

In accordance with the above method embodiment, the present invention provides an electronic device, as shown in fig. 6, which includes a processor 601 and a memory 602, wherein,

the memory 602 is used for storing computer programs;

the processor 601 is configured to implement any step of the above-mentioned method for locating an abnormality causing a surface defect when executing the computer program stored in the memory 602.

The Memory may include a RAM (Random Access Memory) or an NVM (Non-Volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a GPU (Graphics Processing Unit), a CPU (Central Processing Unit), an NP (Network Processor), and the like; but also a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.

The electronic device may further include an image capturing unit, such as a video camera, a still camera, etc., for capturing an image of the product; the electronic equipment can also comprise a display for displaying information such as surface defects, abnormal positions, abnormal reasons and the like; the electronic equipment can also comprise a core database for storing sample images, sensing images, deep learning models and the like.

In this embodiment, the processor of the electronic device can read the computer program stored in the memory and run the computer program to implement: the image to be analyzed is detected and analyzed by acquiring a defect detection result of the image to be detected and acquiring an image to be analyzed matched with defect statistical information in the defect detection result, and abnormal information associated with the defect statistical information is determined, wherein the abnormal information comprises abnormal position information, and the image to be detected is an image shot by image acquisition equipment and comprises a detection object. Because the image to be analyzed can reflect the abnormal information causing the defect, after the defect detection result is obtained through detection, the defect statistical information in the defect detection result is matched with the image to be analyzed, so that the abnormal information related to the defect statistical information can be determined, and the abnormal information is related to the defect statistical information, thereby ensuring that the abnormality causing the surface defect can be accurately positioned.

In addition, corresponding to the method for locating an abnormality causing a surface defect provided in the above embodiments, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements any step of the above method for locating an abnormality causing a surface defect.

In this embodiment, when running, the computer-readable storage medium executes the application program of the method for locating an anomaly that causes a surface defect provided in the embodiment of the present invention, so that the following can be implemented: the image to be analyzed is detected and analyzed by acquiring a defect detection result of the image to be detected and acquiring an image to be analyzed matched with defect statistical information in the defect detection result, and abnormal information associated with the defect statistical information is determined, wherein the abnormal information comprises abnormal position information, and the image to be detected is an image shot by image acquisition equipment and comprises a detection object. Because the image to be analyzed can reflect the abnormal information causing the defect, after the defect detection result is obtained through detection, the defect statistical information in the defect detection result is matched with the image to be analyzed, so that the abnormal information related to the defect statistical information can be determined, and the abnormal information is related to the defect statistical information, thereby ensuring that the abnormality causing the surface defect can be accurately positioned.

For the embodiments of the electronic device and the computer-readable storage medium, since the contents of the related methods are substantially similar to those of the foregoing embodiments of the methods, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the embodiments of the methods.

Corresponding to the above embodiments, an embodiment of the present invention further provides an anomaly locating system for surface defects, as shown in fig. 7, where the anomaly locating system for surface defects may include:

the image acquisition equipment 710 is used for shooting a detection object to obtain an image to be detected;

a plurality of signal acquisition devices 720 for acquiring signal data of each production device on the production line;

the electronic device 730 is used for acquiring a defect detection result of an image to be detected, wherein the defect detection result comprises defect statistical information in the image to be detected, and the image to be detected is an image which is shot by the image acquisition device and comprises a detection object; acquiring an image to be analyzed matched with the defect statistical information; and detecting and analyzing the image to be analyzed, and determining abnormal information associated with the defect statistical information, wherein the abnormal information comprises abnormal position information.

Optionally, when the electronic device 730 is used to obtain a defect detection result of an image to be detected, the electronic device may be specifically used to: acquiring an image to be detected; inputting the image to be detected into a pre-trained deep learning network model to obtain a multi-value diagram corresponding to the image to be detected; and acquiring defect statistical information of the image to be detected from the multi-value image.

the electronic device 730, when being configured to obtain the defect statistical information of the image to be detected from the multi-value map, may be specifically configured to: determining a connected region where each type of defect is located in the multi-value image according to a preset corresponding relation between the pixel value and each type of defect, wherein the pixel values in the connected regions where the same type of defect is located are the same; for each type of defect, counting the number of connected regions where the type of defect is located, the position of the connected regions where the type of defect is located and the area ratio of the connected regions where the type of defect is located in the multi-value graph; and determining the defect types, the number of the defects of each type, the positions of the defects of each type and the area ratio of the defects of each type according to the number of the connected regions where the defects of each type are located, the positions of the connected regions where the defects of each type are located and the area ratio of the connected regions where the defects of each type are located in the multi-value image.

Optionally, when the electronic device 730 is configured to obtain the image to be analyzed that is matched with the defect statistical information, the electronic device may specifically be configured to: inputting a synthesized image counted in a database into the deep learning network model to obtain defect statistical information of the synthesized image, wherein the synthesized image is a two-dimensional image obtained by extracting correlation information according to signal data collected by at least one signal collection device and converting the correlation information into the two-dimensional image, and the synthesized image reflects the state of each production device on a production line of the detection object in the production process; and matching the defect statistical information of the image to be detected with the defect statistical information of the synthetic image, and determining the synthetic image as the image to be analyzed if the defect statistical information of the image to be detected is matched with the defect statistical information of the synthetic image.

Optionally, the electronic device 730, when configured to perform detection analysis on the image to be analyzed and determine the abnormal information associated with the defect statistical information, may be specifically configured to: detecting whether the states of all production equipment on a production line for producing the detection object are abnormal or not according to the image to be analyzed; determining abnormal production equipment on the production line; and determining abnormal position information of the abnormal production equipment according to the equipment state information of the abnormal production equipment.

Optionally, the electronic device 730 may further be configured to: judging whether the defects in the image to be detected are newly generated defects or not or whether the accumulated occurrence frequency of the defects exceeds a preset threshold value or not according to the defect statistical information of the image to be detected; and if the defects are newly generated defects or the accumulated occurrence frequency of the defects exceeds a preset threshold value, acquiring the image to be analyzed matched with the defect statistical information.

Optionally, the electronic device 730 may further be configured to: determining a generation cause of the abnormal information based on the abnormal information; and displaying the abnormal information and/or the reason for generating the abnormal information.

Optionally, the electronic device 730 may include a core database for storing the image to be detected acquired by the image acquisition device, the signal data acquired by each signal acquisition device, the deep learning network model, and the defect statistical information.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, the electronic device, the computer-readable storage medium, and the system embodiment, since they are substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method of locating anomalies that cause surface defects, the method comprising:

acquiring an image to be analyzed matched with the defect statistical information, wherein the image to be analyzed is as follows: the image is obtained by matching defect statistical information in a synthesized image counted in a database with the defect statistical information of the image to be detected, the synthesized image is a two-dimensional image obtained by extracting correlation information according to signal data acquired by at least one signal acquisition device and converting the correlation information into the two-dimensional image, and the synthesized image reflects the state of each production device on a production line of the detection object in the production process;

detecting and analyzing the image to be analyzed to determine abnormal information associated with the defect statistical information, wherein the abnormal information comprises abnormal position information, and the detecting and analyzing the image to be analyzed to determine the abnormal information associated with the defect statistical information comprises: detecting whether the states of all production equipment on a production line for producing the detection object are abnormal or not according to the image to be analyzed; determining abnormal production equipment on the production line; and determining abnormal position information of the abnormal production equipment according to the equipment state information of the abnormal production equipment.

2. The method according to claim 1, wherein the obtaining of the defect detection result of the image to be detected comprises:

acquiring an image to be detected;

3. The method of claim 2, wherein the defect statistics comprise: the type of defects, the number of each type of defects, the positions of each type of defects and the area ratio of each type of defects in the image to be detected;

4. The method of claim 2, wherein the obtaining of the image to be analyzed that matches the defect statistics comprises:

5. The method of claim 1, wherein prior to said obtaining an image to be analyzed that matches said defect statistics, said method further comprises:

6. The method according to any one of claims 1 to 5, wherein after the performing detection analysis on the image to be analyzed and determining abnormal information associated with the defect statistical information, the method further comprises:

7. An anomaly locating device that causes surface defects, said device comprising:

an image obtaining module, configured to obtain an image to be analyzed that is matched with the defect statistical information, where the image to be analyzed is: the image is obtained by matching defect statistical information in a synthesized image counted in a database with the defect statistical information of the image to be detected, the synthesized image is a two-dimensional image obtained by extracting correlation information according to signal data acquired by at least one signal acquisition device and converting the correlation information into the two-dimensional image, and the synthesized image reflects the state of each production device on a production line of the detection object in the production process;

an analysis module, configured to perform detection analysis on the image to be analyzed, and determine abnormal information associated with the defect statistical information, where the abnormal information includes abnormal position information, and is specifically configured to: detecting whether the states of all production equipment on a production line for producing the detection object are abnormal or not according to the image to be analyzed; determining abnormal production equipment on the production line; and determining abnormal position information of the abnormal production equipment according to the equipment state information of the abnormal production equipment.

8. The apparatus according to claim 7, wherein the detection result obtaining module is specifically configured to:

acquiring an image to be detected;

9. The apparatus of claim 8, wherein the defect statistics comprise: the type of defects, the number of each type of defects, the positions of each type of defects and the area ratio of each type of defects in the image to be detected;

the detection result obtaining module is specifically configured to:

10. The apparatus of claim 8, wherein the image acquisition module is specifically configured to:

11. The apparatus of claim 7, further comprising:

12. The apparatus of any one of claims 7 to 11, further comprising:

13. An electronic device comprising a processor and a memory, wherein,

the memory is used for storing a computer program;

the processor, when executing the computer program stored in the memory, is configured to perform the method steps of any of claims 1-6.

14. An anomaly localization system causing surface defects, said system comprising an image acquisition device, a plurality of signal acquisition devices, and an electronic device as claimed in claim 13.