CN114429441A

CN114429441A - Abnormity detection method, abnormity detection device, abnormity detection equipment and storage medium

Info

Publication number: CN114429441A
Application number: CN202011101232.8A
Authority: CN
Inventors: 刘昊淼; 马立永; 金鑫; 涂丹丹
Original assignee: Huawei Cloud Computing Technologies Co Ltd
Current assignee: Huawei Cloud Computing Technologies Co Ltd
Priority date: 2020-10-15
Filing date: 2020-10-15
Publication date: 2022-05-03

Abstract

The application provides an anomaly detection method, which comprises the following steps: acquiring an image to be detected; and determining a detection result of the image to be detected according to the image to be detected and an abnormality detection model, wherein the abnormality detection model is obtained by training according to a first loss function comprising sample weight parameters, and the sample weight parameters represent the importance of normal samples or abnormal samples for training on model training. According to the anomaly detection method, the anomaly of the image to be detected is predicted by the anomaly detection model obtained by training the first loss function comprising the sample weight parameters, so that the problem caused by unbalanced training sample data is effectively solved, and the detection result is more accurate.

Description

Abnormity detection method, abnormity detection device, abnormity detection equipment and storage medium

Technical Field

The present application relates to the field of Artificial Intelligence (AI), and in particular, to an anomaly detection method and apparatus, a device, and a storage medium.

Background

The method for detecting the abnormal conditions has very important significance in various fields such as industrial manufacturing, system routine inspection, monitoring video analysis and the like. For example, in a railway freight scene, in order to ensure safe operation of a train, a large number of images need to be acquired for the train, and the images are analyzed in a manual mode to detect abnormal situations; in a power transmission line inspection scene, in order to ensure normal work of a power transmission line, equipment such as an unmanned aerial vehicle is required to be used for acquiring line images, and lines and related components are detected in a manual mode; in an industrial manufacturing scene, surface quality detection is carried out before products leave a factory, the quality of products leaving the factory can be guaranteed, and customer complaints are reduced.

However, the manual abnormality detection consumes a lot of manpower, which increases the working cost of related enterprises and departments. On the other hand, the work content is boring, and a worker is likely to be fatigued when performing abnormality detection, thereby causing such situations as missing detection, erroneous detection, and the like. In recent years, with the rise of deep learning, artificial intelligence technology has been widely applied in daily life and production practice. The artificial intelligence technology is used for replacing manpower to carry out abnormity detection, so that on one hand, the manpower is saved, and the working cost of related enterprises and departments is reduced; on the other hand, the machine algorithm can not generate fatigue, so that the detection effect is more stable.

In actual production practice, the number of the collected normal samples and the number of the collected abnormal samples are often unbalanced (the number of the normal samples is often large, and the number of the abnormal samples is often small), so the training sample data used for training the machine algorithm model is unbalanced. An anomaly detection model trained by the existing machine learning algorithm model often has the problems that the model training process is unstable, the model training difficulty is extremely high, and the anomaly detection performance of the trained model is difficult to meet the requirements due to the fact that training sample data is unbalanced.

Content of application

The embodiment of the application provides an anomaly detection method and device, and solves the problems that an anomaly detection model is unstable in a model training process due to the fact that training sample data are unbalanced, model training difficulty is high, and the anomaly detection performance of the trained model is difficult to meet requirements.

In a first aspect, the present application provides an anomaly detection method, including first obtaining an image to be detected; and then determining a detection result of the image to be detected according to the image to be detected and an anomaly detection model, wherein the anomaly detection model is obtained by training according to a first loss function comprising sample weight parameters, and the sample weight parameters represent the importance of normal samples or abnormal samples for training on model training.

According to the anomaly detection method, the anomaly of the image to be detected is predicted by the anomaly detection model obtained by training the first loss function comprising the sample weight parameters, so that the problem caused by unbalanced training sample data is effectively solved, and a more accurate detection result can be obtained.

In one possible implementation, in order to facilitate the user to perform the review of the model detection result in the subsequent step, the method further includes: generating a detection result distribution map according to the detection result, wherein the detection result distribution map comprises the distribution of images with normal, abnormal or suspected detection results; and presenting the detection result distribution graph to a user.

In one possible implementation, the test result distribution map further includes a confidence region and an in-doubt region; and providing the images distributed in the in-doubt area for the user to manually verify the in-doubt images, so as to prevent the false detection of the detection model and further guarantee the accuracy of the detection result.

In another possible implementation, in consideration of the improvement of the accuracy of the detection result of the anomaly detection model in the subsequent training optimization process and the workload of the manual verification, the range of the in-doubt area in the detection result distribution map is set to be adjustable based on the instruction of the user, that is, the user can reduce the range of the in-doubt area according to the actual situation in the using process to reduce the workload of the manual verification, or increase the range of the in-doubt area to increase the accuracy of the detection result.

In another possible implementation, the sample weight parameter is determined according to the cluster analysis result of the normal sample set and the abnormal sample set in the training sample data, that is, the abnormal detection model can be dynamically adjusted according to the increase/decrease of the normal sample or the abnormal sample in the training sample data, and the updating efficiency of the abnormal detection model is improved without the participation of an algorithm expert.

In another possible implementation, the anomaly detection model is obtained by training based on a second loss function, where the second loss function is used to constrain the structure of the feature space of the anomaly detection model, so that in the feature space, the distance between the features of the normal samples is shortened, the distance between the features of the anomaly samples and the features of the normal samples is lengthened, the accuracy of the output detection result of the anomaly detection model is improved, and the generation of the detection result distribution diagram is facilitated.

In another possible implementation, the training sample data of the anomaly detection model includes anomaly samples obtained through augmentation, and the augmentation method includes: firstly, acquiring a plurality of normal samples and at least one abnormal sample; then comparing the normal sample with the abnormal sample, and determining an abnormal area of the abnormal sample based on a comparison result; respectively extracting the features of the normal sample and the features of the abnormal sample to obtain a first feature map representing the normal sample and a second feature map representing the abnormal sample; replacing the characteristics of the area corresponding to the abnormal area in the first characteristic diagram with the characteristics of the abnormal area in the second characteristic diagram to obtain a first fusion characteristic diagram; and obtaining an expanded abnormal sample based on the first fusion feature map and the image generation model, enriching the types of the abnormal sample data, and increasing the number of the abnormal samples to balance the number of the abnormal samples and the number of the normal samples.

In another possible implementation, the above-mentioned augmentation method may further include: acquiring a plurality of normal samples, at least one abnormal sample and at least one abnormal feature; comparing the normal sample with the abnormal sample, and determining a replaceable area in the normal sample based on a comparison result; determining a replacement area within the replaceable area based on a user instruction; extracting the characteristics of the normal sample to obtain a third characteristic diagram representing the normal sample; replacing the characteristics of the replacement area in the normal sample with the abnormal characteristics to obtain a second fusion characteristic diagram; and obtaining an augmented abnormal sample based on the second fusion feature map and the image generation model.

In another possible implementation, the above anomaly detection method further includes: and selecting the image with the abnormal detection result output by the artificial verification result and/or the abnormal detection model as an abnormal sample for optimization training, and performing optimization training on the abnormal detection model by using the selected abnormal sample, thereby continuously improving the detection precision of the abnormal detection model and expanding the number of types of the abnormal samples which can be covered by the abnormal detection model.

In a second aspect, the present application also provides an abnormality detection apparatus, including:

the acquisition module is used for acquiring an image to be detected;

and the detection module is used for determining the detection result of the image to be detected according to the image to be detected and the abnormal detection model, wherein the abnormal detection model is obtained by training according to a first loss function comprising a sample weight parameter, and the sample weight parameter represents the importance of a normal sample or an abnormal sample used for training on model training.

In another possible implementation, the apparatus further includes:

the visualization module is used for generating a detection result distribution map according to the detection result, wherein the detection result distribution map comprises the distribution of images with normal, abnormal or suspected detection results;

and presenting the detection result distribution graph to a user.

In another possible implementation, the detection result distribution map further includes a confidence region and an in-doubt region;

the visualization module is further used for providing the images distributed in the in-doubt area for the user, so that the user can manually check the in-doubt images.

In another possible implementation, the apparatus further includes:

and the adjusting module is used for adjusting the range of the in-doubt area in the detection result distribution graph based on the instruction of the user.

In another possible implementation, the sample weight parameter is determined according to a cluster analysis result of a normal sample set and an abnormal sample set in the training sample data.

In another possible implementation, the anomaly detection model is obtained by training based on a second loss function, where the second loss function is used to constrain the structure of the feature space of the anomaly detection model, so that in the feature space, the distance between the features of the normal samples is shortened, and the distance between the features of the anomaly samples and the features of the normal samples is lengthened.

In another possible implementation, the training sample data of the anomaly detection model includes anomaly samples obtained through augmentation, and the augmentation method includes:

acquiring a plurality of normal samples and at least one abnormal sample;

comparing the normal sample with the abnormal sample, and determining an abnormal area of the abnormal sample based on a comparison result;

respectively extracting the features of the normal sample and the features of the abnormal sample to obtain a first feature map representing the normal sample and a second feature map representing the abnormal sample;

replacing the characteristics of the area corresponding to the abnormal area in the first characteristic diagram with the characteristics of the abnormal area in the second characteristic diagram to obtain a first fusion characteristic diagram;

and obtaining an augmented abnormal sample based on the first fusion feature map and the image generation model.

acquiring a plurality of normal samples, at least one abnormal sample and at least one abnormal feature;

comparing the normal sample with the abnormal sample, and determining a replaceable area in the normal sample based on a comparison result;

determining a replacement area within the replaceable area based on a user instruction;

extracting the characteristics of the normal sample to obtain a third characteristic diagram representing the normal sample;

replacing the characteristics of the replacement area in the normal sample with the abnormal characteristics to obtain a second fusion characteristic diagram;

and obtaining an augmented abnormal sample based on the second fusion feature map and the image generation model.

In another possible implementation, the apparatus further includes:

and the optimization module is used for selecting the image with the abnormal artificial verification result as an abnormal sample for optimization training and performing optimization training on the abnormal detection model by using the selected abnormal sample.

In a third aspect, the present application further provides an anomaly detection device, including a memory and a processor, where the memory stores executable codes, and the processor executes the executable codes to implement the method described in the first aspect or any one of the possible implementation manners of the first aspect.

In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect or any one of the possible implementations of the first aspect.

In a fifth aspect, the present application also provides a computer program or a computer program product, characterized in that the computer program or the computer program product comprises instructions that, when executed, implement the method described in the first aspect or any one of the possible implementation manners of the first aspect.

The present application can further combine to provide more implementations on the basis of the implementations provided by the above aspects.

Drawings

Fig. 1 is a schematic diagram of an architecture diagram of an anomaly detection apparatus, a training process and an inference process according to an embodiment of the present application;

fig. 2 is a schematic diagram of an architecture of an abnormal sample amplifying module according to the present embodiment;

fig. 3 is a flowchart of an abnormal sample augmentation method according to an embodiment of the present application;

fig. 4 is a flowchart of another abnormal sample augmentation method provided in the application example provided in the present application;

fig. 5 is a schematic diagram of a process for constructing an anomaly detection model according to an embodiment of the present application;

fig. 6 is a flowchart of a method for constructing an anomaly detection model according to an embodiment of the present application;

FIG. 7 is a schematic diagram of positioning a part of an image to be detected in a railway freight anomaly detection scene according to an embodiment of the present application;

fig. 8 is a detection result visualization diagram of an image to be detected in a railway freight anomaly detection scene provided in the embodiment of the present application;

fig. 9 is a visual diagram of a detection result of an image to be detected in a railway freight anomaly detection scene provided in the embodiment of the present application when the detection result includes an in-doubt result;

FIG. 10 is a schematic diagram of inference of a component detection model in an electronic device assembly detection scenario according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an abnormality detection apparatus according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of an abnormality detection apparatus according to an embodiment of the present application.

Detailed Description

The application provides an anomaly detection method and device, which can generate more vivid anomaly samples to enrich training data through a weakly supervised data generation technology under the condition that the number of the anomaly samples is limited. On the basis, a data self-adaptive anomaly detection method is designed, so that the manual participation in the algorithm model training and updating process can be reduced as much as possible, the automation degree is improved, and the cost is reduced. In the actual use process, through man-machine cooperation, the abnormal detection precision is improved while the manual participation is reduced, the model is continuously updated in an iteration mode, and the advantages of the method and the device are continuously enhanced.

Embodiments of the present application are described below with reference to the accompanying drawings. As can be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solution provided in the embodiments of the present application is also applicable to similar technical problems.

The terms "first," "second," and the like in the description and in the claims of the present application and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely descriptive of the various embodiments of the application and how objects of the same nature can be distinguished. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

It is to be understood that the anomaly detection method provided by the present application can be applied to any processing device having image processing capability. The Processing device may be a terminal having a Central Processing Unit (CPU) and/or a Graphics Processing Unit (GPU), wherein the terminal includes, but is not limited to, a Personal Computer (PC), a workstation, and the like. The processing device may also be a server with a CPU and/or a GPU, which may be independent or a cluster of servers. In some cases, the terminal and the server may also implement the above-mentioned abnormality detection method in a cooperative manner.

Fig. 1 is an architecture diagram of an abnormality detection apparatus according to an embodiment of the present application. As shown in fig. 1, the abnormality detection apparatus includes at least an abnormality sample amplification module 11 and an abnormality detection module 13. The abnormal sample augmentation module 11 is configured to obtain a large number of augmented abnormal samples according to a large number of collected normal samples and a small number of collected abnormal samples, enrich the types of the abnormal samples, balance the number of the abnormal samples and the number of the normal samples preliminarily, form training sample data by the large number of normal samples, the small number of abnormal samples and the large number of augmented abnormal samples, and train the deep learning network model by the training sample data to obtain a qualified abnormal detection model. The anomaly detection module 13 is configured to detect whether an input image to be detected is abnormal, and feed back an image with an abnormal detection result to the training sample data to train and update the anomaly detection model, so as to continuously improve the detection accuracy of the anomaly detection model and expand the number of types of the anomaly samples that can be covered by the anomaly detection model.

In some embodiments, the anomaly detection apparatus further includes a manual screening module 12. The manual screening module 12 is configured to determine an available abnormal sample from a large number of augmented abnormal samples according to a user instruction, and prevent a low-quality sample in the augmented abnormal sample from affecting a training effect.

It can be understood that the normal sample is an image of the object to be detected in the normal state (for example, a train without a fault or a normally assembled workpiece) actually captured, and the abnormal sample is an image of the object to be detected in the abnormal state (for example, a train with a fault or a workpiece assembled in error) actually captured; the available abnormal samples are images which are selected by industry experts or ordinary people according to prior knowledge and have high picture definition and rich characteristic changes such as abnormal region shapes and colors from a large number of expanded abnormal samples, and the normal samples, the abnormal samples and the expanded abnormal samples are respectively called normal sample images, abnormal sample images and expanded abnormal sample images for convenience of description.

In other embodiments, the anomaly detection apparatus further comprises an interactive sample visualization module 14 and a manual review module 15. The interactive sample visualization module 14 performs visualization processing on the detection result to display the detection result for the convenience of the user to check and recheck, then the manual recheck module 15 outputs the final detection result and determines that the abnormal sample image which is missed and false is detected and feeds back to the abnormal sample amplification module 11 according to the received instruction, the abnormal sample amplification module continues to generate more amplified abnormal sample images according to the fed-back abnormal sample image and the fed-back normal sample image, the manual screening module 12 screens the amplified abnormal sample image to obtain an available abnormal sample image, and the screened available abnormal sample image is supplemented to the training sample data to continue to train and update the abnormal detection model, so that the detection precision of the abnormal detection model is continuously improved in a reciprocating iteration manner and the number of the types of the abnormal sample which can be covered by the abnormal detection model is expanded.

The working principle and the specific structure of the abnormal sample amplification module are described below.

FIG. 2 is a schematic diagram of an abnormal sample amplifying module. As shown in fig. 2, the abnormal sample augmentation module at least includes an image feature extraction unit 111, an image feature fusion unit 112, an image generation unit 113, and an image reality discrimination unit 114.

The image sample augmentation module firstly determines an abnormal area of an abnormal sample image, then the image feature extraction unit 111 respectively extracts features of a normal sample image and features of the abnormal sample image, then the extracted features are sent to the image feature fusion unit 112, and the image feature fusion unit 112 replaces image features of a corresponding area in the normal sample image with image features of the abnormal area in the abnormal sample image to obtain fused image features; the fused image features are input to the image generation unit 113, and the image generation unit 113 outputs the generated image of the abnormal sample by using the image generation model therein.

The operation of the abnormal sample amplification module will be described in detail with reference to fig. 3.

Fig. 3 is a flowchart of an abnormal sample augmentation method according to an embodiment of the present disclosure.

The abnormal sample augmenting module acquires a plurality of normal sample images and at least one abnormal sample image in step S301. As can be easily understood, the normal sample image is an image obtained by the camera terminal (e.g., a camera) by shooting the object to be detected in the normal state, and the abnormal sample image is an image obtained by the camera terminal by shooting the object to be detected in the abnormal state. The abnormal sample amplification module may obtain the normal sample image and the abnormal sample image from a photographing terminal, or obtain the normal sample image and the abnormal sample image from a terminal in which the normal sample image and the abnormal sample image are stored.

It is to be explained that, here, the plurality of normal sample images and the abnormal sample image each include an image of the same object to be detected, but the shooting angles, the shooting environments, and the like of the plurality of normal sample images may be different from those of the abnormal sample image.

Through step S302, an abnormal region of the abnormal sample image is determined.

Specifically, the abnormal sample image and the normal sample image are compared, and the abnormal area of the abnormal sample is determined according to the comparison result. Namely, the areas with difference in pixel values or image characteristics of the abnormal sample image and the normal sample image are obtained through comparison. For example, a registration algorithm (e.g., an algorithm such as SIFT + RANSAC, ECC) is used to register the normal sample image and the abnormal sample image, and then a pixel value difference or an image feature difference region is obtained by comparing corresponding regions of the normal sample image and the abnormal sample image, where the difference region is an abnormal region.

And step S303, acquiring a first characteristic diagram representing the normal sample image and a second characteristic diagram representing the abnormal sample image.

The image feature extraction unit 111 extracts features of the normal sample image and features of the abnormal sample image to obtain a first feature map representing the normal sample image and a second feature map representing the abnormal sample image, respectively. The image feature extraction unit 111 may implement the extraction of the features of the normal sample image and the features of the abnormal sample image in any manner. For example, the feature extraction layer of the convolutional neural network is used for extracting the features of the image of the normal sample and the features of the image of the abnormal sample to obtain a first feature map and a second feature map.

It should be further explained that the first feature map and the second feature map are not limited to specific feature maps, and may be, for example, a color feature map, a texture feature map, an intermediate layer feature map extracted by a deep learning model, or a feature map such as an edge feature map of an image.

And step S304, determining a first fusion feature map according to the first feature map, the second feature map and the abnormal area.

The image feature fusion unit 112 replaces the features of the region corresponding to the abnormal region in the first feature map with the features of the abnormal region in the second feature map according to the first feature map representing the normal sample image, the second feature map representing the abnormal sample image and the abnormal region obtained in the above steps, so as to obtain a first fusion feature map.

It should be explained that, in this step, the image feature fusion unit 112 may directly replace the feature of the abnormal region in the second feature map with the feature of the region corresponding to the abnormal region in the first feature map to obtain a first fusion feature map; the first fused feature map may be obtained by processing the features of the abnormal region in the second feature map (for example, processing edge extraction, brightness adjustment, edge smoothing, distortion deformation, and the like), and then replacing the features of the region corresponding to the abnormal region in the first feature map.

Finally, the image feature fusion unit 112 performs step S305 to input the first fusion feature map into the image generation model to obtain an expanded image of the abnormal sample, so as to expand the abnormal sample data.

The image generation model may be obtained by training a deep learning model. For example, the deep learning model may be a variational auto-encoder (VAE), and the training sample pairs are a feature map characterizing a real image and a real image, for example, a normal sample image and/or an abnormal sample image in training sample data may be utilized. Specifically, a feature map of a normal sample image and/or an abnormal sample image is extracted and obtained, the feature map representing the normal sample image and the normal sample image, and/or the feature map representing the abnormal sample image and the abnormal sample image form a training sample pair to train a variational self-encoder to obtain an image generation model.

The feature diagram representing the real image in the training sample pair is input into a variational self-encoder, the variational self-encoder can predict the input real image representing the feature diagram representing the real image, loss is calculated according to the predicted real image and the real image in the training sample pair, and then the parameter of the variational self-encoder is updated based on the loss, so that model training is realized. When the loss meets the training end condition, for example, the loss tends to converge, the training can be stopped, and the trained variational self-encoder can be used as an image generation model to output the generated image of the abnormal sample according to the input fused feature map.

For another example, the deep learning model may be GAN (generic adaptive network, generating confrontation network), and the GAN is composed of two parts, one part is an image generator and the other part is an image truth discriminator. The game and the countermeasure between the image generator and the image truth discriminator achieve the ideal effect of GAN. For example, the image generator of the first generation in the initial training stage can generate some abnormal sample images with poor effects according to the input fused feature map, and the image truth discriminator of the first generation can accurately classify the increased abnormal sample images. In short, the image truth discriminator is a two-classifier, and outputs 0 for the augmented abnormal sample image and 1 for the real image. Then, training of the second generation image generator is started, the second generation image generator can generate slightly better abnormal sample images, and the first generation image reality discriminator can be made to consider the enlarged abnormal sample images as real images. Then, a second-generation image truth discriminator is trained, which can accurately identify the real image and the abnormal sample image generated by the second-generation image generator. By analogy, there are three, four, three, and more generations of image generators and image truth discriminators, and the final image truth discriminator cannot distinguish the augmented abnormal sample image from the real image, so far, GAN is fitted, and the image generator can output the augmented abnormal sample image meeting the requirements according to the fused feature map.

The abnormal sample augmentation module obtains a large number of augmented abnormal sample images by utilizing the trained image generation model, the augmented abnormal sample images are input into the manual screening module 12, and the generated abnormal sample images with the characteristics of high picture definition, rich characteristic changes such as abnormal region shapes and colors, abnormal region positions and shapes and the like according with the prior knowledge of related industry experts and ordinary people are selected according to a user selection instruction and added into training sample data to continuously train and update the abnormal detection model.

It can be understood that, in order to facilitate the augmented abnormal sample image to be used for training the abnormality detection model, the image generation model automatically adds a label to the augmented abnormal sample image (i.e. a label representing that the image is an abnormal sample image) when outputting the augmented abnormal sample image, and reduces the workload of manual labeling.

In another embodiment, another method of outlier sample augmentation is also provided. The amplification method is described in detail below with reference to fig. 4.

As shown in fig. 4, the augmentation method includes the steps of:

s401, acquiring a plurality of normal sample images, at least one abnormal sample image and at least one abnormal feature.

The method for acquiring the plurality of normal sample images and the at least one abnormal sample image is similar to the method for acquiring the abnormal sample image in step S301, and reference may be made to step S301, which is not described herein again. It can be understood that the abnormal feature is a feature that an abnormality may occur on the object to be detected, for example, if the detected object is a train, the abnormality that the train may occur is a crack, and the abnormal feature is an image feature of a crack image.

S402, determining a replaceable area according to the normal sample image and the abnormal sample image.

The normal sample image and the abnormal sample image are registered to determine the image area of the object to be detected in the normal sample image, the area is determined to be a replaceable area, and therefore the situation that the abnormal features are added to a background area to cause the invalidation of the expanded abnormal sample image is avoided.

A replacement area is then determined within the replaceable area based on the user instruction through S403. Different from the previous augmentation method, the method adopts a manual appointed scheme, so that the addition of abnormal features is more flexible, and augmented abnormal sample images are richer and more diverse.

The implementation manner of obtaining the third feature map representing the normal sample image and replacing the feature of the third feature map replacement region with the abnormal feature to obtain the second fusion feature map and the image generation model in steps S404 to S406 is similar to the implementation manner of obtaining the first feature map representing the normal sample image and replacing the feature of the region corresponding to the abnormal region of the first feature map and the image generation model in the previous augmentation method, and reference is made to the above, and details are not repeated here.

The method for constructing the anomaly detection model will be described in detail below with reference to fig. 5.

As shown in fig. 5, the anomaly detection model is trained according to at least training sample data and a first loss function including sample weight parameters. The abnormal detection model trained in the way can solve the problem caused by the unbalanced number of the normal sample images and the abnormal sample images in the training sample data, and a more accurate detection result can be obtained.

The construction process of the anomaly detection model is described in detail below with reference to fig. 6.

First, a plurality of normal sample data and a plurality of abnormal sample data are acquired through step S601. It should be explained that the plurality of abnormal sample data includes an abnormal sample image obtained by amplifying the abnormal sample image by the sample amplification module, and the acquired abnormal sample image. Of course, both the normal sample data and the abnormal sample data are labeled sample images.

Then, in step S602, a cluster analysis is performed on the plurality of normal sample data and the plurality of abnormal sample data, respectively, to obtain a cluster analysis result. Specifically, image features of normal sample data and abnormal sample data are extracted, and then cluster analysis is performed on the extracted image features to obtain a cluster analysis result. In the embodiment of the present application, any clustering manner may be adopted to perform clustering analysis on the image features of the normal samples and the image features of the abnormal samples, and the present application is not limited. For example, the method can be a traditional clustering method such as K-means + automatic category selection and spectral clustering, and can also be any new clustering method.

Step S603, determining a sample weight parameter of the first loss function according to the cluster analysis result.

The cluster analysis result comprises the cluster number of the normal samples, the sample number in each normal sample cluster, the cluster number of the abnormal samples and the sample number in the abnormal sample cluster. See in particular the following formula:

w_p＝C_n/(n_p(C_p+C_n))

w_n＝C_p/(n_n(C_p+C_n))

in the formula: w is a_pA weight for a particular normal sample; w is a_nA weight for a particular anomaly sample; c_pAnd C_nRespectively representing the cluster number of normal samples and abnormal samples, namely the change pattern number of the samples; (ii) a n is_pRepresents the number of samples in the cluster in which the particular normal sample is located; n is_nIndicating the number of samples in the cluster in which the particular outlier sample is located.

In this way, the more samples there are and the less the pattern of change of the current sample (n)_pOr n_nLarger), the lower the weight of the sample; the richer the pattern of change of another class of samples (C)_nOr C_pHigher), the class is weighted higher to balance out, avoiding samples of the class being overwhelmed by an overly rich set of patterns of change.

For a particular sample (i.e., a sample of an object to be detected), the loss function is defined as:

L＝-w_n·y·log(p)-w_p(1-y)log(1-p)

wherein w_pAnd w_nIs the sample weight calculated in S603, y is the sample label (1 represents an abnormal sample, 0 represents a normal sample), p is the probability of the sample abnormality predicted by the model, and log is a logarithmic function.

The first loss function is obtained by adding the loss functions of the plurality of specific samples, and the first loss function is the adaptive weight classification loss function corresponding to fig. 5.

Finally, in step S604, the network model is trained according to the plurality of normal sample data, the plurality of abnormal sample data, and the first loss function to obtain an abnormal detection model. The network model may be a convolutional neural network model, and the extraction of the sample image features in the cluster analysis process may be obtained by extracting convolutional layers of the convolutional neural network model. Specifically, normal sample data and abnormal sample data are input to the convolutional neural network model in batches, and the feature extraction layer of the convolutional neural network model extracts image features of the normal sample and image features of the abnormal sample.

According to the construction method of the anomaly detection model, the anomaly detection model can automatically and dynamically adjust the sample weight parameters according to the increase/decrease of normal samples or abnormal samples in training sample data, so that the manual participation degree in model updating in the initial training process and the training process of the anomaly detection model is reduced, the participation degree of algorithm experts in the using process is reduced, and the maintenance cost is reduced.

In another embodiment, the anomaly detection model is obtained by training based on a second loss function (i.e., an auxiliary recognition loss function corresponding to fig. 5), the auxiliary loss function applies structural constraints to the convolutional neural network feature space based on the idea of metric learning, and by requiring that the distance between the features of the normal samples is short and the features of the abnormal samples are far away from the features of the normal samples, it is ensured that the distributions of the normal samples and the abnormal samples in the learned feature space satisfy certain structural constraints, so that when the subsequent interactive sample visualization module 14 visualizes the samples, the normal samples and the abnormal samples can be efficiently and accurately distinguished.

Specifically, for a pair of samples, the loss is defined as:

wherein x_iAnd x_jThe characteristics of the ith image and the jth image in the training data are obtained; y is_iAnd y_jA label representing an image, as defined in the previous section; alpha is a super parameter which can be adjusted; | "| represents the norm (modulo) of the vector.

The trained anomaly detection model can judge whether an input image to be detected is abnormal or not, the anomaly detection model outputs a judgment result to the interactive sample distribution visualization module 14, and due to the fact that the second loss function is adopted to carry out explicit constraint on the characteristic space structure in the training process of the anomaly detection model, the distribution of normal samples and abnormal samples has certain regularity, workers can be assisted to distinguish the normal samples and the abnormal samples quickly through a visualization mode, and therefore auditing is completed quickly.

Two-dimensional feature points are obtained by performing dimension reduction processing on the features of the normal sample and the features of the abnormal sample in the training process of the abnormal detection model (the dimension reduction processing method can adopt PCA, t-SNE, a self-encoder and the like), feature point distribution is obtained by drawing in the same coordinate system according to the feature points, and the distribution of images with normal, abnormal or suspected detection results is obtained, so that the detection result distribution map is presented to a user. The detection result distribution diagram comprises a confidence area and an in-doubt area, and the interactive sample distribution visualization module 14 can provide and present a sample image of the feature point of the sample to be detected falling into the in-doubt area to a user, so that the user can manually check the in-doubt image.

The detection result distribution map may be obtained according to the distribution of the feature points of the normal sample and the feature points of the abnormal sample in the statistical training process, for example, a prediction classification frame for dividing the feature points of the normal sample and the feature points of the abnormal sample may be obtained, so that the feature points of the normal sample are distributed in a frame selection area of the prediction classification frame, and the feature points of the abnormal sample are distributed outside the frame selection area of the prediction classification frame.

In order to prevent false detection of the abnormal detection model, the center of the prediction classification frame can be kept unchanged and reduced according to a certain proportion according to a user instruction to obtain a confidence frame, the region selected by the confidence frame is a confidence region, and the region between the confidence frame and the prediction classification frame is a questioning region. If the characteristic points of the image to be detected fall into the confidence region, the detection result is directly output to be normal, if the characteristic points of the image to be detected fall into the region outside the prediction classification frame, the detection result is directly output to be abnormal, and if the characteristic points of the image to be detected fall into the doubt region, the image is presented to a user for verification to confirm whether the image is abnormal or not. Therefore, the user only needs to recheck the images corresponding to the feature points in the doubt area, and the purposes of reducing rechecking workload and improving rechecking efficiency are achieved.

The manual review module 15 determines whether the suspected image to be detected is abnormal according to the user instruction, and finally outputs the detection result, and takes the image with the abnormal review result as the abnormal sample of the optimization training to perform the optimization training on the abnormal detection model.

Specifically, the anomaly detection device feeds back the rechecking result as an anomaly sample image to the anomaly sample amplification module, so that the anomaly sample amplification module continues to obtain more anomaly sample images according to the received anomaly sample image amplification, and the anomaly detection model is optimized and updated.

In another embodiment, the anomaly detection device further performs optimization training on the anomaly detection model by using the image with the anomaly detection result as the anomaly sample of the optimization training.

In one embodiment, the anomaly detection device can be applied to a railway freight anomaly detection scene.

The method specifically comprises the following steps: the method comprises the steps of obtaining a normal sample image and an abnormal sample image of a freight train through a camera terminal, analyzing abnormal types corresponding to the images, wherein one abnormal type is an abnormality (such as spring breakage, bearing oil leakage and the like) on a characteristic component, the other abnormal type is an abnormality (such as a foreign matter and the like) with a non-fixed position, and marking the images by adopting different marking strategies according to the abnormal types, wherein the images are the abnormal types on a specific component, the positions of the component in the images (such as the position of the normal component and the position of the abnormal component) need to be marked, the images are the abnormal types with the non-fixed position, and the positions where the abnormality occurs need to be marked.

And then different detection schemes are adopted for detection according to different abnormal types. For the type of anomaly on a particular part, the part detection model is first trained, locating the part in the map (see FIG. 7). On this basis, each component is detected using the above-described abnormality detection apparatus, wherein the abnormality sample amplification module amplifies the number of abnormality sample images for each component. The anomaly detection module then trains an anomaly detection model for each component to determine the abnormal state of the component.

The abnormal type with unfixed position can be directly detected by adopting the abnormal detection device, wherein the abnormal sample amplification module amplifies the number of the abnormal sample images until the number of the abnormal sample images is equal to that of the normal sample images, and then an abnormal detection model is obtained by training the abnormal sample images and the normal sample images to judge the abnormal state of the image to be detected.

The abnormality detection device in both of the above-described abnormality type detection schemes can display the detection result by outputting it (see fig. 8).

The interactive sample distribution visualization module can screen out images of samples needing manual review (see fig. 9, a highlighted image is an image of a screened sample needing manual review), after the images of the samples needing review are reviewed by workers, the images of the samples with identification errors (missing inspection and false inspection) are fed back to the abnormal sample augmentation module, then the abnormal sample augmentation module continues to generate new abnormal sample images according to the fed-back images, and the new abnormal sample images are supplemented into training sample data to continue training the abnormal detection model, so that the purpose of improving the performance of the abnormal detection model is achieved.

In another embodiment, the anomaly detection device can be applied to an electronic equipment assembly detection scene.

The method specifically comprises the following steps: the method comprises the steps of obtaining an image of the electronic equipment in the assembling process through a camera terminal, labeling and assembling correct components on the image (for example, labeling in a labeling frame mode), training a component detection model according to the labeled image as a training sample, judging whether the corresponding component is normal and adding the corresponding component into a normal sample library if the detection frame of the component detection model is overlapped with the labeling frame, and judging whether the corresponding component is abnormal and adding the corresponding component into an abnormal sample library if the detection frame of the component detection model is not overlapped with the labeling frame (see fig. 10).

And then, detecting the sample image in the normal sample library by adopting the abnormality detection device, thereby filtering the sample detected by the model in error.

The image with the abnormal result detected by the abnormal detection device is manually rechecked, and whether the image has the abnormal condition or not is judged by counting the number of the detected normal components in the image and comparing the counted number with the number of the normal components which should appear in the image. And feeding the abnormal image back to the abnormal sample amplification module, then continuously generating a new abnormal sample image by the abnormal sample amplification module according to the fed image, and supplementing the new abnormal sample image to the training sample data to continuously train the abnormal detection model so as to achieve the purpose of improving the performance of the abnormal detection model.

The present application further provides an abnormality detection apparatus 2, fig. 11 is a schematic structural diagram of the abnormality detection apparatus 2, and as shown in fig. 11, the apparatus at least includes:

an obtaining module 21, configured to obtain an image to be detected;

and the detection module 22 is configured to determine a detection result of the image to be detected according to the image to be detected and the anomaly detection model, where the anomaly detection model is obtained by training according to a first loss function including a sample weight parameter, and the sample weight parameter represents an importance of a normal sample or an abnormal sample used for training to model training.

In another possible implementation, the apparatus further includes:

the visualization module 23 is configured to generate a detection result distribution map according to the detection result, where the detection result distribution map includes a distribution of images whose detection results are normal, abnormal, or suspected;

and presenting the detection result distribution graph to a user.

the visualization module 23 is further configured to provide the images distributed in the in-doubt area to the user, so that the user can manually check the in-doubt images.

In another possible implementation, the apparatus further includes:

and an adjusting module 24, configured to adjust a range of the suspicious region in the detection result distribution map based on the instruction of the user.

acquiring a plurality of normal samples and at least one abnormal sample;

In another possible implementation, the apparatus further includes:

and the optimization module 25 is configured to select an image with an abnormal artificial verification result as an abnormal sample for optimization training, and perform optimization training on the abnormal detection model by using the selected abnormal sample.

The anomaly detection apparatus 2 according to the embodiment of the present application may correspond to performing the method described in the embodiment of the present application, and the above and other operations and/or functions of each module in the anomaly detection apparatus 2 are respectively for implementing corresponding flows of each method in fig. 3, fig. 4, and fig. 6, and are not described herein again for brevity.

It should be noted that the above-described embodiments are merely illustrative, wherein the modules described as separate parts may or may not be physically separate, and the parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiments of the apparatus provided in the present application, the connection relationship between the modules indicates that there is a communication connection therebetween, and may be implemented as one or more communication buses or signal lines.

The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform any of the methods described above.

The present application also provides a computer program or computer program product comprising instructions which, when executed, cause a computer to perform any of the methods described above.

Fig. 12 is a schematic structural diagram of an abnormality detection apparatus provided in the present application.

As shown in fig. 12, the abnormality detection apparatus 100 includes a processor 101, a memory 102, a communication interface 103, and a bus 104. The processor 101, the memory 102, and the communication interface 103 communicate with each other via the bus 104, or may communicate with each other via other means such as wireless transmission. The memory 102 stores executable program code and the processor 101 may call the program code stored in the memory 102 to perform the anomaly detection method in the aforementioned method embodiments.

It should be understood that, in the embodiment of the present application, the processor 101 may be a central processing unit CPU, and the processor 101 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or any conventional processor or the like.

The memory 102 may include both read-only memory and random access memory and provides instructions and data to the processor 101. The memory 102 may also include non-volatile random access memory. For example, the memory 102 may also store a training data set.

The memory 102 may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and direct bus RAM (DR RAM).

The bus 104 may include a power bus, a control bus, a status signal bus, and the like, in addition to a data bus. But for clarity of illustration the various buses are labeled as bus 104 in the figures.

It should be understood that the abnormality detection apparatus 100 according to the embodiment of the present application may correspond to an abnormality detection device in the embodiment of the present application, and may correspond to a corresponding main body in executing the methods shown in fig. 3, fig. 4, and fig. 6 according to the embodiment of the present application, and the above-mentioned and other operations and/or functions of each device in the object counting apparatus 100 are respectively to implement corresponding flows of the methods shown in fig. 3, fig. 4, and fig. 6, and are not described herein again for brevity.

It will be further appreciated by those of ordinary skill in the art that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether these functions are performed in hardware or software depends on the particular application of the solution and design constraints. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied in hardware, a software module executed by a processor, or a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The above-mentioned embodiments, objects, technical solutions and advantages of the present application are described in further detail, it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present application, and are not intended to limit the scope of the present application, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. An anomaly detection method, characterized in that it comprises:

acquiring an image to be detected;

and determining a detection result of the image to be detected according to the image to be detected and an abnormality detection model, wherein the abnormality detection model is obtained by training according to a first loss function comprising sample weight parameters, and the sample weight parameters represent the importance of normal samples or abnormal samples for training on model training.

2. The method of claim 1, further comprising: generating a detection result distribution map according to the detection result, wherein the detection result distribution map comprises the distribution of images with normal, abnormal or suspected detection results;

and presenting the detection result distribution graph to a user.

3. The method of claim 2, wherein the test result profile further comprises a confidence region and an in-doubt region;

and providing the images distributed in the in-doubt area for the user to manually check the in-doubt images.

4. The method of claim 3, further comprising:

adjusting the range of the in-doubt area in the detection result distribution map based on the instruction of the user.

5. The method according to any one of claims 1 to 4, wherein the sample weight parameters are determined from the results of cluster analysis of a normal sample set and an abnormal sample set in the training sample data.

6. The method according to any one of claims 1 to 5, wherein the anomaly detection model is trained based on a second loss function, and the second loss function is used for constraining the structure of the feature space of the anomaly detection model, so that the distance between the features of the normal samples is shortened and the distance between the features of the anomaly samples and the features of the normal samples is lengthened in the feature space.

7. The method according to any one of claims 1 to 6, wherein the training sample data of the anomaly detection model includes anomaly samples obtained by augmentation, and the augmentation method includes:

acquiring a plurality of normal samples and at least one abnormal sample;

8. The method according to any one of claims 1 to 6, wherein the training sample data of the anomaly detection model includes anomaly samples obtained by augmentation, and the augmentation method includes:

9. The method according to any one of claims 3-8, further comprising: and selecting the image with the abnormal artificial checking result as an abnormal sample for optimization training, and performing optimization training on the abnormal detection model by using the selected abnormal sample.

10. An abnormality detection device characterized by comprising:

the acquisition module is used for acquiring an image to be detected;

11. The apparatus of claim 10, further comprising:

and presenting the detection result distribution graph to a user.

12. The apparatus of claim 11, wherein the test result distribution map further comprises a confidence region and an in-doubt region;

13. The apparatus of claim 12, further comprising:

14. The apparatus according to any one of claims 10-13, wherein the sample weight parameters are determined according to cluster analysis results of a normal sample set and an abnormal sample set in the training sample data.

15. The apparatus according to any one of claims 10 to 14, wherein the anomaly detection model is further trained based on a second loss function, and the second loss function is used for constraining the structure of the feature space of the anomaly detection model, so that the distance between the features of the normal samples is shortened and the distance between the features of the anomaly samples and the features of the normal samples is lengthened in the feature space.

16. The apparatus according to any one of claims 10 to 15, wherein the training sample data of the anomaly detection model includes anomaly samples obtained by augmentation, and the augmentation method includes:

acquiring a plurality of normal samples and at least one abnormal sample;

17. The apparatus according to any one of claims 10 to 16, wherein the training sample data of the anomaly detection model includes anomaly samples obtained by augmentation, and the augmentation method includes:

18. The apparatus according to any one of claims 12-17, further comprising:

19. An anomaly detection device comprising a memory and a processor, wherein the memory has stored therein executable code, and wherein the processor executes the executable code to implement the method of any one of claims 1-9.

20. A computer-readable storage medium, on which a computer program is stored, which, when the computer program is executed in a computer, causes the computer to carry out the method of any one of claims 1-9.

21. A computer program or computer program product, characterized in that it comprises instructions which, when executed, implement the method of any one of claims 1-9.