CN113674220A - Image difference detection method, detection device and storage medium - Google Patents
Image difference detection method, detection device and storage medium Download PDFInfo
- Publication number
- CN113674220A CN113674220A CN202110860122.8A CN202110860122A CN113674220A CN 113674220 A CN113674220 A CN 113674220A CN 202110860122 A CN202110860122 A CN 202110860122A CN 113674220 A CN113674220 A CN 113674220A
- Authority
- CN
- China
- Prior art keywords
- image
- detected
- feature
- difference
- reference image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an image difference detection method, a detection device and a computer readable storage medium, wherein the method comprises the following steps: acquiring an image to be detected and a reference image, wherein the image to be detected and the reference image are images shot in the same scene at different moments; respectively extracting the features of an image to be detected and a reference image by using a difference feature extraction network to obtain a first feature map of the image to be detected and a second feature map of the reference image, wherein the difference feature extraction network is obtained by training an intervention sample, and the intervention sample comprises an image sample and an interference sample carrying image background information; and comparing the characteristics of the first characteristic diagram and the second characteristic diagram to obtain image difference information of the first characteristic diagram and the reference image. By means of the method, the accuracy of image difference detection can be improved.
Description
Technical Field
The present application relates to the field of image processing, and in particular, to an image difference detection method, an image difference detection apparatus, and a computer-readable storage medium.
Background
With the increase of the industrial automation degree, the quality requirement of the difference detection work is continuously improved. The existing image difference detection method has poor effect in detection of some scenes, is difficult to overcome background interference in specific scenes, and is easy to generate missed detection and false detection.
Disclosure of Invention
The technical problem mainly solved by the application is to provide an image difference detection method, a detection device and a computer readable storage medium, which can improve the accuracy of image difference detection.
In order to solve the technical problem, the application adopts a technical scheme that: there is provided an image difference detection method, the method including: acquiring an image to be detected and a reference image, wherein the image to be detected and the reference image are images shot in the same scene at different moments; respectively extracting the features of an image to be detected and a reference image by using a difference feature extraction network to obtain a first feature map of the image to be detected and a second feature map of the reference image, wherein the difference feature extraction network is obtained by training an intervention sample, and the intervention sample comprises an image sample and an interference sample carrying image background information; and comparing the characteristics of the first characteristic diagram and the second characteristic diagram to obtain image difference information of the first characteristic diagram and the reference image.
The method comprises the following steps of obtaining a first feature map of an image to be detected and a second feature map of a reference image by utilizing the difference feature extraction network to respectively extract features of the image to be detected and the reference image, wherein the difference feature extraction network comprises a convolution network, and the method comprises the following steps: semantic analysis is respectively carried out on the image to be detected and the reference image by utilizing a convolution network to obtain a first semantic segmentation feature map of the image to be detected and a second semantic segmentation feature map of the reference image; and extracting a first feature map from the first semantic segmentation feature map and extracting a second feature map from the second semantic segmentation feature map.
The difference feature extraction network is obtained by training with intervention samples, and comprises the following steps: acquiring an image sample; pasting background information into the partial image sample to obtain an interference sample; performing semantic segmentation and labeling on an undisturbed image sample and an interfered sample; and training the difference feature extraction network by using the marked undisturbed image sample and the marked undisturbed sample.
Before feature extraction is respectively carried out on the image to be detected and the reference image by using a difference feature extraction network to obtain a first feature map of the image to be detected and a second feature map of the reference image, the image difference detection method further comprises the following steps: respectively extracting characteristic points of an image to be detected and a reference image to obtain a first characteristic point of the image to be detected and a second characteristic point of the reference image; and aligning the first characteristic points of the image to be detected and the second characteristic points of the reference image to obtain an aligned image of the image to be detected relative to the reference image.
Wherein, the first characteristic point of the image to be detected and the second characteristic point of the reference image are aligned to obtain an aligned image of the image to be detected relative to the reference image, and the method comprises the following steps: matching the first characteristic points of the image to be detected with the second characteristic points of the reference image to obtain a matching image; sampling the matching points in the matching graph, and calculating to obtain a homography matrix; and converting the image to be detected by utilizing the homography matrix to obtain an aligned image.
The method for matching the first feature point of the image to be detected with the second feature point of the reference image to obtain a matching image includes: acquiring a first feature descriptor of an image to be detected and a second feature descriptor of a reference image; and matching the first characteristic descriptor and the second characteristic descriptor by using a super glue network to obtain a matching graph.
The method for extracting the feature points of the image to be detected and the reference image respectively comprises the following steps: and extracting the feature points of the image to be detected and the reference image by using a feature point detection network, wherein the feature point detection network is obtained by using a corner point labeling method for training, and the corner point labeling information comprises 2D (two-dimensional) graphic information.
The feature comparison between the first feature map and the second feature map to obtain the image difference information between the first feature map and the reference image includes: performing difference processing on the first characteristic diagram and the second characteristic diagram to obtain a difference characteristic diagram of the first characteristic diagram and the second characteristic diagram; and carrying out binarization, corrosion expansion or connected domain contour processing on the difference characteristic diagram to obtain image difference information.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided an image difference detection apparatus comprising a processor and a memory, the memory storing program instructions, the processor being configured to execute the program instructions to implement the image difference detection method described above.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a computer-readable storage medium storing program instructions that can be executed to implement the image difference detection method described above.
The beneficial effect of this application is: in contrast to the prior art, the present application provides an image difference detection method, including: acquiring an image to be detected and a reference image, wherein the image to be detected and the reference image are images shot in the same scene at different moments; respectively extracting the features of an image to be detected and a reference image by using a difference feature extraction network to obtain a first feature map of the image to be detected and a second feature map of the reference image, wherein the difference feature extraction network is obtained by training an intervention sample, and the intervention sample comprises an image sample and an interference sample carrying image background information; and comparing the characteristics of the first characteristic diagram and the second characteristic diagram to obtain image difference information of the first characteristic diagram and the reference image. The difference characteristic extraction network is obtained by training an intervention sample, the intervention sample comprises an image sample and an interference sample carrying image background information, so that the difference characteristic extraction network can inhibit the image background information in the image when extracting the first characteristic diagram of the image to be detected and the second characteristic diagram of the reference image, the influence of the image background information on subsequent difference detection is avoided, the interference of background change on the image difference detection is reduced, and the accuracy of the image difference detection is improved.
Drawings
FIG. 1 is a schematic flowchart of an embodiment of an image difference detection method provided in the present application;
FIG. 2 is a schematic illustration of an image to be detected and a reference image provided herein;
FIG. 3 is a schematic diagram of a differential signature provided herein;
FIG. 4 is a schematic illustration of a characteristic differential heat map provided herein;
FIG. 5 is a schematic illustration of image differences provided herein;
FIG. 6 is a schematic flowchart of another embodiment of an image difference detection method provided in the present application;
FIG. 7 is a schematic diagram of an embodiment of a feature point detection network provided herein;
FIG. 8 is a schematic flowchart of an embodiment of obtaining an alignment image of an image to be detected with respect to a reference image by using a homography matrix;
FIG. 9 is a schematic diagram of one embodiment of a matching graph provided herein;
FIG. 10 is a schematic view of an embodiment of a superglue network as provided herein;
FIG. 11 is a schematic flowchart of an embodiment of feature extraction performed on an image to be detected and a reference image by using a convolutional network according to the present application;
FIG. 12 is a schematic structural diagram of an embodiment of an image difference detection apparatus provided in the present application;
fig. 13 is a schematic structural diagram of a computer-readable storage medium provided in the present application.
Detailed Description
In order to make the purpose, technical solution and effect of the present application clearer and clearer, the present application is further described in detail below with reference to the accompanying drawings and examples.
It should be noted that if descriptions related to "first", "second", etc. exist in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes, and are not to be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of an image difference detection method according to the present application. It should be noted that, if the result is substantially the same, the flow sequence shown in fig. 1 is not limited in this embodiment. As shown in fig. 1, the present embodiment includes:
step S11: and acquiring an image to be detected and a reference image.
The method of the present embodiment is used for detecting differences between two or more images, and the image to be detected and the reference image described herein may be images taken in the same scene at different times. It is to be understood that, in other embodiments, the image to be detected and the reference image may be images captured in different scenes and at different times, in different scenes and at the same time, or in the same scene and at the same time.
In a specific embodiment, as shown in fig. 2, fig. 2 is a schematic diagram of an image to be detected and a reference image provided by the present application, and the method of the present embodiment is used for performing difference analysis on an image including a power meter acquired at a current time and an image including a power meter acquired from the same scene at a previous time, so as to determine whether an abnormal condition exists in the equipment according to the difference of the power meters, avoid missing detection or false detection caused by manual detection, and improve power detection efficiency. When the power meter is used to determine whether the device has an abnormal operation, the image at the first time (previous time) may be used as a reference image, the image at the second time (later time) may be used as an image to be detected, and the image at the second time may be a reference image of an image acquired at the third time as time progresses. It is understood that, in other embodiments, the embodiment may also be used in other application scenarios requiring difference detection, and is not limited in detail herein.
In an embodiment, the image to be detected may be an image at any time and in the same scene as the reference image, and may be specifically acquired from a local storage or a cloud storage, or acquired by acquiring the image at the current time through an image acquisition device.
Step S12: and respectively extracting the features of the image to be detected and the reference image by using a difference feature extraction network to obtain a first feature map of the image to be detected and a second feature map of the reference image.
In this embodiment, the difference feature extraction network is used to perform feature extraction on the image to be detected and the reference image respectively, so as to obtain a first feature map of the image to be detected and a second feature map of the reference image.
The difference characteristic extraction network is obtained by training an intervention sample, wherein the intervention sample comprises an image sample and an interference sample carrying image background information. In one embodiment, an image sample is first acquired; secondly, pasting background information into the partial image sample to obtain an interference sample; thirdly, performing semantic segmentation and labeling on the undisturbed image sample and the disturbed sample; and finally, training the difference feature extraction network by using the labeled undisturbed image sample and the labeled undisturbed sample, so that training the difference feature extraction network by using the interference sample.
Because the difference feature extraction network is obtained based on the training of the intervention sample, the difference feature extraction network trained by the intervention sample can inhibit the interference features related in the intervention sample, reduce the influence of image background information on subsequent difference detection, and improve the accuracy of image difference detection. In other embodiments, the difference feature extraction network may be trained by using the image sample, and then the trained difference feature extraction network may be further trained by using the intervention sample with the interference sample to obtain a difference feature extraction network that can be practically applied. It is to be understood that, in other embodiments, the difference feature extraction network may also be trained in other manners to enable the image background information to be suppressed when the difference feature extraction network is actually used, and the difference feature extraction network may be specifically set according to actual use needs, and is not specifically limited herein.
Optionally, the interference sample carrying the image background information may be specifically set according to an actual application scenario, and is not specifically limited herein. For example, when the method of this embodiment is used in an industrial scene, the interference sample carrying the image background information may be a cloud, a land, vegetation, or ponding, and the interference sample is obtained by adding the cloud, the land, the vegetation, or the like to the image sample in a form of a map, so that the difference feature extraction network obtained by training the interference sample including the cloud, the land, or the vegetation can suppress the interference features of the cloud, the land, or the vegetation, thereby preventing the interference features from being detected as a difference to cause a difference misjudgment, reducing an influence on image difference detection due to a background change, and improving accuracy of image difference detection.
For example, as shown in fig. 2, the right image (b) in fig. 2 is a reference image, the left image (a) is an image to be detected, if the image to be detected includes clouds in the process of acquiring the image to be detected, the cloud is detected as a difference between the two images in the subsequent difference detection, and the actual cloud portion is not a difference that needs to be known by the user, so that the existence of the cloud affects the subsequent difference detection result, the accuracy of the difference detection is reduced, or the calculation amount in the difference detection process is increased. If the difference feature extraction network obtained by training by using the intervention sample is used for feature extraction, the interference feature of the cloud can be inhibited by the difference feature extraction network, so that the influence of the feature of the cloud on the subsequent image difference detection is avoided, and the interference caused by the background change in the subsequent detection process is eliminated.
In another embodiment, the color difference caused by illumination, shadow, rain or fog also affects the subsequent image difference detection to cause misjudgment, and in an embodiment, the image sample can be processed by a data enhancement technology, such as a color disturbance and mode means, to train the difference feature extraction network, so that the difference feature extraction network has universality, the interference of the color difference caused by illumination, shadow, rain or fog and other factors on the subsequent image difference detection can be effectively reduced, the accuracy of the image difference detection is improved, and the occurrence of misjudgment of the difference is reduced. It is to be understood that, in other embodiments, the intervention sample and the data enhancement technology may also be used to train the difference feature extraction network, and may be specifically set according to an actual usage scenario, which is not specifically limited herein.
Step S13: and comparing the characteristics of the first characteristic diagram and the second characteristic diagram to obtain image difference information of the first characteristic diagram and the reference image.
In this embodiment, the first feature map and the second feature map are compared to obtain difference information between the first feature map and the reference image, that is, difference information between the image to be detected and the reference image. The image difference information obtained through the feature comparison may be a part having a difference on the image, or may be a certain area including the difference part, and the embodiment of the image difference information may be specifically set according to the actual use requirement, which is not specifically limited herein.
In an embodiment, the difference feature map of the first feature map and the second feature map can be obtained by performing difference processing on the features of the first feature map and the second feature map, and the difference between the image to be detected and the reference image can be determined based on the difference feature map.
Specifically, as shown in fig. 3 and 4, fig. 3 is a schematic diagram of a differential feature map provided in the present application, fig. 4 is a schematic diagram of a feature difference heat map provided in the present application, a difference process is performed on features in a first feature map and a second feature map, pixel values of corresponding feature points in the first feature map and the second feature map are subtracted to obtain the differential feature map, if the regions are the same, a difference value on the differential feature map is zero, that is, the pixel value is zero, and if the regions are different, a difference value on the differential feature map is not zero, that is, the pixel value is not zero. Because the size of the pixel value reflects the brightness information of the point, the point is darker when the pixel value is zero, and the point is brighter when the pixel value is not zero, the difference of the image to be detected relative to the reference image can be judged by judging the brightness of each position on the difference characteristic diagram. Based on this, as shown in fig. 3, image difference information is determined from the luminance area in the image, for example, an image difference area is determined. In addition, the difference feature map shown in fig. 3 may be subjected to heat processing to obtain a feature difference heat map as shown in fig. 4, so as to more clearly know the image difference region.
It is to be understood that, in other embodiments, the first feature map and the second feature map may be compared in other ways to obtain the image difference information, and the image difference information may be specifically set according to actual use needs, and is not specifically limited herein.
It is to be understood that, in other embodiments, in order to obtain more accurate image difference information, the differential feature map may further be subjected to binarization, erosion expansion, or connected domain contour, and the specific processing manner may be specifically set according to actual usage requirements, and is not specifically limited herein. For example, as shown in fig. 5, fig. 5 is a schematic diagram of image differences provided by the present application, and the difference feature map is subjected to binarization, erosion expansion, connected component contour extraction, and the like to obtain a difference portion selected from a frame in the left image of fig. 5.
In order to determine difference information of the obtained image more accurately based on the feature comparison result of the first feature map and the second feature map, in an embodiment, the registration processing may be further performed on the image to be detected and the reference image before the feature extraction is performed, so that the first feature map and the second feature map can be registered and aligned, and the comparison between corresponding features in the first feature map and the second feature map is facilitated.
Referring to fig. 6, fig. 6 is a schematic flowchart illustrating an image difference detection method according to another embodiment of the present application. It should be noted that, if the result is substantially the same, the flow sequence shown in fig. 6 is not limited in this embodiment. As shown in fig. 6, in this embodiment, the performing feature extraction on the image to be detected and the reference image by using a difference feature extraction network, and performing feature comparison by using a difference processing method on the first feature map and the second feature map, so as to determine and obtain image difference information specifically includes:
step S61: and acquiring an image to be detected and a reference image.
Step S61 is similar to step S11, and will not be described herein.
Step S62: and respectively extracting characteristic points of the image to be detected and the reference image to obtain a first characteristic point of the image to be detected and a second characteristic point of the reference image.
In an embodiment, as shown in fig. 7, fig. 7 is a schematic diagram of an embodiment of a feature point detection network provided in the present application, and a feature point detection network (SuperPoint) may be used to extract feature points of an image to be detected and a reference image. It is to be understood that, in other embodiments, other network models or algorithms may also be used to extract feature points of the image to be detected and the reference image, and the feature points may be specifically set according to actual use needs, which is not specifically limited herein.
The feature point detection network is obtained by utilizing a corner point marking method for training, and the training process of the feature point detection network specifically comprises the following steps: step one, constructing a synthetic data set, wherein the data set is composed of simple 2D graphs, such as triangles, straight lines, ellipses, polygons and the like; step two, training a corner point detection network (MagicPoint) by utilizing the constructed synthetic data set; thirdly, on the synthetic data set, extracting corners by using a corner detection network, namely marking the corners on each graph as key point marking information, which can also be called as interest point self-marking; and step four, training the angular point detection network by using a large number of images on the COCO data set to obtain a feature point detection network.
Step S63: and aligning the first characteristic points of the image to be detected and the second characteristic points of the reference image to obtain an aligned image of the image to be detected relative to the reference image.
In this embodiment, the first feature point of the image to be detected and the second feature point of the reference image may be aligned, so that the image to be detected can be transformed through the aligned first feature point and the aligned second feature point to obtain an aligned image of the image to be detected relative to the reference image.
In an embodiment, as shown in fig. 8, fig. 8 is a schematic flowchart of an embodiment of obtaining an aligned image of an image to be detected relative to a reference image by using a homography matrix according to the present application, and the aligned image of the image to be detected relative to the reference image can be obtained by using the homography matrix generated based on the image to be detected and the reference image; of course, in other embodiments, the image to be detected and the reference image may be aligned in other ways. Obtaining an alignment image of an image to be detected relative to a reference image through a homography matrix, and specifically comprising the following steps:
step S631: and matching the first characteristic points of the image to be detected with the second characteristic points of the reference image to obtain a matching image.
In this embodiment, the first feature point of the image to be detected and the second feature point of the reference image are matched to obtain a matching image. Specifically, as shown in fig. 9, fig. 9 is a schematic diagram of an embodiment of a matching graph provided in the present application, where an image to be detected includes a plurality of first feature points, a reference image includes a plurality of second feature points, and second feature points matching the first feature points are found on the reference image, so that the first feature points of the image to be detected and the second feature points of the reference image correspond to each other one to complete matching of the first feature points of the image to be detected and the second feature points of the reference image, so as to obtain the matching graph.
In an embodiment, as shown in fig. 10, fig. 10 is a schematic diagram of an example of a super glue network provided in the present application, and a matching map is obtained by matching a first feature point of an image to be detected and a second feature point of a reference image using the super glue (super glue) network. Compared with the traditional feature matching algorithm, the super glue network can better focus on the global information of the image, and can realize the registration and alignment of the image under the conditions of weak texture, no texture, low illumination or noise interference and the like, so that the registration effect is more accurate, and the analysis of the difference of the subsequent image is facilitated. Specifically, the feature point detection network extracts a first feature point, a second feature point, a first feature descriptor corresponding to the first feature point, and a second feature descriptor corresponding to the second feature point, and inputs the feature point and the descriptor corresponding to the feature point into the super glue network, so that the super glue network matches the first feature point and the second feature point based on the first feature descriptor and the second feature descriptor, thereby obtaining a matching relationship between the first feature point on the image to be detected and the second feature point on the reference image, that is, a matching result. It is to be understood that, in other embodiments, other network models or algorithms may also be used to perform feature point matching on the first feature points of the image to be detected and the second feature points of the reference image, and the feature points may be specifically set according to actual use requirements, which is not specifically limited herein. It should be noted that when the super glue network matches the first feature point and the second feature point, the abnormal feature point, that is, the feature point without a matching object, is removed, so as to avoid that the abnormal feature point affects the registration of the subsequent image, and therefore, the matching graph shown in fig. 9 only includes feature points that are successfully matched.
The two images generated by the self-supervision learning mode are input into the super glue network to obtain the matching relation of the characteristic points on the two images, and the matching relation is used as a true value to train the super glue network.
Step S632: and sampling the matching points in the matching graph, and calculating to obtain a homography matrix.
And generating a homography matrix corresponding to the image to be detected and the reference image based on the matching points on the matching graph. In an embodiment, the matched first feature point and the second feature point may be sampled by a Random Sample Consensus (RANSAC) algorithm, and a homography matrix may be calculated based on the sampled first feature point and the sampled second feature point having a matching relationship. It is to be understood that, in other embodiments, the image to be detected and the reference image may also be processed through other networks or algorithms to obtain the homography matrix corresponding to the image to be detected and the reference image, which may be specifically set according to actual use needs, and is not specifically limited herein.
Step S633: and converting the image to be detected by utilizing the homography matrix to obtain an aligned image.
In this embodiment, since each feature point on the homography matrix generated by calculation is correspondingly matched, the conversion relationship between the image to be detected and the reference image can be determined according to the obtained homography matrix corresponding to the image to be detected and the reference image, so that the image to be detected can be adjusted to obtain an aligned image of the image to be detected relative to the reference image, that is, the homography matrix can be used to adjust the image to be detected to be aligned with the reference image. It is understood that in other embodiments, the image to be detected and the reference image may be registered and aligned in other manners, and may be specifically set according to actual use requirements, which is not specifically limited herein.
Specifically, after homography matrixes in the image to be detected and the reference image are calculated, all pixel points of the image to be detected are multiplied by the homography matrixes to map the image to be detected onto the reference image, and therefore registration alignment of the image to be detected and the reference image is achieved.
As shown in fig. 2, at present, whether the image to be detected and the reference image have offset changes cannot be found by naked eyes, but due to the fact that a camera for image acquisition is offset, etc., the image to be detected and the reference image may have very fine offset, that is, offset on a pixel level, so that the two images shown in fig. 2 still need to be subjected to registration and alignment processing.
Further, in one embodiment, the registered and aligned images are cropped. After the image to be detected or the reference image is converted to realize registration alignment, the converted image may have a black edge, and the existence of the black edge may be identified as a difference region in the subsequent difference analysis process, so that the black edge of the image after registration alignment is clipped, the interference of the black edge on the image on the detection of the image difference is avoided, and the calculation amount in the image difference detection process is increased.
Step S64: and respectively extracting the features of the image to be detected and the reference image by using a difference feature extraction network to obtain a first feature map of the image to be detected and a second feature map of the reference image.
In the embodiment, the difference feature extraction network obtained by training the intervention sample is used for extracting the features of the image to be detected and the reference image, the intervention sample comprises the image sample and the interference sample carrying the image background information, and the interference sample carrying the image background information can be added to the image sample in a chartlet or other mode to obtain the intervention sample, so that the difference feature extraction network trained by using the intervention sample can inhibit the interference features related in the intervention sample, the influence of the image background change on the subsequent image difference detection is reduced, and the accuracy of the image difference detection is improved. In other embodiments, the difference feature extraction network may be trained by using the image sample, and then the trained difference feature extraction network may be further trained by using the interference sample with the image background information added by intervention. It is to be understood that, in other embodiments, the difference feature extraction network may also be trained in other manners to enable the image background information to be suppressed when the difference feature extraction network is actually used, and the difference feature extraction network may be specifically set according to actual use needs, and is not specifically limited herein.
Optionally, the difference feature extraction network is a Mask-RCNN network model pre-trained on the COCO, and a backbone network ResNet-50 in the Mask-RCNN network model is used as a feature extractor to perform feature extraction on the image to be detected and the reference image. It is understood that, in other embodiments, the difference feature extraction network may also be other network models, and is not limited in detail herein.
It should be noted that, in this embodiment, the image to be detected and the reference image are images after registration alignment, that is, after the image to be detected and the reference image are subjected to registration alignment, feature extraction is performed to obtain the first feature map and the second feature map.
Referring to fig. 11, fig. 11 is a schematic flowchart illustrating a process of extracting features of an image to be detected and a reference image by using a convolutional network according to an embodiment of the present disclosure. In one embodiment, the difference feature extraction network includes a convolutional network, which may be a deep convolutional network, and the convolutional layer of the deep convolutional network has a large receptive field and focuses more on global semantic information; it will be appreciated that in other embodiments, the convolutional network may also be a shallow convolutional network, which focuses more on local details. In which, the output of different convolution layers can be selected as the output of the difference feature extraction network according to the target size in the actual application scene, and the like, and is not limited specifically here. The method for extracting the characteristics of the image to be detected and the reference image by adopting the convolutional network specifically comprises the following steps:
step S641: and performing semantic analysis on the image to be detected and the reference image respectively by using a convolution network to obtain a first semantic segmentation feature map of the image to be detected and a second semantic segmentation feature map of the reference image.
In the embodiment, a convolution network is used for performing global semantic analysis on the image to be detected and the reference image respectively, so that the content belonging to the same semantic in the image is segmented, and a first semantic segmentation feature map of the image to be detected and a second semantic segmentation feature map of the reference image are obtained. Because the target region in the image is labeled when the convolutional network is trained, the trained convolutional network can segment the target region from the background region in the image. Therefore, when the convolution network is used for performing semantic analysis on the image to be detected and the reference image, the target area and the background area in the image to be detected and the reference image are divided, and the background area is suppressed, so that the target area can be conveniently extracted subsequently.
Step S642: and extracting a first feature map from the first semantic segmentation feature map and extracting a second feature map from the second semantic feature map.
In the present embodiment, since the first semantic segmentation feature map and the second semantic segmentation feature map already segment the background region and the target region, the first feature map can be extracted directly from the first semantic segmentation feature map and the second feature map can be extracted directly from the second semantic segmentation feature map. In addition, since the background region is suppressed in the first semantic segmentation feature map and the second semantic segmentation feature map, only the target region is included in the first feature map and the second feature map.
Step S65: and performing difference processing on the first characteristic diagram and the second characteristic diagram to obtain a difference characteristic diagram of the first characteristic diagram and the second characteristic diagram.
In the present embodiment, the difference feature map between the first feature map and the second feature map is obtained by performing difference processing on the features of the first feature map and the second feature map. Specifically, since the background factors in the image to be detected and the reference image in the first feature map and the second feature map are suppressed, the first feature map and the second feature map actually include only the target region, and the difference processing is performed on the features in the first feature map and the second feature map, so that the pixel values of the corresponding feature points in the first feature map and the second feature map are subtracted to obtain the difference feature map.
Step S66: and carrying out binarization, corrosion expansion or connected domain contour processing on the difference characteristic diagram to obtain image difference information.
In the difference feature map, if the regions are the same, the difference value on the difference feature map is zero, that is, the pixel value is zero, and if the regions are different, the difference value on the difference feature map is not zero, that is, the pixel value is not zero. Since the magnitude of the pixel value reflects the luminance information of the point, the point is darker when the pixel value is zero, and the point is brighter when the pixel value is not zero. Therefore, in the present embodiment, the difference information of the image to be detected with respect to the reference image can be determined by determining the brightness of each position on the difference feature map, for example, to obtain the image difference region. It is to be understood that, in other embodiments, the first feature map and the second feature map may be compared in other ways to obtain the image difference information, and the image difference information may be specifically set according to actual use needs, and is not specifically limited herein.
Further, in order to obtain a more accurate image difference region, the difference feature map needs to be subjected to binarization, erosion expansion or connected domain contour extraction, and the specific processing mode adopted may be specifically set according to actual use requirements, and is not specifically limited herein. For example, as shown in fig. 5, the difference feature map is subjected to binarization, erosion expansion, connected component contour extraction, and the like, so as to obtain the difference region selected by the frame in the left image of fig. 5.
It is understood that, in the embodiment, the first feature map and the second feature map are obtained by feature extraction of the registered image to be detected and the reference image, for example, in the embodiment that the registration of the image to be detected and the reference image is not required, the steps S62-S63 and the steps S631-S633 may not be performed.
Referring to fig. 12, fig. 12 is a schematic structural diagram of an embodiment of an image difference detection apparatus provided in the present application. In this embodiment, the image difference detection apparatus 90 includes a processor 91 and a memory 93.
The processor 91 may also be referred to as a CPU (Central Processing Unit). The processor 91 may be an integrated circuit chip having signal processing capabilities. The processor 91 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor 91 may be any conventional processor 91 or the like.
The memory 93 in the image difference detection device 90 is used to store program instructions required for the processor 91 to operate.
The processor 91 is configured to execute the program instructions to implement the methods provided in any of the embodiments of the image difference detection method of the present application and any non-conflicting combinations.
Referring to fig. 13, fig. 13 is a schematic structural diagram of a computer-readable storage medium provided in the present application. The computer readable storage medium 100 of the embodiments of the present application stores program instructions 101, and the program instructions 101 when executed implement the method provided by any of the embodiments of the image difference detection method of the present application and any non-conflicting combinations. The program instructions 101 may form a program file stored in the computer-readable storage medium 100 in the form of a software product, so as to enable a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods according to the embodiments of the present application. And the aforementioned computer-readable storage medium 100 includes: various media capable of storing program codes, such as a usb disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, or terminal devices, such as a computer, a server, a mobile phone, and a tablet. The above description is only for the purpose of illustrating embodiments of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application or are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (10)
1. An image difference detection method, characterized in that the method comprises:
acquiring an image to be detected and a reference image, wherein the image to be detected and the reference image are images shot in the same scene at different moments;
respectively extracting the features of the image to be detected and the reference image by using a difference feature extraction network to obtain a first feature map of the image to be detected and a second feature map of the reference image, wherein the difference feature extraction network is obtained by training an intervention sample, and the intervention sample comprises an image sample and an interference sample carrying image background information;
and comparing the characteristics of the first characteristic diagram and the second characteristic diagram to obtain image difference information of the first characteristic diagram and the reference image.
2. The image difference detection method according to claim 1, wherein the difference feature extraction network comprises a convolution network, and the obtaining of the first feature map of the image to be detected and the second feature map of the reference image by respectively performing feature extraction on the image to be detected and the reference image by using the difference feature extraction network comprises:
semantic analysis is respectively carried out on the image to be detected and the reference image by utilizing the convolutional network to obtain a first semantic segmentation feature map of the image to be detected and a second semantic segmentation feature map of the reference image;
and extracting the first feature map from the first semantic segmentation feature map and extracting the second feature map from the second semantic feature segmentation map.
3. The image difference detection method according to claim 1, wherein the difference feature extraction network is trained by using an intervention sample, and comprises:
acquiring an image sample;
pasting background information into a part of the image sample to obtain the interference sample;
performing semantic segmentation and labeling on the undisturbed image sample and the disturbed sample;
and training the difference feature extraction network by using the marked undisturbed image sample and the marked disturbed sample.
4. The image difference detection method according to claim 1, wherein before the feature extraction is performed on the image to be detected and the reference image respectively by using a difference feature extraction network to obtain the first feature map of the image to be detected and the second feature map of the reference image, the method further comprises:
respectively extracting characteristic points of the image to be detected and the reference image to obtain a first characteristic point of the image to be detected and a second characteristic point of the reference image;
and aligning the first characteristic point of the image to be detected and the second characteristic point of the reference image to obtain an aligned image of the image to be detected relative to the reference image.
5. The image difference detection method according to claim 4, wherein the aligning the first feature point of the image to be detected and the second feature point of the reference image to obtain an aligned image of the image to be detected with respect to the reference image comprises:
matching the first characteristic points of the image to be detected with the second characteristic points of the reference image to obtain a matching image;
sampling the matching points in the matching graph, and calculating to obtain a homography matrix;
and converting the image to be detected by using the homography matrix to obtain the alignment image.
6. The image difference detection method according to claim 5, wherein the matching the first feature point of the image to be detected and the second feature point of the reference image to obtain a matching map comprises:
acquiring a first feature descriptor of the image to be detected and a second feature descriptor of the reference image;
and matching the first characteristic descriptor and the second characteristic descriptor by using a super glue network to obtain the matching graph.
7. The image difference detection method according to claim 4, wherein the performing feature point extraction on the image to be detected and the reference image respectively comprises:
and extracting the feature points of the image to be detected and the reference image by using a feature point detection network, wherein the feature point detection network is obtained by using a corner point marking method for training, and the information marked by the corner points comprises 2D (two-dimensional) graphic information.
8. The image difference detection method according to claim 1, wherein the comparing the first feature map and the second feature map to obtain the image difference information of the first feature map and the reference image includes:
performing difference processing on the first feature map and the second feature map to obtain a difference feature map of the first feature map and the second feature map;
and carrying out binarization, corrosion expansion or connected domain contour processing on the difference characteristic graph to obtain the image difference information.
9. An image difference detection apparatus, characterized in that the image difference detection apparatus comprises a processor and a memory, the memory storing program instructions, the processor being configured to execute the program instructions to implement the image difference detection method according to any one of claims 1-8.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores program instructions that are executable to implement the image difference detection method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110860122.8A CN113674220A (en) | 2021-07-28 | 2021-07-28 | Image difference detection method, detection device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110860122.8A CN113674220A (en) | 2021-07-28 | 2021-07-28 | Image difference detection method, detection device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113674220A true CN113674220A (en) | 2021-11-19 |
Family
ID=78540600
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110860122.8A Pending CN113674220A (en) | 2021-07-28 | 2021-07-28 | Image difference detection method, detection device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113674220A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419039A (en) * | 2022-03-28 | 2022-04-29 | 武汉市融科优品装饰材料有限公司 | Decorative wallpaper defect detection method based on template matching |
| CN115661584A (en) * | 2022-11-18 | 2023-01-31 | 浙江莲荷科技有限公司 | Model training method, open domain target detection method and related device |
-
2021
- 2021-07-28 CN CN202110860122.8A patent/CN113674220A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419039A (en) * | 2022-03-28 | 2022-04-29 | 武汉市融科优品装饰材料有限公司 | Decorative wallpaper defect detection method based on template matching |
| CN115661584A (en) * | 2022-11-18 | 2023-01-31 | 浙江莲荷科技有限公司 | Model training method, open domain target detection method and related device |
| CN115661584B (en) * | 2022-11-18 | 2023-04-07 | 浙江莲荷科技有限公司 | Model training method, open domain target detection method and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108960211B (en) | Multi-target human body posture detection method and system | |
| CN110047095B (en) | Tracking method and device based on target detection and terminal equipment | |
| CN108986152B (en) | Foreign matter detection method and device based on difference image | |
| CN107516322B (en) | Image object size and rotation estimation calculation method based on log polar space | |
| CN109064504B (en) | Image processing method, apparatus and computer storage medium | |
| CN107851196B (en) | Image pattern matching method and device | |
| CN113674220A (en) | Image difference detection method, detection device and storage medium | |
| CN110210584B (en) | High-speed dynamic two-dimensional code identification method based on improved SURF (speeded Up robust features) composite algorithm | |
| Deshpande et al. | A novel modified cepstral based technique for blind estimation of motion blur | |
| CN113724135A (en) | Image splicing method, device, equipment and storage medium | |
| CN113780110A (en) | Method and device for detecting weak and small targets in image sequence in real time | |
| CN108960247B (en) | Image significance detection method and device and electronic equipment | |
| CN111950554A (en) | Identification card identification method, device, equipment and storage medium | |
| Zhang et al. | Depth enhancement with improved exemplar-based inpainting and joint trilateral guided filtering | |
| CN114155285B (en) | Image registration method based on gray histogram | |
| CN104966283A (en) | Imaging layered registering method | |
| CN111179245B (en) | Image quality detection method, device, electronic equipment and storage medium | |
| CN106778822B (en) | Image straight line detection method based on funnel transformation | |
| CN112435278A (en) | Visual SLAM method and device based on dynamic target detection | |
| CN112084365A (en) | Real-time image retrieval method of network camera based on OpenCV and CUDA acceleration | |
| CN111192308A (en) | Image processing method and device, electronic equipment and computer storage medium | |
| Wang et al. | Fast Blur Detection Algorithm for UAV Crack Image Sets | |
| CN112634298B (en) | Image processing method and device, storage medium and terminal | |
| Deng et al. | Texture edge-guided depth recovery for structured light-based depth sensor | |
| CN115345845A (en) | Feature fusion smoke screen interference efficiency evaluation and processing method based on direction gradient histogram and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |