CN117351015B - Tamper detection method and system based on edge supervision and multi-domain cross correlation - Google Patents
Tamper detection method and system based on edge supervision and multi-domain cross correlation Download PDFInfo
- Publication number
- CN117351015B CN117351015B CN202311648686.0A CN202311648686A CN117351015B CN 117351015 B CN117351015 B CN 117351015B CN 202311648686 A CN202311648686 A CN 202311648686A CN 117351015 B CN117351015 B CN 117351015B
- Authority
- CN
- China
- Prior art keywords
- domain
- edge
- tampering
- feature
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 30
- 238000010586 diagram Methods 0.000 claims abstract description 12
- 238000000605 extraction Methods 0.000 claims description 42
- 238000000034 method Methods 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000010606 normalization Methods 0.000 claims description 15
- 230000004913 activation Effects 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 10
- 230000008030 elimination Effects 0.000 claims description 9
- 238000003379 elimination reaction Methods 0.000 claims description 9
- 230000004927 fusion Effects 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 9
- 238000013527 convolutional neural network Methods 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 4
- 238000012360 testing method Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000003708 edge detection Methods 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/95—Pattern authentication; Markers therefor; Forgery detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of image processing, and discloses a tamper detection method and a tamper detection system based on edge supervision and multi-domain cross correlation, wherein the tamper detection method comprises the following steps: step 1, extracting a source domain tampered domain feature, step 2, performing tampered domain edge positioning and tampered domain positioning by using decoupling edge supervision, and extracting a tampered domain feature: step 3, extracting source domain characteristics: the source domain tampering domain characteristics obtained in the step 1 are subjected to difference with the tampering domain characteristics obtained in the step 2, so that source domain characteristics are obtained; step 4, multi-domain cross correlation modeling is carried out, and a source domain tampering domain background diagram is obtained: and carrying out multi-domain cross correlation modeling on the source domain tampering domain features, the tampering domain features and the source domain features to obtain a source domain tampering domain background diagram. The invention improves the accuracy of the positioning of the source domain tampering domain, monitors the final result and improves the accuracy of judging the source domain tampering domain and the background.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a tamper detection method and system based on edge supervision and multi-domain cross correlation.
Background
The image copying-moving image tampering operation refers to copying a certain area in an image and moving to other areas in the same image, thereby achieving the purpose of confusing and decepting the observer. The tampering mode is the same as splicing, and the content of the image is tampered, but compared with the splicing technology, the tampering mode has higher detection difficulty, and is caused by that the copy-move operation is the same image internal operation, and the tampered area is very similar to other areas in the image in statistical properties. Aiming at the characteristics, researchers provide a parallel network for auxiliary positioning of tampered domains and utilization of characteristic information to realize copying mobile image tampering, and the method has the advantages that a main line branch and an auxiliary branch of the parallel network are used for detecting a source domain and a tampered domain, so that the positioning accuracy of a similar target area is improved, and in addition, the edge is used as a supervision condition for detecting the tampered domain, so that the detection accuracy of the tampered domain is improved.
However, this method has the following problems: first, the redundant use of the edge features of the tampered domain as the supervision information results in poor detection of the tampered domain. The prior method uses a tampered domain as an auxiliary domain to optimize a copy-mobile tampered detection result, and the tampered domain is extracted by detecting boundary artifacts left in the tampered process as auxiliary information, but the prior method for directly and simply adding different layer feature detection edges to assist in detecting the tampered domain is unreasonable, which leads to poor detection result of the tampered domain, thereby affecting the whole network. This is because, in the conventional tamper zone detection edge location, in order to make full use of the low-level shallow features, each layer of features is directly added and utilized without processing to perform edge detection, however, the low-level information is already contained in the high-level features, and the direct addition and utilization cause a large amount of redundancy of information and submerge useful information in a large amount of useless information. Second, auxiliary domain feature fusion is not reasonably utilized. The prior method directly adds and fuses the auxiliary domain serving as the tampered domain and the main network for detecting the similar region, thereby realizing the auxiliary function of the tampered domain. However, this direct fusion method does not fully exploit the auxiliary role of the tampered domain for similar region detection. In addition, the existing method simply locates the tampered domain and uses the tampered domain to assist in detecting the main line network of the similar area, and does not use the source domain feature to further assist the main line network.
Disclosure of Invention
Aiming at the defects existing in the prior art, the invention provides a tamper detection method and a tamper detection system based on edge supervision and multi-domain cross correlation, and provides a feature decoupling edge supervision module, wherein when multi-layer features are utilized for edge extraction, deep features and shallow features are utilized simultaneously by a layer-by-layer decoupling method, and effective information of the deep features, which is lost in the shallow features, is excavated while the deep features are utilized; the invention provides a multi-domain cross correlation modeling module, which utilizes the association relation among source domain features, tampering domain features and main line network features to calculate elements, thereby optimizing the main line network features (namely source domain tampering domain features) by utilizing the tampering domain and the source domain, fusing the three optimized features through a gating mechanism to realize reasonable utilization of effective information of the three features and obtain a final source domain tampering domain background diagram.
In order to solve the technical problems, the invention adopts the following technical scheme:
first, the present invention provides a tamper detection method based on edge supervision and multi-domain cross correlation, comprising the steps of:
step 1, extracting source domain tampering domain characteristics:
performing feature extraction on an input image I by using a convolutional neural network, performing feature extraction on different scales by using convolution kernels with different sizes on the obtained features, then fusing the features with different scales, and performing autocorrelation calculation on the fused features to obtain primary source domain tampering domain features;
step 2, performing tampering domain edge positioning and tampering domain positioning by using decoupling edge supervision, and extracting tampering domain features:
carrying out feature extraction on the input image I in different layers by adopting a ResNet network in four layers, and respectively obtaining the output of the first layer to the fourth layer, wherein the output of the fourth layer is used as a tampering domain feature; the obtained four-layer output is subjected to edge feature extraction, redundant feature elimination and splicing through a four-level edge residual error network, and the output of the last-level network is the tampered domain edge feature;
step 3, extracting source domain characteristics:
the source domain tampering domain characteristics obtained in the step 1 are subjected to difference with the tampering domain characteristics obtained in the step 2, so that source domain characteristics are obtained;
step 4, multi-domain cross correlation modeling is carried out, and a source domain tampering domain background diagram is obtained:
the multi-domain cross correlation modeling is to respectively perform pairwise correlation calculation on the source domain tampering domain features, the tampering domain features and the source domain features, calculate the association relationship among elements to obtain three similar features, and fuse the obtained similar features through a gating mechanism to obtain a source domain tampering domain background map.
Further, in step 2, the output size of the first layer of the res net network is 1/4 of the input image I size, the number of channels is the same, the output of the second layer is halved compared with the output size of the first layer, the number of channels is doubled, the output of the third layer is halved compared with the output size of the first layer, the number of channels is doubled, the output of the fourth layer is the same as the output size of the third layer, and the number of channels is doubled.
Further, in step 2, the first-stage edge residual network includes a filter and an edge residual module, and the second-stage edge residual network includes a filter and two edge residual modules; and carrying out edge feature extraction on the obtained four-layer output through a filter and an edge residual error module, carrying out redundant feature elimination and splicing after the edge residual error is carried out on the features of the rear three layers, inputting the obtained four-layer output into an edge residual error module, entering a next-stage network, and outputting the last edge residual error module of the last-stage network to obtain the edge feature of the tampered domain.
Further, the specific method for extracting the edge characteristics of the tampered domain through the edge residual error network is as follows: for an input image I, the four layers of output through a ResNet network are respectively the characteristics M1, M2, M3 and M4, and the four characteristics are respectively processed through a filter module and an edge residual error module to obtain the characteristics J1, J2, J3 and J4; the method comprises the steps of obtaining K1 by taking difference between J2 and J1, obtaining K2 by taking difference between J3 and J2, obtaining K3 by taking difference between J4 and J3, splicing J1 and K1 to obtain P1, obtaining R1 by passing through an edge residual error module, obtaining P2 by splicing R1 and K2, obtaining R2 by passing through an edge residual error module, obtaining P3 by splicing R2 and K3, obtaining R3 by passing through an edge residual error module, and taking R3 as the edge characteristic of a tampered domain.
Further, the two modules of the filter and the edge residual are used for channel reduction and feature extraction, and data obtained after convolution operation, batch normalization operation, L2 normalization operation and activation function activation operation are sequentially multiplied with input data in the filter; in the edge residual error module, first convolution operation is carried out, and then data obtained after second convolution operation, batch normalization operation, linear rectification operation and third convolution operation are added with data obtained after the first convolution operation.
Further, in step 4, the specific method for modeling multi-domain cross correlation is as follows: performing dimension transformation on the source domain tampering domain feature, the tampering domain feature and the source domain feature to respectively obtain three features of a feature F1, a feature F2 and a feature F3, performing correlation calculation on the F1, the F2, the F1, the F3, the F2 and the F3 to obtain three new features, namely a similar feature Z1, a similar feature Z2 and a similar feature Z3, performing gating 1, gating 2 and 1-gating 2 operation on the Z1, the Z2 and the Z3 respectively, multiplying the feature subjected to the gating operation with the corresponding original similar feature Z1, the similar feature Z2 and the similar feature Z3 to obtain features Y1, Y2 and Y3, and finally adding the three features of Y1, Y2 and Y3 to obtain a final source domain tampering domain background image.
The invention further provides a tamper detection system based on edge supervision and multi-domain cross correlation, which is used for realizing the tamper detection method based on the edge supervision and multi-domain cross correlation, and comprises a main line network, an auxiliary branch network and a multi-domain information fusion network, wherein the main line network adopts a convolutional neural network to extract features, the obtained features use convolution kernels with different sizes to extract features with different scales, then the features with different scales are fused, and the fused features use autocorrelation calculation to obtain primary source domain tamper features;
the auxiliary branch network is a tampered domain feature extraction module, the tampered domain feature extraction module adopts a decoupling edge supervision module and comprises a tampered domain feature positioning extraction module for extracting tampered domain features and a tampered domain edge positioning module for extracting tampered domain edge features, the tampered domain feature positioning extraction module adopts a ResNet network, performs feature extraction of different layers on an input image I in four layers, and respectively obtains outputs of a first layer to a fourth layer, wherein the output of the fourth layer is used as the tampered domain features; the tampered domain edge positioning module comprises a four-level edge residual error network, the edge residual error network of the first level comprises a filter and an edge residual error module, the edge residual error network of the latter three levels comprises the filter and two edge residual error modules, the obtained output of four layers of ResNet networks respectively carries out edge feature extraction, redundant feature elimination and splicing through the four-level edge residual error network, and the output of the last level network is the tampered domain edge feature;
the multi-domain information fusion network comprises a multi-domain cross correlation modeling module, the source domain tampering domain characteristics obtained by the main line network and the tampering domain characteristics obtained by the auxiliary branch network are subjected to difference to obtain source domain characteristics, the multi-domain cross correlation modeling is used for carrying out correlation calculation on the source domain tampering domain characteristics, the tampering domain characteristics and the source domain characteristics respectively, calculating the association relation among elements to obtain three similar characteristics, and fusing the obtained similar characteristics through a gating mechanism to obtain a source domain tampering domain background image which is used as an output image of the system.
Further, the filter comprises a convolution layer, a batch normalization layer, an L2 normalization layer and an activation function, and data obtained after the activation operation are multiplied with input data; the edge residual error module comprises a first convolution layer, a second convolution layer, a batch normalization layer, a linear rectification layer and a third convolution layer, and data obtained after the third convolution layer is added with data obtained after the first convolution operation.
Further, the loss Lsum includes two parts, namely edge loss and cross entropy loss, and the loss function formula is as follows:
(1)；
l edges is edge loss, and the formula is as follows:
(2)；
where H and W are the height and width of the input image respectively,for a tampered domain edge feature value of a certain pixel position (i, j) in the input image ++>Is the value of a certain pixel position (i, j) in the real image, < >>For the value of a pixel position (i, j) in the source domain tampering domain background diagram, i and j are the i-th position on the row and the j-th position on the column respectively;
l ce is the cross entropy loss, and the formula is as follows:
(3)；
wherein,representing summing all categories +.>Is the +.f in the true category label vector>The number of elements to be added to the composition,is the probability of predictionFirst->The elements.
Compared with the prior art, the invention has the advantages that:
(1) The edge monitoring module based on characteristic decoupling is designed, and a brand new tampered domain edge positioning mode is used, so that shallow characteristic information can be considered and deep characteristic information can be used when the edge is positioned, redundant information operation is reduced, data content is simplified, and repeated data operation is reduced. And the edge of the tampered domain is used for supervision, so that the accuracy of the edge positioning of the tampered domain is improved, further, the edge learning parameters are optimized, and the accuracy of the positioning of the tampered domain is improved.
(2) The multi-domain cross correlation modeling module is designed, all obtained characteristic information (source domain tampering domain background information, tampering domain background information and source domain background information) is fully utilized, further information mining is carried out on the obtained characteristic information (source domain tampering domain background information, tampering domain characteristics and source domain tampering domain characteristics), and the association relation among elements is calculated by utilizing the source domain characteristics, the tampering domain characteristics and the source domain tampering domain characteristics, so that main line network characteristics (namely source domain tampering domain characteristics) are optimized by utilizing two auxiliary domains of the tampering domain and the source domain, and the optimized three source domain tampering domain background characteristics are fused through a gating mechanism, so that reasonable utilization of effective information of the three source domain tampering domain characteristics is realized, the accuracy of positioning of the source domain tampering domain information is improved, the final result is supervised, and the accuracy of source domain tampering domain and background judgment is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a system block diagram of the present invention;
FIG. 2 is a schematic diagram of a decoupling edge monitoring module according to the present invention;
FIG. 3 is a schematic diagram of a filter structure according to the present invention;
FIG. 4 is a schematic diagram of an edge residual module structure according to the present invention;
FIG. 5 is a schematic diagram of a multi-domain cross-correlation modeling module according to the present invention.
Detailed Description
The invention will be further described with reference to the accompanying drawings and specific examples.
Example 1
With reference to fig. 1, this embodiment provides a tamper detection system based on edge supervision and multi-domain cross correlation, which includes a main line network, an auxiliary branch network and a multi-domain information fusion network, where the input image I of the system has a size of w×h×c, and W, H, C is width, height and dimension, respectively.
Firstly, for a main line network, a convolutional neural network (such as a VGG16 network) is adopted to perform feature extraction, three convolutional kernels with different sizes are used for performing feature extraction with different scales on the obtained features, then the features with different scales are fused, and the fused features are subjected to autocorrelation calculation to obtain the primary source domain tampering domain features.
Secondly, for auxiliary branches, redundant information is reduced for the tampering domain feature extraction module, and meanwhile, edges are used for supervision, so that the accuracy of the whole branch is improved. Specifically, the auxiliary branch adopts a decoupling edge supervision module, and tamper domain edge detection and tamper domain positioning are performed by using decoupling edge supervision to obtain tamper domain characteristics. After extracting the tampered domain features, the source domain tampered domain features of the main line network and the tampered domain features of the auxiliary branches are used for making differences, so that the source domain features are obtained, and further fusion is carried out.
As a preferred embodiment, the decoupling edge supervision module includes a tampered domain feature positioning extraction module for extracting tampered domain features and a tampered domain edge positioning module for extracting tampered domain edge features, as shown in fig. 2, where the tampered domain feature positioning extraction module adopts a res net network (for example, a res net50 backbone network, including four parts of a module 1, a module 2, a module 3 and a module 4, as four network layers), performs feature extraction of different layers on an input image I in four layers, and obtains outputs of the first layer to the fourth layer respectively, where the output of the fourth layer is taken as the tampered domain feature.
It should be noted that, this embodiment is only illustrated by the following proportional relationship between the image size and the number of channels: the output size of the first layer is 1/4 of the input image size, the number of channels is the same, the output of the second layer is halved compared with the output size of the first layer, the number of channels is doubled, the output of the third layer is halved compared with the output size of the first layer, the number of channels is doubled, the output of the fourth layer is the same as the output size of the third layer, and the number of channels is doubled. Under this condition, the technical effect is the best. However, for different network layers, other output image sizes and channel numbers may implement the technical solution, which is not illustrated here.
As a preferred embodiment, the tampered domain edge positioning module comprises a four-stage edge residual network, the edge residual network of the first stage comprises a filter and an edge residual module, the edge residual network of the last stage comprises a filter and two edge residual modules, the four-layer output of the obtained res net network is subjected to edge feature extraction, redundant feature elimination and splicing through the four-stage edge residual network, and the output of the last stage network is the tampered domain edge feature.
As a preferred embodiment, as shown in fig. 3, the filter includes a convolution layer, a batch normalization layer, an L2 normalization layer, and an activation function, and the data obtained after the activation operation is multiplied by the input data.
As a preferred embodiment, as shown in fig. 4, the edge residual module includes a first convolution layer, a second convolution layer, a batch normalization layer, a linear rectification layer, and a third convolution layer, where the data obtained after the third convolution layer is added to the data obtained after the first convolution operation.
And then, the source domain tampering domain characteristics obtained by the main line network and the tampering domain characteristics obtained by the auxiliary branch network are subjected to difference to obtain the source domain characteristics.
Finally, as shown in fig. 5, the multi-domain information fusion network comprises a multi-domain cross correlation modeling module, wherein the multi-domain cross correlation modeling module is used for performing correlation computation on source domain tampered domain features, tampered domain features and source domain features, calculating association relations among elements to obtain three similar features, and fusing the obtained similar features through a gating mechanism to obtain a source domain tampered domain background image serving as an output image of the system.
As a preferred embodiment, three features of the source domain tampering domain feature, the tampering domain feature and the source domain feature are respectively obtained by dimension transformation, namely a feature F1, a feature F2 and a feature F3, three new features are obtained by correlation calculation of the F1, the F2, the F1, the F3, the F2 and the F3, namely a similar feature Z1, a similar feature Z2 and a similar feature Z3, then the Z1, the Z2 and the Z3 are respectively subjected to gating 1, gating 2 and 1-gating 2 operation, the features subjected to the gating operation are multiplied by the corresponding original similar feature Z1, the similar feature Z2 and the similar feature Z3 to obtain features Y1, Y2 and Y3, and finally the three features of Y1, Y2 and Y3 are added to obtain a final source domain tampering domain background map.
The functions and data processing procedures of each module are described in detail in the section of embodiment 2, in conjunction with a tamper detection method based on edge supervision and multi-domain cross correlation.
Example 2
Referring to fig. 1, the present embodiment designs a tamper detection method based on edge supervision and multi-domain cross correlation, which includes the following steps:
and step 1, extracting the source domain tampering domain characteristics.
And performing feature extraction on the input image I by using a convolutional neural network, performing feature extraction on the obtained features by using convolution kernels with different sizes, then fusing the features with different scales, and performing autocorrelation calculation on the fused features to obtain the primary source domain tampering domain features.
And 2, performing tampering domain edge positioning and tampering domain positioning by using decoupling edge supervision, and extracting tampering domain features.
Firstly, for tampered domain positioning, the ResNet network (comprising four parts of a module 1, a module 2, a module 3 and a module 4) is adopted to carry out feature extraction of different layers on an input image I, and outputs of a first layer to a fourth layer are respectively obtained, wherein the output of the fourth layer is used as tampered domain features.
As a preferred embodiment, the output size of the first layer is 1/4 of the input image size, the number of channels is the same, the output of the second layer is halved compared to the output size of the first layer, the number of channels is doubled, the output of the third layer is halved compared to the output size of the first layer, the number of channels is doubled, the output of the fourth layer is the same as the output size of the third layer, and the number of channels is doubled. For example, the first layer is acquired for a given W×H×3 color input image, respectively，/>256, second layer->，/>512, third layer->，/>1024, fourth layer->，/>2048.
Then, tamper domain edge localization is introduced. And respectively carrying out edge feature extraction, redundant feature elimination and splicing on the four-layer output of the ResNet network through a four-level edge residual error network, wherein the output of the last-level edge residual error network is the edge feature of the tampered domain.
Specifically, the first stage of the edge residual network comprises a filter and one edge residual module, and the last three stages of the edge residual network comprises a filter and two edge residual modules. With reference to fig. 3 and fig. 4, two modules of a filter and an edge residual are used for channel reduction and feature extraction, and in the filter, convolution operation (3×3×2 convolution is taken as an example in the figure), batch normalization operation, L2 normalization operation and activation function activation operation are sequentially performed to obtain data, and the data is multiplied by input data; in the edge residual module, first convolution operation (1×1 convolution is exemplified in the figure) is performed, and then data obtained after second convolution operation (3×3 convolution is exemplified in the figure), batch normalization operation, linear rectification operation and third convolution operation (3×3 convolution is exemplified in the figure) are sequentially added with the data obtained after the first convolution operation.
The design idea of the step is as follows: and carrying out edge feature extraction on the output of the four layers of the acquired ResNet network through a filter and an edge residual error module respectively, carrying out redundant feature elimination and splicing after edge residual error is carried out on the features of the rear three layers, inputting the edge residual error module, entering the next-stage network, and obtaining the edge feature of the tampered domain as the output of the last edge residual error module of the last-stage network. That is, the specific method for extracting the edge features of the tampered domain through the edge residual network is as follows:
for a W×H×3 color input image, features M1, M2, M3, and M4 are obtained after passing through modules 1, 2, 3, and 4 of ResNet network, and are respectively sent to a filter and an edge residual module for tampering domain edge features，/>Further extraction of 1. Specifically, the four features are respectively passed through a filter module and an edge residual error module to obtain features J1, J2, J3 and J4; obtaining K1 by taking the difference between J2 and J1, obtaining K2 by taking the difference between J3 and J2, obtaining K3 by taking the difference between J4 and J3, splicing J1 and K1 to obtain P1, obtaining R1 by passing through an edge residual error module, splicing R1 and K2 to obtain P2, obtaining R2 by passing through an edge residual error module, and splicing R2 and K3And obtaining P3, obtaining R3 by the P3 through an edge residual error module, and taking the R3 as the edge characteristic of the tampered domain.
And 3, extracting source domain characteristics.
And (3) making a difference between the source domain tampering domain characteristics obtained in the step (1) and the tampering domain characteristics obtained in the step (2) to obtain source domain characteristics.
And 4, carrying out multi-domain cross correlation modeling to obtain a source domain tampering domain background image.
The multi-domain cross correlation modeling is to respectively perform pairwise correlation calculation on the source domain tampering domain features, the tampering domain features and the source domain features, calculate the association relationship among elements to obtain three similar features, and fuse the obtained similar features through a gating mechanism to obtain a source domain tampering domain background map.
The specific method for modeling the multi-domain cross correlation is as follows: performing dimension transformation on the source domain tampering domain feature, the tampering domain feature and the source domain feature to obtain three features of a feature F1, a feature F2 and a feature F3 respectively, performing correlation calculation on the F1, the F2, the F1, the F3, the F2 and the F3 to obtain three new features, namely a similar feature Z1, a similar feature Z2 and a similar feature Z3 respectively, performing gating 1, gating 2 and 1-gating 2 operation on the Z1, the Z2 and the Z3 respectively, multiplying the feature subjected to the gating operation with the corresponding original similar feature Z1, the similar feature Z2 and the similar feature Z3 to obtain features Y1, Y2 and Y3, and finally adding the three features of Y1, Y2 and Y3 to obtain a final source domain tampering domain background image, as shown in fig. 5.
The obtained source domain tampering domain background image, namely the output image of the invention, comprises a background, a source domain and a tampering domain, for example, blue represents the background, green represents the source domain, and red represents the tampering domain, when the source domain tampering domain background image is only red, green and blue. The network performance is evaluated by using the accuracy, recall and F1-score for the source domain, the tampered domain and the background domain respectively, and the higher the three indexes are, the higher the accuracy of the judgment of the method is, and the more accurate the judgment area is.
It should be noted that, the loss Lsum of the present invention includes two parts, namely edge loss and cross entropy loss, and the loss function formula is as follows:
(1)。
l edges is edge loss, is used for assisting in optimizing edge detection of branch tampering domain, and has the following formula:
(2)；
where H and W are the height and width of the input image I respectively,for the tampered domain edge feature value of a certain pixel position (I, j) in the input image I +.>Is the value of a certain pixel position (i, j) in the real image, < >>The value of a certain pixel position (i, j) in the background map of the source domain, i.e. the output image of the system, i and j are the i-th position on the row and the j-th position on the column respectively. />The absolute error of the output image and the real image is shown, but we are more concerned with the information of the edges, so thatThe unweighted, the duty cycle of the edge portion becomes larger, and this loss can be better optimized for the edge information.
L ce is cross entropy loss and is used for optimizing the final detection result after fusion, and the formula is as follows:
(3)；
wherein,representing summing all categories +.>Is the +.f in the true category label vector>The number of elements to be added to the composition,is the +.f in the predicted probability distribution vector>The elements.
In experiments, five data sets of CASIA-CMFD, coMoFoD-CMFD, MICC-F220, USCISI-CMFD, COVERAGE were used. Training and testing was performed using CASIA-CMFD, and testing was performed using CoMoFoD-CMFD, COVERAGE, MICC-F220, USCISI-CMFD four data sets. The CASIA CMFD dataset contains 1313 counterfeit images and their true counterparts (2626 samples total) divided into training, validation and test sets at a ratio of 8:1:1. The CoMoFoD dataset contained 5000 counterfeit images, 200 base images and 25 operation categories, covering 5 operations and 5 post-processing methods. MICC-F220 contains 22400 images of 220 categories, where each category contains 100 images. The USCISI-CMFD dataset had 100K images for training, validation and testing. An end-to-end batch training method was used, with each batch set to 8.
In summary, the invention designs a tamper detection method and system based on edge supervision and multi-domain cross correlation, which are used for detecting image copy-mobile tamper operation. The invention is based on a characteristic decoupling edge supervision module, and when the edge extraction is carried out by utilizing multi-layer characteristics, the deep characteristics and the shallow characteristics are utilized simultaneously by a layer-by-layer decoupling party, and effective information of the deep characteristics, which is lost by the deep characteristics and exists in the shallow characteristics, is excavated while the deep characteristics are utilized.
The invention provides a multi-domain cross correlation modeling module, which utilizes the association relation among source domain features, tampered domain features and main line network features to calculate elements, thereby optimizing the main line network features by utilizing two auxiliary domains of the tampered domain and the source domain. And fusing the optimized three source domain tampering domain background features through a gating mechanism to realize reasonable utilization of the three effective information (source domain tampering domain background information, tampering domain background information and source domain background information). Thereby improving the accuracy of the positioning of the source domain tampering domain information, supervising the final result and improving the accuracy of judging the source domain tampering domain and the background.
It should be understood that the above description is not intended to limit the invention to the particular embodiments disclosed, but to limit the invention to the particular embodiments disclosed, and that various changes, modifications, additions and substitutions can be made by those skilled in the art without departing from the spirit and scope of the invention.
Claims (5)
1. The tamper detection method based on edge supervision and multi-domain cross correlation is characterized by comprising the following steps of:
step 1, extracting source domain tampering domain characteristics:
performing feature extraction on an input image I by using a convolutional neural network, performing feature extraction on different scales by using convolution kernels with different sizes on the obtained features, then fusing the features with different scales, and performing autocorrelation calculation on the fused features to obtain primary source domain tampering domain features;
step 2, performing tampering domain edge positioning and tampering domain positioning by using decoupling edge supervision, and extracting tampering domain features:
carrying out feature extraction on the input image I in different layers by adopting a ResNet network, and respectively obtaining the output of the first layer to the fourth layer, wherein the ResNet network comprises four parts of a module 1, a module 2, a module 3 and a module 4 as four network layers, and the output of the fourth layer is used as a tampering domain feature; the obtained four-layer output is subjected to edge feature extraction, redundant feature elimination and splicing through a four-level edge residual error network, and the output of the last-level network is the tampered domain edge feature;
the edge residual error network of the first stage comprises a filter and an edge residual error module, and the edge residual error network of the rear three stages comprises the filter and two edge residual error modules; the obtained four-layer output is subjected to edge feature extraction through a filter and an edge residual error module respectively, the features of the rear three layers are subjected to edge residual error, redundant feature elimination and splicing are performed, the obtained four-layer output is input into an edge residual error module, the obtained four-layer output enters a next-stage network, and the output of the last edge residual error module of the last-stage network is the edge feature of the tampered domain;
the filter and the edge residual error are used for channel reduction and feature extraction, and data obtained after convolution operation, batch normalization operation, L2 normalization operation and activation function activation operation are sequentially multiplied with input data in the filter; in the edge residual error module, first convolution operation is carried out, and then data obtained after second convolution operation, batch normalization operation, linear rectification operation and third convolution operation are added with data obtained after the first convolution operation;
the specific method for extracting the edge characteristics of the tampered domain through the edge residual error network comprises the following steps: for an input image I, the four layers of output through a ResNet network are respectively the characteristics M1, M2, M3 and M4, and the four characteristics are respectively processed through a filter module and an edge residual error module to obtain the characteristics J1, J2, J3 and J4; obtaining K1 by taking the difference between J2 and J1, obtaining K2 by taking the difference between J3 and J2, obtaining K3 by taking the difference between J4 and J3, splicing J1 and K1 to obtain P1, obtaining R1 by passing through an edge residual error module, splicing R1 and K2 to obtain P2, obtaining R2 by passing through an edge residual error module, splicing R2 and K3 to obtain P3, obtaining R3 by passing through an edge residual error module, and taking R3 as the edge characteristic of a tampered domain;
step 3, extracting source domain characteristics:
the source domain tampering domain characteristics obtained in the step 1 are subjected to difference with the tampering domain characteristics obtained in the step 2, so that source domain characteristics are obtained;
step 4, multi-domain cross correlation modeling is carried out, and a source domain tampering domain background diagram is obtained:
the multi-domain cross correlation modeling is to respectively perform pairwise correlation calculation on the source domain tampering domain features, the tampering domain features and the source domain features, calculate the association relationship among elements to obtain three similar features, and fuse the obtained similar features through a gating mechanism to obtain a source domain tampering domain background map.
2. The tamper detection method based on edge supervision and multi-domain cross correlation according to claim 1, wherein in step 2, the output size of the first layer of the res net network is 1/4 of the input image I size, the number of channels is the same, the output of the second layer is halved compared to the output size of the first layer, the number of channels is doubled, the output of the third layer is halved compared to the output size of the first layer, the number of channels is doubled, the output of the fourth layer is the same as the output size of the third layer, and the number of channels is doubled.
3. The tamper detection method based on edge supervision and multi-domain cross correlation according to claim 1, wherein in step 4, the multi-domain cross correlation modeling method is as follows: performing dimension transformation on the source domain tampering domain feature, the tampering domain feature and the source domain feature to respectively obtain three features of a feature F1, a feature F2 and a feature F3, performing correlation calculation on the F1, the F2, the F1, the F3, the F2 and the F3 to obtain three new features, namely a similar feature Z1, a similar feature Z2 and a similar feature Z3, performing gating 1, gating 2 and 1-gating 2 operation on the Z1, the Z2 and the Z3 respectively, multiplying the feature subjected to the gating operation with the corresponding original similar feature Z1, the similar feature Z2 and the similar feature Z3 to obtain features Y1, Y2 and Y3, and finally adding the three features of Y1, Y2 and Y3 to obtain a final source domain tampering domain background image.
4. The tamper detection system based on edge supervision and multi-domain cross correlation is characterized by being used for realizing the tamper detection method based on edge supervision and multi-domain cross correlation according to any one of claims 1-3, wherein the system comprises a main line network, an auxiliary branch network and a multi-domain information fusion network, the main line network adopts a convolutional neural network to perform feature extraction, the obtained features use convolution kernels with different sizes to perform feature extraction with different scales, then the features with different scales are fused, and the fused features use autocorrelation calculation to obtain preliminary source domain tamper domain features;
the auxiliary branch network is a tampered domain feature extraction module, the tampered domain feature extraction module adopts a decoupling edge supervision module and comprises a tampered domain feature positioning extraction module for extracting tampered domain features and a tampered domain edge positioning module for extracting tampered domain edge features, the tampered domain feature positioning extraction module adopts a ResNet network, performs feature extraction of different layers on an input image I in four layers, and respectively obtains outputs of a first layer to a fourth layer, wherein the output of the fourth layer is used as the tampered domain features; the tampered domain edge positioning module comprises a four-level edge residual error network, the edge residual error network of the first level comprises a filter and an edge residual error module, the edge residual error network of the latter three levels comprises the filter and two edge residual error modules, the obtained output of four layers of ResNet networks respectively carries out edge feature extraction, redundant feature elimination and splicing through the four-level edge residual error network, and the output of the last level network is the tampered domain edge feature;
the multi-domain information fusion network comprises a multi-domain cross correlation modeling module, the source domain tampering domain characteristics obtained by the main line network and the tampering domain characteristics obtained by the auxiliary branch network are subjected to difference to obtain source domain characteristics, the multi-domain cross correlation modeling is used for carrying out correlation calculation on the source domain tampering domain characteristics, the tampering domain characteristics and the source domain characteristics respectively, calculating the association relation among elements to obtain three similar characteristics, and fusing the obtained similar characteristics through a gating mechanism to obtain a source domain tampering domain background image which is used as an output image of the system.
5. The tamper detection system based on edge supervision and multi-domain cross correlation according to claim 4, wherein the loss Lsum comprises two parts, edge loss and cross entropy loss, the loss function formula is as follows:
Lsum＝Ledges+Lce (1)；
l edges is edge loss, and the formula is as follows:
wherein H and W are the height and width of the input image, E i，j For a tampered domain edge feature value at a pixel position (i, j) in the input image, Y i，j X is the value of a pixel position (i, j) in the real image i，j For the value of a pixel position (i, j) in the source domain tampering domain background diagram, i and j are the i-th position on the row and the j-th position on the column respectively;
lce is the cross entropy loss and the formula is as follows:
Lce＝-∑ g [y g *log(p g )] (3)；
wherein, sigma g Representing the summation of all classes, y g To be the g-th element in the true class label vector, p g Is the g-th element in the predicted probability distribution vector.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311648686.0A CN117351015B (en) | 2023-12-05 | 2023-12-05 | Tamper detection method and system based on edge supervision and multi-domain cross correlation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311648686.0A CN117351015B (en) | 2023-12-05 | 2023-12-05 | Tamper detection method and system based on edge supervision and multi-domain cross correlation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117351015A CN117351015A (en) | 2024-01-05 |
| CN117351015B true CN117351015B (en) | 2024-03-19 |
Family
ID=89357914
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311648686.0A Active CN117351015B (en) | 2023-12-05 | 2023-12-05 | Tamper detection method and system based on edge supervision and multi-domain cross correlation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117351015B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118279913B (en) * | 2024-06-03 | 2024-08-20 | 中国海洋大学 | Intra-domain modeling replication mobile tampering detection method based on semantic segmentation |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014061922A1 (en) * | 2012-10-17 | 2014-04-24 | 에스케이텔레콤 주식회사 | Apparatus and method for detecting camera tampering using edge image |
| CN114022506A (en) * | 2021-11-16 | 2022-02-08 | 天津大学 | Image restoration method with edge prior fusion multi-head attention mechanism |
| CN114972857A (en) * | 2022-05-19 | 2022-08-30 | 中国海洋大学 | Double branch source and target positioning copy mobile detection network and method |
| CN115063373A (en) * | 2022-06-24 | 2022-09-16 | 山东省人工智能研究院 | Social network image tampering positioning method based on multi-scale feature intelligent perception |
| CN116863319A (en) * | 2023-09-05 | 2023-10-10 | 中国海洋大学 | Copy mobile tamper detection method based on cross-scale modeling and alternate refinement |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10778412B2 (en) * | 2017-12-28 | 2020-09-15 | Intel Corporation | Multi-domain convolutional neural network |
-
2023
- 2023-12-05 CN CN202311648686.0A patent/CN117351015B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014061922A1 (en) * | 2012-10-17 | 2014-04-24 | 에스케이텔레콤 주식회사 | Apparatus and method for detecting camera tampering using edge image |
| CN114022506A (en) * | 2021-11-16 | 2022-02-08 | 天津大学 | Image restoration method with edge prior fusion multi-head attention mechanism |
| CN114972857A (en) * | 2022-05-19 | 2022-08-30 | 中国海洋大学 | Double branch source and target positioning copy mobile detection network and method |
| CN115063373A (en) * | 2022-06-24 | 2022-09-16 | 山东省人工智能研究院 | Social network image tampering positioning method based on multi-scale feature intelligent perception |
| CN116863319A (en) * | 2023-09-05 | 2023-10-10 | 中国海洋大学 | Copy mobile tamper detection method based on cross-scale modeling and alternate refinement |
Non-Patent Citations (3)
| Title |
|---|
| "Edge-aware Regional Message Passing Controller for Image Forgery Localization";D. Li等;《2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)》;全文 * |
| 基于深度监督残差网络的肝脏及肝肿瘤分割;张家兵;张耀;徐洪丽;沈舒宁;王冬;刘同波;刘坤;王彬华;;集成技术(第03期);全文 * |
| 基于级联卷积神经网络的图像篡改检测算法;毕秀丽;魏杨;肖斌;李伟生;马建峰;;电子与信息学报(第12期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117351015A (en) | 2024-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111914907B (en) | Hyperspectral image classification method based on deep learning space-spectrum combined network | |
| CN110852316B (en) | Image tampering detection and positioning method adopting convolution network with dense structure | |
| CN117351015B (en) | Tamper detection method and system based on edge supervision and multi-domain cross correlation | |
| CN115063373A (en) | Social network image tampering positioning method based on multi-scale feature intelligent perception | |
| Cao et al. | Ancient mural restoration based on a modified generative adversarial network | |
| CN111611861B (en) | Image change detection method based on multi-scale feature association | |
| CN109801232A (en) | A kind of single image to the fog method based on deep learning | |
| CN110826588A (en) | Drainage pipeline defect detection method based on attention mechanism | |
| CN114332008B (en) | Unsupervised defect detection and positioning method based on multi-level feature reconstruction | |
| CN116645369B (en) | Anomaly detection method based on twin self-encoder and two-way information depth supervision | |
| Li et al. | ConvTransNet: A CNN–transformer network for change detection with multiscale global–local representations | |
| CN115346037B (en) | Image tampering detection method | |
| Pan et al. | Reducing semantic confusion: Scene-aware aggregation network for remote sensing cross-modal retrieval | |
| CN113537023A (en) | Method for detecting semantic change of remote sensing image | |
| CN114998703B (en) | Remote sensing image change detection method based on high-resolution convolutional neural network | |
| CN116994044A (en) | Construction method of image anomaly detection model based on mask multi-mode generation countermeasure network | |
| CN114972202A (en) | Ki67 pathological cell rapid detection and counting method based on lightweight neural network | |
| Wang et al. | SwinCrack: Pavement crack detection using convolutional swin-transformer network | |
| CN118230052A (en) | Cervical panoramic image few-sample classification method based on visual guidance and language prompt | |
| CN118365543A (en) | Crack image shadow removing method based on improvement ENLIGHTENGAN | |
| Xu et al. | Multi-scale Attention Network for Detection and Localization of Image Splicing Forgery | |
| CN115456957B (en) | Method for detecting change of remote sensing image by full-scale feature aggregation | |
| CN113686788B (en) | Conventional water quality monitoring system and method based on remote sensing wave band combination | |
| CN115587967A (en) | Fundus image optic disk detection method based on HA-UNet network | |
| Li et al. | BAFormer: A Novel Boundary-Aware Compensation UNet-like Transformer for High-Resolution Cropland Extraction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |