CN111080628B - Image tampering detection method, apparatus, computer device and storage medium - Google Patents
Image tampering detection method, apparatus, computer device and storage medium Download PDFInfo
- Publication number
- CN111080628B CN111080628B CN201911322233.2A CN201911322233A CN111080628B CN 111080628 B CN111080628 B CN 111080628B CN 201911322233 A CN201911322233 A CN 201911322233A CN 111080628 B CN111080628 B CN 111080628B
- Authority
- CN
- China
- Prior art keywords
- image
- feature map
- tampering
- region
- detected
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/15—Correlation function computation including computation of convolution operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Computational Mathematics (AREA)
- Multimedia (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Image Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Artificial Intelligence (AREA)
- Algebra (AREA)
- Quality & Reliability (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
Abstract
The application relates to an image tampering detection method, an image tampering detection device, a computer device and a storage medium. The method in one embodiment comprises: acquiring an image to be detected, and extracting features of the image to be detected to obtain a suspected tampering feature map; obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map; performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map; positioning a potential tampering area of the reconstruction feature map to obtain rough position information of the tampering area; and processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected. The suspected tampering feature map is reconstructed through the channel weight coefficient and the space weight coefficient, the characterization contrast of the true and false areas of the image is enhanced, and the tampering areas can be distinguished more effectively, so that the detection precision of the image tampering areas is improved.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image tampering detection method, an image tampering detection device, a computer device, and a storage medium.
Background
With the popularization of image modification software and the development of digital image tampering technology, people can easily make realistic false images. Tampering techniques for digital images can be generally divided into two categories: content change and content invariance. The former mainly includes three tamper modes: splice, copy-move, and remove. Stitching refers to stitching a certain region in a real image into another image, copying-moving refers to cloning a certain region in a image and stitching in the same image, removing refers to removing a certain region in an image, and then masking a blank region. Content-invariant modifications, such as: gaussian blur, edge smoothing, color enhancement, etc., are typically added to false images as post-processing operations to mask the modification marks. False images, even if carefully inspected, make it difficult for a person to identify the authenticity of a picture.
These false images are widely spread in social media, misleading people, and threatening the credibility of digital images in information propagation. The field of passive evidence collection of digital image content is receiving widespread attention in the face of increasingly realistic tampered images and the consequent series of information security issues. Passive evidence obtaining of digital image content refers to the process of detecting the authenticity of digital image content without prior knowledge. Researchers want to identify the authenticity of images by means of digital image passive evidence techniques, preventing attackers from using false images to convey misleading information.
The conventional image tampering detection method mostly adopts the steps of extracting manually designed or predefined features at the image block or pixel level, such as judging the authenticity of an image or locating the tampered area of the image through discrete cosine transform feature coefficients, scale-invariant feature transformation, color filter array mode features and the like. Most traditional methods are used for carrying out coarse granularity identification aiming at a certain type of digital image tampering technology, and have low detection precision and cannot be truly applied to image evidence collection.
Disclosure of Invention
In view of the above, it is necessary to provide an image falsification detection method, apparatus, computer device, and storage medium that can improve detection accuracy.
An image tamper detection method, the method comprising:
acquiring an image to be detected, and carrying out feature extraction on the image to be detected to obtain a suspected tampering feature map;
obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map;
performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map;
processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of an interested region so as to obtain coordinates of a potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region;
and processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected.
In one embodiment, the processing the image to be detected based on the tampered region rough position information to obtain a tampered region mask of the image to be detected includes:
acquiring a global feature map of the image to be detected;
cutting the global feature map according to the rough position information of the tampered area to obtain a local feature map of a corresponding position;
and decoding the local feature map to obtain the tampered region mask of the image to be detected.
In one embodiment, the acquiring the global feature map of the image to be detected includes:
performing constraint convolution processing on the image to be detected to obtain a suspected tampering noise image;
acquiring an edge feature map of the suspected tampering noise image;
and carrying out fusion processing on the edge feature map and the reconstruction feature map to obtain a global feature map of the image to be detected.
In one embodiment, the decoding the local feature map to obtain the tampered region mask of the image to be detected includes:
extracting the characteristics of the local characteristic map through a preset residual error network module to obtain a target local characteristic map;
processing the target local feature map through a preset decoder to obtain a pixel classification confidence map;
and obtaining the tampered region mask of the image to be detected based on the pixel classification confidence map and a preset classification threshold.
In one embodiment, the feature extraction of the image to be detected, and obtaining the suspected tampering feature map includes:
performing constraint convolution processing on the image to be detected to obtain a suspected tampering noise image;
and extracting the characteristics of the suspected tampering noise image to obtain a suspected tampering characteristic image.
In one embodiment, after the reconstructing the suspected tampering feature map, the reconstructing feature map further includes:
processing the reconstruction feature map through a preset region recommendation network to obtain a potential tampered region;
and carrying out tampering technology identification on the potential tampering area to obtain tampering technology classification information.
An image tampering detection apparatus, the apparatus comprising:
the feature extraction module is used for obtaining an image to be detected, carrying out feature extraction on the image to be detected, and obtaining a suspected tampering feature map;
the weight acquisition module is used for acquiring a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map;
the feature reconstruction module is used for reconstructing the suspected tampering feature map through the multiplication operation of the channel weight coefficient, the space weight coefficient and the suspected tampering feature map element by element, so as to obtain a reconstructed feature map;
the rough positioning module is used for processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of the region of interest so as to obtain coordinates of a potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region;
and the precise segmentation module is used for processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
acquiring an image to be detected, and carrying out feature extraction on the image to be detected to obtain a suspected tampering feature map;
obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map;
performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map;
processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of an interested region so as to obtain coordinates of a potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region;
and processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
acquiring an image to be detected, and carrying out feature extraction on the image to be detected to obtain a suspected tampering feature map;
obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map;
performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map;
processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of an interested region so as to obtain coordinates of a potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region;
and processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected.
According to the image tampering detection method, the device, the computer equipment and the storage medium, the image to be detected is obtained, and the feature extraction is carried out on the image to be detected, so that a suspected tampering feature map is obtained; obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map; performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map; processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of the region of interest so as to obtain coordinates of the potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region; classifying the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected; the suspected tampering feature map is reconstructed through the channel weight coefficient and the space weight coefficient, the characterization contrast of the true and false areas of the image is enhanced, and the tampering areas can be distinguished more effectively, so that the detection precision of the image tampering areas is improved.
Drawings
FIG. 1 is an application environment diagram of an image tamper detection method in one embodiment;
FIG. 2 is a flow chart of a method of image tamper detection in one embodiment;
FIG. 3 is a schematic diagram of a framework of an image tamper detection model in one embodiment;
FIG. 4 is a flow chart of a tamper zone coarse location acquisition step in one embodiment;
FIG. 5 is a flow chart of a global feature map acquisition step in one embodiment;
FIG. 6 is a block diagram of an image tamper detection device in one embodiment;
fig. 7 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The image tampering detection method provided by the application can be applied to an application environment shown in fig. 1. Wherein the client terminal 102 communicates with the server 104 via a network. The user waits for an image to be detected by the client terminal 102. The server 104 acquires an image to be detected, and performs feature extraction on the image to be detected to acquire a suspected tampering feature map; obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map; performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map; processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of the region of interest so as to obtain coordinates of the potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region; and processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected. The client terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, and tablet computers, and the server 104 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, an image tampering detection method is provided, taking an example that the image tampering detection method is applied to the server in fig. 1, and the method includes the following steps:
step 202, obtaining an image to be detected, and extracting features of the image to be detected to obtain a suspected tampering feature map.
And extracting features of the image to be detected, for example, extracting features of a pre-constructed convolution layer to obtain a suspected tampering feature map. Specifically, constraint processing is carried out on pixel points of the convolution kernel, and feature extraction is carried out on the image subjected to the constraint processing through a preset residual error network, so that a suspected tampering feature map is obtained.
Step 204, obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map.
The channel weight coefficient and the space weight coefficient corresponding to the suspected tampering feature map can be obtained through CBAM (Convolutional Block Attention Module, attention mechanism module of convolution module). The CBAM is a simple and effective lightweight attention module, and calculates weights in a convolution characteristic diagram from two angles of space and a channel to obtain a channel weight coefficient and a space weight coefficient respectively.
Step 206, performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map.
The RPN (Region Proposal Network, regional recommendation network) can better distinguish the tampered region and the real region of the image and output rough suspected tampered region position information. Specifically, a CBAM module can be added after the first convolution operation in the RPN, so as to obtain an RPN network with improved attention mechanism. The CBAM module calculates channel weight coefficients and space weight coefficients of the feature map, performs element wise multiplication operation on the two types of weight coefficients and the feature map, and generates a new feature map to obtain a reconstructed feature map.
And step 208, processing the reconstructed feature map through a preset region recommendation network to generate coordinate information of the region of interest so as to obtain coordinates of the potential tampered region, processing the reconstructed feature map according to the coordinates of the potential tampered region to obtain local features of the potential tampered region, and processing the local features of the potential tampered region through a preset full-connection layer to obtain rough position information of the tampered region.
In the above description, after generating the new feature map, the improved regional recommendation network generates the coordinate information of the RoI (Region of Interest ) by using the new feature map, so as to obtain the location information of the potentially tampered region. And cutting corresponding local features in the reconstruction feature map according to the position information of the potential tampered region, then further processing the local features, and finally obtaining rough position information of the tampered region by using the full connection layer. The potential tampered area is an image area selected from images, the area is focused on by image analysis, the area is delineated for further processing, and the processing time can be reduced and the accuracy can be improved by delineating a target.
Step 210, classifying the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected.
And further learning local characteristics of the region of interest by using rough position information of the tampered region, and finally completing pixel-level tampered region segmentation through a decoder. And carrying out pixel-by-pixel processing on the image to be detected based on the rough position information of the tampered area, and distinguishing the real area and the tampered area of the image to be detected, so as to obtain a tampered area mask of the image to be detected. The mask is a template for an image filter, for example, when extracting the tampered region, the tampered region is highlighted by pixel filtering the image through a matrix of n×n. According to the image tampering detection method, the image to be detected is obtained, and feature extraction is carried out on the image to be detected, so that a suspected tampering feature map is obtained; obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map; reconstructing the suspected tampering feature map through the channel weight coefficient and the space weight coefficient to obtain a reconstructed feature map; positioning a potential tampering area of the reconstruction feature map to obtain rough position information of the tampering area; processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area of the image to be detected; the suspected tampering feature map is reconstructed through the channel weight coefficient and the space weight coefficient, the characterization contrast of the true and false areas of the image is enhanced, and the tampering areas can be distinguished more effectively, so that the detection precision of the image tampering areas is improved.
The application provides an image tampering detection model, wherein the frame of the image tampering detection model is shown in fig. 3 and mainly comprises a tampering feature extraction module, a first-stage rough tampering detection module and a second-stage fine tampering detection module. The model adopts a thick-to-thin system structure, and simultaneously realizes the classification of tampering technology and the segmentation of tampering areas, thereby realizing complete and accurate image evidence collection. First, a learnable falsification feature extractor learns a unified feature expression directly from data. Secondly, the attention area recommendation network in the rough tampering detection module effectively distinguishes the real area and the tampered area of the image so as to carry out subsequent tampering technology classification and tampering area rough positioning. The jump structure of the fine tamper detection sum module then fuses the low-level and high-level information to refine the global features. Finally, the coarse localization information will guide the model to learn further finer local features and segment the tampered region.
In one embodiment, tamper detection is performed on an image to be detected, and obtaining a suspected tamper feature map includes: performing constraint convolution processing on the image to be detected to obtain a suspected tampering noise image; and extracting features of the suspected tampering noise image to obtain a suspected tampering feature map. There are high contrast boundary artifacts at the edges of the tampered region, and there are also differences in pattern noise between the tampered region and the real region. Both of these cues can serve as important evidence of passive evidence of digital image content. The constraint convolution layer can be trained together with the deep learning model to improve the results of non-content image tampering evidence obtaining tasks such as image compression, noise, blurring and the like. The constraint convolution layer can retain richer features, and simultaneously completes the modeling process of pattern noise, so that a double-flow feature extraction structure is not needed, the parameter quantity of a model can be reduced by half, the effect of adaptively extracting image tampering features is achieved, and the generalization capability of the feature extraction module is enhanced. Therefore, a constraint convolution layer is combined with a convolution neural network, and a novel self-adaptive tampering feature extraction module is provided, and can directly model the change relation among pixels of a digital image, and self-adaptively extract image tampering features by combining with a Resnet101 residual network.
Based on the analysis, the self-adaptive tampering feature extraction module can realize tampering detection of the image. As shown in fig. 3, the constrained convolution layer is applied to the res net101 residual network before it is used as an image preprocessing process that can be learned. The feature extraction module takes as input a 3-channel RGB image I of size h×w. Assuming w is a convolution kernel of the constraint convolution layer, the size is 5×5, k is the number of convolution kernels of the constraint convolution layer, k=3 is set, coordinates (0, 0) are coordinates of intermediate elements of the convolution kernel, the value of the intermediate elements of the constraint convolution kernel is-1, and the sum of all other elements is 1, namely the following constraint condition is implemented:
performing a convolution kernel constraint process after back propagation: in each back propagation process, the constraint convolution layer parameters complete weight updating together with the residual error network. Next, each convolution kernel intermediate element is limited to-1, all elements except the intermediate element are normalized, and the sum of the limited elements is 1.
In one embodiment, as shown in fig. 4, step 208, the reconstructed feature map is processed through a preset region recommendation network to generate coordinate information of the region of interest, so as to obtain coordinates of the potentially tampered region; processing the reconstruction feature map according to the coordinates of the potential tampering area to obtain local features of the potential tampering area; processing local features of the potential tampered area through a preset full-connection layer, wherein obtaining rough position information of the tampered area specifically comprises the following steps: step 402, processing the reconstruction feature map through a preset region recommendation network to obtain coordinates of a potential tampered region; step 404, processing the reconstruction feature map according to the coordinates of the potential tampered region to obtain local features of the potential tampered region; and step 406, processing the local features of the potential tampered area through a preset full-connection layer to obtain rough position information of the tampered area. Because the ResNet101 residual network in FIG. 3 performs feature extraction on the noise image generated by the constraint convolution layer, partial semantic information is lost, so that the similarity of the representation among the region classes is larger, and the subsequent bounding box regression and the pixel-level tampered region segmentation precision are influenced. Considering that the region recommendation network can distinguish the tampered region and the real region of the image, the CBAM module is added in the RPN, so that the improved RPN network can endow the tampered region with higher weight, and the characterization contrast between the real region and the false region is enhanced, thereby overcoming the adverse effect of semantic feature deletion on the follow-up boundary regression and segmentation. Specifically, in the constructed image tampering detection model, a CBAM module is added after the first convolution operation in the RPN, so as to obtain the RPN network with improved attention mechanism.
In one embodiment, the reconstructing the suspected tampering feature map, after obtaining the reconstructed feature map, further includes: processing the reconstruction feature map through a preset region recommendation network to obtain a potential tampering region; tamper technique for potentially tampered areasAnd (5) identifying to obtain tampering technology classification information. Specifically, the reconstruction feature map is processed through a preset region recommendation network to obtain coordinates of a potential tampered region; cutting the reconstruction feature map according to the coordinates of the potential tampering area to obtain local features of the potential tampering area, and carrying out tampering technology identification on the local features of the potential tampering area through a preset full-connection layer to obtain tampering technology classification information. The CBAM module calculates the weights of the feature map channel and the space, and performs element-by-element multiplication operation on the two types of weights and the feature map to generate a new feature map. Subsequently, the improved region recommendation network generates coordinates t of the region of interest using the new feature map i And classification probability p of each tampering technique i . And obtaining the classification probability and coordinates of each tampering technology at the boundary box level through the region-of-interest alignment mode and the full connection layer. In the improved RPN network, the tampered area is defined as a foreground, and the real area is a background, so that a specific target may not exist in the output interested area, for example, in the process of removing tampering, the tampered area is selected to be covered by a non-specific target area with a high probability. Whereas in conventional target detection tasks, the RPN defines the region where a specific target exists as a foreground. This indicates that the improved RPN network has the ability to search for tampered areas, rather than searching for targets.
To train the modified RPN network, the loss function of the modified RPN network (RPN-A) is defined as:
wherein p is i Represents the probability that the anchor window anchor i is a tampered area in a training batch miniband,
a category label representing the anchor window anchor i. t is t i Four-dimensional coordinate values representing preliminary regression of anchor window anchor i, ++>
Four-dimensional labels are coordinates of anchor windows anchor i. />
Cross entropy loss representing RPN-A network foreground and background classification, < >>
Indicating the RPN-A output potential bounding box coordinate smoothl 1 (L1 smoothing) loss. N (N) cls For the training batch size in RPN-A, N reg Is the number of anchor window anchor positions. Lambda is a super parameter used to balance the two losses, default to 10. After RPN-A, the RoI alignment module adjusts the local features in the region of interest to the same size, and then achieves tamper technique classification and bounding box regression, i.e., coarse localization, through the fully connected layers and softmax layers.
To complete the first stage training, considering that this is A multi-task learning process, the sum of the RPN-A network loss, the subsequent classification loss, and the bounding box regression loss is taken as the first stage loss, and the first stage loss function is defined as:
wherein, the liquid crystal display device comprises a liquid crystal display device,
loss function for RPN-A network part, < ->
For the final cross entropy classification loss, +.>
Loss for the final bounding box smoothl 1. The three partial loss functions are added to produce the first stage loss function.
In one embodiment, processing an image to be detected based on rough location information of a tampered region to obtain a tampered region mask of the image to be detected includes: acquiring a global feature map of an image to be detected; cutting the global feature map according to the rough position information of the tampered area to obtain a local feature map of a corresponding position; and decoding the local feature map to obtain the tampered region mask of the image to be detected. In the second stage of fine tampering detection, the local features in the bounding box are further learned by using the rough position information obtained by the first stage of rough tampering detection, and finally, pixel level tampering region segmentation is completed through a decoder.
In one embodiment, as shown in fig. 5, acquiring a global feature map of an image to be detected includes: step 502, performing constraint convolution processing on an image to be detected to obtain a suspected tampering noise image; step 504, obtaining an edge feature map of the suspected tampering noise image; and step 506, carrying out fusion processing on the edge feature map and the reconstructed feature map to obtain a global feature map of the image to be detected. As shown in fig. 3, the image tamper detection model designs A new skip structure, and adopts element-by-element addition operation to fuse the edge feature map extracted from the shallow layer of the Resnet101 residual network with the advanced features of the CBAM module in the RPN-A network, so as to further enhance the global feature map. To match the conv5_x portion of the Resnet101 residual network, a single full convolution operation is used to boost the number of channels to 1024, resulting in an enhanced global profile. Meanwhile, the boundary frame coordinates output in the first stage are used as rough positioning information, and the guiding model further focuses on local features in the boundary frame.
In one embodiment, decoding the local feature map to obtain a tampered region mask of the image to be detected includes: extracting the characteristics of the local characteristic map through a preset residual error network module to obtain a target local characteristic map; processing the target local feature map through a preset decoder to obtain a pixel classification confidence map; and obtaining a tampered region mask of the image to be detected based on the pixel classification confidence map and a preset classification threshold. Let the number of coarse position information generated in the first stage be N and the bounding box coordinate dimension be N x 4. The corresponding position features are clipped on the enhanced global feature map using the region of interest alignment RoI Align module and resized to nx7x7x1024. Subsequently, the local feature information is input to the Resnet 101conv5_x for further learning, and the final local feature with dimensions n×7×7×2048 is output. The adjusted local features are input to a decoder that includes a deconvolution operation and two full convolution operations. The partial feature map is up-sampled to nx7x7x256 using a deconvolution operation, and only the real region and the tampered region are distinguished for each pixel due to the class-agnostic pixel classification approach. The first full convolution operation is thus used as a buffer to reduce the number of channels of the feature map, which becomes Nx7x7x64 in dimension. After the last full convolution operation, a pixel confidence map inside the bounding box with dimensions Nx7x7x2 is obtained using the softmax function.
In the training process, binary cross entropy loss is used as a second stage loss function, which is defined as:
wherein, the liquid crystal display device comprises a liquid crystal display device,
label, y representing the i-th pixel i Representing the prediction probability of the ith pixel, N pix And N cls The number of pixels and the number of categories are represented, respectively. />
Is an indication function if +.>
The function is 1 and otherwise 0.
After the first stage and the second stage are completed, three tasks of tamper technology classification, tamper area bounding box rough prediction and pixel level tamper area segmentation are completed. The overall loss function may be defined as:
the image tampering detection model adopts an end-to-end training mode, and in an experiment, if the Resnet101 weight pre-trained in the ImageNet database is directly used for initializing the model, the model can generate gradient explosion phenomenon when the first stage and the second stage are jointly trained. Thus, in the training process, the first stage is first pre-trained in the COCO integrated dataset, and this model is taken as a pre-training model. On four standard tampered image data sets, initializing weights by adopting a pre-training model, and performing end-to-end training in a first stage and a second stage.
The image is scaled to a minimum size of 600 pixels before entering the image tamper detection model. Image flipping is applied to data enhancement. The training batch size of RPN-A was 256, the batch size of the fine tamper detection training was 4, and the test was 8. In the COCO integrated dataset pre-training, the model 110K steps are trained. The initial learning rate was 0.001, and after 40K was reduced to 0.0001,90K, the initial learning rate was reduced to 0.00001. The whole experiment is carried out on a 1080Ti GPU (Graphics Processing Unit, graphic processor), and the experimental result shows that the model achieves the most advanced performance. On the NIST16, COVERAGE and Columbia datasets, the F1 score was increased by 28.4%, 73.2% and 13.3%, respectively.
It should be understood that, although the steps in the flowcharts of fig. 2, 4-5 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps of fig. 2, 4-5 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the sub-steps or stages are performed necessarily occur in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, an image tampering detection apparatus is provided, as shown in fig. 6, which includes a feature extraction module 602, a weight acquisition module 604, a feature reconstruction module 606, a coarse localization module 608, and a fine segmentation module 610. The feature extraction module 602 is configured to obtain an image to be detected, perform feature extraction on the image to be detected, and obtain a suspected tampering feature map. The weight obtaining module 604 is configured to obtain a channel weight coefficient and a spatial weight coefficient corresponding to the suspected tampering feature map. The feature reconstruction module 606 is configured to reconstruct the suspected tampered feature map by performing element-by-element multiplication on the channel weight coefficient and the spatial weight coefficient and the suspected tampered feature map, so as to obtain a reconstructed feature map. The rough positioning module 608 is configured to process the reconstructed feature map through a preset region recommendation network, and generate coordinate information of the region of interest, so as to obtain coordinates of the potential tampered region; processing the reconstruction feature map according to the coordinates of the potential tampering area to obtain local features of the potential tampering area; and processing the local characteristics of the potential tampered area through a preset full-connection layer to obtain rough position information of the tampered area. The precise segmentation module 610 is configured to process the image to be detected based on the rough location information of the tampered region, so as to obtain a tampered region mask of the image to be detected.
In one embodiment, the accurate segmentation module is further configured to obtain a global feature map of the image to be detected; cutting the global feature map according to the rough position information of the tampered area to obtain a local feature map of a corresponding position; and decoding the local feature map to obtain the tampered region mask of the image to be detected.
In one embodiment, the accurate segmentation module is further configured to perform constraint convolution processing on the image to be detected to obtain a suspected tamper noise image; acquiring an edge feature map of the suspected tampering noise image; and carrying out fusion processing on the edge feature map and the reconstructed feature map to obtain a global feature map of the image to be detected.
In one embodiment, the accurate segmentation module is further configured to perform feature extraction on the local feature map through a preset residual error network module, so as to obtain a target local feature map; processing the target local feature map through a preset decoder to obtain a pixel classification confidence map; and obtaining a tampered region mask of the image to be detected based on the pixel classification confidence map and a preset classification threshold.
In one embodiment, the feature extraction module is further configured to perform constraint convolution processing on the image to be detected to obtain a suspected tamper noise image; and extracting features of the suspected tampering noise image to obtain a suspected tampering feature map.
In one embodiment, the image tampering detection device further comprises a tampering technology classification module, which is used for processing the reconstruction feature map through a preset region recommendation network to obtain a potential tampering region; and carrying out tampering technology identification on the potential tampering area to obtain tampering technology classification information.
For specific limitations of the image tampering detection apparatus, reference may be made to the above limitations of the image tampering detection method, and no further description is given here. The respective modules in the above-described image tamper detection apparatus may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer equipment is used for storing data such as images to be detected, image tampering detection models, detection results and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method of image tamper detection.
It will be appreciated by those skilled in the art that the structure shown in fig. 7 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory storing a computer program and a processor implementing steps in the image tamper detection method of any of the embodiments when the computer program is executed.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor implements the steps of the image tamper detection method of any of the embodiments.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.
Claims (10)
1. A method of image tamper detection, the method comprising:
acquiring an image to be detected, and carrying out feature extraction on the image to be detected to obtain a suspected tampering feature map;
obtaining a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map;
performing element-by-element multiplication operation on the suspected tampering feature map through the channel weight coefficient and the space weight coefficient, and reconstructing the suspected tampering feature map to obtain a reconstructed feature map;
processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of an interested region so as to obtain coordinates of a potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region;
and processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected.
2. The method of claim 1, wherein the processing the image to be detected based on the tampered region rough position information to obtain a tampered region mask of the image to be detected comprises:
acquiring a global feature map of the image to be detected;
cutting the global feature map according to the rough position information of the tampered area to obtain a local feature map of a corresponding position;
and decoding the local feature map to obtain the tampered region mask of the image to be detected.
3. The method of claim 2, wherein the acquiring the global feature map of the image to be detected comprises:
performing constraint convolution processing on the image to be detected to obtain a suspected tampering noise image;
acquiring an edge feature map of the suspected tampering noise image;
and carrying out fusion processing on the edge feature map and the reconstruction feature map to obtain a global feature map of the image to be detected.
4. The method according to claim 2, wherein the decoding the local feature map to obtain the tampered region mask of the image to be detected includes:
extracting the characteristics of the local characteristic map through a preset residual error network module to obtain a target local characteristic map;
processing the target local feature map through a preset decoder to obtain a pixel classification confidence map;
and obtaining the tampered region mask of the image to be detected based on the pixel classification confidence map and a preset classification threshold.
5. The method of claim 1, wherein the feature extracting the image to be detected to obtain a suspected tampering feature map comprises:
performing constraint convolution processing on the image to be detected to obtain a suspected tampering noise image;
and extracting the characteristics of the suspected tampering noise image to obtain a suspected tampering characteristic image.
6. The method of claim 1, wherein the reconstructing the suspected tampering feature map, after obtaining a reconstructed feature map, further comprises:
processing the reconstruction feature map through a preset region recommendation network to obtain a potential tampered region;
and carrying out tampering technology identification on the potential tampering area to obtain tampering technology classification information.
7. An image tampering detection apparatus, the apparatus comprising:
the feature extraction module is used for obtaining an image to be detected, carrying out feature extraction on the image to be detected, and obtaining a suspected tampering feature map;
the weight acquisition module is used for acquiring a channel weight coefficient and a space weight coefficient corresponding to the suspected tampering feature map;
the feature reconstruction module is used for reconstructing the suspected tampering feature map through the multiplication operation of the channel weight coefficient, the space weight coefficient and the suspected tampering feature map element by element, so as to obtain a reconstructed feature map;
the rough positioning module is used for processing the reconstruction feature map through a preset region recommendation network to generate coordinate information of the region of interest so as to obtain coordinates of a potential tampering region, processing the reconstruction feature map according to the coordinates of the potential tampering region to obtain local features of the potential tampering region, and processing the local features of the potential tampering region through a preset full-connection layer to obtain rough position information of the tampering region;
and the precise segmentation module is used for processing the image to be detected based on the rough position information of the tampered area to obtain a tampered area mask of the image to be detected.
8. The apparatus of claim 7, wherein the accurate segmentation module is further configured to obtain a global feature map of the image to be detected; cutting the global feature map according to the rough position information of the tampered area to obtain a local feature map of a corresponding position; and decoding the local feature map to obtain the tampered region mask of the image to be detected.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911322233.2A CN111080628B (en) | 2019-12-20 | 2019-12-20 | Image tampering detection method, apparatus, computer device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911322233.2A CN111080628B (en) | 2019-12-20 | 2019-12-20 | Image tampering detection method, apparatus, computer device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111080628A CN111080628A (en) | 2020-04-28 |
| CN111080628B true CN111080628B (en) | 2023-06-20 |
Family
ID=70316033
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911322233.2A Active CN111080628B (en) | 2019-12-20 | 2019-12-20 | Image tampering detection method, apparatus, computer device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111080628B (en) |
Families Citing this family (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111553916B (en) * | 2020-05-09 | 2023-11-14 | 中科计算技术创新研究院 | Image tampering area detection method based on various features and convolutional neural network |
| CN111797737A (en) * | 2020-06-22 | 2020-10-20 | 重庆高新区飞马创新研究院 | Remote sensing target detection method and device |
| CN113537206B (en) * | 2020-07-31 | 2023-11-10 | 腾讯科技（深圳）有限公司 | Push data detection method, push data detection device, computer equipment and storage medium |
| CN112116565B (en) * | 2020-09-03 | 2023-12-05 | 深圳大学 | Method, apparatus and storage medium for generating countersamples for falsifying a flip image |
| CN114338956A (en) * | 2020-09-30 | 2022-04-12 | 北京小米移动软件有限公司 | Image processing method, image processing apparatus, and storage medium |
| CN112233077A (en) * | 2020-10-10 | 2021-01-15 | 北京三快在线科技有限公司 | Image analysis method, device, equipment and storage medium |
| CN112541409B (en) * | 2020-11-30 | 2021-09-14 | 北京建筑大学 | Attention-integrated residual network expression recognition method |
| CN112508039B (en) * | 2020-12-08 | 2024-04-02 | 中国银联股份有限公司 | Image detection method and device |
| CN112614111B (en) * | 2020-12-24 | 2023-09-05 | 南开大学 | Video tampering operation detection method and device based on reinforcement learning |
| CN112561907B (en) * | 2020-12-24 | 2022-11-01 | 南开大学 | Video tampering operation detection method and device based on double-current network |
| CN112733625B (en) * | 2020-12-28 | 2022-06-14 | 华南理工大学 | False face video tampering detection method and system based on time domain self-attention mechanism |
| CN112699888A (en) * | 2020-12-31 | 2021-04-23 | 上海肇观电子科技有限公司 | Image recognition method, target object extraction method, device, medium and equipment |
| CN112750122B (en) * | 2021-01-21 | 2022-08-02 | 山东省人工智能研究院 | Image tampering area positioning method based on double-current boundary perception neural network |
| CN112861960B (en) * | 2021-02-03 | 2022-09-02 | 湖南大学 | Image tampering detection method, system and storage medium |
| CN112950564B (en) * | 2021-02-23 | 2022-04-01 | 北京三快在线科技有限公司 | Image detection method and device, storage medium and electronic equipment |
| CN113205096B (en) * | 2021-04-26 | 2022-04-15 | 武汉大学 | Attention-based combined image and feature self-adaptive semantic segmentation method |
| CN113012152B (en) * | 2021-04-27 | 2023-04-14 | 深圳大学 | Image tampering chain detection method and device and electronic equipment |
| CN113269730B (en) * | 2021-05-11 | 2022-11-29 | 北京三快在线科技有限公司 | Image processing method, image processing device, computer equipment and storage medium |
| CN113469906B (en) * | 2021-06-24 | 2023-02-07 | 湖南大学 | Cross-layer global and local perception network method for image restoration |
| CN113453012B (en) * | 2021-06-25 | 2023-02-28 | 杭州海康威视数字技术股份有限公司 | Encoding and decoding method and device and electronic equipment |
| CN114764858B (en) * | 2022-06-15 | 2022-11-01 | 深圳大学 | Copy-paste image identification method and device, computer equipment and storage medium |
| CN114913401B (en) * | 2022-07-13 | 2022-09-30 | 江苏烨明光电有限公司 | Welding equipment for LED lamp core column and shell and welding quality monitoring method thereof |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544703A (en) * | 2013-10-19 | 2014-01-29 | 侯俊 | Digital image stitching detecting method |
| WO2014042514A2 (en) * | 2012-09-12 | 2014-03-20 | Mimos Berhad | A surveillance system and a method for tampering detection and correction |
| CN106228557A (en) * | 2016-07-26 | 2016-12-14 | 浙江捷尚视觉科技股份有限公司 | Distorted image method for detecting area based on secondary JPEG compression |
| CN110349136A (en) * | 2019-06-28 | 2019-10-18 | 厦门大学 | A kind of tampered image detection method based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9514537B2 (en) * | 2013-12-27 | 2016-12-06 | Xerox Corporation | System and method for adaptive depth map reconstruction |
-
2019
- 2019-12-20 CN CN201911322233.2A patent/CN111080628B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014042514A2 (en) * | 2012-09-12 | 2014-03-20 | Mimos Berhad | A surveillance system and a method for tampering detection and correction |
| CN103544703A (en) * | 2013-10-19 | 2014-01-29 | 侯俊 | Digital image stitching detecting method |
| CN106228557A (en) * | 2016-07-26 | 2016-12-14 | 浙江捷尚视觉科技股份有限公司 | Distorted image method for detecting area based on secondary JPEG compression |
| CN110349136A (en) * | 2019-06-28 | 2019-10-18 | 厦门大学 | A kind of tampered image detection method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111080628A (en) | 2020-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111080628B (en) | Image tampering detection method, apparatus, computer device and storage medium | |
| WO2020238560A1 (en) | Video target tracking method and apparatus, computer device and storage medium | |
| CN112132156B (en) | Image saliency target detection method and system based on multi-depth feature fusion | |
| Zhuo et al. | Self-adversarial training incorporating forgery attention for image forgery localization | |
| KR101896357B1 (en) | Method, device and program for detecting an object | |
| US20230021661A1 (en) | Forgery detection of face image | |
| WO2022105125A1 (en) | Image segmentation method and apparatus, computer device, and storage medium | |
| CN111369581A (en) | Image processing method, device, equipment and storage medium | |
| US11392800B2 (en) | Computer vision systems and methods for blind localization of image forgery | |
| CN111914654B (en) | Text layout analysis method, device, equipment and medium | |
| US20220180624A1 (en) | Method and device for automatic identification of labels of an image | |
| Wang et al. | An unsupervised heterogeneous change detection method based on image translation network and post-processing algorithm | |
| Qin et al. | Face inpainting network for large missing regions based on weighted facial similarity | |
| US20240013516A1 (en) | Method and system for deep learning based image feature extraction | |
| CN116798041A (en) | Image recognition method and device and electronic equipment | |
| Zhang et al. | Symmetry-aware face completion with generative adversarial networks | |
| Wang et al. | A multi-scale attentive recurrent network for image dehazing | |
| CN111582057A (en) | Face verification method based on local receptive field | |
| Vijayalakshmi K et al. | Copy-paste forgery detection using deep learning with error level analysis | |
| CN110781817A (en) | Pedestrian re-identification method for solving component misalignment | |
| CN113920377A (en) | Method of classifying image, computer device, and storage medium | |
| CN114463764A (en) | Table line detection method and device, computer equipment and storage medium | |
| CN111914850B (en) | Picture feature extraction method, device, server and medium | |
| CN114155363A (en) | Converter station vehicle identification method and device, computer equipment and storage medium | |
| Chai et al. | Three-stage image forgery localization with shallow feature enhancement and attention |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |