CN107704877B - Image privacy perception method based on deep learning - Google Patents
Image privacy perception method based on deep learning Download PDFInfo
- Publication number
- CN107704877B CN107704877B CN201710928967.XA CN201710928967A CN107704877B CN 107704877 B CN107704877 B CN 107704877B CN 201710928967 A CN201710928967 A CN 201710928967A CN 107704877 B CN107704877 B CN 107704877B
- Authority
- CN
- China
- Prior art keywords
- privacy
- image
- perception
- bilinear
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000008447 perception Effects 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000013135 deep learning Methods 0.000 title claims abstract description 16
- 238000012549 training Methods 0.000 claims abstract description 20
- 238000013528 artificial neural network Methods 0.000 claims abstract description 15
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 15
- 238000013526 transfer learning Methods 0.000 claims abstract description 3
- 239000013598 vector Substances 0.000 claims description 29
- 238000011176 pooling Methods 0.000 claims description 21
- 238000004364 calculation method Methods 0.000 claims description 11
- 239000011159 matrix material Substances 0.000 claims description 9
- 230000009467 reduction Effects 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000009825 accumulation Methods 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 claims description 2
- 238000012935 Averaging Methods 0.000 claims 1
- 230000002708 enhancing effect Effects 0.000 claims 1
- 238000012360 testing method Methods 0.000 abstract description 7
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 230000006872 improvement Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000000605 extraction Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011217 control strategy Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Bioethics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image privacy perception method based on deep learning, which comprises the following steps: s1, constructing a privacy classification data set with class labels, and training a privacy perception network by using a transfer learning method; s2, completing the identification of the private image by using a deep convolutional neural network facing to privacy perception; and S3, extracting an attention distribution map according to the deep convolution characteristics of the neural network, and positioning the attention concentration area to finish the perception of the image privacy area. The invention has the beneficial effects that: end-to-end training and testing are completed based on the deep neural network, privacy images can be accurately distinguished, privacy areas in the images can be accurately located, and selective protection can be conveniently carried out on privacy information in the images.
Description
Technical Field
The invention relates to artificial intelligence, in particular to an image privacy perception method based on deep learning.
Background
Privacy awareness is an important prerequisite in a privacy protection process, an image is one of the most important information types in the current social network, and privacy awareness of massive image data becomes particularly critical. In view of the strong subjectivity of the concept of privacy, the existing image privacy perception methods usually define the image privacy in a general sense (such as personal certificates, family photos, confidential document snapshots, etc.) or utilize the labeling of user individuals on the social network. The existing method mainly has the following defects:
firstly, in the aspect of private image feature extraction, most of the existing methods use the traditional image feature extraction methods, such as SIFT features, RGB features, color histograms and the like, and have certain limitations on feature expression capability, cannot represent deep semantic-level features, and have poor generalization capability of classification models.
Secondly, some existing methods for perceiving image privacy need to use extra information to achieve acceptable perception effects, such as an image description label of subjective marking of an image by a user or an access control strategy set by the user, however, the information is difficult to obtain under most conditions, requirements for application scenes are also harsh, and the model is poor in universality.
The existing image privacy perception method only completes image-level privacy perception, namely, whether the whole image is a privacy image is distinguished, and an image privacy area is not perceived. However, in practical applications, it is sometimes necessary to perform blocking or blurring processing on a privacy area of an image to achieve the purpose of privacy protection.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides an image privacy perception method based on deep learning.
The invention provides an image privacy perception method based on deep learning, which comprises the following steps:
s1, constructing a privacy classification data set with class labels, and training a privacy perception network by using a transfer learning method;
s2, completing the identification of the private image by using a deep convolutional neural network facing to privacy perception;
and S3, extracting an attention distribution map according to the deep convolution characteristics of the neural network, and positioning the attention concentration area to finish the perception of the image privacy area.
As a further improvement of the present invention, step S1 includes: the method comprises the steps of pre-training a deep convolutional neural network model on a large-scale image data set, then constructing a privacy classification data set, and finely adjusting the pre-trained deep convolutional neural network model on the privacy classification data set.
As a further improvement of the present invention, step S2 includes: and adding a bilinear operation layer after the last convolutional layer of the deep convolutional neural network to enhance the characteristic expression capability of the deep convolutional neural network model, and simultaneously changing the full-connection layer into a pooling layer.
As a further improvement of the present invention, step S3 includes: and obtaining a weighted high-level feature map as an attention distribution map according to the corresponding relation between the weight of each node of the pooling layer and the feature map subjected to bilinear operation, and positioning the privacy area in the original map through scale conversion.
As a further improvement of the invention, the bilinear operation layer mainly calculates the result of dot product between two feature images after convolution,
let original feature map set M ═ { M ═ M1,m2,…,mn1}，
The output bilinear feature map set is M '═ M'1,m’2,…,m’n1×n1},
The formula for the conversion is as follows:
wherein "
"dot product of a representative matrix"
"represents rounding up, n1 represents the number of original feature maps, and i represents the subscript of the bilinear feature map.
As a further improvement of the invention, dimension reduction operation is carried out on the bilinear feature map.
As a further improvement of the invention, the dimension reduction operation is carried out on the bilinear feature map by adopting a Tensor Sketch algorithm.
As a further improvement of the invention, the bilinear feature map is a matrix of c w h, the input of the Tensor Sketch algorithm is a vector, and each position in the bilinear feature map is sequentially calculated when the Tensor Sketch algorithm is used, namely, the w h c-dimensional vectors are respectively operated and remapped into the w h d-dimensional space; firstly, randomly generating a parameter set h for carrying out hash operationk∈{1,…,d}c，sk∈{1,-1}c(k ═ 1,2), where hkFor storing the remapped indices of the input vectors, skRandom negation of each element value of the input vector is realized; according to the parameter set, obtaining a remapped Count Sketch vector through accumulation calculation; as known from the convolution theorem, the convolution of the time domain or the space domain is equal to the corresponding frequency domainThe product of (a); therefore, the two Count Sketch vectors are converted into the frequency domain by using fast fourier transform, the products of the two Count Sketch vectors in the frequency domain are obtained, and then the products are converted back into the space domain by inverse fourier transform, and the convolution of the Count Sketch vectors is obtained through calculation.
As a further improvement of the invention, the fully-connected layer is changed into an average pooling layer, the average pooling layer performs pooling operation on the whole feature map, and the elements of each feature map are averaged to finally obtain a d-dimensional vector.
As a further improvement of the invention, the average pooling layer node has a corresponding relation with the characteristic diagram, and an attention distribution diagram is obtained through the weighted summation of the characteristic diagrams;
let the bilinear feature map sequence P after dimension reduction be { P ═ P1,p2,…,pd}，
And if the finally generated attention distribution graph is A, the calculation formula is as follows:
where n2 is the class label to which the input image is classified,
representing the connection weight for the k-th node of the pooling layer corresponding to category n 2;
and positioning the local of the privacy image according to the result, wherein the specific method comprises the steps of carrying out scale change on the attention distribution map obtained in the step, converting the scale change into the size of an original image, setting a threshold value to complete image binarization, and solving a minimum external matrix of the binarized image as the local perception result of the privacy image.
The invention has the beneficial effects that: the end-to-end training and testing are completed based on the deep neural network, the private image and the non-private image can be accurately distinguished, the private area in the image can be positioned, the private information in the image can be selectively protected conveniently, and good prerequisites are provided for the privacy protection process. From the aspect of method advancement, the method effectively solves the problems of low accuracy, poor generalization capability, dependence on extra information of a user and the like of the traditional privacy perception method, and extends the privacy perception from the integral perception of the image to the perception of the image privacy area under the condition of not increasing a training neural network model.
Drawings
FIG. 1 is a flowchart of an image privacy perception method based on deep learning according to the invention.
FIG. 2 is a deep convolutional neural network structure diagram of an image privacy perception method based on deep learning.
Detailed Description
The invention is further described with reference to the following description and embodiments in conjunction with the accompanying drawings.
As shown in fig. 1 to 2, a method for image privacy perception based on deep learning mainly includes:
constructing a privacy data set: collecting related images and marking according to privacy and non-privacy;
pre-training a neural network: training a deep convolutional neural network on a large-scale image dataset (e.g., ImageNet);
neural network improvement and training: improving the pre-trained neural network, and finely adjusting the private image data set;
image overall privacy perception: automatically judging whether the input image is a privacy image;
image privacy zone perception: a privacy zone in an image is automatically detected.
And in the neural network improvement and training step, the pre-trained convolutional neural network is improved, a bilinear operation layer is added after the last convolutional layer, the characteristic expression capability of the model is enhanced, and meanwhile, the full-connection layer is changed into a pooling layer, so that the foundation is laid for privacy area perception.
Image privacy zone awareness does not require network retraining. According to the invention, a weighted high-level feature map is obtained according to the corresponding relation between the weight of each node of the classification network pooling layer and the feature map subjected to bilinear operation, an attention distribution map is obtained through scale change, and an attention concentrated area is positioned as a privacy area.
The specific implementation mode of each step is as follows:
constructing a privacy data set: in order to improve the data set construction efficiency, the first n alternative images of Baidu and Google image search are selected in a keyword search mode. Keywords mainly relate to categories of certificate photos, family/group photos, file snapshots, etc. In the process of acquiring the keywords, a correlation model (for example, word2vec and GloVe models trained by a large amount of corpora) capable of calculating the similarity between words is used to help generate similar words of the input keywords, so that privacy keywords are increased, and more images can be conveniently searched. Then, the few privacy-independent images obtained by the manual screening search were collected into 4384 privacy images. For non-private images, 200 types of common objects in the ImageNet data set are selected, 4800 images are randomly extracted, and finally, the method comprises the following steps of 1: 1, the training set and the test set are divided, so that the training and the testing of the subsequent neural network are facilitated.
Pre-training a neural network: in the step, the deep convolutional neural network is trained on an ImageNet large-scale image data set. The ImageNet dataset contains about 120 million images, which relate to 1000 classes of common objects. The reason for pre-training is that the private data set is small, and the deep convolutional neural network has more parameters, so that the direct training is difficult to converge. If pre-training is firstly carried out on a large-scale data set, the neural network obtains better initial weight, and meanwhile, certain characteristic expression capability is obtained, so that rapid convergence can be achieved on a small data set, and a better classification effect is obtained. The pretrained neural network uses the VGG16 convolutional neural network with a good effect at present, and the VGG16 comprises 16 convolutional layers and 2 full-connection layers, so that a good effect can be achieved on a general classification task.
Neural network improvement and training: the pre-trained model is first refined and trained on a private data set. The main improvements are as follows:
for the pre-trained VGG16 network, a bilinear operation layer is added after the last convolutional layer, so that the characteristic expression capability of the model is enhanced. The bilinear operation layer mainly calculates the feature map after convolutionThe result of dot multiplication between two points is set as the original feature map set M ═ M1,m2,…,mn1The output bilinear feature map set is M '═ M'1,m’2,…,m’n1×n1And then the formula of the conversion is as follows:
wherein "
"dot product of a representative matrix"
"represents rounding up, n1 represents the number of original feature maps, and i represents the subscript of the bilinear feature map.
However, in the system implementation process, the problem of dimension disaster can occur when the bilinear feature map of the last layer of convolution is directly calculated. If 512 feature maps are arranged in the last layer of the framework, 512 × 512 bilinear feature maps obtained according to the formula (1) bring a large amount of subsequent calculation overhead, and therefore dimension reduction operation needs to be realized on the bilinear feature maps. In contrast, the invention uses a Tensor Sketch algorithm (TS algorithm for short) to realize the dimensionality reduction of data, and the algorithm is a Countsketch-based vector outer product estimation method. Count Sketch is a data hashing method, which is primarily used in the mining of frequent item sets of data streams for the first time, and is subsequently proven by Pham et al to be capable of estimating the outer product of two vectors (i.e., multiplying each element of the vectors by each other) by calculating the convolution of the Count Sketch.
Since the above mentioned feature map is a matrix of c w h, and the input of the TS algorithm is a vector, the present invention calculates each position in the feature map in turn when using the TS algorithm, i.e. calculates w h c-dimensional vectors respectively, and remaps them into w h d-dimensional space. Firstly, randomly generating a parameter set h for carrying out hash operationk∈{1,…,d}c，sk∈{1,-1}c(k ═ 1,2), where hkAfter remapping for storing input vectorIndex of (1), skThe random inversion of the numerical values of the elements of the input vector is realized. According to the parameter set, the remapped Count Sketch vector can be obtained through accumulation calculation. As can be seen from the convolution theorem, the convolution of the time or spatial domain is equal to the product in the corresponding frequency domain. The convolution of the Count Sketch vectors can be computed by converting the two Count Sketch vectors to the frequency domain using a Fast Fourier Transform (FFT), multiplying them in the frequency domain, and then converting back to the spatial domain by an inverse fourier transform. The specific algorithm is as follows.
In addition to the improvement of the bilinear operation layer, the invention also changes the fully-connected layer behind the last convolutional layer in the original network structure into an Average Pooling layer (Average Pooling), the Pooling operation is carried out on the whole feature diagram by the fully-connected layer, the elements of each feature diagram are averaged, and finally the d-dimensional vector is obtained. The pooling layer is used for replacing the full-link layer because the pooling layer has no learnable parameters, so that the model parameters are greatly reduced, the convergence speed is accelerated, and the occurrence of overfitting is avoided to a certain extent. Meanwhile, the corresponding relation between the feature map and the pooled feature vectors is ensured, and conditions are created for subsequently extracting the attention distribution map.
Image overall privacy perception: the step is used for automatically identifying whether an input image is a privacy image, inputting a test image into the trained privacy perception network, and determining whether the test image is the privacy image according to the subordinate probability of each category output by the network.
Image privacy zone perception: this step is used to automatically detect privacy zones in the image. The attention distribution map is extracted mainly through deep convolution characteristics of the network, and the attention focusing area is positioned, so that the perception of the privacy area is completed.
Because the pooling layer node has corresponding relation with the characteristic graphThe attention profile can be obtained by weighted summation of feature maps. Let the bilinear feature map sequence P after dimension reduction be { P ═ P1,p2,…,pdAnd f, if the attention distribution graph generated finally is A, the calculation formula is as follows:
where n2 is the class label to which the input image is classified,
indicating the connection weight for the k-th node of the pooling layer corresponding to category n 2.
According to the result, the local position of the private image is positioned, and the specific method is to carry out scale change on the attention distribution map obtained in the step and convert the attention distribution map into the size of an original image. And setting a threshold value to complete image binarization, and solving a minimum external matrix of the binarized image as a local perception result of the private image.
The invention has wide application, such as:
first, photo sharing has become an increasingly popular way of communicating in social networks. However, there is a certain safety hazard in photo sharing by users, for example, many people, especially young people, directly share photos that may expose their privacy to a social network without considering their own safety enough, and some lawbreakers may use this information to engage in illegal activities, which undoubtedly brings a certain safety threat to themselves or their relatives and friends. In contrast, if the privacy perception mechanism is used, the privacy which may be related to the picture of the uploader can be reminded in time, and the function of preventing the picture from being overlooked is achieved. In addition, in some cases, the user wishes to mask or blur the areas of the public photograph that are related to privacy. The method for sensing the privacy sensitive area of the image can better solve the problems, can automatically position the privacy area in the image, is convenient for subsequent processing, and avoids manual operation.
In the second scheme, the current cloud storage is more and more widely applied, and the cloud platform collects personal information of a large number of users, wherein a large part of the personal information is image data. However, most cloud platforms belong to untrusted systems, and the events of personal data leakage of the cloud platforms are rare. In order to ensure that the privacy of individuals is not leaked, some companies use encryption or data disturbance to protect the privacy, but a large amount of computing resources are required for processing all massive image data. At this time, if the method related in the invention is used for analyzing the image data, the privacy image is distinguished or the privacy sensitive area is positioned first, and the targeted protection is carried out, so that the calculation overhead can be greatly reduced under the condition of ensuring the information security.
The invention improves some defects of the existing image privacy perception method on one hand, and extends the privacy perception problem to the perception of the image privacy area on the other hand so as to meet different requirements. Compared with the traditional privacy perception method, the method only trains the image content characteristics and categories, is not restricted by the image labels and the access strategies set by the user, and can play a role in various application scenes. Meanwhile, the deep convolutional network is used, and compared with the traditional feature extraction method, the method has stronger feature expressiveness and improves the classification accuracy and generalization capability of the model.
The image privacy perception method based on deep learning provided by the invention has the following advantages:
firstly, a good prerequisite is provided for image privacy protection. The invention provides an automatic privacy perception mechanism, which can perceive the privacy of the image and the local part of the image, and meets the diversified requirements of image privacy protection. The privacy image can be selectively protected on the premise of guaranteeing the privacy safety of the user, and the calculation overhead of privacy protection is greatly saved.
The privacy perception data set constructed in the invention contains images searched according to a large number of privacy corpora, so that the model can perceive various common privacy categories including certificate photos, file snapshots and the like, and the method has strong universality.
In the training and testing stages, an end-to-end mode is adopted (the input end is an original picture, the output end is a perception result, and no human interference is needed in the process), the perception of the privacy image and the image privacy area is completed simultaneously by using the same model, the model is convenient to use, and the method is easy to popularize in various practical application scenes.
And fourthly, an optimization strategy of bilinear operation is introduced, the feature expression capability is further improved on the basis of the original model, the image sensing accuracy rate is favorably improved, and the positioning of the privacy area is greatly facilitated.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (6)
1. An image privacy perception method based on deep learning is characterized by comprising the following steps:
s1, constructing a privacy classification data set with class labels, and training a privacy perception network by using a transfer learning method;
s2, completing the identification of the private image by using a deep convolutional neural network facing to privacy perception;
s3, extracting an attention distribution map according to the deep convolution characteristics of the neural network, and positioning an attention concentration area to finish sensing an image privacy area;
step S1 includes: pre-training a deep convolutional neural network model on a large-scale image data set, then constructing a privacy classification data set, and finely adjusting the pre-trained deep convolutional neural network model on the privacy classification data set;
step S2 includes: adding a bilinear operation layer after the last convolution layer of the deep convolution neural network, wherein the bilinear operation layer is used for calculating the result of dot product between two feature images after convolution, enhancing the feature expression capability of a deep convolution neural network model, and simultaneously changing a full-connection layer into an average pooling layer, and is used for pooling the whole feature image, averaging the elements of each feature image and finally obtaining a d-dimensional vector;
step S3 includes: and obtaining a weighted high-level feature map as an attention distribution map according to the corresponding relation between the weight of each node of the average pooling layer and the feature map subjected to bilinear operation, and positioning the privacy area in the original map through scale conversion.
2. The image privacy perception method based on deep learning of claim 1, wherein:
the bilinear operation layer mainly calculates the result of dot product between the feature maps after convolution,
let original feature map set M ═ { M ═ M1,m2,…,mn1}，
The output bilinear feature map set is M '═ M'1,m’2,…,m’n1×n1},
The formula for the conversion is as follows:
wherein
Represents a dot-product of the matrix and,
representing rounding up, n1 represents the number of original feature maps, and i represents the subscript of the bilinear feature map.
3. The image privacy perception method based on deep learning of claim 2, wherein: and performing dimension reduction operation on the bilinear feature map.
4. The image privacy perception method based on deep learning of claim 3, wherein: and (5) performing dimensionality reduction on the bilinear feature map by adopting a TensorSketch algorithm.
5. The image privacy perception method based on deep learning of claim 4, wherein: the bilinear feature map is a matrix of c w h, the input of the Tensor Sketch algorithm is a vector, and each position in the bilinear feature map is calculated in sequence when the Tensor Sketch algorithm is used, namely the w h c-dimensional vectors are respectively operated and remapped into a w h d-dimensional space; firstly, randomly generating a parameter set h for carrying out hash operationk∈{1,…,d}c，sk∈{1,-1}c(k ═ 1,2), where hkFor storing the remapped indices of the input vectors, skRandom negation of each element value of the input vector is realized; according to the parameter set, obtaining a remapped Count Sketch vector through accumulation calculation; as known from the convolution theorem, the convolution of the time domain or the space domain is equal to the product in the corresponding frequency domain; therefore, the two Count Sketch vectors are converted into the frequency domain by using fast fourier transform, the products of the two Count Sketch vectors in the frequency domain are obtained, and then the products are converted back into the space domain by inverse fourier transform, and the convolution of the Count Sketch vectors is obtained through calculation.
6. The image privacy perception method based on deep learning of claim 5, wherein: the average pooling layer node has a corresponding relation with the characteristic graph, and an attention distribution graph is obtained through the weighted summation of the characteristic graphs;
let the bilinear feature map sequence P after dimension reduction be { P ═ P1,p2,…,pd}，
And if the finally generated attention distribution graph is A, the calculation formula is as follows:
where n2 is the class label to which the input image is classified,
representing the connection weight for the k-th node of the average pooling layer corresponding to category n 2;
and positioning the local of the privacy image according to the result, wherein the specific method comprises the steps of carrying out scale change on the attention distribution map obtained in the step, converting the scale change into the size of an original image, setting a threshold value to complete image binarization, and solving a minimum external matrix of the binarized image as the local perception result of the privacy image.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710928967.XA CN107704877B (en) | 2017-10-09 | 2017-10-09 | Image privacy perception method based on deep learning |
| US16/099,836 US11256952B2 (en) | 2017-10-09 | 2017-11-27 | Image privacy perception method based on deep learning |
| PCT/CN2017/113068 WO2019071754A1 (en) | 2017-10-09 | 2017-11-27 | Method for sensing image privacy on the basis of deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710928967.XA CN107704877B (en) | 2017-10-09 | 2017-10-09 | Image privacy perception method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107704877A CN107704877A (en) | 2018-02-16 |
| CN107704877B true CN107704877B (en) | 2020-05-29 |
Family
ID=61184658
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710928967.XA Active CN107704877B (en) | 2017-10-09 | 2017-10-09 | Image privacy perception method based on deep learning |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US11256952B2 (en) |
| CN (1) | CN107704877B (en) |
| WO (1) | WO2019071754A1 (en) |
Families Citing this family (43)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11734567B2 (en) * | 2018-02-13 | 2023-08-22 | Samsung Electronics Co., Ltd. | Method and system for reducing deep neural network architectures |
| CN109101523A (en) * | 2018-06-14 | 2018-12-28 | 北京搜狗科技发展有限公司 | A kind of image processing method, device and electronic equipment |
| CN109145816B (en) * | 2018-08-21 | 2021-01-26 | 北京京东尚科信息技术有限公司 | Commodity identification method and system |
| CN109376757B (en) * | 2018-09-06 | 2020-09-08 | 苏州飞搜科技有限公司 | Multi-label classification method and system |
| CN109743580A (en) * | 2018-12-24 | 2019-05-10 | 秒针信息技术有限公司 | A kind of method for processing video frequency and device, storage medium and processor |
| CN109743579A (en) * | 2018-12-24 | 2019-05-10 | 秒针信息技术有限公司 | A kind of method for processing video frequency and device, storage medium and processor |
| CN109756842B (en) * | 2019-02-19 | 2020-05-08 | 山东大学 | Wireless indoor positioning method and system based on attention mechanism |
| CN109993207B (en) * | 2019-03-01 | 2022-10-25 | 华南理工大学 | Image privacy protection method and system based on target detection |
| CN109993212B (en) * | 2019-03-06 | 2023-06-20 | 西安电子科技大学 | Position privacy protection method in social network picture sharing and social network platform |
| CN110334571B (en) * | 2019-04-03 | 2022-12-20 | 复旦大学 | Millimeter wave image human body privacy protection method based on convolutional neural network |
| US10762607B2 (en) | 2019-04-10 | 2020-09-01 | Alibaba Group Holding Limited | Method and device for sensitive data masking based on image recognition |
| CN110163218A (en) * | 2019-04-10 | 2019-08-23 | 阿里巴巴集团控股有限公司 | Desensitization process method and device based on image recognition |
| CN109903289B (en) * | 2019-04-17 | 2023-05-05 | 广东工业大学 | Terahertz image nondestructive testing method, device and equipment |
| CN110069947B (en) * | 2019-04-22 | 2020-09-15 | 鹏城实验室 | Picture privacy prediction method and device, storage medium and electronic equipment |
| CN111860068A (en) * | 2019-04-30 | 2020-10-30 | 四川大学 | Fine-grained bird identification method based on cross-layer simplified bilinear network |
| CN110175469B (en) * | 2019-05-16 | 2020-11-17 | 山东大学 | Social media user privacy leakage detection method, system, device and medium |
| CN110163177B (en) * | 2019-05-28 | 2022-12-09 | 李峥嵘 | Unmanned aerial vehicle automatic sensing and identifying method for wind turbine generator blades |
| KR102234097B1 (en) * | 2019-07-17 | 2021-04-01 | 부산대학교 산학협력단 | Image processing method and system for deep-learning |
| CN110717856A (en) * | 2019-09-03 | 2020-01-21 | 天津大学 | Super-resolution reconstruction algorithm for medical imaging |
| CN111177757A (en) * | 2019-12-27 | 2020-05-19 | 支付宝(杭州)信息技术有限公司 | Processing method and device for protecting privacy information in picture |
| CN111639359B (en) * | 2020-04-22 | 2023-09-12 | 中国科学院计算技术研究所 | Method and system for detecting and early warning privacy risk of social network picture |
| CN111753885B (en) * | 2020-06-09 | 2023-09-01 | 华侨大学 | Privacy enhanced data processing method and system based on deep learning |
| CN111724424B (en) * | 2020-06-24 | 2024-05-14 | 上海应用技术大学 | Image registration method |
| CN111784757B (en) * | 2020-06-30 | 2024-01-23 | 北京百度网讯科技有限公司 | Training method of depth estimation model, depth estimation method, device and equipment |
| CN111814165B (en) * | 2020-07-07 | 2024-01-26 | 重庆大学 | Image privacy protection method based on deep neural network middle layer |
| CN111967318A (en) * | 2020-07-13 | 2020-11-20 | 北京邮电大学 | Camera-assisted Internet of vehicles wireless communication method based on privacy protection principle |
| CN112101437B (en) * | 2020-09-07 | 2024-05-31 | 平安科技（深圳）有限公司 | Fine granularity classification model processing method based on image detection and related equipment thereof |
| CN115114967A (en) * | 2020-09-21 | 2022-09-27 | 武汉科技大学 | Steel microstructure automatic classification method based on self-organization increment-graph convolution neural network |
| US11704433B2 (en) | 2020-09-21 | 2023-07-18 | International Business Machines Corporation | Dynamic photograph classification |
| CN112347512A (en) * | 2020-11-13 | 2021-02-09 | 支付宝(杭州)信息技术有限公司 | Image processing method, device, equipment and storage medium |
| US12047355B2 (en) * | 2021-03-08 | 2024-07-23 | Adobe Inc. | Machine learning techniques for mitigating aggregate exposure of identifying information |
| CN113095989B (en) * | 2021-03-31 | 2023-07-07 | 西安理工大学 | Zero watermark copyright protection algorithm based on image style migration |
| CN113642717B (en) * | 2021-08-31 | 2024-04-02 | 西安理工大学 | Convolutional neural network training method based on differential privacy |
| CN113837269A (en) * | 2021-09-23 | 2021-12-24 | 中国特种设备检测研究院 | Metallographic structure identification method based on bilinear convolutional neural network |
| CN114091651B (en) * | 2021-11-03 | 2024-05-24 | 支付宝(杭州)信息技术有限公司 | Method, device and system for multi-party combined training of graph neural network |
| CN114821705A (en) * | 2022-03-17 | 2022-07-29 | 南京邮电大学 | Human head posture estimation method based on classification before regression |
| CN114419719B (en) * | 2022-03-29 | 2022-08-12 | 北京爱笔科技有限公司 | Biological characteristic processing method and device |
| CN115062331A (en) * | 2022-05-22 | 2022-09-16 | 北京理工大学 | Privacy protection deep learning method based on additive homomorphic encryption |
| CN114913467A (en) * | 2022-06-14 | 2022-08-16 | 南京邮电大学 | CRNN combined network video privacy protection degree evaluation method for monitoring violent behaviors in home scene |
| CN115906186B (en) * | 2023-02-16 | 2023-05-16 | 广州优刻谷科技有限公司 | Face image privacy protection method, device and storage medium |
| CN116721302B (en) * | 2023-08-10 | 2024-01-12 | 成都信息工程大学 | Ice and snow crystal particle image classification method based on lightweight network |
| CN116740650B (en) * | 2023-08-10 | 2023-10-20 | 青岛农业大学 | Crop breeding monitoring method and system based on deep learning |
| CN118132141B (en) * | 2024-05-06 | 2024-07-30 | 西安电子科技大学 | Function automatic reconstruction method and device based on code characteristic diagram and electronic equipment |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104102929B (en) * | 2014-07-25 | 2017-05-03 | 哈尔滨工业大学 | Hyperspectral remote sensing data classification method based on deep learning |
| US9607217B2 (en) * | 2014-12-22 | 2017-03-28 | Yahoo! Inc. | Generating preference indices for image content |
| US10878320B2 (en) * | 2015-07-22 | 2020-12-29 | Qualcomm Incorporated | Transfer learning in neural networks |
| EP3913591A1 (en) * | 2016-01-29 | 2021-11-24 | KiwiSecurity Software GmbH | Methods and apparatus for using video analytics to detect regions for privacy protection within images from moving cameras |
| CN106295584A (en) * | 2016-08-16 | 2017-01-04 | 深圳云天励飞技术有限公司 | Depth migration study is in the recognition methods of crowd's attribute |
| CN106778740A (en) * | 2016-12-06 | 2017-05-31 | 北京航空航天大学 | A kind of TFDS non-faulting image detecting methods based on deep learning |
| CN106682694A (en) * | 2016-12-27 | 2017-05-17 | 复旦大学 | Sensitive image identification method based on depth learning |
-
2017
- 2017-10-09 CN CN201710928967.XA patent/CN107704877B/en active Active
- 2017-11-27 WO PCT/CN2017/113068 patent/WO2019071754A1/en active Application Filing
- 2017-11-27 US US16/099,836 patent/US11256952B2/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107704877A (en) | 2018-02-16 |
| US20210224586A1 (en) | 2021-07-22 |
| WO2019071754A1 (en) | 2019-04-18 |
| US11256952B2 (en) | 2022-02-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107704877B (en) | Image privacy perception method based on deep learning | |
| Yang et al. | Ml-loo: Detecting adversarial examples with feature attribution | |
| Arietta et al. | City forensics: Using visual elements to predict non-visual city attributes | |
| Parisot et al. | Property inference attacks on convolutional neural networks: Influence and implications of target model's complexity | |
| Thieltges et al. | The devil’s triangle: Ethical considerations on developing bot detection methods | |
| Rehman et al. | Face liveness detection using convolutional-features fusion of real and deep network generated face images | |
| Ratre et al. | Tucker visual search-based hybrid tracking model and Fractional Kohonen Self-Organizing Map for anomaly localization and detection in surveillance videos | |
| Thakur et al. | Machine learning based saliency algorithm for image forgery classification and localization | |
| Chen et al. | Image splicing localization using residual image and residual-based fully convolutional network | |
| Uma et al. | Copy-move forgery detection of digital images using football game optimization | |
| Wang et al. | Suspect multifocus image fusion based on sparse denoising autoencoder neural network for police multimodal big data analysis | |
| Ilyas et al. | E-Cap Net: an efficient-capsule network for shallow and deepfakes forgery detection | |
| Sedik et al. | AI-enabled digital forgery analysis and crucial interactions monitoring in smart communities | |
| Kumar et al. | Image steganography analysis based on deep learning | |
| Mahmood | Defocus Blur Segmentation Using Genetic Programming and Adaptive Threshold. | |
| CN112183299B (en) | Pedestrian attribute prediction method and device, electronic equipment and storage medium | |
| CN114638356A (en) | Static weight guided deep neural network back door detection method and system | |
| Nair et al. | Identification of multiple copy-move attacks in digital images using FFT and CNN | |
| Zheng et al. | Content-adaptive selective steganographer detection via embedding probability estimation deep networks | |
| Dhar et al. | Detecting deepfake images using deep convolutional neural network | |
| Yu et al. | Multi-task learning for hand heat trace time estimation and identity recognition | |
| Lakshminarasimha et al. | Deep Learning Base Face Anti Spoofing-Convolutional Restricted Basis Neural Network Technique | |
| Kalimuthu et al. | Semantic‐Based Facial Image‐Retrieval System with Aid of Adaptive Particle Swarm Optimization and Squared Euclidian Distance | |
| Ramachandra et al. | Face presentation attack detection using multi-classifier fusion of off-the-shelf deep features | |
| Liu et al. | Hybrid network of convolutional neural network and transformer for deepfake geographic image detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |