CN112818360B - Deep neural network encryption reasoning method based on homomorphic encryption technology - Google Patents

Deep neural network encryption reasoning method based on homomorphic encryption technology Download PDF

Info

Publication number
CN112818360B
CN112818360B CN202110064852.7A CN202110064852A CN112818360B CN 112818360 B CN112818360 B CN 112818360B CN 202110064852 A CN202110064852 A CN 202110064852A CN 112818360 B CN112818360 B CN 112818360B
Authority
CN
China
Prior art keywords
convolution
homomorphic
layer
ciphertext
feature map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110064852.7A
Other languages
Chinese (zh)
Other versions
CN112818360A (en
Inventor
刘龙军
王军辉
雷瑞棋
张衔哲
朱劲宇
侯文轩
郑南宁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xian Jiaotong University
Original Assignee
Xian Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xian Jiaotong University filed Critical Xian Jiaotong University
Priority to CN202110064852.7A priority Critical patent/CN112818360B/en
Publication of CN112818360A publication Critical patent/CN112818360A/en
Application granted granted Critical
Publication of CN112818360B publication Critical patent/CN112818360B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/602—Providing cryptographic facilities or services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

A deep neural network encryption reasoning method based on a homomorphic encryption technology is characterized in that a first layer of a ciphertext feature map after convolution operation is obtained by means of homomorphic convolution operation after BN layer is combined into a convolution layer by the cloud end and transmitted to a client; after receiving the input characteristic diagram, the client side obtains a marking matrix by adopting marking operation and transmits the marking matrix to the cloud server, and the cloud server updates the marking matrix to obtain the input characteristic diagram of the homomorphic convolution layer of the second layer; performing homomorphic convolution operation on the second layer to obtain a ciphertext feature map after the convolution operation of the second layer, and transmitting the ciphertext feature map to the client; and repeating the process until all the ciphertext feature maps after convolution layer operation are obtained. The invention uses the GPU to accelerate the homomorphic convolution operation process, thereby avoiding the repeated transmission of data. The invention can reduce the noise increase of the ciphertext, increase the inference layer number of the neural network, and greatly reduce the calculation cost of the ciphertext.

Description

Deep neural network encryption reasoning method based on homomorphic encryption technology
Technical Field
The invention relates to a deep neural network encryption reasoning method based on a homomorphic encryption technology, which can be used as a privacy data protection method in the field of artificial intelligence.
Background
The deep neural network, as an important technology in the field of artificial intelligence, has far exceeded the traditional computer vision processing and identifying methods in applications such as image classification and identification, video target tracking and the like. However, the training and reasoning of the deep neural network facing computer vision needs to collect a large amount of user image data, which easily relates to the privacy of users, and if the user data is leaked or misoperated, on one hand, the privacy of users is likely to be exposed, and even unpredictable property loss and life safety problems are likely to be caused. On the other hand, with the further development of the deep neural network, more user data needs to be collected, and privacy protection is more and more emphasized by the public, and if the privacy protection is not properly processed, the contradiction between the two will hinder the development of the deep neural network technology.
The existing solutions for privacy protection are: differential Privacy (DP), multi-party computing (MPC), homomorphic Encryption (HE). Different solutions have different limitations. Differential privacy is a mathematically rigorous framework that provides some mathematical assurance of the privacy of the user's personal information, which is aimed at reducing the impact of any personal data on the overall result, and can be used to quantify anonymization of sensitive data. But progress in neural network reasoning is not great. The multi-party calculation does not interfere with the calculation process, so that the accuracy and the safety can be ensured. However, both the calculation amount and the communication amount are huge, the requirements on the calculation capacity and the communication bandwidth are high, and the capacity of calculating and testing the communication bandwidth, namely the transmission speed, by multiple parties is considered. Homomorphic encryption can realize reasoning operation of a deep neural network for encrypting data under the condition of protecting privacy. The advantage of homomorphic encryption is the powerful ability to compute under the encrypted domain, but this ability is achieved through a large number of computations. The computational overhead is so large that the current state-of-the-art implementations are still five to six orders of magnitude slower than plain text reasoning.
In recent years, there have been some researches to apply homomorphic encryption technology to encryption reasoning of deep neural networks, cryptoNets are one of the earliest efforts to implement homomorphic encryption on neural networks, and square functions are used instead of activation functions. During training, there are three active layers. The first two active layers use square functions, and the last Sigmoid active function is only used for training and is omitted in actual prediction. Later work typically uses an approximation of the commonly used activation function as the activation function in the homomorphic encryption philosophy. CryptoDL further studies several methods of approximating the commonly used activation functions in CNN (i.e., reLU, sigmoid, and Tanh) using low-order polynomials to find the best approximation. Fast CryptoNets use sparse plaintext ciphertext multiplication and network pruning and quantization to accelerate reasoning in many ways. They also derive an optimal activation function and maximum sparse coding for the approximation of the quantized quadratic function. Carenes proposes a new compact homomorphic CNN architecture that densely packs CNN inputs, weights and activated high-dimensional vectors into FHE encrypted ciphertext and applies highly parallel execution to homomorphic CNN operations.
The existing deep neural network reasoning method for encrypting the data domain by using the homomorphic encryption technology mainly has the following problems. First, the number of layers of a deep neural network based on a homomorphic encryption technique is not deep enough, mainly because homomorphic encryption can only perform limited multiply-add operations under the condition of encryption parameter determination. Secondly, the relu, sigmoid and other activation functions in the deep neural network are difficult to be supported by the existing homomorphic encryption framework, the prior work uses quadratic function fitting, but a new problem is caused, the multiplication between the ciphertext and the ciphertext leads to the multiplication of encryption noise and calculation time, and the precision of the deep neural network is obviously reduced.
Disclosure of Invention
The invention aims to provide a deep neural network encryption reasoning method based on a homomorphic encryption technology, which is used for efficiently realizing deep neural network encryption under the condition of protecting user data privacy.
In order to achieve the purpose, the invention adopts the following technical scheme:
a deep neural network encryption reasoning method based on homomorphic encryption technology comprises the following steps:
(1) Encrypting the picture and transmitting the encrypted picture to a cloud end, laminating the BN into a convolution layer after the cloud end server receives the encrypted image data, performing homomorphic convolution operation according to a convolution method to obtain a ciphertext characteristic diagram after a first layer of convolution operation, and transmitting the ciphertext characteristic diagram to a client end;
(2) After receiving the ciphertext characteristic diagram after the first layer of homomorphic convolution operation, the client side obtains a mark matrix by adopting mark operation and transmits the mark matrix to the cloud server, and the cloud server updates the output characteristic diagram of the first layer of homomorphic convolution layer after receiving the mark matrix, so that the activation function of a ciphertext deep neural network is realized by the cooperation of a mark matrix method of the client side and a characteristic diagram updating method on the cloud server, and finally an input characteristic diagram of a second layer of homomorphic convolution layer is obtained; sequentially transmitting pixel points of the input feature map into a GPU according to the position of the convolution window in the input feature map of the homomorphic convolution layer of the second layer, finishing homomorphic convolution operation of the second layer under a ciphertext domain by adopting a multi-core parallel processing ciphertext polynomial mode, sequentially destroying the pixel points of the input feature map in the GPU to obtain a ciphertext feature map after the convolution operation of the second layer, and transmitting the ciphertext feature map to a client;
(3) And (5) repeating the step (2) until all ciphertext feature maps after convolution layer operation are obtained.
The invention has the further improvement that in the step (1), a homomorphic encryption algorithm is adopted to encrypt the picture; the parameters of the homomorphic encryption algorithm are polynomial modulus, ciphertext modulus and plaintext modulus.
The further improvement of the invention is that in the step (1), the ciphertext feature map after the first layer of convolution operation is calculated by the following formula:
Y i ′＝W′ i *X+B′ i
wherein the content of the first and second substances,
Figure BDA0002903750020000031
Figure BDA0002903750020000032
w 'in the formula' i Is the weight of the ith kernel in the homomorphic convolution kernel, γ i The ith channel, σ, representing a rescaling parameter γ i The ith channel, W, representing the cumulative variance σ i A weight representing an ith convolution kernel; x is an input characteristic diagram; b' i For the bias of the ith kernel in the homomorphic convolution kernel, B i Denotes the offset, μ, of the ith convolution kernel i I channel, beta, representing the cumulative mean mu i The ith channel representing the retranslation parameter beta.
The further improvement of the invention is that in the step (2), the specific process of obtaining the marking matrix by adopting the marking operation is as follows: and (3) representing numbers which are greater than 0 in each point in the characteristic diagram as 1 and numbers which are less than or equal to 0 as 0 by a step marking function by using a step marking method to obtain a marking matrix consisting of 0 and 1.
A further development of the invention is that the step marking function is as follows:
Figure BDA0002903750020000041
/>
where x is the input pixel value and y is the output flag.
The further improvement of the invention is that for convolution layers with more than 5 layers, when the number of 0 in the mark matrix is less than 50%, the point of 1 in the mark matrix and the points with the upper, lower, left and right marks of 1 are updated to be the plaintext after the decryption of the corresponding pixel points, so as to obtain the mark matrix.
The further improvement of the present invention is that, in the step (2), the specific process of sequentially transmitting the pixel points of the input feature map to the GPU at the position of the convolution window in the input feature map of the second layer of homomorphic convolution layer is as follows: when the homomorphic convolution windows are in the first row and the first column of the input feature map, transmitting data of convolution windows 3x3 of all the input channels to the GPU for homomorphic convolution operation, when the homomorphic convolution windows are in the first row and not in the first column of the input feature map, transmitting data of the third column of the convolution windows of all the input channels to the GPU for homomorphic convolution operation, when the homomorphic convolution windows are in the first column and not in the first row of the input feature map, transmitting data of the third row of the convolution windows of all the input channels to the GPU for homomorphic convolution operation, and when the homomorphic convolution windows are not in the first row and not in the first column of the input feature map, transmitting data of the third row and not in the third column of the convolution windows of all the input channels to the GPU for homomorphic convolution operation.
The further improvement of the invention is that in the step (2), the specific process of sequentially destroying the pixel points of the input characteristic diagram in the GPU is as follows: when the homomorphic convolution window is in the last row and the last column of the input feature map, destroying data of convolution windows 3x3 of all input channels after the GPU completes homomorphic convolution operation of the convolution windows, when the homomorphic convolution window is not in the last row of the input feature map, destroying data of the first row of the convolution windows of all input channels after the GPU completes homomorphic convolution operation of the convolution windows, when the homomorphic convolution window is not in the last column of the input feature map, destroying data of the first column of the convolution windows of all input channels after the GPU completes homomorphic convolution operation of the convolution windows, and when the homomorphic convolution window is not in the last row and not in the last column of the input feature map, destroying data of the first row and the first column of the convolution windows of all input channels after the GPU completes homomorphic convolution operation of the convolution windows.
Compared with the prior art, the invention has the following beneficial effects:
the invention adopts an inference mode based on the neural network convolution layer under homomorphic encryption to combine the BN layer into the convolution layer, thereby reducing the homomorphic operation times, greatly reducing the required calculation time and the noise increase of the ciphertext. The invention uses the GPU (graphic processing unit) to accelerate the homomorphic convolution operation process, provides an improvement strategy for the transmission mode of the characteristic diagram data from the CPU to the GPU and the destruction of the characteristic diagram data in the GPU, and avoids the repeated transmission of the data. The invention can reduce the noise increase of the ciphertext, increase the inference layer number of the neural network and greatly reduce the ciphertext calculation cost.
Furthermore, the invention provides a multi-party cooperative activation function, namely a step marking function of the client and data updating on the cloud server to jointly complete a universal relu activation function, so that the noise increase of a ciphertext and the required calculation time are effectively reduced, and the precision of the original deep neural network is kept unchanged.
Further, for convolution layers with more than 5 layers, when the number of 0 in the mark matrix is lower than 50%, the point with 1 in the mark matrix and the adjacent marks of the upper, lower, left and right sides of the point are all 1 is updated to be the plaintext after the corresponding pixel point is decrypted, a new mark matrix is obtained, the number of ciphertexts in the feature diagram is reduced to reduce homomorphic convolution operation amount of the cloud and noise increase of the ciphertexts, and the feature diagram is further guaranteed not to be cracked.
Drawings
Fig. 1 is a cloud edge collaborative homomorphic encryption inference framework diagram.
Fig. 2 is a schematic diagram of a client encryption and transmission cloud.
FIG. 3 is a diagram illustrating a homomorphic convolution operation.
FIG. 4 is a diagram of a GPU convolution accelerated data transfer scheme.
Fig. 5 is a schematic diagram of a GPU convolution acceleration data destruction scheme.
FIG. 6 is a diagram of a multi-party cooperative activation function.
FIG. 7 is a diagram illustrating the homomorphic convolution operation of the nth layer.
Fig. 8 is a schematic diagram of an automatic encryption selector.
Detailed Description
The present invention will be described in detail with reference to the accompanying drawings.
The method comprises the design of encrypting and decrypting user data by using a homomorphic encryption technology at a client side, the design of an automatic parameter selector and an automatic encryption selector, the inference design of encrypted data tightly combined with the inference of a deep neural network at a server side, a homomorphic convolution calculation method for the deep neural network facing ciphertext data and a realization method of a multi-party cooperative activation function, and the design of a GPU acceleration architecture for improving the operation efficiency of the deep neural network and the ciphertext.
As shown in fig. 1, the invention provides a deep neural network encryption reasoning method based on a homomorphic encryption technology by interaction and cooperation of a client and a cloud, wherein an adaptive encryption parameter is selected by an automatic parameter selector at the client to encrypt a user picture and transmit the encrypted user picture to the cloud (after a 5G or even 6G network is popularized, data transmission can be accelerated by means of a new generation of 5G or 6G network transmission technology), a trained deep neural network is deployed at the cloud in advance, and a first layer of homomorphic convolution operation is performed by using a homomorphic convolution operation algorithm provided by the invention and encrypted picture data transmitted by the client to obtain a first layer of output feature map of the deep neural network. All output characteristic graphs in the first five layers of the deep neural network at the cloud (because the automatic encryption selector can enable plaintext to exist in the characteristic graphs) are in a ciphertext form, and therefore privacy of a user is guaranteed. And then, all the encrypted output characteristic graphs are transmitted back to the client, the client decrypts the characteristic graphs in sequence, and the marking operation of the characteristic graphs is completed by utilizing the step marking method in the multi-party cooperative activation function, namely, the value which is greater than 0 in the characteristic graphs is marked as 1, and the value which is less than or equal to 0 in the characteristic graphs is marked as 0, so that a brand-new marking matrix which only contains 0 and 1 is obtained. And finally, returning the mark matrix to the cloud server. And after the cloud server receives the marker matrix, processing the cloud characteristic diagram, and replacing the ciphertext of the characteristic diagram corresponding to the position of the marker matrix 0 with the ciphertext of the corresponding position of the plaintext 0, 1. And taking the processed feature map as a second-layer input feature map of the deep neural network. The same as the homomorphic convolution operation of the first layer, the operations of the second layer, the third layer and the like are completed by interaction with the client. The method and the device have the advantages that (a 14-layer network is realized at present, but the method and the device can realize all the existing deep neural networks, and more than 100 layers of neural networks exist), the deep neural network reasoning and calculating process of the user data based on encryption at the cloud end is finally realized, and the user privacy data are effectively protected from being leaked at the cloud end.
According to the invention, the automatic parameter selector is realized to select the optimal encryption parameters, the convolution layer calculation of the deep neural network model at the cloud end and the activation layer calculation at the client end are optimized. Compared with the existing mode, the mode of finishing the deep neural network inference of the encrypted data by the interaction and cooperation of the client and the cloud can reduce the noise increase of the ciphertext, increase the inference layer number of the neural network and greatly reduce the computation overhead of the ciphertext. In addition, the invention also provides that the homomorphic convolution operation on the cloud server is accelerated in parallel by adopting the GPU, so that the cloud reasoning speed and efficiency are further improved. In addition, the invention also realizes that the automatic encryption selector performs the homomorphic convolution operation after performing the decryption operation on the feature map of the partial layer of the deep neural network, so that the data volume and the calculated amount at the cloud end are reduced, and the original image can not be cracked.
The method comprises the following specific steps:
(1) The picture is encrypted at the client and then transmitted to the cloud: generally, user picture data is acquired through a camera or other scanning equipment, and in order to prevent the user picture data from being divulged of secret when deep neural network reasoning calculation is performed on a cloud server, a client encrypts the user picture data by using a homomorphic encryption algorithm, such as microsoft open source homomorphic encryption base SEAL (Simple Encrypted authenticated atomic Library), and the like. The invention can automatically configure an inference scheme according to a network model, is called an automatic configurable inference framework and consists of an automatic parameter selector and an automatic encryption selector.
Before cloud HE reasoning is carried out, after parameters of an encryption scheme are selected through an automatic parameter selector, picture data are encrypted. The parameters of the encryption scheme are mainly referred to as polynomial modulus, ciphertext modulus and plaintext modulus. Different parameters influence the size of the ciphertext and the complexity of calculation, thereby influencing the speed of the network reasoning speed. The automatic parameter selector determines parameters adopted by an encryption scheme according to the depth and the width of the inference network, so that the inference speed of the neural network based on homomorphic encryption is increased.
Because the data volume of the encrypted picture is very large, the picture of the user is encrypted pixel by pixel, the encrypted picture pixels are transmitted to the cloud server one by one, and the transmission process can adopt a 5G network communication technology to accelerate data transmission, as shown in fig. 2.
(2) The cloud deep neural network performs a first layer of homomorphic convolution operation: after receiving the encrypted image data sent by the client, the cloud server performs homomorphic convolution operation according to a convolution mode, and performs homomorphic convolution operation with a main area of general convolutionThe difference is the weight W 'of the ith kernel in the homomorphic convolution kernel' i And bias B' i The acquisition mode of (1): in the ordinary deep neural network training, an input characteristic diagram is set as X, the weight of an ordinary convolution kernel is set as W, the bias is set as B, the average value accumulated in the BN layer in the training process is set as mu, the variance is set as sigma, the rescaling parameter is set as gamma, the retranslating parameter is set as beta, and the output characteristic diagram passing through the convolution layer and the BN layer is set as Y.
Then there are:
Figure BDA0002903750020000081
wherein, denotes a normal convolution operation, W i And B i Weight and offset, μ, representing the ith convolution kernel i I channel, σ, representing the cumulative mean μ i The ith channel, γ, representing the cumulative variance σ i The ith channel, β, representing a rescaling parameter γ i The ith channel, Y, representing the retranslation parameter beta i Representing the ith channel of the output signature Y.
Considering the linear invariance of convolution operations, the BN layers can be merged into a convolution layer, i.e., two layers of operations are merged into one layer.
Is provided with
Figure BDA0002903750020000082
Namely, it is
Figure BDA0002903750020000083
Wherein, W' i Is the weight, B 'of the ith kernel in the homomorphic convolution kernel' i Is the offset of the ith kernel in the homomorphic convolution kernel. After the weights and biases of the homomorphic convolution kernels are obtained, homomorphic convolution can be computed like the computation of a normal convolution kernel, namely:
Y i ＝W′ i *X+B′ i
wherein, the sum + is respectively the ciphertext multiplied by the plaintext and the ciphertext plus the plaintext in the homomorphic operation.
Based on the common deep neural network training process, the invention obtains the weight W 'of the ith kernel in the homomorphic encryption convolution kernel' i And offset B 'of ith kernel in the homomorphic convolution kernel' i After homomorphic convolution is carried out on the ciphertext feature map Y and the encrypted image data, all ciphertext feature maps Y after the first-layer convolution operation are obtained i And then transmitted to the client through a 5G network communication technology, see fig. 3 for details.
Different from convolution of a deep neural network on a common plaintext image, the data volume of a ciphertext image is particularly large, homomorphic convolution operation is performed on the ciphertext in a large number of polynomial forms, and accordingly a large amount of calculation is caused, and therefore the calculation of cloud ciphertext data is accelerated by using a GPU (graphic processing unit) to improve the cloud calculation efficiency.
(3) Adopting a multi-party cooperative activation function aiming at each layer of output feature map of the deep neural network: when deep neural network reasoning calculation is carried out on the data encrypted by using the homomorphism, homomorphic multiplication between ciphertexts takes the most calculation time. At present, most of work adopts quadratic function fitting activation functions, and the time spent by a deep neural network activation layer is far longer than the calculation time of other layers, so that a ciphertext characteristic diagram of a cloud server is transmitted back to a client side, and after decryption, the client side completes marking operation aiming at a plaintext characteristic diagram. In order to accelerate the computation efficiency of a deep neural network and the interaction efficiency from a client to a cloud, the invention provides a multi-party cooperative activation function, namely a step marking function of the client and a general relu activation function completed together with data updating on a cloud server, wherein the following formula (1) shows the step marking function, firstly, a step marking method is utilized to represent the number which is greater than 0 in each point of a characteristic diagram as 1, the number which is less than or equal to 0 as 0, and a brand-new marking matrix which is composed of 0 and 1 is obtained from the results. And finally, the client transmits the mark matrix back to the cloud. The transmission only transmits the characteristic matrix consisting of 0 and 1, and the network transmission amount is very small.
Step mark function: assuming the input pixel value is x and the output label is y, then there are:
Figure BDA0002903750020000091
and after the cloud receives the feature matrix composed of 0 and 1 sent by the client. And updating the information of the cloud first-layer output characteristic diagram based on the positions of 0 and 1, as shown in fig. 6, the value of the point corresponding to 1 is unchanged, and the value of the point corresponding to 0 is updated to 0. And (3) after the updating is finished, performing homomorphic convolution layer operation of the second layer according to the homomorphic convolution calculation mode of the first layer in the step (2). And repeating the step (2) and the step (3) until all the ciphertext feature maps after the convolution layer operation are obtained.
Because the ciphertext in all the convolution layer input characteristic diagrams in the cloud end consists of 8 polynomials of 8182 degrees, the homomorphic convolution operation amount is far greater than that in the plain text. Therefore, the present invention employs a GPU (graphics processing unit) to accelerate the homomorphic convolution operation of the cloud, as described in step 4.
(4) Carrying out cloud deep neural network reasoning by using a GPU:
when the GPU accelerates the first-layer homomorphic convolution operation, the cloud server receives encrypted image data sent by the client side to obtain an input characteristic diagram of the first-layer homomorphic convolution layer. When the GPU accelerates the homomorphic convolution operation after the first layer, the cloud server receives a mark matrix which is sent by the client and consists of 0 and 1, and updates the output characteristic diagram of the homomorphic convolution layer on the previous layer to obtain the input characteristic diagram of the homomorphic convolution layer; (specifically, for the second layer, the output characteristic diagram of the homomorphic convolutional layer of the first layer is updated to obtain the input characteristic diagram of the homomorphic convolutional layer of the second layer); under the condition that the step length of the convolution window is 1, sequentially transmitting pixel points of the input characteristic graph to a GPU (graphic processing unit) according to the position of the convolution window in the input characteristic graph, accelerating to complete homomorphic convolution operation under a cryptograph domain in a multi-core parallel cryptograph polynomial processing mode, destroying data which are completed in the GPU, and transmitting a homomorphic convolution operation result back to a CPU (central processing unit).
When the homomorphic convolution window is in the first row and the first column of the input feature map, the data of the convolution windows 3x3 of all the input channels are transmitted to the GPU for homomorphic convolution operation, as shown in fig. 4 (a). When the homomorphic convolution window is in the first row but not in the first column of the input feature map, the data of the third column of the convolution windows of all input channels is transmitted to the GPU for homomorphic convolution operation, as shown in (b) of FIG. 4. When the homomorphic convolution window is in the first column but not in the first row of the input feature map, transmitting the data in the third row of the convolution windows of all input channels to the GPU for homomorphic convolution operation, as shown in (c) of FIG. 4. When the homomorphic convolution window is not in the first row and not in the first column of the input feature map, the data in the third row and the third column of the convolution windows of all the input channels are transmitted to the GPU for homomorphic convolution operation, as shown in fig. 4 (d).
When the homomorphic convolution window is in the last row and the last column of the input feature map, the data of the convolution windows 3x3 of all the input channels are destroyed after the GPU completes the homomorphic convolution operation of the convolution window, as shown in fig. 5 (a). When the homomorphic convolution window is in the last column but not in the last row of the input feature map, the data in the first row of the convolution windows of all input channels is destroyed after the GPU completes the homomorphic convolution operation of the convolution window, as shown in fig. 5 (b). When the homomorphic convolution window is in the last row but not in the last column of the input feature map, the data in the first column of the convolution windows of all input channels is destroyed after the GPU completes the homomorphic convolution operation of the convolution window, as shown in (c) of fig. 5. When the homomorphic convolution window is not in the last row and not in the last column of the input feature map, destroying the data in the first row and the first column of the convolution windows of all the input channels after the GPU completes the homomorphic convolution operation of the convolution window, as shown in (d) in fig. 5.
(5) And (3) homomorphic convolution operation of the nth layer of the deep neural network is calculated at the cloud: similar to the homomorphic convolution operation of the second layer, the cloud updates the n-th layer ciphertext feature map and then performs homomorphic convolution operation based on a mark matrix formed by 0 and 1 sent by the client.
In order to further accelerate the cloud homomorphic convolution reasoning speed, the invention further provides a method for accelerating the homomorphic convolution operation speed by reducing the number of cloud ciphertext pixels. The feature map operated by the deep neural network to more than 5 layers is difficult to restore to the original image, namely the ciphertext image is difficult to crack. Therefore, when the homomorphic convolution operation is executed on the feature map after 5 layers, the partially encrypted feature map pixels can be used in a plaintext form, and the operation of the plaintext is far faster than that of the ciphertext, so that the operation speed of homomorphic convolution can be greatly improved. In order to realize fast operation, after the client decrypts the nth layer ciphertext feature map, the plaintext of the selected part of feature map pixels is directly transmitted to the cloud server to execute homomorphic convolution operation, the specific process is as shown in fig. 7, the method is called as the design of the client automatic encryption selector, and the detailed description is as follows:
after the client finishes the step marking function operation in the step (4), based on the distribution of 0 in the marked matrix composed of 0 and 1 after marking, the client selects to decrypt and transmit partial feature map pixel points to the cloud server, so that the cloud server can quickly execute homomorphic convolution operation. And for which pixel points are decrypted and transmitted to the cloud, a strategy is adopted: the total amount of 0 and plaintext in the marking matrix is considered to be more reasonable below 50%, so when the amount of 0 in the marking matrix exceeds 50% or is about 50%, the plaintext of partial characteristic diagram pixel points is not uploaded to the cloud. For convolutional layers with more than 5 layers, when the number of 0 in the mark matrix is lower than 50%, updating the points in the mark matrix, the upper, lower, left and right adjacent marks of which are not 0, into the decrypted plaintext of the corresponding pixel points, and then transmitting the new mark matrix to the cloud server to update the cloud characteristic diagram.
As shown in fig. 8, the corresponding ciphertext pixel points of the dark color will be decrypted and updated in the tag matrix. When homomorphic convolution operation is carried out at the cloud, because the pixel points in the characteristic diagram have a ciphertext form and a plaintext form at the same time, the design of a lookup table is realized, whether the value of each point is a ciphertext or a plaintext or 0 is judged before homomorphic convolution operation, homomorphic convolution operation is carried out on the pixel points corresponding to the ciphertext, and common convolution multiplication and addition operation is carried out on the points corresponding to the plaintext. Therefore, the characteristic graph is not easy to crack, and the computing speed and efficiency of the cloud server homomorphic convolution are improved.

Claims (6)

1. A deep neural network encryption reasoning method based on homomorphic encryption technology is characterized by comprising the following steps:
(1) Encrypting the picture and transmitting the encrypted picture to a cloud, merging BN layers into a convolution layer after the cloud server receives the encrypted image data, performing homomorphic convolution operation according to a convolution method to obtain a ciphertext feature map after a first layer of convolution operation, and transmitting the ciphertext feature map to a client;
(2) After receiving the ciphertext characteristic diagram after the first layer of homomorphic convolution operation, the client decrypts the ciphertext characteristic diagram, obtains a mark matrix by adopting mark operation, transmits the mark matrix to the cloud server, updates the output characteristic diagram of the first layer of homomorphic convolution layer after the cloud server receives the mark matrix, and replaces the ciphertext of the characteristic diagram with the mark matrix of a position 0 with the ciphertext of a position 0 and does not change with the ciphertext of the position 1, so that the activation function of the ciphertext deep neural network is realized by the cooperation of a mark matrix method of the client and a characteristic diagram updating method on the cloud server, and finally the input characteristic diagram of the second layer of homomorphic convolution layer is obtained; sequentially transmitting pixel points of the input feature map into a GPU according to the position of the convolution window in the input feature map of the homomorphic convolution layer of the second layer, finishing homomorphic convolution operation of the second layer under a ciphertext domain by adopting a multi-core parallel processing ciphertext polynomial mode, sequentially destroying the pixel points of the input feature map in the GPU to obtain a ciphertext feature map after the convolution operation of the second layer, and transmitting the ciphertext feature map to a client; the specific process of obtaining the marking matrix by adopting the marking operation comprises the following steps: representing numbers which are larger than 0 in each point in the characteristic diagram as 1 and numbers which are smaller than or equal to 0 as 0 by a step marking function by using a step marking method to obtain a marking matrix which consists of 0 and 1;
for convolution layers with more than 5 layers, when the number of 0 in the mark matrix is lower than 50%, updating the point with 1 in the mark matrix and the points with 1 marks on the upper, lower, left and right sides into the decrypted plaintext of the corresponding pixel point to obtain the mark matrix;
(3) And (5) repeating the step (2) until all ciphertext feature maps after convolution layer operation are obtained.
2. The deep neural network encryption reasoning method based on homomorphic encryption technology as claimed in claim 1, wherein in the step (1), the homomorphic encryption algorithm is adopted to encrypt the picture; the parameters of the homomorphic encryption algorithm are polynomial modulus, ciphertext modulus and plaintext modulus.
3. The deep neural network encryption reasoning method based on homomorphic encryption technology as claimed in claim 1, wherein in step (1), the ciphertext feature map after the first layer of convolution operation is calculated by the following formula:
Y i ′＝W′ i *X+B′ i
wherein the content of the first and second substances,
Figure FDA0003983218270000021
Figure FDA0003983218270000022
w 'in the formula' i Is the weight of the ith kernel in the homomorphic convolution kernel, γ i The ith channel, σ, representing a rescaling parameter γ i The ith channel, W, representing the cumulative variance σ i A weight representing an ith convolution kernel; x is an input characteristic diagram; b' i For the bias of the ith kernel in the homomorphic convolution kernel, B i Denotes the offset, μ, of the ith convolution kernel i I channel, beta, representing the cumulative mean mu i The ith channel representing the retranslation parameter beta.
4. The deep neural network encryption reasoning method based on homomorphic encryption technology of claim 1, wherein the step marking function is as follows:
Figure FDA0003983218270000023
where x is the input pixel value and y is the output flag.
5. The deep neural network encryption inference method based on homomorphic encryption technology as claimed in claim 1, wherein in step (2), the convolution window is located in the input feature map of the homomorphic convolution layer of the second layer, and the specific process of sequentially transmitting the pixel points of the input feature map to the GPU is as follows: when the homomorphic convolution windows are in the first row and the first column of the input feature map, transmitting data of convolution windows 3x3 of all input channels to a GPU for homomorphic convolution operation, when the homomorphic convolution windows are in the first row and not in the first column of the input feature map, transmitting data of the third column of the convolution windows of all input channels to the GPU for homomorphic convolution operation, when the homomorphic convolution windows are in the first row and not in the first column of the input feature map, transmitting data of the third row of the convolution windows of all input channels to the GPU for homomorphic convolution operation, and when the homomorphic convolution windows are not in the first row and not in the first column of the input feature map, transmitting data of the third row of the convolution windows of all input channels and not in the third column to the GPU for homomorphic convolution operation.
6. The deep neural network encryption reasoning method based on homomorphic encryption technology according to claim 1, wherein in the step (2), the specific process of sequentially destroying the pixel points of the input feature map in the GPU is as follows: when the homomorphic convolution windows are located in the last row and the last column of the input feature map, data of convolution windows 3x3 of all input channels are destroyed after the GPU completes homomorphic convolution operation of the convolution windows, when the homomorphic convolution windows are not located in the last column of the input feature map, data of the first row of the convolution windows of all the input channels are destroyed after the GPU completes homomorphic convolution operation of the convolution windows, when the homomorphic convolution windows are not located in the last column of the input feature map, data of the first column of the convolution windows of all the input channels are destroyed after the GPU completes homomorphic convolution operation of the convolution windows, and when the homomorphic convolution windows are not located in the last row and the last column of the GPU, data of the first row and the first column of the convolution windows of all the input channels are destroyed after the homomorphic convolution operation of the convolution windows is completed.
CN202110064852.7A 2021-01-18 2021-01-18 Deep neural network encryption reasoning method based on homomorphic encryption technology Active CN112818360B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110064852.7A CN112818360B (en) 2021-01-18 2021-01-18 Deep neural network encryption reasoning method based on homomorphic encryption technology

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110064852.7A CN112818360B (en) 2021-01-18 2021-01-18 Deep neural network encryption reasoning method based on homomorphic encryption technology

Publications (2)

Publication Number Publication Date
CN112818360A CN112818360A (en) 2021-05-18
CN112818360B true CN112818360B (en) 2023-03-28

Family

ID=75869985

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110064852.7A Active CN112818360B (en) 2021-01-18 2021-01-18 Deep neural network encryption reasoning method based on homomorphic encryption technology

Country Status (1)

Country Link
CN (1) CN112818360B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113541921B (en) * 2021-06-24 2022-06-10 电子科技大学 Method for realizing fully homomorphic encryption by using GPU
CN113435406A (en) * 2021-07-15 2021-09-24 支付宝(杭州)信息技术有限公司 Face recognition method and device for realizing privacy protection
CN114168991B (en) * 2022-02-10 2022-05-20 北京鹰瞳科技发展股份有限公司 Method, circuit and related product for processing encrypted data
CN117077162B (en) * 2023-07-31 2024-04-19 上海交通大学 Privacy reasoning method, system, medium and electronic equipment based on Transformer network model
CN117640249B (en) * 2024-01-23 2024-05-07 工业云制造(四川)创新中心有限公司 Data security sharing method based on opposite side calculation

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6391900B1 (en) * 2017-11-27 2018-09-19 三菱電機株式会社 Homomorphic reasoning apparatus, homomorphic reasoning method, homomorphic reasoning program, and secret information processing system
CN110543901A (en) * 2019-08-22 2019-12-06 阿里巴巴集团控股有限公司 image recognition method, device and equipment
CN111984960A (en) * 2020-07-13 2020-11-24 深圳市捷讯云联科技有限公司 Privacy protection equipment identification model design and use method based on homomorphic encryption

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11087223B2 (en) * 2018-07-11 2021-08-10 International Business Machines Corporation Learning and inferring insights from encrypted data

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6391900B1 (en) * 2017-11-27 2018-09-19 三菱電機株式会社 Homomorphic reasoning apparatus, homomorphic reasoning method, homomorphic reasoning program, and secret information processing system
CN110543901A (en) * 2019-08-22 2019-12-06 阿里巴巴集团控股有限公司 image recognition method, device and equipment
CN111984960A (en) * 2020-07-13 2020-11-24 深圳市捷讯云联科技有限公司 Privacy protection equipment identification model design and use method based on homomorphic encryption

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Privacy-Preserving Deep Learning and Inference;M. Sadegh Riazi等;《2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)》;20190103;全文 *
Towards the AlexNet Moment for Homomorphic Encryption: HCNN, the First Homomorphic CNN on Encrypted Data With GPUs;Ahmad Al Badawi等;《 IEEE Transactions on Emerging Topics in Computing》;20200806;第9卷(第3期);全文 *
一种处理隐私保护数据的神经网络;王启正等;《密码学报》;20190415(第02期);全文 *

Also Published As

Publication number Publication date
CN112818360A (en) 2021-05-18

Similar Documents

Publication Publication Date Title
CN112818360B (en) Deep neural network encryption reasoning method based on homomorphic encryption technology
Brutzkus et al. Low latency privacy preserving inference
Chou et al. Faster cryptonets: Leveraging sparsity for real-world encrypted inference
US20200366459A1 (en) Searching Over Encrypted Model and Encrypted Data Using Secure Single-and Multi-Party Learning Based on Encrypted Data
CN110751291B (en) Method and device for realizing multi-party combined training neural network of security defense
Li et al. Optimizing privacy-preserving outsourced convolutional neural network predictions
CN107633175B (en) Chaos-based rapid color image encryption method
CN108199828B (en) Method and device for encrypting color picture
CN111597574A (en) Parallel image encryption system and method based on spatial diffusion structure
Devi et al. Entropy influenced RNA diffused quantum chaos to conserve medical data privacy
Raghuvanshi et al. A data encryption model based on intertwining logistic map
Ibarrondo et al. Banners: Binarized neural networks with replicated secret sharing
CN112906052B (en) Aggregation method of multi-user gradient permutation in federated learning
CN110807484A (en) Privacy protection VGG-based secret image identification method and system
Mohamad Data hiding by using AES Algorithm
CN115580687B (en) Multi-image encryption method based on variable parameter hyperchaotic system and S-shaped diffusion
CN115865307B (en) Data point multiplication operation method for federal learning
CN109559269B (en) Image encryption method and terminal
Perusheska et al. Deep learning-based cryptanalysis of different AES modes of operation
Mousa Bat-genetic encryption technique
He et al. Cryptoeyes: Privacy preserving classification over encrypted images
Cheng et al. Private inference for deep neural networks: a secure, adaptive, and efficient realization
WO2022137447A1 (en) Confidential information processing system, and confidential information processing method
Brindha Periodicity analysis of Arnold Cat Map and its application to image encryption
Fu et al. A novel parallel image encryption scheme using chaos

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant