CN114726498B - Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy - Google Patents

Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy Download PDF

Info

Publication number
CN114726498B
CN114726498B CN202210340042.4A CN202210340042A CN114726498B CN 114726498 B CN114726498 B CN 114726498B CN 202210340042 A CN202210340042 A CN 202210340042A CN 114726498 B CN114726498 B CN 114726498B
Authority
CN
China
Prior art keywords
layer
cloud terminal
user
model
matrix
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210340042.4A
Other languages
Chinese (zh)
Other versions
CN114726498A (en
Inventor
徐小平
杨震
陈渝文
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing University of Technology
Original Assignee
Beijing University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing University of Technology filed Critical Beijing University of Technology
Priority to CN202210340042.4A priority Critical patent/CN114726498B/en
Publication of CN114726498A publication Critical patent/CN114726498A/en
Application granted granted Critical
Publication of CN114726498B publication Critical patent/CN114726498B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/008—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols involving homomorphic encryption
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
    • H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
    • H04L63/0442—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload wherein the sending and receiving network entities apply asymmetric encryption, i.e. different keys for encryption and decryption
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/08—Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords
    • H04L9/0861—Generation of secret information including derivation or calculation of cryptographic keys or passwords
    • H04L9/0869—Generation of secret information including derivation or calculation of cryptographic keys or passwords involving random numbers or seeds
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Complex Calculations (AREA)

Abstract

The invention discloses an intelligent home data analysis method based on network layering and adopting homomorphic encryption for interaction, which comprises the following steps: training a DNN model aiming at a specific task at a cloud server; dividing the model by a selected middle layer, deploying the first half section to the user local, and keeping the second half section at the cloud; transmitting the input acquired by devices such as sensors in the intelligent home system into a first half network, and obtaining the output of an intermediate layer; interaction is carried out with the second half section of the cloud network locally in a Paillier homomorphic encryption mode, and finally an output result of the network is obtained; and the cloud server responds to the intelligent home system in time according to the output of the network. According to the method, based on the data analysis by using the traditional neural network, the interactive analysis method of local and cloud encryption is adopted, and the security of homomorphic encryption and the small calculation cost of network layering on the local are utilized, so that the security problem in the data analysis of the intelligent home system is solved. The interactive data analysis method using network layering and encryption has high privacy security and applicability of model deployment scenes.

Description

Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy
Technical Field
The invention relates to the technical field of privacy protection of deep learning and Paillier homomorphic encryption. In particular to an intelligent home data analysis method based on a hierarchical deep neural network.
Background
The Internet of things is regarded as a new information technology industry revolution after computers and the Internet, and people and objects are closely connected together by generalizing things in a scene constructed by the Internet of things. The intelligent home and related industry chains are fully developed under the background of the development of the Internet of things. The intelligent home is connected with a plurality of devices (audio-video equipment, a lighting system, telephone remote control and an anti-theft alarm system) in the home through the internet of things technology, and various means and functions such as home appliance control, telephone remote control, multimedia intelligent recommendation, environment real-time monitoring, fireproof anti-theft alarm, heating ventilation control and timing programming control are provided. A typical smart home system has a hardware architecture diagram shown in fig. 1.
However, in the process of high-speed development and large-scale application, a series of security problems are exploded at the same time, and when the deep learning model stored in the cloud is utilized to perform reasoning service, the input uploaded to the cloud model often contains privacy information of the user. For example, in home fall detection, a camera installed in the home may take a photograph of the elderly in the home and upload it to the cloud, and disclosure of such private data is undesirable to the user. Although the intelligent home equipment in the internet of things has small-scale computing capability by means of cloud technology in the mobile field, due to the requirement of real-time updating and the model with larger computing power, the mode of storing the model locally cannot be widely applied to a general intelligent home system, so that the diversified requirements of users cannot be met.
Disclosure of Invention
The invention is used for solving the problem that the use of a deep learning model in an intelligent home system is affected by the risk of user data privacy disclosure, and particularly provides an interactive data analysis method based on a hierarchical network by utilizing middle layer output and cloud, and meanwhile, protecting the interaction process by utilizing a Paillier homomorphic encryption method. According to the method, based on layering of the deep learning network, the safety of interactive model prediction is utilized, and private data of a local end user is protected from being stolen.
In order to achieve the above purpose, the technical scheme adopted by the invention is based on an interactive data analysis method for carrying out local and cloud on a depth model to be layered, and as shown in the figure, the implementation steps of the method are as follows:
Aiming at the functional requirements of users, training a DNN model F aiming at a specified task in a smart home scene by a cloud terminal, wherein the DNN model F has a structure of a convolutional neural network with k layers;
Step (2) for realizing interactive calculation of the local and cloud terminals, selecting an intermediate layer according to a layer separation mechanism of the neural network, sending a model part before the intermediate layer of the model F, comprising the intermediate layer, as a feature extraction layer to a local user, and simultaneously, keeping the latter half part at the cloud terminal;
step (3), the intelligent equipment in the family inputs the acquired information into a local feature extraction layer, and a user obtains output and carries out data preprocessing on the output information;
Step (4) the user generates a public and private key pair and random parameters required by Paillier homomorphic encryption, and the cloud terminal generates corresponding random parameters;
Step (5) the user carries out homomorphic encryption on the output of the feature extraction layer by using a public key and transmits the homomorphic encryption to a server of the cloud terminal, the cloud terminal converts the convolution kernel into a weight matrix, carries out interactive homomorphic encryption calculation based on ciphertext, and then transmits a calculation result to a local user by the cloud terminal;
Step (6), the user obtains the output of the feature extraction layer and the interactive calculation result of the model F i+1 layer on the cloud terminal, takes the result as the input of the interactive calculation with the model F i+2 layer until the last layer of the model F is calculated, and the cloud terminal obtains the last result and sends out instructions to the intelligent household equipment;
Further, the step (1) further includes: and the cloud terminal provided by the intelligent home service provider is in communication connection with the local Internet of things equipment and is used for remotely controlling the intelligent home equipment. The intelligent household equipment comprises any one or combination of the following equipment: camera, sensor and switching value acquisition terminal.
Further, the step (2) further includes: the intermediate layer of the model F should be selected to beLayer to/>And the layer k is the model F network layer number. The cloud terminal selects an ith layer of the model F, the cloud terminal sends the first layer to the ith layer of the model F as a characteristic extraction layer of the network to the local for the safety requirement of a user on data collected by the local intelligent equipment, the rest part of the model F is reserved at the cloud terminal, and the process of computing the characteristic extraction layer and the cloud terminal network in a homomorphic encryption mode is called interactive computing. The output of the feature extraction layer is preprocessed to obtain/>Homomorphic encryption calculation is carried out on the model F and the weight parameter W i+1 of the (i+1) th layer, namely the 1 st layer of the cloud terminal, and the input/>, of the (i+2) th layer is obtained through local activation and pooling operationKnown as completing a round of interactive calculations;
Further, the step (3) further includes: because the cloud terminal splits the model F, the model F cannot perform general convolution layer calculation. In order to achieve the computational requirements of the convolutional layer in F, the user needs to preprocess the output X i of the feature extraction layer,
Using the results after the treatmentConverting convolution calculation into multiplication of matrix, wherein the preprocessing in the step (3) specifically comprises:
the output of the user characteristic layer is X i, which is a three-dimensional tensor with the size of M i×Ni×Di, wherein M i、Ni respectively represents the row and column sizes of the tensor X i, D i represents the number of channels output by the ith layer of the model F, and the ith layer of the model F, namely the convolution kernel size of the 1 st layer of the cloud terminal is Wherein/>The row and column sizes of the convolution kernel are represented, respectively. The specific first preprocessing process is that a user uses X i to convert three-dimension Zhang Liangbian into a two-dimensional matrix according to the channel sequence, takes the matrix with the convolution kernel size on the matrix of each channel of X i and carries out vectorization longitudinal splicing, and then transversely splices the result of each channel to finally obtain/>I.e. the output after pretreatment is of a size/> Of (3), wherein/>Representing the size of a matrix row,/>Representing the size of the matrix array. Pair/>The m-th line element calculation formula is as follows:
Wherein: Representing the pre-processed feature layer output, vector changing the matrix into a Vector, concat c representing the longitudinal stitching of the matrix, concat r representing the transverse stitching of the matrix,/> The horizontal fetch size on the j-th channel of X i is shown as/>The matrix is changed into a horizontal vector and then is spliced longitudinally, and the whole formula shows that the matrix which is changed on the 1 st to the D i th channels is spliced transversely. The end user gets the matrix/>, which is preprocessedI.e. input of model F ith layer +1. Let/>Is of the size/>Wherein:
The preprocessing is performed first in each subsequent round of interactive computation, and differs from the first preprocessing in that the input of the preprocessing in the jth round of interactive computation is the output of the jth-1 round of computation The required convolution kernel size is the i+j layer of the model F, namely/>, of the j layer of the cloud
Further, the step (4) specifically includes the following sub-steps:
step (4-1): the user randomly generates two independent large prime numbers p and q, meets gcd (pq, (p-1) (q-1))=1, and the attribute is to ensure that the lengths of the two prime numbers are equal, wherein gcd (a and b) represents the maximum common factor of a and b, and the user calculates n=p×q and randomly selects an integer Where Z * represents a non-zero integer set. Generating a public key/>
Step (4-2): lambda is obtained from lambda=lcm (p-1, q-1), and lambda is obtained fromGenerating a private keyWherein L is defined as L (x) = (x-1)/n, where x is an argument, n is the corresponding n value in step (4-1), lcm (a, b) represents the greatest common multiple of a, b, mod represents the modulo operation;
step (4-3): the user calculates the generated random matrix r= [ r i+1,…,rk-1,rk ] locally for each round of interaction in the step (2), which is a random matrix with the size of The cloud terminal synchronously generates a random matrix s= [ s i,…sk-1,sk ], which is a two-dimensional random matrix with the size of/>Where k represents the number of layers of the model F, D i+1 represents the number of i+1 layers of the model F, i.e., the convolution kernel Ke i+1 of the 1 st layer of the cloud terminal, whose size is equal to the number of channels output by that layer.
Further, step (5) completes the interactive computation of the model F convolutional layer, comprising the following sub-steps:
step (5-1) the user uses the public key The method comprises the steps of sending the cloud terminal to the cloud terminal, storing a private key, and carrying out interactive calculation with i+1 to k layers of the cloud terminal, wherein the calculation of a user and a model F in the i+1 layer of the cloud terminal is taken as an example: first the user locally makes use of the public key/>Encrypting r i+1 and transmitting the encrypted result to the cloud terminal, wherein the specific encryption formula is as follows:
Wherein r i+1 represents a random matrix generated by a user for the i+1st layer of the model F, namely the cloud terminal layer 1 interaction calculation, and z 1 represents a selected random number, wherein Enc (r i+1) represents the result of encrypting the random matrix r i+1;
Step (5-2) the cloud terminal changes the (i+1) th layer of the model F, namely a convolution kernel Ker i+1 of the (1) th layer of the cloud terminal into a weight matrix W i+1,Keri+1 with the size of Of (3), wherein/>D i is the size of Ker i+1 rows, columns and channel numbers respectively, and as known from the step (4-3), the total number of convolution kernels is D i+1, and a specific calculation formula is as follows:
wherein: w i+1 [ m ] represents the m-th column of the matrix, namely the result obtained by calculation of the m-th convolution kernel, The (D i -1) channel of the (m) th convolution kernel is represented, the matrix of the (m) th convolution kernel along the channel direction is vectorized one by a right side formula, and is longitudinally spliced into a vector and filled in the (m) th column of the matrix W i+1,
Wi+1＝Concatr(Wi+1[1],…Wi+1[Di+1-1],Wi+1[Di+1-1])
Wherein: w i+1 represents the result of the longitudinal concatenation after all convolution kernels are transformed into vectors, which is a size ofFrom step (3), the size of W i+1, i.e./>
Step (5-3) the cloud terminal utilizes the public keyEncrypting the weight matrix W i+1 of the (i+1) th layer, then carrying out homomorphic encryption multiplication and addition calculation with enc (r i+1), and sending an encryption calculation result to a user, wherein the specific formula is as follows:
the cloud terminal encrypts the W i+1 by using the public key:
The cloud terminal performs homomorphic multiplication calculation on W i+1、ri+1:
And the cloud terminal carries out homomorphic addition calculation on r i+1*Wi+1、si+1:
wherein: z 2 represents a random number used to encrypt s i+1;
Step (5-4) the user uses the private key Decrypting the encryption calculation result sent by the cloud terminal, wherein the purpose is to obtain r i+1*Wi+1-si+1 containing W i+1 after decryption, and the specific formula is as follows:
ri+1*Wi+1-si+1＝L(enc(ri+1*Wi+1-si+1)λmod n2)*μ
Wherein: the function L (x) is a decryption function, and the expression is L (x) = (x-1)/n
The user willSent to cloud terminal, cloud terminal calculates/>And sends the result to the user, who finally gets/>Namely, the convolution calculation result/>, of the ith layer+1 of the model FIt is of size/>Is a matrix of (a);
Step (5-5) the user inserting the matrix Conversion to three-dimensional tensor/>Enabling a user to perform activation and pooling operations locally, three-dimensional tensors/>The conversion operation of (2) is specifically as follows:
In the method, in the process of the invention, Representation matrix/>The elements of row h (m-1) +n and column t,Represent tensor/>Element value of mth row and nth column in t-th channel,/>Is a tensor of size h x D i+1;
step (5-6) the user downloads the bias from the cloud terminal and locally pairs Performing an activate function operation:
Wherein: b represents the bias downloaded by the user from the cloud terminal;
Step (5-7) the user performs pooling operation locally, the result obtained in step (5-6) realizes pooling by matrix multiplication, completes the interactive calculation with the (i+1) th layer of the model F, namely the (1) th layer of the cloud terminal, and takes the result as the input of the interactive calculation between the user and the (i+2) th layer of the model F, namely the (2) th layer of the cloud terminal
Advantageous effects
According to the method, on the basis of a pre-trained intelligent home data analysis model, the model is stored on a local user and a cloud server in a layered and segmented mode, and the model is used for analyzing a home environment through a homomorphic encryption interaction method. The interactive data analysis method using layered and homomorphic encryption remarkably protects the privacy information of the user compared with the original data sharing type data analysis mode.
Drawings
FIG. 1 is a schematic diagram of a smart home system architecture
FIG. 2 is a block diagram of a distributed model
FIG. 3 is a convolution calculation process of matrix multiplication
FIG. 4 is a homomorphic encryption interactive prediction process
Detailed description of the preferred embodiments
The invention aims to provide an intelligent home data analysis method based on network layering, which is used for outputting a result of a model which is used for protecting user privacy more on a pre-trained deep learning model.
The technical scheme provided by the present invention will be described in detail with reference to the specific examples, and it should be understood that the following specific examples are only for illustrating the present invention and are not intended to limit the scope of the present invention
Because the data exchange between the local equipment and the cloud server can be generated by the model required by the running task of the Internet of things equipment in the running process, the data are acquired by malicious attackers in the cloud, and the attackers can infer equipment attributes, equipment states and events in the intelligent home environment and sensitive information of users by directly acquiring plaintext information or carrying out data speculation on intermediate information, so that the privacy of the users is damaged. The intelligent home system is jointly deployed on a cloud server and a local area, and can be conveniently installed in an intelligent home system under the condition of networking. The invention can be used by plug and play, and can start to ensure the privacy safety of users only by being deployed in the network data transmission physical range of the intelligent home environment. The invention firstly acquires the family information data acquired in the sensor equipment for analysis, inputs the family information data into a first half network stored locally, and obtains the output of the middle layer. And then, according to the privacy protection strategy of Paillier homomorphic encryption, the invention uses the middle layer to output and store the model second half section in the cloud for interactive calculation. Because the data are processed in advance to be in a matrix form, the calculation of the convolution layer can be expressed in a matrix multiplication form, so that the aim of protecting the privacy of the user through safe interaction is fulfilled.
Specifically, in the step (1), a deep learning model F for early warning of a user or family behavior in an intelligent home system is pre-trained in a cloud terminal, for example, in a "fall-prevention alarm system", in order to prevent possible fall injury of old people in home, the model carries out corresponding warning prompt locally by the cloud terminal according to pictures captured by a camera in the intelligent home system.
And (2) segmenting the model trained in the cloud terminal from the middle. Specifically, a complete Deep Neural Network (DNN) model includes a convolutional layer, a pooled layer, and a fully connected layer, with several layers in between called hidden layers. For selection of middle layer, according to middle layer selection method (Chen Z,Fu A,Zhang Y,et al.Secure collaborative deep learning against GAN attacks in the Internet of Things[J].IEEE Internet of Things Journal,2020,8(7):5839-5849.) mentioned in paper Secure collaborative DEEP LEARNING AGAINST GAN ATTACKS, in order to make the cost of the user in the local area smaller to ensure model reusability, the invention selects a lower network layer as the middle layer, in this embodiment, a 12-layer image recognition convolutional neural network for a "fall-prevention alarm system" comprises 9 convolutional layers and 3 fully-connected layers, and selects the third layer of the neural networkLayer to layerLayer, where k is the number of network layers, we choose layer 4 as the middle layer.
And (3) inputting the input acquired by the sensor in the intelligent household equipment into a network and preprocessing the output result. Specifically, for example, in the "old people fall-prevention early warning system", a picture shot by a camera is input into local equipment, and after passing through a plurality of hidden layers, an output X i is obtained. In the present embodiment, the sensor device takes a picture-in model F of size 64×64×3 and obtains an intermediate layer output X of size 32×32×8 at the intermediate layer i
In this step, to meet the requirement that the convolution operation becomes matrix multiplication in the interactive model prediction, firstly, the feature output of the intermediate layer is preprocessed, the output of the user feature layer is X i, which is a three-dimensional tensor with size of M i×Ni×Di, where M i、Ni represents the row and column sizes of tensor X i, D i represents the number of channels output by the ith layer of the model F, and the ith+1th layer of the model F, i.e. the convolution kernel size of the 1 st layer of the cloud terminal isWherein/>The row and column sizes of the convolution kernel are represented, respectively. The specific first preprocessing process is that a user uses X i to convert three-dimension Zhang Liangbian into a two-dimensional matrix according to the channel sequence, takes the matrix with the convolution kernel size on the matrix of each channel of X i and carries out vectorization longitudinal splicing, and then transversely splices the result of each channel to finally obtain/>I.e. the output after pretreatment is of a size ofOf (3), wherein/> Representing the size of a matrix row,/>Representing the size of the matrix array. Pair/>The calculation formula of (2) is as follows:
Wherein: Representing the pre-processed feature layer output, vector changing the matrix into a Vector, concat c representing the longitudinal stitching of the matrix, concat r representing the transverse stitching of the matrix,/> The horizontal fetch size on the j-th channel of X i is shown as/>The matrix is changed into a horizontal vector and then is spliced longitudinally, and the whole formula shows that the matrix which is changed on the 1 st to the D i th channels is spliced transversely. The end user gets the matrix/>, which is preprocessedI.e. input of model F ith layer +1. Let/>Is of the size/>Wherein the method comprises the steps of
The preprocessing is performed first in each subsequent round of interactive computation, and differs from the first preprocessing in that the input of the preprocessing in the jth round of interactive computation is the output of the jth-1 round of computationThe required convolution kernel size is the i+j layer of the model F, namely/>, of the j layer of the cloudIn this embodiment, for the intermediate layer output of 32×32×16 and the convolution kernel of 3×3×16, the matrix/>, with the dimension of 900×144 for each channel, is obtained after preprocessing
In the step (4), the user and the cloud terminal generate a public and private key pair for Paillier homomorphic encryptionAnd a random matrix r, s,/>, calculated for each interactionIs a public key,/>Is a private key, and specifically comprises the following substeps:
Step (4-1): the user randomly generates two independent large prime numbers p and q, meets gcd (pq, (p-1) (q-1))=1, and the attribute is to ensure that the lengths of the two prime numbers are equal, wherein gcd (a and b) represents the maximum common factor of a and b, and the user calculates n=p×q and randomly selects an integer Where Z * represents a non-zero integer set. Generating a public key/>
User calculation n=p×q, randomly selecting integersWhere Z * represents a non-zero integer set. Generating public keys
Step (4-2): λ is obtained from λ=lcm (p-1, q-1), and a private key is generated from μ= (L (g λmod n2))-1)Wherein L is defined as L (x) = (x-1)/n, where x is an argument, n is the corresponding n value in step (4-1), lcm (a, b) represents the greatest common multiple of a, b, mod represents the modulo operation;
step (4-3): the user calculates the generated random matrix r= [ r i+1,…,rk-1,rk ] locally for each round of interaction in the step (2), which is a random matrix with the size of The cloud terminal synchronously generates a random matrix s= [ s i,…sk-1,sk ], which is a two-dimensional random matrix with the size of/>Where k represents the number of layers of the model F, D i+1 represents the number of i+1st layers of the model F, i.e., the convolution kernel Ker i+1 of the 1 st layer of the cloud terminal, whose size is equal to the number of channels output by that layer. In this embodiment, the size of the matrix r is 900×144, and the size of the matrix s is 900×6.
And (5) performing interactive calculation on the local user and the cloud server, wherein in the embodiment, the network of the 'old man anti-falling early warning system' interacts the result output by the feature processing layer with the cloud terminal. The method comprises the following steps:
step (5-1) the user uses the public key The method comprises the steps of sending the cloud terminal to the cloud terminal, storing a private key, and carrying out interactive calculation with i+1 to k layers of the cloud terminal, wherein the calculation of a user and a model F in the i+1 layer of the cloud terminal is taken as an example: first the user locally makes use of the public key/>Encrypting r i+1 and transmitting the encrypted result to the cloud terminal, wherein the specific encryption formula is as follows:
Wherein r i+1 represents a random matrix generated by a user for the i+1st layer of the model F, namely the cloud terminal layer 1 interaction calculation, and z 1 represents a selected random number, wherein Enc (r i+1) represents the result of encrypting the random matrix r i+1;
Step (5-2) the cloud terminal changes the (i+1) th layer of the model F, namely a convolution kernel Ker i+1 of the (1) th layer of the cloud terminal into a weight matrix W i+1,Keri+1 with the size of Of (3), wherein/>D i is the size of Ker i+1 rows, columns and channel numbers respectively, and as known from the step (4-3), the total number of convolution kernels is D i+1, and a specific calculation formula is as follows:
wherein: w i+1 [ m ] represents the m-th column of the matrix, namely the result obtained by calculation of the m-th convolution kernel, The (D i -1) channel of the (m) th convolution kernel is represented, the matrix of the (m) th convolution kernel along the channel direction is vectorized one by a right side formula, and is longitudinally spliced into a vector and filled in the (m) th column of the matrix W i+1,
Wi+1＝Concatr(Wi+1[1],…Wi+1[Di+1-1],Wi+1[Di+1-1])
Wherein: w i+1 represents the result of the longitudinal concatenation after all convolution kernels are transformed into vectors, which is a size ofFrom step (3), the size of W i+1, i.e./>In this embodiment, the number of convolution kernels of the i+1th layer of the model F is 6, so the size of the processed matrix W i+1 is 144×6.
Step (5-3) the cloud terminal utilizes the public keyEncrypting the weight matrix W i+1 of the (i+1) th layer, then carrying out homomorphic encryption multiplication and addition calculation with enc (r i+1), and sending an encryption calculation result to a user, wherein the specific formula is as follows:
the cloud terminal encrypts the W i+1 by using the public key:
The cloud terminal performs homomorphic multiplication calculation on W i+1、ri+1:
And the cloud terminal carries out homomorphic addition calculation on r i+1*Wi+1、si+1:
wherein: z 2 represents a random number used to encrypt s i+1;
Step (5-4) the user uses the private key Decrypting the encryption calculation result sent by the cloud terminal, wherein the purpose is to obtain r i+1*Wi+1-si+1 containing W i+1 after decryption, and the specific formula is as follows:
ri+1*Wi+1-si+1＝L(enc(ri+1*Wi+1-si+1)λmod n2)*μ
Wherein: the function L (x) is a decryption function, and the expression is L (x) = (x-1)/n
The user willSent to cloud terminal, cloud terminal calculates/>And sends the result to the user, who finally gets/>Namely, the convolution calculation result/>, of the ith layer+1 of the model FIt is of size/>Is a matrix of (a);
Step (5-5) the user inserting the matrix Conversion to three-dimensional tensor/>The aim is to enable the user to perform activation and pooling operations locally and to perform preprocessing operations in the next round of interactive computation, three-dimensional tensor/>The transformation method of (2) is specifically as follows:
In the method, in the process of the invention, Representation matrix/>The elements of row h (m-1) +n and column t,Represent tensor/>Element value of mth row and nth column in t-th channel,/>Is a tensor of size h x D i+1; in this example,/>Is a matrix of size 900 x 6,/>Is a tensor of size 30 x 6.
To more visually demonstrate the matrix multiplication transformation for the convolution computation, as shown in fig. 3, for the user and the interactive computation of the i+j layer of the model F, i.e. the j layer of the cloud terminal, the input X i+j of this layer that has not been preprocessed, it is a three-dimensional tensor with the size of 3 multiplied by 3, firstly, the three-dimensional tensor is preprocessed, the size of a matrix in a convolution kernel of a j-th layer of the cloud terminal is taken, i.e. 2×2, taking a matrix of size 2×2 on each channel of X i+j and vectorizing, filling in the processed inputIt is a size of 4 x 12. The j-th layer of the cloud terminal has two convolution kernels, which are three-dimensional tensors with the size of 2 multiplied by 3, three matrixes are vectorized into 4 multiplied by 1 vectors along the channel direction for the first convolution kernel, the vectors are longitudinally spliced and then filled in the 1 st column of the weight matrix W i+j, and the same operation is carried out for the second convolution kernel, so that the weight matrix with the size of 12 multiplied by 2 is finally obtained. For W i+j and/>Homomorphic encryption calculation is carried out to obtain an output matrix/>, the size of which is 4 multiplied by 2For the next round of interactive computation, it is converted into two matrices by column, the resulting output is a tensor of size 2 x 2, which is consistent with the results of conventional convolution calculations;
step (5-6) the user downloads the bias from the cloud terminal and locally pairs Performing an activate function operation:
Wherein: b represents the bias downloaded by the user from the cloud terminal;
Step (5-7) the user performs pooling operation locally, the result obtained in step (5-6) realizes pooling by matrix multiplication, completes the interactive calculation with the (i+1) th layer of the model F, namely the (1) th layer of the cloud terminal, and takes the result as the input of the interactive calculation between the user and the (i+2) th layer of the model F, namely the (2) th layer of the cloud terminal In this embodiment, model F layer i+1 pair/>The size of the pooling matrix for pooling is 2×2, and the pooling/>The size is 15×15×6.
The step (6) of full-connection layer can realize interactive full-connection layer calculation by using the homomorphic encryption mode of the step (5) because convolution operation is not involved, and the step five is different in that the full-connection layer has no convolution kernel but a weight matrix, in the interactive calculation with the first full-connection layer, a user needs to input a three-dimensional tensor to be processed into a one-dimensional vector in a data preprocessing stage, and in the interactive calculation of the rest full-connection layer, because the input of each layer is a vector, the input preprocessing is not needed. For example, the (i+j) th layer of the model, i.e. the j th layer of the cloud terminal, is the first fully connected layer of the model F, its inputIt is a three-dimensional tensor with the size of M i+j×Ni+j×Di+j, and in the preprocessing process of the interactive calculation of the full connection layer, the method is characterized by comprising the following steps ofThe matrix on each channel is converted into a vector line by line and is transversely spliced, and the vector processed by each channel is transversely spliced to obtain the pretreated/>Is a row vector of length M i+jNi+jDi+j. In this embodiment, layer 10 is the first fully connected layer, its input/>Is a three-dimensional tensor with the size of 5 multiplied by 16, a row vector with the size of 400 is obtained after pretreatment, the weight matrix W i+1 of the layer is a homomorphic encryption calculation with the size of 400 multiplied by 120 of a user at a local and cloud terminal, and the output/> -of the 10 th layer is finally obtainedWhich is a vector of length 120. And then the output/>, of the prediction model is obtained after the calculation of the full connection layers of the 11 th layer and the 12 th layerIt is a length 5 vector, each value in the vector representing one of the conditions that may occur in a "fall prevention alarm system".
The cloud terminal finally obtains an image analysis result, and controls the intelligent home equipment to make corresponding reactions according to a specific output result given by the model, in the embodiment, when the model is output and displayed that the old people in the family fall down, the alarm device can be started to remind other people in the house to rescue the old people. Meanwhile, the safety data analysis method based on the Internet of things can realize the intelligent monitoring function of home appliances, the monitoring function of home environment, the intelligent processing function of home accidents and the like on the premise of protecting the privacy of users from being acquired by cloud, and realize the customization, combination and association of the functions, thereby further improving the generalization of the functions. The data analysis method adopts a local and cloud interactive method, has low local operation cost, takes care of the limited computing power budget of families, is convenient for deploying more large models, and is easy to popularize.
In the embodiments provided in the present application, it should be understood that the disclosed method may be implemented in other ways without exceeding the spirit and scope of the application. The present embodiments are to be considered in all respects as illustrative and not restrictive, and the intention is not to be limited to the details given.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (6)

1. The intelligent home data analysis method based on the hierarchical network and protecting the privacy of the user is characterized by comprising the following steps of: the method comprises the following steps:
Aiming at the functional requirements of users, training a DNN model F aiming at a specified task in a smart home scene by a cloud terminal, wherein the DNN model F has a structure of a convolutional neural network with k layers;
Step (2) for realizing interactive calculation of the local and cloud terminals, selecting an intermediate layer according to a layer separation mechanism of the neural network, sending a model part before the intermediate layer of the model F, comprising the intermediate layer, as a feature extraction layer to a local user, and simultaneously, keeping the latter half part at the cloud terminal;
step (3), the intelligent equipment in the family inputs the acquired information into a local feature extraction layer, and a user obtains output and carries out data preprocessing on the output information;
Step (4) the user generates a public and private key pair and random parameters required by Paillier homomorphic encryption, and the cloud terminal generates corresponding random parameters;
Step (5) the user carries out homomorphic encryption on the output of the feature extraction layer by using a public key and transmits the homomorphic encryption to a server of the cloud terminal, the cloud terminal converts the convolution kernel into a weight matrix, carries out interactive homomorphic encryption calculation based on ciphertext, and then transmits a calculation result to a local user by the cloud terminal;
Step (6), the user obtains the output of the feature extraction layer and the interactive calculation result of the model F i+1 layer on the cloud terminal, takes the result as the input of the interactive calculation with the model F i+2 layer until the last layer of the model F is calculated, and the cloud terminal obtains the last result and sends out instructions to the intelligent household equipment;
Because the cloud terminal splits the model F, the model F cannot perform general convolution layer calculation, and a user needs to preprocess the output X i of the feature extraction layer to realize the calculation requirement of the convolution layer in the F and utilize the processed result Converting convolution calculation into multiplication of matrix, wherein the preprocessing in the step (3) specifically comprises:
the output of the user characteristic layer is X i, which is a three-dimensional tensor with the size of M i×Ni×Di, wherein M i、Ni respectively represents the row and column sizes of the tensor X i, D i represents the number of channels output by the ith layer of the model F, and the ith layer of the model F, namely the convolution kernel size of the 1 st layer of the cloud terminal is Wherein/>The first preprocessing process comprises the steps that a user uses X i to convert three dimensions Zhang Liangbian into a two-dimensional matrix according to the channel sequence, the matrix of each channel of X i is taken, the matrix of the size of the convolution kernel is vectorized and longitudinally spliced, then the result of each channel is transversely spliced, and finally/>I.e. the output after pretreatment is of a size/> Of (3), wherein/>Representing the size of a matrix row,/>Represents the size of the matrix array, pair/>The calculation formula is as follows:
Wherein: Representing the pre-processed feature layer output, vector changing the matrix into a Vector, concat c representing the longitudinal stitching of the matrix, concat r representing the transverse stitching of the matrix,/> The horizontal fetch size on the j-th channel of X i is shown as/>Is changed into a horizontal vector and then is longitudinally spliced, the whole formula shows that the matrix which is changed on the 1 st to the D i th channels is transversely spliced, and the end user obtains the preprocessed matrix/>I.e. input of model F ith layer +1. Let/>Is of the size/>Wherein the method comprises the steps of
The preprocessing is performed first in each subsequent round of interactive computation, and differs from the first preprocessing in that the input of the preprocessing in the jth round of interactive computation is the output of the jth-1 round of computationThe required convolution kernel size is the i+j layer of the model F, namely/>, of the j layer of the cloud
2. The method for analyzing smart home data based on a hierarchical network and protecting privacy of a user according to claim 1, wherein the step (1) further comprises: the cloud terminal provided by the intelligent home service provider is in communication connection with the local Internet of things equipment and is used for remotely controlling the intelligent home equipment; the intelligent household equipment comprises any one or combination of the following equipment: camera, sensor and switching value acquisition terminal.
3. The hierarchical network-based intelligent home data analysis method for protecting user privacy according to claim 1, wherein: the step (2) further comprises: the intermediate layer of the model F should be selected to beLayer to/>The layer k is the number of layers of the model F network; the cloud terminal selects an ith layer of the model F as a middle layer, the cloud terminal sends the first layer to the ith layer of the model F to the local as a characteristic extraction layer of a network for the safety requirement of a user on data collected by the local intelligent equipment, the rest part of the model F is reserved in the cloud terminal, the process of computing the characteristic extraction layer and the cloud terminal network in a homomorphic encryption mode is called interactive computing, and the output of the characteristic extraction layer is preprocessed to obtain/>Homomorphic encryption calculation is carried out on the model F and the weight parameter W i+1 of the (i+1) th layer, namely the 1 st layer of the cloud terminal, and the input/>, of the (i+2) th layer is obtained through local activation and pooling operationKnown as completing a round of interactive calculations.
4. The hierarchical network-based intelligent home data analysis method for protecting user privacy according to claim 1, wherein: the step (4) specifically comprises the following sub-steps:
step (4-1): the user randomly generates two independent large prime numbers p, q, satisfying gcd (pq, (p-1) (q-1)) =1, this attribute is to ensure that the lengths of the two prime numbers are equal, wherein gcd (a, b) represents the maximum common factor of a, b, the user calculates n=p×q, randomly selects the integer g, Wherein Z * represents a non-zero set of integers; generating a public key/>
Step (4-2): λ is obtained from λ=lcm (p-1, q-1), and a private key is generated from μ= (L (g λmodn2))-1)Wherein L is defined as L (x) = (x-1)/n, where x is an argument, n is the corresponding n value in step (4-1), lcm (a, b) represents the greatest common multiple of a, b, mod represents the modulo operation;
Step (4-3): the user calculates the generated random matrix r= [ r i+1,…,rk-1,rk ] locally for each round of interaction in the step (2), which is a random matrix with the size of The cloud terminal synchronously generates a random matrix s= [ s i,…sk-1,sk ], which is a two-dimensional random matrix with the size of/>Where k represents the number of layers of the model F, D i+1 represents the number of i+1st layers of the model F, i.e., the convolution kernel Ker i+1 of the 1 st layer of the cloud terminal, whose size is equal to the number of channels output by that layer.
5. The hierarchical network-based intelligent home data analysis method for protecting user privacy according to claim 1, wherein: the step (5) comprises the following sub-steps:
step (5-1) the user uses the public key The method comprises the steps of sending the cloud terminal to the cloud terminal, storing a private key, carrying out interactive calculation with the i+1 to k layers of the cloud terminal, and calculating the i+1 layers of the cloud terminal by a user and a model F as follows: first the user locally makes use of the public key/>Encrypting r i+1 and transmitting the encrypted result to the cloud terminal, wherein the specific encryption formula is as follows:
wherein r i+1 represents a random matrix generated by a user for the i+1st layer of the model F, namely the cloud terminal layer 1 interaction calculation, and z 1 represents a selected random number, wherein Enc (r i+1) represents the result of encrypting the random matrix r i+1;
Step (5-2) the cloud terminal changes the (i+1) th layer of the model F, namely a convolution kernel Ker i+1 of the (1) th layer of the cloud terminal into a weight matrix W i+1,Keri+1 with the size of Of (3), wherein/>The sizes of Ker i+1 rows, ker i+1 columns and Ker i+1 channels are known from the step (4-3), and the total number of convolution kernels is D i+1, and the specific calculation formula is:
wherein: w i+1 [ m ] represents the m-th column of the matrix, namely the result obtained by calculation of the m-th convolution kernel, The (D i -1) channel of the (m) th convolution kernel is represented, the matrix of the (m) th convolution kernel along the channel direction is vectorized one by a right side formula, and is longitudinally spliced into a vector and filled in the (m) th column of the matrix W i+1,
Wi+1＝Concatr(Wi+1[1],…Wi+1[Di+1-1],Wi+1[Di+1-1])
Wherein: w i+1 represents the result of the longitudinal concatenation after all convolution kernels are transformed into vectors, which is a size ofFrom step (3), the size of W i+1, i.e./>Step (5-3) cloud terminal utilizes public key/>Encrypting the weight matrix W i+1 of the (i+1) th layer, then carrying out homomorphic encryption multiplication and addition calculation with enc (r i+1), and sending an encryption calculation result to a user, wherein the specific formula is as follows:
the cloud terminal encrypts the W i+1 by using the public key:
The cloud terminal performs homomorphic multiplication calculation on W i+1、ri+1:
And the cloud terminal carries out homomorphic addition calculation on r i+1*Wi+1、si+1:
wherein: z 2 represents a random number used to encrypt s i+1;
Step (5-4) the user uses the private key Decrypting the encryption calculation result sent by the cloud terminal, wherein the purpose is to obtain r i+1*Wi+1-si+1 containing W i+1 after decryption, and the specific formula is as follows:
ri+1*Wi+1-si+1＝L(enc(ri+1*Wi+1-si+1)λmod n2)*μ
Wherein: the function L (x) is a decryption function, and the expression is L (x) = (x-1)/n
The user willSent to cloud terminal, cloud terminal calculates/>And sends the result to the user, who finally gets/>Namely, the convolution calculation result/>, of the ith layer+1 of the model FIt is a single-sizedIs a matrix of (a);
Step (5-5) the user inserting the matrix Conversion to three-dimensional tensor/>Enabling a user to perform activation and pooling operations locally, three-dimensional tensors/>The conversion operation of (2) is specifically as follows:
In the method, in the process of the invention, Representation matrix/>The elements of row h (m-1) +n and column t,Represent tensor/>Element value of mth row and nth column in t-th channel,/>Is a tensor of size h x D i+1;
step (5-6) the user downloads the bias from the cloud terminal and locally pairs Performing an activate function operation:
Wherein: B represents the bias downloaded by the user from the cloud terminal; step (5-7) the user performs pooling operation locally, the result obtained in step (5-6) realizes pooling through a matrix multiplication mode, the i+1 layer of the model F, namely the cloud terminal layer 1, is completed, and the result is used as input/>, of the user, the i+2 layer of the model F, namely the cloud terminal layer 2, for interactive calculation
6. The method for analyzing intelligent home data based on hierarchical network and protecting user privacy according to claim 1, wherein in the step (6), since the calculation of the full connection layer in the model F is also matrix multiplication, the calculation can be performed according to the homomorphic encryption method in the step (5).
CN202210340042.4A 2022-03-31 2022-03-31 Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy Active CN114726498B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210340042.4A CN114726498B (en) 2022-03-31 2022-03-31 Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210340042.4A CN114726498B (en) 2022-03-31 2022-03-31 Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy

Publications (2)

Publication Number Publication Date
CN114726498A CN114726498A (en) 2022-07-08
CN114726498B true CN114726498B (en) 2024-06-25

Family

ID=82242241

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210340042.4A Active CN114726498B (en) 2022-03-31 2022-03-31 Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy

Country Status (1)

Country Link
CN (1) CN114726498B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105577357A (en) * 2015-12-21 2016-05-11 东南大学 Intelligent household data privacy protection method based on full homomorphic encryption
CN113761557A (en) * 2021-09-02 2021-12-07 积至(广州)信息技术有限公司 Multi-party deep learning privacy protection method based on fully homomorphic encryption algorithm

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110598438B (en) * 2019-07-19 2023-05-30 福州大学 Cloud protection outsourcing data privacy protection system based on deep convolutional neural network
CN113869499A (en) * 2021-10-15 2021-12-31 浙江大学 High-efficiency conversion method for unintentional neural network

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105577357A (en) * 2015-12-21 2016-05-11 东南大学 Intelligent household data privacy protection method based on full homomorphic encryption
CN113761557A (en) * 2021-09-02 2021-12-07 积至(广州)信息技术有限公司 Multi-party deep learning privacy protection method based on fully homomorphic encryption algorithm

Also Published As

Publication number Publication date
CN114726498A (en) 2022-07-08

Similar Documents

Publication Publication Date Title
Dhawan et al. SSII: secured and high-quality steganography using intelligent hybrid optimization algorithms for IoT
Mishra et al. Delphi: A cryptographic inference system for neural networks
CN109194507B (en) Non-interactive privacy protection neural network prediction method
CN106600518A (en) Image encryption method and image decryption method with visual security and data security based on compressed sensing
EP2874341B1 (en) Secure evaluation of a program
CN104040934B (en) Image privacy is protected when being manipulated by cloud service
Li et al. Optimizing privacy-preserving outsourced convolutional neural network predictions
CN110139000B (en) Image compression encryption method based on compressed sensing and life game scrambling
US20130318351A1 (en) Similarity degree calculation system, similarity degree calculation apparatus, computer program, and similarity degree calculation method
CN112417467A (en) Image encryption method based on anti-neurocryptography and SHA control chaos
CN113946858B (en) Identity security authentication method and system based on data privacy calculation
Bhatnagar et al. Biometric inspired multimedia encryption based on dual parameter fractional fourier transform
CN114826564B (en) Secret image sharing and recovering method based on block chain
CN109635580A (en) Image encryption method, device, electronic equipment and computer storage medium
CN115102683A (en) Method, system and equipment for encrypting and decrypting face image based on target detection technology
Kumar Technique for security of multimedia using neural network
Shankar et al. Secure image transmission in wireless sensor network (WSN) applications
Tiken et al. A comprehensive review about image encryption methods
CN114726498B (en) Intelligent home data analysis method based on hierarchical network and capable of protecting user privacy
Manisha et al. CBRC: a novel approach for cancelable biometric template generation using random permutation and Chinese Remainder Theorem
US10027479B2 (en) Generating device, encryption device, decryption device, generating method, encryption method, decryption method, and computer program product
Shankar et al. Multiple share creation with optimal hash function for image security in wsn aid of ogwo
US20210132949A1 (en) Method and apparatus for an advanced convolution on encrypted data
CN116527278A (en) Block chain hidden communication method based on generation type hidden network and image double hidden
CN115001654A (en) Chaos-based thread pool and GPU combined optimization batch image encryption method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant