CN105989161A - Big data processing method and apparatus - Google Patents

Abstract

Embodiments of the present invention provide a method and device for processing big data. The method includes: receiving a query instruction sent by a client, and determining a query function K according to the query instruction; querying the large data set D according to the query function K to obtain a query result R, the query result R={R _j }, Wherein 1≤j≤m, m is a positive integer greater than or equal to 1; obtain the sensitivity S(K) of the query function K, the sensitivity S(K) characterizes the sensitivity of the query function K; according to the query result R and sensitivity S(K) determine the noise N that needs to be added to the query result R, noise N={N _j }, the noise component N _j of noise N corresponds to the query result component R _j one by one; according to the noise component N _j The query result component R _j is subjected to noise-adding processing to obtain a noisy query result R'={R' _j }. The embodiment of the present invention proposes a method capable of adding noise to big data, and can output data query results with differential privacy.

Description

A method and device for processing big data

technical field

The present invention relates to the field of data processing, and more particularly, to a method and device for processing big data.

Background technique

The purpose of privacy-preserving data mining is to protect personal privacy data while promoting data sharing among users. Differential privacy is a rigorous theoretical model used to describe and analyze data publishing methods, with the aim of providing efficient methods to maximize the accuracy of statistical query information from statistical databases, while minimizing the chance of identifying individual records.

The current feasible data processing process with differential privacy can only be applied to small-scale data, but for large data, each component of the query result vector has independent coordinates, and each independent coordinate is an exponential Therefore, there is no effective way to implement differential privacy on large-scale data.

Contents of the invention

Embodiments of the present invention provide a method and device for processing big data, which can achieve the purpose of querying with differential privacy on large-scale data.

In the first aspect, the embodiment of the present invention provides a method for processing big data, including: receiving a query instruction sent by a client, and determining a query function K according to the query instruction; Query to obtain query result R, the query result R={R _j }, wherein 1≤j≤m, m is a positive integer greater than or equal to 1; obtain the sensitivity S(K) of the query function K, the The sensitivity S(K) characterizes the sensitivity of the query function K; the noise N that needs to be added to the query result R is determined according to the query result R and the sensitivity S(K), and the noise N={N _j }, the noise component N _j of the noise N is in one-to-one correspondence with the query result component R _j ; according to the noise component N _j , the query result component R _j is subjected to a noise-adding process to obtain a noisy query result R' = {R' _j }.

With reference to the first aspect, in the first possible implementation manner of the first aspect, the acquiring the sensitivity S(K) of the query function K(x) includes: acquiring the query result K(D1) of the data set D1 and the query result K (D2) of the data set D2; the minimum value of the difference between the query result K (D1) and the query result K (D2) in a metric space is used as the sensitivity S (K) Value, wherein, the data set D1 and the data set D2 are two different subsets of the large data set D, and the difference between the data set D1 and the data set D2 is at most one record data.

With reference to the first aspect or the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the determination of the noise according to the query result R and the sensitivity S(K) N includes: generating noise N' satisfying the Laplace noise distribution according to the query result R, wherein each noise component in the noise N' is independent of each other; correcting the noise N' according to the sensitivity S(K) Then the noise N is obtained, wherein the noise component N _j satisfies the Laplace noise distribution of the sensitivity S(K).

In combination with the first aspect or the first to second possible implementation methods of the first aspect, in a third possible implementation manner of the first aspect, the query function K is a hash function F, and the method includes: According to the training set of the large data set D, the hash function F is obtained through training; wherein, the training set is a subset of the large data set D, and the training set also includes an attribute set X and a classification label Y, the attribute set X is a collection of data representing element attributes in the training set, and the classification label Y is a collection of data representing element classification results in the training set.

In a second aspect, an embodiment of the present invention provides a device for processing big data, including: a receiving module, configured to receive a query instruction sent by a client, and determine a query function K according to the query instruction; A first determination module, the first determination module is used to query the large data set D according to the query function K to obtain a query result R, the query result R={R _j }, where 1≤j≤m, m is a positive integer greater than or equal to 1; an acquisition module, the acquisition module is used to acquire the sensitivity S(K) of the query function K determined by the first determination module, and the sensitivity S(K) represents the The sensitivity of the query function K; the second determination module, the second determination module is used to determine the need to add query results R according to the query results R and the sensitivity S (K) obtained according to the acquisition module noise N, the noise N={N _j }, the noise component N _j of the noise N corresponds to the query result component R _j one-to-one; the noise adding module is used to determine according to the second The noise N _j determined by the module adds noise to the query result component R _j to obtain a noisy query result R'={R' _j }.

With reference to the second aspect, in the first possible implementation of the second aspect, the acquisition module is specifically configured to: acquire the query result K(D1) of the dataset D1 and the query result K(D2) of the dataset D2; Setting the norm minimum value of the difference between the query result K(D1) and the query result K(D2) as the value of the sensitivity S(K), wherein the data set D1 and the data set D2 are two different subsets of the large data set D, and there is at most one record data difference between the data set D1 and the data set D2.

, wherein, in combination with the second aspect or the first to possible implementation manners of the second aspect, in the second possible implementation manner of the second aspect, the second determining module is specifically configured to: according to the query result R Generate noise N' that satisfies the Laplace noise distribution, wherein the noise components in the noise N' are independent of each other; the noise N' is obtained after correcting the noise N' according to the sensitivity S(K), where the The noise component _Nj of the noise N satisfies the Laplace noise distribution of the sensitivity S(K).

In combination with the second aspect or the first to second possible implementations of the second aspect, in the third possible implementation of the second aspect, the query function K is a hash function F, and the first determination The module is also used for: training the hash function F according to the training set of the large data set D; wherein, the training set is a subset of the large data set D, and the training set includes an attribute set X and classification label Y, the attribute set X is a collection of data representing element attributes in the training set, and the classification label Y is a collection of data representing element classification results in the training set.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentially private. Noise processing, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required in the embodiments of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

Fig. 1 is a schematic diagram of a system scenario example where an embodiment of the present invention can be applied.

Fig. 2 is a flowchart of a method for processing big data according to an embodiment of the present invention.

Fig. 3 is a flowchart of a method for processing big data according to another embodiment of the present invention.

Fig. 4 is a flowchart of a method for processing big data according to another embodiment of the present invention.

Fig. 5 is a schematic block diagram of an apparatus for processing big data according to an embodiment of the present invention.

Fig. 6 is a schematic block diagram of an apparatus for processing big data according to another embodiment of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

Fig. 1 is a schematic diagram of a system scenario example where an embodiment of the present invention can be applied.

When noise processing with differential privacy is not performed, the client user directly submits a precise query request 1 to the database with original sensitive data, as shown by the dotted arrow 1 in the figure, and the database returns the precise query result 2 to the client, as shown in the figure As shown by the dotted arrow 2, it is easy to leak the individual privacy data of the original sensitive data on the client.

By adding the query mechanism, the client user can submit a statistical query request 3 to the database with original sensitive data through the query mechanism, as shown by the solid line arrow 3 in the figure, the database can return the statistical query result 4 to the client through the query mechanism, in Before returning to the client, the statistical query result is subjected to sensitivity noise processing, and the statistical query result 5 after adding noise is returned to the client, as shown by the solid line arrow 5 in the figure, so that the query result has privacy protection and avoids as much as possible Disclosure of personal data privacy.

Fig. 2 is a flowchart of a method for processing big data according to an embodiment of the present invention. As shown in Figure 2, the method includes:

Step 210, receiving a query instruction sent by the client, and determining a query function K according to the query instruction.

Step 220, query the large data set D according to the query function K to obtain a query result R, the query result R={R _j }, where 1≤j≤m, m is a positive integer greater than or equal to 1.

Step 230, acquiring the sensitivity S(K) of the query function K, where the sensitivity S(K) characterizes the sensitivity of the query function K.

Step 240, determine the noise N to be added to the query result R according to the query result R and the sensitivity S(K), noise N={N _j }, the noise component N _j of the noise N corresponds to the query result component R _j one-to-one.

Step 250: Perform noise-adding processing on the query result component R _j according to the noise component N _j to obtain a noisy query result R'={R' _j }.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, noise is determined based on the sensitivity, and the query result of the original large data set can be noise-added with differential privacy, and finally the query result with differential privacy is obtained. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Specifically, in step 210, after receiving the query instruction sent by the client, a specific statistical query function K is selected by obtaining the data and information query requirements of the relevant user data set from the client. For example, the statistical query function can be summation Functions such as (sum) or average (average) can also be hash functions trained based on classification query results, wherein the query result R is obtained through statistical query in the large data set D according to the query function K.

Specifically, in step 230, the sensitivity S(K) of the query function K is obtained, the sensitivity S(K) characterizes the sensitivity (sensitive) of the query function K, and the sensitivity S(K) for any function F( F) is defined as: the minimum value that satisfies the condition ||F(D ₁ )-F(D ₂ )|| _M ≤ S(F), where the difference between data sets D1 and D2 is at most one record data, and M represents a metric space , it should be understood that the difference of one record data between data sets D1 and D2 means that the value or value type of a certain element is different when the number of data elements in D1 and D2 is the same. Optionally, as an embodiment of the present invention, in step 220, obtaining the sensitivity S(K) of the query function K includes: calculating the query result K(D1) of the data set D1 and the query result K(D2) of the data set D2 ); the minimum value of the difference between the query result K (D1) and the query result K (D2) in a metric space is used as the value of the sensitivity S (K), wherein the difference between the data set D1 and the data set D2 is at most A record data, data set D1 and data set D2 are two different subsets of the large data set D, it should be noted that the difference between query result K(D1) and query result K(D2) in a metric space refers to is the absolute value of the difference between the query result K(D1) and the query result K(D2).

Specifically, obtaining the sensitivity S(K) of the query function K includes calculating the sensitivity S(K) according to the following formula: S(K)=min||K(D ₁ )-K(D ₂ )|| _M , where the difference between dataset D1 and dataset D2 is at most one record data, and M represents a metric space.

Optionally, as an embodiment of the present invention, the determining the noise N according to the query result R and the sensitivity S(K) includes: generating a noise N' satisfying a Laplacian noise distribution according to the query result, where Each noise component in the noise N' is independent of each other; the noise N is obtained after correcting the noise N' according to the sensitivity S(K), and the noise component N _j satisfies the Laplace noise distribution of the sensitivity S(K).

Specifically, in step 230, according to Laplace's differential privacy theorem, a noise adding mechanism is selected to generate noise N'=[N' ₁ ,...,N' _j ,...,N' _m ], where the noise N' Each noise component is independent of each other, and the noise is calibrated according to the sensitivity S(K) of K to obtain the calibrated noise N=[N ₁ ,...,N _j ,...,N _m ], where the calibrated noise N is The individual noise components are independent of each other.

Specifically, adding noise to the query result R _j according to the noise N _j to obtain the query result R' with differential privacy means adding the calibrated noise to the statistical query result and outputting the query result with privacy protection R' ₌ [ R' ₁ , ..., R' _j , ..., R' _m ] = R = [R ₁ , ..., R _j , ..., R _m ] + [N ₁ , ..., N _j , ..., N _m ].

Optionally, as an embodiment of the present invention, according to the query requirements of the client for the large data set D, the query function K of the query result R is determined to be a hash function F, and the query result component of the query result R is R _j , where 1 ≤j≤m, m is a positive integer greater than or equal to 1.

Optionally, as an embodiment of the present invention, according to the client's query requirements for the large data set D, determining the query function K of the query result R as a hash function F includes: according to the training set of the large data set D, training to obtain the hash function K Hide function F, and generate the first hash classification table T of hash function F; Wherein, the training set includes attribute set X and classification label Y, and described attribute set X is the collection of the data that characterizes element attribute in described training set, The classification label Y is a collection of data representing element classification results in the training set.

Optionally, as an embodiment of the present invention, performing noise addition processing on the query result component R _j according to the noise N _j to obtain the query result R' with differential privacy includes: according to the noise N _j to the hash function F(x) Noise-adding processing is performed on the first hash classification table T to obtain a second hash classification table T' with differential privacy for the query result R'.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentially private. Noise processing, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Fig. 3 is a flowchart of a method for processing big data according to an embodiment of the present invention. As shown in Figure 3, the method includes:

Step 310, receiving a query instruction sent by the client, and determining a query function F according to the query instruction, where the query function F is a hash function.

Step 320, query the large data set D according to the query function F to obtain a query result R, the query result R={R _j }, where 1≤j≤m, m is a positive integer greater than or equal to 1.

Step 330, acquiring the sensitivity S(K) of the query function F, where the sensitivity S(K) characterizes the sensitivity of the query function F.

Step 340, determine the noise N to be added to the query result R according to the query result R and the sensitivity S(K), noise N={N _j }, the noise component N _j of the noise N corresponds to the query result component R _j one-to-one.

Step 350: Perform noise-adding processing on the query result component R _j according to the noise component N _j to obtain a noisy query result R'={R' _j }.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentiated Noise processing for privacy, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

It should be understood that in step 110, the query requirement of the above-mentioned large data set R refers to, for example, a statistical query function obtained by summing a certain attribute in a certain subset of R, where 1≤j≤m, m is a positive integer greater than or equal to 1, and can be trained to construct a hash function F by constructing Differentially Private Random Decision Hashing (English: Differentially Private Random Decision Hashing, abbreviated: DPRDH) according to the query requirements of the large data set R.

Optionally, as an embodiment of the present invention, obtaining the hash function F of the large data set R according to the large data set R includes: constructing a random decision hash with differential privacy according to the training set of the large data set D, and obtaining Hash function F, and generate the first hash classification table T of hash function F; wherein, the training set includes an attribute set X and a classification label Y, and the attribute set X is the data representing the attribute of an element in the training set The classification label Y is a collection of data representing element classification results in the training set. It should be understood that according to the query requirements for the large data set D, m hash classification tables can be obtained through training, and the m hash classification tables are all the first hash classification table T. It should be understood that the first hash classification table T is a class of hash classification table that contains at least one hash classification table obtained through the training set of the large data set D.

Specifically, construct a random decision-making hashing process with differential privacy as follows: input the training set {attribute set X, classification label Y}, the first hash classification table contains m initial hash classification tables and L classes, where, The attribute set X can be numerical, categorical, and binary, and the classification label Y is the label symbol obtained after classification according to the attribute set X. Under the classification label Y, it corresponds to L classes, L is a positive integer greater than or equal to 1; output m hash classification tables, the set of m hash classification tables is T=[T ₁ ,...,T _j ,...,T _m ], that is, the first hash classification table is obtained T, for any one of the sub-tables T _j =[bk _key1 , bk _key2 , . . . , bk _keyL ].

Specifically, according to the above input and output parameters, construct a random decision hash with differential privacy as follows:

1. Randomly generate m mask vectors (maskvector), wherein any mask vector is maskvector _j ;

2. The attribute set X in the input training set is uniformly encoded as a binary type, and m binary types Xbinary are obtained, wherein any binary type is encoded as Xbinary _j ;

3. The training process for constructing a random hash with differential privacy is as follows:

For i=1; i≤|X|; ++i do

For j=1, j≤m; ++j do

Calculate the key value, key=maskvector _j AndXbinary _j ;

Assign the key value to the hash classification table, bk _key = T _j [key];

Adjust the key value bk _key(Y) +=1 in the hash classification table;

end

end

It should be understood that, corresponding to the outer loop in step 3, it refers to performing an inner loop on each element in the attribute set X to assign their corresponding key values to m hash classification tables; Corresponding to the inner loop, according to the result of the mask vector logic and binary encoding as the key value, assign each key value to any hash classification table T _j , and loop m times to obtain m hash classifications Table T=[T ₁ , . . . , T _j , . . . , T _m ].

4. After step 3, m hash classification tables can be obtained, T = [T ₁ , ..., T _j , ..., T _m ], any sub-table T _j = [bk _key1 , bk _key2 , ..., bk _keyL ] corresponds to Table 1, and each column vector Y=[Y _1bi ,...,Y _ibi ,...,Y _Lbn ] of this table is called a mask vector.

Table 1

Classification label (L class) bucket 1 Barrel 2 ... bucket i ... barrel n Y ₁ Y _1b1 Y _1b2 Y _1bi Y _1bn Y ₂ Y _2b1 Y _2b2 Y _2bi Y _2bn … Y _i Y _ib1 Y _{ib2 Yibi} Y _ibn … Y _L Y _Lb1 Y _Lb2 Y _L Y _Lbn

Optionally, as an embodiment of the present invention, calculating the sensitivity S(F) of the query function F(x) includes: calculating the query result K(D1) of the data set D1 and the query result K(D2) of the data set D2; The minimum value of the difference between the query result K(D1) and the query result K(D2) in a metric space is used as the value of the sensitivity S(K), where there is at most one record difference between the data set D1 and the data set D2 Data, dataset D1 and dataset D2 are two different subsets of the large dataset D.

Optionally, as an embodiment of the present invention, the sensitivity S(F) of the above hash function F(x) is calculated by the following formula: S(F)=min||F(D ₁ )-F(D ₂ )|| _M , where the difference between data sets D1 and D2 is at most one record data, and M represents a metric space.

Optionally, the determining the noise N according to the query result R and the sensitivity S(K) includes: generating a noise N' satisfying the Laplace noise distribution according to the query result, wherein each noise component in the noise N' is independent of each other; according to After the sensitivity S(K) corrects the noise N', the noise N is obtained, and the noise component N _j satisfies the Laplace noise distribution of the sensitivity S(K), that is, the noise component N _j satisfies Lap(S(F)/ε) , so that the query result R' after adding noise has ε-differential privacy.

Optionally, as an embodiment of the present invention, according to the client's query requirements for the large data set D, determining the query function K(x) of the query result R as a hash function F(x) includes: according to the large data set D The training set is trained to obtain the hash function F(x), and generates the first hash classification table T of the hash function F(x); wherein, the training set includes the attribute set X and the classification label Y of the large data set D .

Optionally, as an embodiment of the present invention, performing noise addition processing on the query result component _Rj according to the noise _Nj , and obtaining the query result R' with differential privacy includes:

Noise processing is performed on the first hash classification table T of the hash function F(x) according to the noise N _j to obtain a second hash classification table T' corresponding to the query result R' with differential privacy.

Optionally, as an embodiment of the present invention, by constructing a Differentially Private Random Decision Hashing Classifier (English: Differentially Private Random Decision Hashing Classifier, abbreviated: DPRDHC), the query result R' with differential privacy can be predicted and output.

Specifically, construct a differentially private random hash classifier to predict the output query result R' as follows:

1. Input m second sets of hash classification tables, T'=[T' ₁ ,...T' _j ,...T' _m ] and the classified identification column X';

2. Initialize the classification label vector (label vectors), classification statistics (label count and label average);

3. Encode the classified column X';

4. The prediction process of constructing a random hash classifier with differential privacy is as follows:

For j=1; j≤m; ++j do

Calculate the key value, key=maskvector _j AndXbinary _j ;

Assign the key value to the hash classification table, bk _key = T _j [key];

Adjust the key value label count+=bk _key in the hash classification table;

end

5. Calculate the arithmetic mean of the classification labels in m second-class hash classification tables, labelavg=label count/m;

6. Take the maximum value among the m labels as the classification label value, Y'=argmax(label avg);

7. Output the classification label value Y'.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentiated Noise processing for privacy, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

The following describes the embodiment of the present invention in more detail in combination with specific steps.

Fig. 4 is a flowchart of a method for processing big data according to another embodiment of the present invention. As shown in Figure 4, the method 400 includes the following steps:

In step 401, a statistical query function F is obtained.

Step 402, generating mutually independent noises N'=[N' ₁ , ..., N' _j , ..., N' _m ].

Step 403, calculating the standard deviation D=[D ₁ , . . . , D _j , . . . , D _m ] of the noise N'.

Step 404, calculating the sensitivity S(F) of the statistical query function.

Step 405, by calibrating the standard deviation D of the noise N', the calibrated noise N=[N ₁ , . . . , N _j , . . . , N _m ] is obtained.

Step 406, obtain statistical query results R=[R ₁ , . . . , R _j , . . . , R _m ].

Step 407, the calibrated noise N is added to the statistical query results, and the privacy-protected query results are output R'=[R' ₁ ,...,R' _j ,...,R' _m ]=R=[R ₁ ,...,R _j , . . . , R _m ]+[N ₁ , . . . , N _j , . . . , N _m ].

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentiated Noise processing for privacy, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Optionally, in step 401, according to the aggregation information query requirements of the relevant user data sets of the client, select a specific statistical query function F, such as a summation or averaging function, etc., which can also be obtained by training based on classification query results hash function.

Optionally, in step 402, according to the query result of the statistical query function F, an appropriate noise mechanism can be selected to generate mutually independent noise N'=[N' ₁ ,...,N' _j ,...,N' _m ], each component of the noise N' is independent of each other, for example, N' can satisfy the Laplace noise distribution, then each component of N' is mutually independent and satisfies the Laplace noise distribution of. It should be understood that selecting an appropriate noise mechanism refers to selecting a noise addition mechanism according to Laplace's differential privacy theorem.

Optionally, in step 403, the standard deviation of each independent component in the noise N' is calculated separately to obtain the standard deviation D=[D ₁ , . . . , D _j , . . . , D _m ].

Optionally, in step 404, calculate the query result F(D1) of the data set D1 and the query result F(D2) of the data set D2; take the query result F(D1) and the query result F(D2) in a metric space The minimum value of the inner difference is used as the value of the sensitivity S(K), wherein the difference between the data set D1 and the data set D2 is at most one record data. Specifically, calculating the sensitivity S(F) of the statistical query function F includes calculating the sensitivity S(F) according to the following formula: S(F)=min||F(D ₁ )-F(D ₂ )|| _M , where the data set D1 and the data set D2 differ by at most one record data, M represents a metric space, and the data set D1 and the data set D2 are two different subsets of the large data set D.

Optionally, in step 405, the calibrated noise N=[N ₁ ,...,N _j ,...,N _m ] is obtained by calibrating the standard deviation D of the noise N', so that each of the calibrated noise N A component N _j satisfies Lap(S(F)/ε), so as to output query results with ε-differential privacy, where ε ranges between [0,1], and ε can be specified by the user.

Optionally, in step 406, the statistical query result R=[R ₁ ,...,R _j ,...,R _m ] is obtained according to the statistical query function F. It should be understood that this step can also be obtained before generating the noise N', The present invention is not limited thereto.

Optionally, in step 407, the calibrated noise N is added to the statistical query result, and the privacy-protected query result R'=[R' ₁ ,...,R' _j ,...,R' _m ]=R= [R ₁ ,...,R _j ,...,R _m ]+[N ₁ ,...,N _j ,...,N _m ], since each component in the noise N is calibrated according to the statistical function sensitivity S(F) The latter is obtained and satisfies the Lap(S(F)/ε) distribution, therefore, the output query result R' has ε-differential privacy.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentiated Noise processing for privacy, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Figures 1 to 4 describe in detail the specific process of processing big data from the perspective of methods, and the following describes the device for processing big data in detail in conjunction with Figures 5 to 6 .

Fig. 5 is a schematic block diagram of an apparatus for processing big data according to an embodiment of the present invention. As shown in FIG. 5 , the apparatus 500 includes: a receiving module 510 , a first determining module 520 , a calculating module 530 , a second determining module 540 and a noise adding module 550 .

The receiving module 510 is configured to receive a query instruction sent by the client, and determine a query function K according to the query instruction.

The first determination module 520, the first determination module is used to query the large data set D according to the query function K to obtain the query result R, the query result R={R _j }, where 1≤j≤m, m is greater than or equal to 1 positive integer of .

An acquiring module 530, configured to acquire the sensitivity S(K) of the query function K determined by the first determining module, where the sensitivity S(K) characterizes the sensitivity of the query function K.

The second determination module 540, the second determination module is used to determine the noise N that needs to be added to the query result R according to the query result R and the sensitivity S(K) obtained by the acquisition module, noise N={N _j }, noise N The noise component N _j is in one-to-one correspondence with the query result component R _j .

A noise adding module 550, configured to add noise to the query result component R _j according to the noise N _j determined by the second determining module, to obtain a noisy query result R'={R' _j }.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentiated Noise processing for privacy, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Specifically, the receiving module 510 selects a specific statistical query function K by acquiring the data and information query requirements of the relevant user data sets from the client. For example, the statistical query function may be sum or average, etc. can also be a hash function trained based on classification query results, wherein the query result R is obtained through statistical query in the large data set D according to the query function K.

Optionally, as an embodiment of the present invention, the acquisition module 530 is specifically configured to: calculate the query result K(D1) of the data set D1 and the query result K(D2) of the data set D2; combine the query result K(D1) with the query The minimum value of the difference of the result K(D2) is set as the value of the sensitivity S(K), wherein there is at most one record data difference between the data set D1 and the data set D2, and the data set D1 and the data set D2 are large data Two different subsets of set D, it should be understood that the difference of one record data between data sets D1 and D2 means that when D1 and D2 have the same number of data elements, the value or value type of a certain element is different. At the same time, it should be noted that the difference between the query result K(D1) and the query result K(D2) refers to the absolute value of the difference between the two.

Specifically, the acquisition module 520 is also used to calculate the sensitivity S(K) according to the following formula: S(K)=min||K(D ₁ )-K(D ₂ )|| _M , where the data sets D1 and D2 There is at most one record data difference, the data set D1 and the data set D2 are two different subsets of the large data set D, and M represents a metric space.

Optionally, as an embodiment of the present invention, the second determination module 530 is specifically configured to: generate noise N' satisfying the Laplace noise distribution according to the query result, wherein each noise component in the noise N' is independent of each other; according to The noise N is obtained after the sensitivity S(K) corrects the noise N', and the noise component N _j satisfies the Laplace noise distribution of the sensitivity S(K).

Optionally, as an embodiment of the present invention, the first determination module 510 is further configured to: determine the query function K(x) of the query result R as the hash function F(x ), the query result component of the query result R is R _j , where 1≤j≤m, m is a positive integer greater than or equal to 1.

Optionally, as an embodiment of the present invention, the first determination module 510 is also configured to: obtain the hash function F(x) through training according to the training set of the large data set D, and generate the first hash function F(x) A hash classification table T; wherein, the training set is a subset of the large data set D, the training set includes an attribute set X and a classification label Y, and the attribute set X is a collection of data representing element attributes in the training set , the classification label Y is a collection of data representing element classification results in the training set.

Optionally, as an embodiment of the present invention, the first determination module 510 is further configured to perform noise addition processing on the first hash classification table T of the hash function F(x) according to the noise N _j to obtain the query result R ' _j a second hash sorting table T' with differential privacy.

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentiated Noise processing for privacy, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Fig. 6 is a schematic block diagram of an apparatus for processing big data according to another embodiment of the present invention. It should be noted that the device shown in FIG. 6 corresponds to the embodiments in FIGS. 2 to 4, and can implement various processes of the method for processing big data in the embodiments in FIGS. 1 to 4, and detailed descriptions are appropriately omitted to avoid repetition. An apparatus for processing big data as shown in FIG. 6 includes: a processor 610 , a memory 620 and a bus 630 . Wherein, the processor 610 and the memory 620 are connected through a bus 630 , the memory 620 is used for storing instructions, and the processor 610 is used for executing the instructions stored in the memory 620 . Specifically, the processor 610 is configured to: receive the query instruction sent by the client, and determine the query function K according to the query instruction; query the large data set D according to the query function K to obtain the query result R, and the query result R={R _j }, where 1≤j≤m, m is a positive integer greater than or equal to 1; obtain the sensitivity S(K) of the query function K, the sensitivity S(K) represents the sensitivity of the query function K; according to the query result R Determine the noise N that needs to be added to the query result R with the sensitivity S(K), noise N={N _j }, the noise component N _j of the noise N is in one-to-one correspondence with the query result component R _j ; according to the noise component N _j , the query The result component R _j is subjected to noise-adding processing to obtain a noisy query result R'={R' _j }.

Optionally, as an embodiment of the present invention, the processor 610 is configured to acquire the query result K(D1) of the data set D1 and the query result K(D2) of the data set D2; combine the query result K(D1) and the query result K (D2) The minimum value of the difference in a metric space is taken as the value of the sensitivity S(K), where there is at most one record data difference between the data set D1 and the data set D2, and the data set D1 and the data set D2 are large data sets Two different subsets of D.

Specifically, the processor 610 is used to obtain the sensitivity S(K) according to the following formula: S(K)=min||K(D ₁ )-K(D ₂ )|| _M , wherein, the data set D1 and the data set D2 differs by at most one record data, dataset D1 and dataset D2 are two different subsets of large dataset D, and M represents a metric space.

Optionally, as an embodiment of the present invention, the processor 610 is configured to generate noise N' satisfying the Laplacian noise distribution according to the query result, wherein each noise component in the noise N' is independent of each other; according to the sensitivity S(K) After correcting the noise N', the noise N is obtained, and the noise component N _j satisfies the Laplace noise distribution of the sensitivity S(K).

In the embodiment of the present invention, by determining the sensitivity of the query function of the large data set, based on the sensitivity, the noise that needs to be added to the query result is determined and the noise is added to the query result, so that the query result of the original large data set can be differentiated Noise processing for privacy, and finally get query results with differential privacy. Therefore, the implementation of the present invention can add noise to large-scale large data, avoid the leakage of sensitive data as much as possible, and realize the purpose of differential privacy query.

Those of ordinary skill in the art can realize that, in combination with the various method steps and units described in the embodiments disclosed herein, they can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the possibility of hardware and software For interchangeability, in the above description, the steps and components of each embodiment have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each particular application, but such implementation should not be regarded as exceeding the scope of the present invention.

The methods or steps described in connection with the embodiments disclosed herein may be implemented by hardware, software programs executed by a processor, or a combination of both. The software program can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

Although the present invention has been described in detail in conjunction with preferred embodiments with reference to the accompanying drawings, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, those skilled in the art can make various equivalent modifications or replacements to the embodiments of the present invention, and these modifications or replacements should all fall within the scope of the present invention.

Claims (8)

1. A method for processing big data, comprising: receiving the query instruction sent by the client, and determining the query function K according to the query instruction; Query the large data set D according to the query function K to obtain a query result R, the query result R={R _j }, wherein 1≤j≤m, m is a positive integer greater than or equal to 1; Acquiring the sensitivity S(K) of the query function K, the sensitivity S(K) representing the sensitivity of the query function K; Determine the noise N that needs to be added to the query result R according to the query result R and the sensitivity S(K), the noise N={N _j }, the noise component N _j of the noise N and the query result component R _j one-to-one correspondence; Noise-adding processing is performed on the query result component R _j according to the noise component N _j to obtain a noisy query result R'={R' _j }. 2. method according to claim 1, is characterized in that, the sensitivity S (K) of described acquisition described query function K comprises: Obtain the query result K(D1) of dataset D1 and the query result K(D2) of dataset D2; Taking the minimum value of the difference between the query result K(D1) and the query result K(D2) in a metric space as the value of the sensitivity S(K), wherein the data set D1 and the data Set D2 is two different subsets of the large data set D, and the difference between the data set D1 and the data set D2 is at most one record data. 3. The method according to any one of claims 1 and 2, wherein said determining noise N according to said query result R and said sensitivity S(K) comprises: Generate noise N' satisfying Laplace noise distribution according to the query result R, wherein each noise component in the noise N' is independent of each other; The noise N is obtained after correcting the noise N' according to the sensitivity S(K), wherein the noise component N _j of the noise N satisfies the Laplace noise distribution of the sensitivity S(K). 4. The method according to any one of claims 1 to 3, wherein the query function K is a hash function F, and the method comprises: According to the training set of the large data set D, the hash function F is obtained through training; Wherein, the training set is a subset of the large data set D, and the training set also includes an attribute set X and a classification label Y, and the attribute set X is a collection of data representing element attributes in the training set, The classification label Y is a collection of data representing element classification results in the training set. 5. A device for processing big data, comprising: A receiving module, the receiving module is used to receive the query instruction sent by the client, and determine the query function K according to the query instruction; A first determination module, the first determination module is used to query the large data set D according to the query function K to obtain a query result R, the query result R={R _j }, where 1≤j≤m, m is a positive integer greater than or equal to 1; An acquisition module, the acquisition module is used to acquire the sensitivity S(K) of the query function K determined by the first determination module, the sensitivity S(K) characterizes the sensitivity of the query function K; A second determination module, the second determination module is used to determine the noise N that needs to be added to the query result R according to the query result R and the sensitivity S(K) obtained by the acquisition module, the noise N ={N _j }, the noise component N _j of the noise N is in one-to-one correspondence with the query result component R _j ; A noise adding module, configured to add noise to the query result component R _j according to the noise N _j determined by the second determining module, to obtain a noisy query result R'={R' _j }. 6. The device according to claim 5, wherein the acquiring module is specifically used for: Obtain the query result K(D1) of dataset D1 and the query result K(D2) of dataset D2; Setting the norm minimum value of the difference between the query result K(D1) and the query result K(D2) as the value of the sensitivity S(K), wherein the data set D1 and the data set D2 are two different subsets of the large data set D, and there is at most one record data difference between the data set D1 and the data set D2. 7. The device according to any one of claims 5 or 6, wherein the second determination module is specifically configured to: Generate noise N' satisfying Laplace noise distribution according to the query result R, wherein each noise component in the noise N' is independent of each other; The noise N is obtained after correcting the noise N' according to the sensitivity S(K), wherein the noise component N _j of the noise N satisfies the Laplace noise distribution of the sensitivity S(K). 8. The device according to any one of claims 7 to 7, wherein the query function K is a hash function F, and the first determination module is also used for: According to the training set of the large data set D, the hash function F is obtained through training; Wherein, the training set is a subset of the large data set D, the training set includes an attribute set X and a classification label Y, and the attribute set X is a collection of data representing element attributes in the training set, so The classification label Y is a collection of data representing element classification results in the training set.