CN111382459B

CN111382459B - Private data integration method and server

Info

Publication number: CN111382459B
Application number: CN201910485170.6A
Authority: CN
Inventors: 高铭智; 王邦杰; 游家牧; 吕品慧; 刘凯诚
Original assignee: Industrial Technology Research Institute ITRI
Current assignee: Industrial Technology Research Institute ITRI
Priority date: 2018-12-27
Filing date: 2019-06-05
Publication date: 2022-05-17
Anticipated expiration: 2039-06-05
Also published as: CN111382459A

Abstract

The disclosure provides a private data integration method and a server. The privacy data integration method comprises the following steps. The first processing device and the second processing device respectively obtain a first generative model and a second generative model according to the first privacy data and the second privacy data. The server generates first generation data and second generation data through the first generation model and the second generation model respectively. The server integrates the first generated data and the second generated data to generate composite data.

Description

Private data integration method and server

Technical Field

The disclosure relates to a private data integration method and a server.

Background

For some business purposes, companies may need to share customer data with each other. However, the fields of different customer data may be different, so data integration is a very difficult task. Therefore, it is necessary to provide a data integration method to perform the work of data integration.

Furthermore, the customer data may contain some private information. There may be concerns about revealing customer private data during the data integration process. Therefore, how to develop a data integration method with privacy protection has become an important development direction of big data technology.

Disclosure of Invention

The disclosure relates to a private data integration method and a server.

According to an embodiment of the present disclosure, a method for integrating private data is provided. The privacy data integration method comprises the following steps. The first processing device and the second processing device respectively obtain a first generative model and a second generative model according to the first privacy data and the second privacy data. The server generates first generation data and second generation data through the first generation model and the second generation model respectively. The server integrates the first generated data and the second generated data to generate composite data.

According to an embodiment of the present disclosure, a server is provided. The server is used for executing the privacy data integration method. The privacy data integration method comprises the following steps. First and second production data are generated by the first and second production models, respectively. The first generative model and the second generative model are obtained according to the first privacy data and the second privacy data respectively. The first generated data and the second generated data are integrated to generate synthesized data.

In order that the manner in which the above recited and other aspects of the present disclosure are obtained can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to the embodiments thereof which are illustrated in the appended drawings.

Drawings

FIG. 1 is an architecture of lateral data integration according to an embodiment;

FIG. 2A is a diagram illustrating a Database linking algorithm (Database J)_oin Algorithm) performs one embodiment of lateral data integration;

FIG. 2B is a block diagram illustrating one embodiment of a Record Link Algorithm (Record Link Algorithm) performing horizontal data integration;

FIG. 2C is a flow diagram illustrating one embodiment of a Statistical matching Algorithm (Statistical Match Algorithm) performing horizontal data integration;

FIG. 3 illustrates a one-to-one processing device, a two-to-one processing device, and a server according to an embodiment;

FIG. 4 is a flow diagram of a method of private data integration, according to an embodiment;

FIG. 5 is a flow diagram of a database linking algorithm, record linking algorithm, or statistical matching algorithm, chosen in accordance with one embodiment;

FIG. 6 is a flowchart illustrating a process for obtaining a joint probability distribution according to one embodiment;

FIG. 7 is a joint probability distribution according to another embodiment.

Description of the symbols:

100: first processing device

200: second processing device

300: server

900: network

A. B, C, D, DA, EC, IC, ID, NR, X, Y, Z: field(s)

CT 53: linked list

GD 51: first generation data

GD 52: second generation data

GM 51: first generation model

GM 52: second generative model

HV 1: first hash value

HV 2: second hash value

LK: link scores

JPD53, JPD 53': joint probability distribution

NCT 53: noise list table

ND: noisy data

PD1l, PD21, PD31, PD41, PD 51: first privacy data

PD12, PD22, PD32, PD42, PD 52: second privacy data

RV1, RV 2: random vector

S110, S120, S130, S131, S132, S133, S134, S135, S136, S140, S150: step (ii) of

SD13, SD23, SD33, SD43, SD 53: synthesizing data

SP 53: sampling data

Detailed Description

Please refer to fig. 1, which is a structure of horizontal data integration according to an embodiment. The first private data PD11 has a field Y and a field X, which include "(Y11, X11), (Y12, X12), (Y13, X13)", and the second private data PD12 has a field Z and a field X, which include "(Z21, X21), (Z22, X22), (Z23, X23)".

The first privacy data PD11 and the second privacy data PD12 may be integrated into a composite data SD13 having a field Y, X, Z. For example, synthetic data SD13 includes "(y 31, x31, z31), (y32, x32, z32), (y33, x33, z 33)". The field Y, X of the synthesized data SD13 and the field Y, X of the first privacy data PD11 have similar joint probability distributions, and the field Z, X of the synthesized data SD13 and the field Z, X of the second privacy data PD12 have similar joint probability distributions. Therefore, the synthesized data SD13 may represent the first privacy data PD11 and the second privacy data PD12 at the same time.

In addition, "(y 11, x11), (y12, x12), (y13, x 13)" of the first private data PD11 and "(z 21, x21), (z22, x22), (z23, x 23)" of the second private data PD12 are not directly displayed in the synthesized data SD 13. Therefore, the result of data integration has a function of privacy protection.

Please refer to fig. 2A, which illustrates an embodiment of a Database linking Algorithm (Database Join Algorithm) for performing horizontal data integration. A first privacy data PD21 has fields EC, ID, IC. The field EC is an Energy Consumption Level (Energy Consumption Level), the field ID is a User identity (User Identification), and the field IC is an Income Level (Income Level). The user identity is a Direct Identifier column (Direct Identifier), and the energy consumption level and the income level are Indirect Identifier columns (Indirect Identifier column). The direct identification column may point directly to a person; and a non-direct identification column cannot point directly to a person. A second privacy data PD22 has fields NR, ID, IC. The field NR is the Total Number of rooms (Total Number of Rooms). The total number of rooms is an indirect identification column. In the database link algorithm, a field ID and an IC are connection Key items (Joint keys). For example, the contents of the field ID of the first privacy data PD21 (or the second privacy data PD22) are filled in the field ID of a composite data SD 23. According to the content of the field ID of the first privacy data PD21, the content of the field EC of the first privacy data PD21 is correspondingly filled in the field EC of the synthetic data SD 23. The contents of the field NR of the second privacy data PD22 are correspondingly filled in the field NR of the synthesized data SD23 in accordance with the contents of the field ID of the second privacy data PD 22. The contents of the field IC of the first privacy data PD21 (or the second privacy data PD22) are correspondingly filled in the field IC of the synthesized data SD23 in accordance with the contents of the field ID of the first privacy data PD21 (or the second privacy data PD 22).

Please refer to fig. 2B, which illustrates an embodiment of performing horizontal data integration by Record linking Algorithm (Record linking Algorithm). A first privacy data PD31 has fields EC, IC, DA. The field EC is the energy consumption level, the field IC is the income level, and the field DA is the total liability. The energy consumption level, income level and total liability are indirect identification columns. A second privacy data PD32 has fields NR, IC, DA. The field NR is the total number of rooms. The total number of rooms is an indirect identification column. In the record linking algorithm, the fields IC, DA are used to calculate a linking Score (linking Score) LK. For example, the linking score LK of the first column of the first privacy data PD31 and the first column of the second privacy data PD32 is 1.8. The first column of the first privacy data PD21 has a link score of 0.8 with the seventh column of the second privacy data PD 22. The first private data PD31 is linked with the second private data PD32 by a linking score LK to obtain synthetic data SD33 having fields EC, IC, NR.

Please refer to fig. 2C, which illustrates an embodiment of a Statistical matching Algorithm (Statistical Match Algorithm) performing horizontal data integration. The first privacy data PD41 has fields EC, IC, DA. The field EC is the energy expenditure level, the field IC is the income level, and the field DA is the total liability. The energy consumption level, income level and debt total are the indirect identification columns. A second privacy data PD42 has fields NR, IC, DA. The field NR is the total number of rooms. The total number of rooms is an indirect identification column. In the statistical matching algorithm, a common field DA is used to calculate an Absolute Value of Error (Absolute Value of Error). For example, the absolute value of the error with respect to 302 (the first column of the field DA of the first privacy data PD 41) is "2, 13, 189, 77, 49, 4, 142" for the field DA of the second privacy data PD 42. The absolute value of the error with respect to 310 (second column of the field DA of the first privacy data PD 41) is "189, 204, 2, 114, 240, 177, 49" for the field DA of the second privacy data PD 42. The first private data PD41 and the second private data PD42 are linked by the absolute values of the errors, and a synthesized data SD43 having the fields EC, IC, NR is obtained.

Please refer to fig. 3 to 4. Fig. 3 shows a first processing device 100, a second processing device 200, and a server 300 according to an embodiment. FIG. 4 is a flow diagram of a method of private data integration, according to an embodiment. The first processing device 100 and the second processing device 200 are, for example but not limited to, a computer, a chip or a circuit board. The first processing device 100 is installed in one company, and the second processing device 200 is installed in another company. The server 300 is, for example but not limited to, a computer, a cloud Computing center, a Computing Cluster System (Computing Cluster System) or an Edge Computing System (Edge Computing System). The server 300 is provided to a third party. The first processing device 100 and the server 300 may communicate via the network 900, and the second processing device 200 and the server 300 may communicate via the network 900. The private data matching method is described with reference to the first processing device 100, the second processing device 200, and the server 300.

In step S110, the first processing device 100 and the second processing device 200 respectively obtain a first generation model GM51 and a second generation model GM52 according to a first privacy data PD51 and a second privacy data PD 52. For example, the first privacy data PD51 has a field A, B, C, and the second privacy data PD52 has a field D, B, C. The generative model is given a target value of "Y" for the Y variable, which is the conditional probability (i.e., X | Y ═ Y). A category content of the first privacy data PD51 or the second privacy data PD52 is converted into a numerical content. The first privacy data PD51 and the second privacy data PD52 are not directly transmitted to the server 300. In fact, only the parameters of the first generative model GM51 and the parameters of the second generative model GM52 are transmitted to the server 300.

Next, in step S120, the server 300 generates the first and second generated data GD51 and GD52 using the first and second generation models GM51 and GM 52. The first generation model GM51 or the second generation model GM52 is obtained by a generation Algorithm (generic Algorithm), such as a Variational Auto-Encoder (VAE) Algorithm, a Generic Adaptive Network (GAN) Algorithm, an information generating adaptive Network (Info-GAN), an adaptive Auto-Encoder (AAE) Algorithm, or an Adaptive Learning (ALI) Algorithm. In this step, after a random vector RV1 is input to the first generation model GM51, the first generation model GM51 outputs first generation data GD 51. The first generated data GD51 is not identical to the first privacy data PD51, but has a similar joint probability distribution. After the other random vector RV2 is input to the second generation model GM52, the second generation model GM52 outputs second generation data GD 52. The second generated data GD52 is not identical to the second privacy data PD52, but has a similar joint probability distribution.

Then, in step S130, the server 300 integrates the first generated data GD51 and the second generated data GD52 to obtain a combined data SD 53. In step S130, the first generated data GD51 and the second generated data GD52 may be obtained through a database linking algorithm (e.g., the manner described in fig. 2A), a record linking algorithm (e.g., the manner described in fig. 2B), or a statistical matching algorithm (e.g., the manner described in fig. 2C). The final synthesized data SD53 contains category contents that were converted into numerical contents.

Referring now to FIG. 5, a flowchart illustrating a database linking algorithm, record linking algorithm, or statistical matching algorithm selected according to one embodiment is shown. In step S131, the server 300 obtains the First Hash Value (First Hash Value) HV1 (shown in fig. 3) of the First private data PD51 and the Second Hash Value (Second Hash Value) HV2 (shown in fig. 3) of the Second private data PD52 from the First processing apparatus 100 and the Second processing apparatus 200. The first hash value HV1 is obtained by encoding the content of a directly identified column or a representative indirectly identified column of the first privacy data PD 51. The second hash value HV2 is obtained by encoding the content of a directly identified column or a representative indirectly identified column of the second privacy data PD 52.

In step S132, the server 300 compares the first hash value HV1 with the second hash value HV2 to determine whether the overlapping ratio of the first generated data GD51 and the second generated data GD52 is higher than a predetermined value. The overlapping rate of the first generated data GD51 and the second generated data GD52 is a rate of repeated contents. If the overlap ratio is not higher than the predetermined value, the process goes to step S136; if the overlap ratio is higher than the predetermined value, the process proceeds to step S133.

In step S133, the server 300 determines whether the first generated data GD51 and the second generated data GD52 have at least one connection Key (Joint Key). If the first generated data GD51 and the second generated data GD52 have a connection key item, proceed to step S134; if the first generated data GD51 and the second generated data do not have the connection key item, the process proceeds to step S135.

In step S134, the server 300 integrates the first generated data GD51 and the second generated data GD52 by using a database link algorithm (the method described in fig. 2A).

In step S135, the server 300 integrates the first generated data GD51 and the second generated data GD52 by using a record linking algorithm (the method described in fig. 2B). In this step, the recording link algorithm is adopted to integrate the first generated data GD51 with the second generated data GD52 without using the connection key item.

In step S136, the server 300 integrates the first generated data GD51 and the second generated data GD52 by using a statistical matching algorithm (the method described in fig. 2C).

In step S140 of fig. 4, the server 300 obtains a Joint Probability Distribution (Joint Probability Distribution) JPD53 of the synthesized data SD 53. In step S140, a noise data ND is added to the joint probability distribution JPD 53. Referring to fig. 6, a process of obtaining a joint probability distribution JPD53 according to an embodiment is described. First, the synthetic data SD53 is converted into a linked list (containment Table) CT 53. The number of times of various combinations of fields EC, IC, NR is filled in the listing table CT 53. Next, the noise data ND is added to the netlist CT53 to obtain a noise netlist (noise containment Table) NCT 53. Then, the degree of the noise list NCT53 is converted into a probability value to obtain a joint probability distribution JPD 53.

Furthermore, in another embodiment, the dimension of the joint probability distribution JPD53 may be reduced. Please refer to fig. 7, which shows a joint probability distribution JPD 53' according to another embodiment. The joint probability distribution JPD53 of dimension 3 is converted into a joint probability distribution JPD 53' of dimension 2. Therefore, the complexity can be reduced from 5^3 to 5^2+5^2, so that the operation load and the operation time can be effectively reduced.

Next, in step S150 of fig. 4, the server 300 samples the resultant data SD53 according to the joint probability distribution JPD53 (or the joint probability distribution JPD 53') to obtain a sampled data SP 53. The content of the sampled data SP53 approximates the content of the first privacy data PD51 and the second privacy data PD 52.

According to the privacy data integration method described above, the sample data SP53 is obtained by integration of the first privacy data PD51 and the second privacy data PD 52. The sampled data SP53 can represent the approximate contents of the first privacy data PD51 and the second privacy data PD52, but does not leak any privacy data of the client. This is quite useful for big data technologies. Furthermore, the number of privacy data is not intended to limit the disclosure. For example, three or more than three private data may also be performed by the above-mentioned private data integration method.

In summary, although the present disclosure has been described with reference to the above embodiments, the disclosure is not limited thereto. Various modifications and adaptations may occur to those skilled in the art without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure should be determined by the following claims.

Claims

1. A method for integrating private data is characterized in that the method for integrating private data comprises the following steps:

the first processing device and the second processing device respectively obtain a first generative model and a second generative model according to the first privacy data and the second privacy data;

the server generates first generation data through the first generation model and generates second generation data through the second generation model; and

the server integrates the first generated data and the second generated data through a database linking algorithm, or through a statistical matching algorithm, or through a record linking algorithm to generate synthetic data;

wherein the first generative model or the second generative model is obtained by a generative algorithm.

2. The private data integration method of claim 1, wherein in the step of obtaining the first generative model and the second generative model, the first generative model or the second generative model is obtained by a variational autoencoder algorithm, a generative confrontation network algorithm, an information generating confrontation network, a confrontation autoencoder algorithm, or a confrontation inference learning algorithm.

3. The method according to claim 1, wherein in the step of obtaining the first generative model and the second generative model, the category content of the first private data or the second private data is converted into numerical content, and the composite data includes the category content.

4. The private data integration method of claim 1, wherein the step of integrating the first generated data and the second generated data comprises:

and if the first generated data and the second generated data have at least one connection key item, integrating the first generated data and the second generated data by adopting the database link algorithm.

5. The private data integration method of claim 4, wherein the step of integrating the first generated data and the second generated data comprises:

and if the overlapping rate of the first generated data and the second generated data is not higher than a preset value, integrating the first generated data and the second generated data by adopting the statistical matching algorithm.

6. The private data integration method of claim 5, wherein the step of integrating the first generated data and the second generated data comprises:

and if the overlapping rate of the first generated data and the second generated data is higher than a preset value and the first generated data and the second generated data do not have the connection key item, integrating the first generated data and the second generated data by adopting the record link algorithm.

7. The private data syndication method according to claim 1, further comprising:

a joint probability distribution of the composite data is obtained.

8. The method of claim 7, wherein the joint probability distribution is transformed to reduce dimensionality.

9. The method of claim 7, wherein in the step of obtaining the joint probability distribution, noise data is added to the joint probability distribution.

10. The private data syndication method according to claim 7, further comprising:

the resultant data is sampled to obtain a sampled data.

11. The privacy data integration method according to claim 10, wherein contents of the sample data approximate contents of the first privacy data and contents of the second privacy data.

12. A server for performing a private data integration method, the private data integration method comprising:

generating first generating data through a first generating model, and generating second generating data through a second generating model, wherein the first generating model and the second generating model are respectively obtained according to first privacy data and second privacy data; and

integrating the first generated data and the second generated data through a database linking algorithm, or through a statistical matching algorithm, or through a record linking algorithm to generate synthetic data;

13. The server of claim 12, wherein the first generative model or the second generative model is obtained by a variational autoencoder algorithm, a generative confrontation network algorithm, an information generative confrontation network, a confrontation autoencoder algorithm, or a confrontation inference learning algorithm.

14. The server according to claim 12, wherein category contents of the first privacy data or the second privacy data are converted into numerical contents, and the composite data contains the category contents.

15. The server of claim 12, wherein the step of integrating the first generated data and the second generated data comprises:

16. The server of claim 15, wherein the step of integrating the first generated data and the second generated data comprises:

17. The server of claim 16, wherein the step of integrating the first generated data and the second generated data comprises:

and if the overlapping rate of the first generated data and the second generated data is higher than a preset value and the first generated data and the second generated data do not have the connection key item, integrating the first generated data and the second generated data by adopting a record linking algorithm.

18. The server of claim 12, wherein the private data syndication method further comprises:

a joint probability distribution of the composite data is obtained.

19. The server of claim 18 wherein the joint probability distribution is transformed to reduce dimensionality.

20. The server of claim 18, wherein in the step of obtaining the joint probability distribution, noise data is added to the joint probability distribution.

21. The server of claim 18, wherein the private data syndication method further comprises:

the resultant data is sampled to obtain sampled data.

22. The server of claim 21, wherein the content of the sampled data approximates the content of the first private data and the content of the second private data.