CN111382459A - Privacy data integration method and server - Google Patents

Abstract

This disclosure provides a privacy data integration method and server. The privacy data integration approach includes the following steps. The first processing device and the second processing device respectively obtain the first generation model and the second generation model based on the first privacy data and the second privacy data. The server generates first generated data and second generated data through the first generated model and the second generated model respectively. The server integrates the first generated data and the second generated data to generate composite data.

Description

Privacy data integration method and server

technical field

This disclosure relates to a privacy data integration method and server.

Background technique

Companies may need to share customer data with each other for certain business purposes. However, the fields of different customer data may be different, so data integration is a very difficult task. Therefore, there is a need to provide a data integration method to perform the work of data integration.

In addition, customer data may contain some private information. There may be concerns about leaking private customer data during the data integration process. Therefore, how to develop data integration methods with privacy protection has become an important development direction of big data technology.

SUMMARY OF THE INVENTION

This disclosure relates to a privacy data integration method and server.

According to an embodiment of the present disclosure, a privacy data integration method is provided. The privacy data integration method includes the following steps. The first processing device and the second processing device obtain the first generation model and the second generation model according to the first privacy data and the second privacy data, respectively. The server generates the first generation data and the second generation data through the first generation model and the second generation model respectively. The server integrates the first generated data and the second generated data to generate composite data.

According to an embodiment of the present disclosure, a server is provided. The server is used to execute the privacy data integration method. The privacy data integration method includes the following steps. The first generation data and the second generation data are generated by the first generation model and the second generation model, respectively. The first generation model and the second generation model are obtained according to the first privacy data and the second privacy data, respectively. The first generated data and the second generated data are integrated to generate composite data.

In order to have a better understanding of the above and other aspects of the present disclosure, the following embodiments are given and described in detail in conjunction with the accompanying drawings as follows.

Description of drawings

FIG. 1 is an architecture of horizontal data integration according to an embodiment;

2A illustrates an embodiment of a Database Join Algorithm performing horizontal data integration;

FIG. 2B illustrates an embodiment of the Record Linkage Algorithm performing horizontal data integration;

2C illustrates an embodiment of a Statistical Match Algorithm performing horizontal data integration;

FIG. 3 shows one processing device, one two processing devices, and a server according to an embodiment;

4 is a flowchart of a privacy data integration method according to an embodiment;

5 is a flowchart of a database linking algorithm, a record linking algorithm or a statistical matching algorithm selected according to an embodiment;

6 is a diagram illustrating a process for obtaining a joint probability distribution according to an embodiment;

Figure 7 is a joint probability distribution according to another embodiment.

Symbol Description:

100: First processing device

200: Second processing device

300: server

900: network

A, B, C, D, DA, EC, IC, ID, NR, X, Y, Z: Fields

CT53: Contingency Table

GD51: First generated data

GD52: Second generated data

GM51: The first generative model

GM52: Second generative model

HV1: first hash value

HV2: second hash value

LK: Link Score

JPD53, JPD53’: joint probability distribution

NCT53: Noise Contingency Table

ND: noise data

PD11, PD21, PD31, PD41, PD51: first privacy data

PD12, PD22, PD32, PD42, PD52: second privacy data

RV1, RV2: random vector

S110, S120, S130, S131, S132, S133, S134, S135, S136, S140, S150: Steps

SD13, SD23, SD33, SD43, SD53: Composite data

SP53: Sampling Data

Detailed ways

Please refer to FIG. 1 , which is an architecture of horizontal data integration according to an embodiment. The first privacy data PD11 has field Y and field X, which includes "(y11, x11), (y12, x12), (y13, x13)", and the second privacy data PD12 has field Z and field X, which includes "( z21, x21), (z22, x22), (z23, x23)".

The first privacy data PD11 and the second privacy data PD12 can be integrated into a composite data SD13 having fields Y, X, and Z. For example, the composite data SD13 includes "(y31, x31, z31), (y32, x32, z32), (y33, x33, z33)". The fields Y and X of the synthetic data SD13 have similar association probability distributions with the fields Y and X of the first privacy data PD11 , and the fields Z and X of the synthetic data SD13 have similar association probability with the fields Z and X of the second privacy data PD12 distributed. Therefore, the composite data SD13 can represent the first privacy data PD11 and the second privacy data PD12 at the same time.

In addition, "(y11, x11), (y12, x12), (y13, x13)" of the first privacy data PD11 and "(z21, x21), (z22, x22), (z23, x23)" is not directly displayed in synthetic data SD13. Therefore, the result of data integration has the function of privacy protection.

Please refer to FIG. 2A , which illustrates an embodiment of horizontal data integration performed by a Database Join Algorithm. A first privacy data PD21 has fields EC, ID, IC. The field EC is Energy Consumption Level, the field ID is User Identification, and the field IC is Income Level. The user identity is a direct identifier column (Direct Identifier), and the energy consumption level and the income level are an indirect identifier column (Indirect Identifier). A direct identity column can point directly to a person; a non-direct identity column cannot point directly to a person. A second privacy data PD22 has fields NR, ID, IC. The field NR is the Total Number of Rooms. The total number of rooms is an indirect identification column. In the database linking algorithm, the field ID and IC are the joint key items (Joint Key). For example, the content of the field ID of the first privacy data PD21 (or the second privacy data PD22) is filled in the field ID of a composite data SD23. According to the content of the field ID of the first privacy data PD21, the content of the field EC of the first privacy data PD21 is correspondingly filled in the field EC of the composite data SD23. According to the content of the field ID of the second privacy data PD22, the content of the field NR of the second privacy data PD22 is correspondingly filled in the field NR of the composite data SD23. According to the content of the field ID of the first privacy data PD21 (or the second privacy data PD22 ), the content of the field IC of the first privacy data PD21 (or the second privacy data PD22 ) is correspondingly filled in the field IC of the composite data SD23 middle.

Please refer to FIG. 2B , which illustrates an embodiment of the Record Linkage Algorithm performing horizontal data integration. A first privacy data PD31 has fields EC, IC, DA. The field EC is the energy consumption level, the field IC is the income level, and the field DA is the total liability. Energy consumption levels, income levels, and total liabilities are indirect identification columns. A second privacy data PD32 has fields NR, IC, DA. The field NR is the total number of rooms. The total number of rooms is an indirect identity column. In the record linking algorithm, the fields IC and DA are used to calculate a Linkage Score LK. For example, the link score LK of the first column of the first privacy data PD31 and the first column of the second privacy data PD32 is 1.8. The link score of the first column of the first privacy data PD21 and the seventh column of the second privacy data PD22 is 0.8. The first privacy data PD31 and the second privacy data PD32 are linked through the link score LK to obtain synthetic data SD33 having fields EC, IC, NR.

Please refer to FIG. 2C , which illustrates an embodiment of horizontal data integration performed by a Statistical Match Algorithm. The first privacy data PD41 has fields EC, IC, DA. The field EC is the energy consumption level, the field IC is the income level, and the field DA is the total liability. Energy consumption level, income level and total liabilities are indirect identification columns. A second privacy data PD42 has fields NR, IC, DA. The field NR is the total number of rooms. The total number of rooms is an indirect identity column. In the statistical matching algorithm, the common field DA is used to calculate an Absolute Value of Error. For example, for the field DA of the second privacy data PD42, the absolute value of the error relative to 302 (the first column of the field DA of the first privacy data PD41) is "2, 13, 189, 77, 49, 4 , 142”. For the field DA of the second privacy data PD42, the absolute value of the error relative to 310 (the second column of the field DA of the first privacy data PD41) is "189, 204, 2, 114, 240, 177, 49". The first privacy data PD41 and the second privacy data PD42 are connected by the absolute values of these errors to obtain a composite data SD43 having fields EC, IC, NR.

Please refer to Figures 3 to 4. FIG. 3 shows a first processing device 100 , a second processing device 200 and a server 300 according to an embodiment. FIG. 4 is a flowchart of a privacy data integration method according to an embodiment. The first processing device 100 and the second processing device 200 are, for example (but not limited to) a computer, a chip or a circuit board. The first processing device 100 is installed in a certain company, and the second processing device 200 is installed in another company. The server 300 is, for example (but not limited to) a computer, a cloud computing center, a computing cluster system or an edge computing system. The server 300 is installed in a third party. The first processing device 100 and the server 300 can communicate through the network 900 , and the second processing device 200 and the server 300 can also communicate through the network 900 . The private data integration method is described by the first processing device 100 , the second processing device 200 and the server 300 .

In step S110, according to a first privacy data PD51 and a second privacy data PD52, the first processing device 100 and the second processing device 200 obtain a first generation model GM51 and a second generation model GM52, respectively. For example, the first privacy data PD51 has fields A, B, and C, and the second privacy data PD52 has fields D, B, and C. The generative model is that the target value of the given Y variable is "y", and the X variable is the conditional probability (ie X|Y=y) c. A type of content of the first privacy data PD51 or the second privacy data PD52 is converted into a numerical content . The first privacy data PD51 and the second privacy data PD52 are not directly transmitted to the server 300 . In fact, only the parameters of the first generation model GM51 and the parameters of the second generation model GM52 are transmitted to the server 300 .

Next, in step S120, the server 300 generates the first generation data GD51 and the second generation data GD52 using the first generation model GM51 and the second generation model GM52. The first generative model GM51 or the second generative model GM52 is obtained by a generative algorithm (Generative Algorithm), such as a variational auto-encoder (Variational Auto-Encoder, VAE) algorithm, a generative adversarial network (Generative Adversarial Network, GAN) ) algorithm, an information generation adversarial network (Info-GAN), an AAE algorithm or an ALI algorithm. In this step, after a random vector RV1 is input to the first generation model GM51, the first generation model GM51 outputs the first generation data GD51. The first generated data GD51 is not the same as the first private data PD51, but has a similar joint probability distribution. Then, after another random vector RV2 is input to the second generation model GM52, the second generation model GM52 outputs the second generation data GD52. The second generated data GD52 is not the same as the second private data PD52, but has a similar joint probability distribution.

Then, in step S130, the server 300 integrates the first generated data GD51 and the second generated data GD52 to obtain a composite data SD53. In step S130, the first generated data GD51 and the second generated data GD52 may use a database linking algorithm (eg, as described in FIG. 2A ), a record linking algorithm (eg, as described in FIG. 2B ), or a statistical matching algorithm (for example, in the manner described in FIG. 2C ). The final composite data SD53 contains the category contents that have been converted into numerical contents.

Please refer to FIG. 5 , which is a flowchart of a database linking algorithm, a record linking algorithm or a statistical matching algorithm selected according to an embodiment. In step S131 , the server 300 obtains the first hash value (First Hash Value) HV1 (shown in FIG. 3 ) of the first privacy data PD51 and the second privacy data PD52 from the first processing device 100 and the second processing device 200 The second hash value (Second Hash Value) HV2 (shown in Figure 3). The first hash value HV1 is obtained by encoding the content of the direct identification column or the representative non-direct identification column of the first privacy data PD51. The second hash value HV2 is obtained by encoding the content of the direct identification column or the representative non-direct identification column of the second privacy data PD52.

In step S132, the server 300 compares the first hash value HV1 with the second hash value HV2 to determine whether the overlap ratio of the first generated data GD51 and the second generated data GD52 is higher than a predetermined value. The overlap ratio of the first generated data GD51 and the second generated data GD52 is the ratio of the overlapping content. If the overlap ratio is not higher than the predetermined value, proceed to step S136; if the overlap ratio is higher than the predetermined value, proceed to step S133.

In step S133, the server 300 determines whether the first generated data GD51 and the second generated data GD52 have at least one joint key. If the first generated data GD51 and the second generated data GD52 have a connection key item, go to step S134; if the first generated data GD51 and the second generated data do not have a connection key item, go to step S135.

In step S134, the server 300 integrates the first generated data GD51 and the second generated data GD52 using a database linking algorithm (the method described in FIG. 2A).

In step S135, the server 300 integrates the first generated data GD51 and the second generated data GD52 using a record linking algorithm (the method described in FIG. 2B). In this step, the record linking algorithm is used to integrate the first generated data GD51 and the second generated data GD52 without using the link key.

In step S136, the server 300 integrates the first generated data GD51 and the second generated data GD52 using a statistical matching algorithm (the method described in FIG. 2C).

In step S140 of FIG. 4 , the server 300 obtains a joint probability distribution (JointProbability Distribution) JPD53 of the composite data SD53. In step S140, a noise data ND is added to the joint probability distribution JPD53. Please refer to FIG. 6, which illustrates the process of obtaining the joint probability distribution JPD53 according to an embodiment. First, the composite data SD53 is converted into a contingency table (Contingency Table) CT53. The number of times of various combinations of fields EC, IC, NR is filled in the contingency table CT53. Next, the noise data ND is added to the contingency table CT53 to obtain a noise contingency table (Noisy Contingency Table) NCT53. Next, the times of the noise contingency table NCT53 are converted into probability values to obtain a joint probability distribution JPD53.

Furthermore, in another embodiment, the dimensionality of the joint probability distribution JPD53 may be reduced. Please refer to FIG. 7, which is a joint probability distribution JPD53' according to another embodiment. The joint probability distribution JPD53 of dimension 3 is transformed into a joint probability distribution JPD53' of dimension 2. In this way, the complexity can be reduced from 5∧3 to 5∧2+5∧2, so that the operation load and operation time can be effectively reduced.

Next, in step S150 of FIG. 4, the server 300 samples the paired data SD53 according to the joint probability distribution JPD53 (or the joint probability distribution JPD53') to obtain a sampled data SP53. The content of the sample data SP53 is similar to the content of the first privacy data PD51 and the second privacy data PD52.

According to the above privacy data integration method, the sampling data SP53 is obtained by integrating the first privacy data PD51 and the second privacy data PD52. The sample data SP53 can represent the approximate contents of the first privacy data PD51 and the second privacy data PD52, but will not leak any customer's privacy data. This is quite useful for big data technology. In addition, the amount of private data is not intended to limit this disclosure. For example, three or more private data may also be performed by the above-mentioned private data integration method.

To sum up, although the present disclosure has been disclosed above with embodiments, it is not intended to limit the present disclosure. Those skilled in the art to which the present disclosure pertains can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure should be determined by the claims.

Claims (22)

1. a privacy data integration method, is characterized in that, this privacy data integration method comprises: The first processing device and the second processing device respectively obtain the first generation model and the second generation model according to the first privacy data and the second privacy data; The server generates first generated data and second generated data through the first generation model and the second generation model, respectively; and The server integrates the first generated data and the second generated data to generate synthetic data. 2. The privacy data integration method according to claim 1, wherein in the step of obtaining the first generative model and the second generative model, the first generative model or the second generative model is passed through a variational autoencoder ( Variational Auto-Encoder, VAE) algorithm, Generative Adversarial Network (GAN) algorithm, Information Generative Adversarial Network (Info-GAN), AAE algorithm or ALI algorithm. 3. The privacy data integration method according to claim 1, wherein in the step of obtaining the first generation model and the second generation model, the category content of the first privacy data or the second privacy data is converted into numerical values content, the synthetic data contains that category of content. 4. The privacy data integration method according to claim 1, wherein the step of integrating the first generated data and the second generated data comprises: If the first generated data and the second generated data have at least one joint key, a database join algorithm (Database Join Algorithm) is used to integrate the first generated data and the second generated data. 5. The privacy data integration method according to claim 4, wherein the step of integrating the first generated data and the second generated data comprises: If the overlap ratio of the first generated data and the second generated data is not higher than a predetermined value, a statistical matching algorithm (Statistical Match Algorithm) is used to integrate the first generated data and the second generated data. 6. The privacy data integration method according to claim 5, wherein the step of integrating the first generated data and the second generated data comprises: If the overlap ratio of the first generated data and the second generated data is higher than a predetermined value, and the first generated data and the second generated data do not have the connection key item, a Record Linkage Algorithm is used to integrate The first generated data and the second generated data. 7. The privacy data integration method according to claim 1, further comprising: A Joint Probability Distribution is obtained for the synthetic data. 8. The privacy data integration method of claim 7, wherein the joint probability distribution is transformed to reduce dimensionality. 9. The privacy data integration method according to claim 7, wherein in the step of obtaining the joint probability distribution, noise data is added to the joint probability distribution. 10. The privacy data integration method according to claim 7, further comprising: The synthetic data is sampled to obtain a sampled data. 11. The privacy data integration method according to claim 10, wherein the content of the sample data is similar to the content of the first privacy data and the content of the second privacy data. 12. A server for executing a privacy data integration method, wherein the privacy data integration method comprises: The first generation data and the second generation data are respectively generated by the first generation model and the second generation model, and the first generation model and the second generation model are obtained according to the first privacy data and the second privacy data respectively; and The first generated data and the second generated data are integrated to generate composite data. 13. The server according to claim 12, wherein the first generative model or the second generative model is implemented by a variational auto-encoder (Variational Auto-Encoder, VAE) algorithm, a generative adversarial network (Generative Adversarial Network, GAN) algorithm, Information Generative Adversarial Network (Info-GAN), AAE algorithm or ALI algorithm is obtained. 14. The server of claim 12, wherein the category content of the first privacy data or the second privacy data is converted into numerical content, and the synthetic data includes the category content. 15. The server of claim 12, wherein the step of integrating the first generated data and the second generated data comprises: If the first generated data and the second generated data have at least one joint key, a database join algorithm is used to integrate the first generated data and the second generated data. 16. The server of claim 15, wherein the step of integrating the first generated data and the second generated data comprises: If the overlap ratio of the first generated data and the second generated data is not higher than a predetermined value, a statistical matching algorithm (Statistical Match Algorithm) is used to integrate the first generated data and the second generated data. 17. The server of claim 16, wherein the step of integrating the first generated data and the second generated data comprises: If the overlap ratio of the first generated data and the second generated data is higher than a predetermined value, and the first generated data and the second generated data do not have the connection key item, a Record Linkage Algorithm is used to integrate The first generated data and the second generated data. 18. The server according to claim 12, wherein the privacy data integration method further comprises: A Joint Probability Distribution is obtained for the synthetic data. 19. The server of claim 18, wherein the joint probability distribution is transformed to reduce dimensionality. 20. The server of claim 18, wherein in the step of obtaining the joint probability distribution, noise data is added to the joint probability distribution. 21. The server according to claim 18, wherein the privacy data integration method further comprises: The synthetic data is sampled to obtain sampled data. 22. The server of claim 18, wherein the content of the sample data is similar to the content of the first privacy data and the content of the second privacy data.

Images