KR20160066454A - Data Processing Method for Distributable and Unidentifiable Big Data - Google Patents

Abstract

The present invention relates to a method for producing and managing unidentifiable big data for distribution capable of being used in a variety of ways by linking produced data in a separate environment and of being freely distributed to an external system, preventing the leakages of private information. According to the present invention, information can be exclusively selected and processed for each group by grouping data on private information in private information representative records; extracting private information required by individuals included in each group as a set of individual log data by personal acting log data; and matching or integrating the same into the private information representative record after integrating the personal acting log data with a certain standard and making single type of representative log data. Also, by separately grouping personal information and personal acting information and integrating the same, the probability of leakages of personal information caused by back tracking or combination can be significantly reduced compared to the existing method of grouping the whole information under a certain standard or local or spatial features.

Description

Technical Field [0001] The present invention relates to a data processing method for distributing non-

The present invention relates to a method of processing non-identifying big data for distribution, and more particularly, to a method for processing non-identification big data for distribution, which can freely circulate the data to an external system without fear of leakage of personal information, RTI ID = 0.0 > non-identified < / RTI > big data for distribution.

Big data refers to electronic data such as electronic commerce data, metadata, web logs, radio frequency identification (RFID) data, sensor network data, social network data, social data, and Internet text Data that includes both informal and semi-structured data that have not been used before, such as documents, Internet search indexes, and the like. Such data generally has a level of data that can not be handled by ordinary software tools and computer systems (Big Data).

As Big Data began to create various values, people began to compare Big Data to crude oil. It means that you can not spend the information age without big data just like the equipment does not run without oil. Gartner, a US market researcher, said the data is the 21st century crude oil that will influence future competitiveness, and companies must understand and prepare for the upcoming data economic era. The most important asset for a 21st century company is data, which means that you can not survive if you do not manage and value it.

In recent years, not only companies but also governments have been actively trying to utilize various collected big data information and statistical analysis data utilizing them in decision making and policy making, A large data processing technique is being actively studied.

Big data is analyzed and utilized within the organization in which the data are collected. However, there is a difference in the attributes of the data collected according to the organization that collects the data, so it is necessary to utilize the data of another organization. In the case of organizations that lack the ability or system, there is a need to analyze the information unique to the organization from the big data of other organizations or their combination and use it for decision making.

However, because of the nature of Big Data, not only is the amount of data enormous, but most Big Data contains inevitably information about personal information, and there is a possibility that a legal dispute will arise due to the leakage of personally identifiable information Therefore, there is a limit to the exchange and distribution of Big Data between organizations. Accordingly, in order to avoid the occurrence of legal disputes due to the leakage of personal information, an organization capable of collecting Big Data is required to process Big Data for business purposes In addition, it is difficult to acquire analytical data, which is essential for the unique business environment of the organization, from the viewpoint of the organization that needs the utilization of the big data.

In order to solve the above problems, some big data processing systems and methods for grouping data, or non-identifying personal attributes through masking, substitution, semi-discrimination, and typing have been applied in some fields.

The masking is to mask or delete the object information (e.g., 670101-10491910 **************), and the replacement is to replace the information generated corresponding to the object information (e.g. 670101-10491910 ID2311331), the semi-discrimination is semi-discriminated to represent only a part of the object information (for example, 670101-10491910 67-1), and the type is a method of classifying and distinguishing object information (for example, 670101-10491910 man) .

However, according to the conventional big data processing method, in the grouping of data, attribute values of similar records are aggregated and converted into representative records (total processing), and grouping is performed simply by grouping with attribute range values (categorization) , It is difficult to appropriately select and process only the information that is uniquely required in the respective destinations. Even if the personal information is unidentified by masking, substitution, semi-identification, or type, There is a risk that there is a risk of leakage of personal information through backtracking, and there is a problem that it is difficult to perform linkage analysis of big data independently generated in separate environments.

An object of the present invention is to provide a non-discriminating big data processing method for distribution which can selectively process only necessary information uniquely required by individual destinations .

It is a further object of the present invention to provide a method and system for providing personal information to a user by making it possible to completely re-identify a particular individual at the time of distribution of big data, And to provide a big data processing method for distribution that can be safely used for distribution without having to obtain permission.

It is still another object of the present invention to provide a method for processing non-identifying big data for distribution, which enables two or more big data independently generated in separate environments to be used in various analysis as needed.

A method for processing large data for distribution according to the present invention is a method for processing large data in a data server having a communication unit, a processing unit and a storage unit, And storing the original data collected in the storage unit of the data server; Grouping a plurality of individual individual record into one personal representative record by using at least one attribute of the personal data contained in original data stored in the storage unit as a grouping key; Generating a personal identification list by associating, for each grouped personal representative record, an attribute that can specify each individual included in the personal representative record; Wherein the processing unit extracts individual log data of individual actions of all individuals belonging to the respective individual identification lists from the individual activity log data included in the stored original data, Generating a log data set; The processing unit integrating the individual behavior individual log data sets based on at least one or more attributes to generate one personal behavior representative log data; And storing the personal representative representative record and the personal behavior representative log data in the storage unit.

According to the present invention, the individual personality data is grouped into individual personal representative records, and the personal behavior log data is obtained by extracting only the personal behavior information required for each person included in each group, And integrates them based on specific attributes, converts them into one personal behavior representative log data, and then matches / integrates them with the personal representative record, so that only the information that is uniquely required by each individual destination can be appropriately screened and provided .

In addition, by grouping and merging personal information and personal behavior information separately, it is possible to significantly reduce the possibility of leakage of personal information by combination or backtracking compared to the conventional method of grouping the whole information on the basis of time, space, .

In another aspect of the present invention, the step of grouping into the individual personal representative record includes a step of non-identifying the personal information, wherein the non-identifying step includes the step of identifying an individual from the individual individual record included in the original data An uniqueness removal step of removing attribute values that can be specified; If the attribute value of the grouped individual personal representative record is the same as the corresponding attribute value of the individual personal individual record included in the original data, the corresponding attribute value of the individual personal representative record is corrected, The attribute value of the record is different from the attribute value of the personal information included in the original data; If a similarity value between the personal representative record and one or more individual personal records applied to generation thereof is greater than a predetermined maximum similarity threshold value based on a similarity calculation function predefined by the user, Removing similarity; A continuity removal step of preventing individual individual record included in the individual personal representative record from being regrouped into the same group for the same person after a predetermined period of time has elapsed; And at least one of the above-mentioned steps.

According to this aspect of the present invention, in addition to the grouping of personal information, further performing the de-personalization, i.e., non-discrimination, in the grouping process makes it impossible for the individual to be re-identified at the time of distribution of the big data , It is possible to safely use the sensitive personal information for distribution without having to obtain permission from the individual.

Meanwhile, in another aspect of the present invention, the grouping step into the individual personal representative record is performed by dividing the attribute value distribution information of the group personal information of the individual personal individual records grouped in the same personal representative record into the group Adding to the personal representative record; Adding at least one attribute value distribution information among attributes of the individual person individual records grouped in the same individual person representative record to the personal representative record as a group distribution attribute; And at least one of the steps of:

According to this aspect of the present invention, two or more big data independently generated in a separate environment via the grouping of the pinky distribution attribute and / or the group distribution attribute can be used for analysis in various ways as needed.

According to the present invention, individual personality information is grouped into individual personal representative records, and individual behavior log data is obtained by extracting only individual personal behavior information required for each person included in each group, , And then integrates them based on specific attributes to match and merge them to individual personal representative records as one personal behavior representative log data so that only the information that is uniquely required by each individual destination can be selectively processed and provided .

In addition, by grouping and merging personal information and personal behavior information separately, it is possible to significantly reduce the possibility of leakage of personal information by combination or backtracking compared to the conventional method of grouping the whole information on the basis of time, space, .

In addition, in addition to the grouping of personal information, it is possible to carry out de-personalization or non-discrimination in the grouping process, thereby making it impossible to re-identify a specific individual at the time of distribution of big data, It can be safely used for distribution without having to obtain permission from individual persons.

In addition, two or more big data independently generated in a separate environment can be variously linked and used for analysis as needed via the grouping of the pinky key distribution attribute and / or the group distribution attribute.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is an exemplary diagram illustrating a data-centric computing environment that forms the Big Data Processing System of the present invention.
2 is a block diagram showing a main configuration of the data server shown in FIG.
FIG. 3 is a block diagram schematically showing steps of generating non-distribution large data for distribution performed in a processing unit of a data server; FIG.
FIG. 4 shows an example of a representative record generated through the steps of FIG. 3. FIG. 4 illustrates an example in which representative records are generated by grouping five persons by using age groups and genders as grouping keys.

Referring to FIG. 1, a data-centric computing environment for forming a big data processing system of the present invention may be constructed by a plurality of user terminals 120 connected to a data server 110 and a data server via a wired or wireless network .

The data-centric computing environment utilizes data generated in real time from a plurality of user terminals 120 to provide a variety of functions such as a social network service (SNS), a smart grid, an intelligent home appliance, Means a technology based on big data processing that can provide various applications such as real-time streaming or real-time decision making.

The big data processing system and method according to the present invention are implemented by a data server 110 connected to a plurality of user terminals 120 and collect data generated by a plurality of user terminals 120 and process the data And provides the stored data to the user terminal 120 that takes up the data, thereby establishing an environment in which data-centric computing applications can be performed.

The user terminal 120 may be a computer having a communication device connected to the data server 110 and having an information processing function for generating data according to the operation of the user terminal 120, , Mobile communication terminals such as smart phones, tablet PCs, and PDAs (personal digital assistants), smart home appliances, radio frequency identification (RFID) data, vehicles such as black boxes or navigation systems, trains, airplanes But is not limited thereto.

2, the data server 110 is connected to a plurality of user terminals 120 through a communication unit 113, such as a wired / wireless network such as short-range wireless communication, Wi-Fi, 3G (3Generation) A cloud server or a web server that collects data generated in the user terminal 120 and stores the data in the storage unit 112 and processes and stores the data collected by the processing unit 111, But is not limited thereto.

Original data collected through the communication unit 113 from the terminals 120 connected via the wire / wireless network is stored in the storage unit 112 of the data server 110.

In recent years, due to the development of network technology and the proliferation of smart phones, the amount of logs produced per day in a big data processing system operated by enterprises and institutions has been increased to tens to hundreds of gigabytes, It is also possible to have a range of exabytes or zeta bytes exceeding a byte.

The processing unit 111 according to the present invention appropriately processes the large data stored in the storage unit 112 to select and group the data necessary for analysis so that the capacity is reduced and the data is unidentified, In order to more reliably prevent the outflow possibility, a separate non-identifying process is performed at the time of grouping so as to be processed into non-identifying big data for distribution and stored in the storing unit 112, and stored in a relatively small capacity in the storing unit 112 The non-distribution big data for distribution is transmitted to the requester requiring analysis and utilization thereof through the communication unit 113 of the server and the communication network.

FIG. 3 is a block diagram schematically showing a generation step of distribution non-identifying big data performed in the processing unit of the data server. For convenience of description, generation of one representative record for distribution is used as a reference. Hereinafter, a method for generating non-identifying big data for distribution of the present invention will be described in detail with reference to FIG.

First, the processing unit 111 groups at least one attribute of the personal image data included in the original data stored in the storage unit into a single personal representative record by using a grouping key as a grouping key do.

Individual personal representative records grouped in this way are stored in a storage unit 112 in a form of a set of records in the form of a table.

The original data is composed of personal data of individual individuals and activity log data collected during the service of the operating system. The personal information is data including the data uniquely classified by the individual or corresponding personal information, (Wage, age, height, illness, etc.), and personal activity log data is composed of personal information (social security number, passport number, medical insurance number, vehicle registration number, name, address, Service request / provision / use history generated by utilizing the service.

The grouping is a representation of a plurality of (N) records as one representative record in order to secure the integrity of the individual non-discrimination. The grouping is a minimum scale of grouping similar records to maximize the validity and usability of data ) Is preferably selected.

The grouping key refers to an attribute that serves as a reference for performing grouping among the attributes (fields) of the original individual individual record. When a continuous numeric data type attribute is used as a grouping key, a histogram, a binning, It is preferable to convert the attribute into discrete card type data and then group them.

 In case of performing N: 1 grouping, when the number of original individual individual records having the same group pinky value is M, approximately M / N representative records are generated for the grouping attribute value.

If the attributes included in the representative record are numeric as attributes other than the group pinky, apply general aggregate functions (sum, mean, median, maximum value, sampling, etc.) to the representative attribute values representing the grouped N attribute values After conversion, it is assigned to the attribute value of the representative record.

A process of grouping the original data into the personal representative record at the processing unit of the data server will now be described in detail.

Table 1 is an example of personal information extracted from original data, and is set as an attribute included in the representative record, such as resident registration number, age name, address, and income.

Table 1

Table 2 shows an example of conversion to un-identify individual attributes.

 Table 2

Table 3 shows an example in which three records (N = 3) are grouped based on the age, sex, and city, and one ID is generated for each group, and IDs are assigned to the respective groups. An identifier (ID) value is generated to uniquely distinguish each representative record.

Table 3

Table 4 shows an example of grouping two personalities (N = 2) of personal information in Table 2, that is, groups of age, sex, and city as groups, differently from Table 3, 40s South Seoul> group and 2 <40s Seoul> groups are created.

In addition, by applying the average function to the income attribute and assigning the income attribute value of the representative record, individual personal representative records are generated as follows, and for each grouped personal representative record, And generates an individual identification list by associating an attribute (e.g., a resident registration number) that can specify an individual.

Table 4

By the grouping process as described above, it is possible to reduce the capacity of the data and discriminate the individual. However, in order to more reliably prevent the leakage of the personal information, the grouping process, as well as the uniqueness removal, It is preferable that the personal information non-discrimination process be performed.

Uniqueness removal refers to the removal of identification information (e.g., a resident registration number) that can identify an individual in a source individual identification individual record, as illustrated in Table 5.

Table 5

In the case where the attribute value of the personal representative record generated in the N: 1 grouped group is the same as the corresponding attribute value of the individual record, as illustrated in Table 6, the attribute value of the representative record is corrected, The attribute value of the individual record and the attribute value of the representative record are different from each other.

If the attribute value is a continuous numeric data type, it can be converted into a discontinuous Cartesian type data type by preprocessing with histogram, binning, clustering, etc. to remove the identity.

Table 6

Similarity removal is based on a predefined similarity function and if the similarity value between the personal representative record and one or more original individual personal records is greater than the maximum similarity threshold value ε, will be..

If the original individual record is x and the processed individual record is y, the similarity between x and y is defined as follows.

Struc_sim (x, y) = the similarity calculation function between the original individual individual record x and the grouped individual personal representative record y

(0 < = Struc_sim (x, y) < = 1,

        Struc_sim (x, y) == 1 => x and y are the same record)

Continuity removal is performed by changing the record identifier mapping scheme used in the generated distribution big data to a different method after a predetermined period of time elapses so that the N: 1 representative record identifier value is continuously used for the same value .

In other words, the re-grouping is performed by changing the mapping method after a certain period of time so that the individual individual records used in generating the personal representative record are prevented from being grouped into the same group for the same personal personal data in the future.

The above describes the method of generating the personal representative record by the processing unit 111. Next, the individual log data of the individuals belonging to each individual personal representative record is extracted from the individual activity log data on the original data , The process of generating personal behavior representative log data will be described in detail with reference to an example.

Table 7 shows an example of the personal behavior log data. As mentioned above, the personal behavior log data is composed of personal service request / provision / use history generated while utilizing the service, and generally, The action log data basically has the person identification attribute, the time attribute, and the space attribute because it indicates the change of the individual's spatio-temporal action state. The action log items have the semi-regular form as the log attribute value have.

Table 7

First, individual log data of individual actions is extracted for all individuals in the individual identification list obtained for each individual personal representative record, and is generated as a set of personal behavior individual log data of the corresponding individual personal representative record.

As described in conjunction with Table 4 above, the personal identification list is generated by associating, with each grouped personal representative record, an attribute (e.g., a resident registration number) that can identify each individual included in the corresponding personal representative record do.

For example, in Table 4, the personal identification list of the personal representative record id321 is as shown in Table 8, and it can be generated as shown in Table 9 for the personal behavior individual log data set of the individual personal representative record id321.

Table 8

Table 9

In this way, individual log data of individual actions extracted for each individual personal representative record are integrated into one personal behavior representative log data, that is, grouped.

When integrating individual log data of individual behavior extracted for each individual personal representative record into one personal behavior representative log data, the integration target can be selected by the space or the time close by giving time or space condition as a constraint.

The integrated log data of the selected individual activities are integrated into one personal behavior representative log data by applying various integration functions.

Integration functions include union, intersection, sampling, frequent elements, clustering, and histogram.

For example, the personal behavior representative log data generated when various integration functions are applied to the individual behavior individual log data set (Table 9) of the individual personal representative record id321 in Table 4 is as follows.

In the case of merging into a union as a whole,

In the case of integration into an intersection as a whole,

An example of selectively integrating time or space constraints by constraints is the union of individual behaviors within 7 minutes,

The union of individual acts by the same administrative division

As described above, the generated individual personal representative record and the personal behavior representative log data are sequentially stored in the form of a table in the storage unit 112 to form big data for distribution, And is formed by individual representative records by matching and integrating personal behavior representative log data.

The matching / integration to the representative record may be performed in a server providing big data for distribution or in a server where big data is used.

FIG. 4 shows an example of a representative record generated by the above method, and shows an example in which 5 persons are grouped by age group and gender as a grouping key to generate a representative record.

On the other hand, according to the distribution big data formed by the present invention, the group fingerprint of the non-identified individual personal representative record can be used for linkage analysis by combining with other personal identification data for distribution which is not identified by the same grouping key have.

That is, by using the method of the present invention, it is possible to use two or more distribution big data independently generated in separate environments.

 According to another aspect of the present invention, in order to improve the linking accuracy by linking representative records of similar characteristics in linkage analysis of data, a distribution attribute of group pinky and a distribution attribute of group are used for linkage analysis.

The distribution information of the grouping of the individual individual individual records grouped into one group with the same representative record can be added to the representative record as a new attribute. This added attribute is defined as a group pinky distribution property.

Similarly, a new attribute may be added to the representative record that represents the distribution of the original individual individual record attribute values grouped into a group for attributes other than the grouping key. These attributes are defined as grouping distribution properties.

These two types of distribution property values can be expressed as a function representing a general distribution. Typical types are standard deviation, median, quartile-quartile distance (Q3-Q1), maximum value, | maximum-minimum value | Or the number of different attribute values.

Hereinafter, a method of connecting data sets that are processed independently of each other using a grouping policy distribution attribute and a grouping distribution attribute will be described in detail.

First, the distribution type personal image data A containing the average income information and the distribution personal image data B containing the average flow property information are all converted by the grouping method of the present invention separately as the age attribute and the sex attribute of the same grouping I suppose.

It is also assumed that the maximum income of each group is added to the grouping distribution attribute in A, and the maximum value of the liquid asset is added to the grouping distribution attribute in B.

If the age distribution attribute, which is a group Pinky distribution property, is equally defined as a middle age median value in the group, distribution attributes are additionally generated for each representative record of A and B, as illustrated in Table 10. [

Table 10

If only the representative records satisfying the condition that the difference of the age distribution attribute value of the group of the two sets of data set is within 2 are inputted while joining the converted data sets A and B with the grouping key of the age and sex, Likewise, two personal-linked representative records are linked to income and liquid assets representative records.

Table 11

As described above, the distribution big data formed by the present invention can be used in conjunction with two or more distribution big data independently generated in a separate environment, and the process is as follows.

Two different distribution big data A (personal representative representative record set A _S , personal behavior representative log data set A _L ) and B (personal representative record set B _S , personal behavior representative log data set B _L ) when, in the personal a representative set of records a and _S B are representative personal record set _S B are connected as described above as an example in Table 10, 11 of the above.

Based on this result, the representative log data A and B are newly created by combining the individual personal representative representative log records A _L and B's personal behavior representative log data record B _L , And performs a big data linkage analysis on the data.

Representative personal record x∈A _S and personal records representing y∈B _S is coupled as previously illustrated if x represents individual behavior log data v∈A _L and y represent individual actions of the log data of the same _L w∈B considered as a personal act of history, and when the linkage analysis of two big data a and B with two actions represent the log data <v, w> is behavior analysis to assume that means linked to and integrated Big data (a _L B _L) .

On the other hand, in the process of generating personal behavior representative log data from individual log data of the individual behavior, non-identification processes such as uniqueness removal, identity removal, similarity removal, and continuity removal are performed similarly to the grouping generation process of the personal representative record .

The elimination of uniqueness in the generation of personal behavior representative log data is to remove identification information (eg, resident registration number, etc.) that can identify an individual from the individual log data of the semi-structured personal conduct extracted for each individual personal representative record, , And deletes all representative log records identical to at least one personal behavior individual log data for the representative log data integrated for each personal representative record.

Similarity removal is performed based on a predefined similarity function, if the similarity value between the personal representative record and one or more original personal personal record is greater than the maximum similarity threshold value (?), The corresponding personal representative record is deleted and not created I will not ..

When the individual log data of the original personal behavior is T and the representative personal behavior representative data is S, the similarity between T and S is defined as follows.

          log_sim (S, T) = | TS | / | S | (: Intersection, | |: number of elements)

Continuity removal is defined by associating the log identifier of the personal behavior representative log data set with the identifier number of the corresponding personal representative record to prevent the representative log data identifier value from being continuously used to the same value for the same individual .

100: data server 120: user terminal
111: Processor 112:
113:

Claims (3)

A big data processing method performed in a data server having a communication unit, a processing unit, and a storage unit,
Storing original data collected through a communication unit from a terminal connected through a wire / wireless network in a storage unit of the data server;
Grouping the plurality of individual person individual records into a single personal representative record by grouping at least one or more attributes of the personal information data included in the original data stored in the storage unit as the grouping key;
The processing unit generates an individual identification list by associating, with each grouped individual personal representative record, an attribute capable of specifying each individual included in the individual personal representative record
Wherein the processing unit extracts individual log data of individual actions of all individuals belonging to the respective individual identification lists from the individual activity log data included in the stored original data, Generating a log data set;
The processing unit integrating the individual behavior individual log data sets based on at least one or more attributes to generate one personal behavior representative log data;
Storing the personal representative representative record and the personal behavior representative log data in the storage unit;
Identifying a big non-identifying data for distribution.
The method according to claim 1,
The step of grouping into the personal representative record includes a step of non-identifying the personal information,
Wherein the non-
An uniqueness removal step of removing an attribute value that can specify an individual from the individual individual record included in the original data;
If the attribute value of the grouped individual personal representative record is the same as the corresponding attribute value of the individual personal individual record included in the original data, the corresponding attribute value of the individual personal representative record is corrected, The attribute value of the record is different from the attribute value of the personal information included in the original data;
If the similarity value between the individual personal representative record and one or more individual personal record applied to the generation of the personal representative representative record is greater than a predetermined maximum similarity threshold value based on a similarity calculation function predefined by the user, Removing similarity;
A continuity removal step of preventing individual individual record included in the individual personal representative record from being regrouped into the same group for the same person after a predetermined period of time has elapsed; The method comprising the steps of; 3. The method according to claim 1 or 2,
The grouping into the personal representative record may include:
Adding the attribute value distribution information of the group fingerprint of the individual person individual records grouped in the same individual person representative record to the personal name representative record as a group fingerprint distribution attribute;
Adding at least one attribute value distribution information among attributes of the individual person individual records grouped in the same individual person representative record to the personal representative record as a group distribution attribute; The method comprising the steps of;

Images