KR20190053616A - Data merging device and method for bia datda analysis - Google Patents

Abstract

Provided is a data merging apparatus for big data analysis, which comprises: a database including a plurality of data sets; a data matching unit determining a first data set and a second data set to perform data matching among a plurality of data sets included in the database, selecting a plurality of column items among column items of the first data set and the second data set to calculate a degree of similarity between the selected column items, and matching data rows of the first data set and the second data set based on the degree of similarity; and a data merging unit merging the first data set and the second data set based on the matching result of the data rows.

Description

TECHNICAL FIELD [0001] The present invention relates to a data merging apparatus and method for analyzing large data,

The present invention relates to a data merging apparatus and method for big data analysis.

For big data analysis, 70% to 80% of the total effort is used for data preprocessing. Big data analysis is explosively growing, but the development of data preprocessing technology is slow as much as the development of big data analysis technology. Therefore, there is a need to develop automated data preprocessing technology.

It is emphasized that there is a necessity of re-introducing information value through mash-up with external data in conjunction with a public information open environment outside the relational database (RDBMS) based and internal data analysis environment.

However, the merge of data is impossible if the viewpoint and purpose of internal / external data generation are different, or the data is semantically different from the same data or expression due to the lack of data standards. In this case, it is necessary to input excessive human resources to match the data, and it is impossible to normalize the data.

Therefore, it is necessary to study data merging technique using data matching algorithm to activate big data analysis. However, current data matching algorithm only matches one data item, I can not. Especially, the accuracy of Korean data is low. In addition, there is a problem that the data is matched even though the data is required to be corrected by the user.

In addition, the data matching algorithm, which is one of the data preprocessing processes, is the fuzzy data matching algorithm. This algorithm is an algorithm that matches the data using the calculated values based on the edit distance (Levenshtein Distance). However, since existing algorithms match through comparison between single keys, it is difficult to match structurally in a table using multiple keys. In addition, since it is an algorithm developed based on English, there is a limit to accuracy by using existing algorithms for Korean data.

The background technology of the present application is disclosed in Korean Patent Laid-Open Publication No. 2011-0099783 (Publication Date: 2001.11.09).

The present invention has been made to solve the above problems occurring in the prior art, and it is an object of the present invention to provide a data merge device for merging a plurality of data by performing a similarity calculation by selecting a plurality of column items from a plurality of data sets, And a method is provided.

It is another object of the present invention to provide a data merging apparatus and method for merging different kinds of data more accurately by providing matching information to a user for items that can not be matched when a plurality of data sets are matched The purpose is to provide.

In addition, the present invention has been made to solve the above-mentioned problems of the prior art, and it is an object of the present invention to provide a method and apparatus for performing a first matching similarity calculation and a second matching similarity calculation, And more particularly, to a data merging apparatus and method for merging a plurality of data more accurately.

It should be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided a data merge apparatus for analyzing big data, comprising: a data base including a plurality of data sets; a first data set for performing data matching among a plurality of data sets included in the data base; Determining a data set and a second data set, selecting a plurality of column items of the column items of the first data set and the second data set to calculate a degree of similarity between the selected column items, And a data merge unit for merging the first data set and the second data set based on the matching result of the data row.

According to an embodiment of the present invention, the data matching unit derives a first similarity rank data set by performing a first matching similarity calculation based on the first column of the first data set and the second column of the second data set, 1 similarity rank order data set to derive a second similarity rank order data set by performing a second matching similarity degree calculation based on the first similarity degree ranking data set and comparing the first similarity degree ranking data set and the second similarity degree ranking data set, The data row of the first data set and the data row of the second data set may be matched.

According to an embodiment of the present invention, the first similarity degree ranking data set and the second degree of similarity ranking data set calculated through the data matching unit may include ranking information of each data row of the second data set.

According to an embodiment of the present invention, the data matching unit is configured to combine the first row data of the first data set and the second row data of the second data set located in the first rank of the first similarity rank order data set, Ranking in the similarity ranking data set and ranking in the first similarity ranking data set of the row data of the second data set located in the first ranking of the second similarity ranking data set are both 1, When the row data of the second data set located at the first rank of the first similarity rank order data set matches the row data of the second data set located at the first rank of the second similarity rank order data set, It is possible to determine that the row data of the second data set located in the first data set matches the first row data.

According to an embodiment of the present invention, the data matching unit is configured to combine the first row data of the first data set and the second row data of the second data set located in the first rank of the first similarity rank order data set, The ranking in the similarity ranking data set and the row data of the second data set in which the ranking in the first similarity ranking data set of the row data of the second data set located at the first rank of the second similarity ranking data set is all 1 When there is a plurality of rows, it is possible to determine that the row data of the second data set matching the first row data can not be matched.

According to one embodiment of the present invention, the data matching unit is configured to combine the first row data of the first data set and the first row data of the second data set, Ranking of the second similarity ranking data set of the first row data of the second data set is 1 and the ranking of the second similarity ranking data set of the second row data of the second data set is 1 , It can be determined that the first row data of the second data set is matched with the first row data.

According to an embodiment of the present invention, a data merging apparatus for analyzing big data includes a profiling unit for analyzing data types and character string patterns of a plurality of column items of the first data set and the second data set through data analysis, And a similar item analyzer for analyzing similar items between the first data set and the second data set based on the analysis result of the profiling unit.

According to an embodiment of the present invention, the data merge unit may exclude column items that are duplicated in the merging of the plurality of column items of the first data set and the second data set, based on the analysis result of the similar- And perform the merging of the first data set and the second data set.

According to an embodiment of the present invention, there is provided a data merge method for analyzing big data, comprising: determining a first data set and a second data set to perform data matching among a plurality of data sets included in a database; Deriving a first similarity rank order data set by performing a first matching similarity degree computation based on a first column of the first data set and the first column of the second data set, performing a second matching similarity calculation based on the first similarity degree rank data set Deriving a second similarity ranking data set, comparing the first and second similarity ranking data sets, and matching the data rows of a second data set matching each data row of the first data set And merging the first data set and the second data set based on the result of the step and the matching.

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-mentioned problem solving means of the present invention, a plurality of data items can be merged by performing a similarity calculation by selecting a plurality of column items from a plurality of data sets.

Further, according to the above-mentioned problem solving means of the present invention, it is possible to merge a plurality of data more accurately by providing matching information to a user for items that can not be matched in a plurality of data set matching.

According to the present invention, after the first matching similarity calculation and the second matching similarity calculation are performed, the first matching similarity calculation result is compared with the second matching similarity calculation to perform final data matching, A plurality of data can be merged more accurately.

1 is a block diagram schematically illustrating a configuration of a data merge apparatus according to an embodiment of the present invention.
2A to 2D are schematic diagrams for explaining a first matching similarity calculation according to an embodiment of the present invention.
FIG. 3 is a schematic diagram for explaining a second matching similarity calculation according to an embodiment of the present invention; FIG.
4 is a diagram illustrating exemplary first matching similarity calculation and second similarity calculation according to an embodiment of the present invention.
FIGS. 5A and 5B are schematic diagrams for illustrating a method of comparing a first similarity degree ranking data set and a second similarity ranking ranking data set according to an exemplary embodiment of the present invention to match data rows of a first data set and a second data set.
6A and 6B are schematic diagrams for explaining an embodiment of data analysis and similar item analysis according to an embodiment of the present invention.
FIG. 7 is a schematic diagram for explaining an embodiment of data merging for big data analysis according to an embodiment of the present invention; FIG.
8 is a flowchart illustrating a data merging method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when a part is referred to as being "connected" to another part, it is not limited to a case where it is "directly connected" but also includes the case where it is "electrically connected" do.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as " including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

FIG. 1 is a block diagram schematically illustrating the configuration of a data merge apparatus according to an embodiment of the present invention. FIGS. 2A to 2D are schematic diagrams for explaining a first matching similarity calculation according to an embodiment of the present invention, FIG. 4 is a diagram illustrating a first matching similarity calculation result and a second matching similarity calculation result according to an embodiment of the present invention, and FIG. 4 illustrates a result of a second matching similarity calculation according to an embodiment of the present invention FIGS. 5A and 5B are schematic diagrams for explaining matching of data rows of a first data set and a second data set by comparing a first similarity degree ranking data set and a second similarity degree ranking data set according to an embodiment of the present invention .

1, the data merge apparatus 100 includes a data base 110, a data matching unit 120, a profiling unit 130, a similar item analysis unit 140, and a data merge unit 150 . As another example, the data merge apparatus 100 may receive data necessary for data merge from an external server, but the present invention is not limited thereto.

The data merge apparatus 100 determines a first data set and a second data set among a plurality of data sets, selects a plurality of column items from the column items of the first data set and the second data set, . The data merge apparatus 100 may match the data rows of the first data set and the second data set based on the similarity. In addition, the data merge device 100 may provide the user with a matching determination whether or not the user is unacceptable based on whether or not to match the plurality of data, and may merge the plurality of data based on the matching determination information. The data merge apparatus 100 may merge the first data set and the second data set based on the matching result of the data row.

According to an embodiment of the present invention, the data merge apparatus 100 may merge data by matching a first data set and a second data set using a data matching algorithm based on artificial intelligence in a plurality of data sets.

In addition, the data merge apparatus 100 may be configured such that the first data and the second data of the second data set are overlapped and matched to the first data, which is a problem that arises in the result of data matching performed through the calculation of similarity between existing single column items The first row data of the first data set and the first row data of the second data set may be matched by performing data matching through a similarity calculation between a plurality of column items. ) Analyzes and matches the data similarity of a plurality of data sets, so data merging can be possible without intervention of a data engineer specialist and a business person. The data merge apparatus 100 can improve the accuracy of data matching by learning and analyzing the amount of data that grows under the big data environment and the data characteristics of the specific business domain.

In addition, the data merge apparatus 100 can perform more accurate data matching and merging by selecting matching candidates directly to the user of a data item that is determined to be impossible to match, and correcting the result.

The database 110 may include a plurality of data sets used for merging big data. The database 110 may include unstructured data.

The data matching unit 120 may determine a first data set 10 and a second data set 20 to perform data matching among a plurality of data sets included in the data base 110. According to one embodiment of the present invention, the first data set 10 and the second data set 20 are data that includes a plurality of column items of similarity degree information set in advance based on the similarity calculation result of the similarity analysis unit 140 Can be set. According to another example, the first data set 10 and the second data set 20 may be a data set determined by the user's selection. The data matching unit 120 may determine the first data set 10 and the second data set 20 that include items required to be merged among a plurality of data sets to perform the similarity calculation.

The data matching unit 120 may select a plurality of column items among the column items of the first data set and the second data set to calculate the similarity between the selected column items. The data matching unit 120 may perform an operation using an artificial intelligence based algorithm when performing a similarity operation. The data matching algorithm of the data matching unit 120 according to an embodiment of the present invention may perform similarity calculation using a fuzzy data matching algorithm, but the present invention is not limited thereto. The Fuzzy Data Matching algorithm is an algorithm that performs matching between data using the calculated values based on the edit distance (Levenshtein Distance).

2A and 2B, reference numeral (a) in FIGS. 2A and 2B is a first data set 10 and reference numeral (b) in FIGS. 2A and 2B is a second data set (10). The first data set may include four column entries 11-14. Also, the second data set may include four column entries 21 to 24. The column items included in the first data set and the second data set can be divided into representative keys. For example, the representative key of the first column item 11 of the first data set may be a 'parking lot name', and the first column item 11 may be a set of columns included in the representative key 'parking lot name' . That is, the representative key of the first column item 11 included in the first data set 10 is the 'parking lot name' and the representative key of the second column item 12 of the first data set 10 is' Operation start time ', the representative key of the third column item 13 of the first data set 10 is the' parking lot operation end time ', and the representative of the fourth column item 14 of the first data set 10 The key can be 'parking fee (10 minutes, unit level)'. The representative key of the first column item 21 of the second data set 20 is the parking lot and the representative key of the second column item 22 of the second data set 20 is the parking lot operation start time , The representative key of the third column item 23 of the second data set 20 is the parking lot operation end time and the representative key of the fourth column item 24 of the second data set 20 is the address of the ' Lt; / RTI >

According to one embodiment of the present invention, the data matching unit 120 may include a first column of the first data set 10 of the plurality of column items of the first data set 10 and the second data set 20, The degree of similarity between the selected column items can be calculated by selecting the third column items 11 to 13 and selecting the first to third column items 21 to 23 of the second data set 20. At this time, the number of the plurality of column items of the first data set 10 and the second data set 20 must match. That is, the data matching unit 120 can determine the number of column items to be matched with the data matching. As another example, the user may select a column item for which to perform data matching.

The data matching unit 120 may select a representative column item among a plurality of selected column items. 2C, the data matching unit 120 selects the first column item 11 of the first data set 10 as a representative column item and the second column item 11 of the second data set 20 as a representative column item. The column item 21 can be selected as the representative column item to perform the similarity calculation. As another example, the representative column entry may be selected by the user. The data matching unit 120 may perform data matching using a fuzzy algorithm when performing a similarity operation by selecting a representative column item among a plurality of selected column items, but the present invention is not limited thereto.

The data matching unit 120 performs a first matching similarity calculation based on the first columns 11 and 21 of the first data set 10 and the second data set 20 to generate a first similarity rank order data set 30 Can be derived. For example, the first column of the first data set and the second data set may be the parking lot name 11 and the parking lot 21. The first column is not limited to the parking lot 11 and the parking lot name 21 but may be provided in the data matching unit 120 or at the user's choice based on the first column 11 and 21 The first degree of similarity degree calculation may be performed to derive the first degree of similarity ranking data set 30.

The data matching unit 120 may perform a similarity calculation between each row data of the first data set 10 and all row data of the corresponding column items of the second data set 20. [ The first matching similarity computation is performed on the first column data 11 of the first data set 10 and the first column data 21 of the second data set 20 May be an operation for comparing the degree of similarity. Illustratively, referring to FIG. 2C, the first column item 11 of the first data set 10 is a representative key column name, and the first column item 11 of the first data set 10 , The first column item 21 of the first data item 'trainee' and the second item data set 20 is a representative key parking item, and the first column item 21 of the second data set 20 is' , 'Seoul Station 2 Parking Lot', 'Seoul Station 3 Parking Lot', 'Cheonggyecheon Doosan We've the Zenith', 'First Bank', 'Training Center Park', 'New Shinhae Station (Link to Line 9)', Car park, and 'Golden Tower'. That is, the data matching unit 120 compares the respective row data of the first column item 11 of the first data set 10 with the first column item 21 of the second data set 20 .

The first order of likeness ranking data set 30 is included in each row data included in the first column item 11 of the first data set 10 and in the first column item 21 of the second data set 20 The data set may be sorted in the order of higher degree of similarity compared with the respective row data. The first degree of similarity ranking data set 30 according to one embodiment of the present application can be described with reference to FIG. 2D. Column A of FIG. 2D is each row data included in the first column item 11 of the first data set 10. The first degree of similarity Rank 1 of the first order similarities Rank 1 to A of the first data set 10 and the first order similarity Rank 2 of the first column items 11 of the second data set 20 ) Is the data included in the first column item 21 of the second data set 20 arranged in descending order of the value of the result of the similarity calculation between the first column 21 and the second column 21. When the values of the similarity calculation results between the respective row data of the first column items 11 of the first data set 10 and the first column items 21 of the second data set 20 are overlapped, Is included in the first column item 21 of the second data set 20 matched to the data of the first data item 11 of the first data set 10 The data matching unit 120 sets the first column item of the second data set 20 in the second data set 20 to the first data item 20 in the second data set 20, 21) in the order of the column indexes of the first similarity rank order data set.

For example, the data matching unit 120 may store the data similarity calculation result of the 'training source' of the first column item 11 of the first data set 10 into the first column item 21 of the second data set 20 ), (0, 0) (0, 1) of the row data included in the training data ('Training Park', 90 and 5) A ranking data set can be derived. The data matching unit 120 may derive the data of the first matching degree ranking data set 30 including the similarity calculation result value and the column index of the second data set 20 to the first similarity ranking data set. That is, the result of similarity between the 'training center' and the 'training center park' included in the first column item 11 is 90, and the column index in the second data set 20 of 'training center park' have. When the row data included in the first column item 11 and the result value of the similarity of the first column item 21 of the second data set 20 coincide with each other, Can be listed in the order of. For example, the calculation result of 'Seoul Station Parking Window', 'Seoul Station 2 Parking Lot' and 'Seoul Station 3 Parking Lot' is 93, 'Seoul Station 2 Parking Lot' has 1 column index and 'Seoul Station 3 parking lot' The data matching unit 120 may list the operation results in the order of the column indexes arranged in the second data set 20. [

The data matching unit 120 may derive the second similarity rank order data set 40 by performing a second matching similarity calculation based on the first similarity rank order data set 30. In this case, the first similarity ranking data set 30 and the second similarity ranking data set 40 may include ranking information of each data row of the second data set 20. The first similarity ranking data set 30 and the second similarity ranking data set 40 may include respective data rows of the second data set 20 in the order of similarity value.

 The data matching unit 120 compares a plurality of column items of the first data set 10 and the second data set 20 based on the first matching degree of similarity ranking data set 30 to obtain a second similarity ranking data set . The data matching unit 120 derives the first similarity rank order data set 30 by performing a first matching similarity degree calculation based on the first columns 11 and 21, The second matching similarity calculation may be performed for each row data included in the plurality of remaining column items selected from among the column items of the first data set 10 and the second data set 20 according to the rank. The data matching unit 120 sequentially arranges the remaining column data of each row data of the second data set 20 included in the first similarity rank order data set 30 in order of similarity of the first similarity rank order data set 30 (Row data) of the first data set 10 and the remaining column data (each row data) of the first data set 10 to calculate a second matching similarity degree.

Illustratively, referring to FIG. 3, FIG. 3 (a) may be a process of deriving a second similarity ranking data set 40. The data matching unit 120 performs a similarity calculation between the column items based on the first columns 11 and 21 as shown in (1) of FIG. 3 (a) in the second matching similarity degree calculation, The similarity degree calculation between the column items is performed on the basis of the second columns 12 and 22 as shown in (2) of FIG. 3 (a) 23), the similarity calculation between the column items can be performed. In other words, the data matching unit 120 calculates the first matching degree of the second data set 20 based on the first row data 'training source' of the first column item 11 of the first data set 10, And the result of the similarity calculation may be derived as a first similarity ranking data set 30 as shown by (1) in FIG. 3 (a). The data matching unit 120 includes a second column item 12 and a third column item 13 located in the same row as the first row data 'training source' of the first column item 11 of the first data set 10, And a second column item (12) located on the same row as the first data to third data 'training center park' to 'Seoul station 2 parking lot' of the first column item (21) of the second data set (20) Item 13 can be performed. The second matching similarity computation includes a computation of similarity with all the row data of the second data set 20 for each row data of the first data set 10.

The data matching unit 120 performs a second matching similarity calculation to determine whether or not the 'training person' of the column item of the 'parking lot name' of the first data set 10 and the 'parking lot' of the second data set 20 (1), which is the row data of the column item, and the data included in the column item of the 'parking lot operation start time' of the row data of the 'training source' of the first data set 10 (②) of the data included in the 'parking lot operation start time' which is the row data of the 'training center park' of the first data set 10 and the data of the 'training center' of the first data set 10, (3) calculating the similarity between the data included in the column item of the 'parking lot operation end time' and the data included in the 'parking lot operation end time' which is the row data of the 'training center park' of the second data set 20 To determine a second order of similarity ranking and derive a second order of similarity ranking data set 40 .

The data matching unit 120 may derive the data of the first column item 11 and the calculation results of the respective row data of the first similarity rank order data set 30 at the time of deriving the second similarity rank order data set 40 . 3 (b), the data matching unit 120 determines that the column index in the second data set 20 of the training center park is 5, and the column index in the first data set 20 of the training center The matching similarity degree calculation result value is 90, and the second matching similarity degree result value (?, And the average value of the similarity degree result) can be analyzed as 96.67. According to an embodiment of the present invention, it can be seen that the second matching similarity value calculated by considering the first to third column items is increased compared to the first matching similarity value calculated by considering only the first column item, More accurate data matching can be performed through comparison between column items. The data matching unit 120 may match the data rows of the first data set and the second data set based on the similarity.

Referring to FIG. 4, (a) of FIG. 4 illustrates a result of performing a first matching similarity calculation based on the first column 11 of the first data set 10 and the second column of the second data set 20 And may be a diagram showing the result of the data with the highest degree of similarity. For example, the similarity degree of the first degree of similarity calculation result between the 'training center' included in the first data set 10 and the 'training center park' included in the second data set 20 can be calculated to be 90%. For example, if the name of the data overlapping the first column 11 of the first data set 10 exists, the 'Seoul station parking lot' stores the data of the second column 21 of the second data set 20 The matching results are different from each other, and the similarity value can also be equal to 93%. That is, a plurality of data may be matched to the 'Seoul Station 2 parking lot' and the 'Seoul Station 3 parking lot' in the first column 11 of the first data set 10.

4B may show the result of performing the second matching similarity calculation by adding the second column item. Based on the results of performing based on the first column 11 and the second column 21 of the first data set 10 and the second data set 20 at the first matching degree of similarity, , The similarity calculation can be performed. The data matching unit 120 may perform a similarity calculation based on a plurality of columns to derive a result of a similarity calculation more accurately than a result of a single column.

4, the first matching similarity calculation and the second matching similarity calculation are performed based on the first column and the second column. However, when performing the second matching similarity calculation, a plurality of (for example, three) columns are selected, Value can be calculated. The above-described columns may be selected by the data matching unit 120, but the present invention is not limited thereto. It is possible to determine the column items of the first data set 10 and the column items of the second data set 20, Can be performed.

In one embodiment herein. In order to more accurately match the duplicated data generated after the first matching similarity calculation, a second matching similarity calculation is performed based on the first similarity calculation result after the first matching similarity calculation is performed, and the first matching similarity calculation result And the second matching similarity calculation to perform final data matching. The data matching unit 120 compares the first matching similarity degree calculation result and the second matching similarity degree calculation result to perform final data matching so that data matching is performed between the first data set 10 and the second data set 20 The matching accuracy is improved.

The data matching unit 120 compares the first similarity rank order data set 30 and the second similarity order ranking data set 40 and outputs the first data set 10 and the second data set 20 can be matched.

Hereinafter, a is a first rank (Rank 1) in the first similarity ranking data set 30, and a 'is a ranking (Rank 1) in the second similarity ranking ranking data set 40, Lt; / RTI > N is the rank in the second similarity rank order data set 40 of the first rank rank order data set 30 in the first similarity rank order data set 30 and N ' (Rank 1) in the first similarity rank order data set 30.

Referring to FIG. 5A, according to an embodiment of the present invention, the data matching unit 120 may compare N and N 'in step S510. If the comparison result of N and N 'coincides with each other, the data matching unit 120 may compare a and a' in step S520. The data matching unit 120 determines that the row data of the second data set 20 located at the first rank of the first similarity rank order data set 30 is equal to the row data of the second data set 20 when N and N ' It can be determined that the first row data of the first data set 10 is matched with the first row data of the first data set 10. According to one embodiment of the present invention, the data matching unit 120 may associate with the first row data (e.g., A1) of the first data set 10, (N) in the second similarity ranking data set 40 of the row data of the second data set 20 located in the ranking and the second data set (N) located in the first ranking of the second similarity ranking data set 40 20 are all 1 and the rank N 'of the second data set 20 located in the first rank of the first similarity rank order data set 30 is 1 When the row data a and the row data a 'of the second data set 20 located at the first rank of the second similarity rank order data set 40 coincide with each other, It can be determined that the row data of the second data set 30 located in the first rank matches with the first row data of the first data set 10. [

For example, referring to (a) of FIG. 5B, the data matching unit 120 determines that 'N' and 'N' match, and 'a' and 'a' can do. That is, in the second similarity rank order data set 40 of the 'Trainee Park' having the rank 1 in the first similarity rank order data set 30 in association with the 'trainee', the ranking is 1 and the second similarity rank order data set 40, the ranking in the first similarity rank order data set 30 of the 'Trainee Park' having rank 1 is 1, so it can be determined that N and N 'match. In addition, in association with the 'training center', the 'training center' having the rank of the degree of similarity value of the first similarity degree ranking data set 30 and the ' The data matching unit 120 stores the data of the first data set 10 including the training data and the data of the first data set 10 including the training data and the second data including the training data The row data of the set 20) is matched.

In addition, according to one embodiment of the present invention, when N and N 'match in step S510, the data matching unit 120 may compare a and a' in step S520. If it is determined in step S520 that a and a 'are not coincident with the comparison results a and a, the row data of the second data set 20 matched with the first row data can be determined as unmatchable. In other words, in conjunction with the first row data of the first data set 10, the second similarity rank data of the row data of the second data set 20 located in the first rank of the first similarity rank order data set 30 The ranking in the set 40 and the ranking in the first similarity ranking data set 30 of the row data of the second data set 20 located in the first ranking of the second similarity ranking data set 40 are all 1 When there are a plurality of row data of the second data set 20, row data of the second data set 20 matched with the first row data can be determined as unmatchable.

For example, referring to (b) of FIG. 5B, the data matching unit 120 performs matching when N and N 'are equal to 1, and a and a' having N and N ' It can be judged to be impossible. The data matching unit 120 compares the data items included in the second data set 20 in association with the'Opicia 'so that the first similarity degree rank data set 30 ('Optia 1', 89, 13) and (Opicia 1 ', 13, 89, 94.5) and (' Opicia 2 ', 14, 89, and 94), and the second similarity ranking data set 40 ), And the like. The data matching unit 120 determines that the data having the similarity calculation result of 1 in the first similarity rank order data set 30 is'Opithia 1 'and'Opecia 2' (that is, The data having the first degree of similarity calculation result in the second degree of similarity ranking data set 40 can be judged to be 'Opisia 1' and 'Opisia 2' (that is, And the row data having the second highest order of similarity rankings are a plurality of 'Occia 1' and 'Occia 2'). In (b) of FIG. 5B, RANK 1 and RANK 2 may be the number of data, not the order of similarity. Since the data matching unit 120 has a plurality of data that can be matched with 'Occia', the data matching unit 120 provides the determination result to the user to obtain a more accurate matching result, and the first row data of the first data set 10 The row data of the second data set can be matched. That is, when there are a plurality of row data of the second data set 20 in which the first similarity rank order and the second similarity rank order are both first in relation to the specific row data of the first data set 10, The row data of the second data set 20 matching the specific row data of the first data set 10 can not be determined. According to one embodiment of the present invention, when the first row data of the first data set 10 can not be matched with the row data of the second data set to be matched, the first data set 10 is corrected or selected by the user, The row data of the second data set to be matched can be determined.

Also, according to one embodiment of the present invention, if N and N 'do not match in step S510, the data matching unit 120 may compare N and N' in step S530. The data matching unit 120 matches the data having the rank 1 of the second similarity rank order data set 40 with the first row data of the first data set 10 when the N and N 'comparison result N is greater than or equal to N' . In other words, in conjunction with the first row data (e.g., A1) of the first data set 10, the first row data of the second data set 20 and the first similarity ranking data set of the second row data (30) is all 1, the second similarity ranking data set (40) of the first row data of the second data set (20) is 1, and the second row data of the second data set It can be determined that the first row data of the second data set 20 is matched with the first row data if the ranking of the second similarity ranking data set 40 of the second data set 20 is not 1. [ The data matching unit 120 may calculate and compare the rank information N. N 'of each row data a, a'.

For example, referring to (c) of FIG. 5B, the data matching unit 120 may compare N and N 'rankings to determine that data is matched when N and N' do not match. The first row data of the first data set can be determined as " Seoul station parking lot ". The data matching unit 120 compares the calculation result of the first similarity rank order data set 30 of the 'Seoul Station Parking Lot' ('Seoul Station 2 parking lot', 93,1) and ('Seoul Station 3 parking lot', 93,2) (3 parking lots in Seoul Station, 2, 93, and 97.67) and (2 parking lots in Seoul Station 2, 1, 93, and 97.67 in the second similarity degree ranking data set 40) 84.33) or the like to derive the second similarity ranking data set 40. The data matching unit 120 may determine a as a parking lot at the Seoul Station 2, determine a 'as a parking lot at the Seoul Station 3, determine N as 2, and N' as 1. That is, the data matching unit 120 determines that N and N 'are in disagreement and N is positioned higher than N', and data located at 1 in the second similarity rank order data set 40 ) Matches the first row data of the first data set (Seoul station parking lot). The data matching unit 120 may calculate and compare the rank information N. N 'of each row data a, a'.

The data matching unit 120 associates the first row data of the second data set 20 and the first row data of the second data set 20 in association with the first row data of the first data set 10 When the order of the similarity ranking data set 30 is all 1, but the order of the first row data and the second row data in the second similarity order data is different, the order of the rows of the second data set 20 having a higher similarity order It can be determined that the data is matched with the first row data of the first data set 10.

6A and 6B are schematic diagrams for explaining an embodiment of data analysis and similar item analysis according to an embodiment of the present invention.

Referring to FIG. 6A, the profiling unit 130 may analyze data types and character string patterns of a plurality of column items of the first data set 10 and the second data set 20 through data analysis. The profiling unit 130 may analyze the data of a plurality of data sets included in the database 110 in a data type and a character string pattern for each column item. The profiling unit 130 may perform an analysis for shaping the unstructured data included in the database 110.

Referring to FIG. 6B, the similar item analysis unit 140 may analyze similar items between the first data set 10 and the second data set 20 based on the analysis result of the profiling unit 130. For example, the similar item analysis unit 140 may include a representative key of a plurality of column items 11 to 14 included in the first data set 10 and a plurality of column items included in the second data set 20 21 to 24) can be analyzed to perform similarity analysis. The similarity analysis result of the representative key of the 'parking lot operation start time' included in the first data set 10 and the representative key of the 'parking lot operation start time' included in the second data set 20 is calculated to be 99% . That is, the column corresponding to the 'parking lot operation start time' is included in the first data set 10 and is also included in the second data set 20. The similar item analysis unit 140 may calculate 80% of the similarity between the 'parking lot name' of the first data set 10 and the 'parking lot' of the second data set 20. The similar item analyzing unit 140 uses a similar character string as a result of the character pattern analysis of the 'parking lot name' and the 'parking lot', but the same column item is included in the first data set 10 and the second data set 20 It can be judged that it is not.

The data matching unit 120 and the similar item analysis unit 140 may perform similarity analysis based on the Levenshtein distance).

FIG. 7 is a schematic diagram for explaining an embodiment of data merging for big data analysis according to an embodiment of the present invention; FIG.

The data merge unit 150 may merge the first data set 10 and the second data set 20 based on the matching result of the data row. The data merging unit 150 may extract the first data set and the second data set from the first data set and the second data set based on the analysis result of the similar item analyzing unit 140, The combination of the set 10 and the second set of data 20 can be performed. The data merge unit 150 may merge the column items using the matching row data of the first data set 10 and the second data set 20 according to the matching result of the data row.

7, the data merge unit 150 may perform the merging of the first data set 10 and the second data set 20 based on the column items required for the big data analysis . The data merge unit 150 receives the first data set 10 from the column of the parking lot name 11, the parking lot operation start time 12, the parking lot operation end time 13, and the parking fee 14 And the second data set 20 may perform a merge by selecting a column item of the parking lot 21, the parking lot operation start time 22, the parking lot operation end time 23, and the address 24 . For example, the data merge unit 150 may merge the column items based on the column items selected by the user's decision, but the present invention is not limited thereto.

The data merge unit 150 may calculate the parking lot operation start times 12 and 22 overlapping the first data set 10 and the second data set 20 based on the analysis result of the similarity analysis unit 140, It is possible to perform the data merge by selecting one of the two column items of the operation end time 13 and 23.

In one embodiment of the present invention, the data merge unit 150 receives data from the parking lot 11, the parking lot operation start time 12, the parking lot operation end time 13, the parking fee (14) However, it is possible to perform a data merge by selecting a plurality of column items among a plurality of column items included in the first data set 10 and the second data set 20.

8 is a flowchart illustrating a data merging method according to an embodiment of the present invention. The data merge method shown in FIG. 8 is performed by the data merge apparatus 100 described with reference to FIG. 1 through FIG. Therefore, even if omitted below, the description of the data merge apparatus 100 through FIGS. 1 through 7 can also be applied to FIG.

In step S801, the data merge apparatus 100 may determine a first data set 10 and a second data set 20 to perform data matching among a plurality of data sets.

In step S802, the data merge apparatus 100 performs a first matching similarity calculation based on the first column of the first data set 10 and the second data set 20 to generate a first similarity ranking data set 30, Can be derived.

In step S803, the data merge apparatus 100 may derive a second similarity ranking data set 40 by performing a second matching similarity calculation based on the first similarity ranking data set 30. [

In step S804, the data merge apparatus 100 compares the first similarity rank order data set 30 and the second similarity rank order data set 40, and compares the second data that is matched with each data row of the first data set 10 The data row of the set 20 can be matched.

In step S805, the data merge apparatus 100 may merge the first data set 10 and the second data set 20 based on the matching result.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

10: first data set 20: second data set
30: first degree of similarity ranking data set 40: second degree of similarity ranking data set
100: Data merge device
110: Database
120: Data matching unit
130: Profiling portion
140:
150: Data merging unit

Claims (9)

A data merge device for big data analysis,
A database comprising a plurality of data sets;
Determining a first data set and a second data set to perform data matching among a plurality of data sets included in the database and selecting a plurality of column items among the column items of the first data set and the second data set A data matching unit operable to calculate similarities between selected column items and to match the data rows of the first data set and the second data set based on the similarity; And
A data merging unit for merging the first data set and the second data set based on the matching result of the data row,
And a data merge device. The method according to claim 1,
Wherein the data-
Based on the first similarity rank order data set, deriving a first similarity rank order data set by performing a first matching similarity degree calculation based on the first column of the first data set and the second column of the second data set, To derive a second degree of similarity ranking data set,
And comparing the first similarity degree ranking data set and the second similarity degree ranking data set to match the data rows of the first data set and the second data set based on data having the highest degree of similarity. 3. The method of claim 2,
Wherein the first similarity degree ranking data set and the second degree of similarity ranking data set include ranking information of each data row of the second data set. The method of claim 3,
Wherein the data-
In association with the first row data of the first data set, a ranking in a second similarity rank order data set of row data of a second data set located in a first rank of the first similarity rank order data set, Wherein the ranking of the row data of the second data set located at the first rank of the data set is all 1 and the rank of the row data of the second data set located at the first rank of the first rank of similarity ranking data set is & And if the row data of the second data set located in the first order of the second order of similarity ranking data set coincides with the row data of the second data set located in the first order of the first similarity degree ordering data set, And determines that the data is matched with the row data. The method of claim 3,
Wherein the data-
In association with the first row data of the first data set, a ranking in a second similarity rank order data set of row data of a second data set located in a first rank of the first similarity rank order data set, If there are a plurality of row data of the second data set whose rank is all 1 in the first similarity rank order data set of the row data of the second data set located in the first rank of the data set, 2 < / RTI > dataset to be non-matching. The method of claim 3,
Wherein the data-
Wherein the ranking of the first similarity degree rank data set of the first row data and the second row data of the second data set is all 1 in association with the first row data of the first data set, If the ranking of the second similarity ranking data set of the row data is 1 and the ranking of the second similarity ranking data set of the second row data of the second data set is not 1, And determines that the data matches the first row data of the first data set. The method according to claim 1,
A profiling unit for analyzing data types and character string patterns of a plurality of column items of the first data set and the second data set through data analysis; And
A similar item analyzer for analyzing similar items between the first data set and the second data set based on the analysis result of the profiling unit,
Further comprising: The method according to claim 6,
Wherein the data merge unit comprises:
Wherein the first data set and the second data set are combined based on the analysis result of the similarity breakdown analysis unit, except for a column item which is duplicated when a plurality of column items of the first data set and the second data set are merged, To the data merge device. A method for merging data for big data analysis,
Determining a first data set and a second data set to perform data matching among a plurality of data sets included in the database;
Deriving a first similarity rank order data set by performing a first matching similarity degree calculation based on the first data set and the first column of the second data set;
Deriving a second similarity ranking data set by performing a second matching similarity calculation based on the first similarity ranking data set;
Comparing the first similarity degree ranking data set and the second degree of similarity ranking data set to match data rows of a second data set matching each data row of the first data set; And
Merging the first data set and the second data set based on a matching result,
/ RTI >

Images