KR102033151B1 - Data merging device and method for bia datda analysis - Google Patents

Abstract

The data merging apparatus for big data analysis may determine a database including a plurality of data sets, a first data set and a second data set to perform data matching among the plurality of data sets included in the database, Selecting a plurality of column items among the column items of the first data set and the second data set to calculate similarity between the selected column items and matching data rows of the first data set and the second data set based on the similarity. And a data merger configured to merge the first data set and the second data set based on a matching result of the data matcher and the data row.

Description

Data merging device and method for big data analysis {DATA MERGING DEVICE AND METHOD FOR BIA DATDA ANALYSIS}

The present invention relates to a data merging apparatus and method for big data analysis.

For big data analysis, 70% to 80% of the total effort is used for data preprocessing. Big data analysis is exploding, but the development of technology related to data preprocessing is as slow as the development of big data analysis technology. Therefore, the development of automated data preprocessing technology is emerging.

In addition to the relational database (RDBMS) -based and internal data analysis environment, the necessity of reloading information value through mash-up with external data is emphasized.

However, merging of data is impossible when it is difficult to match mechanically because the data are different from each other in terms of the purpose and use of internal and external data generation, or because of the absence of a data standard. In this case, excessive human resources must be input to match the data, and it is not always possible to make it constant.

Therefore, research on data merging technology using data matching algorithm is needed to activate big data analysis.However, the current data matching algorithm only matches between single data items. Can't. In particular, in the case of Korean data, there is a problem that the accuracy is poor. In addition, despite the data that needs to be corrected by the user, there was a problem of matching the data.

Also, Fuzzy Data Matching algorithm is the most commonly used data matching algorithm in the data integration process. This algorithm is an algorithm for matching data using calculated values based on the edit distance (Levenshtein Distance). However, existing algorithms have a disadvantage in that structural matching is difficult in the case of tables using multiple keys because they are matched through comparison between single keys. In addition, since the algorithm was developed based on the English language, there is a limit of accuracy only by using an existing algorithm for Korean data.

The background technology of the present application is disclosed in Korean Patent Laid-Open Publication No. 2011-0099783 (published date: 2001.11.09).

The present invention is to solve the above-described problems of the prior art, a data merging device for merging a plurality of data by performing a similarity operation by selecting a plurality of column items from a plurality of data sets when matching is difficult due to similarity analysis of a single data And to provide a method.

In addition, the present application is to solve the above-mentioned problems of the prior art, and provides a data merging apparatus and method for merging heterogeneous data more accurately by providing matching information to the user for the items that cannot be matched when matching a plurality of data sets It is intended to provide.

In addition, the present application is to solve the above-described problems of the prior art, the first matching similarity operation and the second matching similarity operation after performing the first matching similarity operation matching result and the second matching similarity operation to compare the final data An object of the present invention is to provide a data merging apparatus and method for merging a plurality of data more accurately.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, a data merging apparatus for big data analysis, the database including a plurality of data sets, the first to perform data matching among a plurality of data sets included in the database Determine a data set and a second data set, select a plurality of column items among the column items of the first data set and the second data set, calculate a similarity between the selected column items, and based on the similarity, The data matching unit may match a data row of a set and the second data set, and a data merger may merge the first data set and the second data set based on a matching result of the data rows.

According to an embodiment of the present disclosure, the data matching unit may derive a first similarity ranking data set by performing a first matching similarity operation based on first columns of the first data set and the second data set, A second similarity ranking data set is derived by performing a second matching similarity operation based on a first similarity ranking data set, and is compared with the first similarity ranking data set and the second similarity ranking data set based on the highest similarity data. The data rows of the first data set and the second data set may be matched.

According to one embodiment of the present application, the first similarity ranking data set and the second similarity ranking data set calculated through the data matching unit may include rank information of each data row of the second data set.

According to an embodiment of the present application, the data matching unit, in association with the first row data of the first data set, the second of the row data of the second data set located in the first rank of the first similarity ranking data set The ranking in the first similarity ranking data set of the ranking in the similarity ranking data set and the row data of the second data set located in the first ranking of the second similarity ranking data set is all 1, and the first similarity ranking data set. A first rank of the first similarity rank data set when the row data of the second data set located in the first rank of and the row data of the second data set located in the first rank of the second similarity rank data set match. The row data of the second data set located at may be determined to match the first row data.

According to an embodiment of the present application, the data matching unit, in association with the first row data of the first data set, the second of the row data of the second data set located in the first rank of the first similarity ranking data set The row data of the second data set having a rank of the first similarity rank data set of both the rank in the similarity rank data set and the row data of the second data set located in the first rank of the second similarity rank data set is 1; When there are a plurality of row data, row data of a second data set matching the first row data may be determined as non-matchable.

According to one embodiment of the present application, the data matching unit, in association with the first row data of the first data set, of the first similarity ranking data set of the first row data and the second row data of the second data set The ranking is all 1, the ranking of the second similarity ranking data set of the first row of data of the second data set is one, and the ranking of the second similarity ranking data set of the second row of data of the second data set is one. If not, it may be determined that the first row data of the second data set matches the first row data.

According to an embodiment of the present disclosure, a data merging device for big data analysis may include: a profiling unit configured to analyze data types and string patterns of a plurality of column items of the first data set and the second data set through data analysis; The apparatus may further include a similar item analyzer configured to analyze similar items between the first data set and the second data set based on analysis results of the profiling unit.

According to an embodiment of the present disclosure, the data merger is based on an analysis result of the quasi-yield analysis unit, except for overlapping column items when merging a plurality of column items of the first data set and the second data set. Merging of the first data set and the second data set may be performed.

According to an embodiment of the present disclosure, a data merging method for big data analysis may include: determining a first data set and a second data set to perform data matching among a plurality of data sets included in a database, the first data set; Deriving a first similarity ranking data set by performing a first matching similarity operation based on a first column of a data set and the second data set, and performing a second matching similarity operation based on the first similarity ranking data set Deriving a second similarity ranking data set, comparing the first similarity ranking data set and the second similarity ranking data set, and matching data rows of a second data set matching each data row of the first data set; Merging the first data set and the second data set based on the matching result.

The above-mentioned means for solving the problems are merely exemplary and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the aforementioned problem solving means of the present application, a plurality of data can be merged by selecting a plurality of column items from a plurality of data sets and performing a similarity calculation.

In addition, according to the above-described problem solving means of the present application, by providing matching information to the user for the item that can not be matched when matching a plurality of data, it is possible to merge a plurality of data more accurately.

In addition, according to the above-described problem solving means of the present invention, after performing the first matching similarity operation and the second matching similarity operation, the first matching similarity operation by comparing the matching result and the second matching similarity operation to perform the final data matching, It is possible to merge a plurality of data more accurately.

1 is a block diagram schematically illustrating a configuration of a data merging device according to an embodiment of the present disclosure.
2A to 2D are schematic diagrams for describing a first matching similarity calculation according to an embodiment of the present application.
3 is a schematic diagram illustrating a second matching similarity calculation according to an embodiment of the present application.
4 is a diagram exemplarily illustrating a result of a first matching similarity calculation and a second similarity calculation according to an exemplary embodiment of the present application.
5A through 5B are schematic diagrams for comparing data rows of a first data set and a second data set by comparing a first similarity rank data set and a second similarity rank data set according to an exemplary embodiment of the present disclosure.
6A to 6B are schematic diagrams for describing an embodiment of data analysis and similarity analysis according to an embodiment of the present application.
7 is a schematic diagram illustrating an embodiment of data merging for big data analysis according to an embodiment of the present application.
8 is a flowchart illustrating a data merging method according to an embodiment of the present application.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted for simplicity of explanation, and like reference numerals designate like parts throughout the specification.

Throughout this specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. do.

Throughout this specification, when a member is said to be located on another member "on", "upper", "top", "bottom", "bottom", "bottom", this means that any member This includes not only the contact but also the presence of another member between the two members.

Throughout this specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding the other components unless specifically stated otherwise.

1 is a block diagram schematically showing the configuration of a data merging apparatus according to an embodiment of the present application, Figures 2a to 2d is a schematic diagram for explaining a first matching similarity calculation according to an embodiment of the present application, Figure 3 is 4 is a schematic diagram illustrating a second matching similarity calculation according to an embodiment of the present disclosure, and FIG. 4 is a diagram illustrating a first matching similarity calculation result and a second matching similarity calculation result according to an embodiment of the present disclosure. 5A through 5B are schematic diagrams for comparing data rows of a first data set and a second data set by comparing a first similarity rank data set and a second similarity rank data set according to an exemplary embodiment of the present application. .

Referring to FIG. 1, the data merging device 100 may include a database 110, a data matching unit 120, a profiling unit 130, a similar item analysis unit 140, and a data merging unit 150. Can be. As another example, the data merging device 100 may receive data required for data merging from an external server, but is not limited thereto.

The data merging apparatus 100 determines the first data set and the second data set among the plurality of data sets, selects a plurality of column items among the column items of the first data set and the second data set, and determines the similarity between the selected items. Can be calculated. The data merging apparatus 100 may match data rows of the first data set and the second data set based on the similarity. In addition, the data merging apparatus 100 may provide a user with a determination of a match at the time of a non-determination based on whether a match is determined when merging a plurality of data, and merge the plurality of data based on information on whether the user has determined the match. The data merging apparatus 100 may merge the first data set and the second data set based on the matching result of the data rows.

According to an embodiment of the present disclosure, the data merging device 100 may merge data by matching the first data set and the second data set by using an artificial intelligence-based data matching algorithm in a plurality of data sets.

In addition, the data merging apparatus 100 may overlap the first data and the second data of the second data set with the first data, which is a problem that occurs when a data matching result is performed through a similarity operation between existing single column items. The data matching may be performed by performing a similarity operation between a plurality of column items to match the first row data of the first data set with the first row data of the second data set. ) Analyzes and matches the data similarity of multiple data sets, so that data can be merged without the involvement of data engineer experts and staff. The data merging apparatus 100 may improve data matching accuracy by learning and analyzing an increasing amount of data and data characteristics of a specific business domain in a big data environment.

In addition, the data merging apparatus 100 may perform more accurate data matching and merging by correcting the result by allowing a user to directly select matching candidates for a data item determined to be impossible to match.

The database 110 may include a plurality of data sets used for big data merging. The database 110 may include unstructured data.

The data matching unit 120 may determine the first data set 10 and the second data set 20 to perform data matching among the plurality of data sets included in the database 110. According to the exemplary embodiment of the present application, the first data set 10 and the second data set 20 may include data including a plurality of column items of similarity information preset based on the similarity calculation result of the similarity analysis unit 140. It can be three. According to another example, the first data set 10 and the second data set 20 may be data sets determined according to a user's selection. The data matching unit 120 may determine a first data set 10 and a second data set 20 including items that need to be merged among a plurality of data sets and perform a similarity operation.

The data matching unit 120 may select a plurality of column items among column items of the first data set and the second data set to calculate the similarity between the selected column items. The data matching unit 120 may perform an operation using an artificial intelligence based algorithm when performing the similarity calculation. According to an embodiment of the present disclosure, the data matching algorithm of the data matching unit 120 may perform a similarity operation using a Fuzzy Data Matching algorithm, but is not limited thereto. Fuzzy Data Matching Algorithm is an algorithm that performs matching between data using the result value calculated based on the editing distance (Levenshtein Distance).

2A to 2B, reference numeral a of FIGS. 2A to 2B denotes a first data set 10, and a reference numeral b of FIGS. 2A to 2B denotes a second data set. (10). The first data set may include four column items 11 to 14. In addition, the second data set may include four column items 21 to 24. Column items included in the first data set and the second data set may be separated by the representative key. For example, the representative key of the first column item 11 of the first data set may be 'parking name', and the first column item 11 may be a set of columns included in the representative key 'parking name'. . That is, the representative key of the first column item 11 included in the first data set 10 is 'parking name', and the representative key of the second column item 12 of the first data set 10 is 'parking lot'. Operation start time ', the representative key of the third column item 13 of the first data set 10 is' parking lot operation end time', and the representative of the fourth column item 14 of the first data set 10. The key may be a 'parking fee (per 10 minutes)'. The representative key of the first column item 21 of the second data set 20 is 'parking lot', and the representative key of the second column item 22 of the second data set 20 is 'parking operation start time' The representative key of the third column item 23 of the second data set 20 is 'parking operation end time', and the representative key of the fourth column item 24 of the second data set 20 is 'address'. Can be.

According to an exemplary embodiment of the present disclosure, the data matching unit 120 may include the first column items of the first data set 10 among the plurality of column items of the first data set 10 and the second data set 20. The similarity between the selected column items may be calculated by selecting the third column items 11 to 13 and selecting the first to third column items 21 to 23 of the second data set 20. At this time, the number of column items of the first data set 10 and the second data set 20 must match. That is, the data matching unit 120 may determine the number of column items to be matched with data. As another example, the user may select a column item for data matching.

The data matching unit 120 may select a representative column item among the plurality of selected column items. For example, referring to FIG. 2C, the data matching unit 120 selects the first column item 11 of the first data set 10 as a representative column item, and selects a second column of the second data set 20. Similarity calculation may be performed by selecting the column item 21 as a representative column item. As another example, the representative column item may be selected by the user. The data matching unit 120 may perform data matching using a Fuzzy algorithm when performing a similarity calculation by selecting a representative column item among a plurality of selected column items, but is not limited thereto.

The data matching unit 120 performs a first matching similarity operation based on the first columns 11 and 21 of the first data set 10 and the second data set 20, and thus, the first similarity ranking data set 30. ) Can be derived. For example, the first column of the first data set and the second data set may be the parking lot name 11 and the parking lot 21. Only one embodiment of the present application, the first column is not limited to the parking lot 11 and the parking lot name 21, but based on the first column (11 and 21) by the data matching unit 120 or the user's selection A first similarity ranking data set 30 may be derived by performing a first matching similarity operation.

The data matching unit 120 may perform a similarity operation between each row data of the first data set 10 and all row data of a corresponding column item of the second data set 20. The first matching similarity calculation is performed with the first row data of the first column item 11 of the first data set 10 (eg, the training staff) and the first column item 21 of the second data set 20. It may be an operation for comparing the similarity of. For example, referring to FIG. 2C, the first column item 11 of the first data set 10 is a representative key runner name and is included in the first column item 11 of the first data set 10. The first column item 21 of the first data 'trainer' and the second data set 20 are representative key parking lots, and the 'ultra-building' included in the first column item 21 of the second data set 20. (Tower), 'Seoul Station 2nd Parking Lot', 'Seoul Station 3rd Parking Lot', 'Cheonggyecheon Doosan We've the Zenith', 'Cheil Bank (Main Branch)', 'Training Park', 'Shinbanghwa Station (Line 9)', 'Cheil It may be to perform the similarity calculation with the parking lot, 'Golden Tower'. That is, the data matching unit 120 may compare the similarity between each row data of the first column item 11 of the first data set 10 and the first column item 21 of the second data set 20. Can be.

The first similarity ranking data set 30 is included in each row data included in the first column item 11 of the first data set 10 and in the first column item 21 of the second data set 20. It may be a data set that is ranked in a high degree of similarity compared with each row data. The first similarity ranking data set 30 according to an embodiment of the present disclosure may be described with reference to FIG. 2D. Column A of FIG. 2D is each row data included in the first column item 11 of the first data set 10. The first similarity Rank A of A to the first similarity rank 3 of A are each row data of the first column item 11 of the first data set 10 and the first column item 21 of the second data set 20. ) Is the data included in the first column item 21 of the second data set 20 in which the values of the similarity arithmetic results are arranged in ascending order. When the value of the similarity calculation result between each row data of the first column item 11 of the first data set 10 and the first column item 21 of the second data set 20 overlaps, for example Included in the first column item 21 of the second data set 20 matched with the data of the first data (eg, Seoul station parking lot) of the first column item 11 of the first data set 10. If the data is duplicated (for example, Seoul Station 2 Parking Lot, Seoul Station 3 Parking Lot), the data matching unit 120 may include the first column item (eg, the second data set 20 disposed in the second data set 20). The first similarity ranking data set may be derived in the order of the column index of 21).

For example, the data matching unit 120 may calculate a data similarity calculation result of the 'trainer' of the first column item 11 of the first data set 10 and then use the first column item 21 of the second data set 20. 1) Similarity is determined in order of ('Training Park', 90, 5), ('Ultra Building (Tower)', 0,0) ('Seoul Station 2 Parking Lot', 0, 1) A ranking data set can be derived. The data matching unit 120 may derive data of the first matching similarity ranking data set 30 by including a similarity calculation result value and a column index of the second data set 20 in the first similarity ranking data set. That is, the similarity result between 'training center' and 'training park' included in the first column item 11 is 90, and the column index in the second data set 20 of the 'training park' data is 5 days. have. In addition, when the similarity result values of the row data included in the first column item 11 and the first column item 21 of the second data set 20 coincide, the column index in the second data set 20 is matched. Can be listed in the order of. For example, the calculation result of 'Seoul Station Parking Lot', 'Seoul Station Parking Lot 2' and 'Seoul Station Parking Lot 3' is the same as 93.The column index of 'Seoul Station Parking Lot 2' is 1 and 'Seoul Station Parking Lot 3' is 2 In this case, the data matching unit 120 may list the operation results in the order of the column indexes arranged in the second data set 20.

The data matching unit 120 may derive the second similarity ranking data set 40 by performing a second matching similarity operation based on the first similarity ranking data set 30. In this case, the first similarity ranking data set 30 and the second similarity ranking data set 40 may include ranking information of each data row of the second data set 20. The first similarity ranking data set 30 and the second similarity ranking data set 40 may include each data row of the second data set 20 in a list according to the ranking of similarity values.

 The data matching unit 120 compares a plurality of column items of the first data set 10 and the second data set 20 based on the first matching similarity ranking data set 30 to compare the second similarity ranking data set. Can be derived. The data matching unit 120 derives the first similarity ranking data set 30 by performing a first matching similarity operation based on the first columns 11 and 21, and the similarity degree of the first similarity ranking data set 30. According to the ranking, a second matching similarity calculation may be performed on each row data included in the plurality of remaining column items selected from the column items of the first data set 10 and the second data set 20. The data matching unit 120 sequentially stores the remaining column data of each row data of the second data set 20 included in the first similarity ranking data set 30 in the order of similarity ranking of the first similarity ranking data set 30. The second matching similarity calculation may be performed by calculating the similarity between each row data) and the remaining column data (each row data) of the first data set 10.

For example, referring to FIG. 3, FIG. 3A may refer to a process of deriving a second similarity ranking data set 40. In the second matching similarity calculation, the data matching unit 120 performs a similarity operation between column items based on the first columns 11 and 21 as shown in ① of FIG. 3 (a). In the difference matching similarity calculation, a similarity operation is performed between column items based on the second columns 12 and 22 as shown in (a) of FIG. 3, and the third column 13 and as shown in (3) of FIG. Based on 23), the similarity operation between column items can be performed. In other words, the data matching unit 120 based on the first row data 'trainer' of the first column item 11 of the first data set 10 when calculating the first matching similarity, the second data set 20. The similarity between the first column items 21 may be calculated, and the similarity calculation result may be derived as the first similarity ranking data set 30 as shown in ① of FIG. The data matching unit 120 includes the second column item 12 and the third column item 13 located in the same row as the first row data 'trainer' of the first column item 11 of the first data set 10. And the second column item 12 and the third column located in the same row as the first to third data 'training park' and 'Seoul station 2 parking lot' of the first column item 21 of the second data set 20. A second matching similarity operation between items 13 may be performed. The second matching similarity operation includes a similarity operation with all the row data of the second data set 20 for each row data of the first data set 10.

When performing the second matching similarity calculation, the data matching unit 120 performs the 'training center', which is the row data of the column item of the 'parking name' of the first data set 10, and the 'parking lot' of the second data set 20. Similarity with 'training park', which is the row data of the column item, is calculated (1), and data included in the column item of 'parking operation start time' of the row data of 'training station' of the first data set 10 and 2 Calculate the similarity between the data included in the 'Parking lot start time', which is the row data of the 'Training Park' of the data set 20, and calculate the similarity between the row data of the 'Training Center' of the first data set 10. Average of the result of calculating (3) the similarity between the data included in the column item of the parking lot operation end time and the data included in the parking lot operation end time, which is the row data of the training park in the second data set 20 To determine the second similarity ranking and derive the second similarity ranking data set 40. Can be.

When deriving the second similarity ranking data set 40, the data matching unit 120 includes a result of calculation of each data of the first column item 11 and each row data of the first similarity ranking data set 30. Can be. That is, referring to FIG. 3B, the data matching unit 120 has a column index of 5 in the second data set 20 of the training park, and is the first of the training center and the training park. The matching similarity calculation result value is 90, and the second matching similarity result value (①, and an average value of the similarity result) may be analyzed as 96.67. According to one embodiment of the present application, it can be seen that the second matching similarity value calculated in consideration of the first to third column items is increased compared to the first matching similarity value calculated in consideration of only the first column item. More accurate data matching may be performed by comparing column items. The data matching unit 120 may match data rows of the first data set and the second data set based on the similarity.

Referring to FIG. 4, (a) of FIG. 4 is a result of performing a first matching similarity calculation based on the first column 11 of the first data set 10 and the second data set 20. It may be a diagram showing a result of data having the highest similarity. For example, as a result of the first similarity calculation between the 'training center' included in the first data set 10 and the 'training park' included in the second data set 20, the similarity may be calculated as 90%. When there is a name of overlapping data in the first column 11 of the first data set 10, for example, 'Seoul station parking lot' is the data of the second column 21 of the second data set 20. The matching results are different from each other, and the similarity value may also be 93%. That is, a plurality of data may be matched to 'Seoul Station Parking Lot' and 'Seoul Station Parking Lot 3' in the 'Seoul Station Parking Lot' of the first column 11 of the first data set 10.

FIG. 4B illustrates a result of performing a second matching similarity calculation by adding a second column item. Based on the results of performing the first columns 11 and 21 of the first data set 10 and the second data set 20 in the first matching similarity, including the second columns 12 and 22. , Similarity operation can be performed. The data matching unit 120 may derive a similarity calculation result more accurate than a result of the operation performed in a single column by performing a similarity operation based on a plurality of columns.

In FIG. 4, although the first matching similarity operation and the second matching similarity operation are performed based on the first column and the second column, similarity results are obtained by selecting a plurality of (eg, three) columns when performing the second matching similarity operation. Can compute a value. The column described above may be selected by the data matching unit 120, but is not limited thereto. The column item of the first data set 10 and the column item of the second data set 20 may be determined by a user. The similarity operation of may be performed.

In one embodiment of the present application. To more accurately match duplicate data generated after performing the first matching similarity operation, after performing the first matching similarity operation, the second matching similarity operation is performed based on the first similarity operation result, and the first matching similarity operation result. And the second matching similarity operation may be compared to perform final data matching. The data matching unit 120 compares the first matching similarity calculation result and the second matching similarity calculation result and performs final data matching to perform data matching between row data between the first data set 10 and the second data set 20. This increases the matching accuracy.

The data matching unit 120 compares the first similarity ranking data set 30 and the second similarity ranking data set 40 to the first data set 10 and the second data set based on the highest similarity data. 20 rows of data can be matched.

Hereinafter, in terms used in the description of FIG. 5A, a is data of rank 1 in the first similarity ranking data set 30, and a ′ is rank 1 in the second similarity ranking data set 40. Can be data. N is the rank in the second similarity rank data set 40 of the data having the first rank (Rank 1) in the first similarity rank data set 30, and N 'is the first rank in the second similarity rank data set 40. (Rank 1) may be a rank in the first similarity ranking data set 30.

Referring to FIG. 5A, according to an embodiment of the present disclosure, the data matching unit 120 may compare N with N ′ in step S510. When the comparison result of N and N 'is matched, the data matching unit 120 may compare a with a' in step S520. When N and N 'coincide, and a and a' coincide, the data matching unit 120 matches the row data of the second data set 20 located in the first rank of the first similarity rank data set 30. It may be determined that the first row data of the first data set 10 matches. According to an exemplary embodiment of the present application, the data matching unit 120 is connected to the first row data (eg, A1) of the first data set 10, and thus, the first data of the first similarity ranking data set 30 may be used. A ranking N in the second similarity ranking data set 40 of the row data of the second data set 20 located in the ranking and a second data set located in the first ranking of the second similarity ranking data set 40 ( In the first similarity ranking data set 30 of the row data of 20), the ranking N 'is all 1, and the second data set 20 in the first ranking of the first similarity ranking data set 30 is located. If the row data (a ') of the second data set 20 located in the first rank of the row data (a) and the second similarity rank data set 40 matches, the first similarity rank data set 30 It may be determined that the row data of the second data set 30 positioned in the first rank matches the first row data of the first data set 10.

For example, referring to FIG. 5B (a), when N and N 'match and a and a' match, the data matching unit 120 determines that 'training center' and 'training park' match. can do. That is, the ranking in the second similarity ranking data set 40 of the 'training park' which is ranked 1 in the first similarity ranking data set 30 in association with the 'training center' is 1, and the second similarity ranking data set ( In (40), since the ranking in the first similarity ranking data set 30 of the 'training park' with the ranking is 1, it may be determined that N and N 'match. In addition, the training center having a ranking of the similarity values of the training park and the second similarity ranking data set 40 in which the ranking of the similarity value of the first similarity ranking data set 30 is 1 is associated with the training center. Park 'matches, the data matching unit 120 is a training center (row data of the first data set 10 including' training center ') and' training park '(second training data including' training park ') It may be determined that the row data of the set 20 is matched.

In addition, according to an exemplary embodiment of the present disclosure, when N and N 'match in step S510, the data matching unit 120 may compare a and a' in step S520. If the result of the comparison a and a 'does not match in step S520, the row data of the second data set 20 matching the first row data may be determined as not matchable. In other words, the second similarity ranking data of the row data of the second data set 20 positioned in the first rank of the first similarity ranking data set 30 in association with the first row data of the first data set 10. The rank in the first similarity ranking data set 30 of the rank data in the set 40 and the row data of the second data set 20 located in the first rank of the second similarity rank data set 40 are all one. When there is a plurality of row data of the second data set 20, the row data of the second data set 20 matching the first row data may be determined as not matchable.

For example, referring to FIG. 5B (b), the data matching unit 120 matches when N and N 'are 1, but a plurality of data of a and a' having N and N '1 are matched. It can be judged impossible. The data matching unit 120 compares the data items included in the second data set 20 in association with 'opia', so that the first similarity ranking data set 30 is ('opia1', 89, 13) and ('Ophisia 2', 89, 14) and the like, and the second similarity ranking data set 40 includes ('Ophisia 1', 13, 89, 94.5) and ('Opiasia 2', 14,89,94.5). May be three, listed). The data matching unit 120 determines the data having a similarity calculation result of the first similarity ranking data set 30 as 'Ophicia 1' and 'Ophisia 2' (that is, the row data having the first similarity ranking first). Is a plurality of 'Opcia 1' and 'Opcia 2') and the data having the first similarity calculation result in the second similarity ranking data set 40 may be determined as 'Opcia 1' and 'Opcia 2' (ie, The second similarity ranking ranks first with plural number of 'opia1' and 'opia2'. In FIG. 5B, RANK 1 and RANK 2 may be the number of data rather than the similarity rank. Since the data matching unit 120 has a plurality of data that can be matched with 'Opcia', the data matching unit 120 provides the determination result to the user to obtain a more accurate matching result, and the data of the first row of the first data set 10 is matched. The row data of the second data set may be matched. That is, when the data matching unit 120 has a plurality of row data of the second data set 20 having the first similarity rank and the second similarity rank with respect to the specific row data of the first data set 10, It is not possible to determine the row data of the second data set 20 matching the specific row data of the first data set 10. According to an exemplary embodiment of the present disclosure, when the row data of the second data set that matches the first row data of the first data set 10 cannot be matched, the first data set 10 may be corrected or selected by a user. The row data of the second data set that matches the first row data of may be determined.

In addition, according to an exemplary embodiment of the present application, when N and N 'do not coincide in step S510, the data matching unit 120 may compare N and N ′ in step S530. The data matching unit 120 matches the first row data of the first data set 10 with the first rank data of the second similarity ranking data set 40 when N is greater than or equal to N 'as a result of comparing N and N'. It can be judged. In other words, the first similarity ranking data set of the first row data and the second row data of the second data set 20 in association with the first row data (eg, A1) of the first data set 10. All of the ranks of 30 are 1, the second similarity ranking data set 40 of the first row of data of the second data set 20 has a rank of 1, and the second row of data of the second data set 20. When the rank of the second similarity ranking data set 40 of the other than 1 is determined, it may be determined that the first row data of the second data set 20 matches the first row data. The data matching unit 120 may calculate and compare the ranking information N. N 'of each row data a and a'.

For example, referring to (c) of FIG. 5B, when N and N 'do not match, the data matching unit 120 may compare the ranks of N and N' and determine that the data is matched. The first row data of the first data set may be determined as 'Seoul station parking lot'. The data matching unit 120 calculates the calculation result of the first similarity ranking data set 30 of the 'Seoul station parking lot' ('Seoul Station 2 parking lot', 93,1) and ('Seoul Station 3 parking lot', 93,2), etc. The data set can be derived, and the result of the calculation in the second similarity ranking data set 40 can be obtained from ('Seoul Station 3 Parking Lot', 2, 93, 97.67) and ('Seoul Station 2 Parking Lot', 1, 93, 84.33) may be used to derive the second similarity ranking data set 40. The data matching unit 120 may determine a as the second parking lot in Seoul Station, a 'as the third parking lot in Seoul Station, and determine N as 2 and N as 1. That is, the data matching unit 120 determines that N is inconsistent with N 'and that N is located in a higher rank than N', and data located in 1 of the second similarity ranking data set 40 (Seoul Station 3 Parking Lot). ) May be determined to match the first row data of the first data set (Seoul station parking lot). The data matching unit 120 may calculate and compare the ranking information N. N 'of each row data a and a'.

The data matching unit 120 is connected to the first row data (eg, A1) of the first data set 10, and thus, the first row data and the first row data of the second row data of the second data set 20. If the ranking of the similarity ranking data set 30 is all 1, but the ranking of the first row data and the second row data in the second similarity ranking data are different, the rows of the second data set 20 having the higher similarity ranking are higher. The data may be determined to match the first row data of the first data set 10.

6A to 6B are schematic diagrams for describing an embodiment of data analysis and similarity analysis according to an embodiment of the present application.

Referring to FIG. 6A, the profiling unit 130 may analyze data types and string patterns of a plurality of column items of the first data set 10 and the second data set 20 through data analysis. The profiling unit 130 may analyze data of a plurality of data sets included in the database 110 by data type and string pattern for each column item. The profiling unit 130 may perform an analysis for shaping unstructured data included in the database 110.

Referring to FIG. 6B, the similar item analyzer 140 may analyze similar items between the first data set 10 and the second data set 20 based on the analysis result of the profiling unit 130. For example, the similar item analyzer 140 may represent the representative keys of the plurality of column items 11 to 14 included in the first data set 10 and the plurality of column items included in the second data set 20 ( Similarity analysis can be performed by analyzing the string pattern of the representative key of 21 to 24). The similarity analysis result of the representative key of the 'parking operation start time' included in the first data set 10 and the representative key of the 'parking operation start time' included in the second data set 20 may be calculated with a similarity 99%. Can be. That is, it means that the column corresponding to 'parking start time' is included in the first data set 10 and the second data set 20. The similar item analyzer 140 may calculate the similarity between the 'parking name' of the first data set 10 and the 'parking lot' of the second data set 20 at 80%. Similar item analysis unit 140, but the similar character string is used as a result of the string pattern analysis of the 'parking name' and 'parking lot', the same column item is included in the first data set 10 and the second data set 20 You can judge that.

The data matching unit 120 and the similar item analyzer 140 may perform the similarity analysis based on the similarity analysis based on the Levenshtein distance.

7 is a schematic diagram illustrating an embodiment of data merging for big data analysis according to an embodiment of the present application.

The data merger 150 may merge the first data set 10 and the second data set 20 based on the matching result of the data rows. In addition, the data merging unit 150 may exclude the column items that are overlapped when merging a plurality of column items of the first data set and the second data set based on the analysis result of the similar item analysis unit 140, and the first data. Merging of the set 10 and the second data set 20 may be performed. The data merger 150 may merge each column item by using matching row data of the first data set 10 and the second data set 20 according to the matching result of the data rows.

For example, referring to FIG. 7, the data merger 150 may merge the first data set 10 and the second data set 20 based on column items necessary for big data analysis. . The data merger 150 may include a column of the first data set 10 including the parking lot name 11, the parking lot operation start time 12, the parking lot operation end time 13, and the parking fee (10 minutes per unit). The second data set 20 may include columnar items of the parking lot 21, the parking lot operation start time 22, the parking lot operation end time 23, and the address 24 to perform merging. . For example, the data merging unit 150 may merge the column items based on the column item selected by the user's decision, but is not limited thereto.

The data merging unit 150 based on the analysis result of the similar item analysis unit 140, the parking lot operating start times 12 and 22 overlapped with the first data set 10 and the second data set 20, and the parking lot. Data merging may be performed by selecting one column item among two column items of the operation end times 13 and 23.

In one embodiment of the present application, the data merging unit 150 is a parking lot 11, a parking lot operation start time 12, a parking lot operation end time 13, a parking fee (per 10 minutes, per unit), address (24) Although the column items of) are selected and merged, data merging may be performed by selecting a plurality of column items among a plurality of column items included in the first data set 10 and the second data set 20.

8 is a flowchart illustrating a data merging method according to an embodiment of the present application. The data merging method illustrated in FIG. 8 is performed by the data merging apparatus 100 described with reference to FIGS. 1 to 7. Therefore, although omitted below, the contents described with respect to the data merging device 100 through FIGS. 1 to 7 may also be applied to FIG. 8.

In operation S801, the data merging apparatus 100 may determine the first data set 10 and the second data set 20 to perform data matching among the plurality of data sets.

In operation S802, the data merging apparatus 100 performs a first matching similarity operation based on the first columns of the first data set 10 and the second data set 20 to perform a first similarity ranking data set 30. Can be derived.

In operation S803, the data merging apparatus 100 may derive the second similarity ranking data set 40 by performing a second matching similarity operation based on the first similarity ranking data set 30.

In operation S804, the data merging apparatus 100 compares the first similarity ranking data set 30 and the second similarity ranking data set 40 to match the second data to each data row of the first data set 10. Data rows of the set 20 may be matched.

In operation S805, the data merging apparatus 100 may merge the first data set 10 and the second data set 20 based on the matching result.

The above description of the present application is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above description, and it should be construed that all changes or modifications derived from the meaning and scope of the claims and their equivalents are included in the scope of the present application.

10: first data set 20: second data set
30: first similarity ranking data set 40: second similarity ranking data set
100: data merge device
110: database
120: data matching unit
130: profiling unit
140: analog analysis unit
150: data merge unit

Claims (9)

In the data merging device for big data analysis,
A database comprising a plurality of data sets;
Determine a first data set and a second data set to perform data matching among a plurality of data sets included in the database, and represent a representative column among a plurality of column items included in the first data set and the second data set. Selecting an item, performing a similarity operation for analyzing the string pattern of the row data of each of the selected representative column item of the first data set and the data of the entire representative column item of the second data set, and based on the similarity, A data matcher for matching data rows of the first data set and the second data set; And
Based on a similarity analysis result of analyzing a string pattern of a representative key of a plurality of column items included in the first data set and a representative key of a plurality of column items included in the second data set, the first data set and The first data set and the second data set according to a matching result including a similarity rank of data rows of the first data set and the second data set, except for overlapping column items when merging a plurality of column items of the second data set; A data merging unit for merging column items of the first data set and the second data set using matching row data of the second data set;
Including, the data merging device. The method of claim 1,
The data matching unit,
A first similarity ranking data set is derived by performing a first matching similarity operation based on the first column of the first data set and the second data set, and a second matching similarity based on the first similarity ranking data set. Perform a calculation to derive a second similarity rank data set,
And comparing the first similarity ranking data set and the second similarity ranking data set to match data rows of the first data set and the second data set based on data having the highest similarity. The method of claim 2,
And the first similarity ranking data set and the second similarity ranking data set include ranking information of each data row of the second data set. The method of claim 3, wherein
The data matching unit,
A second similarity rank and a second similarity rank of a second similarity rank data set of row data of a second data set positioned in a first rank of the first similarity rank data set in association with first row data of the first data set; The first similarity rank data sets of the row data of the second data set located at the first rank of the data set are all one, and the row data of the second data set located at the first rank of the first similarity rank data set. And when the row data of the second data set located at the first rank of the second similarity rank data set is matched, the row data of the second data set located at the first rank of the first similarity rank data set is matched to the first data. And determine to match the row data. The method of claim 3, wherein
The data matching unit,
A second similarity rank and a second similarity rank of a second similarity rank data set of row data of a second data set positioned in a first rank of the first similarity rank data set in association with first row data of the first data set; A first similarity degree of the row data of the second data set positioned in the first rank of the data set, when there is a plurality of row data of the second data set having a rank of all 1 in the rank data set, the first matched to the first row data; And determining row data of the two data sets as non-matchable. The method of claim 3, wherein
The data matching unit,
In association with the first row data of the first data set, the first similarity ranking data set of the first row data and the second row data of the second data set are both one, and the first of the second data set is first. The first row of the second data set when the rank of the second similarity rank data set of row data is 1 and the rank of the second similarity rank data set of second row data of the second data set is not one. And determine that data matches the first row data of the first data set. The method of claim 1,
A profiling unit which analyzes data types and string patterns of a plurality of column items of the first data set and the second data set through data analysis; And
A similar item analysis unit analyzing a similar item between the first data set and the second data set based on an analysis result of the profiling unit;
Further comprising, the data merging device. delete In the data merging device, in the data merging method for big data analysis,
Determining a first data set and a second data set to perform data matching among a plurality of data sets included in the database;
A representative column item is selected from a plurality of column items included in the first data set and the second data set, and the row data of each of the representative column items of the selected first data set and the representative column item of the second data set Deriving a first similarity ranking data set by performing a first matching similarity operation analyzing the string pattern of the entire data;
A second similarity similarity operation by performing a second matching similarity operation that compares a string pattern of data of all the representative column items of the first data set selected based on the first similarity ranking data set and the representative column items of the second data set; Deriving a rank data set;
Comparing the first similarity ranking data set and the second similarity ranking data set to match data rows of a second data set matching each data row of the first data set; And
Based on a similarity analysis result of analyzing a string pattern of a representative key of a plurality of column items included in the first data set and a representative key of a plurality of column items included in the second data set, the first data set and The first data set and the second data set according to a matching result including a similarity rank of data rows of the first data set and the second data set, except for overlapping column items when merging a plurality of column items of the second data set; Merging column items of the first data set and the second data set using matching row data of the second data set;
Data merging method comprising a.

Images