JP6549786B2 - Data cleansing system, method and program - Google Patents

Description

  The present invention relates to data cleansing.

  Solutions that integrate data scattered in a company and use it for business analysis are attracting attention. In order to realize that, it is necessary to have master data in which attributes included in transaction data are stored without excess or deficiency. The master data needs to have a one-to-one correspondence between the instance of the entity it represents and the record. That is, each record needs to be uniquely identifiable with respect to other records.

  However, master data of many companies is often not suitable for utilization. That is, since many master data are created based on data input by humans, the uniqueness of records is lost due to various causes such as input errors and operational defects, and duplicate records are included. Often Other than that, duplicate records may become a problem when integrating multiple master data in a business merger or the like.

  It should be noted that the uniqueness of a record is the semantic uniqueness of an entity of master data. For example, in the case of master data in which a person is a unit of an entity, such as an employee master or a customer master, data such as personal names and address notations have ambiguities as expressions such as fluctuation of names, alias or omission. obtain. In addition, ambiguity may also occur due to variations in character encoding, such as so-called distinction between half-width characters and full-width characters. Or even if they have the same first and last name, they may be different people. Therefore, in master data in which a person is a unit of entity, it is assumed that there is only one record, assuming that such a typographical ambiguity exists (possibly by unifying the notation according to certain rules if possible). It is important that it corresponds to a person and that the same person does not exist in multiple records.

  The duplicate record in the master data means that the entity indicated by the record is duplicated regardless of the notation difference. In order to utilize in-house data, it is necessary to manually search and correct such duplicate records that may be included in the master data. This task is generally called data cleansing.

  A technique for detecting similar records is known as a technique for supporting data cleansing. Duplicated records can not always be detected by simple string matching because they contain a typographical ambiguity. However, it is highly possible that the records having small notational differences between records, that is, similar records are duplicate records. Patent Document 1 discloses a technique for detecting similar records by generating a feature vector from each record of the data table and comparing them with each other.

  In the data table, a column that makes a record uniquely identifiable is called a "key". The key may be artificially created, but if all the values stored in the column (referred to as "column value") are different (that is, uniqueness is guaranteed), the column can be adopted as the key It is. In addition, even if uniqueness can not be guaranteed in one column, if the combination of multiple columns guarantees uniqueness, the combination of multiple columns can be adopted as a composite key. Such a combination of columns that allows a record to be uniquely identified is called UCC (Unique Column Combination). Patent Document 2 discloses a technique for detecting UCC.

US Patent Application Publication No. 2007/0005556 International Publication No. 2014/191059

  Even if the above-mentioned technique for detecting similar records is adopted in order to support data cleansing of master data, should a certain record be regarded as a duplicate record and be deleted, or the record itself be left, writing fluctuation etc. The final decision on whether or not to make corrections is made by the person responsible for data cleansing. Data cleansing personnel need to visually detect duplicate records and decide whether to delete records or correct the notation. However, in-house master data often consists of a large number of columns, such as more than 100, or many columns have unknown applications. In such a case, the workload of the person in charge of data cleansing is very large.

  In addition, as a result of correcting the data table, such as unification of writing fluctuation and unification of full-width characters and half-width characters, it may occur that keys (columns) that guarantee uniqueness of records no longer guarantee uniqueness. . In particular, if a composite key that guarantees uniqueness of a record no longer guarantees uniqueness due to the modification of the data table, it is possible that the data cleanser may overlook the loss of uniqueness of the composite key. high.

  That is, in the prior art, the work load of the data cleansing person is heavy, and the loss of the key or the composite key that may occur due to the correction work of the person in charge may be overlooked. Therefore, an object of the present invention is to improve the working efficiency and / or working accuracy of data cleansing.

  A data cleansing system according to one embodiment comprises a processor and a memory. The processor reads the data table from the memory, calculates the degree of similarity between the records of the data table, and detects a UCC (Unique Column Combination) which is a set of columns that enables each record of the data table to be uniquely identified. .

  According to the present invention, the working efficiency and / or working accuracy of data cleansing can be improved.

The physical structural example of a data cleansing system is shown. The structural example of the function which a data cleansing system has is shown. An example of a function included in a metadata generation unit is shown. It is a flowchart which shows the example of a data display process. An example of a data table is shown. An example of a similar record matrix is shown. An example of a UCC list is shown. It is a flow chart which shows an example of metadata generation processing. An example of a hash matrix is shown. An example of a MinHash signature is shown. An example of a position list index (PLI) is shown. It is a flowchart which shows an example of a production | generation process of a similar record matrix. It is a flowchart which shows an example of the extraction process of a UCC candidate column. It is a flow chart which shows an example of generation processing of a UCC list. It is a flow chart which shows an example of data edit processing. It is a flowchart which shows an example of a metadata regeneration process. An example of a data table display screen is shown. The example of an improvement data table display screen is shown.

  Examples will be described below. In the following description, information may be described by the expression “xxx table” or “xxx list”, but the information may be expressed by any data structure. That is, "xxx table" or "xxx list" can be called "xxx information" to indicate that the information does not depend on the data structure. Furthermore, when describing the contents of each information, the expressions “identification information”, “identifier”, “name”, “name”, “ID” are used, but they can be mutually replaced.

  Further, in the following description, processing may be described with “program” as the subject, but the program is executed by a processor (for example, a CPU (Central Processing Unit)) to appropriately determine the processing defined. The subject of the processing may be a processor, or an apparatus having the processor, in order to use at least one of a storage resource (for example, a memory) and a communication interface device. Some or all of the processing performed by the processor may be performed by hardware circuitry. The computer program may be installed from a program source. The program source may be a program distribution server or storage medium (eg, portable storage medium).

  Further, in the following description, when different elements of the same type are described separately, like “PLI 303A” and “PLI 303 B”, reference numerals are used, and when the same elements are not distinguished, “ Only the common numbers of the reference signs may be used as in PLI 303 ".

  FIG. 1 shows a physical configuration example of the data cleansing system 100.

  The data cleansing system 100 is an example of a computer, and includes a processor 101, a memory 102, a storage 103, a network interface 104, and a console 105. An example of the data cleansing system 100 is a personal computer, a rack mount server, a blade server or the like. The processor 101 is bi-directionally connected to the memory 102, the storage 103, the network interface 104, and the console 105. The data cleansing system 100 may have only a part of these components or may have a plurality of the same components.

  The processor 101 is an arithmetic device based on hardware such as a central processing unit (CPU), reads a program from the memory 102, and executes the program.

  The memory 102 is composed of volatile semiconductor memory and holds programs, data, and the like. Examples of the memory 102 are a dynamic random access memory (DRAM), a magnetoresistive random access memory (MRAM), and a ferroelectric random access memory (FeRAM).

  The storage 103 is configured of a non-volatile storage device, and holds programs, data, and the like. An example of the storage 103 is a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

  The network interface 104 is configured by a communication device such as a network interface controller (NIC), for example, and is connected to the network 106. The network interface 104 controls a protocol for communicating with other devices via the network 106. An example of the network 106 is Ethernet (registered trademark), a wireless network based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard, a wide area network based on the SONET / SDH (Synchronous Optical Network / Synchronous Digital Hierarchy) standard, or It is a network that combines these multiple network technologies.

  The console 105 includes, for example, an input device such as a keyboard and a mouse, and a display device such as a liquid crystal display panel. The console 105 may receive an operation signal corresponding to the operation input from the input device, and notify the processor 101 of the content of the operation signal. Then, the console 105 displays text information, text and images based on graphical information, and the like output from the processor 101 on the display device.

  An operating system (OS) and a user program stored in the storage 103 may be read out to the memory 102 when the data cleansing system 100 starts up or when it is executed. Then, when the processor 101 executes the OS and the user program read into the memory 102, various functions of the data cleansing system 100 may be realized. The program executed by the processor 101 may be introduced into the data cleansing system 100 via removable media (CD-ROM, flash memory, etc.) or a network, and stored in the storage 103. To this end, the data cleansing system 100 may have an interface for reading data from removable media.

  FIG. 2 shows an exemplary configuration of functions of the data cleansing system 100.

  The data cleansing system 100 has a display unit 202, an operation unit 203, a data editing unit 204, an input / output unit 205, and a metadata generation unit 206 as functions. These functions may be realized by the processor 101 executing a program stored in the memory 102.

  The operation unit 203 interprets the operation content input through the console 105 as various commands. The operation unit 203 may pass a data display instruction to the display unit 202 and may pass a data editing instruction to the data editing unit 204.

  The input / output unit 205 reads the data file 107 stored in the storage 103 and stores it as the data table 201 in the memory 102.

  The metadata generation unit 206 generates the metadata 207 based on the data table 201 stored in the memory 102. The metadata generation unit 206 may store the generated metadata 207 in the memory 102. Details of the metadata generation unit 206 will be described later.

  The display unit 202 displays information on data cleansing through the console 105. The display unit 202 may display information related to the data table 201 and the metadata 207 through the console 105. Further, the display unit 202 may change the display mode of the information in accordance with the content of the data display instruction received from the operation unit 203.

  Note that part or all of the above-described functions may be distributed to a plurality of data cleansing systems 100 for the purpose of distribution of processing load, improvement of availability, and the like. In addition, the data cleansing system 100 may be configured by one physical computer or a plurality of logical or physical computers. When the data cleansing system 100 is configured by a plurality of physical computers, the above-described functions may be realized by the plurality of processors 101 communicating via the network 106.

  FIG. 3 illustrates an example of functions included in the metadata generation unit 206.

  The metadata generation unit 206 may include the similar record detection unit 208, the UCC detection unit 209, and the hash matrix generation unit 210 as functions. The metadata 207 may include a similarity record matrix 250 and a UCC list 260.

  The hash matrix generation unit 210 generates a hash matrix 301 (see FIG. 9) from the data table 201.

  The similar record detection unit 208 uses the hash matrix 301 generated by the hash matrix generation unit 210 to calculate the degree of similarity between the records included in the data table 201. The similar record calculating unit 208 may store the calculated degree of similarity in the similar record matrix 250.

  The UCC detection unit 209 detects the UCC (set of columns) from the data table 201 using the hash matrix 301 generated by the hash matrix generation unit 210. The UCC detection unit 209 may register the detected UCC in the UCC list 260.

  The similar record detection unit 208 and the UCC detection unit 209 may use the same hash matrix 301 generated by the hash matrix generation unit 210. As a result, the processing amount of the entire system can be reduced compared to the case where the hash matrix for similar record detection and the hash matrix for UCC detection are separately generated.

  FIG. 4 is a flowchart showing an example of data display processing.

  The input / output unit 205 reads the data file 107 from the storage 103 (step S402).

  The input / output unit 205 parses the data file 107 serialized in a format such as, for example, a CSV (Comma-Separated Values) format, and generates the data table 201 (see FIG. 5) (step S404).

  The metadata generation unit 206 generates the metadata 207 from the generated data table 201 and stores it in the memory 102 (step S406). Details of the process will be described later.

  The display unit 202 displays the data table 201 and the metadata 207 through the console 105 (step S408). The display example will be described later (see FIGS. 17 and 18).

  FIG. 5 shows an example of the data table 201.

  The data table 201 is target data of data cleansing in the present embodiment. The data table 201 may store any data. The data table 201 includes a plurality of records and a plurality of columns, and each column of the records may store a value (referred to as a column value or a cell value).

  In this embodiment, for the sake of explanation, each record is assigned a record ID (R001, R002,...) Capable of uniquely identifying the record. Further, each column is assigned a column ID (C001, C002,...) Which can uniquely identify the column. The record ID and / or the column ID need not be included in the original data file 107, and may be assigned by the parsing process of the input / output unit 205. The record ID may be called a record name. The column ID may be called a column name.

  In the example of FIG. 5, the record of the record ID "R001" has the column value "AAA" in the column ID "C001", the column value "CCC" in the column ID "C002", and the column value "0" in the column ID "C003" , Column ID "C004" has a column value "0".

  FIG. 6 shows an example of the similar record matrix 250. The similar record matrix 250 may be included in the metadata 207.

  The similar record matrix 250 manages the degree of similarity between two records included in the data table 201.

  Each record ID included in the data table 201 may be assigned to each row and each column in the similar record matrix 250. In the cell at the intersection of the row record ID and the column record ID, the similarity between the record of the record ID of the row and the record of the record ID of the column may be stored.

  The similarity may be a value that can range from 0 to 1, indicating that the larger the value, the more similar. The example of FIG. 6 indicates that the similarity between the record IDs “R002” and “R001” is “0.80” (relatively similar).

  FIG. 7 shows an example of the UCC list 260. The UCC list 260 may be included in the metadata 207.

  The UCC list 260 manages a set of column IDs (that is, UCC) that can uniquely identify each record of the data table 201.

  For example, when all the records in the data table 201 can be uniquely identified by a combination of the column value of column ID "C001" and the column value of column ID "C002", the column IDs "C001" and "C002" are used. The set is UCC. In this case, the UCC list 260 stores a set of column IDs “C001” and “C002”.

  In the UCC list 260, each UCC may be assigned a UCC ID (U001, U002,...) That can uniquely identify UCC (a set of column IDs).

  FIG. 8 is a flowchart illustrating an example of metadata generation processing.

  The metadata generation unit 206 calculates a hash value of each column value of the data table 201 using a certain hash function (step S602).

  The metadata generation unit 206 generates a hash matrix 301 (see FIG. 9) for the data table 201 using the calculated hash value (step S604).

  The metadata generation unit 206 generates the similar record matrix 250 (see FIG. 6) and the UCC list 260 (see FIG. 7) using the generated hash matrix 301 (step S606).

  FIG. 9 shows an example of the hash matrix 301.

  The hash matrix 301 is a matrix composed of hash values calculated by applying a hash function to each column value of the data table 201. The hash matrix 301 may be generated for each different hash function. In this embodiment, an ID capable of uniquely identifying each hash function is called a "hash function ID".

  The hash matrix 301 may have each record ID of each data table 201 and each column ID in each row and each column. The hash value of the column value of the column ID in the record of the record ID of the data table 201 may be stored in the cell of the intersection of the record ID of the row and the column ID of the column.

  In FIG. 9, for the sake of explanation, a record ID is given to a row and a column ID is given to a column. Such an ID may not be assigned to the hash matrix 301 actually stored in the memory.

  FIG. 10 shows an example of the MinHash signature 302.

  The MinHash signature 302 is used for the MinHash method. The MinHash signature 302 may be generated based on the hash matrix 301.

  For the purpose of description, each record ID of the hash matrix 301 may be assigned to each row of the MinHash signature 302 of FIG. In addition, each hash function ID (h1, h2,...) Described in FIG. 9 may be assigned to each column.

  In the cell at the intersection of the record ID of the MinHash signature 302 and the hash function ID, the smallest hash value among a plurality of hash values belonging to the record ID in the hash matrix 301 generated from the hash function of the hash function ID Is stored. For example, assuming that the hash matrix 301 of FIG. 9 is generated from the hash function of the hash function ID “h1”, the record ID “R001” and the hash function ID “h1” in the MinHash signature 302 of FIG. The smallest hash value “1234” among the plurality of hash values “1234”, “4122”, “5628”,... Belonging to the record of the record ID “R001” of the hash matrix 301 of FIG. Is stored. Similarly, in the column of the intersection of the record ID "R001" and the hash function ID "h2", a hash function of the hash function ID "h2" is used to generate a hash function among a plurality of hash values belonging to the record ID "R001" of the generated hash matrix. The smallest hash value of is stored.

  The hash value calculated by the hash function of the hash function ID “h1” is cyclically shifted, XOR is calculated between the cyclically shifted value and the random number, and the calculated value is set as the hash function ID “h2”. You may use as a value corresponded to the hash value which concerns. In this case, in the MinHash signature 302, the minimum value among the values corresponding to the hash value relating to the hash function ID “h2” may be stored in the cell of the hash function ID “h2”.

  FIG. 11 shows an example of a position list index (PLI) 303.

  The PLI 303 may be generated for each column ID of the data table 201. In the example of FIG. 11, PLI 303A is PLI related to the column ID “C001” of the data table 201. The same applies to PLI 303B, 303C, and 303D.

  The PLI 303 associated with a certain column ID associates and manages a plurality of record IDs having the same hash value in the column ID column in the hash matrix 301 and the same hash value.

  In the example of FIG. 11, the PLI 303A indicates that, in the hash matrix 301, a plurality of record IDs "R001" and "R003" having the same hash value "1234" exist in the column of the column ID "C001".

  It should be noted that the structure of PLI 303 is similar to the data structure commonly known as a hash table. The PLI 303 may be a hash table generated by utilizing the hash values of the hash matrix 301, and only the bucket having two or more entries may be extracted.

  FIG. 12 is a flowchart showing an example of generation processing of the similar record matrix 250. The said process is corresponded to the process of FIG.8 S606.

  The metadata generation unit 206 generates the MinHash signature 302 corresponding to the data table 201 (step S804). The MinHash signature 302 may be generated as described in FIG. 10 above.

  The metadata generation unit 206 may divide the plurality of columns of the generated MinHash signature 302 into several groups. Here, each divided group is called a "band" (step S806).

  The metadata generation unit 206 may combine, for each record ID of the MinHash signature 302, the hash value of the column belonging to the band. Then, the metadata generation unit 206 may apply a predetermined hash function to the combined hash value to calculate the hash value (step S808). The metadata generation unit 206 may execute this process for each band. The process of calculating the hash value may be an algorithm known as so-called LSH (Locality Sensitive Hashing). In this case, it is known that sets of records having the same hash value are likely to be similar.

  The metadata generation unit 206 executes the process of step S 814 for each set of all records having the same hash value (LOOP 2). The set of records selected in each loop process is called "set of selected records".

  The metadata generation unit 206 calculates the probability that the hash values of the MinHash signature 302 of the set of selected records match (step S 814). This probability is known to approximate an index of similarity between two sets called Jaccard distance. Therefore, this probability is regarded as similarity, and is stored in the similar record matrix 250.

  FIG. 13 is a flowchart showing an example of UCC candidate column extraction processing.

  This process is a process of extracting a column likely to be included in the UCC (referred to as “UCC candidate column”) before the generation process of the UCC list 260 (see FIG. 14). By performing this process, the amount of UCC detection processing can be reduced.

  The metadata generation unit 206 executes steps S904 to S908 for each of all the columns of the hash matrix 301 (LOOP1). The column selected in each loop processing is called "selected column".

  The metadata generation unit 206 calculates the cardinality of the selected column of the data table 201 using the hash value of the selected column of the hash matrix 301 (step S904). The cardinality of a column may be the number (different number) of types of column values stored in the column. As a method of approximating the cardinality from the hash value, the HyperLog Log algorithm may be adopted.

  The metadata generation unit 206 determines whether the calculated cardinality is less than or equal to a predetermined threshold (step S906). If the determination result is affirmative (step S906: YES), the metadata generation unit 206 excludes the selected column from the UCC candidates (step S908). This is because columns with low cardinality have a small number of different column values, and thus are unlikely to constitute UCC.

  If the determination result is negative (step S906: NO), the metadata generation unit 206 does not need to do anything. UCC candidate columns are extracted by the above process.

  FIG. 14 is a flowchart showing an example of the generation process of the UCC list 260. The said process is corresponded to the process of FIG.8 S606. This process is an example of a process of generating a UCC list from the UCC candidate columns extracted in the process of FIG.

  The metadata generation unit 206 executes steps S1004 to S1008 for each of all UCC candidate columns (LOOP1). The UCC candidate column selected in each loop process is referred to as "selected UCC candidate column".

  The metadata generation unit 206 generates the PLI 303 for the selected UCC candidate column using the hash value of the hash matrix 301 (step S1004).

  The metadata generation unit 206 determines whether there is an entry having two or more record IDs in the PLI 303 (step S1006). If there is no entry having two or more record IDs in the PLI 303 (step S1006: NO), the metadata generation unit 206 registers the selected UCC candidate column in the UCC list 260 (step S1008). This is because the selected UCC candidate column can guarantee the uniqueness of the record alone. If there is an entry having two or more record IDs in the PLI 303 (step S1006: YES), the metadata generation unit 206 may do nothing in particular.

  Next, the metadata generation unit 206 executes steps S1012 to S1016 for each of all the sets of the remaining UCC candidate columns not registered in the UCC list in the above-described processing (LOOP2). A set of UCC candidate columns selected in each loop process is referred to as a “selected UCC candidate column set”.

  The metadata generation unit 206 regards each entry of the PLI 303 related to the set of selected UCC candidate columns as a set of record IDs, and calculates a common set (step S1012).

  The metadata generation unit 206 determines whether the calculated common set is an empty set (step S1014). If the common set is an empty set (step S1014: YES), the metadata generation unit 206 registers the set of selected UCC candidate columns in the UCC list 260. This is because the set of selected UCC candidate columns can guarantee the uniqueness of the record in the set of columns. If the common set is not an empty set (step S1014: NO), the metadata generation unit 206 may do nothing in particular.

  FIG. 15 is a flowchart showing an example of the data editing process.

  When the data table 201 is edited (corrected), the contents of the metadata 207 may also change accordingly. The present processing is an example of data editing processing and metadata regeneration processing generated accordingly.

  When the data editing unit 204 receives the data editing instruction (step S1102), the data editing unit 204 edits the data table 201 (step S1104). The data editing instruction may be passed from the operation unit 203 which has received the data editing input operation through the console 105 to the data editing unit 204.

  The metadata generation unit 206 regenerates the metadata 207 (step S1106). Details of the process will be described later (see FIG. 16).

  The display unit 202 displays the content of the edited data table 201 and the regenerated metadata 207 through the console 105 (step S1108).

  FIG. 16 is a flowchart illustrating an example of the metadata regeneration process. The process corresponds to the process of step S1106 in FIG.

  Here, an example in the case of receiving the data editing instruction related to the record deletion and an example in the case of receiving the data editing instruction related to the cell update are shown. The data editing instruction related to the record deletion is issued, for example, when deleting a record determined to be a duplicate record in the cleansing operation. A data editing instruction related to cell update is issued, for example, when data is rewritten in order to unify the notation.

  The metadata generation unit 206 determines whether the received data editing instruction is a record deletion or a cell update (step S1202).

  First, the case where a data editing instruction related to record deletion is received will be described. In the process, a process of updating the PLI 303 and the similar record matrix 250 is performed.

  The metadata generation unit 206 acquires the hash value of each column value belonging to the record to be deleted from the hash matrix 301 (step S1206).

  The metadata generation unit 206 deletes the acquired hash value and the record ID to be deleted from the PLI 303 of each column (step S1208).

  The metadata generation unit 206 deletes the record ID to be deleted from the hash matrix 301, the MinHash signature 302, and the similar record matrix 250 (step S1210). Then, the process ends.

  Next, the case where a data editing instruction related to the update of the cell value is received will be described. In the process, an update process of the UCC list 260 is performed. In the description of this process, the updated cell value is referred to as "updated cell value".

  The metadata generation unit 206 calculates a hash value from the updated cell value, and updates the hash matrix 301 using the calculated hash value (steps S1222 to S1224).

  The metadata generation unit 206 updates the PLI 303 of the column including the updated cell value (step S1226).

  The metadata generation unit 206 determines whether a column ID (referred to as “update column ID”) including the updated cell value is included in the UCC list 260 (step S1228). If the determination result is affirmative (S1228: YES), the metadata generation unit 206 proceeds to the process of the next step S1230, and if negative (step S1228: NO), this process ends.

  The metadata generation unit 206 performs the process of the following steps S1232 to S1236 for each UCC including the update column ID (step S1230).

  That is, the metadata generation unit 206 acquires an entry of the updated hash value from the PLI 303 of the update column ID. Then, the metadata generation unit 206 calculates a common set between the record ID group of the acquired entry and the record ID group of the entry of the other PLI 303 (step S1232).

  The metadata generation unit 206 determines whether the calculated common set is an empty set (step S1234). That is, the metadata generation unit 206 determines whether the acquired record ID group is not included in the record ID group of any other PLI 303 or included in the record ID group of any other PLI 303. Determine if it exists.

  If the common set is not an empty set (step S1234: NO), the metadata generation unit 206 deletes the set of column IDs of the non-empty PLI 303 from the UCC list 260 (step S1236). This is because the column ID set is not UCC due to the update of the cell value.

  If the common set is an empty set (step S1234: YES), the metadata generation unit 206 does not have to do anything in particular.

  The metadata generation unit 206 may execute an update process of the similar record matrix 250 in addition to the above process.

  In the above process, the UCC list 260 is updated only when the column including the updated cell value belongs to the UCC. That is, according to the present embodiment, it is possible to reduce the amount of update processing of the UCC list 260 when the cell value is updated.

  FIG. 17 shows an example of the data table display screen 400. This screen may be displayed by the process of step S408 of FIG. 4 or step S1108 of FIG.

  The display unit 202 may generate a data table display screen 400 as shown in FIG. 17 based on the data table 201 and may display it on the console 105. On the data table display screen 400, the record ID and the column ID of the data table 201 may be displayed together.

  FIG. 18 shows an example of the improved data table display screen 401. This screen may be displayed by the process of step S408 of FIG. 4 or step S1108 of FIG.

  The display unit 202 may generate an improved data table display screen 401 as shown in FIG. 18 based on the data table 201 and the metadata 207, and may display it on the console 105.

  The improved data table display screen 401 may include a button 402 capable of selecting a column ID or a column ID group belonging to the UCC list 260. When a person in charge of data cleansing presses this button 402, a column corresponding to the column ID or column ID group selected by the button 402 in the data table can be distinguished from other columns (for example, in different colors) It may be highlighted (see shaded area in FIG. 18).

  Further, in the similarity display area 403 between records of the improved data table display screen 401, the similarity between the records in the similar record matrix 250 may be displayed. At this time, the display unit 202 may display records having high similarity as high as possible. For example, the display unit 202 may sort and display records in descending order of similarity.

  Note that the improved data table display screen 401 of FIG. 18 is merely an example in which the information included in the metadata 207 is displayed on the console 105, and the display mode is not limited to this.

  According to this embodiment, records with high similarity can be displayed at the top. This allows data cleansing personnel to easily find the records that may need data cleansing.

  Moreover, according to the present embodiment, the columns belonging to UCC can be displayed in a recognizable manner. This allows the data cleansing agent to easily recognize which cell value would cause UCC's relationship to be lost.

  Furthermore, according to the present invention, records having high similarity and columns belonging to UCC can be displayed together. This enables the person in charge of data cleansing to correct data while matching the semantic uniqueness of the record with the notional uniqueness. That is, the person in charge of data cleansing can efficiently perform the data cleansing task.

  The embodiment described above is an illustration for explaining the present invention, and is not intended to limit the scope of the present invention to the embodiment. Those skilled in the art can practice the present invention in various other aspects without departing from the scope of the present invention.

  For example, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. The configuration of another embodiment may be added to the configuration of one embodiment. Other configurations may be added to, deleted from, or replaced with some of the configurations of the respective embodiments.

  In addition, each configuration, function, processing unit, processing means, and the like in the above-described embodiment may be realized by hardware, for example, by designing part or all of them with an integrated circuit or the like. It may be realized by software by interpreting and executing a program that realizes a function. Information such as programs, tables, and files for realizing each function can be stored in a memory, a hard disk drive, a storage device such as a solid state drive (SSD), or a recording medium such as an IC card, an SD card, or a DVD. .

  Also, in the drawings, control lines and / or information lines which are considered to be necessary for explanation are shown, and not all control lines and / or information lines necessary for implementation are shown. That is, even if not shown, almost all the components may be connected to each other.

  100: data cleansing system 201: data table 206: metadata generation unit 208: similar record detection unit 209: UCC detection unit 210: hash matrix generation unit

Claims (9)

A system for performing data cleansing, comprising a processor and a memory,
The processor is
Read data table from the memory,
Calculate the similarity between records in the data table,
Detecting a UCC (Unique Column Combination) which is a set of columns that enables each record of the data table to be uniquely identified,
A data cleansing system that displays the similarity and UCC. The processor is
Generate a hash matrix from the data table,
The data cleansing system according to claim 1, wherein the calculation of the degree of similarity and the detection of the UCC are performed using the generated hash matrix. The data cleansing system according to claim 2, wherein the processor calculates the similarity between records of the data table based on the MinHash method. The processor is
The cardinality of each column of the data table is calculated from the generated hash matrix,
The data cleansing system according to claim 2, wherein a column whose calculated cardinality is equal to or less than a predetermined threshold is excluded from UCC candidates. The processor is
In displaying the contents of the data table, a record having a high degree of similarity is displayed at the top, and a column included in the UCC is displayed in a distinguishable manner from other columns.
The data cleansing system according to claim 1, which receives a change in the value of the data table. The processor is
If there is more than one UCC, accept the UCC selection,
The data cleansing system according to claim 5, wherein the columns included in the selected UCC are displayed in a distinguishable manner from other columns. The processor is
The data cleansing system according to claim 5, wherein when a value of a column included in the UCC is changed, UCC redetection is performed. A method of data cleansing,
Get the data table,
Calculate the similarity between records in the data table,
Detecting a UCC (Unique Column Combination) which is a set of columns that enables each record of the data table to be uniquely identified,
A data cleansing method for displaying the similarity and UCC. In a system that performs data cleansing,
Get the data table,
Calculate the similarity between records in the data table,
Detecting a UCC (Unique Column Combination) which is a set of columns that enables each record of the data table to be uniquely identified,
A computer program for executing displaying the similarity and UCC.

Images