CN110147393A - The entity resolution method in data-oriented space - Google Patents

Abstract

A data space-oriented entity resolution method, the invention relates to an entity resolution method. The purpose of the present invention is to solve the existing problem that when performing entity parsing in data space, it is necessary to compare records. For record pairs in different fields, the matching probability is very small, and pairwise comparison will waste resources. The process is: Step 1. Construct the record map; Step 2. Use the pruning method to simplify the record map; Step 3. Block the pruned record map; Step 4. Establish an attribute mapping cluster; Step 5. Calculate the attribute map The goodness of the set; step six, after obtaining the goodness of each mapping set in the attribute mapping cluster, perform entity resolution in the block. The invention is used in the field of data entity analysis.

Description

Entity Resolution Method Oriented to Data Space

technical field

The present invention relates to entity resolution methods.

Background technique

Entity resolution refers to the process of identifying different description forms of the same entity. It aims to ensure data quality and is a key technology in data cleaning, data integration and data mining ^[1] (Vasilis Efthymiou, Kostas Stefanidis, Vassilis Christophides.Big Data Entity Resolution :From Highly to Somehow Similar Entity Descriptions in the Web[C]//Proceeding Big Data'15Proceedings of the 2015IEEE International Conference on Big Data,2015,11(1):401-410P). In traditional entity resolution work, most work relies on schema or semantic mapping between data. Data space is a new data integration method. It does not have strict data schema and semantic mapping, but gradually incorporates data and establishes relationships according to the needs of the subject. It is a heterogeneous data collection, which is characterized by data from multiple Data source ^[2] (Ge Jingjun, Hu Changjun, Liu Xin. A virtual data space sharing model for domain science[J]. Small Microcomputer System, 2014,35(3):514-519PGE Jingjun, HU Changjun, LIU Xin.Virtual Data Space Sharing Model for Domain Science [J]. Minicomputer System, 2014, 35(13): 514-519P). When entity resolution is performed in data space, a powerful tool for entity resolution, semantic mapping, is lost. Entity resolution needs to compare records. For record pairs in different fields, the matching probability is very small, and pairwise comparison will waste resources.

Contents of the invention

The purpose of the present invention is to solve the existing problem of comparing records when performing entity analysis in data space. For record pairs in different fields, the matching probability is very small, and pairwise comparison will waste resources. entity resolution method.

The specific process of the data space-oriented entity resolution method is as follows:

Step 1. Build a record map:

Step 2, using the pruning method to simplify the record map;

Step 3, performing block processing on the clipped recording image;

Step 4. Establish an attribute mapping cluster;

Step 5, calculating the goodness of the attribute mapping set;

Step 6: After obtaining the superiority of each mapping set in the attribute mapping cluster, perform entity resolution in the block.

The beneficial effects of the present invention are:

The present invention proposes block technology ^[3] (Batya Kening, Avigdor Gal.MFIBlocks:Aneffective blocking algorithm for entity resolution[J].Information Systems,2013,38(6):908-926P), that is, to use a cost-effective The low calculation method predicts the data, that is, the data records that may belong to the same entity are placed in a block, and the records are compared only within the block. It solves the problem that when performing entity parsing in the data space, it is necessary to compare records. For record pairs in different fields, the matching probability is very small, and pairwise comparison will waste resources.

This paper conducts theoretical research on multi-source heterogeneous data entity parsing for data space. Considering that even under the non-semantic mapping, two records pointing to the same entity have common points in their attribute values, and the relationship between records is included in the calculation, and the record graph is constructed by combining the two. According to the record collections in different situations, the record graph is simplified through the applicable pruning method, and an algorithm for dividing the record graph according to the pruned record graph is proposed.

When doing entity parsing in a block, use the attribute value to map the attribute, and obtain the information referred to by the attribute name of the overall data record in the block to distinguish the data that has a common value with the existing data in the block but still does not match Come, and propose a method similar to regular expressions, calculate the similarity of attribute values, and merge the attribute values of the mapped attributes of the matching records, so as to return a more comprehensive entity information to the user.

It is verified by experiments that the method proposed in the present invention has a certain positive effect on entity resolution.

Description of drawings

Fig. 1 is the flow chart of the construction record diagram of the present invention;

Fig. 2 is that the present invention carries out pruning flowchart to record map;

Fig. 3 is to carry out block flow chart according to the recording chart after pruning;

Figure 4a is a data map of heterogeneous attribute mapping;

Figure 4b is a global attribute map of heterogeneous attribute mapping;

Figure 4c is a graph of the goodness calculation of the attribute mapping cluster of heterogeneous attribute mapping;

Figure 5a is a graph of the changes of the two methods on the two data sets;

Figure 5b is a graph of the changes of the two methods on the two data sets;

Figure 6 is a comparison diagram of the entity generation of the two algorithms.

Detailed ways

Specific implementation mode 1: The specific process of the entity resolution method oriented to the data space in the implementation mode of the present invention is as follows:

Step 1. Build a record map:

Step 2, using the pruning method to simplify the record map;

Step 3, performing block processing on the clipped recording image;

Step 4. Establish an attribute mapping cluster;

Step 5, calculating the goodness of the attribute mapping set;

Step 6: After obtaining the superiority of each mapping set in the attribute mapping cluster, perform entity resolution in the block, so as to exclude data records that are mistakenly included in this block and point to other entities.

Embodiment 2: The difference between this embodiment and Embodiment 1 is that the record map is constructed in the step 1; the specific process is:

Use a label method to represent data records, and regard data records as a collection of attribute values; at this time, based on a common-sense assumption ^[4] ([4]S.Prabhakar Benny,S.Vasavi Dr,P.Anupriya.Hadoop FrameworkFor Entity Resolution Within High Velocity Streams[J].Procedia ComputerScience,2016,85:550-557P), if two records point to the same entity, they must contain some same attribute values. And calculate the relationship between data records to improve accuracy ^[5] (Xiao Qihua, Chen Ke, Huang Dongmei. Data spatial feature extraction method considering spatial correlation [J]. Computer Simulation, 2014, 31(12): 425-428, 433P XIAO Qihua, CHEN Ke, HUANG Dongmei. Data Spatial Feature Extraction Method Considering Spatial Relevance [J]. Computer Simulation, 2014, 31(12): 425-428, 433P). A record graph model is used to represent record nodes and record node relationships in the data space; by calculating the similarity between two records, an edge is drawn between the two records, and the edge weight is the similarity value. This label-style chunking method, because of its simple representation method, only needs to obtain the attribute value of the record, and does not rely on fixed data patterns and hard-line semantics, so it can be used on heterogeneous data sets facing the data space. It has great applicability.

Assuming a record set R={r ₁ {FullName:Tom Lloyd Malik; Job:producer,Actor;Address:LA}, r ₂ {Name:Tom Malik;Producer;birthPlace:LA},r ₃ {Label:MikeStyles;Profession :producer; Place_of_birth:LA; Place_of_birth:1964},r ₄ {MikeHarry Styles;birthPlace:LA;Gender:male},r ₅ {FullName:Harry Green;Address:LOS;Sex:male;Profession:Writher},r ₆ {Label: Harry Green; Gender: male; birthYear: 1980; married}}. An overview of record graph partitioning methods based on record set R is shown in Figure 1, Figure 2, and Figure 3 (in order to simplify the example diagram, the record sets in the figure do not mark their relationship for the time being).

Step 11, calculating the similarity between two records;

Steps 1 and 2, constructing a record graph according to records and similarities in the data space.

Other steps and parameters are the same as those in Embodiment 1.

Specific embodiment three: the difference between this embodiment and specific embodiment one or two is that the similarity between the two records is calculated in the step one by one; the specific process is:

The tag conversion function tag() can convert a record into a tag set (namely tag:r _i →T(r _i )). Calculate the ratio of the intersection and union sizes of the two sets to get the label similarity of the two records.

Step 111, calculate the label similarity:

Use the tag conversion function tag() to convert the record into a tag set, and calculate the tag similarity between two records, which is recorded as sim _tag (r _i , r _j ):

Among them, T(r _i ) is the normalized label set converted by the record r _i through the label conversion function; T(r _j ) is the normalized label set converted by the record r _j through the label conversion function;

Step 112, calculate the similarity of the relationship:

Integrate the comprehensive similarity of all relations of two records, denoted as sim _rel (r _i ,r _j ):

Among them, Nbr(r _i ) represents the record set connected with record r _i in rel relation, Nbr(r _j ) represents the record set connected with record r _j in rel relation, REL represents the record set connected with record r ₁ , r A collection of all record relationships appearing on ₂ , such as cooperation relationship, teacher-student relationship, teaching relationship, etc.;

Step 113: Integrate the label similarity and relationship similarity to obtain the comprehensive similarity sim(r _i , r _j ):

sim(r _i , r _j )=α·sim _tag (r _i ,r _j )+(1-α)·sim _rel (r _i ,r _j )

Among them, α represents the weight of label similarity.

Other steps and parameters are the same as those in Embodiment 1 or Embodiment 2.

Embodiment 4: This embodiment differs from Embodiment 1 to Embodiment 3 in that, in the step 1 or 2, a record map is constructed according to the records and similarities of the data space; the specific process is:

Given a record set R in a data space, construct an undirected graph G=(R,E), which is called a record graph;

Among them, R is the record set, which represents the records in the data space; E is the edge set, and there is an edge between two records to represent the similarity of the record pair.

After the record graph is constructed, the matching probability of record pairs with smaller edge weights in the graph is lower. By processing the edges of the record graph, unnecessary record pair comparisons are reduced.

Other steps and parameters are the same as those in Embodiments 1 to 3.

Specific implementation mode five: this implementation mode is different from one of specific implementation modes one to four in that the pruning method is adopted in the step 2 to simplify the record diagram; the specific process is:

Delete edges with smaller weights according to certain rules, that is, pruning the graph to reduce redundant matching times.

For the convenience of the following description, the definition of the point area is given.

Definition 4. Point area: A record r appears in the record graph G as a node, and the area formed by the record node itself, its neighbor records with edges and the edges connecting them becomes a point area. The dot region is a subgraph in the record graph G, which is denoted as G _r ={{r}∪R _r ,E _r }. Among them, R _r is the set formed by the neighbor nodes connected with r, and E _r is the set formed by the edges connecting r and the neighbor nodes.

The pruning process has two main components: pruning centers and pruning rules.

The pruning center can be divided into two categories: edge centralization, which selects the global best pair to be compared by traversing the edge set of the graph, so as to filter out the edges that do not meet the pruning rules; node centralization, which traverses All the nodes in the graph aim to find the best pairing set to be compared with a record node in its point area—that is, several records with the largest edge weights connected to this record.

Pruning rules are divided into weight threshold and cardinality threshold according to function. The weight threshold specifies the minimum weight of retained edges, and deletes all edges lower than this weight; the cardinality threshold gives the maximum number of retained edges in the graph, and retains the edges with top-k edge weights. Among them, the cardinality threshold limits the number of pairs to be compared, which is suitable for applications with limited time resources. The weight threshold determines whether to prune the edges connecting record pairs according to the matching probability of the record pair itself, which is suitable for applications that value validity. According to the scope of action, the pruning rules are divided into global threshold and local threshold. The global threshold applies to the entire graph, that is, all edges in the graph; while the local threshold applies to a subset of the graph, that is, the point region of a node.

The pruning method adopts one of edge-centered cardinality pruning, node-centered cardinality pruning, edge-centered threshold pruning, or node-centered threshold pruning;

(1) Radix pruning with edge centralization:

The global cardinality threshold k specifies the total number of edges to be retained in the record graph, and retains the k edges with the largest weight; the edge set can be sorted in descending order according to the weight, and the edges with low weight can be effectively deleted.

(2) Cardinality pruning of node centralization:

For each node r _i , keep the edge connecting the top-k weight of node r _i (that is, the k edges with the largest weight); for node r _i , record the point area G _ri of node r _i = {{r _i }∪R _ri ,E _ri } and the cardinality threshold k _ri of the current record node, traverse the edges in the subgraph, and retain the weight top-k (a total of n edges connected to ri, retain the highest weight k edges of , delete other edges) and delete other edges connecting r _i at the same time;

Among them, R _ri is the set of neighbor nodes connected to the record node r _i by edges, E _ri is the set of edges connecting the record node r _i and neighbor nodes; the point area G _ri is the record graph a subgraph in G;

Generally speaking, the cardinality threshold of each node should depend on the edge set size of its point region (eg k _ri =0.1×|E _ri |).

(3) Edge-centered threshold pruning:

Use the weight threshold to perform pruning in the global scope, select the minimum edge weight w _min (obtained by experiment), traverse all the edges in the graph, and delete the edges with weights lower than w _min ;

It traverses all the edges in the graph and deletes the edges whose weight is lower than the preset threshold w _min , keeps the remaining edges in the graph and outputs them. In general, the weight between matching records is greater than that between unmatched records, so choosing the w _min target is to determine the balance point between the two.

(4) Node-centralized threshold pruning:

For the selection of the pruning range, if a global threshold is used, a unified threshold is used for all nodes in the record graph. At this time, the pruning process is the same as the edge-centered weight pruning scheme; if a local threshold is used, it can be According to the user's needs, a specific threshold is selected for a special node. In essence, it applies the edge-centered weight pruning to the point area of the node r _i . The main difference from the edge-centered threshold pruning scheme is that it can use different thresholds for each node. It first obtains the point region G _ri of r _i ={{r _i }∪R _ri ,E _ri }, and then specifies the minimum edge weight of subgraph pruning according to the input local threshold standard; then, traverses the edges in E _ri , delete the edge whose weight is less than the set threshold.

Other steps and parameters are the same as in one of the specific embodiments 1 to 4.

Specific embodiment six: this embodiment is different from one of the specific embodiments one to five in that in the step three, the clipped record map is processed in blocks; the specific process is:

The graph after pruning is G={R,E}, R is the record set, E is the edge set;

Take any record r _i , create a block b _i and put r _i in b _i , if the node r _j in the point area of r _i is connected to all the nodes in b _i , then place r _j in in b _i , and delete the connected edges between r _j and all nodes in b _i , repeat this operation until all neighbor nodes of r _i are traversed; at this time, if the node in b _i becomes an isolated in the graph If there is no node connected to it, delete this node from the graph; repeat step 3 until the graph G is empty; at this time, the block work is completed, and the block set B={b ₁ ,b ₂ ,... ,b _|B| };

Record comparison and connection based on attribute mapping

After being divided into blocks, the record information in the block contains a large number of repeated attribute values, which is helpful for mapping attributes. By observing the characteristics of the global data and processing the semantic mapping, it can be found that there are some differences in the information of one or several data records. At this time, such records can be divided to improve accuracy. Integrate the matching record information, and users can get what they need, which can reduce the time for processing record information.

Mapping of heterogeneous attributes

A common method in entity resolution is to use uniform attributes to calculate attribute value similarity. However, in the multi-source and heterogeneous environment of data space, there is no accurate attribute mapping, so attribute values can be used instead for attribute matching.

Instantiation: An entity record consists of a set of attributes and values, and the value of an attribute may have misspellings or null values. Null values cannot be compared. An attribute is said to instantiate an entity if it is described by an attribute whose value is not null. The entire attribute semantic mapping process is shown in Figures 4a, 4b, and 4c.

Other steps and parameters are the same as one of the specific embodiments 1 to 5.

Embodiment 7: The difference between this embodiment and one of Embodiments 1 to 6 is that the attribute mapping cluster is established in the step 4; the specific process is:

Step 41. For two attributes from different entities, calculate the similarity value of the attribute:

1) Attribute name similarity (you can use edit distance or other suitable string similarity measurement functions), denoted as S _L , obtained by comparing two normalized attribute names (according to the characteristics of the data set, you can standardize the case, expand the abbreviation is the full name);

Whether to include this part in the calculation can be decided according to the similarity of attribute names in the data set.

2) Attribute value similarity (two attributes, they may have different values in different records, such as attribute att1 has 3 values in the record set, attribute att2 has 4 values in the record set, then select att1 and The two most similar values of att2 are taken as the attribute value similarity of the two attributes), recorded as S _V , compare all the values of the two attributes, and keep the highest similarity score.

Attribute matching pairs are obtained based on attribute name similarity and attribute value similarity. From the set of attribute matching pairs, the attribute matching set is obtained through calculation. The set of attribute matching sets is called attribute matching cluster.

Attributes in an attribute match set match each other exactly. The method rejects that an attribute mapping set contains many loosely related attributes and follows the widely used no-repetition assumption ^[6] (Imen Megdiche, Oliver Teste, CassiaTrojahn. An extensible linear approach for holistic ontology matching [C].InISWC, Part I, vol.LNCS 9981.Springer, Kobe, Japan, 2016:393-410P), at this time, each attribute under a namespace (an entity, or a data source with semantic restrictions) can match at most another namespace An attribute of , setting a global 1:1 matching constraint ^[7] (Chuncheng Xiang, Baobao Chang, ZhifangSui.An ontology matching approach based on affinity-preserving random walks[C]//Proceeding IJCAI'15Proceedings of the 24th International Conference onArtificial Intelligence, 2015:1471-1477P). However, the derivation of attribute mapping sets under the global 1:1 matching constraint is not a straightforward process, since an attribute often involves more than one matching attribute pair, and simply selecting the pair with the highest matching probability estimate may lead to conflicts.

The process of getting the entire cluster of attribute maps is called global attribute map, which takes a set of attribute matching pairs as an input and returns a cluster of attribute maps. Let I be the number of namespaces, J be the number of matching attribute pairs, for two different namespaces n _s , n _t , the number of attributes under n _s , n _t is denoted as M _s , M _t , under n _s The a-th attribute and the b-th attribute under n _t are denoted as p _a ^s and p _b ^t respectively, and the overall attribute matching is optimized by maximizing the total matching probability of all matching attribute pairs, so as to meet the global 1:1 restriction:

Where σ(p _a ^s ,p _b ^t ) is the matching probability of the attribute pair p _a ^s and p _b ^t ; Θ(p _a ^s ,p _b ^t ) is the indicator function, when the attribute pair p _a ^s and p _b ^t are selected To form an attribute mapping set, the function value is 1, otherwise it is 0.

For Algorithm 1, which forms attribute matching clusters by matching attribute pairs:

Step 42: Arrange the set of attribute matching pairs in descending order of matching probability (line 2), and process them sequentially. For attribute pairs p _a and p _b , if there is no attribute mapping set in the attribute mapping cluster N that contains p _a and p _b respectively N _i , N _j , add attribute mapping set {p _a , p _b } to N (line 6-7); if attribute mapping cluster N contains attribute mapping set N _i of p _a and attribute mapping cluster N contains Attributes from the same namespace as p _b (a data source is a namespace, if you are not sure which data source the record comes from, the record itself is a namespace. Attributes from the same namespace as pb, that is, This attribute and pb belong to a record, which are two attributes included in a record.), delete the attribute pair p _a ≈ p _b (line 8-9); otherwise (for the negation of the above two ifs, the first if That is, if there are no attribute mapping sets N _i and N _j respectively containing p _a and p _b in the attribute mapping cluster N, the second if means that if there is an attribute mapping set N _i containing p _a in the attribute mapping cluster N and The attribute mapping cluster N contains attributes from the same namespace as p _b . Otherwise, the two conditions are not satisfied.), merge N _i , N _j into a larger attribute mapping set N _k , and set N Add _k to N, and delete N _i and N _j from N at the same time (line 11-12); repeat steps 4 and 2 until J is empty.

Other steps and parameters are the same as one of the specific embodiments 1 to 6.

Embodiment 8: The difference between this embodiment and one of Embodiments 1 to 7 is that the step 5 is to calculate the goodness of the attribute mapping set; the specific process is:

At this point, the attribute map cluster has been obtained. All the attributes in a mapping set form a semantic mapping with each other, and the information corresponding to them is a type of information. Zhen Lingmin et al. ^[8] (Zhen Lingmin, Yang Xiaochun, Wang Bin, etc. Entity Resolution Technology Based on Attribute Weights [J]. Computer Research and Development, 2013, 50 (Suppl.): 281-289P ZHEN Lingmin, YANGXiaochun, WANG Bin, et al. Entity Analysis Technology Based on Attribute

Weight[J].Computer Research and Development,2013,50(Suppl.):281-289P) believes that by assigning higher weights to some important attributes, some unimportant attributes can also be discarded to increase entity resolution accuracy and efficiency. Calculate the superiority of the attribute mapping set, corresponding to the above weights. When performing subsequent entity resolution, weighted calculations are performed to improve its accuracy.

Goodness of attribute mapping set (good()): An attribute mapping set, in which the attribute names may be different, but correspond to one type of information. A class of information pointed to by an attribute mapping set, and its corresponding attribute value information itself, is called the goodness of the attribute mapping set to help entity parsing.

The present invention calculates the attribute mapping set goodness through the following three aspects:

a. Discrimination:

An attribute mapping set, if its value is more changeable and the span is larger, will not help entity resolution very much. The value corresponding to the attribute map set is more useful for entity resolution if it changes in a relatively small range. Let R be the record set, and for an attribute mapping set N _i , define a discrimination function on R, denoted as discr(N _i ):

Among them, val(r, N _i ) extracts the attribute C _i recorded in the attribute mapping set (the attributes in the attribute mapping set come from different namespaces, and through calculation, it is assumed that they point to the same type of information. For this mapping set Giving weights to increase the accuracy of entity parsing. Discrimination is one of the weight calculation methods. If the attributes of this mapping set have a large difference in the corresponding attribute values, it is considered that this attribute mapping set is less helpful, and they The probability of pointing to a type of information is also low, and a lower weight should be assigned to avoid affecting the accuracy. ), norm() is to normalize the attribute values from different sources; U is the union symbol;

b. Richness:

The more values an attribute mapping set has, the more information it provides for entity resolution. That is to say, the value of an attribute recorded in this attribute mapping set is not empty, which is beneficial to entity resolution. Let R be a record set, and for an attribute mapping set N _i , define an abundance function on R, denoted as abund(N _i ):

At this time there is Θ() represents the function name, if the record r has a value in the attribute set N _i , the value of this function is 1, and if there is no value, it is 0.

Sum discrimination and richness, sort from large to small, and calculate diversity in order from large to small;

c. Diversity:

To increase diversity and reduce duplication among different attribute mapping sets, the goodness of attribute mapping sets with redundant information can be reduced. Every time an attribute mapping set is selected, it needs to be compared with the previously selected mapping set. If its information has a lot of duplication with the previous mapping set, its diversity will be reduced. Let N _i be the current mapping set, N _selected is the selected mapping cluster; for a given N _selected , the diversity of Ni is recorded as div(N _i |N _selected ):

Among them, S _i represents the record set instantiated by N _i (if for the attribute mapping set Ni, there are n attributes {att1,att2,...,attn}, if record r contains one of the attributes, and this attribute value is not empty, it is said that the attribute mapping set Ni instantiates this record r.), that is S _j represents the set of records instantiated by N _j , i.e. S _v (v _x , v _y ) is the similarity of v _x and v _y (calculated by edit distance calculation or other suitable string similarity functions), v _x is the attribute value of record r under N _i , v _y is to record the attribute value of r under N _j ; div(N _i |N _j ) indicates that after selecting N _j as the calculation mapping set for calculating record similarity, when considering whether to select N _i as the calculation mapping set for calculating record similarity, consider The repetition degree of the information of N _i and the information of N _j ;

Combining goodness in two steps. The first step combines discrimination and diversity, which reflect the static goodness of an attribute mapping set, and the descending order of the static goodness of the attribute mapping set combines diversity to update the goodness. The second step incorporates diversity to update the goodness. Let N be the sorted attribute mapping cluster. For a mapping set N _i ∈ N, the overall goodness of the mapping set N _i is good(N _i ):

comb(N _i )=α·discr(N _i )+(1-α)·abund(N _i )

Among them, α is the weight of discriminative goodness, γ is the weight of static goodness, 0≤α, γ≤1; comb(N _i ) is the goodness of static attribute set, combining discrimination and richness.

Other steps and parameters are the same as one of the specific embodiments 1 to 7.

Embodiment 9: The difference between this embodiment and one of Embodiments 1 to 8 is that in the step 6, after obtaining the superiority of each mapping set in the attribute mapping cluster, the entity resolution work is performed in the block, so as to eliminate misincorporation This block points to the data records of other entities; the calculation process is as follows:

There are record pairs r _i , r _j , at this time r _i has m attributes, r _j has n attributes, among them, there are p attributes that pass the mapping, and p≤min(m,n), the mapped attributes are { att ₁ ,att ₂ ,...,att _p }, the similarity between r _i and r _j is:

Among them, N _l is the attribute mapping set, sim _content (r _i .att _l , r _j .att _l ) is the attribute value similarity of r _i and r _j two record mapping attributes; att _l is the corresponding Attributes, this mapping set contains the mapping between r _i and a certain attribute in r _j ; compare the similarity of two records with the preset threshold λ, if it is greater than the threshold λ, it is considered a match; due to sequential processing , so, when the record pair matches, it is merged into a new record, and the new record covers the information of the original record pair; the calculation method and integration process of sim _content () are described in detail below.

The specific solution process of the sim _content (r _i .att _l , r _j .att _l ) is: the integration of record information similar to regular expressions

In the process of merging records, a method similar to regular expressions can be used to merge information, because this kind of information merging has some similarities with regular expressions, and both determine a rule for a type of information. When merging record pairs, the two values of the mapped attribute are combined into a regular expression-like.

Regular expression-like concepts

Since the values of the mapped attributes are similar, the same part can be unified and different parts can be reserved to form a regular expression, which can effectively merge matching information.

Similar to Regular Expression (StRE): Let ∑ be an alphabet and ε be an empty character. A regular expression-like StRE=S[1]S[2]...S[n]. where for any i (1≤i≤n), element S[i]={c _i,1 ,c _i,2 ,...,c _i,ni }, for j (1≤j≤n _i ), c _i,j ∈∑∪{ε}. Hereinafter, the class regular expression is simply referred to as expression.

R is an attribute-value pair, S is an expression derived from an attribute-value pair, and G is a collection of S instantiations. At this time, R∈G. For example, the class regular expression {t,n}ight can instantiate tight and night.

An expression for a property value is its value itself. However, an expression formed by combining two attribute values, or a regular expression formed by combining an attribute value and an expression requires inference calculation.

(1), calculate the similarity of regular expressions; the specific process is:

The expression of an attribute value is itself, and the similarity between two expressions can be calculated by using the edit distance; record the edit distance function as D(i, j), where i and j represent the lengths of two strings a and b respectively, using Dynamic programming to calculate the edit distance; get the edit distance similarity function sim _edit (a,b) between strings:

Computes the similarity between an expression and an attribute value, or between two expressions, similar to edit distance. Since there may be multiple elements or empty characters in the expression, the edit distance function is slightly modified. The principle of modification is as follows: (1) For multiple characters, if two expression elements contain the same character, the expression element matches ;(2) For the null character, if there is no match, the expression element containing the null character takes the null character, which does not account for the length at this time, and has no effect on the edit distance.

Through the above principles, the minimum edit distance of two expressions is obtained; the edit distance D(|S ₁ |,|S ₂ |) of expressions S ₁ and S ₂ is as follows:

S[i] represents the i-th element of the expression, S ₁ [i] represents the i-th element of the expression S ₁ , S ₂ [j] represents the j-th element of the expression S ₂ , and has initial conditions:

Among them, the function MU corresponds to principle (1), as long as the expression elements contain the same character, then match; principle (1) for multiple characters, if two expression elements contain the same character, then the expression elements match;

The function NU corresponds to the principle (2). If there is a null character in the expression element, it will be ignored directly, which will not affect the calculation of the edit distance, and the edit distance can be kept at the minimum; principle (2) for the null character, if it does not match, include The expression element of a null character takes a null character, which does not take up the length at this time and has no effect on the edit distance. The edit distance of the last two expressions is D(|S ₁ |,|S ₂ |);

At this point, the edit distance similarity between expressions is sim _edit (S ₁ ,S ₂ ):

Among them, sim _edit (S ₁ , S ₂ ) is sim _content (), S ₁ and S ₂ are two attribute values att _l in the function sim _content (r _i .att _l , r _j .att _l );

During the comparison and merging process, the recorded attribute values are regarded as expressions for calculation, and after the comparison calculation is completed, the attribute information is generated according to the expressions.

(2), the generation of regular expressions

Generate a new expression from two expressions, the principle is to introduce the least irrelevant examples. For example, for the attribute value pair cute kid and cut kind, there can be an expression S ₁ =cut{e,ε}ki{d,n}{d,ε}, and S ₁ can get 8 instances, introducing 6 irrelevant Instances, there may also be an expression S ₂ =cut{e,ε}ki{n,ε}d, 4 instances can be obtained, and 2 irrelevant instances are introduced. So S ₂ is better than S ₁ .

The present invention utilizes the edit distance matrix M of the expression to start from M[|S ₁ |,|S ₂ |] and go back to M[0,0] to obtain the expression of the expression. The generation rules are as follows :

Particularly

Among them, k is a certain position in the backtracking process, S[k] is the element of the current position, and when i=j=0, the backtracking ends; at this time, the expressions S ₁ and S ₂ are combined into a regular expression of S =...S[k]...S[0] (where k is arranged in the reverse order of the backtracking order). And when the three generated conditions are satisfied at the same time, the priority of the function MU is higher than that of the function NU. At this time, it is helpful to merge the same characters and reduce the introduction of irrelevant instances. NU(S ₂ [j]) is the function corresponding to the jth element of the expression S ₂ , NU(S ₁ [i]) is the function corresponding to the i-th element of the expression S ₁ , MU(S ₁ [i ], S ₂ [j]) is the function corresponding to the j element of expression S ₂ and the i element of expression S ₁ ;

An example is used to illustrate the similarity calculation and generation process of class regular expressions. Set attribute values attvalue ₁ = "cute kid", attvalue ₂ = "cut kind". The corresponding expressions are respectively S ₁ =cutekid and S ₂ =cut kind. At this time, the distance matrix can be obtained according to the generation formula of the regular expression, as shown in Table 1.

Table 1 Similarity calculation and generation matrix of regular expressions

Each value M[i,j] in the matrix represents the edit distance between the first i elements of S ₁ and the first j elements of S _2. Since the expression is the attribute value itself at this time, the edit distance of the expression is equal to the attribute value edit distance. The edit distance between S ₁ and S ₂ is M[7,7]=2, and the expression similarity is 1-2/7=5/7. Assuming that the similarity of the two records exceeds the threshold and they are confirmed to match, the two attribute values need to be merged. According to the expression generation rules, on the basis of the expression matrix M, start backtracking from M[7,7]. M[7,7]=M[6,6]+MU(S ₁ [6], S ₂ [6]), so backtrack to the position of M[6,6], and so on. The backtracking paths are marked in bold in Table 1. Finally, the expression cut{e,ε}ki{ε,n}d is obtained.

(3) Generation of entity information

After the block processing is completed, the attribute values in the attribute mapping set are generated through step-by-step record comparison and information merging, and finally a final expression of the attribute mapping set is obtained. The occurrence frequency of each element can be recorded in the generation process of the regular expression-like, and the character with the highest frequency of occurrence can be selected in each regular-expression-like element as the value of the element by using the regular expression with frequency. Generate a string with the highest frequency as the attribute value of this attribute map set. Such as three attribute values Mike Doe, M.Doe, Mike D., get the expression M{i:2,.:1}{k:2,ε:1}{e:2,ε:1}D{o :2,.:1}{e:2,ε:1}, and finally get the attribute value Mike Doe according to the frequency.

For this type of regular expression, for wrong characters caused by typesetting, spelling, etc., the frequency of such wrong characters must be less than that of correctly spelled characters, so taking the character with the highest frequency helps to filter noise.

Other steps and parameters are the same as those in Embodiments 1 to 8.

Adopt the following examples to verify the beneficial effects of the present invention:

Embodiment one:

This embodiment is specifically prepared according to the following steps:

Experimental dataset

The chunking experiment uses two data sets, referred to as D ₁ and D ₂ for short. D ₁ is extracted from the movie information dataset shared by DBPedia and IMDB. D ₂ is extracted from the dataset composed of two versions of DBPedia ^[9] (George Papadakis, Jonathan Svirsky, Avigdor Gal, Themis Palpanas. Comparative analysis of approximate blocking techniques for entity resolution [J]. Proceedings of the VLDB Endowment, 2016, 9 (9):684-695P). Wherein D1 has 50796 records containing ₂₂₄₀₃ entities and D2 contains ₃₃₅₄₇₉ records containing 89258 entities in total.

Experimental Evaluation Criteria

The evaluation criteria used in this paper ^[10] (Chirag Nagpal, Kyle Miller, Benedikt Boecking, Artur Dubrawski.An Entity Resolution approach to isolate instances of Human Trafficking online[J].Computer Science,2017,3(18):10-18P):

Blocking criteria: pair integrity (PC), reduction rate (RR), F value (F=2×PC×RR/(PC+RR));

Analysis criteria: precision (P), recall (R) and F value (F=2×P×R/(P+R)).

Experimental Analysis of Blocking Method

In the block process, the present invention describes four kinds of pruning methods, and the parameter α setting value of calculating record similarity is 0.6, and then pruning: for the edge-centered threshold pruning scheme (ECWP), in the data set When the upper threshold w _min is 0.6 and 0.5 respectively, the F value is the highest; for the node-centralized threshold pruning scheme (NCWP), the experiment is based on an even distribution assumption, using a unified weight threshold, the execution process and the threshold and ECWP The same; for the edge-centralized cardinal pruning scheme (ECCP), a total of k edges need to be retained during pruning, and k varies with the total number |E|. When k=0.5*|E|, the two data The F value on the set reaches the highest; for the node-centralized cardinal pruning scheme (NCCP), during the pruning process, k _ri pieces are kept for the edges connected to each node r _i , which is still based on the average distribution Assume that when k _ri =0.5*|E _ri |, the F values all reach the highest. At this time, for the four pruning methods, the threshold with the best performance is selected for comparison, as shown in Table 2.

Table 2 PC and RR values of the pruning method on the two data sets

It can be seen that the edge-centralized pruning method cuts out more useless record pairs, focuses on efficiency, and is suitable for large-scale entity resolution tasks, especially when there are fewer matching entities; while for node-centralized The pruning method keeps more matching record pairs. The edge-centralized algorithm retains edges whose weight is top-k or whose weight is greater than a preset threshold, while the node-centralized algorithm ensures that each node is connected to its most similar record, which is more suitable for applications that focus on accuracy ;The weight threshold algorithm retains record pairs with high similarity, which ensures the accuracy of the algorithm; the cardinality threshold controls the number of record pairs to be compared, and retains the top-k edge, which will affect the accuracy. But there are guarantees for method efficiency.

The overall accuracy is maintained above 97%, and the reduction rate is also maintained above 60%, indicating that pruning the block graph and discarding a certain amount of useless record pairs guarantees the efficiency of the algorithm to a certain extent.

Experimental Analysis of Attribute Mapping and Expression

When calculating the mapping goodness of attribute clusters, the experiment sets the parameter values as α=0.5 and λ=0.4. The shortest distance method was selected for comparison of results. The results of two stages are evaluated: one is the result of entity resolution, but the generation of real entity information.

For the results of entity parsing, the accuracy and recall of the two methods are shown in Figure 5a and 5b. It can be seen that as the threshold increases, the recall rate decreases and the precision rate increases significantly. On the two data sets, when the threshold value is 0.6, the method in this paper has achieved a higher F value. Methods On the two data sets, the recall rate of the two methods is not much different, while the processing method based on attribute mapping and regular expression in this paper makes the recall rate slightly better. The accuracy rate is significantly better than the method based on the shortest distance, and the accuracy rate is higher. It shows that the method in this paper is more suitable for the recording characteristics of the data space. The shortest distance is more dependent on the semantics of the data, and it can play a greater role in an environment with strong semantics.

For the generation phase of entity information, most of the entity resolution work focuses on the analysis results, but not much attention is paid to the generation of entity information and merging and entities. Using the data set D ₁ , the number of generated entities and the actual number of entities on it are shown in Figure 6 . Entity generation is performed using the optimal threshold generated during entity resolution. It can be seen from the figure that the analytical results of this paper are more accurate, and the number of generated entities is closer to the number of real entities. But its accuracy is still significantly higher than the shortest distance method.

The present invention can also have other various embodiments, without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations are all Should belong to the scope of protection of the appended claims of the present invention.

Claims (9)

1. the entity resolution method in data-oriented space, it is characterised in that: the method detailed process are as follows:

Step 1: building record figure；

Step 2: simplifying record figure using pruning method；

Step 3: carrying out piecemeal processing to the record figure after cutting；

Step 4: establishing attribute mapping cluster；

Step 5: the goodness of computation attribute mapping ensemblen；

Step 6: obtain attribute mapping cluster in each mapping ensemblen goodness after, entity resolution is carried out in block.

2. the entity resolution method in data-oriented space according to claim 1, it is characterised in that: constructed in the step 1 Record figure；Detailed process are as follows:

Similarity between two step 1 one, calculating records；

Step 1 two constructs record figure according to the record and similarity of data space.

3. the entity resolution method in data-oriented space according to claim 2, it is characterised in that: the step 1 one is fallen into a trap Calculate the similarity between two records；Detailed process are as follows:

Step 1 one by one, calculate label similarity:

Record is switched into tag set by label transfer function tag (), the label similarity of two records is calculated, is denoted as sim_tag(r_i,r_j):

Wherein, T (r_i) it is that r will be recorded by label transfer function_iThe standardization tally set being converted into；T(r_j) it is to be turned by label Exchange the letters number will record r_jThe standardization tally set being converted into；

Step 1 one or two, calculated relationship similarity:

Two comprehensive similarities recorded in possessed all relationships are incorporated, sim is denoted as_rel(r_i,r_j):

Wherein, Nbr (r_i) indicate and record r_iThere are the set of records ends of connection, Nbr (r in rel relationship_j) indicate and record r_j? There is the set of records ends of connection in rel relationship, REL is indicated in record r₁、r₂All record set of relationship of upper appearance；

Step 1 one or three integrates label similarity and relationship similarity, obtains comprehensive similarity sim (r_i,r_j):

sim(r_i,r_j)=α sim_tag(r_i,r_j)+(1-α)·sim_rel(r_i,r_j)

Wherein, α indicates the weight of label similarity.

4. the entity resolution method in data-oriented space according to claim 3, it is characterised in that: root in the step 1 two According to record and similarity the building record figure of data space；Detailed process are as follows:

The set of records ends R of a given data space, constructs a non-directed graph G=(R, E), referred to as record figure；

Wherein R is set of records ends, represents the record in data space；E is side collection, represents and remembers there are a line between two records The similarity of record pair.

5. the entity resolution method in data-oriented space according to claim 4, it is characterised in that: used in the step 2 Pruning method simplifies record figure；Detailed process are as follows:

Pruning method use while centralization radix beta pruning, node centralization radix beta pruning, while centralization threshold value beta pruning or One of threshold value beta pruning of node centralization；

(1) the radix beta pruning of side centralization:

Global radix threshold value k, which specifies record figure, will retain the sum on side, retain the side of k maximum weight；

(2) the radix beta pruning of node centralization:

For each node r_i, retain link node r_iTop-k weight side；

(3) the threshold value beta pruning of side centralization:

Beta pruning is carried out in global scope using weight threshold, chooses minimum side right w_min, weight is lower than by all sides in traversing graph w_minEdge contract；

(4) the threshold value beta pruning of node centralization:

One unified threshold value, the weight beta pruning scheme phase of beta pruning process and side centralization are used to all nodes of record figure Together.

6. the entity resolution method in data-oriented space according to claim 5, it is characterised in that: to cutting in the step 3 Record figure after change carries out piecemeal processing；Detailed process are as follows:

Figure after beta pruning is G={ R, E }, and R is set of records ends, and E is side collection；

Appoint and takes a record r_i, create a block b_iAnd by r_iIt is placed in b_iIn, if r_iNode r in point region_jWith b_iIn all knots Point has Bian Xianglian, then by r_jIt is placed in b_iIn, and delete r_jWith b_iIn all nodes connected side, repeat this operation until time Go through r_iAll neighbor nodes；At this point, if b_iIn node become an isolated node in figure, it is boundless to be attached thereto, then from This node is deleted in figure；Step 3 is repeated until figure G is sky；At this point, piecemeal work is completed, set of blocks B={ b is obtained₁, b₂,...,b_|B|}。

7. the entity resolution method in data-oriented space according to claim 6, it is characterised in that: established in the step 4 Attribute maps cluster；Detailed process are as follows:

Step 4 one, for two attributes from different entities, the similar value of computation attribute:

The similar value of attribute, is denoted as S_V, all values of two attributes are compared, retains highest similarity score, obtains attributes match It is right；

If I is NameSpace number, J is matched attributes match logarithm, the NameSpace n different for two_s、n_t, n_s、n_tUnder Attribute number is denoted as M_s、M_t, n_sUnder a-th of attribute, n_tUnder b-th of attribute be denoted as p respectively_a ^s、p_b ^t, by maximizing all Properties pair of total matching probability matches to optimize integrity attribute, meets global 1:1 limitation with this:

Wherein σ (p_a ^s,p_b ^t) it is attribute to p_a ^sAnd p_b ^tMatching probability；Θ(p_a ^s,p_b ^t) it is indicator function, when selection attribute pair p_a ^s、p_b ^tCome when forming an attribute mapping ensemblen, otherwise functional value 1 is 0；

Step 4 two is arranged attributes match by matching probability descending set, is sequentially handled, for attribute to p_a、p_bIf P is not separately included in attribute mapping cluster N_a、p_bAttribute mapping ensemblen N_i、N_j, then attribute mapping ensemblen { p is added into N_a、p_b}； If it includes p in cluster N that attribute, which maps,_aAttribute mapping ensemblen N_iAnd comprising coming from and p in attribute mapping cluster N_bSame NameSpace Attribute, delete p_a≈p_bThis attribute pair；Otherwise, merge N_i、N_jThe attribute mapping ensemblen N bigger as one_k, by N_kIt is added to In N, while N is deleted from N_i、N_j；Step 4 two is repeated until J is sky.

8. the entity resolution method in data-oriented space according to claim 7, it is characterised in that: calculated in the step 5 The goodness of attribute mapping ensemblen；Detailed process are as follows:

Pass through following three aspects computation attribute mapping ensemblen goodness:

A, discrimination property:

Set of records ends is let R be, for an attribute mapping ensemblen N_i, a discrimination goodness function is defined on R, is denoted as discr (N_i):

Wherein, val (r, N_i) be extracted and be recorded in attribute C in attribute mapping ensemblen_iOn attribute value, norm () be to separate sources Attribute value standardize；U is union symbol；

B, rich:

Set of records ends is let R be, for an attribute mapping ensemblen N_i, an abundant goodness function is defined on R, is denoted as abund (N_i):

Have at this timeΘ () representative function name, if record r is in property set N_iOn have value, Then this functional value is 1, and no value is then 0；

Distinguish property and it is rich sum up, be ranked up from big to small, by from big to small sequence calculate diversity；

C, diversity:

If N_iFor current mapping ensemblen, N_selectedFor the mapping cluster having been selected；For giving N_selected, N_iDiversity note For div (N_i|N_selected):

Wherein, S_iIt indicates by attribute mapping ensemblen N_iThe set of records ends of instantiation, i.e.,S_jIndicate by Attribute mapping ensemblen N_jThe set of records ends of instantiation, i.e.,S_v(v_x,v_y) it is v_x、v_ySimilarity, vx For attribute value of the record r at Ni, v_yIt is record r in N_jUnder attribute value；div(N_i|N_j) indicate to choose N_jIt is recorded as calculating After the calculating mapping ensemblen of similarity, whether N is chosen considering_iWhen recording similar mapping ensemblen as calculating, N is considered_iInformation and N_jInformation multiplicity；

If N is the attribute mapping cluster of sequence, for a mapping ensemblen N_i∈ N, mapping ensemblen N_iWhole goodness is good (N_i):

comb(N_i)=α discr (N_i)+(1-α)·abund(N_i)

Wherein, α is the weight of discrimination property goodness, and γ is the weight of static goodness, 0≤alpha, gamma≤1；comb(N_i) it is static belong to Property collection goodness, incorporate discrimination property and it is rich.

9. the entity resolution method in data-oriented space according to claim 8, it is characterised in that: obtained in the step 6 In attribute mapping cluster after the goodness of each mapping ensemblen, entity resolution is carried out in block；Calculating process is as follows:

There is record to r_i、r_j, r at this time_iThere are m attribute, r_jThere is n attribute, wherein have p, and p≤min by the attribute of mapping (m, n), the attribute of mapping are { att₁,att₂,...,att_p, r_iWith r_jSimilarity are as follows:

Wherein, N_lFor attribute mapping ensemblen, sim_content(r_i.att_l,r_j.att_l) it is r_i、r_jThe attribute value of two record attribute mappings Similarity；att_lFor attribute corresponding to an attribute mapping ensemblen；

The similarity that two record is compared with preset threshold value λ, if more than threshold value λ, then it is assumed that matching；Due to sequentially Processing, so, when record is to matching, a new record is merged into, new record covers the information of former record pair.