TW201905733A - Multi-source data fusion method and device - Google Patents

Abstract

Embodiments of the specification provide a multisource data fusion method and apparatus. The multisource data fusion method can be used for obtaining data belonging to a same entity from a data set; for any entity, at least one associated attribute of an entity attribute of the entity can be obtained; an attribute similarity between associated attributes of two entities can be obtained; and if the attribute similarity is greater than a similarity threshold, it can be determined that the two entities are a same entity, and entity attributes of the two entities are associated with the same entity.

Description

Multi-source data fusion method and device

This disclosure relates to the field of network technology, in particular to a multi-source data fusion method and device.

When data analysis is performed on an entity, a large amount of attribute data describing the entity needs to be obtained. These attribute data can include multiple sources, for example, from information input by the user himself, or from web crawler collection, or from multiple sources Different channel providers. Data from different sources can have different data standards, and the description of the same entity may be different. For example, assume that two data sources are actually descriptions of the same entity, but the description of the name of the entity is described in the descriptions of the two sources. Different or different address descriptions. When analyzing data for an entity, you can associate a large amount of attribute data describing the same entity to the entity, that is, perform multi-source data fusion of the entity, and then analyze the entity based on these multi-source data. There is a need to provide a solution that can more accurately merge multiple sources of the same entity.

In view of this, the embodiments of the present specification provide a multi-source data fusion method and device to accurately and quickly perform multi-source data fusion. Specifically, the present disclosure is implemented through the following technical solutions: The first aspect provides a multi-source data fusion method, which is used to obtain data belonging to the same entity from a data set, including multiple entities Data, each entity's data includes at least one entity attribute; the method includes: for any entity, obtain at least one associated attribute of each entity attribute; obtain the attribute similarity of the associated attributes of the two entities; If the attribute similarity is greater than the similarity threshold, it is determined that the two entities are the same entity, and the entity attributes of the two entities are related to the same entity. In a second aspect, a multi-source data fusion device is provided. The device is used to obtain data belonging to the same entity from a data set. The data set includes data belonging to multiple entities, and the data of each entity includes at least one entity attribute. The device includes: attribute acquisition module for acquiring any at least one associated attribute of each entity attribute for any entity; similarity calculation module for acquiring attribute similarity of the associated attributes of two entities Association processing module, used to determine that the two entities are the same entity if the attribute similarity is greater than the similarity threshold, and associate the entity attributes of the two entities to the same entity. The multi-source data fusion method and device provided in the embodiments of this specification construct a similarity calculation method based on the associated attributes of entity attributes to measure the similarity relationship between two entities, so that the differences in entity attribute descriptions will not affect the same The identification of entities can quickly and accurately complete the acquisition of multi-source data of the same entity; there is an effective measurement method for multi-source data with different data formats, which can realize the identification and fusion of the same entity data, so that the entity The information is more complete.

In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification. The technical solution is described clearly and completely. Obviously, the described embodiments are only a part of the embodiments, but not all the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by a person of ordinary skill in the art without making progressive labor shall fall within the scope of protection of the present disclosure. In data analysis, it is often about collecting data from the same entity from multiple channels, and conducting a more accurate business analysis of the entity based on these data. In actual implementation, even if the data from the above multiple sources all describe the characteristics of the same entity, they may be described in different ways. For example, for the same physical store, the name in source L1 is m, and the name in source L2 is n. In fact, the names m and n refer to the same store, but the literal description is different; for example, the same entity above For stores, the description of the store address in source L1 and source L2 may also be different. When performing multi-source data fusion, it is necessary to obtain data belonging to the same entity and associate these data with the same entity in order to analyze the entity based on these data. However, the data from different sources mentioned in the above example may not be related to the same entity due to the inconsistent description of the entity. The multi-source data fusion method provided by one or more embodiments of this specification will be used to solve this problem, so that even if the entity description methods are different, the data of the same entity can be associated. In the multi-source data fusion method of one or more embodiments of this specification, "entity attributes" and "associated attributes" will be referred to. Among them, the entity attributes are the direct attributes of an entity collected from various sources, and the related attributes may be other attributes related to the entity attributes. The following example: For example, an entity attribute can be the address of a physical store. The "latitude and longitude coordinates" corresponding to the "address" are the associated attributes of the "address", or the "information of the province and city where the address belongs" is also the associated attribute of the "address". For another example, an entity attribute may be a contact number of a physical store. And the "province of the number corresponding to the contact phone" is the associated attribute of the "contact phone"; or, "the name of the common consignee corresponding to the contact phone" is also the associated attribute of the "contact phone"; or, "the contact phone The corresponding contact mailbox is also an associated attribute of "contact phone". There are many ways to obtain related attributes, which can be the entity attributes of other entities, or they can be obtained from the big data information collected by history. For example, the collected and stored historical transaction data can be used to obtain the common use of a contact phone. Information such as the address of the consignee or the name of the commonly used consignee. The number of associated attributes corresponding to one entity attribute may be at least one. For quick and convenient access to related attributes in subsequent processing, a graph database can be established in advance. FIG. 1 illustrates a part of a graph database. The graph database may include multiple attribute nodes, such as attribute node 11, attribute node 12, attribute node 13, and attribute node 14 in FIG. In addition, the attribute nodes that have an association relationship are connected by an edge. For example, the attribute node 11 and the attribute node 12 are connected by an edge, indicating that the province to which the number belongs is related to the number; between the attribute nodes that have no association relationship can be Not connected by edges. The edges used to connect attribute nodes in the graph database can help to quickly find the attribute nodes related to a certain attribute node, and be used in the search for related attributes. For example, assuming that the attribute node 11 is an entity attribute, the attributes corresponding to at least one attribute node connected to the attribute node 11 can be determined as the associated attributes of the contact phone according to the node connection relationship, for example, the number of the province, The common consignee name corresponding to the number, etc., are all related attributes of the contact phone. The establishment of the graph database can be constructed by applying the entity attributes of other entities or the big data information collected by history. On the basis of the above description of "entity attributes" and "associated attributes", the multi-source data fusion method of one or more embodiments of this specification will be described below with reference to FIG. 2, in which the method will be based on the "association between different entities" The calculation of similarity of "attributes" to measure the similarity between entities. As mentioned earlier, different entities may be described differently (different entities here are only used to represent different data sources, and may actually be the same entity). This difference is usually the description of the entity's "entity attributes", and The entity similarity judgment in the method of this example is not based on entity attributes but on associated attributes, so that different descriptions of entity attributes will not lead to misjudgment of entity differences, and entity similarity usually has a higher degree of similarity in associated attributes. In step 202, the data in the data set is processed in a unified data format. For multi-source heterogeneous data sets, standardization and structured preprocessing can be carried out in order to standardize the description attributes of entities. Due to different data sources, the description of the information may be different, and the format standards of the data may also be different. For example, English letters, separators, simplified and traditional characters, etc., need to be processed uniformly to improve the quality of the data. For entity information, a corresponding data model can be constructed. For example, for a store, the standard attribute range of the store can be determined, such as phone, business license, address, etc., and extract as much valuable information as possible. In step 204, the data of different entities that meet the predetermined conditions are divided into the same data set. In order to avoid the expansion of the amount of data calculation caused by the Cartesian product caused by the subsequent similarity calculation, a preliminary classification of the data set can be performed to gather the data with higher likelihood of similarity. This process can be called data bucketing. For example, entities with unique characteristics that are completely consistent can be directly determined to be the same entity, such as store name, business license number, etc. The remaining data that are not directly determined to be consistent can be preliminarily classified by a strong rule classification bucket. For example, the data of different entities that meet the predetermined conditions can be divided into the same data set, for example, the data in the same data set The store entity is in the same city, the landline number area is the same, or the store service type (food, service, shopping) is the same. The multiple predetermined conditions of the strong rule classification bucket can be implemented in batches. For example, in a specific implementation, a data set can be divided according to the city where the store is located, and the processing of steps 206 to 210 can be performed on the data set to extract The data of the same entity; and then the remaining data of the data set can be consistent with the landline number area to obtain a sub-data set, and perform the processing of steps 206 to 210 again on the sub-data set to extract the data of the same entity. In step 206, for any entity, at least one associated attribute of each entity attribute is obtained. In this step, at least one associated attribute related to the attribute of the entity can be found in the graph database of the example in FIG. 1 according to the connection relationship between the attribute nodes. For example, you can find an entity attribute in the graph database first, and the entity attribute is one of the attribute nodes in the graph database, and then use the attribute of at least one attribute node connected to the attribute edge of the entity as its associated attribute. In step 208, the attribute similarity of the associated attributes of the two entities is obtained. For example, suppose entity A has attributes a ₀ , a ₁ … an _n , and entity B has attributes b ₀ , b ₁ … b _n . Generally, a ₀ and b ₀ can be the same attribute, but the values are different, for example, both are mobile phone numbers, but the mobile phone numbers are different. Similarly, a ₁ and b _{1 have} the same attributes. For example, both are store addresses, but the specific address information is different. In this example, attribute pairs like “a ₀ and b ₀ ” and “a ₁ and b ₁ ” may be referred to as “corresponding entity attributes” of the two entities, that is, referring to the same entity attributes. Taking one of the corresponding entity attributes as an example, "a ₀ and b ₀ ". Assume that the associated attributes of attribute a ₀ include: α ₀ , α ₁ , ... α _n ; the associated attributes of attribute b ₀ include: β ₀ , β ₁ ,… Β _n . Similarly, α ₀ and β ₀ can be the same attribute but have different values. For example, both are mailboxes associated with mobile phone numbers, but the mailboxes are different. In this example, attribute pairs like “α ₀ and β ₀ ” can be called “corresponding association attributes”, that is, referring to the same association attribute, and “α ₀ and β ₀ ” are “corresponding entity attributes” “a ₀ and b ₀ "One of the" corresponding associated attributes ". Based on the above concepts of "corresponding entity attributes" and "corresponding associated attributes", the following explains how to calculate the attribute similarity of two entities. The attribute similarity between any two corresponding associated attributes can be calculated separately, and the calculation formula can be as shown in the following formula (1). α _i and β _i are two corresponding correlation attributes, when α _{i is} not equal to β _i , the similarity is 0, when α _i = β _i , the similarity . Where e is the natural base, N is the number of other attribute values associated with the corresponding associated attribute, for example, a ₀ , b ₀ are the mobile phone number, α ₀ , and β _{0 are} the mailboxes associated with the mobile phone number, when α ₀ = β ₀ When it is found that the mailbox has 4 mobile phone numbers related to it, N = 4. θ is the concentration adjustment parameter. For hotspot data, such as the city information corresponding to the mobile phone, a city may correspond to many related mobile phone numbers, then the value of θ can be set larger, otherwise, if the possibility of repeated data such as mailbox is not high, θ The value can be set smaller. For any corresponding associated attribute of any corresponding entity attribute, it can be calculated according to formula (1). For example, for one of the corresponding entity attributes "a ₀ and b ₀ ", the attribute similarity of α ₀ and β ₀ can be calculated, the attribute similarity of α ₁ and β ₁ can be calculated, and so on. Next, the attribute similarity of the two entities can be obtained according to the attribute similarity between the corresponding associated attributes and the attribute weight of the corresponding entity attribute. For example, as shown in formula (2), the instance is the similarity between the attributes of entity A and entity B Calculation. Among them, m is the number of effective attributes of A and B, that is, the corresponding attributes have values. In the above example, entity A has attributes a ₀ , a ₁ ... an _{n respectively} , and entity B has attributes b ₀ , b ₁ ... b _n . Assuming that at least one of a ₁ and b ₁ does not obtain an attribute value, then this attribute is an invalid attribute. If both a ₀ and b ₀ can obtain the attribute value, it is a valid attribute, and at most n valid attributes. For a pair of "corresponding entity attributes" (for example, a ₀ and b ₀ ), n is the number of effective corresponding associated attributes of the corresponding entity attribute. Similarly, suppose the associated attributes of attribute a ₀ include: α ₀ , α ₁ ,… α _n ; the associated attributes of attribute b ₀ include: β ₀ , β ₁ , ... β _n , and there are at most n valid corresponding associated attributes. Is "corresponding associated attribute" (for example, " ") The associated" corresponding entity attributes "(for example, a ₀ and b ₀ ) attribute weights can be set higher for important corresponding entity attributes and lower for non-important corresponding entity attributes. Represents the average value of the attribute similarity of the "corresponding associated attributes" associated with a "corresponding entity attribute". In step 210, if the attribute similarity is greater than the similarity threshold, it is determined that the two entities are the same entity, and the entity attributes of the two entities are related to the same entity. For example, when the value of sim (A, B) is greater than the threshold σ, the two can be considered as the same entity. After recognizing that two are the same entity, you can associate the entity attributes of the two entities to the same entity. The multi-source data fusion method of this example builds a similarity calculation method based on the associated attributes of entity attributes to measure the similarity relationship between two entities, so that the difference in entity attribute descriptions does not affect the identification of the same entity. Quickly and accurately complete the acquisition of multi-source data of the same entity; there is an effective measurement method for multi-source data with different data formats, which can realize the identification and fusion of the same entity data, thereby making the entity data more complete. The execution order of each step in the flow shown in FIG. 2 above is not limited to the order in the flowchart. In addition, the description of each step can be implemented in the form of software, hardware, or a combination thereof. For example, those skilled in the art can implement it in the form of software code, which can be a computer capable of implementing the logical functions corresponding to the steps. Execute instructions. When implemented in software, the executable instructions can be stored in memory and executed by the processor in the device. For example, corresponding to the above method, one or more embodiments of this specification simultaneously provide a data processing device, which may include a processor, a memory, and computer instructions stored on the memory and executable on the processor. The processor executes the instructions to implement the following steps: for any entity, obtain at least one associated attribute of each entity attribute; obtain the attribute similarity of the associated attributes of the two entities; if the attribute If the similarity is greater than the similarity threshold, it is determined that the two entities are the same entity, and the entity attributes of the two entities are related to the same entity. One or more embodiments of this specification also provide a multi-source data fusion device, which can be applied to implement a multi-source data fusion method of one or more embodiments of this specification. As shown in FIG. 3, the device may include: an attribute acquisition module 31, a similarity calculation module 32, and an association processing module 33. The attribute obtaining module 31 is used to obtain at least one associated attribute of each entity attribute for any entity; the similarity calculation module 32 is used to obtain the attribute similarity of the associated attributes of the two entities; association processing The module 33 is configured to determine that the two entities are the same entity if the attribute similarity is greater than the similarity threshold, and associate the entity attributes of the two entities to the same entity. In one example, the attribute acquisition module 31 is specifically used to: obtain the entity attribute from a pre-built graph database, the entity attribute is one of the attribute nodes in the graph database, the graph data The library includes multiple attribute nodes, and the attribute nodes that have an association relationship are connected by edges; the attributes corresponding to at least one attribute node connected to the entity attribute edges are determined as the associated attributes of the entity attributes. In one example, the similarity calculation module 32 is specifically used to: for the corresponding entity attributes of two entities, determine the corresponding associated attributes of the corresponding entity attributes; respectively calculate the attribute similarity between any two corresponding associated attributes Obtaining the attribute similarity of the two entities according to the attribute similarity between the corresponding associated attributes and the attribute weight of the corresponding entity attribute. In one example, as shown in FIG. 4, the device may further include: a data classification module 34 for classifying data of different entities that meet predetermined conditions into the same data set. In one example, as shown in FIG. 4, the device may further include: a data preprocessing module 35, which is used to unify the data in the data set. The device or module explained in the above embodiments may be realized by a computer chip or entity, or by a product with a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an e-mail sending and receiving device, a game A console, tablet, wearable device, or any combination of these devices. For the convenience of description, when describing the above device, the functions are divided into various modules and described separately. Of course, when implementing one or more embodiments of this specification, the functions of each module may be implemented in one or more software and / or hardware. Those skilled in the art should understand that one or more embodiments of this specification may be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of this specification can be used on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code The form of the implementation of computer program products. It should also be noted that the terms "include", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also Other elements not explicitly listed, or include elements inherent to this process, method, commodity, or equipment. Without more restrictions, the element defined by the sentence "include one ..." does not exclude that there are other identical elements in the process, method, commodity, or equipment that includes the element. One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices. One or more embodiments of this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiment of the data processing device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method embodiment. The above is only one or more embodiments of this specification and is not intended to limit this disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this disclosure should be included in this Expose the scope of protection.

11‧‧‧Attribute node

12‧‧‧Attribute node

13‧‧‧Attribute node

14‧‧‧Attribute node

202‧‧‧Step

204‧‧‧Step

206‧‧‧Step

208‧‧‧Step

210‧‧‧Step

31‧‧‧Attribute acquisition module

32‧‧‧Similarity calculation module

33‧‧‧Related processing module

34‧‧‧Data classification module

35‧‧‧Data preprocessing module

In order to more clearly explain one or more embodiments of the specification or the technical solutions in the prior art, the following will briefly introduce the drawings required in the description of the embodiments or the prior art. Obviously, the appended The drawings are only some of the embodiments described in one or more embodiments of this specification. For those of ordinary skill in the art, without paying progressive labor, other drawings can also be obtained based on these drawings. 1 is a partial schematic diagram of a graph database provided by one or more embodiments of the specification; FIG. 2 is a schematic flowchart of a multi-source data fusion method provided by one or more embodiments of the specification; FIG. 3 is one or more of the specification FIG. 4 is a schematic structural diagram of a multi-source data fusion device provided by one or more embodiments of this specification.

Claims (11)

A multi-source data fusion method, the method is used to obtain data belonging to the same entity from a data set, the data set includes data belonging to multiple entities, and the data of each entity includes at least one entity attribute; the method includes: For any entity, obtain at least one associated attribute of each entity attribute; Obtain the attribute similarity of the associated attributes of the two entities; If the attribute similarity is greater than the similarity threshold, determine the two The entity is the same entity, and the entity attributes of the two entities are related to the same entity. According to the method described in item 1 of the patent application scope, the acquiring associated attributes of each entity attribute includes: acquiring the entity attribute from a pre-established graph database, the entity attribute is in the graph database One of the attribute nodes in the graph database includes a plurality of attribute nodes, and the attribute nodes that have an association relationship are connected by edges; The attribute corresponding to at least one attribute node connected to the entity attribute edge is determined to be The associated attributes of the entity attributes. According to the method described in item 1 of the patent application scope, the obtaining the attribute similarity of the associated attributes of the two entities includes: For the corresponding entity attributes of the two entities, determining the corresponding associated attributes of the corresponding entity attributes; Calculate the attribute similarity between any two corresponding associated attributes separately; According to the attribute similarity between the corresponding associated attributes and the attribute weights of the corresponding entity attributes, obtain the attribute similarity of the two entities . According to the method described in item 1 of the patent application scope, the method further includes: dividing the data of different entities that meet predetermined conditions into the same data set. According to the method described in item 1 of the patent application scope, the method further includes: unifying the data in the data set in a data format. A multi-source data fusion device, the device is used to obtain data belonging to the same entity from a data set, the data set includes data belonging to multiple entities, and the data of each entity includes at least one entity attribute; the device includes: Attribute acquisition module, used to obtain at least one associated attribute of each entity attribute for any entity; Similarity calculation module, used to obtain attribute similarity of the associated attributes of two entities; Association processing module , Used to determine that the two entities are the same entity if the attribute similarity is greater than the similarity threshold, and associate the entity attributes of the two entities to the same entity. According to the device described in item 6 of the patent application scope, the attribute acquisition module is specifically configured to: acquire the entity attribute from a pre-established graph database, and the entity attribute is one of the graph databases An attribute node, the graph database includes a plurality of attribute nodes, and the attribute nodes that have an association relationship are connected by edges; the attribute corresponding to at least one attribute node connected to the entity attribute edge is determined as the entity The associated attribute of the attribute. According to the device described in item 6 of the patent application scope, the similarity calculation module is specifically used to: for the corresponding entity attributes of two entities, determine the corresponding associated attributes of the corresponding entity attributes; calculate any two correspondences respectively Attribute similarity between related attributes; according to the attribute similarity between the corresponding related attributes and the attribute weight of the corresponding entity attribute, the attribute similarity of the two entities is obtained. According to the device described in item 6 of the scope of the patent application, the device further includes: a data classification module for dividing data of different entities that meet predetermined conditions into the same data set. According to the device described in item 6 of the patent application scope, the device further includes: a data preprocessing module, which is used for unifying the data format of the data in the data set. A data processing device, the device includes a memory, a processor, and computer instructions stored on the memory and executable on the processor. The processor implements the following steps when executing the instructions: For any entity, obtain separately At least one associated attribute of each entity attribute; obtain the attribute similarity of the associated attributes of the two entities; if the attribute similarity is greater than the similarity threshold, determine that the two entities are the same entity, The entity attributes of the two entities are related to the same entity.