CN113760995A

CN113760995A - Entity linking method, system, equipment and storage medium

Info

Publication number: CN113760995A
Application number: CN202111054119.3A
Authority: CN
Inventors: 黄艳香
Original assignee: Shanghai Minglue Artificial Intelligence Group Co Ltd
Current assignee: Shanghai Minglue Artificial Intelligence Group Co Ltd
Priority date: 2021-09-09
Filing date: 2021-09-09
Publication date: 2021-12-07

Abstract

The application discloses an entity linking method, which comprises the following steps: and attribute discrimination calculation: calculating attribute discrimination of at least one attribute of a plurality of data source entities based on an attribute discrimination algorithm; a linking step: and judging whether the plurality of data source entities are matched or not based on the attribute zone division of the data source entities, and completing the linkage of the plurality of data source entities. The method of the invention provides attribute zone indexing, provides a quantitative calculation mode for the importance degree of the attribute in the entity link, assists the entity link algorithm to improve the accuracy and reduces the design cost of the manual algorithm.

Description

Entity linking method, system, equipment and storage medium

Technical Field

The present application relates to the field of data processing, and in particular, to an entity linking method, system, computer device, and computer-readable storage medium.

Background

Currently, structured and semi-structured data are widely available in various enterprises, but due to changes of managers, scattered physical layout, system autonomy and the like, the data have the problems of complicated sources (different types of relational databases, data of different departments and the like), structural isomerism (SQL, NoSQL databases, text files, Hive big data and the like) and the like. The integration and fusion of multi-source heterogeneous data are necessary basic conditions for enterprises to make upper-layer application, and entity linkage is a very important ring in the process. For example, the data source 1 has "zhang san, male, age 30, science and technology", the data source 2 has "zhang san, male, age 28, second hand", it is determined whether two data sources "zhang san" are the same person, and the entity data representing the same zhang san "in all data sources are linked, so that the task object of entity linking is realized. Entities are often described by a plurality of attribute-value pairs, such as "name: zhang III; sex: "male" describes a male entity named zhang san, wherein "name" and "gender" are attribute names, specifically, in a row-column database, attributes may be stored in a column name form, in a key-value database, attributes may be key names, and for convenience of description, in the following, we collectively refer to data representing attributes in various data structures by attribute-values.

The existing entity linking techniques can be generally classified into the following categories:

1) entity linking based on business rules: the domain expert proposes a matching strategy based on experience and observation of the data.

2) Feature vectors plus traditional machine learning: the classification model is typically trained using labeling data by constructing various similarity-based feature vectors, and then predicting the pair of entities to be matched based on the model.

3) Word embedding (word embedding) method plus deep learning: the feature expression of the words is generally learned through a word embedding method, a deep learning classification model is trained based on similarity calculation of the optimized entity pair by using labeled data, and then the entity pair to be matched is predicted based on the model.

4) Method based on natural language model: and splicing each field and data of the entity, converting each entity into a sentence, training a model based on the labeled data, and judging whether the two entity pairs are matched in a mode of judging whether the semantics of the sentences are similar by using a natural language model.

At present, aiming at the following bottlenecks in the related art, no effective solution is proposed:

in the existing method, although the accuracy of deep learning and a method based on a natural language model is higher, the problems of high algorithm complexity, more required resources, long operation time and the like exist. Therefore, the entity link based on the rules and the traditional machine learning method are still widely applied, and the main problems are that the accuracy is not high enough, the universality is not strong, and the required labor cost is high.

Therefore, based on the problems in the prior art, the method provided by the invention mainly solves the problems that the existing entity linking method based on rules or machine learning is not high in accuracy and high in required labor cost. By providing the algorithm for quantitatively calculating the attribute discrimination, more data insight information is provided for the design of the entity link algorithm, the entity link accuracy is improved in an auxiliary manner, and the manual analysis design cost is reduced.

Disclosure of Invention

The embodiment of the application provides random grouping storage of user privacy data, and a recommendation implementation mode which does not depend on user information storage of a recommendation server side can ensure that the user privacy data are protected, complete storage cannot be realized at a client side, and retention cannot be realized at the server side.

In a first aspect, an embodiment of the present application provides an entity linking method, including:

and attribute discrimination calculation: calculating attribute discrimination of at least one attribute of a plurality of data source entities based on an attribute discrimination algorithm;

a linking step: and judging whether the plurality of data source entities are matched or not based on the attribute zone division of the data source entities, and completing the linkage of the plurality of data source entities.

In some embodiments, the attribute zone division algorithm is:

property region scale v:

the data set comprises a plurality of data source entities R, the data source entities R have attributes A, n is the number of the data source entities R after entity linking, and m is the number of all non-repeated values of the data source entities R on the attributes A.

In some embodiments, the linking step comprises:

and judging whether the multiple data source entities are matched or not based on the attribute similarity and a preset threshold value, and if the matching is successful, linking the multiple data source entities.

In some embodiments, the linking step further comprises:

and screening the importance degree of the attributes based on the attribute division, and selecting important attributes to link the plurality of data source entities.

In a second aspect, an embodiment of the present application provides an entity linking system, which employs the entity linking method described above, and includes:

attribute discrimination calculation module: calculating attribute discrimination of at least one attribute of a plurality of data source entities based on an attribute discrimination algorithm;

a linking module: and judging whether the plurality of data source entities are matched or not based on the attribute zone division of the data source entities, and completing the linkage of the plurality of data source entities.

In some embodiments, the attribute zone division algorithm is:

property region scale v:

In some embodiments, the linking module comprises:

In some embodiments, the linking module further comprises:

In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the entity linking method according to the first aspect when executing the computer program.

In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the entity linking method according to the first aspect.

Compared with the prior art, the invention provides the calculation of the attribute zone division and the use of the attribute zone division in the entity link;

1) the invention provides a concept of attribute discrimination, and provides a quantitative calculation mode for the importance degree of attributes in entity links;

2) the invention provides quantitative insight to attributes, assists in entity link algorithm to improve accuracy or reduce design cost of manual algorithm.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 is a flow chart of a method for entity linking according to the present invention;

FIG. 2 is a schematic diagram of a physical link system according to the present invention;

fig. 3 is a hardware structure diagram of a computer device according to an embodiment of the present application.

In the above figures:

100 physical linking system

10 attribute discrimination calculation module, 20 linking module

81. A processor; 82. a memory; 83. a communication interface; 80. a bus.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.

It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.

The entity linking method, the entity linking device, the computer equipment or the storage medium provide a quantitative calculation mode for the importance degree of the attribute in the entity linking, and simultaneously provide quantitative insight for the attribute to assist the entity linking algorithm in improving the accuracy or reducing the design cost of the manual algorithm.

The entity linking method and the entity linking device solve the problems that in the prior art, an entity linking method based on rules or machine learning is not high in accuracy and high in required labor cost. By providing the algorithm for quantitatively calculating the attribute discrimination, more data insight information is provided for the design of the entity link algorithm, the entity link accuracy is improved in an auxiliary manner, and the manual analysis design cost is reduced.

Fig. 1 is a schematic flow chart of the method of the present invention, and as shown in fig. 1, the present embodiment provides an entity linking method, including:

attribute discrimination calculation step S10: calculating attribute discrimination of at least one attribute of a plurality of data source entities based on an attribute discrimination algorithm;

linking step S20: and judging whether the plurality of data source entities are matched or not based on the attribute zone division of the data source entities, and completing the linkage of the plurality of data source entities.

In some embodiments, the attribute zone division algorithm is:

property region scale v:

In some embodiments, the linking step S20 includes:

In some embodiments, the linking step S20 further includes:

The following describes in detail a specific embodiment of the data integration method of the present invention with reference to the accompanying drawings:

(1) attribute zone partition quantitative calculation

Firstly, we propose the concept of "attribute zone division" to measure the degree of distinction of the attribute value to the entity, for example, for the entity of a person, one identity card number only corresponds to one person, one person has and only has one identity card number, we say that the attribute of the identity card number has very good degree of distinction for the entity of the person, relatively, one mobile phone number corresponds to one person, but one person may have a plurality of mobile phone numbers, one name may correspond to a plurality of persons, one person may also have aliases such as names used in great numbers, so the attribute zone division of the mobile phone number is inferior to the identity card number, but is superior to the name. Generally, a business expert or an algorithm expert considers the degree of distinction of attributes when making a rule or designing an algorithm, but intuitively considers the reference in an experience or the like. And 3, intuitive and intuitive feeling. However, when the data is excessive or the professional expert does not have clear and definite rules, the setting of the attributes is often determined by a brain-shooting bag or is tried continuously. There is a lack of quantitative computation.

For a class of entities R and an attribute a, we propose to define its attribute zone division v in the following way:

wherein n is the unique entity number of R in the data set, that is, the entity number here is the entity number after entity linking, and m is the number of all non-duplicated values of these entities on the attribute a. By this calculation, the closer the attribute values are to the entities, the closer their degree of distinction is to 1.

(2) Attribute differentiation assisted entity linking

The attribute discrimination provides quantitative insight information for the data, and can be used for assisting entity link algorithm design, improving algorithm accuracy or reducing manual analysis design cost. Several application modes are listed below:

1) and (5) assisting the business expert to make rules. Through the calculation of the attribute discrimination, the service expert can have more quantitative measurement on the 'important' attribute in the visual sense so as to formulate more scientific and perfect service rules.

2) As attribute weights. Calculating all attribute similarities and setting a threshold to judge whether two entities are matched are also a widely applied method, in the method, weight setting and similarity threshold setting of different attributes are important factors influencing the algorithm effect, and attribute discrimination provides important references for attribute weight and threshold setting, for example, attribute region division can be directly used as attribute weight.

3) Assisting the machine learning algorithm. On one hand, under the condition of more and disordered attributes, the attribute distinguishing degree can help algorithm personnel to distinguish and screen out important attributes and non-important attributes, only the important attributes are selected as features, computing resources are saved, and interference of the non-important attributes is reduced. On the other hand, the attribute region division can also be directly added into the feature vector as an image feature, so that more dimensional information is provided for the model.

In addition, fig. 2 is a schematic diagram of the entity linking system of the present invention, and as shown in fig. 2, an embodiment of the present application provides an entity linking system, which adopts the entity linking method described above, including:

In some embodiments, the attribute zone division algorithm is:

property region scale v:

In some embodiments, the linking module comprises:

In some embodiments, the linking module further comprises:

In a third aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the data integration method according to the first aspect is implemented.

In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the data integration method according to the first aspect.

In addition, the information recommendation method of the embodiment of the present application described in conjunction with fig. 1 may be implemented by a computer device. Fig. 3 is a hardware structure diagram of a computer device according to an embodiment of the present application.

The computer device may comprise a processor 81 and a memory 82 in which computer program instructions are stored.

Specifically, the processor 81 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 82 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 82 may include a Hard Disk Drive (Hard Disk Drive, abbreviated to HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 82 may include removable or non-removable (or fixed) media, where appropriate. The memory 82 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 82 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, Memory 82 includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (EAROM), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.

The memory 82 may be used to store or cache various data files for processing and/or communication use, as well as possible computer program instructions executed by the processor 81.

The processor 81 implements any of the entity linking methods in the above embodiments by reading and executing computer program instructions stored in the memory 82.

In some of these embodiments, the computer device may also include a communication interface 83 and a bus 80. As shown in fig. 3, the processor 81, the memory 82, and the communication interface 83 are connected via the bus 80 to complete communication therebetween.

The communication interface 83 is used for implementing communication between modules, devices, units and/or equipment in the embodiment of the present application. The communication port 83 may also be implemented with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.

Bus 80 includes hardware, software, or both to couple the components of the computer device to each other. Bus 80 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 80 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

In addition, in combination with the entity linking method in the foregoing embodiments, the embodiments of the present application may provide a storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the entity linking methods in the above embodiments.

Compared with the prior art, the invention provides the calculation of the attribute zone division and the use of the attribute zone division in the entity link; the invention provides a concept of attribute discrimination, and provides a quantitative calculation mode for the importance degree of attributes in entity links; the invention provides quantitative insight to attributes, assists in entity link algorithm to improve accuracy or reduce design cost of manual algorithm.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. An entity linking method, comprising:

2. The entity linking method according to claim 1, wherein the attribution differentiation algorithm is:

property region scale v:

3. The entity linking method according to claim 1, wherein said linking step comprises:

4. The entity linking method of claim 1, wherein said linking step further comprises:

5. An entity linking system using the entity linking method according to any one of claims 1 to 4, comprising:

6. The entity linking system of claim 5, wherein the attribution differentiation algorithm is:

property region scale v:

7. The entity linking system of claim 1, wherein the linking module comprises:

8. The entity linking system of claim 1, wherein the linking module further comprises:

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer device implements the entity linking method of any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the entity linking method according to any one of claims 1 to 4.