JP6440542B2 - Knowledge engine for managing large amounts of complex structured data - Google Patents

Description

  The present invention relates to information storage and retrieval, and more particularly to structured data storage and retrieval.

[Related applications]
This application claims the benefit of US Provisional Application No. 61 / 955,077, filed Mar. 18, 2014, the contents of which are hereby incorporated by reference in their entirety.

  The storage of structured data in relational databases that can be queried using Structured Query Language (SQL) has evolved gradually since the 1970s. For example, a large merchant may use a relational database to store customer profiles and other histories. Such a database can be quite large, but only works efficiently when the range of data retrieved is relatively narrow, such as a customer profile.

  A "knowledge engine" has been developed that can query a knowledge database corresponding to a wide range of matters against a conventional relational database. However, such knowledge engines are so complex that their complexity is unmanageable due to the wide range of entities and attributes to be queried. For example, one user may ask "How many years was Jeff Jefferson Clinton born?" And another user might ask the same knowledge engine "What is the population of Barrow, Alaska in 2012?" . Accordingly, there is a need in the art for a knowledge engine that can efficiently accommodate the complexity and scope of structured data to be queried and a corresponding knowledge database.

  In addition, knowledge databases are becoming large and difficult to handle because of the amount of structured data that they are required to store. Therefore, there is a need in the art for a knowledge database that has an organization that supports efficient and fast searches by corresponding knowledge engines while allowing efficient storage.

  The following exemplary embodiments are directed to the use of Freebase as a source of structured data used to build the disclosed knowledge database. However, it will be appreciated that in alternative embodiments, Wikipedia, a wide variety of other sources of information such as online and electronic dictionaries and encyclopedias can be used instead of Freebase. For example, the knowledge database can be driven by a system for generating resources as disclosed in International Application No. PCT / US14 / 67479, the contents of which are hereby incorporated by reference in their entirety. Is done. Therefore, the knowledge database can be loaded with various links to other resources such as hyperlinks to a plurality of other items in the Wikipedia database.

  In Freebase, data is organized in triples of Resource Description Framework (RDF). The first part of the RDF triple is an entity, which is the subject described or depicted. The second part of the triple is an attribute or predicate, which is the type of relationship for the entity being described. Finally, the third part of a triple is a value or object, which is the object referenced by that triple. For example, consider the following example sentence: “Joe is a friend of Tom.” In this example, Tom is the subject or entity, friendship is the attribute or predicate, and Joe is the value. Such a triple can be easily expressed in the form of a connection graph, where Joe and Tom are nodes connected by an arc from Joe to Tom, representing the attribute “is a friend of”. A number of similar triples, such as Tom's age, occupation, and interest, can extend from the entity “Tom”.

  An exemplary method may include forming a category index for an encoded category for a set of encoded resource description format entities, attributes, values, and encoded categories, wherein the category index includes: Each encoded category includes a list of corresponding encoded entities. The method may also include determining, for each encoded attribute, an encoded entity that has an encoded value for the encoded attribute. The method may also include forming an attribute index of the encoded attributes and their corresponding encoded entities and encoded values in response to the determination. The method can also include storing the category index and the attribute index to form a knowledge database.

  The method can also include decomposing the structured query into a plurality of simple queries. For each of a plurality of simple queries, the method may also include accessing a knowledge database category index or attribute index to determine a list of encoded entities. The method may also include determining an encoded entity that is responsive to the structured query for a common set of encoded entity lists. The method may also include converting the encoded entity responsive to the structured query to the original entity.

FIG. 6 is a flowchart for loading an exemplary knowledge database. FIG. 6 is a flowchart when an exemplary knowledge engine queries a knowledge database. FIG. 3 is a diagram of an exemplary search architecture that includes the knowledge engine of FIGS. 1 and 2. 4 is a diagram of an exemplary system that includes the knowledge engine of FIGS. 1-3. FIG. FIG. 5 is a diagram of an exemplary server device that includes the knowledge engine of FIGS. FIG. 5 is a diagram of an exemplary server device that includes the knowledge engine of FIGS.

  Referring now to the drawings, an unprocessed triple is received from a corresponding database, such as Freebase, as shown in step 101 of FIG. Step 102 removes redundant entities, attributes, and RDF triples from the unprocessed triple received in step 101 to obtain a non-redundant (also referred to herein as “clean”) RDF triple 103. including. For the RDF triple 103, those entities are ranked in step 104. For example, the rank of an entity can be generated by analyzing Wikipedia access count output, frequency of entity name appearance, entity name popularity, and the like.

  At step 105, the clean RDF triple 103 is encoded as a unique integer identification (ID). Each entity, attribute, and value is so encoded so that a single triple can be represented by an integer triplet, resulting in encoded data at step 106. In order to distinguish between the original version and the encoded version, the integer ID representing the encoded entity is also referred to herein as an “entity ID”. Similarly, an integer ID representing an encoded attribute is also referred to as “attribute ID” in this specification. Similarly, an integer ID representing an encoded value is also referred to as a “value ID” in this specification. It is also possible to assign the integer ID of the entity to the rank of the entity obtained in step 104. In addition, entities can be organized according to categories so that each category can also be encoded into a category ID. A hash table H (not shown) stores all associations for entities, attributes, values, entity rankings, and categories in relation to their integer representation. Similarly, the uncoded state corresponding to the integer ID, that is, the original entity, attribute, value, entity ranking, and association with the category are stored in the array list L. Therefore, the hash table H and the array list L are opposite to each other. The encoded RDF triple can then be loaded into memory 108 at step 107. Step 107 may also include indexing all entity IDs according to their category ID and attribute ID. A category represents a type of entity, for example, the type is “movie star”, whereas an entity of such a category can consist of individual movie stars. In step 109, additional indexing can be performed with respect to indexing the entity ID and value ID for each attribute ID. The result of this indexing is represented by the category ID and attribute ID index stored in the knowledge database or memory at step 110.

  An exemplary category index lists entity IDs by category. Similarly, the attribute index lists the entity ID along with the value ID for each attribute. The category ID index and the attribute ID index are sorted by entity ID for each category ID and for each attribute ID. The resulting encoded and indexed RDF triples in the knowledge database formed as described with respect to FIG. 1 can be advantageously retrieved by the knowledge engine 200 as shown in FIG. At step 201, a structured query 201 is received. Such structured queries include specified entities, attributes with comparison values, and categories, as well as output attributes and sort order. In step 202, the structured query 201 is parsed into a simple query 203, which is advantageous. Each simple query 203 can correspond to one category and one attribute. Further, each simple query 203 can have only one entity, or only one attribute with a comparison, or only one category. For example, a user may want to know the identity of all movie stars born after 1982 in the database. Such a query can be structured into several simple queries, such as all entities in the category “Movie Star” and all entities with attributes born after 1982. It should be noted that if each simple query 203 corresponds to one category and one attribute, the number of categories and attributes can far exceed general relational database constraints. Specifically, the data set in this embodiment may exceed several hundred gigabytes, and a large number of computer clusters may be required to parse the simple query 203.

  The simple query 203 is created using the encoded integer value of the hash table. That is, rather than searching for the category “movie star”, the actual search is based on the integer coded representation (category ID) of the category “movie star”. Similarly, in the search for the attribute “release date”, the search is performed using the attribute ID corresponding to the “release date”. Step 204 includes listing the output attributes and sort attributes of the query. This list may be empty in some embodiments.

  The comparison step 205 is performed for each simple query 203. For example, the simple query 203 can include listing all entity IDs corresponding to a particular category ID. Alternatively, the simple query 203 may relate to listing all entity IDs and corresponding value IDs for a particular attribute ID. Then, using the array list L, the value ID can be decrypted to the original value. The resulting value can then be compared to some query parameter to determine whether the corresponding entity responds to the simple query. For example, such a simple query can include a comparison value whose value must be greater than, equal to, or smaller depending on the inherent nature of the simple query. Each time there is a comparison or match in step 205, a corresponding entity list 206 is generated.

  In step 207, an entity list 208 is obtained by obtaining a common set of various entity lists 206. For example, in the above exemplary search that identifies all movie stars born after 1982, one entity list 206 can include all movie stars found in memory 108. Another list may include all entities in the database that were born after 1982. A common set of these lists corresponds to the required answer.

  For example, in step 207, a common set of entity A (denoted as E (A)) and entity B (denoted as E (B)) is identified by a common set E (A) ∩E (B). Can be asking. If there is no common set between the entities, there may be no relationship between entities A and B. When this common set exists, the union of E (A) and E (B) represented by E (A) ∪E (B), and the expression −log ((E (A) ∩E (B ) / E (A) ∪E (B)) can be calculated to obtain a similarity score between entities A and B.

  Such similarity score is inversely proportional to the similarity between entities. In particular, the most closely related entities are considered to correspond to the common set of referenced entities being the same as their union (the logarithm of 1 is zero regardless of the base of the logarithm). As the common set becomes smaller compared to the union, the resulting logarithm of the ratio becomes more negative and the inverse of the logarithm becomes more positive. In that way, for each entity, an ordered score of the related entities can be generated. In some embodiments, a threshold is applied to the ordered score to determine the subset of entities most relevant to a given entity, and possibly a list 208 of entities that are responsive to the structured query 201. Ask for. Regardless of whether a threshold is applied, the similarity calculation makes it easy to obtain a list 208 of entities that are responsive to the query 201 and also facilitates sorting and ranking 209 of related entities. be able to.

  Additional algorithms can be implemented in place of or in combination with the logarithmic method described above. For example, the numerical similarity between entities can be calculated using the Jaccard method or the PMI method. Also, using a given entity being an element of a category, other elements in that category can also be selected as related entities.

  At step 209, based on the output format determined at step 204, the entities in the final list 208 can be sorted according to their designated attributes. Alternatively, step 209 may include ranking the entities by entity rank if the output format is empty. From the sorting or ranking in step 209, an output list of entities of the type specified in step 210 can be obtained, which can be displayed to the user as a search result in step 211.

  Then, some example queries are considered to describe the search process in more detail. The structured query 201 has two parts, an input format and an output format. For example, assume that the user wants to know the titles of all the movies released in the week from September 5, 2013 to September 12, 2013. The resulting structured query 201 can have an input format of category = movie, release date> = 2013-09-05, release date <2013-09-12. The output format can be entity name, release date: sorted in descending order.

  The resulting structured query 201 can then be decomposed into the next simple query 203. The first simple query lists all entities of category = movie. As described above, the knowledge database stores the category index and the attribute index in a compressed form so that the RDF triplet value can be replaced with, for example, an integer triplet. It will be appreciated that other types of data compression may be used. For this simple query, the hash table H is first accessed to find an integer representation of the category “movie”. A search from the knowledge database then simply matches the integer in the hash table corresponding to the category “movie” with the encoded category index. For example, it is assumed that the category “movie” is represented by an integer ID “j”. Therefore, searching the knowledge database is a relatively fast and easy task of retrieving the category index encoded as integer ID “j”. This j-th index is therefore a movie index, resulting in a list of corresponding entity IDs.

  The second simple query that results from the decomposition of the example structured query is to list all entities that have a publication date attribute, and the attribute value is 2013-09-05 or higher (such a simple query). Note that a query has attributes, comparisons, and values). Therefore, an integer representation of the attribute “release date” is acquired from the hash table H. For example, it is assumed that the attribute ID of “release date” is represented by an integer y. Then, the knowledge engine retrieves the attribute index corresponding to the integer y from the knowledge database. Such a search only matches integers and is therefore much faster than prior art methods. The attribute index obtained from the search lists all entities that have a value corresponding to the “publication date” attribute, but is encoded like any other item stored in the knowledge database. In other words, the attribute index obtained by the search is a list of pairs, and each pair is an entity ID and a corresponding value ID. The knowledge engine can then create a new entity ID list E. For each value ID “n” of the attribute index obtained by the search, the knowledge engine acquires the original value (public date in this example) from the array list L, and the query date whose public date is “2013-09-05”. Can be compared to the same or later date. If the result of the comparison is true (the date the publication date is the same as or later than the query date “2013-09-05”), the knowledge engine can put the corresponding entity ID into the entity ID list E. This entity ID list E becomes a list that is responsive to the second simple query.

The third simple query is similar to the second simple query, and thus covers the list of all entities with the publication date attribute, and the attribute value is less than the date 2013-09-12. This third simple query is therefore processed in the same manner as described for the second simple query. Then, the common set of the three entity lists obtained by these three simple queries can be performed as described with respect to step 207 to obtain the output entity list 208. In this example, there is no need for sorting or ranking as described with respect to step 209 in FIG. The array list L can then be used to decrypt the result to obtain the original entities and their values. Table 1 below shows search results listing movies according to this example structured query.

  The actual list of search results generally depends on the output format 204 specified in the query, such as whether values should be listed in ascending or descending order.

  Knowledge engine 200 may consist of a server, multiple servers, or other type of suitable computer. To perform the steps shown in FIG. 2, the server implementing the knowledge engine 200 can be coded in Java or other suitable programming language. Triples stored in Freebase are represented using a form known in the field of RDF.

  Knowledge engine 200 can be incorporated into the system architecture as shown in FIG. A semantic engine 305 receives a natural language query from a user and provides a corresponding structured query 201 to the knowledge engine 200. The knowledge engine then interacts with the database 108 described with respect to FIG. 2 to obtain the requested search results and then displays the results on a user device 310 such as a smartphone, smartwatch, tablet, or other suitable device. Can do.

  Knowledge engine 200 can also be incorporated into the system as shown in FIG. System 400 includes a plurality of computing devices, such as server device 402 and client devices 404 and 406, each of which can be configured to send and receive data / data packets 416 and 418 over communications network 408. . For example, the communication interface 410 can send and receive raw RDF data and structured queries as described above in connection with FIGS. Memory 412 is volatile, non-volatile, removable, and / or non-removable, such as magnetic, optical, or flash storage, further configured to store one or more entity lists, for example, as described above. One or more of the storage components may be included. As such, communication interface 410, knowledge engine 200, semantic engine 305, knowledge base 108, and memory 412 can be communicatively coupled via a system bus, network, or other connection mechanism 414.

  Client devices 404 and 406 can take a variety of forms, such as personal computers (PCs), smartphones, among other computing devices capable of sending structured queries and receiving search results, for example. , Wearable computers, laptop / tablet computers, smart watches with appropriate computer hardware resources, head mounted displays, and other types of wearable devices. Client devices 404 and 406 can include various components, such as, for example, input / output (I / O) interfaces 430 and 440, communication interfaces 432 and 442, processors 434 and 444, and data storage mechanisms, respectively. 436 and 446, all of which can be communicatively coupled to each other via a system bus, network, or other connection mechanism 438 and 448, respectively.

  The I / O interfaces 430 and 440 can be configured to facilitate interaction between the client devices 404 and 406 and the user of the client devices 404 and 406, respectively. For example, the I / O interfaces 430 and 440 can be configured to access queries received from a user and provide search results to the user. Thus, I / O interfaces 430 and 440 may include input hardware, such as a microphone that receives voice commands, a touch screen, a touch-sensitive panel, a computer mouse, a keyboard, and / or other input hardware.

  As described above, the knowledge engine 200 can comprise a server, multiple servers, or other types of suitable computers. The knowledge engine 200 can be integrated with the server device as shown in FIGS. 5A and 5B due to its expandability. FIG. 5A illustrates an exemplary server device 500 configured to support a set of trays, according to one embodiment. As shown, the server device 500 can include a housing 502 that supports the trays 504 and 506 and possibly can support a plurality of other trays. The housing 502 can include slots 508 and 510 configured to hold trays 504 and 506, respectively. The housing 502 can be connected to the power supply 512 via connections 514 and 516 to supply power to the slots 508 and 510, respectively. Enclosure 502 can also connect to communication network 518 via connections 520 and 522 to provide network connectivity to slots 508 and 510, respectively. Accordingly, trays 504 and 506 can be inserted into slots 508 and 510, respectively, to receive power from power source 512 and to connect to communication network 518.

  FIG. 5B illustrates a tray 504 configured to support one or more components. The tray 204 can include a knowledge engine 200, a semantic engine 305, a knowledge base 108, a communication interface 410, and a memory 412. Tray 204 can include a connector 526 that can be coupled to connection 514 or 516 to provide power to tray 504. The tray 504 can also include a connector 528 that can be coupled to the connection 520 or 522 to provide the tray 504 with connectivity to the network. As such, knowledge engine 200, semantic engine 305, and knowledge base 108 can be configured to perform the operations described herein and illustrated in the accompanying drawings.

  DESCRIPTION OF SYMBOLS 108 ... Memory, 200 ... Knowledge engine, 201 ... Structured query, 203 ... Simple query, 206, 208 ... Entity list, 305 ... Semantic engine, 310 ... User device, 402 ... Server device, 404, 406 ... Client device, 408 Communication network 416 418 Data / data packet 410 Communication interface 412 Memory 414 Connection mechanism 430 440 I / O interface 432 442 Communication interface 434 444 Processor 436 446: Data storage mechanism, 438, 448 ... Connection mechanism, 500 ... Server device, 502 ... Housing, 504, 506 ... Tray, 508, 510 ... Slot, 514, 516 ... Connection, 512 ... Power supply, 518 ... Communication network , 52 ... connector, 528 ... connector.

Claims (13)

A knowledge database configured to store an attribute index for an encoded attribute, an encoded entity corresponding to the encoded attribute, and an encoded value, the encoded Said knowledge database further configured to store a category index of categories, wherein each encoded category in said category index includes a list of corresponding encoded entities;
A knowledge engine configured to retrieve an encoded entity list responsive to a simple query using the category index and the attribute index stored in the knowledge database;
A system comprising: Encoded attributes, encoded entity, encoded values, and each coded categories, according to claim 1 comprising a corresponding integer system. The system is
A semantic engine configured to receive a natural language query and provide a corresponding structured query to the knowledge engine;
The system of claim 1 comprising: The system of claim 1 , wherein the knowledge engine interacts with the knowledge database to cause a user device to display the encoded entity list that is responsive to the simple query. The system of claim 1 , wherein the knowledge engine is configured to obtain a list of the encoded entities that are responsive to the simple query for a common set of the encoded entities. The system of claim 1 , wherein the knowledge engine is configured to determine a common set of one entity list in memory and another entity list in the knowledge database. The knowledge engine is configured to determine that a similarity score between the list of encoded entities is greater than a threshold and obtain the list of encoded entities that are responsive to a structured query The system of claim 1 , wherein: Such that the knowledge engine determines that a similarity score between the list of encoded entities is greater than a threshold and obtains the list of encoded entities that are responsive to the structured query. And the similarity score is determined based on -log ((E (A) ∩E (B) / E (A) ∪E (B)), where E (A) is an entity The system of claim 7 , wherein E (B) is another entity. A non-transitory computer-readable recording medium storing program instructions,
The program instructions are sent to the knowledge engine,
For a set of encoded resource description format entities, attributes, values, and encoded categories, creating a category index for the encoded category, wherein each encoded category in the category index is The creating step, including a list of corresponding encoded entities;
Determining, for each encoded attribute, an encoded entity having an encoded value for the encoded attribute;
Forming an attribute index with respect to the encoded attribute and the encoded entity and the encoded value corresponding to the encoded attribute in response to the determination result;
Storing the category index and the attribute index to form a knowledge database;
Is a program instruction for executing
A non-transitory computer-readable recording medium. The program instructions are sent to the knowledge engine,
Decomposing a structured query into multiple simple queries,
For each of the plurality of simple queries, accessing the category index or the attribute index of the knowledge database to determine a list of encoded entities;
Determining a common set of a plurality of lists of the encoded entities to determine the encoded entities that are responsive to the structured query;
Converting the encoded entity responsive to the structured query to an original entity;
Is a program instruction for further executing
The non-transitory computer-readable recording medium according to claim 9 . Obtaining a common set of the encoded list of entities;
The non-transitory computer-readable recording medium according to claim 10 , comprising: obtaining a common set of one entity list in a memory and another entity list in the knowledge database. The program instructions are sent to the knowledge engine,
Determining that the similarity score between the list of encoded entities is greater than a threshold to determine the encoded entities responsive to a structured query;
Is a program instruction for further executing
The non-transitory computer-readable recording medium according to claim 9 . Determining that the similarity score between the list of encoded entities is greater than a threshold and determining the encoded entities responsive to the structured query comprises:
-Log (E (A) ∩E (B) / E (A) ∪E (B))
The non-transitory computer-readable recording medium according to claim 12 , wherein the E (A) is one entity and the E (B) is another entity.