JP6767825B2 - Data management equipment, data management methods, and data management programs - Google Patents

Description

The present invention relates to a data management device, a data management method, and a data management program.

Conventionally, a technique for searching a desired data from a huge amount of data has been used in a web browser, an email software, or the like (see, for example, Patent Document 1). In such a technology, information such as searched or unsearched is managed by a table for each data to be searched. For example, there is known a method of preparing a bit array table having a one-to-one correspondence with the data to be searched and evaluating whether or not the data to be searched is already searched data. Further, there is known a method of evaluating whether or not the data to be searched is the data that has already been searched by registering the identifier of the searched data in an existing hash table implemented in C language or the like.

Japanese Patent Application Laid-Open No. 2009-211263

In the conventional technique, for example, when a bit array table is used, the time required to initialize the table at the start of the search increases as the number of data to be searched increases.

On the other hand, when an existing hash table is used, it takes time to acquire the data because the data structure of the list structure that manages the data identifiers by associating them with pointers is used.

The present invention has been made in view of the above problems, and an object of the present invention is to speed up a search from a huge amount of data.

One aspect of the present invention is a data management device that determines the necessity of predetermined processing for the target data, and uses a primary determination table having an area for storing the identification information of the target data and the identification information of the target data. Whether or not the identification information is registered is determined at the addresses of the secondary determination table stored in the list structure and the primary determination table corresponding to the hash value obtained by hashing the identification information of the target data with the first hash function. When it is determined that the identification information is not registered, the identification information of the target data is registered at the address of the primary determination table as the predetermined processing is performed, and the identification information of the target data is registered. If it is determined that the target data is present, it is determined that the predetermined processing related to the target data is unnecessary, and if the registration determination unit determines that the identification information is registered in the secondary determination table, the target data It is a data management device including a data management unit that refers to the secondary determination table using a hash value obtained by hashing identification information with a second hash function as a key, and determines the necessity of the predetermined processing.

According to one aspect of the present invention, it is possible to speed up a search from a huge amount of data.

It is a block diagram centering on the data management apparatus 100 in 1st Embodiment. It is a figure which shows an example of the contents of the existing data object 132. It is a figure which shows an example of the data structure of the graph index 134. It is a figure which shows typically the content of the search process by the index search unit 114. It is a figure which shows an example of the data structure of the primary determination table 122. It is a figure which shows an example of the data structure of the secondary determination table 124. It is a flowchart which shows an example of the flow of the process of determining whether or not the distance to a new object is calculated, which is executed by the index search unit 114. It is a flowchart which shows an example of the flow of the process of determining whether or not the distance to a new object is calculated, which is executed by the index search unit 114. It is a figure which shows another example of the primary determination table 122.

Hereinafter, a data management device, a data management method, and a data management program to which the present invention is applied will be described with reference to the drawings.

[Overview]
The data management device is realized by one or more processors. The data management device is, for example, a device that manages data for discriminating between already searched data and non-searched data when searching for data desired by a user in an Internet search.

(First Embodiment)
[overall structure]
FIG. 1 is a configuration diagram centered on the data management device 100 according to the first embodiment. The data management device 100 in the first embodiment is connected to one or more client terminals 10 via a network NW. The client terminal 10 is a personal computer, a mobile phone such as a smartphone, a tablet terminal, or the like. The network NW includes a wireless base station, a public line, a dedicated line, a provider terminal, the Internet, and the like. The network NW includes a wireless base station, a public line, a dedicated line, a provider terminal, the Internet, and the like.

When the data management device 100 receives the query data from the client terminal 10, it searches for data similar to the query data and returns the search result data to the client terminal 10. The query data is, for example, a keyword input in the search window of a web browser executed on the client terminal 10, data describing data search conditions, and the like. The data returned by the data management device 100 may be the data itself (for example, image data having an extension of jpg or the like), or an identifier for referencing the data (for example, a URL (Uniform Resource Locator)). May be good. Further, the query data and the target data for the search may be any kind of data such as image data, voice data, and text data.

The data management device 100 includes, for example, a network interface 102, an input / output device 104, a control unit 110, a storage unit 120, and a data server 130. The network interface 102 is, for example, a NIC (Network Interface Card).

The input / output device 104 receives an input operation by the administrator of the data management device 100. For example, the input / output device 104 includes an input device such as a mouse, a keyboard, and a touch panel, and an output device such as an LCD (Liquid Crystal Display), an organic EL (Electroluminescence) display device, and a speaker.

[Control unit]
The control unit 110 includes, for example, a data object generation unit 112 and an index search unit (an example of a data management unit) 114. The components of these control units 110 are realized by, for example, a processor such as a CPU (Central Processing Unit) executing a program. In addition, some or all of these components may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array). Software and hardware may work together. The control unit 110 does not have to be realized by one processor, and may perform distributed processing for each function. For example, the data object generation unit 112 and the index search unit 114 may be realized by separate processors.

The data object generation unit 112 generates an input data object based on the input data received from, for example, the client terminal 10. When the input data is an image, the object is, for example, a local feature amount, a BoF (Bag of Features), a color histogram, or a combination thereof. When the input data is text data, the object is, for example, vector data of the frequency of occurrence of words, a meaning vector extracted using a neural network, or a vector based on the frequency of occurrence with other words or other relative relationships. And so on. The object may be any data as long as the distance between the objects can be defined.

The index search unit 114 searches the data server 130 and extracts an object similar to the object (hereinafter referred to as a new object) generated by the data object generation unit 112 from the existing data object 132. For example, the existing data object 132 and the graph index 134 are stored in the data server 130. The graph index 134 is an index for searching an object at high speed, and is data having a graph structure composed of edges and objects (nodes).

The data server 130 is realized by a storage device such as an HDD or a flash memory. Further, the data server 130 may be realized by an external storage device such as a NAS (Network Attached Storage) device that can be accessed from the data management device 100 via the network NW.

The existing data object 132 and the graph index 134 are stored in the data server 130. The existing data object 132 and the graph index 134 are expanded (loaded) from the data server 130 into a high-speed accessible memory such as a RAM (Random Access Memory) and used. The existing data object 132 is an object generated from the data to be searched. The existing data object 132 is associated with the data from which the existing data object 132 is generated or an identifier that refers to the data. The data management device 100 transmits to the client terminal 10 an existing object obtained by querying the new object, that is, data associated with an object similar to the new object, or an identifier that refers to the data.

FIG. 2 is a diagram showing an example of the contents of the existing data object 132. As shown in the figure, the existing data object 132 refers to the object (vector data in the figure), the data from which the object is generated, or the data with respect to the object identification information (object ID in the figure). It is the data associated with the identifier to be used.

The graph index 134 is information about an edge connecting a plurality of existing vector data 132, and is data having a graph structure formed by a plurality of edges connecting any two of the existing vector data 134. The edge included in the graph index 134 is, for example, a bidirectional edge (undirected edge). FIG. 3 is a diagram showing an example of the contents of the graph index 134. As shown in the figure, the graph index 134 is data in which the object IDs of the existing objects at both ends to which the edges are connected are associated with the edge IDs that are the identification information of each edge. The objects may be connected by a unidirectional edge (directed edge) instead of the bidirectional edge.

Hereinafter, the processing of the index search unit 114 will be described in more detail. The index search unit 114 extracts a predetermined number of objects as objects similar to the new object in order from the object having the shortest distance to the new object among the objects that can be traced by the edge defined by the graph index 134. When the graph index 134 is composed of directed edges, the index search unit 114 has a short distance to the new object among the objects that can be traced according to the direction from the reference source object to the reference destination object. A predetermined number of objects are extracted in order from the beginning as objects similar to the new object. The distance between objects is defined as, for example, when the objects are vectors, the Lp norm (p = 1, 2, ...) Is obtained for the difference between the vector elements.

Further, the index search unit 114 may extract an object within a predetermined distance from the new object among the objects that can be traced by the edge defined by the graph index 134 as an object similar to the new object. Good. Then, the index search unit 114 transmits the data associated with the extracted object or the identifier thereof in the existing data object 132 to the client terminal 10 via the network interface 102 and the network NW.

FIG. 4 is a diagram schematically showing the content of the search process by the index search unit 114. First, the index search unit 114 inputs the graph structure (graph index) G, the new object q, and the search range r (S100). Next, the index search unit 114 randomly generates a set S of objects from the graph structure G (S102).

Next, the index search unit 114 clears the set R that is the output data (S104). Next, the index search unit 114 selects the object closest to the new object q from the objects included in the set S, sets it as the object s (S106), and excludes the object s from the set S (S108).

Next, the index search unit 114 determines whether or not the distance between the object s and the new object q exceeds the search radius r (S110). When the distance between the object s and the new object q does not exceed the search radius r, the index search unit 114 selects one object from the neighboring object set N (G, s) of the object s and sets it as the object o (S112). ). Then, the index search unit 114 determines whether or not the object o is included in the set C (S114). The set C is a set set to avoid duplicate selection, and data is deleted at the start or end of the processing of this flowchart.

When the object o is not included in the set C, the index search unit 114 determines whether or not the distance between the object o and the new object q is equal to or less than the search radius r (S118). When the distance between the object o and the new object q is less than or equal to the search radius r, the object o is put in each of the set S and the set R (S120). When the distance between the object o and the new object q exceeds the search radius r, the processing of S120 is skipped. Further, when it is determined in S114 that the object o is included in the set C, the processes of S116 to S120 are skipped.

Next, the index search unit 114 determines whether or not all the objects have been selected from the neighboring object set N (G, s) of the objects s (S122). If all the objects are not selected from the neighboring object set N (G, s) of the objects s, the process is returned to S112.

When all the objects are selected from the neighboring object set N (G, s) of the object s, the index search unit 114 determines whether or not the set S is an empty set (S124). If the set S is not an empty set, processing is returned to S106.

When it is determined in S124 that the set S is an empty set, or when it is determined in S110 that the distance between the objects s and q exceeds the search radius r, the index search unit 114 outputs the set R (S126). .. The objects included in this set R are objects similar to the new objects. The content of the search process described here is just an example, and the search process may be performed by another method.

Here, the index search unit 114 has the primary determination table 122 and the primary determination table 122 stored in the storage unit 120 in order to determine in S114 whether or not the object is included in the set C in S116 and the distance is calculated in S118. The secondary determination table is used to determine whether or not the object is an "object whose distance has already been calculated". The storage unit 120 is realized by, for example, a readable / writable memory such as a RAM or a flash memory.

FIG. 5 is a diagram showing an example of the data structure of the primary determination table 122. The primary determination table 122 is a table having a one-dimensional array structure, and is a table stored at a fixed address in the memory area. Each area obtained by dividing the primary determination table 122 is associated with a hash value obtained by hashing the object ID with the first hash function. In the figure, the start address is illustrated as information showing each area as a representative. In the area corresponding to each hash value, the object ID of the object that is the source of the hash value and the flag indicating whether or not the data is stored in the secondary determination table 124 are stored. The first hash function is, for example, a function for obtaining a remainder obtained by dividing an object ID by 2 ⁿ . In order to realize this, the index search unit 114 can perform processing at a higher speed than actually performing an operation by using, for example, the lower n bits of the binary string of the input data as a hash value. When the first hash function is a function for obtaining the remainder obtained by dividing the object ID by 2 ⁿ , N = 2 ⁿ in the figure. For example, it is preferable that N is set to a value of about 1/10 of the total number of existing vector data 132.

The index search unit 114 first determines "whether or not the distance to a new object has been calculated" for the object of interest (hereinafter referred to as the "object of interest") as an object for determining whether or not the distance has already been calculated. It is determined whether or not the interest ID is stored (registered) in the address corresponding to the hash value obtained by hashing the object ID (hereinafter, the attention ID) of the object of interest. When the focus ID is stored, the index search unit 114 determines that the focus object has "calculated the distance to the new object".

When the object ID of interest is not stored, the index search unit 114 determines whether or not the object ID of another object is stored at the address corresponding to the hash value from the object of interest. When the object IDs of other objects are not stored (that is, when nothing is stored), the index search unit 114 determines that the object of interest "has not calculated the distance to the new object". In this case, the index search unit 114 calculates the distance between the object of interest and the new object, and writes the ID of interest in the address corresponding to the hash value from the object of interest.

On the other hand, when the object ID of another object is stored in the address corresponding to the hash value from the object of interest, the index search unit 114 refers to the flag of the primary determination table 122 and displays the secondary determination table 124. It is determined whether or not one or more object IDs from which the same hash value is derived (hash collision) are stored by the first hash function. For example, if the flag value is 1, it means that one or more object IDs are stored in the secondary determination table 124 (hash collision has occurred), and if the flag value is 0, the object ID is Indicates that it is not stored (no hash collision has occurred).

Therefore, if the flag value is 0, the index search unit 114 determines that the object of interest has not calculated the distance to the new object. In this case, the index search unit 114 calculates the distance between the object of interest and the new object, and puts the ID of interest in the secondary determination table 124 into an area corresponding to the hash value hashed by the second hash function (described later). , Write the focus ID. If the flag value is 1, the index search unit 114 refers to the secondary determination table 124 and determines "whether or not the distance to the new object has been calculated" for the object of interest.

FIG. 6 is a diagram showing an example of the data structure of the secondary determination table 124. The secondary determination table 124 has a one-dimensional array structure data storage area (array portion in the figure) for the possible value of the hash value derived by the second hash function, and has a list structure data storage area. (List part in the figure). The address (link destination) at the beginning of the list part is stored in the array part. The second hash function is, for example, a function for deriving any of M hash values sufficiently smaller than N from the input data. The second hash function is, for example, a function that uses the lower m bits of the binary string of the object ID as a hash value (n> m).

Object IDs are sequentially registered in the list portion of the secondary determination table 124. When the object ID is registered in the list part, for example, the address of the list part to be referred to next is stored in the end part following the object ID in the array part. This relationship is referred to as the pointer indicating the address to be referenced next. In the figure, the arrow indicates a pointer.

The index search unit 114 refers to the secondary determination table 124 and determines whether or not the attention ID is stored while tracing the pointer. If the focus ID is stored at any time, the index search unit 114 determines that the focus object has "calculated the distance to the new object".

On the other hand, when the pointer is traced to reach the last area, the index search unit 114 determines that the object of interest has not calculated the distance to the new object. The index search unit 114 determines that "the last area has been reached" when data that does not store the address of the list part to be referred to next appears. In this case, the index search unit 114 stores the object ID of the object of interest at an arbitrary address, and adds a pointer indicating the address in which the object ID of the object of interest is stored to the immediately preceding area.

7 and 8 are flowcharts showing an example of a flow of processing executed by the index search unit 114 to determine whether or not the distance to the new object has been calculated.

First, the index search unit 114 erases (clears) the data in the primary determination table 122 and the secondary determination table 124 (S200).

Next, the index search unit 114 hashes the attention ID with the first hash function (S202). Next, the index search unit 114 refers to the primary determination table 122 (S204), and determines whether or not the data is registered at the address corresponding to the hash value obtained in S202 (S206). If the data is not registered at the address corresponding to the hash value obtained in S202, the focus ID is registered in the primary judgment table (S208), and it is determined that the focus object "has not calculated the distance to the new object". (S210), the process of this flowchart ends.

When it is determined in S206 that the data is stored in the address corresponding to the hash value obtained in S202, the index search unit 114 determines whether or not the attention ID is registered in the address corresponding to the hash value obtained in S202. (S212). When the focus ID is registered at the address corresponding to the hash value obtained in S202, the index search unit 114 determines that the focus object has "calculated the distance to the new object" (S214), and processes this flowchart. To finish.

When the attention ID is not registered in the address corresponding to the hash value obtained in S202 (hereinafter, FIG. 8), the index search unit 114 uses the hash value obtained by the first hash function in S202 in the primary determination table 122. It is determined whether the flag given to the region corresponding to is 1 or 0 (S216). When the flag is 0, the index search unit 114 hashes the attention ID with the second hash function (S218), and registers the attention ID in the secondary determination table 124 (S224). At this time, the index search unit 114 assigns a pointer indicating the address in which the attention ID is registered to the area corresponding to the hash value obtained in S218 in the array portion of the secondary determination table 124. Then, the index search unit 114 determines (hereinafter, FIG. 7) that the object of interest has not calculated the distance to the new object (S210), and ends the process of this flowchart.

On the other hand, when the flag is 1, the index search unit 114 hashes the focus ID with the second hash function (S220), and the focus ID sequentially traces the pointer from the address corresponding to the hash value obtained in S220. Is determined whether or not is registered (S222). When the attention ID is not registered at the destination where the pointers are sequentially followed, the index search unit 114 performs the processing of S224 or less. When the focus ID is registered at the destination where the pointers are sequentially followed, the index search unit 114 (hereinafter, FIG. 7) determines that the focus object has "calculated the distance to the new object" (S214), and this flowchart. Ends the processing of.

According to the first embodiment described above, the storage unit 120 in which the primary determination table 122 having the one-dimensional array structure and the secondary determination table 124 having the one-dimensional array structure and the list structure are stored. When determining whether or not the distance between the object of interest (target data) and the new object has been calculated, the address corresponding to the hash value obtained by hashing the interest ID with the first hash function in the primary determination table 122 is used. It is determined whether or not the focus ID or the object ID of another object is registered, and if any of them is not registered, the focus ID is registered at that address, and if the focus ID is registered, the focus is paid. If it is determined that the distance between the object and the new object has been calculated and the object ID of another object is registered, the secondary determination is made using the hash value obtained by hashing the object ID of the object of interest with the second hash function as a key. High-speed processing can be realized by providing the index search unit 122 that refers to the table 122.

Here, a comparison with a case where the same determination as above is performed only with the sequence structure or the list structure will be described.

If the number of arrays in the primary determination table 122 is prepared to be the same as the number of objects included in the existing data object 132, for example, the process of tracing the pointer in the list structure becomes unnecessary. Therefore, the process of determining whether or not a new object has already been selected can be performed at high speed, but the time required for the "process of erasing data" shown in S200 of FIG. 7 becomes long, and the search can be performed. The time required for the entire process becomes long.

On the other hand, when the same determination is made only with the list structure, the process of tracing the pointer occurs even for the object in which the hash collision does not occur, so that the time required for each determination process becomes long. Further, since the pointer is added every time a hash collision occurs, the time required for the determination process becomes longer.

Usually, the target range (search range) of the search process is a limited range with respect to the entire graph index 134. Therefore, if the array structure of the number of arrays corresponding to the search range is set, the time required for the determination process can be sufficiently shortened. This part corresponds to the primary determination table 122. The processing time becomes long by using the secondary judgment table 124, but if the number of arrays in the primary judgment table 122 has a margin, the possibility of hash collision can be reduced, and as a whole, the possibility of hash collision can be reduced. The search process can be speeded up.

Although the primary determination table 122 has been described as having a one-stage array structure, it may have a multi-stage array structure. FIG. 9 is a diagram showing another example of the primary determination table 122. In this case, the object ID is stored in the first stage if the first stage is empty, in the second stage if the first stage is not empty and the second stage is empty, and second if the second stage is also empty. The next determination table 124 is referred to.

In addition, "a flag indicating whether or not data is stored in the secondary judgment table 124 is added to each entry of the primary judgment table 122", but the addition of this flag is omitted and the index search unit is used. When the ID of the object other than the attention ID is stored in the primary determination table 122, 114 refers to the first data of the secondary determination table 124 and determines whether or not the data is stored in the secondary determination table 124. May be confirmed.

Further, the "predetermined process" for determining whether or not the data management device 100 has already executed is a node corresponding to the query (the target data is set as the search target node in the data search process using the graph structure data). The process is not limited to the process of calculating the distance to the new object), and may be another type of process.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

10 ... Client terminal, 100 ... Data management device, 102 ... Network interface, 104 ... Input / output device, 110 ... Control unit, 112 ... Data object generation unit, 114 ... Index search unit, 120 ... Storage unit, 122 ... Primary judgment table , 124 ... Secondary judgment table, 130 ... Data server, 132 ... Existing vector data, 134 ... Graph index

Claims (11)

A data management device that determines the necessity of predetermined processing for target data.
A primary determination table having an area for storing the identification information of the target data, and
A secondary judgment table that stores the identification information of the target data in a list structure,
Whether or not the identification information is registered is determined at the address of the primary determination table corresponding to the hash value obtained by hashing the identification information of the target data with the first hash function.
When it is determined that the identification information is not registered, the identification information of the target data is registered in the address of the primary determination table as the predetermined processing is performed.
When it is determined that the identification information of the target data is registered, it is determined that the predetermined processing related to the target data is unnecessary.
When it is determined that the identification information is registered in the secondary determination table, the secondary determination table is referred to by using the hash value obtained by hashing the identification information of the target data with the second hash function as a key. A data management unit that determines the necessity of predetermined processing,
A data management device equipped with. The data management unit refers to a flag stored in the primary determination table, which indicates whether or not the identification information is stored in the secondary determination table, and whether the identification information is registered in the secondary determination table. Judge whether or not
The data management device according to claim 1. The data management unit sequentially traces the address indicated by the pointer stored in the address of the secondary determination table corresponding to the hash value obtained by hashing the identification information of the target data with the second hash function. It is determined whether or not the identification information of the target data is registered in the secondary determination table.
The data management device according to claim 1 or 2. When the identification information of the target data is registered in the secondary determination table, the data management unit determines that the predetermined processing is necessary.
The data management device according to claim 3. The number of possible values of the hash value obtained by the first hash function is larger than the number of possible values of the hash value obtained by the second hash function.
The data management device according to any one of claims 1 to 4. The first hash function is a function whose hash value is the lower n bits of the binary string of the input data.
The data management device according to any one of claims 1 to 5. The primary determination table has a plurality of one-dimensional array structures.
The data management device according to any one of claims 1 to 6. The predetermined process is a process of calculating the distance to a node corresponding to a query by using the target data as a search target node in a data search process using graph-structured data.
The data management device according to any one of claims 1 to 7. The number of arrays in the primary determination table is set to a value of about 1/10 of the number of nodes to be searched.
The data management device according to claim 8. It has a primary determination table having an area for storing the identification information of the target data and a secondary determination table for storing the identification information of the target data in a list structure, and determines the necessity of predetermined processing for the target data. Data management device
Whether or not the identification information is registered is determined at the address of the primary determination table corresponding to the hash value obtained by hashing the identification information of the target data with the first hash function.
When it is determined that the identification information is not registered, the identification information of the target data is registered at the address as the predetermined processing is performed.
When it is determined that the identification information of the target data is registered, it is determined that the predetermined processing related to the target data is unnecessary.
When it is determined that the identification information is registered in the secondary determination table, the secondary determination table is referred to by using the hash value obtained by hashing the identification information of the target data with the second hash function as a key. Judging the necessity of a predetermined process,
Data management method. It has a primary determination table having an area for storing the identification information of the target data and a secondary determination table for storing the identification information of the target data in a list structure, and determines the necessity of predetermined processing for the target data. To the data management device
The address of the primary determination table corresponding to the hash value obtained by hashing the identification information of the target data with the first hash function is made to determine whether or not the identification information is registered.
When it is determined that the identification information is not registered, the identification information of the target data is registered at the address by performing the predetermined process.
When it is determined that the identification information of the target data is registered, the predetermined processing relating to the target data is determined to be unnecessary.
When it is determined that the identification information is registered in the secondary determination table, the secondary determination table is referred to by using the hash value obtained by hashing the identification information of the target data with the second hash function as a key. Lets determine the necessity of predetermined processing,
Data management program.

Images