JPH11203183A - Data management device and record medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To execute the identification processing of identifiers at high speed. SOLUTION: An identifier input means 1 inputs a first identifier set and an identifier division means 2 divides the respective identifiers belonging to the first identifier set into the segments of a prescribed data length. A tag generation means 3 generates a tag by computing the exclusive OR of the segments obtained from the respective identifiers and an index generation means 4 generates an index from the tag belonging to the first identifier set. Then, in the case that a second identifier set is inputted so as to perform the identification processing with the first identifier, the tags corresponding to the respective identifiers of the second identifier set are generated by a similar processing. Then, whether or not the same tag is present is judged by referring to the index, and when it is present, the identifiers are compared to each other.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a data management device and a recording medium, and more particularly to a data management device and a recording medium for managing data by using an identifier.

[0002]

2. Description of the Related Art When managing a plurality of data recorded in a recording apparatus, it is often necessary to perform arithmetic processing on a data set.

[0003] Generally, in an arithmetic processing for a data set, a data identification processing for determining that the data is the same is frequently used. Conventionally, data and an identifier for uniquely identifying the data are managed in association with each other, and the identifiers are compared to determine the identity of the identifier and identify the data.

Normally, the data length of an identifier is very small as compared with the data length of the data itself, so that the data identification process using the identifier has been efficient. However, when the number of pieces of data to be managed increases, for example, when the data management system is operated in a distributed environment, the data length representing an identifier for identifying the data has been gradually increased.

Accordingly, a method has been proposed in which the position where the data (file) is recorded is specified by a path name. That is, in such a method, data called a file is identified by a path name.

By the way, since a path name is given to a user-friendly name, the length of the path name (the data length of the path name) tends to increase as the number of files to be managed increases. Therefore, as the file hierarchy increases, the path name naturally becomes longer. At this time, since the file (data) identification process is a process of comparing path names (character strings), the processing cost increases as the path name becomes longer.

As a method of solving such a problem,
Japanese Patent Application Laid-Open No. 4-338844 discloses a "file path name management control method". In the present invention, an ID corresponding to a path name is created, and this ID is used instead of the path name. When creating an ID, the ID has fewer bytes than the path name,
In addition, care must be taken so that the correspondence between the ID and the path name is unique. By using this ID instead of the path name, it is possible to save memory and to speed up the processing. As a result, the IDs are compared with each other so that the ID can be compared with the case where the path name is used. It becomes possible to identify a file quickly.

On the other hand, "try" is known as an efficient search method for a key having a long data length. In the try, characters included in a long character string are made to correspond to nodes of a tree structure as an index. Then, the target character string can be searched for by comparing the character string to be searched for each node of the tree structure.

For example, in Japanese Patent Laid-Open No. 5-2607 "High-speed search method using a tree-structured data structure", a character string is replaced with an identifier and a character is replaced with a block by dividing a long identifier into compact blocks. Is applying a try. In other words, the block is expressed as a node of tree-structured data as an index, and the cost of the comparison process is reduced by comparing the block sequence instead of comparing the identifier.

By the way, in the search using the B-tree, it is necessary to compare the entire identifier several times for each node of the tree. In the method disclosed in JP-A-5-2607, the cost of the comparison process is reduced by comparing each block having a small data length.

[0011]

However, Japanese Patent Laid-Open Publication No.
In the method disclosed in US Pat. No. 38,844, it is necessary to extract a path name from an ID in order to access a file. Therefore, it is necessary to provide a table and management means for managing (associating) the path name and the ID. This table is
Naturally, since it is necessary to include all path names, the size of the correspondence table between path names and IDs becomes large in proportion to the number of files. Therefore, there is a problem that the occupied memory capacity increases as the number of files increases.

On the other hand, the method disclosed in Japanese Patent Application Laid-Open No. 5-2607 is intended to reduce the number of times of comparing identifiers in a search process using a tree structure, and not to improve the efficiency of the identifier comparison process itself. Absent. The original purpose of this method is to improve the efficiency of search processing when a key having a long data length is used. Therefore, it is difficult to use this method as it is in a situation where the comparison processing of the identifiers has to be repeatedly performed many times as in the set operation processing.

However, if the method is to be used for set operation processing, a method of creating a tree structure serving as an index from the identifier set to be operated and searching for elements of another identifier set one by one will be adopted. Become. For this reason, since an index created for the entire identifier in advance cannot be used, a process of creating an index on the spot is required, and as a result, there is a problem that the processing speed is reduced.

The present invention has been made in view of such a point, and even for an identifier having a long data length, the identity of the identifier is quickly determined from a plurality of identifiers in a data set operation process. And a data management device capable of identifying data.

[0015]

According to the present invention, in order to solve the above-mentioned problems, in a data management apparatus which manages data by using an identifier, an identifier input means for inputting an identifier, and the input identifier is divided into a plurality of segments. An identifier dividing means for dividing, a tag generating means for performing a predetermined logical operation on the obtained segment to generate a tag having a data length shorter than the identifier, and a tag generated by the tag generating means. A data management device, comprising: index generation means for generating an index based on the data.

Here, the identifier is input from the identifier input means. The identifier dividing means divides the input identifier into a plurality of segments. The tag generating means performs a predetermined logical operation on the obtained segment to generate a tag having a data length shorter than the identifier. The index generation unit generates an index based on the tag generated by the tag generation unit.

[0017]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a principle diagram for explaining the principle of the data management device of the present invention. In this figure, an identifier for identifying data is input from the identifier input means 1. The identifier dividing means 2 divides the input identifier into a plurality of segments and outputs the segment. Tag generation means 3
Performs a predetermined logical operation on the obtained segment to generate a tag having a data length shorter than the identifier. The index generation unit 4 generates, for example, an index having a tree structure based on the generated tags.

Next, an example of an embodiment of the present invention will be described with reference to FIG. In this figure, an identifier for identifying data is input from an identifier input unit 11. The identifier dividing means 12 divides the input identifier into a plurality of segments having the same length as the tag and outputs the segment. The tag generation means 13 generates a tag having a shorter data length than the identifier by calculating an exclusive OR between the obtained segments. The index generation unit 14 generates an index having a tree structure based on the generated tags. When a new identifier is input, the same tag presence determination unit 15 determines whether a tag corresponding to the identifier already exists in the index. Identifier identity determination means 16
When the same tag existence determining means 15 determines that the same tag already exists, the identifier corresponding to the tag is compared with the newly input identifier to determine whether they are the same. And outputs the result.

Next, the operation of the above embodiment will be described. In the following, the operation will be described with an example of an identification process for identifying an identifier between the first and second sets of identifiers when they are input.

Such an identification process is used when calculating a logical sum or a logical product between the identifier sets. FIG. 3 is a flowchart illustrating an example of a process performed when the identification process is performed in the embodiment illustrated in FIG. When this flowchart is started, the following processing is executed. [S1] The identifier input means 11 inputs a first identifier set. [S2] The identifier dividing means 12 and the tag generating means 13 generate a tag.

That is, the identifier dividing means 12 acquires only one identifier from the input identifier set, and divides the acquired identifier into segments having the same data length as the tag. The tag generation means 13 calculates an exclusive OR between a plurality of segments obtained from one identifier, and sets the obtained result as a tag of the identifier.

Incidentally, another hash function may be used instead of the exclusive OR. [S3] The index generation unit 14 generates an index based on the generated tags. That is, the index generation unit 14 generates an index (binary tree) corresponding to the first set of identifiers by using the generated tags. FIG. 4 shows an example of an index generated by the above processing. In this example, the leftmost index is a part created by the above processing.
This index is composed of nodes n1 to n7, and information necessary for branching is given to each node. For such a binary tree,
There is a detailed description in Iwanami Course Information Science 11, "Data Management Algorithm", P33.

In the rightmost nodes (n4 to n7) of the index,
Tags are associated at least one by one. In this example, the tags t1 and t2 are associated with the node n4, and the tags t3 and t5 are associated with the nodes n5 and n7, respectively. Further, each tag is associated with an identifier. In this example, the tags t1 to
Identifiers i1 to i5 are respectively associated with t5. In this embodiment, the index, the tag, and the identifier as shown in FIG.

By using such an index, the first
Among the identifiers belonging to the set of identifiers, identifiers having the same tag (identifiers with collision of tags) can be efficiently collected. Note that the index actually created is larger than that in FIG. [S4] The identifier input means 11 inputs a second identifier set. [S5] The identifier dividing means 12 and the tag generating means 13 generate tags corresponding to the respective identifiers belonging to the input second identifier set by the same processing as described above. [S6] The identical tag existence determining means 15 and the identifier identity determining means 16 cooperate to execute identifier identification processing. That is, the same tag existence determination means 15 determines whether or not the same tag exists by referring to the index. As a result, when it is determined that the same tag exists, the identifier identity The determining means 16 further determines whether those identifiers are the same. The details of this process are described in FIG.
It will be described later with reference to FIG. [S7] The identifier identity determination means 16 determines whether or not all identifiers have been identified. As a result, if the identification of all identifiers has been completed, the process is completed, and if not, the process returns to step S6. Next, FIG.
The details of the identification processing shown in FIG. 3 will be described with reference to FIG.

This flowchart is called and executed when the "identification process" of step S6 shown in FIG. 3 is started. When this flowchart is started, the following processing is executed. [S21] The same tag existence determination unit 15 selects one tag corresponding to the identifier belonging to the second identifier set generated by the tag generation unit 13, and the first identifier set generated by the index generation unit 14. Is searched for the same tag by referring to the index corresponding to. [S22] The same tag presence determination unit 15 determines in step S2
As a result of the process 1, if it is determined that the same tag exists, the process proceeds to step S23. If it is determined that the same tag does not exist, the process proceeds to step S25. [S23] If the same tag exists, the identifier identity determination unit 16 determines the identifier (identifier belonging to the first identifier set) corresponding to the tag and processes the same. Identifiers (identifiers belonging to the second identifier set) are compared as binary data to determine whether they are the same. As a result, when it is determined that these identifiers are the same, the process proceeds to step S24, and when it is determined that they are not the same, the process proceeds to step S25.

If it is determined in step S22 that a plurality of tags are the same for the tag to be processed, the tag is compared with the first identifier. Compare identifiers with tags.
If they are the same, it is determined to be the same identifier,
Ignore the remaining identifiers with the same tag. [S24] The identifier identity determination unit 16 saves the identified identifier as a third identifier set. [S25] The same tag presence determination unit 15 determines whether there are any more tags belonging to the second identifier set (tags to be identified). As a result, when it is determined that there is still a tag, the process returns to step S21, and the identification processing for the next tag is performed. If it is determined that there is no tag, the process returns to the process of FIG.

According to the above-described processing, after it is determined at a high speed whether or not the same tag exists using the index, if the same tag exists, the corresponding identifiers are compared. Since the number of identifiers having the same tag is sufficiently smaller than the number of elements in the identifier set, the identification process can be executed quickly.

The third set of identifiers generated as described above is the logical product of the input first set of identifiers and the second set of identifiers. Further, since the second identifier set from which the duplicate identifiers of the first identifier set are removed does not have duplicate identifiers, the first and second identifier sets are simply combined to form the first and second identifier sets. An OR of the second set of identifiers can be created.

Next, a configuration example of the second embodiment of the present invention will be described with reference to FIG. In this figure, parts corresponding to those in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted.

In this embodiment, as compared with the case of FIG. 2, a data structure information recording means 20 is newly added, an identifier dividing means 12, a tag generating means 1
3 and the processing in the identifier identity determination means 16 is different. Other configurations are the same as those in FIG.

The data structure information recording means 20 records information relating to the data structure of the identifier. When a request is made, necessary information is stored in the identifier dividing means 12, the tag generating means 13, or the identifier identical information. It is supplied to the sex determination means 16.

The identifier dividing means 12 divides an identifier according to a data structure. The tag generation unit 13 refers to the information recorded in the data structure information recording unit 20 and allocates more tag data areas to the parts where the value of the identifier has a large variation. Generate

If the same tag exists, the identifier identity determination means 16 determines whether the data structure information recording means 20
, The identifiers are sequentially compared from the part having the lower comparison cost.

Next, the operation of the above embodiment will be described. In the following processing, as in the case described above,
The description will be made by taking the identification processing of the first and second identifier sets as an example. The description of the same processing as that described above is omitted.

When an identifier is input, the identifier dividing means 1
Reference numeral 2 refers to the structure information recorded in the data structure information recording means 20 and divides the identifier for each area having the same attribute.

The tag generation means 13 generates a corresponding tag from all the input identifiers. That is, the tag generation unit 13 refers to the structure information of the identifier recorded in advance in the data structure information recording unit 20, and generates a tag according to the structure of the identifier. For example, by allocating more tag data areas to portions where there is a large variation (statistical variation) in identifier values, optimization is performed so that tags generated from different identifiers are unlikely to overlap.

Now, consider the case where a tag is generated with reference to the structure information as shown in FIG. In this example, the identifier is constituted by the following data. (A) length ... 4-byte integer data, and all identifiers have the same value (fixed value). (B) name ... a character string having a fixed value. (C) unknown: 2048-byte binary data, which takes a specific value. (D) number: Array data in which 16 4-byte data are arranged, and each data represents all numerical values that can be represented by 4 bytes.

[0038] Instead of the above structure information,
It is also possible to use structure information as shown in FIG.
That is, in the case of such structure information, an identifier that is variable-length data can be expressed by determining the length of another field based on the “length” or “delimiter” of the “role”. (A) length ... 4-byte integer, indicating the length of name. (B) name ... a character string, and the data length is represented by the length described above. (C) unknown1... Binary data, and the end of the data is determined by the field boundary. (D) boundary ... This is a 4-byte integer, and its value is 0. Represents the separation of data before and after. (E) unknown2... Binary data, the head of which is determined by the field bo undary.

Here, the structure information shown in FIG. 7 will be described. In this example, as shown in the item of the degree of variation, the variation of “length” and “name” is a fixed value, and therefore it is desirable to allocate the tag area only to the other fields.

Therefore, first, the identifier dividing means 12 converts the identifiers into “length”, “name”, “unkn” according to the structure information.
Divide into four areas of own and number. Then, the tag generation unit 13 refers to the “variation degree” of the structure information, allocates a large area to the tag portion corresponding to the number field, and assigns the fields corresponding to length and name having fixed values to the tag generation. Do not use. Here, assuming that the data length of the tag is 4 bytes and the weight of the degree of variation is as shown in FIG. 9, the overall weight is 5 (= 4 + 1).

As a result, unknown is 4 × 1 ÷ 5 = 0.8
While number is 4 × 4 は 5 = 3.2,
1 byte and 3 bytes are allocated after rounding. And, for example, by a hash function,
Is converted into a 1-byte hash value, and number is converted into a 3-byte hash value.

As described above, by determining the tag generation method using the structure information, the collision of tags can be reduced. Note that each field is equivalently 1
When bytes are allocated, le in the tag area
The 2 bytes of the area corresponding to ngth and name have the same value for all tags, and the tag value that can be expressed in effect is 2
(= 4-2) bytes, so that tag collisions are likely to occur.

In the above embodiment, the fields are assigned according to the weights. However, for example, the hash function may be changed according to other information such as the data type.

The index generation means 14 generates an index (index for the first set of identifiers) based on the tags created as described above. Next, an identification process when a second identifier set is input will be described.

When the second identifier set is input, the identifier dividing means 12 divides each identifier belonging to the second identifier set by the same division method as in the case of the first identifier set. The tag generation means 13 also generates a tag by the same processing as described above.

The same tag presence determination means 15 determines whether the same tag exists by referring to the index for the first identifier set. As a result, when it is determined that the same tag exists, the identifier identity determination unit 16 determines whether the identifiers corresponding to these tags are the same. That is, the identifier identity determination means 16 determines whether the tags have accidentally become the same identifier or the same identifier.

The identifier identity judging means 16 first obtains the structure information of the identifier from the data structure information recording means 20, and determines the field comparison order. That is, comparison is performed from a field having a large value variation or a field having a small comparison processing cost such as a small number of bytes. A field having a fixed value (having the same value in all identifiers) is not set as a comparison target. In our example, "number"
Since the degree of variation of the field corresponding to is large, the comparison process is performed with priority given to this field. And if no identity is detected in this field,
Next, the fields corresponding to "unknown" are compared.
Since “length” and “name” are fixed values,
These fields are not compared. If at least one field is different, it is determined that the two identifiers are different at that time. On the other hand, if all the fields to be compared are the same, it is determined that the identifiers are also the same.

In the process of determining the identity of an identifier, the field determination based on the degree of variation may be performed once even when determining a plurality of identifiers. According to the above-described embodiment, more tag data areas are allocated to portions having large variations according to the data structure of identifiers, thereby reducing tag collisions. Furthermore, in the identification processing, when it is determined that there is the same tag, the comparison processing is performed preferentially from a part having a low identity and a part having a low comparison cost by referring to the data structure.
The identification process can be executed at high speed.

Next, a third embodiment of the present invention will be described with reference to FIG. In this figure, parts corresponding to those in FIG. 6 are denoted by the same reference numerals, and their description is omitted.

In this embodiment, a structure description data input means 30, a structure description data analysis means 31, and a writing means 32 are newly added. In addition,
Other configurations are the same as those in FIG.

The structure description data input means 30 inputs structure description data (see FIG. 11) which is data describing the structure of the identifier. The structure description data analysis means 31 analyzes the structure description data input from the structure description data input means 30 and generates structure information of the identifier (see FIGS. 7 and 8 described above).

The writing means 32 writes the structure information of the identifier obtained by the structure description data analyzing means 31 in the data structure information recording means 20 and records it. Next, the operation of the above embodiment will be described. In the following description, as in the case described above, the identification processing of the first and second identifier sets will be described as an example. The description of the same processing as that described above is omitted.

In this embodiment, when the user knows the data structure of the identifier, the user inputs the structure description data describing the data structure, analyzes the structure description data, generates the structure information, and generates the structure information. Write to the recording means 20.
Then, by referring to the structure information, the tag generation processing and the identification processing are performed by the same processing as in the second embodiment. The identifier identification method is the second method.
Since this embodiment is the same as the embodiment, only the registration procedure of the structure information will be described below.

For example, the structure description data for the identifier having the structure information shown in FIG. 7 is as shown in FIG. "Int length 4 * 1 cons" shown in the first line of this example
"t;" indicates that this field is of integer type (int) and 4
It has a length of bytes (4 * 1) and a fixed value (const)
Has been shown. As described above, the structure description data indicates the data type, the field name, the data length, and the degree of variation.

Similarly, “string” represents a character string type,
“Bin” indicates a binary type, “change” indicates a small variation, and “volatile” indicates a large variation. FIG. 12 shows an example of the structure description data for the identifier having the structure information shown in FIG. In this example,
On the fourth line, “bound” indicating “separation” of data is shown. This “bound” indicates that it is a “delimiter”. The syntax of the structure description data does not matter. It is only necessary that the structure information can be represented by the structure description data.

Such structure description data is created by, for example, a text editor or the like, and is input from the structure description data input unit 30. The input structure description data is
It is analyzed by the structure description data analysis means 31. That is, the structure description data analysis means 31 parses (structures) the input structure description data and creates an analysis tree having structure information. Then, structure information is created from the parse tree obtained by the parse.

The structure information created as described above is
The data structure information recording means 20 is written by the writing means 32.
Is written in a predetermined area. The structure information created in this way is referred to in the tag creation processing and the identification processing as described in the second embodiment.

Next, a configuration example of the fourth embodiment of the present invention will be described with reference to FIG. In this figure, parts corresponding to those in FIG. 10 are denoted by the same reference numerals, and the description thereof is omitted.

In this embodiment, a data structure analyzing means 40, an instruction information providing means 41, and a candidate narrowing means 42 are newly added, and the processing of the identifier identity determining means 16 is different. . Other configurations are the same as those in FIG.

The data structure analysis means 40 analyzes the data structure of a plurality of identifiers input from the identifier input means 11 by a statistical method. When the same tag presence determination unit 15 determines that the same tag exists, the instruction information provision unit 41 compares the identifier corresponding to the tag with the newly input identifier, and differs between the two. The part is specified, and instruction information indicating the specified part is added to the index.

When the same tag presence determination means 15 determines that a plurality of identical tags exist, the candidate narrowing means 42 refers to the instruction information provided by the instruction information providing means 41 to determine only different portions. The candidates are narrowed down by comparison.

The identifier identity determination means 16 determines the identity between the narrowed-down candidate and the newly input identifier by referring to the data structure of the identifier. Next, the operation of the above embodiment will be described. In the following description, as in the case described above, the identification processing of the first and second identifier sets will be described as an example. The description of the same processing as that described above is omitted.

The data structure analysis means 40 analyzes a plurality of identifiers input from the identifier input means 11 by a statistical method, and generates identifier structure information. FIG. 14 is a flowchart illustrating an example of a process performed by the data structure analysis unit 40. When this flowchart is started, the following processing is executed. [S41] An identifier to be analyzed is input from the identifier input means 11.

Here, it is assumed that 100 identifiers have been input. [S42] All identifiers input from the identifier input unit 11 are divided into segments of a size suitable for processing.

FIG. 15 shows this division. Each identifier is divided into n segments of 4 bytes each. The index is a number assigned to identify each segment. [S43] The statistical information for each segment is obtained from all the segments.

That is, the distribution of the value of the i-th (1 ≦ i ≦ n) segment is examined to detect how many identical values exist. Then, this is collectively used as an analysis result.
FIG. 16 shows an example of the analysis result. In this example, there is only one type of value (value distribution) for indexes 1 to 4 (all 100 segments have the same value). Also, it can be seen that there are seven types of values for the indexes 261 to 272, and the maximum number of the same values is 79. [S44] Structural information is generated based on the analysis result in step S43.

That is, from the bias of the value of each segment,
Estimate the degree of variation between data types and values. At this time, if the data type and the degree of variation of the adjacent segments are the same, they are put together. Note that, here, for example, the boundary between “large” and “medium” is set to 20 types of values, and the boundary between “small” and “medium” is set to 5 types.

For example, if the degree of variation of the index before and after the analysis result shown in FIG. 16 is the same, by executing a process of combining the indexes, structural information as shown in FIG. 17 can be obtained. . [S45] The data structure analysis unit 40 outputs the generated structure information to the writing unit 32.

By the above processing, even when the structure of the identifier is unknown, it is possible to estimate the structure information from a plurality of identifiers. These processes can be executed in advance, not at the same time as the identifier set identification process, to obtain structural information.

Next, a process for generating a tag and an index will be described with reference to FIG. When this flowchart is started, the following processing is executed. [S61] The tag generator 13 generates a tag from the input identifier based on the structure information recorded in the data structure information recorder 20. [S62] The same tag presence determination unit 15 refers to the index and determines whether the same tag exists. As a result, if the same tag exists, step S6
The process proceeds to step S3, and if not, the process proceeds to step S65. [S63] The instruction information providing unit 41 compares the identifiers corresponding to the colliding tags, and identifies the different parts. [S64] The instruction information providing unit 41 adds, to the index, instruction information indicating a different part of the identifier specified in step S63. [S65] The tag generation unit 13 determines whether or not the tag generation processing for all identifiers has been completed. As a result, when the generation of the tag is completed, the process is completed, and when it is determined that the generation is not completed, step S61 is performed.
Return to

According to the above processing, when tag collision occurs, identifiers corresponding to those tags are compared,
Differences are identified. Then, instruction information indicating the specified part is added to the index.

Next, processing for identifying an identifier with reference to the index created as described above will be described. FIG. 19 is a flowchart illustrating an example of a process for identifying an identifier with reference to the index added with the instruction information created by the process of FIG. 18. When this flowchart is started, the following processing is executed. [S81] The identical tag existence determining unit 15 refers to the index and acquires the same tag as the tag of the identifier to be compared. [S82] The identical tag existence determining means 15 determines whether only one identical tag exists. As a result, if only one identical tag exists, step S8
5 and if there are a plurality, step S
Go to 83. [S83] The candidate narrowing means 42 acquires the instruction information from the index. [S84] The candidate narrowing-down unit 42 refers to the obtained instruction information, compares only portions having different identifiers, and narrows the candidates to one. [S85] The identifier identity determination unit 16 refers to the structure information recorded in the data structure information recording unit 20,
Identify the part to compare identifiers. [S86] The identifier identity determination unit 16 determines the difference between the candidate identifier narrowed down by the candidate narrowing down unit 42 and the identifier to be compared by comparing the part specified in step S85.

According to the above-described processing, at the time of creating an index, information indicating fields having different values is provided in the index between identifiers having the same tag, and at the time of identifying an identifier, this information is used instead of the structural information. Since the identifiers are compared with each other, even if a plurality of identifiers appear as the same identifier candidate due to the collision of tags, the identifiers can be narrowed down to one identifier by a small number of comparison processes.

In the above embodiment, an index is created for one set of identifiers, and operations are performed with a number of sets of identifiers, or when tags are likely to collide (for example, when there are many unknown parts in identifiers). , When the processing is not so optimized).

Next, a case where the above embodiment is applied to file copy management will be described. In the following, 2
In one data management device, one data management device A
Let's consider a case where the data is copied (transferred) to the other data management apparatus B, and the correspondence is managed in order to manage the consistency between the data.

When there is an identifier set A as a search result from the data management device A and an identifier set B as a search result from the data management device B, the identifier a and the identifier set There is a possibility that the identifier b as the element indicates the same data (original data and copied data). Therefore, if the search results are merged as they are, the copied data will be duplicated. Such problems become more serious as the number of management systems used simultaneously and the number of copies of data increase.

FIG. 20 shows an example of processing for coping with such a case. When this flowchart is started, the following processing is executed. [S101] An identifier set A is obtained as a search result from the data management device A, and an identifier set B is obtained as a search result from the data management device B. [S102] The elements of the identifier set B are converted into the corresponding original data identifiers using the correspondence table between the identifiers of the original data and the copy data. [S103] In step S102, an identifier set that cannot be converted to original data is set as an identifier set B '. On the other hand, the converted identifier set is referred to as an identifier set B ″. [S104] According to the embodiment of FIG. 2, FIG. 6, FIG. 10, or FIG. To check. [S105] The logical sum of the identifier set A and the identifier set B ″ is calculated according to the embodiment of FIG. 2, FIG. 6, FIG. 10, or FIG. 13. [S106] The result obtained in step S105 and the identifier Merge with set B ′.

According to such processing, search results without duplication can be obtained. As described above, if the logical sum of the identifier set, which is a result of using the present invention, is used, it is possible to obtain a search result without duplication of data across a plurality of data management devices by a simple process.

The above processing functions can be realized by a computer. In this case, the processing contents of the functions that the data management device should have are described in a program recorded on a computer-readable recording medium, and the above processing is realized by the computer by executing this program on the computer. Is done.

As a computer-readable recording medium, there are a magnetic recording device, a semiconductor memory, and the like. When distributing in the market, CD-ROM (Compact Disk Read
The program can be stored and distributed on a portable recording medium such as Only Memory) or a floppy disk, or stored in a storage device of a computer connected via a network, and transferred to another computer via the network. . When the program is executed by the computer, the program is stored in a hard disk device or the like in the computer, and is loaded into the main memory and executed.

[0081]

As described above, according to the present invention, an identifier is inputted from the identifier input means, the identifier dividing means divides the inputted identifier into a plurality of segments, and the tag generating means divides the inputted segment into a plurality of segments. By performing a predetermined logical operation on the tag, a tag having a data length shorter than the identifier is generated, and the index generating unit generates the index based on the tag generated by the tag generating unit. Can be executed at high speed.

[Brief description of the drawings]

FIG. 1 is a principle diagram showing the principle of the present invention.

FIG. 2 is a diagram illustrating a configuration example of a first embodiment of the present invention.

FIG. 3 is a flowchart illustrating an example of a process performed in the embodiment shown in FIG. 2;

FIG. 4 is a diagram showing an example of an index generated in the embodiment shown in FIG.

FIG. 5 is a flowchart illustrating an example of an identification process performed in the embodiment shown in FIG. 2;

FIG. 6 is a diagram illustrating a configuration example according to a second embodiment of the present invention.

FIG. 7 is a diagram showing an example of structure information used in the embodiment shown in FIG. 6;

FIG. 8 is a diagram showing another example of the structure information.

FIG. 9 is a diagram showing a relationship between a variation and a weight.

FIG. 10 is a diagram illustrating a configuration example of a third embodiment of the present invention.

11 is a diagram showing an example of structure description data used in the embodiment shown in FIG.

FIG. 12 is a diagram showing another example of the structure description data used in the embodiment shown in FIG. 10;

FIG. 13 is a diagram illustrating a configuration example according to a fourth embodiment of the present invention.

FIG. 14 is a diagram showing an example of processing executed in the embodiment shown in FIG.

FIG. 15 is a diagram showing a state of an identifier divided into segments according to the embodiment shown in FIG. 13;

FIG. 16 is a diagram showing an example of segment information analyzed by the embodiment shown in FIG. 13;

FIG. 17 is a diagram illustrating an example of structural information generated from the analysis result illustrated in FIG. 16;

FIG. 18 is a flowchart illustrating an example of a process performed in the embodiment shown in FIG.

FIG. 19 is a flowchart illustrating another example of the processing executed in the embodiment shown in FIG.

FIG. 20 is a flowchart illustrating an example of processing when the present invention is applied to data copy management.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Identifier input means 2 Identifier division means 3 Tag generation means 4 Index generation means 11 Identifier input means 12 Identifier division means 13 Tag generation means 14 Index generation means 15 Identical tag existence determination means 16 Identifier identity determination means 20 Data structure information recording means Reference Signs List 30 Structure description data input means 31 Structure description data analysis means 32 Writing means 40 Data structure analysis means 41 Instruction information adding means 42 Candidate narrowing means

Claims (11)

[Claims] 1. A data management device for managing data by an identifier, an identifier input means for inputting an identifier, an identifier dividing means for dividing the input identifier into a plurality of segments, By performing a logical operation of, tag generation means for generating a tag having a shorter data length than the identifier, based on the tag generated by the tag generation means,
A data management device, comprising: index generation means for generating an index. 2. The identifier division unit divides the identifier into segments having the same data length as the tag, and the tag generation unit generates the tag by calculating exclusive OR between the segments. 2. The data management device according to claim 1, wherein: 3. The identifier dividing means divides an identifier into segments according to the data structure of the identifier, and the tag generating means responds to a variation state of the value of each segment obtained by the identifier dividing means. 2. The data management device according to claim 1, wherein a data area of the tag is assigned to each segment. 4. A data structure information recording unit for recording information on a data structure of the identifier, wherein the identifier division unit and the tag generation unit refer to information recorded in the data structure information recording unit. 4. The data management device according to claim 3, wherein the processing is performed. 5. A structure description data input unit to which structure description data describing the data structure is input, and a structure description data analysis unit configured to analyze the input structure description data and generate information on a data structure of the identifier. 5. The data management apparatus according to claim 4, further comprising: writing means for writing the information on the data structure of the identifier to the data structure information recording means. 6. A data structure analyzing means for analyzing, by a statistical method, a data structure of a plurality of identifiers inputted from said identifier input means, and said data structure information recording means for storing information on the obtained data structure of said identifier. 5. The data management device according to claim 4, further comprising: a writing unit that writes data into the data management device. 7. When a tag of a newly input identifier is generated by the tag generation unit, it is determined whether or not the same tag exists by referring to the index generated by the index generation unit. Identical tag presence determining means, and when it is determined that the same tag exists, identifier identity determining means for determining whether an identifier having the tag and the newly input identifier are the same, 2. The data management device according to claim 1, further comprising: 8. The method according to claim 7, wherein the identifier identity determination unit sequentially compares the identifiers with reference to a data structure of the identifiers, starting from a portion having a low comparison cost.
Data management device as described. 9. When the same tag presence determination means determines that the same tag exists, an identifier corresponding to the tag is compared with a newly input identifier, and a different part is identified between them. 8. The data management apparatus according to claim 7, further comprising an instruction information assigning unit that assigns, to the index, instruction information that specifies the specified part. 10. When the same tag presence determination means determines that there are a plurality of identical tags, comparing only portions having different identifiers with reference to the instruction information provided by the instruction information providing means. Thus, the apparatus further comprises candidate narrowing-down means for narrowing down candidates, wherein the identifier identity determining means determines the identity between the narrowed-down candidate and the newly input identifier by referring to a data structure of the identifier. The data management device according to claim 9, wherein 11. A computer-readable recording medium storing a data management program for causing a computer to manage data by an identifier, an identifier input means for inputting the identifier, and an identifier for dividing the input identifier into a plurality of segments. Dividing means, by performing a predetermined logical operation on the obtained segment, a tag generating means for generating a tag having a data length shorter than the identifier, based on the tag generated by the tag generating means,
A computer-readable recording medium that records a data management program for causing a computer to function as index generation means for generating an index.