JPWO2007132564A1 - Data processing apparatus and method - Google Patents

Abstract

A technique for searching data similar to certain data at high speed is provided. When the data acquisition unit 41 acquires the target data, the search unit 42 searches whether the target data exists in the database 60. If it does not exist in the database 60, first, the element number comparison unit 44 excludes data whose difference in the number of elements from the target data is a predetermined value or more from the candidates. Next, the eigenvalue comparison unit 45 calculates an eigenvalue of the target data, and excludes data having a false distance between eigenvalues equal to or greater than a predetermined value from candidates. Further, the use element comparison unit 46 excludes data whose difference in use element from the target data is a predetermined value or more from the candidates. The edit distance calculation unit 47 calculates an edit distance between the narrowed-down data and the target data, and extracts data whose edit distance is equal to or less than a predetermined value as similar data. The candidate presenting unit 48 presents the extracted candidate data to the user.

Description

  The present invention relates to a data processing technique, and more particularly to a technique for retrieving data similar to certain data.

  One of the functions provided in a word processor or the like is a spell checker that detects misspellings or typographical errors in English words and presents correction candidates. Conventional spell checkers rule out mistake patterns and the like that are easy for a user to fall, and when spelling errors are detected, correction candidates are extracted and presented based on the rules.

  However, since the user does not necessarily make the same mistake as the ruled pattern, it may not be possible to guess an appropriate correction candidate. A technique for extracting similar data by a more objective and faster algorithm is required.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for searching data similar to certain data at high speed.

  One embodiment of the present invention relates to a data processing apparatus. The data processing apparatus includes a search unit that searches whether or not target data is stored in a database, and candidate data similar to the target data when the target data is not stored in the database. An extraction unit that extracts from the data, the extraction unit calculates a distance between the target data and the data stored in the database, and data with the distance smaller than a predetermined upper limit as the candidate data When the component of data is classified into a plurality of groups before the distance calculator calculates the distance, and whether the component belonging to each group is included in the target data Is calculated for each group, and the specific numerical value of the target data and the data stored in the database are fixed. An eigenvalue comparison unit that calculates a pseudo-distance between the numerical values and excludes data in which the pseudo-distance is larger than the predetermined upper limit from a target from which the distance calculation unit calculates the distance. And

  According to this aspect, objectively similar data can be extracted using the distance between the data. In addition, before calculating the distance between data, it is possible to extract similar data at high speed by narrowing down the data to be calculated using the pseudo distance between the eigenvalues defined in each data. it can.

  The eigenvalue may be a binary number having the same number of digits as the number of the group, and the eigenvalue comparison unit assigns a bit to each of the groups, and a component belonging to the group is included in the data In this case, the bit assigned to the group may be set to “1”, and if not included, the bit assigned to the group may be set to “0” to calculate the eigenvalue. The eigenvalue comparison unit calculates the false distance between two eigenvalues by bit-inverting one eigenvalue and then calculating the logical product of both, and the number of “1” included in the other The pseudo-distance may be the larger of the numbers of “1” included in the bit string obtained by performing the logical inversion of the two after bit-reversing the eigenvalue. When calculating the pseudo distance between two eigenvalues, the eigenvalue comparison unit bit-inverts the eigenvalue of the bit string of the two eigenvalues with the larger “1” and then calculates the logical product of the two. The number of “1” included in the bit string may be set as the false distance.

  The extraction unit excludes data whose difference in the number of components exceeds the predetermined upper limit from the target for which the eigenvalue comparison unit calculates the pseudo distance before the eigenvalue comparison unit calculates the pseudo distance. It may further include an element number comparison unit. The extraction unit includes the number of components included in the target data and not included in the data stored in the database before the distance calculation unit calculates the distance, and stored in the database. The number of components included in the data and not included in the target data is calculated, and any one of the data exceeding the predetermined upper limit is excluded from the target from which the distance calculation unit calculates the distance An element comparison unit may be further included. Thereby, similar data can be extracted at higher speed.

  The database may store the data classified for each number of components and for each eigenvalue. Thereby, the search efficiency of the database can be improved, and the time required for the search can be shortened. Similarly, the efficiency and speed of extracting similar data from the database can be improved.

  The data processing device includes a learning unit that arranges the target data, the data extracted as the candidate data, or the data selected by the user from the candidate data at a higher rank in a data group having the same eigenvalue. Further, it may be provided. A data group having the same eigenvalue as the target data, the data extracted as the candidate data, or the data selected by the user from the candidate data is arranged higher in the data group having the same number of components A learning unit may be further provided. Thereby, the search efficiency of a database can be improved. In addition, when presenting similar data candidates, the display order can be optimized so that frequently used data is higher.

  The distance calculation unit may calculate the minimum number of procedures necessary to transform one data into the other data by inserting, deleting, or replacing a component, and may use the distance as the distance.

  Another aspect of the present invention also relates to a data processing apparatus. The data processing apparatus acquires a data group to be stored in a database, and in the acquired data group, calculates a usage frequency of a component constituting each data, and uses the configuration based on the usage frequency. Classification generator for classifying elements into a plurality of groups, and for each data, eigenvalue calculation for calculating for each group whether or not a component belonging to the group is included in the target data And a data sorting unit that classifies data included in the data group according to the number of elements used and the unique numerical value and stores the data in the database.

  The eigenvalue may be a binary number having the same number of digits as the number of the group, and the eigenvalue calculating unit assigns a bit to each of the groups, and a component belonging to the group is included in the data In this case, the bit assigned to the group may be set to “1”, and if not included, the bit assigned to the group may be set to “0” to calculate the eigenvalue.

  Yet another embodiment of the present invention relates to a data processing method. This data processing method includes a step of searching whether target data is stored in a database, and candidate data similar to the target data from the database when the target data is not stored in the database. Extracting, calculating the distance between the target data and the data stored in the database, and extracting the data whose distance is smaller than a predetermined upper limit as the candidate data And, when the data components are classified into a plurality of groups before the distance is calculated, the unique data indicating whether the target data is included in the target data. Calculate the numerical value, and the unique number of the target data and the unique number of the data stored in the database Calculating a pseudorange between said data larger than pseudorange said predetermined upper limit, characterized in that it comprises a and a step of excluding from the target of calculating the distance.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, etc. are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which searches the data similar to a certain data at high speed can be provided.

It is a figure which shows the structure of the data processor which concerns on embodiment. It is a figure which shows the classification | category of a component. It is a figure which shows the example of the internal data of a database. It is a figure which shows the data by which the candidate was narrowed down by the element number comparison part. It is a figure which shows the data by which the candidate was narrowed down by the intrinsic | native numerical value comparison part. It is a figure which shows the data by which the candidate was narrowed down by the use element comparison part. It is a figure which shows the candidate data extracted by the edit distance calculation part. It is a flowchart which shows the procedure of the data processing method which concerns on embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Data processor, 30 Database production | generation part, 31 Usage frequency calculation part, 32 Classification production | generation part, 33 Eigenvalue calculation part, 34 Data sort part, 41 Data acquisition part, 42 Search part, 43 Candidate extraction part, 44 Element number comparison Part, 45 eigenvalue comparison part, 46 use element comparison part, 47 edit distance calculation part, 48 candidate presentation part, 49 learning part, 60 database.

  The data processing apparatus according to the embodiment searches whether or not the target data exists in the database, and if not, extracts similar data from the database and presents it. For example, it is possible to provide a spell checker function that checks whether or not the spelling of an English word is correct using a dictionary database in which English words are registered, and presents correction candidates when it is determined that the spelling is not correct. In addition, using a DNA database in which DNA base sequences are registered, you can identify similar genes in different species, and estimate the time that has elapsed since the species diverged by measuring their distance. The function to do can be provided. Furthermore, similar images and music can be extracted using a database of images and music.

  In this embodiment, in order to extract similar data, the “distance” between the target data and the data registered in the database is calculated, and those having a short distance are determined to be similar. The distance between the data only needs to reflect a difference between the data. For example, a known technique such as a Hamming distance (signal distance) or a Levenstein distance (editing distance) can be used. Alternatively, the similarity of local alignment may be calculated using Smith Waterman algorithm or the like. Hereinafter, an example in which the spelling check of English words is performed using the edit distance will be described.

  The edit distance is the minimum number of steps required for transforming one character string into another character string by inserting, deleting, or replacing characters, and can be generally calculated using an algorithm based on dynamic programming. However, as more English words are registered in the dictionary in order to improve the accuracy of the spell checker, the number of objects for calculating the edit distance increases. In order to detect a spelling error of one English word, when the correction distance is calculated and the correction candidate is presented with respect to that word and all the English words registered in the dictionary, It takes a lot of time, and there is a risk that the convenience of the user may be impaired.

  In the present embodiment, before actually calculating the edit distance, those having low similarity are excluded from the calculation target in advance, and the target is narrowed down and then the edit distance is calculated, thereby increasing the number of registered dictionaries. However, we propose a technique for extracting and presenting correction candidates in a short time. In this embodiment, an eigenvalue is defined for each data, and a rough distance between the data is measured by calculating a pseudo-distance between eigenvalues, and those whose pseudo-range exceeds a predetermined upper limit are excluded in advance. To do.

  FIG. 1 shows a configuration of a data processing apparatus according to the embodiment. The data processing apparatus 10 includes a database generation unit 30, a data acquisition unit 41, a search unit 42, a candidate extraction unit 43, a candidate presentation unit 48, a learning unit 49, and a database 60. The database generation unit 30 includes a use frequency calculation unit 31, a classification generation unit 32, a unique numerical value calculation unit 33, and a data sort unit 34. The candidate extraction unit 43 includes an element number comparison unit 44, an eigenvalue comparison unit 45, a used element comparison unit 46, and an edit distance calculation unit 47. In terms of hardware components, these configurations are realized by a CPU of a computer, a memory, a program loaded in the memory, and the like, but here, functional blocks realized by their cooperation are illustrated. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  First, a procedure for generating the database 60 storing dictionary data of English words will be described. The usage frequency calculation unit 31 acquires a set of data to be stored in the database 60 and calculates how much data each element is used in the data set. In this example, when the usage frequency calculation unit 31 acquires a list of English words, the usage frequency calculation unit 31 calculates the usage frequency of characters used in the English words. When the same element is used multiple times in one data, in this embodiment, the usage frequency is counted as 1. However, depending on the content of the data, the number of times of the used element may be counted. Good.

  The classification generation unit 32 classifies elements used for data into a plurality of groups. At this time, the classification generation unit 32 classifies the elements so that the total use frequency of the elements belonging to each classification is substantially uniform. For example, in a list of 10 English words, if “a” is used 8 times, “b” is used 4 times, “c” is used 6 times, and “d” is used 10 times, “a” and “c” ”,“ B ”, and“ d ”, the total usage frequency of characters belonging to each classification is 14 times, which is uniform. The number of groups into which the classification generation unit 32 classifies elements is determined in consideration of the efficiency of search and candidate extraction described later. In the present embodiment, in order to simplify the description, as shown in FIG. 2, the 26-character alphabet is classified into five groups.

  The eigenvalue calculation unit 33 calculates, for each data, an eigenvalue composed of a bit string having the same number of digits as the number of classifications classified by the classification generation unit 32. The eigenvalue calculation unit 33 assigns one bit to each classification, and if an element belonging to the classification is used in the data, the bit assigned to the classification is set to “1”. The bits assigned to the classification are set to “0”. In the present embodiment, as shown in FIG. 2, since the alphabet is classified into five groups, the eigenvalue is a 5-bit binary number. If any of the letters “a”, “b”, “c”, “d”, “e” is used in the English word, the bit of the “1” bit from the most significant bit is set to “1”. “0” if not used. Similarly, if any letter from “f” to “j” is used in the English word, the bit of the “2” bit from the most significant bit is set to “1”, and if not used, “0”. And In this manner, the unique numerical value calculation unit 33 calculates the specific numerical values of all English words included in the list. For example, the specific value of the English word “test” is “10010”.

  The data sorting unit 34 sorts the data included in the data set by the number of elements used, and further collects data having the same unique numerical value in the data set having the same number of elements. That is, the list of English words is sorted by the number of characters, and English words having the same number of characters are classified by eigenvalues. In this way, English dictionary data is generated. An example of the generated dictionary data is shown in FIG.

  Next, the procedure for checking the spelling of English words will be described. The data acquisition unit 41 acquires data to be searched. Here, an English word to be subjected to spell check is acquired. The search unit 42 searches whether the acquired data exists in the database 60. Here, as described above, the English words are sorted by the number of characters in the dictionary stored in the database 60 and classified and stored by the specific numerical value. Therefore, the search unit 42 determines the number of characters of the acquired English words. And the specific numerical value is calculated, and the corresponding record in the database 60 is searched. For example, when the data acquisition unit 41 acquires the English word “test”, the search unit 42 only needs to search for records having the number of elements “4” and the unique numerical value “10010”. Thereby, data can be searched efficiently.

  If the acquired data exists in the database 60, the process ends. If it does not exist, the candidate extraction unit 43 extracts data similar to the acquired data from the data existing in the database 60. In the case of spell check, if the acquired English word does not exist in the database 60, there is a possibility of a spelling error, so the candidate extraction unit 43 extracts correction candidates. The candidate extraction unit 43 determines the similarity based on the distance between the data, and extracts data whose distance is closer than a predetermined value as a candidate. Hereinafter, an example of extracting English words whose edit distance to the target data “tewt” is 2 or less will be described.

  The element number comparison unit 44 calculates the number of elements of the target data and compares it with the data in the database 60. When data whose edit distance is n or less is extracted, data whose difference in the number of elements is n + 1 or more is excluded from candidates without calculating the edit distance. Therefore, since the number of characters of the target data “tewt” is “4”, the number-of-elements comparison unit 44 excludes English words with the number of characters “1” and English words with the number of characters “7” or more from the candidates. As a result, the data in the database 60 shown in FIG. 3 is narrowed down as shown in FIG.

  The eigenvalue comparison unit 45 calculates the eigenvalue of the target data and compares it with the data in the database 60. The eigenvalue comparison unit 45 calculates the “false distance” between the eigenvalue of the target data and the eigenvalue of the data in the database 60 as follows, and the calculated “false distance” is used as a candidate. Data larger than the upper limit of the edit distance to be extracted is excluded from the candidates. The pseudo distance between eigenvalues described below is called a pseudo distance because it does not satisfy symmetry in the distance axiom.

  The eigenvalue comparison unit 45 calculates the logical product of the number of “1” included in the bit string obtained by performing the logical inversion of one of the eigenvalues and the other eigenvalue after performing the bit inversion of the other eigenvalue. The larger one of the number of “1” included in the bit string is set as a false distance. For example, when the false distance between the eigenvalue “10011” of the target data “tewt” and the eigenvalue “00100” is calculated, first, the logical product of the inverted “01100” of the former “10011” and the latter “00100” The number of “1” in “00100” is “1”. The number of “1” s of the logical product “10011” of the inverted “11011” of the latter “00100” and the former “10011” is “3”. Therefore, “3”, which is the larger of “1” and “3”, is a pseudo distance between the eigenvalues “10011” and “00100”.

  The reason why the false distance is calculated instead of the Hamming distance between eigenvalues is that the substitution process is counted as “1” when the edit distance between the data is calculated. When considering only deletion or insertion of data, the Hamming distance may be used, but when considering substitution, the Hamming distance obtained from the exclusive OR between eigenvalues is compared with the upper limit of the editing distance, and data candidates are selected. When narrowing down, there is a possibility of excluding data whose edit distance does not exceed the upper limit. For example, the unique value of the character string “afp” is “11010”, and the unique value of the character string “up” is “00011”. Since these exclusive ORs are “11001”, the Hamming distance between eigenvalues is “3”. However, if “a” in “afp” is replaced with “u” and “f” is deleted, “up” is obtained, so the edit distance between the data is “2”. Therefore, the Hamming distance between eigenvalues may be larger than the editing distance between data. On the other hand, when the above pseudo distance is calculated, the logical product of the eigenvalue inversion “00101” of “afp” and the eigenvalue “00011” of “up” is “00001”, and the inversion of the latter “11100” Since the logical product of the former “11010” is “11000”, the false distance between eigenvalues is “2”. Thus, since the pseudo distance between eigenvalues does not exceed the edit distance between data, it can be used for narrowing down data.

  When calculating the pseudo distance between two eigenvalues, the eigenvalue comparison unit 45 performs bit-inversion on the eigenvalue having the larger “1” in the bit string of the two eigenvalues and then calculates the logical product of the two. The number of “1” included in the bit string may be a false distance. For example, if the distance between the eigenvalue “10011” of the target data “tewt” and the eigenvalue “00100” is calculated, first, the latter eigenvalue has a smaller number of “1” s, so this is bit-inverted. And “11011” is obtained. When the logical product of this and “10011” is calculated, it becomes “10011”, and the number of “1” becomes “3”. Since this is larger than the upper limit “2” of the edit distance, the eigenvalue comparison unit 45 excludes English words whose eigenvalue is “11011” from the candidates. As a result, the candidate data shown in FIG. 4 is further narrowed down as shown in FIG.

  The use element comparison unit 46 compares the constituent elements of the target data with the constituent elements of the data in the database. The used element comparison unit 46 calculates the difference between the element used for the target data and the element used for the data in the database 60, and the calculated difference is the upper limit of the edit distance to be extracted as a candidate. Data larger than are excluded from the candidates. The used element comparison unit 46 uses the number of elements that are used by the target data and are not used by the data in the database 60 and the elements that are used by the data in the database 60 and are not used by the target data. The number is calculated, and if any of them exceeds the upper limit of the edit distance, the data is excluded from the candidates. For example, the difference in the use elements of the target data “tewt” and the data “word” is as follows. There are two elements “t” and “e” that are used in the target data “tewt” but are not used in the data “word”. They are used in the data “word” and are included in the target data “tewt”. There are three elements “o”, “r”, and “d” that are not used. Therefore, the difference between the elements used is “3”, which is larger than the upper limit “2” of the edit distance, and therefore the data “word” is excluded from the candidates. As a result, the candidate data shown in FIG. 5 is further narrowed down as shown in FIG.

  The edit distance calculation unit 47 calculates the edit distance between the candidates narrowed down as described above and the target data, and extracts data whose edit distance is closer than a predetermined value as candidates. Instead of the edit distance calculation unit 47, a configuration may be provided in which the distance or similarity between data is calculated by other methods such as a signal distance or a similarity by Smith Waterman algorithm. In general, the distance becomes smaller as the similarity between the data is higher, and the similarity becomes larger as the similarity between the data is higher. Here, “higher similarity” means that the distance is higher. expressing. Therefore, when the edit distance is calculated, data whose calculated value is smaller than a predetermined upper limit is extracted, and when the similarity is calculated, data whose calculated value is larger than a predetermined lower limit is extracted. The candidate data shown in FIG. 7 is extracted by the above procedure.

  The candidate presenting unit 48 presents the candidates extracted by the candidate extracting unit 43 to the user. The candidate presenting unit 48 preferably displays the data so that the calculated distance is close to the top. Thereby, candidates with higher similarity can be displayed at the top. The candidate presenting unit 48 may present candidate data together with the calculated distance value. Thereby, when selecting data from the candidate data presented by the user, the similarity can be determined with reference to the distance value. Data selected by the user from the presented candidate data is output to a word processor or the like.

  The learning unit 49 causes the database 60 to learn by rearranging the data so that the candidate data extracted by the candidate extraction unit 43 and the data selected by the user thereafter are arranged on the upper level of the database 60. The learning unit 49 places the data group having the same eigenvalue as the target data, the data extracted as the candidate data, or the data selected by the user from the candidate data, in the data group having the same number of components. Move to be placed. Further, the learning unit 49 moves the target data, the data extracted as the candidate data, or the data selected by the user from the candidate data so as to be placed higher in the data group having the same eigenvalue. Let As a result, data that is used or data that is extracted or selected as data that is similar to the data that is used can be searched more quickly in the next and subsequent searches. And speed can be improved. In addition, when the candidate presentation unit 48 presents candidate data, the display order can be optimized so that data that is frequently used, extracted, or selected has a higher rank. Thereby, a user's convenience can be improved.

  Since the process of comparing the number of elements, eigenvalues, and used elements can be performed faster than calculating the edit distance, it is similar by narrowing down the candidates in advance by these processes before calculating the edit distance. The process of extracting data can be speeded up. The process of comparing the number of elements, eigenvalues, and elements used increases the processing time in this order, so by executing faster processing first, the efficiency of narrowing down can be improved, and similar data can be obtained. The extraction process can be further accelerated.

  FIG. 8 is a flowchart illustrating a procedure of the data processing method according to the embodiment. First, when the data acquisition unit 41 acquires target data (S10), the search unit 42 searches whether the target data exists in the database 60 (S14). If it does not exist in the database 60 (N in S14), the candidate extraction unit 43 extracts data similar to the target data from the database 60. First, the number-of-elements comparison unit 44 calculates the number of elements used in the target data, and narrows down the target by excluding data whose difference in the number of elements is a predetermined value or more from candidates (S16). Next, the eigenvalue comparison unit 45 calculates an eigenvalue of the target data, and excludes data whose pseudo distance between eigenvalues is equal to or greater than a predetermined value from candidates to narrow down the target (S18). Further, the used element comparison unit 46 compares the element used for the target data with the element used for the data in the database 60, and excludes data having a difference of a predetermined value or more from the candidates to narrow down the target. (S20). The edit distance calculation unit 47 calculates an edit distance between the narrowed-down data and the target data, and extracts data having an edit distance equal to or less than a predetermined value as similar data (S22). The candidate presentation unit 48 presents the extracted candidate data to the user (S24). If the target data exists in the database 60 (Y in S14), the process of extracting similar data is skipped. The learning unit 49 stores the database 60 so that the data extracted as the target data, the data similar to the target data, or the data selected by the user from the extracted data is arranged on the upper side of the database 60. The database 60 is learned by moving the position at (S26).

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  The present invention can be used in a data processing apparatus that extracts data similar to certain data.

Claims (14)

A search unit for searching whether the target data is stored in the database;
An extraction unit that extracts candidate data similar to the target data from the database when the target data is not stored in the database;
The extraction unit includes:
A distance calculation unit that calculates a distance between the target data and data stored in the database, and extracts data having the distance smaller than a predetermined upper limit as the candidate data;
When the component of data is classified into a plurality of groups before the distance calculation unit calculates the distance, whether or not the component belonging to each group is included in the target data is indicated for each group. Calculating an eigenvalue, calculating a false distance between the eigenvalue of the target data and the eigenvalue of the data stored in the database, and calculating the data in which the false distance is greater than the predetermined upper limit, An eigenvalue comparison unit that the calculation unit excludes from the target for calculating the distance;
A data processing apparatus comprising:   The eigenvalue is a binary number having the same number of digits as the number of the group, and the eigenvalue comparison unit assigns a bit to each of the groups, and when a component belonging to the group is included in the data, 2. The data processing apparatus according to claim 1, wherein the eigenvalue is calculated by setting a bit assigned to the group to “1” and, if not included, a bit assigned to the group to “0”. .   The eigenvalue comparison unit calculates the false distance between two eigenvalues by bit-inverting one eigenvalue and then calculating the logical product of both, and the number of “1” included in the other 3. The data processing apparatus according to claim 2, wherein a larger one of the number of “1” included in the bit string obtained by calculating the logical product of both of the eigenvalues after bit inversion is set as the false distance.   When calculating the pseudo distance between two eigenvalues, the eigenvalue comparison unit bit-inverts the eigenvalue of the bit string of the two eigenvalues with the larger “1” and then calculates the logical product of the two. 3. The data processing apparatus according to claim 2, wherein the number of “1” included in the bit string is the false distance.   The extraction unit excludes data whose difference in the number of components exceeds the predetermined upper limit from the target for which the eigenvalue comparison unit calculates the pseudo distance before the eigenvalue comparison unit calculates the pseudo distance. 5. The data processing apparatus according to claim 1, further comprising an element number comparison unit.   The extraction unit includes the number of components included in the target data and not included in the data stored in the database before the distance calculation unit calculates the distance, and stored in the database. The number of components included in the data and not included in the target data is calculated, and any one of the data exceeding the predetermined upper limit is excluded from the target from which the distance calculation unit calculates the distance 6. The data processing apparatus according to claim 1, further comprising an element comparison unit.   The data processing apparatus according to any one of claims 1 to 6, wherein the database classifies and stores the data for each number of components and for each unique numerical value.   It further comprises a learning unit that arranges the target data, the data extracted as the candidate data, or the data selected by the user from the candidate data in a data group having the same eigenvalue. The data processing apparatus according to claim 7.   A data group having the same eigenvalue as the target data, the data extracted as the candidate data, or the data selected by the user from the candidate data is arranged higher in the data group having the same number of components The data processing apparatus according to claim 7, further comprising a learning unit.   The distance calculation unit calculates the minimum number of steps required to transform one data into another data by inserting, deleting, or replacing a component, and sets the distance as the distance. The data processing device according to any one of 1 to 9. A usage frequency calculation unit that acquires a data group to be stored in the database and calculates the usage frequency of the constituent elements constituting each data in the acquired data group;
A classifying unit that classifies the components into a plurality of groups based on the usage frequency;
For each data, a unique numerical value calculation unit for calculating a specific numerical value for each group indicating whether or not a component belonging to the group is included in the data;
A data sorting unit that classifies the data included in the data group according to the number of elements used and the unique numerical value and stores the data in the database;
A data processing apparatus comprising:   The eigenvalue is a binary number having the same number of digits as the number of the group, and the eigenvalue calculation unit allocates a bit to each of the groups, and when a component belonging to the group is included in the data, 12. The data processing apparatus according to claim 11, wherein the unique numerical value is calculated by setting a bit assigned to the group to “1” and, if not included, a bit assigned to the group to “0”. . Searching whether the target data is stored in the database;
Extracting the candidate data similar to the target data from the database when the target data is not stored in the database,
The extracting step includes:
Calculating a distance between the target data and data stored in the database, and extracting data having the distance smaller than a predetermined upper limit as the candidate data;
Before calculating the distance, when the data components are classified into a plurality of groups, a unique numerical value is calculated for each group indicating whether or not a component belonging to each group is included in the target data. , Calculating a pseudo distance between the eigenvalue of the target data and the eigenvalue of the data stored in the database, and the data for which the pseudo distance is greater than the predetermined upper limit from the target for calculating the distance Step to exclude,
A data processing method comprising: A function to search whether the target data is stored in the database;
A function for extracting candidate data similar to the target data from the database when the target data is not stored in the database;
The extracting function is:
A function of calculating a distance between the target data and data stored in the database, and extracting data having the distance smaller than a predetermined upper limit as the candidate data;
Before calculating the distance, when the data components are classified into a plurality of groups, a unique numerical value is calculated for each group indicating whether or not a component belonging to each group is included in the target data. , Calculating a pseudo distance between the eigenvalue of the target data and the eigenvalue of the data stored in the database, and the data for which the pseudo distance is greater than the predetermined upper limit from the target for calculating the distance The features to exclude,
A data processing program comprising:

Images