JP2009169689A - Data classification method and data processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To shorten the processing time of similarity calculation by efficiently calculating similarity between document data. <P>SOLUTION: A data processing apparatus 100 creates an inverted index 150d associating word IDs in document data with document data containing words corresponding to the word IDs, and executes sequential pattern extraction on the inverted index 150d. The occurrence frequency of combinations of document data occurring after the execution of sequential pattern extraction is determined (especially, only the occurrence frequency of patterns of length 2 in the sequential pattern extraction is determined), and similarity between the document data is calculated according to the determination. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a data processing apparatus that classifies document data according to the similarity of each document data stored in a storage device, and a data classification method thereof.

  Conventionally, in a document search system composed of a server device and a client device, for example, when the server device acquires a search keyword from the client device, a document corresponding to the search keyword from a plurality of document data stored in the database. Data is searched and the search result is provided to the client device.

  In recent years, the number of document data stored in the database has increased dramatically, and the number of document data searched based on search keywords has become enormous. In order to make it easy to refer to the document data, the degree of similarity of each document data is calculated using a technique such as a vector space model (for example, see Non-Patent Document 1), and the document data is calculated for each similar document data. Classification.

  For example, in the vector space model described in Non-Patent Document 1, the presence / absence of a plurality of keywords is converted as a vector for each document data, and the similarity is calculated for all combinations of converted vectors.

G. Sallon, A, Wong and C. S. Yang Cornell University "A Vector Space Model for Automatic Indexing", Communications of the ACM November 1975 Volume 18 Number 11

  However, if the similarity of document data is calculated using a vector space model as in the conventional technique described above, the calculation is performed between all combinations of document data, and thus the square of the number of document data is calculated. There is a problem in that the amount of processing occurs and the processing time for calculating the similarity becomes long.

  That is, it is an extremely important issue to reduce the processing time required for similarity calculation by efficiently calculating the similarity of document data.

  The present invention has been made to solve the above-described problems caused by the prior art, and is a data classification method capable of efficiently calculating the similarity of document data and reducing the processing time required for the similarity calculation. And it aims at providing a data processor.

  In order to solve the above-described problems and achieve the object, this data classification method is a data classification method for a data processing device that classifies document data according to the similarity of each document data stored in a storage device, A data processing device reads out document data stored in the storage device, creates a list in which a keyword in each document data and a document data including the keyword are associated with each other, and a sequential with respect to the list A determination step of performing pattern extraction and determining the number of occurrences of the combination of the document data that has appeared, and a calculation step of calculating the similarity between the document data based on the determination result of the determination step Is a requirement.

  Further, this data classification method is that, in the above-described data classification method, the determination step is required to determine only the number of occurrences of the pattern of length 2 by sequential pattern extraction.

  Further, the data classification method further includes a classification step of classifying the document data by obtaining a combination of each document data that maximizes the similarity value in the data classification method, and the classification step includes: It is a requirement to classify the document data so that the number of classifications specified by the user is equal to the number of classifications of document data.

  Further, the data processing device is a data processing device that classifies document data according to the similarity of each document data stored in the storage device, and reads out the document data stored in the storage device. A list creating means for creating a list in which a keyword is associated with document data including the keyword, and a determination means for executing sequential pattern extraction on the list and determining the number of occurrences of a combination of appearing document data And a calculation means for calculating the similarity between the document data based on the determination result of the determination means.

  Further, this data processing apparatus is characterized in that, in the above data processing apparatus, the determination means determines only the number of occurrences of a length 2 pattern by sequential pattern extraction.

  According to this data classification method, a list in which keywords in each document data are associated with document data corresponding to the keywords is created, and sequential pattern extraction is performed on the list. Then, the number of occurrences of the combination of document data that appears as a result of sequential pattern extraction is determined, and the similarity between each document data is calculated based on the determination result, so that the similarity of document data can be calculated efficiently. The processing time for calculating the similarity can be shortened.

  Further, according to this data classification method, the similarity between each document data is calculated by determining only the number of occurrences of a length 2 pattern in sequential pattern extraction, so that the processing load on the apparatus is greatly increased. It can be reduced.

  Further, according to this data classification method, document data is classified so that the number of classifications designated by the user is equal to the number of classifications of document data, so that it is possible to provide information according to user preferences. It becomes possible.

  Exemplary embodiments of a data classification method and a data processing device according to the present invention will be explained below in detail with reference to the accompanying drawings.

  First, the outline and features of the data processing apparatus according to this embodiment will be described. The data processing apparatus according to the present embodiment creates an inverted index list in which a keyword in each document data is associated with document data including the keyword, and executes sequential pattern extraction for the inverted index list.

  Then, the data processing apparatus determines the number of occurrences of the combination of document data appearing as a result of executing the sequential pattern extraction (in particular, in this embodiment, only the number of occurrences of the length 2 pattern is determined in the sequential pattern extraction). Then, the similarity between each document data is calculated based on the determination result.

  As described above, the data processing apparatus according to the present embodiment calculates the similarity between the respective document data by performing sequential pattern extraction on the transposed index list. It is possible to calculate, and the processing time for calculating the similarity can be shortened.

  Next, the configuration of the search system according to the present embodiment will be described. FIG. 1 is a diagram illustrating a configuration of a search system. As shown in the figure, this search system includes a terminal device 50 and a data processing device 100, and the terminal device 50 and the data processing device 100 are connected via a network 10.

  Among these, the terminal device 50 is a device that transmits a search keyword to the data processing device 100 when a search keyword is received from a user via an input device or the like. When the terminal device 50 acquires the search result from the data processing device 100, the terminal device 50 displays the acquired search result on the display.

  When the search keyword is acquired from the terminal device 50, the data processing device 100 searches the storage device for document data corresponding to the search keyword, performs the above-described sequential pattern extraction, and classifies the document data. This is a device that outputs the retrieved document data to the terminal device 50 as a search result.

  Here, the configuration of the data processing apparatus 100 shown in FIG. 1 will be described in detail. FIG. 2 is a functional block diagram illustrating the configuration of the data processing apparatus 100 according to the present embodiment. As shown in FIG. 2, the data processing apparatus 100 includes an input unit 110, an output unit 120, a communication control IF unit 130, an input / output control IF unit 140, a storage unit 150, and a control unit 160. Configured.

  Among these, the input unit 110 is an input unit that inputs various types of information, and includes a keyboard, a mouse, a microphone, and the like. The output unit 120 is an output unit that outputs various types of information, and includes a monitor (or display, touch panel), a speaker, and the like.

  The communication control IF unit 130 is a unit that mainly controls communication with the terminal device 50 (see FIG. 1). The input / output control IF unit 140 is a unit that controls data input / output by the input unit 110, the output unit 120, the communication control IF unit 130, the storage unit 150, and the control unit 160.

  The storage unit 150 is a storage unit that stores data and programs necessary for various types of information processing by the control unit 160. In particular, as closely related to the present invention, as shown in FIG. , A word index 150b, a word ID management table 150c, a transposed index 150d, a similarity table 150e, and a cluster table 150f.

  The document management data 150a is data for storing various document data. FIG. 3 is a diagram showing an example of the data structure of the document management data 150a. As shown in the figure, the document management data 150a stores a document ID for identifying each document data and the document data in association with each other.

  The word index 150b is data that stores the document ID and each word ID (word ID string) included in the document data identified by the document ID in association with each other. FIG. 4 is a diagram illustrating an example of a data structure of the word index 150b.

  As shown in the figure, the word index 150b stores a document ID and a word ID string in association with each other. For example, in the first row of FIG. 4, information indicating that the document data identified by the document ID “A” includes the word identified by the word ID “1, 2, 3” is registered. Yes.

  The word ID management table 150c is a table that stores a word ID and a word corresponding to the word ID in association with each other. FIG. 5 is a diagram illustrating an example of a data structure of the word ID management table 150c.

  The transposed index 150d is data that stores the word ID and the document ID of the document data including the word of the word ID in association with each other. FIG. 6 is a diagram illustrating an example of the data structure of the transposed index 150d. For example, in the first row of FIG. 6, it is registered that document data having a word identified by the word ID “1” is document data identified by the document ID “A”.

  The similarity table 150e is a table that stores the similarity between each piece of document data. FIG. 7 is a diagram illustrating an example of a data structure of the similarity table 150e. “A” to “E” shown in FIG. 7 are document IDs, and each numerical value is a similarity. Referring to FIG. 7, for example, information indicating that the similarity between the document data with the document ID “A” and the document data with the document ID “B” is “2” is registered.

  The cluster table 150f is data used when classifying document data based on the similarity of each document data. FIG. 8 is a diagram illustrating an example of the data structure of the cluster table 150f. “A” to “E” shown in FIG. 8 are document IDs, and each numerical value is a similarity. The process of classifying document data based on the cluster table 150f will be described later.

  The control unit 160 has an internal memory for storing programs and control data that define various processing procedures, and is a control means for executing various processes by these, and is particularly closely related to the present invention. Includes a document data search unit 160a, an inverted index creation unit 160b, a similarity table creation unit 160c, and a clustering processing unit 160d.

  Among these, the document data search unit 160a is a means for searching the document management data 150a for document data including the received search keyword when the search keyword is received from the terminal device 50. The document data search unit 160a outputs the document ID of the searched document data to the transposed index creation unit 160b.

  The transposed index creation unit 160b is a unit that acquires a document ID from the document data search unit 160a and creates a transposed index 150d corresponding to the acquired document ID. Specifically, the transposed index creating unit 160b executes a word index creating process for creating the word index 150b and the word ID management table 150c, and a transposed index creating process for creating the transposed index 150d. Hereinafter, the word index creation process and the transposition index creation process executed by the transposed index creation unit 160b will be described in order.

  First, word index creation processing executed by the transposed index creation unit 160b will be described. When the transposed index creation unit 160b obtains the document ID from the document data search unit 160a, the transposed index creation unit 160b obtains the document data corresponding to the document ID from the document management data 150a, and stores the obtained document data (hereinafter, document data group). A morphological analysis is performed on the result.

  Then, the transposed index creation unit 160b assigns a word ID to the word obtained as a result of executing the morphological analysis on the document data group, associates the word ID with the word corresponding to the word ID, and the word ID management table 150c. Register with.

  The transposed index creation unit 160b compares the word ID management table 150c with the document data group to determine document data including words in the word ID management table 150c, and creates the word index 150b (see FIG. 4). To do.

  Next, a transposed index creation process executed by the transposed index creation unit 160b will be described. The transposed index creation unit 160b obtains the word index 150b, and creates the transposed index 150d (see FIG. 6) by determining the document ID of the document data having the word ID for each word ID.

  The similarity table creation unit 160c performs sequential pattern extraction on the transposed index 150d, and determines the number of occurrences of the combination of document data (document ID) that appears as a result of the sequential pattern extraction (particularly in this embodiment). In the sequential pattern extraction, only the number of occurrences of the length 2 pattern is determined), and the similarity table 150e is created based on the determination result.

  In the following, the processing of the similarity table creation unit 160c will be specifically described. First, the similarity table creation unit 160c extracts a document ID (in other words, each document ID included in the transposed index 150d) that is a frequent series of 1 item from the transposed index 150d. For example, when a document ID (hereinafter referred to as a series document ID) that is a frequent series with 1 item is extracted from the transposed index 150d shown in FIG. 6, the document IDs “A” and “B” are used as the series document IDs. , “C” is extracted.

  Subsequently, the similarity table creation unit 160c creates projection data by projecting the transposed index 150d with the series document ID.

Here, the definition of projection will be described. There sequence _{s = <a 1 ,a 2 ,···,a m} >, to Item _{a, a 1 ≠ a, a} 2 ≠ a, ···, a j-1 ≠ a, and _a j = a If made such integer j (1 ≦ j ≦ m) is present, is defined as sequence _<a 1 _,a 2 _{,···,a j>} a prefix for a of s (prefix (s, a) ), series _{<a j+1 ,···,a m>} the postfix for a of s (postfix (s, a) ) to define. If j does not exist, prefix and postfix are undefined.

  Then, to project a certain series database S (a database including a plurality of data of the above series) with an item a to create projection data S | a, postfix (s, a) is created and defined as a series database.

  Specifically, the similarity degree table creation unit 160c applies items (series document IDs “A” and “B” to each document ID column (corresponding to the series s) included in the transposed index 150d (corresponding to the series database S). ”,“ C ”). For example, the similarity table creation unit 160c performs projection in the order of the sequence document IDs “A”, “B”, and “C”.

(Projection by series document ID “A”)
When the similarity table creation unit 160c performs projection with the sequence document ID “A” on the transposed index 150d (see FIG. 6), the sequence document ID among the document ID columns including the sequence document ID “A”. A document ID string excluding “A” (corresponding to postfix (s, a) of the sequence document ID “A”) and a word ID corresponding to the document ID string are extracted to create projection data.

  FIG. 9 is a diagram illustrating an example of projection data created by projection of the sequence document ID “A”. Referring to the projection data in FIG. 9, the number of occurrences of the combination of the sequence document ID “A” and the document ID “B” is “2”, and the combination of the sequence document ID “A” and the document ID “C” Since the occurrence count is “3”, the similarity table creation unit 160c sets the similarity between the document data with the document ID “A” and the document data with the text ID “B” to “2” and the document ID “A”. The similarity between the document data and the document data with the sentence ID “C” is determined as “3”.

  Thereafter, the similarity table creation unit 160c updates the transposed index 150d (see FIG. 6) in association with the projection data (see FIG. 9). Specifically, the document ID string with the word ID “2” of the transposed index 150d is “B, C”, the document ID string with the word ID “7” is “B, C”, and the document ID string with the word ID “10”. Is updated to “C”. FIG. 10 is a diagram (1) illustrating an example of the data structure of the updated inverted index 150d.

(Projection by series document ID “B”)
When the similarity table creation unit 160c performs projection using the sequence document ID “B” on the transposed index 150d (see FIG. 10), the sequence document ID “B” in each document ID column including the sequence document “B” is displayed. A document ID string excluding “B” (corresponding to postfix (s, a) of the sequence document ID “B”) and a word ID corresponding to the document ID string are extracted to create projection data.

  FIG. 11 is a diagram illustrating an example of projection data created by projection of the sequence document ID “B”. Referring to the projection data in FIG. 11, since the number of occurrences of the combination of the series document ID “B” and the document ID “C” is “4”, the similarity table creation unit 160c determines the document with the document ID “B”. The degree of similarity between the data and the document data of the sentence ID “C” is determined as “4”.

  Thereafter, the similarity table creation unit 160c updates the transposed index 150d (see FIG. 10) in association with the projection data (FIG. 11). Specifically, the document ID string of the word IDs “2”, “5”, “7”, “9” of the transposed index 150d is updated to “C”. FIG. 12 is a diagram (2) illustrating an example of the data structure of the updated inverted index 150d.

(Projection by series document ID “C”)
When the similarity table creation unit 160c performs projection on the transposed index 150d (see FIG. 12) using the sequence document ID “C”, the sequence document ID “C” includes the sequence document ID “C”. Since there is no document ID string excluding “C” (corresponding to postfix (s, a) of the series document ID “C”) and the word ID corresponding to the document ID string, the similarity table creation unit 160c performs the projection process. Exit.

  Next, the similarity table creation unit 160c creates the similarity table 150e based on the similarity between the document data obtained as a result of executing the projection using the sequence document IDs “A”, “B”, and “C”. To do. Here, as an example, the case where the similarity is calculated from the sequence document IDs “A”, “B”, and “C” has been described. For example, in addition to the sequence document ID, the sequence document ID “D” is used. , “E” and the like are included, the similarity is calculated by executing the above-described sequential pattern extraction.

  The clustering processing unit 160d is means for classifying the document data group based on the similarity table 150e. Note that the clustering processing unit 160d classifies the document data group so that the number of groups to be classified (hereinafter, the number of clusters) is equal to the number of clusters specified by the user (hereinafter, the number of designated clusters). The user may specify the specified number of clusters using the input unit 110, or the user of the terminal device 50 inputs the specified number of clusters, and the terminal device 50 inputs the specified number of clusters to the data processing device 100. May be sent.

  Hereinafter, the processing of the clustering processing unit 160d will be specifically described. FIG. 13 is a diagram for explaining the processing of the clustering processing unit 160d. Here, as an example, a case where the number of designated clusters is “3” will be described.

  First, the clustering processing unit 160d initializes the cluster table 150f, and then copies the data of the similarity table 150e (see FIG. 7) to the cluster table 150f, so that FIG. 8 (or the upper left corner of FIG. 13) is obtained. The cluster table 150f shown is generated.

  Then, the clustering processing unit 160d detects the document ID pair having the maximum similarity from the cluster table 150f, and deletes the row and column data corresponding to the detected document ID of the pair from the cluster table 150f.

  In the example shown in FIG. 13, the document ID pair having the maximum similarity is a pair of document IDs “A” and “D” having the similarity “8”. The row and column data corresponding to “A” and “D” are deleted from the cluster table 150f (see step S10 in FIG. 13).

  The clustering processing unit 160d creates a cluster composed of a pair having the maximum similarity (deleted pair) and adds it to the cluster table 150f. In the example illustrated in FIG. 13, a cluster “A, D” including document IDs “A” and “D” is created, and the created cluster “A, D” is added to the cluster table 150f (see step S20 in FIG. 13). ).

  The clustering processing unit 160d registers the maximum similarity among the similarities between the document IDs included in the cluster added to the cluster table 150f and other document IDs in the row corresponding to the cluster. Then, the clustering processing unit 160d registers the similarity in the cluster column in association with the similarity registered in the cluster row.

  An example will be described with reference to FIG. The similarity between the cluster “A, D” and the document ID “B” is the similarity “2” between the document ID “A” and the document ID “B”, the document ID “D”, and the document ID “B”. Among the similarities “0” with “B”, the maximum similarity is registered, so the similarity “2” is registered in the location.

  The similarity between the cluster “A, D” and the document ID “C” is the similarity “5” between the document ID “A” and the document ID “C”, and the document ID “D” and the document ID “C”. Among the similarities “3” with “C”, since the maximum similarity is registered, the similarity “5” is registered in the location.

  The similarity between the cluster “A, D” and the document ID “E” is the similarity “1” between the document ID “A” and the document ID “E”, and the document ID “D” and the document ID “E”. Among the similarities “2” with “E”, the maximum similarity is registered, and therefore the similarity “2” is registered in the location. Then, the clustering processing unit 160d registers the similarity in the cluster column in association with the similarity registered in the cluster row (see step S30 in FIG. 13).

  The clustering processing unit 160d repeats the above processing until the number of clusters in the cluster table 150f is equal to the designated number of clusters. Since the number of clusters is “4” and the designated number of clusters is “3” at the end of step S30 in FIG. 13, the clustering processing unit 160d repeats the above process once again.

  That is, in the cluster table 150f shown in the upper right corner of FIG. 13, the document ID pair having the maximum similarity is the document ID “C” and “E” pair having the similarity “6”. The unit 160d deletes the row and column data corresponding to the document IDs “C” and “E” from the cluster table 150f (see step S40 in FIG. 13).

  Then, the clustering processing unit 160d creates a cluster “C, E” including the document IDs “C” and “E”, and adds the created cluster “C, E” to the cluster table 150f (step S50 in FIG. 13). reference).

  The clustering processing unit 160d determines that the similarity between the document ID “C” and the document ID “B” is “3”, and the similarity between the document ID “E” and the document ID “B” is “4”. Therefore, the maximum similarity “4” is registered as the similarity between the cluster “C, E” and the document ID “B”.

  Also, the clustering processing unit 160d has a similarity between the document ID “C” and the document ID “A, D” of “5”, and a similarity between the document ID “E” and the document ID “A, D”. Since the degree is “2”, the maximum degree of similarity “5” is registered as the degree of similarity between the cluster “C, E” and the document ID “A, D” (see step S60 in FIG. 13).

  When the processing of step S60 in FIG. 13 is completed, the number of clusters in the cluster table 150f is “3”, which is equal to the designated number of clusters “3”. Therefore, the clustering processing unit 160d performs document processing based on the cluster table 150f. Classify data groups. In the example illustrated in FIG. 13, the document data group is classified into document data with a document ID “B”, document data with document IDs “A” and “D”, and document data with document IDs “C” and “E”. Will be.

  The clustering processing unit 160 d outputs the classified document data to the terminal device 50 as an answer to the search keyword transmitted from the terminal device 50.

  Next, a processing procedure of the data processing apparatus 100 according to the present embodiment will be described. FIG. 14 is a flowchart illustrating the processing procedure of the data processing apparatus 100 according to the present embodiment. As shown in the figure, the data processing device 100 acquires a search keyword from the terminal device 50 (step S101), and searches for document data corresponding to the search keyword (step S102).

  The data processing apparatus 100 executes an inverted index creation process (step S103), executes a similarity table creation process (step S104), executes a clustering process (step S105), and performs a clustering result (classified document data). ) Is output to the terminal device 50 (step S106).

  Next, the transposed index creation process shown in step S103 of FIG. 14 will be described. FIG. 15 is a flowchart showing the inverted index creation processing. As shown in the figure, in the data processing apparatus 100, the transposed index creation unit 160b performs morphological analysis on each document data (step S201).

  The transposed index creation unit 160b creates the word index 150b (step S202), and creates the transposed index 150d using the word ID as a key (step S203).

  Next, the similarity table creation process shown in step S104 of FIG. 14 will be described. FIG. 16 is a flowchart showing the similarity table creation processing. As shown in the figure, in the data processing apparatus 100, the similarity table creation unit 160c initializes the similarity table 150e (step S301), and extracts a series document ID that becomes a frequent series of item number 1 from the transposed index 150d. (Step S302).

  Then, the similarity table creation unit 160c creates projection data based on the sequence document ID and the transposed index 150d (step S303), and calculates the appearance frequency of the pair (number of occurrences of the combination) (step S304). The transposed index 150d is updated (step S305).

  The similarity table creation unit 160c determines whether or not all series document IDs have been selected (step S306). If all series document IDs have not been selected (step S307, No), unselected series document IDs are selected. Select (step S308), the process proceeds to step S303.

  On the other hand, when all the series document IDs are selected (step S307, Yes), the similarity of each document pair is registered in the similarity table 150e (step S309).

  Next, the clustering process shown in step S105 of FIG. 14 will be described. FIG. 17 is a flowchart showing the clustering process. As shown in the figure, the clustering processing unit 160d copies the similarity table 150e to the cluster table 150f (step S401), and determines whether or not the number of clusters in the cluster table 150f is equal to the designated cluster number (step S402). ).

  If the number of clusters is equal to the number of designated clusters (step S403, Yes), the clustering process is terminated. On the other hand, when the number of clusters is different from the number of designated clusters (No in step S403), the maximum value is determined in the cluster table 150f (step S404).

  The clustering processing unit 160d deletes the row and column of the pair having the maximum value (step S405), generates a cluster including the pair having the maximum value, and adds the cluster to the cluster table 150f (step S406).

  Subsequently, the clustering processing unit 160d selects an unselected element from each element of the added cluster row (step S407), refers to the similarity table 150e, and determines the maximum value of the similarity between the added clusters. Is registered as an element (step S408).

  Then, the clustering processing unit 160d determines whether or not all the elements have been selected (step S409), and if not all the elements have been selected (step S410, No), the process proceeds to step S407, When the element is selected (step S410, Yes), the value of each element of the row added to the cluster table 150f is registered in each element of the corresponding column (step S411), and the process proceeds to step S402.

  As described above, the data processing apparatus 100 according to the present embodiment creates the transposed index 150d in which the word ID in each document data is associated with the document data including the word corresponding to the word ID, and the transposed index. Sequential pattern extraction is executed for 150d. Then, the number of occurrences of the combination of document data appearing as a result of executing the sequential pattern extraction is determined (particularly, in the present embodiment, only the number of occurrences of the length 2 pattern is determined in the sequential pattern extraction), and the determination result is obtained. Based on this, since the similarity between the document data is calculated, the similarity of the document data can be calculated efficiently, and the processing time for calculating the similarity can be shortened as compared with the conventional technique.

For example, as in the prior art, the similarity calculation based on the vector space model calculates the similarity between documents by comparing the documents, and thus represents the number of comparisons when the number of documents is n. Formula F (n) is
F (n) = (n × (n-1)) / 2
It can be expressed by a “quadratic function expression” such as For example, when the number of documents is 4, the similarity needs to be calculated six times (for example, if the documents are a, b, c, d, ab, ac, a- d, bc, bd, and cd need to be calculated a total of 6 times).

And in order to estimate the amount of calculation of the vector space model, an expression representing the number of comparisons is expressed in O notation.
O (n) = n ^ 2
Can be expressed as

On the other hand, in the data processing apparatus 100 according to the present embodiment, the similarity between each document is calculated by comparing each document with the transposed index 150d, so that the amount of calculation that increases as the number of documents increases. Is increased according to the “linear function” (for example, when the number of documents increases by one, the number of comparisons with the transposed index 150d increases by one), and an expression representing the number of comparisons is expressed in O notation.
O (n) = n
It can be expressed as.

  Therefore, when the number of documents n is doubled, the processing time is about n ^ 2 in the conventional technique, whereas in the data processing apparatus 100 according to the present embodiment, the processing time is about n times. Therefore, the processing time for calculating the similarity can be shortened as compared with the conventional technique.

  By the way, among the processes described in the present embodiment, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being performed manually can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

  Also, each component of the data processing apparatus 100 shown in FIG. 2 is functionally conceptual and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Furthermore, each processing function performed by each device may be realized by a CPU and a program that is analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  FIG. 18 is a diagram illustrating a hardware configuration of the computer 40 configuring the data processing apparatus 100 according to the embodiment. As shown in FIG. 18, the computer (article search device) 40 includes an input device 41, a monitor 42, a RAM (Random Access Memory) 43, a ROM (Read Only Memory) 44, and a medium reading device 45 that reads data from a storage medium. A communication device 46 that transmits and receives data to and from other devices (for example, the terminal device 50), a CPU (Central Processing Unit) 47, and an HDD (Hard Disk Drive) 48 are connected by a bus 49.

  The HDD 48 stores a clustering processing program 48b that exhibits the same function as that of the data processing apparatus 100 described above. When the CPU 47 reads and executes the clustering processing program 48b, the clustering processing process 47a is activated. Here, the clustering process 47a corresponds to the document data search unit 160a, the transposed index creation unit 160b, the similarity table creation unit 160c, and the clustering processing unit 160d shown in FIG.

  The HDD 48 stores various data 48a corresponding to the document management data 150a, the word index 150b, the word ID management table 150c, the transposition index 150d, the similarity table 150e, and the cluster table 150f. The CPU 47 reads out various data 48 a stored in the HDD 48, stores it in the RAM 43, and classifies each document data using the various data 43 a stored in the RAM 43.

  Incidentally, the clustering processing program 48b shown in FIG. 18 is not necessarily stored in the HDD 48 from the beginning. For example, a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into a computer, or a hard disk drive (HDD) provided inside or outside the computer. The clustering processing program 48b is stored in the “fixed physical medium” of “the other computer (or server)” connected to the computer via a public line, the Internet, a LAN, a WAN, or the like. The computer may read and execute the clustering processing program 48b from these.

  The following additional notes are disclosed with respect to the embodiments including the above-described examples.

(Supplementary note 1) A data classification method for a data processing device for classifying document data according to the similarity of each document data stored in a storage device,
The data processing device includes:
A list creation step of reading out the document data stored in the storage device and creating a list in which keywords in each document data are associated with document data including the keywords;
A determination step of performing sequential pattern extraction on the list and determining the number of occurrences of a combination of document data that has appeared;
A calculation step of calculating a similarity between the document data based on the determination result of the determination step;
A data classification method characterized by including:

(Supplementary note 2) The data classification method according to supplementary note 1, wherein in the determination step, only the number of occurrences of the pattern of length 2 is determined by sequential pattern extraction.

(Supplementary Note 3) The method further includes a classification step of classifying the document data by obtaining a combination of the respective document data that maximizes the similarity value, and the classification step includes the classification number designated by the user and the document data. The data classification method according to appendix 1 or 2, wherein the document data is classified so that the number of classifications is equal.

(Supplementary note 4) A data processing device for classifying document data according to the similarity of each document data stored in a storage device,
List creation means for reading out the document data stored in the storage device and creating a list in which keywords in each document data are associated with document data including the keywords;
A determination unit that performs sequential pattern extraction on the list and determines the number of occurrences of a combination of document data that has appeared;
Calculation means for calculating the similarity between the document data based on the determination result of the determination means;
A data processing apparatus comprising:

(Supplementary note 5) The data processing apparatus according to supplementary note 4, wherein the determination unit determines only the number of occurrences of a pattern of length 2 by sequential pattern extraction.

(Additional remark 6) It further has a classification means for classifying document data by obtaining a combination of each document data that maximizes the similarity value, and the classification means includes the classification number designated by the user and the document data. 6. The data processing apparatus according to appendix 4 or 5, wherein the document data is classified so that the number of classifications is equal.

  As described above, the data classification method and data processing apparatus according to the present invention are useful for a search system for searching for document data, and in particular, when it is necessary to classify each document data without taking processing time. Suitable for

It is a figure which shows the structure of a search system. It is a functional block diagram which shows the structure of the data processor concerning a present Example. It is a figure which shows an example of the data structure of document management data. It is a figure which shows an example of the data structure of a word index. It is a figure which shows an example of the data structure of a word ID management table. It is a figure which shows an example of the data structure of an inverted index. It is a figure which shows an example of the data structure of a similarity table. It is a figure which shows an example of the data structure of a cluster table. It is a figure which shows an example of the projection data produced by projection of series document ID "A". It is a figure (1) which shows an example of the data structure of the updated inverted index. It is a figure which shows an example of the projection data produced by projection of series document ID "B". It is a figure (2) which shows an example of the data structure of the updated inverted index. It is a figure for demonstrating the process of a clustering process part. It is a flowchart which shows the process sequence of the data processor concerning a present Example. It is a flowchart which shows a transposition index creation process. It is a flowchart which shows a similarity table preparation process. It is a flowchart which shows a clustering process. It is a figure which shows the hardware constitutions of the computer which comprises the data processor concerning an Example.

Explanation of symbols

10 network 40 computer 41 input device 42 monitor 43 RAM
43a, 48a Various data 44 ROM
45 Media reader 46 Communication device 47 CPU
47a Clustering process 48 HDD
48b Clustering processing program 49 Bus 50 Terminal device 100 Data processing device 110 Input unit 120 Output unit 130 Communication control IF unit 140 Input / output control IF unit 150 Storage unit 150a Document management data 150b Word index 150c Word ID management table 150d Transposed index 150e Similar Degree table 150f cluster table 160 control unit 160a document data search unit 160b transposed index creation unit 160c similarity table creation unit 160d clustering processing unit

Claims (5)

A data classification method for a data processing device for classifying document data according to the similarity of each document data stored in a storage device,
The data processing device includes:
A list creation step of reading out the document data stored in the storage device and creating a list in which keywords in each document data are associated with document data including the keywords;
A determination step of performing sequential pattern extraction on the list and determining the number of occurrences of a combination of document data that has appeared;
A calculation step of calculating a similarity between the document data based on the determination result of the determination step;
A data classification method characterized by including:   2. The data classification method according to claim 1, wherein in the determination step, only the number of occurrences of a length 2 pattern is determined by sequential pattern extraction.   The method further includes a classification step of classifying the document data by obtaining a combination of the respective document data having the maximum similarity value, and the classification step includes a classification number designated by the user and a classification number of the document data. The data classification method according to claim 1, wherein the document data is classified so that the two are equal to each other. A data processing device for classifying document data according to the similarity of each document data stored in a storage device,
List creation means for reading out the document data stored in the storage device and creating a list in which keywords in each document data are associated with document data including the keywords;
A determination unit that performs sequential pattern extraction on the list and determines the number of occurrences of a combination of document data that has appeared;
Calculation means for calculating the similarity between the document data based on the determination result of the determination means;
A data processing apparatus comprising:   The data processing apparatus according to claim 4, wherein the determination unit determines only the number of occurrences of a pattern of length 2 by sequential pattern extraction.

Images