JP2009031931A - Search word clustering device, method, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique capable of accurately classifying search words without depending on the accuracy of morphological analysis. <P>SOLUTION: A search log acquiring part 211 acquires a search log that records combinations of search words from a search server 30. Based on the search log, a co-occurrence query list creating part 221 creates a co-occurrence query list consisting of search words, co-occurrence queries and frequencies of co-occurrence. Based on the co-occurrence query list, a co-occurrence query matrix creating part 231 creates a co-occurrence query vector where the co-occurrence frequencies of the co-occurrence queries of the search words are vector components. A frequency analysis-vector standardization part 232 makes the vector lengths of the co-occurrence query vectors equal. An Euclidean distance calculating part 234 calculates the Euclidean distance between the co-occurrence query vectors. A hierarchical clustering part 235 performs clustering based on the Euclidean distance calculated. In this way, it is possible to accurately classify the search words without being affected by the accuracy of morphological analysis of a document. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a search term clustering device, a search term clustering method, a search term clustering program, and a recording medium, and more particularly to a search term clustering device, a search term clustering method, a search term clustering program, and a recording medium for clustering search terms. .

  In recent years, the Internet has exploded in popularity, and websites (web sites) including various web pages (web pages) are useful. Various search engines have been developed for users to search for desired Web pages on the Web. The current mainstream search algorithm is a full-text search based on a match between a search term input by a user and a word in a document on a Web page. In this method, there is a case where a mismatch occurs between the search word and the word in the document of the Web page due to the diversity of words, and the search rate may be lowered.

As countermeasures, search term expansion such as search term classification (search term clustering) such as a thesaurus is used to perform a search using a search term similar to the search term in the front end. For example, the following Non-Patent Document 1 and Non-Patent Document 2 propose a clustering method for estimating the similarity between different search terms based on the frequency with which different search terms are included in the same document, and classifying the search terms. ing. Search term classification using dependency information in documents has also been proposed.
J. Wen, J. Nie, and H. Zhang? “Query clustering using userlogs”, ACM Transactions on Information Systems (ACMTOIS), January 2002, Vol.20, No.1, pp.59.81. D. Beeferman and A. Berger, “Agglomerative clustering of search engine query log”, The 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2000), August 2000.

  However, in the search term classification as described above, the accuracy of the search rate and the search term classification varies greatly depending on the accuracy of the morphological analysis that determines the break between words or clauses in the document. Is dependent on the accuracy of morphological analysis. Therefore, since it is necessary to perform morphological analysis on a document with high accuracy, there is a problem that a processing load is large when classifying search terms.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a search word clustering apparatus, a search word clustering method, and a search that can accurately classify search words without depending on the accuracy of morphological analysis. The object is to provide a word clustering program and a recording medium.

The present invention relates to a search log for acquiring a search log that is a record of a search performed by each of a plurality of combinations of search terms in a search server that searches a web page by a combination of a plurality of search terms specified by an unspecified user. Based on the search log acquired by the acquisition unit and the search log acquisition unit, for each search term, a co-occurrence query that is another search term combined with the search term and used for the search, and the co-occurrence query is the search term Based on the co-occurrence query list created by the co-occurrence query list creation means for creating the co-occurrence query list associated with the co-occurrence frequency that is the frequency used in the search in combination with A co-occurrence query vector which is an n-dimensional vector having a numerical value based on the co-occurrence frequency of each of the n co-occurrence queries of the search word as a vector component is generated. Query vector creation means, vector standardization means for standardizing each co-occurrence query vector for each search term created by the co-occurrence query vector creation means, and search terms standardized by the vector standardization means Euclidean distance calculation means for calculating the Euclidean distance between each of the co-occurrence query vectors for, and clustering means for clustering each search word based on the Euclidean distance calculated by the Euclidean distance calculation means,
Is a search word clustering device.

  According to this configuration, the search log acquisition means calculates the frequency of performing a search by each of a plurality of combinations of search terms in a search server that searches a web page by a combination of a plurality of search terms specified by an unspecified user. The recorded search log is acquired, and the co-occurrence query list creation means uses the other search terms combined with the search word and used for the search based on the search log acquired by the search log acquisition means. Because a co-occurrence query list is created in which a co-occurrence query is associated with the co-occurrence frequency, which is the frequency that the co-occurrence query is combined with the search term and used in the search, it is affected by the accuracy of morphological analysis on the document. Instead, it is possible to acquire the co-occurrence query combined with the search word by an unspecified user and used for the search, and the co-occurrence frequency of the co-occurrence query.

  Further, the co-occurrence query vector creation means uses a numerical value based on the co-occurrence frequency of each of the n co-occurrence queries of the search word as a vector component based on the co-occurrence query list created by the co-occurrence query list creation means. Since the co-occurrence query vector which is an n-dimensional vector is created, the co-occurrence query list can be quantified so as to be suitable for the subsequent analysis. In addition, the vector standardization unit standardizes the co-occurrence query vectors so that the vector lengths of the co-occurrence query vectors are the same for each search term created by the co-occurrence query vector creation unit. It is possible to prevent an error from occurring in search term clustering.

  Further, the Euclidean distance calculation means calculates the Euclidean distance between the co-occurrence query vectors for each search term standardized by the vector standardization means, and the clustering means searches based on the Euclidean distance calculated by the Euclidean distance calculation means. Since each word is clustered, it can be inferred that search terms related to co-occurrence query vectors with close Euclidean distances are very similar in terms of the degree of coincidence of the co-occurrence queries and their co-occurrence frequencies. The search terms can be classified with high accuracy based on the co-occurrence query combined with the search term by an unspecified user and used for the search, and the co-occurrence frequency of the co-occurrence query. .

  In this case, the co-occurrence query list creation unit extracts each search term from the search log acquired by the search log acquisition unit, and the co-occurrence frequency of the co-occurrence query and the co-occurrence query collected by the search server for each of the extracted search terms. Can be used to create a co-occurrence query list.

  According to this configuration, the co-occurrence query list creation unit extracts each search term from the search log acquired by the search log acquisition unit, and shares the co-occurrence query collected by the search server outside the apparatus for each of the extracted search terms. Since the co-occurrence query list is created by acquiring the co-occurrence frequency of the occurrence query, it is not necessary for the co-occurrence query list creation means itself to aggregate the search log, and the processing load of the search term clustering apparatus can be reduced. it can. Moreover, according to this structure, it is not necessary to have a search server as a structure connected to the search word clustering apparatus.

  Alternatively, the co-occurrence query list creation unit creates a co-occurrence query list by aggregating the co-occurrence query and the co-occurrence frequency of the co-occurrence query for each search term from the search log acquired by the search log acquisition unit. I can do it.

  According to this configuration, the co-occurrence query list creation unit performs co-occurrence by tabulating the co-occurrence query and the co-occurrence frequency of the co-occurrence query for each search term from the search log acquired by the search log acquisition unit. Since a query list is created, it is possible to create a co-occurrence query list suitable for subsequent processing in the search word clustering apparatus.

  On the other hand, the vector dimension number reduction means for deleting the numerical value based on the co-occurrence frequency of the co-occurrence query whose numerical value based on the co-occurrence frequency is equal to or less than the threshold from the co-occurrence query vector generated by the co-occurrence query vector generating means. It is preferable to further include

  According to this configuration, the vector dimension number reduction means calculates the co-occurrence frequency of the co-occurrence query whose numerical value based on the co-occurrence frequency from the co-occurrence query vector created by the co-occurrence query vector creation means is equal to or less than a threshold from the vector component. Since deletion is performed, a co-occurrence query whose co-occurrence frequency that is not suitable for clustering of search terms is too low can be excluded from the clustering process, so that the accuracy of clustering can be improved and the processing load of the apparatus can be reduced.

  In this case, it is preferable that the clustering means performs clustering for each search word by the Ward method.

  According to this configuration, since the clustering means performs clustering for each search word by the Ward method, clustering can be performed with high accuracy.

  The present invention also provides a search log in which a computer performs a search using a combination of a plurality of search terms in a search server that searches a web page using a combination of a plurality of search terms specified by an unspecified user. And a co-occurrence query that is another search word used in the search combined with the search word for each search word based on the search log acquired by the computer in the search log acquisition step A co-occurrence query list creating step for creating a co-occurrence query list in which a co-occurrence query is associated with a co-occurrence frequency that is a frequency used in a search by combining the co-occurrence query with a search word, and the computer Based on the co-occurrence query list created in the creation step, a numerical value based on the co-occurrence frequency of each of the n co-occurrence queries of the search term The co-occurrence query vector creation step for creating a co-occurrence query vector, which is an n-dimensional vector as a vector component, and the vector length of each of the co-occurrence query vectors for each search term created by the computer in the co-occurrence query vector creation step are A vector standardization step for standardizing to be the same, a Euclidean distance calculation step for calculating the Euclidean distance between the co-occurrence query vectors for each search term standardized by the computer in the vector standardization step, and a computer for calculating the Euclidean distance And a clustering step for performing clustering for each of the search terms based on the Euclidean distance calculated in the step.

  In this case, in the co-occurrence query list creation step, the computer extracts each search term from the search log acquired in the search log acquisition step, and the co-occurrence query and the co-occurrence query are totaled by the search server for each extracted search term. By obtaining the co-occurrence frequency, a co-occurrence query list can be created.

  Alternatively, in the co-occurrence query list creation step, the computer calculates the co-occurrence query list by aggregating the co-occurrence query for each search term and the co-occurrence frequency of the co-occurrence query from the search log acquired in the search log acquisition step. Can be created.

  On the other hand, a vector dimension in which the computer deletes a numerical value based on the co-occurrence frequency of the co-occurrence query whose numerical value based on the co-occurrence frequency is equal to or less than a threshold from the co-occurrence query vector generated in the co-occurrence query vector generation step from the vector component It is preferable to further include a number reduction step.

  In the clustering step, it is preferable that the computer performs clustering for each search word by a Ward method.

  Furthermore, the present invention acquires a search log that is a record of a search performed by each combination of a plurality of search terms in a search server that searches a web page by a combination of a plurality of search terms specified by an unspecified user. Based on the search log acquisition step and the search log acquired in the search log acquisition step, for each search word, a co-occurrence query that is another search word combined with the search word and used for the search is A co-occurrence query list creation step for creating a co-occurrence query list associated with a co-occurrence frequency that is a frequency used in a search combined with a search term, and a co-occurrence query list created in the co-occurrence query list creation step The co-occurrence vector is an n-dimensional vector having a numerical value based on the co-occurrence frequency of each of the n co-occurrence queries of the search term as a vector component A co-occurrence query vector creation step for creating a revector, a vector standardization step for standardizing the vector length of each co-occurrence query vector for each search term created in the co-occurrence query vector creation step, and a vector standardization step A Euclidean distance calculating step for calculating the Euclidean distance between the co-occurrence query vectors for each of the search terms standardized in step E, a clustering step for performing clustering for each of the search terms based on the Euclidean distance calculated in the Euclidean distance calculating step, Is a search word clustering program for causing a computer to execute

  In this case, in the co-occurrence query list creation step, the computer extracts each search term from the search log acquired in the search log acquisition step, and the co-occurrence query and the co-occurrence query aggregated by the search server for each of the extracted search terms. By acquiring the co-occurrence frequency, a co-occurrence query list can be created.

  Alternatively, in the co-occurrence query list creation step, the co-occurrence query list is obtained by causing the computer to aggregate the co-occurrence query and the co-occurrence frequency of the co-occurrence query from the search log acquired in the search log acquisition step. Can be created.

  On the other hand, the vector dimension which deletes the numerical value based on the co-occurrence frequency of the co-occurrence query whose numerical value based on the co-occurrence frequency is equal to or less than the threshold from the co-occurrence query vector generated in the co-occurrence query vector generation step from the vector component. It is preferable to further execute the number reduction step.

  In the clustering step, it is preferable to cause the computer to execute clustering for each search word by the Ward method.

  In addition, the present invention is a computer-readable recording medium on which the search word clustering program of the present invention is recorded.

  According to the search word clustering device, the search word clustering method, the search word clustering program, and the recording medium of the present invention, the search word clustering program is combined with the search word by an unspecified user and used for the search without being affected by the accuracy of the morphological analysis for the document. Based on the obtained co-occurrence query and the co-occurrence frequency of the co-occurrence query, the search terms can be classified with high accuracy.

  Hereinafter, a search word clustering device, a search word clustering method, a search word clustering program, and a recording medium according to embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of a search system according to the embodiment. As shown in FIG. 1, the search word clustering apparatus and the search word clustering method of this embodiment are configured such that a plurality of unspecified user terminals 10 such as personal computers and workstations, a search server 30 and an information host computer 40 are connected to a network 20. Applied in a search system interconnected via In the search system of FIG. 1, the user terminal 10 transmits a search query that is a combination of a plurality of search terms designated by the user to the search server 30. The search server 30 searches the Web page on the Web site from the information host computer 40 based on the search query transmitted from the user terminal 10 and returns the search result to the user terminal 10.

  FIG. 2 is a block diagram illustrating functional blocks of the search server according to the embodiment. As shown in FIG. 2, the search server 30 receives a search query transmitted from the user terminal 10 by the search control unit 301, and retrieves a Web page on the Web site from the information host computer 40 via the information collection control unit 303. Search for. A combination of search terms included in the search query transmitted from the user terminal 10 is recorded as a search log together with the time transmitted from the user terminal 10 in the search log DB 302.

  In response to a request from the user terminal 10, the search control unit 301 refers to the search log DB 302, and searches for a co-occurrence query that is a search term that is combined as a search query and used for a search term. A service to be presented to the user terminal 10 in the order in which the co-occurrence frequency, which is the number of times combined with the search word and used for the search, is higher. The information collection control unit 303 sequentially accumulates information retrieved from the information host computer 40 based on a command from the search control unit 301 in the collected information DB 304, and returns it as a search result to the user terminal 10 via the search control unit 301. To do.

  FIG. 3 is a block diagram illustrating a hardware configuration of the user terminal according to the embodiment. As shown in FIG. 3, the user terminal 10 of this embodiment includes an interface 101, a keyboard 102, a mouse 103, a hard disk 104, a CD-ROM drive 105, a memory card slot 106, a ROM 107, a RAM 108, a display device 109, and a CPU 110. It is configured as a general personal computer.

  The user terminal 10 is inserted into the CD-ROM drive 105 or the memory card slot 106 with a recording medium on which the search word clustering program of this embodiment is recorded. The search word clustering program read from the recording medium inserted into the CD-ROM drive 105 or the memory card slot 106 is recorded by the hard disk 104 and then read into the RAM 108 and executed. By executing the search term clustering program, the user terminal 10 functions as a search term clustering device.

  FIG. 4 is a block diagram showing functional blocks of the user terminal according to the embodiment, and shows a state in which the search word clustering program of the present embodiment is being executed. As shown in FIG. 4, the user terminal 10 of this embodiment includes a log analysis block 210, a co-occurrence query acquisition block 220, a cluster analysis block 230, and an output block 240.

  The log analysis block 210 includes a search log acquisition unit 211, a search log analysis unit 212, a co-occurrence information storage unit 213, and a search word list storage unit 214. The search log acquisition unit 211 is for acquiring a search log including a record in which a search is performed using a combination of a plurality of search terms from the search server 30 via the network 20. The search log analysis unit 212 performs search log analysis processing, such as noise filtering, co-occurrence query acquisition, and aggregation processing, on the search log acquired by the search log acquisition unit 211, and the co-occurrence query for each search word and its co-occurrence. This is for analyzing the occurrence frequency.

  The co-occurrence query and the co-occurrence frequency for each search term analyzed from the search log by the search log analysis unit 212 are recorded in the co-occurrence information storage unit 213. In addition, the search terms designated by the unspecified number of user terminals 10 analyzed from the search log by the search log analysis unit 212 are recorded in the search term list storage unit 214 in the order of frequency used for the search. . The search log acquisition unit 211 and the search log analysis unit 212 function as a search log acquisition unit described in the claims.

  The co-occurrence query acquisition block 220 includes a co-occurrence query list creation unit 221 and a co-occurrence query list storage unit 222. The co-occurrence query list creation unit 221 determines each search word from the co-occurrence query for each search word recorded in the co-occurrence information storage unit 213 and its co-occurrence frequency, or the search word recorded in the search word list storage unit 214. Is a co-occurrence query list in which the co-occurrence queries for and the co-occurrence frequencies are associated. The co-occurrence query list created by the co-occurrence query list creation unit 221 is recorded in the co-occurrence query list storage unit 222. The co-occurrence query list creation unit 221 functions as a co-occurrence query list creation unit described in the claims.

  The cluster analysis block 230 includes a co-occurrence query matrix creation unit 231, a frequency analysis / vector standardization unit 232, a vector dimension number reduction unit 233, an Euclidean distance calculation unit 234, and a hierarchical clustering unit 235.

Based on the co-occurrence query list recorded in the co-occurrence query list storage unit 222, the co-occurrence query matrix creation unit 231 takes m search terms (w ₁ , w ₂ ,... W _m ) as rows and n Co-occurrence queries (c ₁ , c ₂ ,... C _n ) as columns, and a co-occurrence query in which the logarithm of the co-occurrence frequency at which the search word w _i and the co-occurrence query c _j co-occur is the component a _ij Create a matrix. That is, this co-occurrence query matrix is an n-dimensional co-occurrence with the logarithm of the co-occurrence frequency of _n co-occurrence queries (c ₁ , c ₂ ,..., C _n ) as vector components for the search word w _i . A query vector (a _i1 , a _i2 ,... A _in ) is a row vector. The co-occurrence query matrix creation unit 231 functions as a co-occurrence query vector creation unit described in the claims.

  The frequency analysis / vector standardization unit 232 standardizes the co-occurrence query vectors created by the co-occurrence query matrix creation unit 231 and having different vector lengths depending on the co-occurrence frequencies of the components so that the vector lengths are the same. Is for. The frequency analysis / vector standardization unit 232 functions as vector standardization means described in the claims.

  The vector dimensionality reduction unit 233 is suitable for clustering with respect to the co-occurrence query vectors that are standardized by the frequency analysis / vector standardization unit 232, and the co-occurrence frequency of the co-occurrence query is so low that the co-occurrence frequency is below a preset threshold. Therefore, it is removed from the vector component to reduce the number of vector dimensions. The vector dimension number reduction unit 233 functions as vector dimension number reduction means described in the claims. Note that the vector dimension number reduction unit 233 may remove the co-occurrence frequency of the co-occurrence query that is extremely high as the co-occurrence frequency is equal to or higher than a preset threshold value from the vector components because it is not suitable for clustering.

  The Euclidean distance calculation unit 234 is for calculating the Euclidean distance between the co-occurrence query vectors whose vector dimension number has been reduced by the vector dimension number reduction unit 233. The Euclidean distance calculation unit 234 functions as Euclidean distance calculation means described in the claims. Note that the distance between the co-occurrence query vectors can be expressed not only by the Euclidean distance but also by a cosine scale or the like.

  The hierarchical clustering unit 235 is for performing hierarchical clustering by the Ward method for each search word based on the Euclidean distance between the co-occurrence query vectors calculated by the Euclidean distance calculation unit 234. The hierarchical clustering unit 235 functions as clustering means described in the claims. The hierarchical clustering unit 235 may perform hierarchical clustering by the nearest neighbor method, the farthest neighbor method, the group average method, the centroid method, the median method, and the Mcquitty method in addition to the Ward method. In addition, the Mcquitty method enables high-precision clustering after the Ward method. In this embodiment, a threshold is set for the Euclidean distance between the co-occurrence query vectors, and a search term related to the co-occurrence query vector having a Euclidean distance equal to or less than the threshold is similar, and a normal synonym dictionary format is used. Can also be output.

  The output block 240 includes a tree diagram output unit 241, a cluster formation order analysis unit 242, and a hierarchical structure number data storage unit 243. The tree diagram output unit 241 is for outputting the result of clustering as a tree diagram when the scale of the data output by the hierarchical clustering unit 235 is small.

  The cluster formation order analysis unit 242 is for analyzing the cluster formation order and outputting the clustering result as a hierarchical structure number when the scale of the data output by the hierarchical clustering unit 235 is small. is there. The hierarchical structure number output by the cluster formation order analysis unit 242 is stored in the hierarchical structure number data storage unit 243.

  Hereafter, the operation | movement which clusters the search term of the user terminal 10 of this embodiment is demonstrated.

  FIG. 5 is a flowchart showing the operation of the user terminal according to the embodiment. The processing shown in FIG. 5 is roughly divided into a log analysis phase (S11 to S12), a co-occurrence query acquisition phase (S13 to S15), a cluster analysis phase (S16 to S20), and an output phase (S21 to S24).

  First, the search log acquisition unit 211 of the user terminal 10 acquires a search log from the search server 30 via the network 20 (S11). The search log is information including, for example, a record shown in FIG. 6 and a search performed by a combination of a plurality of search terms.

  The search log analysis unit 212 performs search log analysis (S12). In the search log analysis process, the search log analysis unit 212 first performs noise filtering that deletes search terms that are searched for less than a predetermined threshold among the search terms included in the search log. Thereby, for example, it is possible to remove words that do not need to be subjected to search word clustering such as incorrect wording due to user input mistakes.

  Next, in the search log analysis process, the search log analysis unit 212 extracts a search term from the search log. As a result, the search log analysis unit 212 can obtain co-occurrence information that is a combination of a search word and a co-occurrence query as shown in FIG. In FIG. 7, for example, “mail order” is combined as a co-occurrence query for the search term “fishing food” and used in the search. The acquired co-occurrence information is recorded in the co-occurrence information storage unit 213. On the other hand, the search log analysis unit 212 determines the order in which the search terms are used in descending order of the number of times used for the search as shown in FIG. The search terms and their order of use are recorded in the search term list storage unit 214. As a result, clustering can be performed preferentially from search terms that are frequently used for search.

  Based on the search log, the co-occurrence query list acquisition unit 221 sets, for each search word, a co-occurrence query that is another search word combined with the search word and used for the search, and the co-occurrence query is combined with the search word. A co-occurrence query list associated with the co-occurrence frequency, which is the frequency used for the search after being combined, is created (S13 to S15). There are the following two methods for creating the co-occurrence query list.

When the service of the search server 30 described above is not used (S13), the co-occurrence query list creation unit 221 aggregates the co-occurrence information recorded in the co-occurrence information storage unit 213 and searches the search word list storage unit. A co-occurrence query list is preferentially created from search terms frequently used in the search recorded in 214 (S14). FIG. 9 is a diagram illustrating a co-occurrence query list according to the embodiment. For m search terms (w ₁ , w ₂ ,... W _m ), n co-occurrence queries (c ₁ , c ₂ , ... c _n ) and the logarithm a _{ij of the} co-occurrence frequency are associated with each other. Specifically, for example, as shown in FIG. 10, a list in which the co-occurrence queries “Haneda Airport”, “service status”, “fare”, and the like are associated with the search word “JAL” together with the co-occurrence frequency. Is created. The created co-occurrence query list is stored in the co-occurrence query list storage unit 222.

  On the other hand, when using the service of the search server 30 described above (S13), the co-occurrence query list creating unit 221 searches the search server 30 for a single search term in response to a request to the search server 30. Co-occurrence queries, which are search terms combined as a query and used for a search, are presented to the user terminal 10 in the order in which the co-occurrence frequency, which is the number of times combined with the search word and used for the search, is higher. To do. The co-occurrence query list creation unit 221 creates a co-occurrence query list based on the presented information (S15). In this case, the co-occurrence query list creation unit 221 can only obtain the rank used for the search by combining the co-occurrence query with the search term, and cannot obtain a specific numerical value of the co-occurrence frequency. By causing the search server 30 to process the search log aggregation, the processing load on the user terminal 10 side is reduced. In this case, the search server 30 may not be provided as a configuration connected to the search word clustering apparatus.

Based on the co-occurrence query list recorded in the co-occurrence query list storage unit 222, the co-occurrence query matrix creation unit 231 takes m search terms (w ₁ , w ₂ ,... W _m ) as rows and n Are co-occurrence queries (c ₁ , c ₂ ,..., C _n ), and a co-occurrence query matrix having a component a _ij as a logarithm of the co-occurrence frequency at which the search word w _i and the co-occurrence query c _j co-occur. Create (S16). Thus, by creating a co-occurrence query matrix having the logarithm of the co-occurrence frequency as a component a _ij , the subsequent clustering process can be performed with high accuracy. Note that when the co-occurrence query list is created using the service of the search server 30 described above, a specific numerical value of the co-occurrence frequency of the co-occurrence query cannot be obtained. As shown in FIG. 11, for example, when the co-occurrence query “dress” is combined with the search word “Rakuten” and used for the search, the component is set to “1”.

The frequency analysis / vector standardization unit 232 calculates a vector length from the co-occurrence frequencies of the co-occurrence query vectors (a _i1 , a _i2 ,... A _in ) that are row vectors of the co-occurrence query matrix, and each co-occurrence query vector Are standardized to be the same (S17). The vector dimension number reduction unit 233 removes the co-occurrence frequency of the co-occurrence query, which is so low that the co-occurrence frequency is equal to or lower than a preset threshold value, from the vector component, and reduces the vector dimension number (S18). In the process of reducing the number of vector dimensions, there are other principal expressions such as Harwell-Boeing format, efficient eigenvalue calculation methods such as Ranchos method, principal component analysis using singular value calculation methods and latent meanings. By performing analysis, it is possible to perform processing at high speed in a small memory space, and by treating co-occurrence queries with high correlation by the basis conversion as the same base, an improvement in clustering accuracy can be expected. Alternatively, in the process of reducing the number of vector dimensions, a method of calculating only dimensions other than non-zero elements can be performed. According to this method, the Euclidean distance can be calculated at high speed with simpler logic.

  The Euclidean distance calculation unit 234 calculates the Euclidean distance between the co-occurrence query vectors (S19). FIG. 12 is a table expressing the distance between co-occurrence query vectors based on the Euclidean distance according to the embodiment. As shown in FIG. 12, for example, the Euclidean distance between the search term “Yahoo” and the search term “YAHOO” is small as 1.414214 because the degree of coincidence between the co-occurrence query and the co-occurrence frequency is large. On the other hand, it can be seen that the Euclidean distance between the search term “map” and the search term “GOOGLE” is as large as 2.449490 because the degree of coincidence between the co-occurrence query and the co-occurrence frequency is small.

  The hierarchical clustering unit 235 performs hierarchical clustering by the Ward method for each search word based on the calculated Euclidean distance between the co-occurrence query vectors (S20).

  If the data obtained by clustering is small (S21), the tree diagram output unit 241 outputs the result of hierarchical clustering as a tree diagram as shown in FIGS. 13 and 15 (S22). For example, in FIG. 13, the search term “Yahoo” and the search term “YAHOO” are shown adjacently in the tree diagram, and the search term “OCN” is shown adjacently above them. The words are shown to be similar words.

  On the other hand, when the data obtained by the clustering is large (S21), the cluster formation order analysis unit 242 analyzes the cluster formation order from the result of the hierarchical clustering (S23), and is shown in FIGS. Such a hierarchical structure number is output (S24). FIG. 16 corresponds to the tree diagram of FIG. The number of the hierarchical structure expresses the branching direction of the tree diagram as “0” and “1”. For example, the hierarchical structure number “0000” of the search word “Shinjuku” and the hierarchical structure number “0001” of the search word “Ikebukuro” in FIG. This indicates that “0” and “1” are separated in different directions.

  According to the present embodiment, the search log acquisition unit 211 performs a search by each of a plurality of combinations of search terms in the search server 30 that searches a web page by a combination of a plurality of search terms specified by an unspecified user. The co-occurrence query list creation unit 221 used the search log in combination with the search word for each search word based on the search log acquired by the search log acquisition unit 211. Morphological analysis on documents to create a co-occurrence query list that associates co-occurrence queries, which are other search terms, and co-occurrence frequencies, which are frequencies that are combined with the search terms and used in the search. Acquire the co-occurrence query used in the search by combining with the search term by an unspecified user and the co-occurrence frequency of the co-occurrence query without being affected by the accuracy of It can be.

  In addition, the co-occurrence query matrix creation unit 231 has an n dimension based on the co-occurrence frequency of each of the n co-occurrence queries of the search word based on the co-occurrence query list created by the co-occurrence query list creation unit 221. Since the co-occurrence query vector, which is a vector, is created, the co-occurrence query list can be quantified so as to be suitable for later analysis. Further, the frequency analysis / vector standardization unit 232 performs standardization so that the vector lengths of the co-occurrence query vectors for the search terms created by the co-occurrence query matrix creation unit 231 are the same. Depending on the length, it is possible to prevent an error from occurring in search word clustering.

  Further, the Euclidean distance calculation unit 234 calculates the Euclidean distance between the co-occurrence query vectors for each search term standardized by the frequency analysis / vector standardization unit 232, and the hierarchical clustering unit 235 includes the Euclidean distance calculation unit 234. Clustering is performed for each search term based on the calculated Euclidean distance, so it is assumed that the search terms related to the co-occurrence query vectors with close Euclidean distances have a high degree of coincidence between the co-occurrence queries and the co-occurrence frequencies. Based on the co-occurrence query used in the search combined with the search word by an unspecified user and the co-occurrence frequency of the co-occurrence query without being affected by the accuracy of the morphological analysis on the document, Search terms can be classified with high accuracy. As a result, it is possible to acquire search term classification information suitable for the purpose of assisting Web search.

  That is, according to the present embodiment, since there is no need for document morphological analysis, the processing load is reduced, and high-precision and large-scale clustering processing is possible. Further, according to the present embodiment, there is no need for morphological analysis of a document, and clustering is performed based on a search log obtained by searching for an unspecified number of users, so that it is also effective for unknown words. Furthermore, according to the present embodiment, clustering can be performed with a simpler algorithm than before, and each search term is expressed in the form of a co-occurrence query matrix. Therefore, other clustering methods, similarity determination, etc. Various analyzes can be easily performed.

  Further, according to the present embodiment, the co-occurrence query list creation unit 221 extracts each search term from the search log acquired by the search log acquisition unit 211, and the search server 30 outside the apparatus totals each extracted search term. The co-occurrence query list can be created by acquiring the co-occurrence query and the co-occurrence frequency of the co-occurrence query, so that the co-occurrence query list creation unit 211 itself does not need to add up the search log, and processing load Can be reduced. Further, as a configuration connected to the search word clustering device, the search server 30 need not be provided. Alternatively, in the present embodiment, the co-occurrence query list creation unit 221 adds up the co-occurrence query for each search term and the co-occurrence frequency of the co-occurrence query from the search log acquired by the search log acquisition unit 211. Thus, a co-occurrence query list can be created, so that a co-occurrence query list suitable for subsequent processing can be created.

  Furthermore, according to the present embodiment, the vector dimension number reduction unit 233 calculates the co-occurrence frequency of the co-occurrence query whose co-occurrence frequency is equal to or less than the threshold from the co-occurrence query vector created by the co-occurrence query matrix creation unit 231 as a vector component. Therefore, a co-occurrence query whose co-occurrence frequency that is not suitable for search word clustering is too low can be excluded from the clustering process, the clustering accuracy can be improved, and the processing load of the apparatus can be reduced.

  In addition, in the present embodiment, the hierarchical clustering unit 235 performs clustering with respect to each search word by the Ward method, so that clustering can be performed with high accuracy.

  According to the search word clustering apparatus, search word clustering method, search word clustering program, and recording medium of the present embodiment, the search query at the front end in the search application such as search word expansion in search and navigation to upper and lower hierarchies Can be used for expansion. In addition, according to the present embodiment, by adding a hierarchical classification number as metadata to the index word used for the search, it is possible to perform a high-speed search in consideration of the meaning of the word. In addition, in the present embodiment, the classification information itself can be used as a thesaurus or as an aid for creating a thesaurus.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made. For example, in the above-described embodiment, the description has been made centering on a mode in which a search log is directly acquired from a search server via a network. However, the present invention is not limited to this, for example, when performing search word clustering. Alternatively, a recording medium such as a hard disk in which the search log is recorded may be moved from the search server to the search word clustering apparatus, and the search log may be read from the recording medium by the search word clustering apparatus.

  Further, in the above-described embodiment, the description has been focused on the mode in which the user terminal functions as a search word clustering device. However, the present invention is not limited to this, and for example, is separate from the search server or the search server. In the server, search word clustering may be performed based on the search log by the search word clustering method of the present invention. In this case, the search term clustering method of the present invention is performed based on the search log obtained as a result of performing a search based on a request from the user terminal using the similarity relationship between the search terms obtained by the search term clustering. Search word clustering and thesaurus updates can be performed by performing search word clustering as needed.

It is a figure showing composition of a search system concerning an embodiment. It is a block diagram which shows the functional block of the search server which concerns on embodiment. It is a block diagram which shows the structure of the hardware of the user terminal which concerns on embodiment. It is a block diagram which shows the functional block of the user terminal which concerns on embodiment. It is a flowchart which shows operation | movement of the user terminal which concerns on embodiment. It is a figure which shows the example of the search log of the search server which concerns on embodiment. It is a figure which shows the co-occurrence information of the search word which concerns on embodiment. It is a figure which shows the ranking of the search word which concerns on embodiment. It is a figure which shows the co-occurrence query list which concerns on embodiment. It is a figure which shows the example of the co-occurrence query list which concerns on embodiment. It is a figure which shows the example of the co-occurrence | production query matrix which concerns on embodiment. It is the table | surface expressing the distance between the co-occurrence query vectors by the Euclidean distance which concerns on embodiment. It is a tree diagram which shows the hierarchical structure of the word produced using the service of the search server which concerns on embodiment. It is the figure which showed the hierarchical structure of the word produced using the service of the search server which concerns on embodiment with the number. It is a dendrogram showing a hierarchical structure of words created by aggregating co-occurrence information of search logs according to the embodiment. It is the figure which showed the hierarchical structure of the word created by totaling the co-occurrence information of the search log which concerns on embodiment with the number.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... User terminal, 20 ... Network, 30 ... Search server, 40 ... Information host computer, 101 ... Interface, 102 ... Keyboard, 103 ... Mouse, 104 ... Hard disk, 105 ... CD-ROM drive, 106 ... Memory card slot, 107 ... ROM, 108 ... RAM, 109 ... display device, 110 ... CPU, 210 ... log analysis block, 211 ... search log acquisition unit, 212 ... search log analysis unit, 213 ... co-occurrence information storage unit, 214 ... store search term list 220: Co-occurrence query acquisition block, 221 ... Co-occurrence query list creation unit, 222 ... Co-occurrence query list storage unit, 230 ... Cluster analysis block, 231 ... Co-occurrence query matrix creation unit, 232 ... Frequency analysis / vector standardization Part, 233... Vector dimension number reduction part, 234. Separate calculation unit, 235 ... Hierarchical clustering unit, 240 ... Output block, 241 ... Tree diagram output unit, 242 ... Cluster formation order analysis unit, 243 ... Hierarchical structure number data storage unit, 301 ... Search control unit, 302 ... Search Log DB, 303... Information collection control unit, 304... Collection information DB.

Claims (16)

Search log acquisition means for acquiring a search log that is a record of a search performed by each combination of the plurality of search terms in a search server that searches a web page by a combination of a plurality of search terms specified by an unspecified user; ,
Based on the search log acquired by the search log acquisition means, for each of the search terms, a co-occurrence query that is another search term used in a search combined with the search term, and the co-occurrence query is the A co-occurrence query list creation means for creating a co-occurrence query list associated with a co-occurrence frequency that is a frequency used in a search combined with a search term;
Based on the co-occurrence query list created by the co-occurrence query list creation means, an n-dimensional vector having a numerical value based on the co-occurrence frequency of each of the n co-occurrence queries of the search word as a vector component A co-occurrence query vector creating means for creating a co-occurrence query vector;
Vector standardization means for standardizing the co-occurrence query vectors for each of the search terms created by the co-occurrence query vector creation means so that the vector lengths of the co-occurrence query vectors are the same;
Euclidean distance calculation means for calculating the Euclidean distance between the co-occurrence query vectors for each of the search terms standardized by the vector standardization means;
Clustering means for performing clustering for each of the search terms based on the Euclidean distance calculated by the Euclidean distance calculating means;
A search term clustering device comprising:   The co-occurrence query list creation unit extracts each of the search terms from the search log acquired by the search log acquisition unit, and the co-occurrence query and the co-occurrence tabulated by the search server for each of the extracted search terms. The search word clustering apparatus according to claim 1, wherein the co-occurrence query list is created by acquiring a co-occurrence frequency of a query.   The co-occurrence query list creation unit aggregates the co-occurrence query and the co-occurrence frequency of the co-occurrence query for each of the search terms from the search log acquired by the search log acquisition unit. The search word clustering apparatus according to claim 1, wherein the search word clustering apparatus creates a query list.   A vector dimension that deletes a numerical value based on the co-occurrence query of the co-occurrence query whose numerical value based on the co-occurrence frequency is equal to or less than a threshold from the co-occurrence query vector generated by the co-occurrence query vector generating unit from a vector component The search word clustering device according to any one of claims 1 to 3, further comprising number reduction means.   The search word clustering apparatus according to claim 1, wherein the clustering means performs clustering for each of the search words by a Ward method. A search log for acquiring a search log that is a record of a search performed by each combination of a plurality of search terms in a search server that searches a web page by a combination of a plurality of search terms specified by an unspecified user An acquisition step;
Based on the search log acquired in the search log acquisition step by the computer, for each of the search terms, a co-occurrence query that is another search term combined with the search term and used for the search, and the co-occurrence A co-occurrence query list creation step for creating a co-occurrence query list associated with a co-occurrence frequency that is a frequency used in a search by combining a query with the search term;
Based on the co-occurrence query list created in the co-occurrence query list creation step by the computer, n-dimensions having numerical values based on the co-occurrence frequencies of the n co-occurrence queries of the search term as vector components A co-occurrence query vector creation step for creating a co-occurrence query vector that is a vector;
A vector standardization step in which a computer standardizes the vector length of each of the co-occurrence query vectors for each of the search terms created in the co-occurrence query vector creation step;
A Euclidean distance calculating step in which a computer calculates an Euclidean distance between the co-occurrence query vectors for each of the search terms standardized in the vector standardization step;
A clustering step in which the computer performs clustering for each of the search terms based on the Euclidean distance calculated in the Euclidean distance calculation step;
Search word clustering method including   In the co-occurrence query list creation step, the computer extracts each of the search terms from the search log acquired in the search log acquisition step, and the co-occurrence queries totaled by the search server for each of the extracted search terms and The search word clustering method according to claim 6, wherein the co-occurrence query list is created by acquiring the co-occurrence frequency of the co-occurrence query.   In the co-occurrence query list creation step, the computer totals the co-occurrence query and the co-occurrence frequency of the co-occurrence query for each of the search terms from the search log acquired in the search log acquisition step. The search word clustering method according to claim 6, wherein the co-occurrence query list is created.   The computer deletes the numerical value based on the co-occurrence query of the co-occurrence query whose numerical value based on the co-occurrence frequency is equal to or less than a threshold from the co-occurrence query vector generated in the co-occurrence query vector generation step from the vector component The search word clustering method according to any one of claims 6 to 8, further comprising a vector dimension number reduction step.   10. The search word clustering method according to claim 6, wherein in the clustering step, the computer performs clustering for each of the search words by a Ward method. 11. A search log acquisition step of acquiring a search log that is a record of a search performed by each combination of the plurality of search terms in a search server that searches a web page by a combination of a plurality of search terms specified by an unspecified user; ,
Based on the search log acquired in the search log acquisition step, for each of the search terms, a co-occurrence query that is another search term combined with the search term and used for the search, and the co-occurrence query is the A co-occurrence query list creation step for creating a co-occurrence query list associated with a co-occurrence frequency that is a frequency used in a search combined with a search term;
Based on the co-occurrence query list created in the co-occurrence query list creation step, an n-dimensional vector having a numerical value based on the co-occurrence frequency of each of the n co-occurrence queries of the search word as a vector component A co-occurrence query vector creation step for creating a co-occurrence query vector;
A vector standardization step for standardizing the co-occurrence query vectors so as to have the same vector length for each of the search terms created in the co-occurrence query vector creation step;
A Euclidean distance calculation step of calculating a Euclidean distance between the co-occurrence query vectors for each of the search terms standardized in the vector standardization step;
A clustering step of performing clustering for each of the search terms based on the Euclidean distance calculated in the Euclidean distance calculation step;
Search term clustering program to make computer execute.   In the co-occurrence query list creation step, the computer extracts each of the search terms from the search log acquired in the search log acquisition step, and the co-occurrence queries totaled by the search server for each of the extracted search terms and The search word clustering program according to claim 11, wherein the co-occurrence query list is created by acquiring the co-occurrence frequency of the co-occurrence query.   In the co-occurrence query list creation step, by causing the computer to aggregate the co-occurrence query and the co-occurrence frequency of the co-occurrence query for each of the search terms from the search log acquired in the search log acquisition step. The search word clustering program according to claim 11, wherein the co-occurrence query list is created.   The computer deletes, from the vector component, a numerical value based on the co-occurrence frequency of the co-occurrence query whose numerical value based on the co-occurrence frequency is equal to or less than a threshold from the co-occurrence query vector generated in the co-occurrence query vector generation step. The search word clustering program according to claim 11, further causing a vector dimension number reduction step to be executed.   The search word clustering program according to any one of claims 11 to 14, wherein in the clustering step, the computer is caused to perform clustering by a Ward method for each of the search words.   The computer-readable recording medium which recorded the search word clustering program of any one of Claims 11-15.

Images