JP4915021B2 - Search device and control method of search device - Google Patents

Description

  The present invention relates to a technique for appropriately rearranging the display order of a search result list in a search service such as a Web page according to the degree of harmfulness of a resource hit in a search.

  Currently, various Web pages are open on the Internet, and a user can easily obtain desired information by accessing these Web pages from a terminal. However, on the other hand, there are many web pages that contain adult content or violent content, and it has been a big problem in recent years that young people can easily access such harmful web pages. Is discussed.

In order to control access to such harmful Web pages, a technique called “filtering” is provided. In this “filtering”, for example, filtering software in which NG words and URLs are registered in advance is installed in the terminal. Then, it is determined whether or not the web page accessed by the terminal is a harmful web site according to a matching processing result indicating whether the web page includes the NG word or the domain of the URL. If it is an Internet search server device, for example, as disclosed in Patent Document 1, processing that does not include those determined to be harmful Web sites in the search results is performed.
JP 2007-128119 A

  However, the filtering technique using the registered NG word has a problem that all Web pages including the NG word are excluded from the search result. For example, when the NG word is set to “suicide”, the method for preventing suicide and the page for accepting consultations are actually omitted from the search results, and access to such a healthy web page is also blocked. There is a possibility.

  In addition, as described above, sites that are suspected of being harmful based on NG words and the like are not included in the search results regardless of the degree of harmfulness, that is, the configuration in which Web pages are classified and selected in a binary manner In particular, there is a problem that it is not suitable for an Internet search service. This is because the primary purpose of the Internet search service is to provide users with desired information. Therefore, while it is meaningful to block access to a completely harmful web page, the so-called “gray zone” web page, which is vague whether it is harmful, is excluded. The purpose is meaningless and cannot be achieved.

  Moreover, although it is important to control access to harmful Web pages for young people with poor judgment ability by the above configuration, in a modern information society where information is selected and selected by those who have a certain degree of judgment. It can also take away the opportunity to develop the necessary abilities.

  In order to solve the above problems, the present invention provides a function capable of determining the harmfulness of a Web page by “vector comparison”, which is more effective than simple NG word matching, and a value index. By using a label that indicates the degree of harm in a stepwise manner, for example, instead of omitting Web pages that are considered harmful from the search results, the order in the search result list The search apparatus which makes it difficult to access by lowering is provided.

  Specifically, a hit word vector generation unit that generates a hit word vector that is a word vector having a word extracted from a character string included in each page of the search hit list, and a label indicating a value index are associated with each other A reference word vector holding unit that holds a plurality of reference word vectors, which are reference word vectors, a plurality of reference word vectors that are held, and a hit word vector generated by a hit word vector generation unit The similarity calculation unit that calculates the similarity, the calculation expression storage unit that stores the calculation expression for calculating the similarity, and the identification information of the page that generated the hit word vector are most similar by the calculation Sorted based on the value index of the reference word vector label And sorting the display unit that is a retrieval device having a.

  In order to reduce the processing load of sorting the search result list, for example, a search apparatus having a function of sorting the search results up to the top 100 according to the degree of harmfulness is also provided. Specifically, in addition to the above configuration, the hit word vector generation unit has higher-order generation means for generating hit word vectors for pages included in the search hit list up to a predetermined upper rank in the search hit list. The sort display unit is a search device having an upper sort display means for displaying at least the search hit list up to the upper predetermined order according to the display order generated by the upper generation means.

  In addition, a search device that uses a character string included in a URL of a Web page as a vector feature for determining the degree of harmfulness is also provided. Specifically, the word vector generation unit is a search device having URL character string utilization means for generating a hit word vector using a character string included in the URL of a search hit list page as a feature.

  Also provided is a search device having a function for generating a vector by using a vector feature for judging the degree of harmfulness as an appearance word or the like in a Web page and giving the appearance frequency of the word as a feature value. . Specifically, in addition to the above configuration, the word vector generation unit further determines the size of the word axis in the vector space in accordance with the appearance frequency of the same word included in the page. Is a search device having weighting means for generating.

  In the above description, the search target of the search device of the present invention is not limited to the Web page, and may be other binary data such as moving image / still image data, music data, program data, and the like.

  According to the present invention having the above-described configuration, it is possible to determine the degree of harmfulness in relation to a web page hit by a search step by step, and thereby to sort the arrangement order in the search result list. Here, according to a survey by AOL (registered trademark) that provides an Internet search service, the rate of access to sites listed in the search results of the 8th or lower in the search result list is 3% or less, and the total is 20% or less. Has been reported. As described above, even if the sites that are considered harmful are rearranged below the search result list, the access can be sufficiently suppressed.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to these embodiments, and can be implemented in various modes without departing from the spirit of the present invention.

  In the first embodiment, claims 1 and 5 will be mainly described. In the second embodiment, claims 2 and 6 will be mainly described. In the third embodiment, claims 3 and 7 will be mainly described. In the fourth embodiment, claims 4 and 8 will be mainly described.

Example 1
<Overview>
FIG. 1A is an example of a search hit list returned from a normal search server device when a search query “suicide” is input to a search Web page. As shown in this figure, a search hit list by a normal search server device includes a web page that promotes suicide such as “suicide manual (easy death)” or “overseas suicide image” or a web that does not match public order and morals. Pages may appear at the top of the list.

  This is because, in a normal search server device, the display order of Web pages that hit the search query is determined according to the display order determination algorithm (for example, rearrangement of display order according to the authority of the page determined recursively). This is because the search hit list is generated by rearranging and the basic part of the display order determination algorithm is not designed to take into account the article content itself of the Web page.

  On the other hand, FIG. 1B is an example of a search hit list returned from the search server device in which the search device of this embodiment is incorporated when the same search query “suicide” is input. As shown in this figure, in this search hit list, the Web page that seems to be harmful as described above is displayed in the lower part of the list.

  This is because, in the search device of this embodiment, the similarity of the web page hit by the search is determined by using the reference vector for judging the degree of harmfulness and the vector of the web page hit by the search. This is because the degree of harmfulness is judged and the search hit list is rearranged accordingly.

  As described above, the display order can be automatically rearranged according to the degree of harmfulness of pages in the search hit list using the search device of this embodiment. Therefore, it is possible to suppress access to Web pages that are considered harmful.

<Functional configuration>
FIG. 2 is a diagram illustrating an example of functional blocks in the search device according to the present embodiment. The “search device” is a device having a function of accepting a predetermined search query, specifying a resource including the search query, and presenting identification information and location information of the specified resource to a search query input person. The search device of this embodiment may be incorporated into a server device on a network that provides a so-called Internet search service, and may be realized as a search server device. Or you may implement | achieve the service which is integrated in a terminal device of an end user and searches the resource in a user terminal.

  Further, the search target in the present embodiment is not limited to the Web page exemplified in the outline, and may be other document data, or binary data such as a still image / moving image, sound, and a program.

  As shown in this figure, the “search device” (0200) of this embodiment includes a “hit word vector generation unit” (0201), a “reference word vector holding unit” (0202), and a “similarity calculation”. Part "(0203)," arithmetic expression storage part "(0204), and" sort display part "(0205).

  Note that the functional blocks of the search device described below can be realized as hardware, software, or both hardware and software. Specifically, if a computer is used, a CPU, a main memory, a bus, or a secondary storage device (a hard disk, a non-volatile memory, a storage medium such as a CD or a DVD, a read drive for the medium, etc.), information Input devices used for input, printing equipment, display devices, other hardware components such as external peripheral devices, interfaces for external peripheral devices, communication interfaces, driver programs for controlling these hardware, Other examples include application programs and user interface applications.

  Then, through the arithmetic processing of the CPU according to the program developed on the main memory, the data input from the input device or other interface etc. and stored in the memory or hard disk is processed and stored, or each of the above hardware and software An instruction for controlling the above is generated. In addition, the present invention can be realized not only as a search device but also as a method. A part of the invention can be configured as software. Furthermore, a software product used for causing a computer to execute such software and a recording medium in which the product is fixed to a recording medium are naturally included in the technical scope of the present invention (the same applies throughout the present specification). Is).

  The “hit word vector generation unit” (0201) has a function of generating a hit word vector for each page of the search hit list, and can be realized by a CPU, a main memory, a hit word vector generation program, or the like. The “search hit list” refers to a list of identification information (for example, resource title, URL, ID number of an arbitrary character string, etc.) of resources hit in a search by an input search query. Further, the “hit word vector” refers to a word vector whose feature is a word extracted from a character string included in each page of the search hit list.

  FIG. 3 is a diagram for explaining an example of generation of hit word vectors. As shown in this figure, for example, the CPU interprets the morpheme analysis program included in the hit word vector generation program, and performs morpheme analysis on the text data of the Web page hit in the search query ((a) in FIG. 3). Specifically, a method of performing morphological analysis processing by word decomposition processing of a sentence by pattern matching using a word dictionary or a syntax rule dictionary, or word decomposition processing by scoring using a stochastic language model such as a hidden Markov model Is mentioned. The word extracted by this processing may be an idiomatic phrase composed of a plurality of words depending on the setting of an analysis rule for morphological analysis.

  Then, the word extracted as a result of the morpheme is set as a vector feature of the Web page, as shown in FIG. In other words, the Web page (psychology of suicide prevention) identified by ID: W001 is represented by a vector whose features are words such as “suicide”, “prevention”, “hydrogen sulfide”, “not beautiful”, and “bullying”. It is said that it is done. Then, the contents of the Web page and the like are represented by vectors in this way, and comparison judgment processing with reference word vectors to be described later is performed, so that more effective judgment processing is possible than simple NG word registration.

  The same processing is performed for other pages that have been hit by the search. For example, the web page identified by ID: W002 is “suicide”, “hanging neck”, “medicine”, “jumping”, “easy”, etc. The web page identified by ID: W003 is vectorized with features such as “suicide”, “glo”, “image”, “fun”, “click”, etc. as features. That's it. In addition, the extracted words may be configured to refer to a synonym (synonym) dictionary or the like and collect a plurality of synonyms as one word.

  In addition, a value corresponding to the degree of harmfulness or the like may be given in advance with respect to a word, and a vector feature value may be used. With this configuration, similarity can be determined after weighting according to words in similarity determination by vector comparison described later. Or you may comprise so that a value may be given to the feature of the said vector according to the appearance frequency of a word so that it may mention later in Example 4. FIG.

  Note that the hit word vector generation processing is not limited to the one based on morphological analysis. For example, as described later in the second embodiment, a character string included in the URL of a Web page may be used as a vector feature. If the resource that hits the search is a voice, for example, a morpheme included in the voice may be extracted by frequency analysis using pattern matching to generate a hit word vector related to the voice.

  Alternatively, when the resource that hits the search is a moving image or a still image, for example, a hit image characteristic vector having an image characteristic as its feature instead of a word may be generated. Specifically, for example, the image data is frequency-converted, and an edge (contour) image is generated from the high-frequency component extracted therefrom. Compared with a reference image for pattern matching held in advance, for example, a vector is generated using an object included in the image such as “ax”, “tree”, “mosaic pattern” as a feature. If the resource that hits the search is a program, the source code is extracted, and the processing contents such as “system file execution / copy”, “system area rewriting”, “address book data acquisition and A hit execution vector having a feature such as “mail transmission addressed to the address” may be generated instead of the hit word vector.

  The “reference word vector holding unit” (0202) has a function of holding a plurality of reference word vectors, and is realized by various storage devices such as an HDD (hard disk drive), a nonvolatile memory, an optical recording medium, and its reading drive. be able to.

  The “reference word vector” refers to a reference word vector associated with a label indicating a value index. For example, as shown in FIG. 4, “suicide”, “prevention”, “consultation”, etc. are used as vector features. If it is included, the label “5” indicating the value index indicating safety is associated. On the other hand, when “suicide”, “fun”, “glo”, etc. are vector features, a label “1” indicating a harmful value index is held in association with each other, and “suicide”, “tell me” "," Easy ", etc., the label" 3 "is held in association with the subtle degree of harmfulness.

  The label shown in FIG. 4 is a value index indicating the degree of harmfulness of the Web page, but the value index is not limited thereto. For example, the value index indicating the degree of expertise may be used as the label, and the search hit list may be rearranged in the order of expertise, or the value index indicating the degree of reliability or usefulness may be used as the label, and the trust The search hit list may be rearranged in the order of sex and usefulness.

  Moreover, even if the resource hit in the search is a moving image / still image, sound, program, etc., the vector feature (object image for moving image / still image, frequency data of morpheme, program for sound) If so, a part of the source code) may be stored in association with a label indicating a value index. Specifically, for example, when the object image data of “mosaic pattern” or “shadow” is used as a feature vector, it is assumed that the moving image is not suitable for under 18 years old, so the label is held as “1”. Alternatively, if the vector is characterized by audio frequency data such as “suicide”, “glo”, or broadcast-prohibited terms, it is stored in association with the label “1”, “system file execution / copy”, “system area If it is a vector having a source code for “rewriting” as a feature, it is stored in association with the label “1” because there is a possibility that it is a highly dangerous program such as a virus.

  In this way, by associating a label with a reference word vector and holding it, even if a Web page or the like in a search hit list is an unknown resource, the label is assigned and ranked by determining similarity with the reference. Is possible.

  It should be noted that the generation of the reference word vector and the association of the labels are preferably initially created and input by a service provider or service user and used as a learning example. And after that, it is good to comprise so that the number may be expanded automatically by feedback etc. That is, the hit word vector to which a label is given according to the similarity to the reference word vector by the processing of the search device of the present embodiment described later will be used as a reference word vector in the future in association with the given label. That's it. Alternatively, for example, the same processing is performed on a Web page collected by a program that automatically collects resources on the network (a so-called “crawler program”), and the number of reference word vectors to be held is expanded. Also good. That is, the reference word vector is used as learning data corresponding to a teacher in so-called “supervised machine learning”. By this machine learning, it is possible to increase the accuracy of labeling an unknown resource by vector comparison described later.

  The “similarity calculation unit” (0203) has a function of calculating the similarity between each of the plurality of reference word vectors held and the hit word vector generated by the hit word vector generation unit. Or a main memory, a similarity calculation program, or the like.

  FIG. 5 is a diagram for explaining an example of similarity calculation processing of a reference word vector and a hit word vector in the similarity calculation unit. As shown in FIG. 5A, the hit word vector identified by the ID “W001” in FIG. 3B is arranged in a vector space having the feature as its axis. Subsequently, the reference word vector identified by the ID “R001” in FIG. 4 is arranged in the vector space. If the angle between both vectors is θ, cos θ = (vector W001 × vector R001) / (| vector W001 | × | vector R001 |). The smaller the angle θ between both vectors is, the more similar the features of both vectors are, that is, the similarity between both vectors is indicated by the value of cos θ calculated above.

  Then, regarding the hit word vector of W001, the similarity (cos θ) with the other reference word vectors “R002”, “R003”,. Then, as shown in FIG. 5B, the value of cos θ is close to 1, that is, the top k pieces (k is a predetermined natural number; two in the example shown in the figure) having high similarity to the hit word vector “ R001 "and" R004 "are specified. Then, an average value (rounded up, etc.), median, or mode of the label “5” held in association with “R001” and the label “4” of “R004” in the reference word vector holding unit By using the label of the hit word vector “W001”, a label that becomes a reference for the rearrangement display by the next sort display unit can be given to the resource hit in the search.

  The labeling process for the hit word vector is a similarity determination process called the k nearest neighbor method, but the present embodiment is of course not limited to this method. For example, the similarity may be determined using a support vector machine (SVM) or the like.

  Similarly, other hit word vectors “W002”, “W003”,... Shown in FIG. 3B are similar to the reference word vectors “R001”, “R002”,. Determination processing and label assignment according to the degree of similarity are executed. Then, as shown in FIG. 6, with respect to resources such as Web pages shown in the search hit list, labels of approximate reference word vectors are associated and stored in the storage device.

  In addition, the hit word vector newly provided with the label according to the similarity to the reference word vector as described above may be held as it is in the reference word vector holding unit and used after the next time. With this configuration, the number of reference word vectors held can be automatically expanded.

  The “arithmetic expression storage unit” (0204) has a function of storing an arithmetic expression for calculating the similarity, and is realized by various storage devices such as an HDD, a nonvolatile memory, an optical recording medium, and a reading drive thereof. can do. Examples of the arithmetic expression held here include an arithmetic expression for obtaining cos θ of a vector as described in the similarity determination unit and an arithmetic expression for executing the k nearest neighbor method. However, the arithmetic expression stored here is not limited to this, and any arithmetic expression can be used as long as it determines the similarity between vectors and assigns a label indicating a value index according to the similarity. It doesn't matter.

For example, if the resource is limited to a text-based resource, there is one that uses a so-called “Levenstein distance (editing distance)” indicating the degree of difference between two character strings as another arithmetic expression. Specifically, h ₁ , h ₂ ,... Are set as hit word vector features (character strings), and r ₁ , r ₂ ,. The SIM (similarity) of each morpheme is calculated by the arithmetic expression shown in FIG. The average value is used as a value indicating the similarity between the hit word vector and the reference word vector.

Note that the Levenshtein distance LD (h _a , r _a ) may be calculated using, for example, an algorithm based on dynamic programming as in the conventional case. Also, instead of calculating the SIM for each feature (morpheme) of both vectors, the hit word vector and the reference word vector are considered as one character string having the feature as a character (for example, the vector is “suicide”, “ If “consultation” and “prevention” are used as features, the number of characters is considered to be “a suicide consultation prevention”.

  The “sort display section” (0205) has a function of sorting and displaying the identification information of the page on which the hit word vector is generated in the order of the value index of the label of the reference word vector that is most similar by the above calculation. For example, it can be realized by a CPU, a main memory, a sort display program, and the like.

  Specifically, as shown in FIG. 6, the search hit list as shown in FIG. 1B is obtained by sorting the resources such as Web pages in the search hit list newly assigned with labels in the order of the labels. Is generated. Note that the sorting process of the identification information in the list is not performed using only the label, but may be performed by adding the label value to the normal search hit list sorting rule. In that case, the label value may be used as a variable included in a function for determining a normal sort order.

  In this way, it is possible to sort the order of arrangement in the search result list by stepwise judging the value index such as the degree of harmfulness of the web page hit in the search. Therefore, the search hit list including the web page in the gray zone can be presented to the search user.

  Further, for example, by using a vector having a feature such as a word included in a document of a Web page, a label is given to an unknown resource in the search hit list, and the search hit list is sorted in the order of the labels. it can. In addition, the reference word vector for giving a label to the unknown resource can be automatically expanded by feedback or automatic collection of crawlers, and the labeling accuracy can be improved without taking time.

<Hardware configuration>
FIG. 7 is a schematic diagram illustrating an example of a configuration in the search device when the functional components described above are realized as hardware. The operation of each hardware component in the search hit list display sort process will be described with reference to FIG.

  As shown in this figure, the search device is a hit word vector generation unit, a similarity calculation unit, and a sort display unit, and also executes “CPU (Central Processing Unit)” (0701) that executes various other calculation processes. , “Main memory” (0702). In addition, “HDD” (0703) which is a reference word vector holding unit and arithmetic expression storage unit, “communication IF (interface)” (0704) connected via a network to a search terminal which transmits a search query, and the like I have. They are connected to each other via a data communication path such as a “system bus” to transmit / receive information and process information.

  A program is read into the “main memory”, and the “CPU” interprets the read program, and executes various arithmetic processes according to the interpreted procedure. In addition, a plurality of addresses are assigned to each of the “main memory” and “flash memory”, and in the calculation processing of the “CPU”, the addresses are specified and accessed to store the data. It is possible to perform arithmetic processing using.

  Here, the search query input from the search terminal via the search Web page or the like is received by the “communication IF” and stored in the address 1 of the “main memory”. Then, the search device searches the search index, cache data, etc. for search that are collected in advance by the crawler and stored in the “HDD” by the logical operation process of the “CPU” as in the normal search process. Then, the identification information of the Web page including the character string that matches the search query is stored as the search hit ID at the address 2,... Of “main memory”.

  Subsequently, the “CPU” interprets the hit word vector generation program, and acquires HTML document data of the Web page indicated by the search hit ID from the cache data in “HDD”, for example. Then, the “CPU” refers to a word dictionary (not shown) stored in the “HDD”, and performs pattern matching processing between “words in the dictionary” and “words in the HTML document” by, for example, the longest match method. Next, for the extracted word, the “CPU” similarly refers to a grammar dictionary (not shown), and executes a process for determining whether the word is correctly extracted from the word part-of-speech shown in the grammar dictionary and the connection relationship. As for the extracted word determined to be incorrect as a result of the determination, another delimiter is found by the re-pattern matching process, and the word in the sentence is extracted in a grammatically correct form. Here, the “CPU” may refer to the synonym dictionary and perform a process of collecting the synonyms out of the extracted words into one word. Then, a hit word vector is generated using the extracted word as a vector feature of the Web page. The same processing is performed for Web pages related to other search hit IDs that have been searched, and the generated hit word vectors are stored at addresses 3,.

  Further, the “CPU” interprets the similarity calculation program and reads the reference word vector 1 (R001) stored in the “HDD” to the address 4 of the “main memory” and stores it in the address 3 according to the interpretation result. The similarity with the hit word vector (W001) is calculated. Specifically, for example, each vector value is substituted into an arithmetic expression such as “cos θ = (vector W001 × vector R001) / (| vector W001 | × | vector R001 |)” stored in “HDD”. The cosine distance (cos θ) is calculated. Then, for the hit word list W001, cos θ is calculated in the same manner with other reference word vectors R002, R003,..., And the top k reference word vectors whose cos θ values are close to 1 are specified. Then, by the k nearest neighbor method, the label of the Web page from which the hit word vector is generated is determined as, for example, the mode value label among the labels associated with the top k reference word vectors. The information is stored in association with the identification information of the Web page at address 5 of “memory”.

  In addition, the labels of other Web pages indicated by the search hit IDs are determined by the same processing, and stored in the “main memory” in association with the identification information of each Web page.

  Then, the “CPU” interprets the sort display program, and sorts the identification information of the Web pages stored in association with each other in the order of the value index indicated by the label to generate a sorted search hit list. Store in address 6 of “main memory”. Then, the sorted search hit list is returned to the search terminal that has transmitted the search query via the “communication IF”.

  Further, the label stored at the address 5 of the “main memory” is associated with the hit word vector generated from the Web page, and held in the “HDD” so that it can be used as a reference word vector from the next time. May be.

  If the search target resource is binary data such as a moving image instead of a Web page, hit image characteristic vector generation and similarity determination processing may be performed as described in the above functional configuration.

  The above configuration is an example of a hardware configuration when the search device of the present embodiment is incorporated in a search server device on a network. When the search device according to the present embodiment is incorporated in the terminal device of the end user, for example, the search query may be input through an input device (not shown) instead of the “communication IF”. In addition, the sorted search hit list may be output and displayed directly on the display of the terminal itself instead of being output from the “communication IF”.

<Process flow>
FIG. 8 is a flowchart illustrating an example of a processing flow in the search device according to the present embodiment. The steps shown below may be steps executed by each hardware configuration of the computer as described above, or may be processing steps that constitute a program for controlling the computer recorded on a medium. I do not care.

  As shown in this figure, first, when a search query is received (step S0801), page titles that hit the received search query are listed, and a search hit list is acquired (step S0802). Subsequently, for each page of the search hit list, for example, morphological analysis is performed, and a hit word vector having the morpheme as a feature is generated (step S0803). Then, the similarity between each of the stored reference word vectors and the hit word vector is calculated using, for example, a cosine distance using an arithmetic expression stored in advance (step S0804).

  Then, the page titles and the like in the search hit list are sorted and displayed in the order of the value index indicated by the reference word vector label that is most similar by the calculation (step S0805). Further, with respect to a page to which a label is given in this way, the hit word vector generated from the page may be associated with the label and stored in the HDD or the like so that it can be used as a reference word vector from the next time.

  Of course, this processing flow may be a processing flow in a search server device on a network or a processing flow in an end user terminal. In the former case, step S0801 is a step of accepting a search query input at the terminal via the network, and in the latter case, step S0801 is a step of accepting a search query input directly to the terminal via an input device or the like. It becomes. Further, step S0805 is a step of displaying the search hit list on the terminal by returning the sorted list in the former case to the terminal that has transmitted the search query. In the latter case, the search is directly performed on its own display. This is a step of displaying a hit list (the same applies to the processing flow of the following embodiments).

<Brief description of effect>
As described above, the search device according to the present embodiment sorts the arrangement order in the search result list by making a step-by-step determination using the value index such as the degree of harmfulness regarding the Web page hit by the search. Can do. Therefore, the search hit list including the web page in the gray zone can be presented to the search user.

  Further, for example, by using a vector having a feature such as a word included in a document of a Web page, a label is given to an unknown resource in the search hit list, and the search hit list is sorted in the order of the labels. it can. In addition, the reference word vector for giving a label to the unknown resource can be automatically expanded by feedback or automatic collection of crawlers, and the labeling accuracy can be improved without taking time.

<< Example 2 >>
<Overview>
The present embodiment is a search apparatus that is based on the above-described embodiment, and that is configured to perform only a partial sort display, for example, one page is displayed as a search hit list.

  FIG. 9 is a diagram illustrating an example of a search hit list sorted by the search device according to the present embodiment. As shown in FIG. 9 (a), among 60000 hits in the search query, 1 to 10 hit results are displayed as a 1-page search hit list. The calculation process for sort display as described in Example 1 is performed. However, as shown in FIG. 9B, the calculation processing for the sort display is not performed for the subsequent hit results from 11 to 20 cases. The search hit list on the second page is configured such that, for example, the user clicks “Next” on the first page and performs a display operation on the second page, and then performs a calculation process for sorting display again. May be. Alternatively, while the user is browsing the first page, a calculation process for sorting and displaying the search hit list after the next page may be performed in the background process.

  In this embodiment, with this configuration, it is possible to reduce or distribute the processing load when sorting the page titles in the search hit list, and to increase the search hit list display speed.

<Functional configuration>
FIG. 10 is a diagram illustrating an example of functional blocks in the search device according to the present embodiment. As shown in this figure, the “search device” (1000) of the present embodiment is based on the first embodiment and includes a “hit word vector generating unit” (1001), a “reference word vector holding unit” (1002), It has a “similarity calculation unit” (1003), an “arithmetic expression storage unit” (1004), and a “sort display unit” (1005). In addition, since these structural requirements have already been described in the above embodiment, the description thereof will be omitted.

  The search device according to the present embodiment is characterized in that the hit word vector generation unit further includes “upper generation unit” (1006) and the sort display unit further includes “upper sort display unit” (1007). To do.

  The “upper generation means” (1006) has a function of generating hit word vectors for pages included in the search hit list up to a predetermined rank in the search hit list. Specifically, before generating a hit word vector, a search hit list in which the display order is determined according to the same rules as in the past is acquired, and for example, the top 100 web pages from the search hit list, etc. The hit word vector is generated for the target.

  The "predetermined upper rank" can be appropriately determined to reduce the sort processing load of the search device. However, as described above in the overview, the search hit list is divided into a plurality of pages and displayed. In this case, it is preferable to generate a hit word vector for the pages in the upper predetermined order included in the one page display of the search hit list.

  Further, when the upper generation unit is configured as such, as described in the overview, when an operation for displaying the next search hit list page is received, the Web included in the next search hit list page is included. The hit word vector generation unit of the present embodiment may further include “next page operation generation means” for specifying a page and generating the hit word vector. Alternatively, while displaying the search hit list sorted according to the label, the “next page background generation means” that specifies the Web page included in the next search hit list page in the background and generates the hit word vector. May further be included in the hit word vector generation unit of the present embodiment.

  The “upper sort display means” (1007) sorts and displays at least the search hit list up to the upper predetermined rank based on the value index of the label of the reference word vector that is most similar by the calculation. It has a function. Further, “next page sort display means” may be further provided in correspondence with the “next page operation generation means” and “next page background generation means” described above.

  In this way, in this embodiment, the processing load when sorting the page titles in the search hit list can be reduced or distributed, and the display speed of the sorted search hit list can be increased.

<Process flow>
FIG. 11 is a flowchart illustrating an example of a processing flow in the search device according to this embodiment. The steps shown below may be steps executed by each hardware configuration of the computer as described above, or may be processing steps that constitute a program for controlling the computer recorded on a medium. I do not care.

  As shown in this figure, first, when a search query is received (step S1101), page titles that hit the received search query are listed, and a search hit list is acquired (step S1102). Subsequently, for each page included in the search hit list, for example, the display order is determined according to the same rules as in the prior art, and then the hit pages up to the upper predetermined order are specified (step S1103). Then, for example, morpheme analysis is performed on each of the identified hit pages, and a hit word vector having the morpheme as a feature is generated (step S01104). Then, the similarity between each of the stored reference word vectors and the hit word vector is calculated using, for example, a cosine distance using an arithmetic expression stored in advance (step S1105).

  Then, at least the pages in the search hit list up to a predetermined upper rank are sorted and displayed in the order of the value index indicated by the reference word vector label that is most similar by the calculation (step S1106).

<Brief description of effect>
As described above, in the search device according to the present embodiment, at least initially, vector generation and similarity calculation processing are executed for pages up to a predetermined upper rank in the search hit list. Therefore, it is possible to reduce or distribute the processing load when sorting the page titles in the search hit list, and to increase the display speed of the sorted search hit list.

Example 3
<Overview>
FIG. 12 is a diagram for explaining an example of the hit word vector generation process in the search device according to the present embodiment. As shown in FIG. 12A, for example, a URL (Uniform Resource Locator) is acquired for a Web page that has been hit by a search. In general, the URL often includes a character string having relevance to the contents of the Web page. In addition, when the same domain name is included, a plurality of pages constituting one Web site, for example, a “page that collects images of Southeast Asia” that constitutes one “site introducing overseas suicide images”, “an African image” There is a high possibility that the web page has the same content, such as “a page that collects“.

  Thus, as shown in FIG. 12B, the search device of this embodiment generates a hit word vector using a character string included in the URL as a feature and uses it for labeling for sorting display. To do.

<Functional configuration>
FIG. 13 is a diagram illustrating an example of functional blocks in the search device according to the present embodiment. As shown in this figure, the “search device” (1300) of the present embodiment is based on the first embodiment and includes a “hit word vector generation unit” (1301), a “reference word vector holding unit” (1302), It has a “similarity calculation unit” (1303), an “arithmetic expression storage unit” (1304), and a “sort display unit” (1305). Further, on the basis of the second embodiment, “upper generation means” and “upper sort display means” (not shown) may be provided. In addition, since these structural requirements have already been described in the above embodiment, description thereof will be omitted.

  The search device according to the present embodiment is characterized in that the hit word vector generation unit further includes “URL character string use means” (1306).

  The “URL character string using means” (1306) has a function of generating a hit word vector using a character string included in the URL of the page of the search hit list as a feature. The URL is made up of a communication protocol name, a host machine name holding the resource, a path name indicating a directory structure holding the resource in the host machine, and the like. As described above, this URL often includes a character string associated with the contents of the Web page. Alternatively, when the same domain name is included, the same Web site is configured, so that there is a high possibility that the Web pages have similar contents. In other words, since it is possible to determine the similarity of Web pages by comparing character strings included in the URL, the URL character string using means generates a hit word vector using the character string included in the URL as a vector feature. It is a state of doing.

  In the URL, character strings are separated by slashes (/), dots (.), And the like. Therefore, it is possible to easily specify a character string that is a vector feature by referring to these delimiters without performing complicated morphological analysis processing using a word dictionary or a grammar dictionary. Also, in this embodiment, a protocol name such as “http”, an extension such as “.html”, a domain such as “.com”, or a path name included in the URL for general purposes regardless of the page contents such as TOP. These character strings may be registered in advance so that they do not have vector features.

  In the present embodiment, the reference word vector held in the reference word vector holding unit is naturally configured to feature “picture”, “movie”, etc. in accordance with the generated hit word vector. It is.

<Process flow>
FIG. 14 is a flowchart illustrating an example of a processing flow in the search device according to the present embodiment. The steps shown below may be steps executed by each hardware configuration of the computer as described above, or may be processing steps that constitute a program for controlling the computer recorded on a medium. I do not care.

  As shown in this figure, first, when a search query is received (step S1401), page titles that hit the received search query are listed, and a search hit list is acquired (step S1402). Next, for each page included in the search hit list, character strings are distinguished by referring to the delimiters of the URL, and a hit word vector having the character string as a feature is generated (step S01403). Then, the similarity between each of the stored reference word vectors and the hit word vector is calculated using, for example, a cosine distance using an arithmetic expression stored in advance (step S1404).

  Then, the page titles and the like in the search hit list are sorted and displayed in the order of the value index indicated by the reference word vector label that is most similar by the calculation (step S1405).

<Brief description of effect>
As described above, the display order can be sorted using the character string included in the URL of a resource such as a Web page by the search device of this embodiment. Therefore, the similarity can be determined using, for example, whether the content is included in the same Web site.

  In addition, it is possible to generate a hit word vector by determining a feature by processing such as “refer to a delimiter” that is lighter in load than morphological analysis.

Example 4
<Overview>
FIG. 15 is a diagram for explaining an example of the hit word vector generation process in the search device according to the present embodiment. As shown in this figure, the present embodiment is characterized in that a feature value is assigned according to the appearance frequency of the word regarding the feature of the hit word vector. Since the vector characteristic is expressed more clearly by giving the vector feature value for each feature in this way, the similarity can be judged more strictly by comparison with the reference word vector.

<Functional configuration>
FIG. 16 is a diagram illustrating an example of functional blocks in the search device according to the present embodiment. As shown in this figure, the “search device” (1600) of the present embodiment is based on the first embodiment and includes a “hit word vector generation unit” (1601), a “reference word vector storage unit” (1602), It has a “similarity calculation unit” (1603), an “arithmetic expression storage unit” (1604), and a “sort display unit” (1605). Further, on the basis of the second and third embodiments, it may have “upper generation means”, “upper sort display means”, or “URL character string use means” (not shown). In addition, since these structural requirements have already been described in the above embodiment, description thereof will be omitted.

  The search apparatus according to the present embodiment is characterized in that the hit word vector generation unit further includes “weighting means” (1606).

  The “weighting means” (1606) has a function of generating a hit word vector in which the size of the word axis in the vector space is determined in accordance with the appearance frequency of the same word included in the page.

  FIG. 17 is a diagram for explaining an example of hit word vector generation processing and similarity determination processing with reference word vectors by this weighting means. As shown in FIG. 17A, regarding the Web page “W001” included in the search hit list, the frequency of appearance of words obtained as a result of morphological analysis is counted. Then, the appearance frequency is stored as a feature value in a flash memory or the like. Similarly, for other Web pages “W002”, “W003”,..., Etc., the appearance frequency of words is similarly counted, and a hit word vector is generated and stored using the count value as a feature value.

  On the other hand, as shown in FIG. 17B, a feature value is appropriately given for the vector feature of the reference word vector. Of course, if the reference word vector is generated based on some web page or the like, the feature value may be given by the appearance frequency.

  Then, as shown in FIG. 17C, the similarity between “W001” and “R001” is shown by an arithmetic expression such as cos θ = (vector W001 × vector R001) / (| vector W001 | × | vector R001 |). In order to calculate the index, the index value cos θ varies depending on the feature value. Therefore, the similarity can be judged more strictly.

  The size of the axis determined according to the appearance frequency may be corrected by referring to a part of speech or a correction table set according to the content of the character. Specifically, the size of a word that seems to be harmful, for example, a word such as “glo” is determined by doubling the appearance frequency. Or, when there is a combination of words such as “suicide” and “easy method”, the appearance frequency is similarly corrected.

  In the weighting means of this embodiment, when the search target is other than a Web page, the size of the vector axis is determined by using its feature other than a word, for example, the appearance frequency of a predetermined frequency if it is speech as a value. It is good to decide.

<Process flow>
FIG. 18 is a flowchart illustrating an example of a processing flow in the search device according to the present embodiment. The steps shown below may be steps executed by each hardware configuration of the computer as described above, or may be processing steps that constitute a program for controlling the computer recorded on a medium. I do not care.

  As shown in this figure, first, when a search query is received (step S1801), page titles that hit the received search query are listed, and a search hit list is acquired (step S1802). Subsequently, for each page of the search hit list, for example, morphological analysis is performed, and the appearance frequency of words included in the page is counted. The morpheme is used as a feature, and a hit word vector is generated using the counted value as the feature value (step S1803). Then, the similarity between each of the stored reference word vectors and the hit word vector is calculated using, for example, a cosine distance using an arithmetic expression stored in advance (step S1804).

  Then, the page titles and the like in the search hit list are sorted and displayed in the order of the value index indicated by the reference word vector label that is most similar by the calculation (step S1805).

<Brief description of effect>
As described above, the search device of this embodiment generates a hit word vector to which a feature value is assigned according to the appearance frequency of words. Therefore, since the vector characteristic is expressed more clearly, the similarity can be judged more strictly by comparison with the reference word vector.

The figure for demonstrating an example of the sorting display of the search hit list by the search device of Example 1. The figure showing an example of the functional block in the search device of Example 1. The figure for demonstrating an example of the production | generation of the hit word vector in the hit word vector production | generation part of the search device of Example 1. The figure showing an example of the reference word vector currently hold | maintained at the reference word vector holding part of the search device of Example 1. The figure for demonstrating an example of the similarity calculation process of the reference word vector and hit word vector in the similarity calculation part of the search device of Example 1. The figure showing an example of labeling to the Web page hit by the search by similarity judgment of the search device of Example 1 1 is a diagram illustrating an example of a hardware configuration in a search device according to Embodiment 1. FIG. 7 is a flowchart illustrating an example of a processing flow in the search device according to the first embodiment. The figure for demonstrating an example of the sort display of the search hit list by the search device of Example 2. The figure showing an example of the functional block in the search device of Example 2. 7 is a flowchart illustrating an example of a process flow in the search device according to the second embodiment. The figure for demonstrating an example of the generation processing of the hit word vector in the search device of Example 3. The figure showing an example of the functional block in the search device of Example 3. 7 is a flowchart illustrating an example of a process flow in the search device according to the third embodiment. The figure for demonstrating an example of the generation processing of the hit word vector in the search device of Example 4. The figure showing an example of the functional block in the search device of Example 4. The figure for demonstrating an example of the generation process of the hit word vector by the weighting means of the search device of Example 4, and the similarity judgment process with a reference word vector 10 is a flowchart illustrating an example of a processing flow in the search device according to the fourth embodiment.

Explanation of symbols

0200 Search device 0201 Hit word vector generation unit 0202 Reference word vector holding unit 0203 Similarity calculation unit 0204 Calculation expression storage unit 0205 Sort display unit

Claims (8)

A hit word vector generation unit that generates a hit word vector that is a word vector having a word extracted from a character string included in each page of the search hit list;
A reference word vector holding unit that holds a plurality of reference word vectors that are reference word vectors associated with a label indicating a value index;
A similarity calculator that calculates the similarity between each of the plurality of reference word vectors held and the hit word vector generated by the hit word vector generator;
An arithmetic expression storage unit that stores an arithmetic expression for calculating the degree of similarity;
A sort display unit for sorting and displaying the identification information of the page that generated the hit word vector based on the value index of the label of the reference word vector that is most similar by the calculation;
A search device having: The hit word vector generation unit
In the search hit list, there is an upper generation means for generating a hit word vector for pages included in the search hit list up to the upper predetermined order,
Sort display section
2. The high-order sort display means for sorting and displaying at least high-order search hit lists up to a predetermined rank based on a value index of a reference word vector label that is most similar by the calculation. Search device.   The search device according to claim 1, wherein the hit word vector generation unit includes URL character string use means for generating a hit word vector using a character string included in a URL of a search hit list page as a feature.   The hit word vector generation unit includes weighting means for generating a hit word vector that defines the size of the word axis in the vector space in accordance with the appearance frequency of the same word included in the page. 4. The search device according to any one of 3. A control method for a search device having a reference word vector holding unit that holds a plurality of reference word vectors, which are reference word vectors associated with a label indicating a value index,
A hit word vector generation step for generating a hit word vector that is a word vector having a word extracted from a character string included in each page of the search hit list as a feature;
Using the arithmetic expression for calculating the similarity stored in advance in the arithmetic expression storage unit, each of the plurality of reference word vectors held, and the hit word vector generated by the hit word vector generation unit, A similarity calculation step for calculating the similarity of
Sort display step for sorting and displaying the identification information of the page that generated the hit word vector based on the value index of the label of the reference word vector that is said to be most similar by the calculation,
Of a search apparatus for causing a computer to execute the above. The hit word vector generation step
In the search hit list, including an upper generation step of generating a hit word vector for pages included in the search hit list up to the upper predetermined order,
Sort display step is
6. The high order sort display step of sorting and displaying at least high-order search hit lists up to a predetermined high order based on a value index of a reference word vector label that is most similar by the calculation. Method for controlling a search apparatus of a computer.   7. The search device control according to claim 5, wherein the hit word vector generation step includes a URL character string use step of generating a hit word vector using a character string included in a URL of a search hit list page as a feature. Method.   6. The hit word vector generation step includes a weighting step of generating a hit word vector in which the size of the word axis in the vector space is determined according to the appearance frequency of the same word included in the page. The control method of the search device according to any one of 7.

Images