JP2009277156A - Filtering device and filtering method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem of a conventional filtering technology of using a registered NG word and URL wherein "Omission in filtering" arises as a result and a sound Web site may also be filtered as a result. <P>SOLUTION: A harmful Web site used as a model of comparison is previously prepared, and the similarity between the content of the Web site and the content of the Web site collected by a crawler is determined synthetically using "load result of appearance frequency of a word resulting from morpheme analysis of both Web documents". This filtering device has a function of automatically creating a reference database for harmful/harmless determination of the Web site according to the determination result. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for effectively performing a so-called “filtering” technique for prohibiting access to harmful websites and the like.

  Currently, various websites are open on the Internet, and a user can easily know desired information by accessing these websites from a terminal. However, on the other hand, there are many websites that contain adult content, violent content, etc., and in recent years it has been easy for young people to access such harmful websites. Has been discussed as a big problem.

  In order to control access to such harmful Web sites, a technique called “filtering” is provided. In this “filtering”, for example, if an NG word or URL is registered in the filtering software in advance and the Web site accessed by the terminal contains the NG word or the domain of the URL, it is harmful. A process is performed in which it is determined that the web site is not displayed and no display is performed.

A similar function is also implemented in the search server device, and in Patent Document 1, a matching process for matching a registered NG word, URL, or the like is performed on a website that has been hit by a search, and the website is determined to be a harmful website. Things are not included in the search results.
JP 2007-128119 A

  However, in the filtering technique using the registered NG word or URL, it is necessary to register the NG word or URL as appropriate. In particular, regarding the registration of URLs, considering the current state of networks where the number of Web sites has increased explosively on a daily basis, it is clear that the number of URL registrations for filtering has not caught up with the speed of the increase in sites. Therefore, there is a problem that “filtering omission” occurs as a result.

  Further, in the case of matching determination processing using NG words, even if an NG word matches, an unknown newly opened website can be targeted for filtering. However, for example, in the case of matching at the registered word (word) level, even a Web site including the word “tobacco”, for example, may actually be a site for enlightening the adverse effects of underage smoking. Therefore, there is a problem that a sound website may be filtered as a result.

  In addition, by the Ministry of Internal Affairs and Communications that questioned the effectiveness of the above filtering technology, at the beginning of 2008, mobile phone sites equipped with SNS (Social Networking Service), blogs, etc. were collectively classified as dangerous sites. A notice to prohibit access has been issued to mobile phone carrier companies. However, such a collective prohibition process, as a matter of course, uniformly prohibits many sites including healthy sites, which may result in hindering the development of the network-related market.

  In order to solve the above problems, it has the following features, so that it has a function of automatically registering and updating registration words and URLs so that there is no filtering omission, and even if it is an unknown website, Provided is a filtering device capable of performing filtering that is more effective than filtering based on simple NG word comparison.

  That is, a harmful website serving as a comparison model is prepared in advance, and the similarity between the contents of the website and the contents of the website collected by the crawler is obtained as a result of morphological analysis of both web documents. A comprehensive determination is made using the “weighted result of word appearance frequency”. A filtering device is provided that has a function of automatically generating a reference database for determining whether a website is harmful or harmless according to the determination result.

  And in order to implement | achieve the said characteristic function, the filtering apparatus of this invention specifically performs the morphological analysis process of the analysis object sentence obtained by the crawler part which acquires the analysis object sentence, and the crawler part in the following structures. A morpheme analysis processing unit, an analysis target weighted result acquisition unit that obtains an analysis target weighted result that is a result of weighting a keyword obtained by morpheme analysis processing based on an appearance frequency in the analysis target sentence, and a morphological analysis of a sample sentence A sample weighted result acquisition unit that acquires a sample weighted result that is a result of weighting the keyword obtained by the above based on the appearance frequency in the sample sentence, and the analysis target weighted result and the sample weighted result are assigned values. An arithmetic expression storage unit for storing a predetermined arithmetic expression to be calculated, an analysis target weight result, a sample weight result, and an arithmetic expression And the similarity calculation unit for obtaining the similarity between the analysis target sentence and the sample sentence, and if the calculation result in the similarity calculation part satisfies the predetermined condition, the identification information of the analysis target sentence is used as the sample sentence. And a similar sentence accumulating unit for accumulating that the sentence is similar to.

  In addition, when performing a search using a search server or a terminal, a filtering device further provided with a function of performing effective filtering on the search result by performing the above-described processing on the Web site of the search result is provided. Specifically, the filtering device includes, in addition to the above configuration, a search sentence identification information acquisition unit that acquires search sentence identification information indicating a search result, and search sentence identification information acquired by the search sentence identification information acquisition unit. And a comparison unit for comparing with the identification information stored in the similar text storage unit.

  Also provided is a filtering device that further has a function of acquiring the URL of the access destination at the terminal and determining whether the Web site indicated by the URL to be accessed is a harmful site by determining the similarity. . Specifically, the filtering device includes: a URL receiving unit that receives a URL; an analysis target sentence acquiring unit that acquires a sentence of a web page identified by the received URL as an analysis target sentence; And an identification information output unit that outputs the fact to the URL transmission source when the result is a calculation result that satisfies a predetermined condition.

  According to the present invention having the above-described configuration, if a harmful Web page that is a model to be compared is prepared in advance, the URL of the harmful Web site can be automatically updated and registered. Therefore, it is possible to deal with newly opened unknown websites without omission of filtering. Moreover, in order to comprehensively determine the similarity to the harmful website as a model, using a weighted result of the appearance frequency of words obtained as a result of morphological analysis of both Web documents, rather than simply comparing registered words, More effective filtering can be realized.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to these embodiments, and can be implemented in various modes without departing from the spirit of the present invention.

  The first embodiment will mainly describe claims 1, 5, 7, and 11. In the second embodiment, claims 2, 3, 8, and 9 will be mainly described. In the third embodiment, claims 4, 6, 10, and 12 will be mainly described.

  Example 1

    <Overview>

  FIG. 1 is a conceptual diagram for explaining an example of determination processing of a filtering target website by the filtering device of the present embodiment. As shown in FIG. 1 (a), the filtering device of the present embodiment performs morphological analysis processing on document data (sample text) of a model site “Purchasing agent for overseas cigarettes” of a harmful website registered in advance. Do. As a result of the morphological analysis, the top 100 keywords having the highest number of appearances are extracted in the form of “cigarette”, “circle”,. Then, according to the appearance frequency of each keyword in the Web text, for example, “cigarette 64”, “circle 31”,.

  Further, as shown in FIG. 1B, the morphological analysis process is similarly performed on a “cigarette risk” site that is automatically acquired by a crawler program or the like and is unclear as to whether it is a harmful site or not. Then, the appearance keywords in the Web document, such as “tobacco 72”, “nicotine 28”,.

  In the filtering device of this embodiment, instead of simply determining whether the extracted keywords match, the weighted points given to the keywords are multiplied for each matching keyword, and the total value of the weighted point products is used. The similarity between both document data is judged. In the above example, both websites contain the same word, such as cigarettes, while other words are different. For this reason, the total weighted point product of both document data is lower (than the predetermined value), and therefore the two documents are not similar in content, and it is determined that this “cigarette risk” site is a safe site. Is done.

  On the other hand, for a website that has a high weighted score and whose document content is determined to be similar to the sample text, the URL is databased as a harmful website. As described above, the filtering device of this embodiment can automatically perform update registration (database creation) of URLs of harmful websites. In addition, regarding the process for determining whether or not to register in the database, a harmful Web that serves as a model is used by using a weighted result of the appearance frequency of words obtained as a result of morphological analysis of both Web documents, rather than simply comparing registered words. Since the similarity to the site is comprehensively determined, more effective filtering can be realized.

    <Functional configuration>

  FIG. 2 is a diagram illustrating an example of functional blocks in the filtering device of the present embodiment. As shown in this figure, the “filtering device” (0200) of this embodiment includes a “crawler unit” (0201), a “morpheme analysis processing unit” (0202), and an “analysis target weighted result acquisition unit” (0203). ), “Sample weighted result acquisition unit” (0204), “arithmetic expression storage unit” (0205), “similarity calculation unit” (0206), and “similar sentence storage unit” (0207). .

  Note that the functional blocks of the apparatus described below can be realized as hardware, software, or both hardware and software. Specifically, if a computer is used, a CPU, a main memory, a bus, or a secondary storage device (a hard disk, a non-volatile memory, a storage medium such as a CD or a DVD, a read drive for the medium, etc.), information Input devices used for input, printing equipment, display devices, other hardware components such as external peripheral devices, interfaces for external peripheral devices, communication interfaces, driver programs for controlling these hardware, Other examples include application programs and user interface applications.

  Then, through the arithmetic processing of the CPU according to the program developed on the main memory, the data input from the input device or other interface etc. and stored in the memory or hard disk is processed and stored, or each of the above hardware and software An instruction for controlling the above is generated. In addition, the present invention can be realized not only as an apparatus or a system but also as a method. A part of the invention can be configured as software. Furthermore, a software product used for causing a computer to execute such software and a recording medium in which the product is fixed to a recording medium are naturally included in the technical scope of the present invention (the same applies throughout the present specification). Is).

  The “crawler unit” (0201) has a function of acquiring an analysis target sentence, and can be realized by acquiring document data of a new website on the network as an analysis target sentence by a general crawler program, for example. Of course, the function of the crawler unit is not limited to the acquisition of the analysis target text by the crawler program. For example, the crawler unit is realized by a user input device, a GUI, and the like. You may comprise so that the input sentence may be acquired.

  In the filtering apparatus according to the present embodiment, the document content of the new website acquired in this way is compared with the content of sample text indicating a model case such as a harmful website with the following configuration.

  The “morpheme analysis processing unit” (0202) has a function of performing a morpheme analysis process on the analysis target text obtained by the crawler unit, and can be realized by an arithmetic device such as a CPU or a morpheme analysis program. As for specific processing examples in this morphological analysis processing unit, as in the past, for example, word decomposition processing of sentences by pattern matching using a word dictionary, a syntax rule dictionary, etc., or a probabilistic language model such as a hidden Markov model It is good to carry out by word decomposition processing by scoring using. Of course, the extracted word may be an idiomatic phrase composed of a plurality of words depending on the setting of analysis rules for morphological analysis.

  And about the word extracted in this way, the filtering apparatus of a present Example assign | provides a weighted score with the following structures for the comparison with a sample sentence, It is characterized by the above-mentioned.

  The “analysis target weighted result acquisition unit” (0203) has a function of acquiring the analysis target weighted result, and can be realized by an arithmetic unit such as a CPU or an analysis target weighted result acquisition program, for example. “Analysis target weighted result” refers to a result obtained by weighting a keyword obtained by morphological analysis processing based on the appearance frequency in the analysis target sentence. Further, “weighting based on the appearance frequency” means a process of assigning a different value for each word according to the appearance frequency in the sentence. For example, the number of occurrences of all words in a sentence (for example, only a noun case may be used, and particles and auxiliary verbs may be excluded) is used as a parameter, and the number of occurrences of each of words A, B, C,. It is a condition that the ratio which shows is made into the weight value. In addition, the number of appearances may be used as a weight value.

  The extracted words may be configured such that a plurality of synonyms (synonyms) are collected as one word by referring to a synonym (synonyms) dictionary and the like and weighted according to the appearance frequency. Further, for example, according to the part of speech or the content of the character, a correction table or the like may be referred to correct the weight value for each word. Specifically, if there is a word or combination that seems to be harmful, such as “adult”, “drug” and “purchase” or a combination of words, the weight is corrected to be higher, etc. is there.

  The “sample weighted result acquisition unit” (0204) has a function of acquiring a sample weighted result, and can be realized by an arithmetic unit such as a CPU, a storage device such as a flash memory, a sample weighted result acquisition program, or the like. . The “sample weighted result” refers to a result obtained by weighting a keyword obtained by morphological analysis of a sample sentence based on the appearance frequency in the sample sentence. “Sample text” refers to text indicating a model site that can be filtered, and of course may be text data of an actual website or text data of a fictitious website. . Further, the sample text needs to be input and registered in advance by a manager or the like. Further, since the sample sentences are already registered in this way, only the weighted result may be calculated in advance to acquire a database. Alternatively, if the sample sentence is an actual harmful website and the contents of the sentence are frequently updated, a new acquisition of the sample sentence and a calculation acquisition process of the sample weighted result are performed each time the comparison with the analysis target sentence is performed. You may comprise as follows.

  Note that the sample weighted result obtained by the sample weighted result obtaining unit is only the difference between the sentence being the sample sentence and the sentence to be analyzed, and the calculation method of the weighted result itself is the same as that of the above-mentioned analysis object weighted result. Therefore, the description is omitted.

  In the filtering apparatus of the present embodiment, the analysis target weighted result and the sample weighted result calculated and acquired in this way are associated with each sentence ID to generate table data and the like, and a flash memory or the like for comparing both sentence data Hold at.

  FIG. 3 is a diagram illustrating an example of the data table of the analysis target weighted result and the sample weighted result stored in the flash memory or the like as described above. As shown in FIG. 3, for example, the source address (URL, etc.) of the sentence, the appearance keyword, and the weight value thereof are shown in association with each “sentence ID”. That is, in the top record in the table data shown in FIG. 3A, “tobacco (weight 72)” or “forbidden symptom” (extracted word) is used as the extracted word in the analysis target sentence indicated by “sentence ID: A001”. The weight value 4) ”is shown.

  Further, as shown in FIG. 3B, “tobacco (weight 64)”, “recommendation (weight 2)”, and the like are shown as extracted words in the sample sentence indicated by “sentence ID: S001”. That is.

  In the filtering device of the present embodiment, the analysis target sentence is determined by using the analysis target weighted result and the sample weighted result held in this manner and determining the similarity between the two sentence data with the following configuration. It is determined whether or not such a website should be subject to filtering.

  The “arithmetic expression storage unit” (0205) has a function of storing a predetermined arithmetic expression that uses the analysis target weighted result and the sample weighted result as substitution values. For example, a flash memory, an HDD (hard disk drive), an optical disk, It can be realized by the reading drive or the like. In addition, the “arithmetic expression” is, for example, based on an expression that calculates the product of weighted values of the same word between the analysis target sentence and the sample sentence, and adds all the calculated values of all words. Examples include a formula for multiplying each parameter by a correction value or adding a correction value. Then, when the same word frequently appears in each of the sentence data, the calculation result of the above-described arithmetic expression takes a larger value than when the same word does not frequently appear. Therefore, by comparing the calculation results based on such calculation expressions, it is possible to determine the similarity of appearing words between two sentences. Then, the similarity of contents between two sentences can be estimated from the similarity of the appearing words.

In addition, specific examples of the arithmetic expressions stored in the arithmetic expression storage unit include the following expressions.

In addition, the said arithmetic formula has shown the following arithmetic processes. Specifically, as shown in FIG. 3, first, an identification number n (1 to k) is assigned to a word extracted from the analysis target sentence α held in a flash memory or the like, and the weight value of the word is set to P _n . . Similarly, an identification number m (1 to 1) is assigned to a word extracted from the sample sentence β to be compared, and the weight value of the word is set to C _m . Then, whether or not the keyword n (1 to k) of the sentence α and the keyword m (1 to l) of the sentence β coincide (synonymous with each other) is determined brute-force.

As a result of the determination process, when the values match, the product of the weight values is calculated as f (n, m) = (P _n × (C _m × U), and when they do not match, f (n, m ) = 0 In this way, the product of the weighted values of the words whose keywords match between the two sentences is obtained, and the products of the weighted values are integrated.

  The “similarity calculation unit” (0206) has a function of calculating the similarity between the analysis target sentence and the sample sentence by calculating according to the analysis target weighting result, the sample weighting result, and the arithmetic expression. It can be realized by a calculation device or a similarity calculation program. Specifically, the weighting value for each word is stored in the main memory from the arithmetic expression stored in the flash memory as described above and the data table as shown in FIG. Then, the CPU executes the arithmetic processing of the arithmetic expression using the weight value as a variable, and calculates the arithmetic result. Then, for example, the CPU determines whether the calculation result value is larger than a predetermined value by a logical calculation process of the CPU. If the calculation result value is larger than the predetermined value, it is determined that the similarity between sentences is high.

  Alternatively, the determination of the similarity is not limited to an absolute similarity determination using a predetermined value as a determination criterion. For example, each calculation is performed between a plurality of analysis target sentences for one sample document. A result value may be calculated, and a relative similarity determination may be performed in which the analysis target sentence having the maximum value is determined to be most similar to the sample sentence.

  FIG. 5 is a conceptual diagram showing an example of calculation processing for similarity determination in the similarity determination unit. As shown in this figure, the word “cigarette” appearing in the analysis target sentence (A001) and the sample sentence (S001) is referred to, for example, table data as shown in FIG. 64 "is acquired. Then, the weighted result product “4608” is calculated by multiplying the values according to the arithmetic expression. In addition, the weighted result products “0”, “0”,... Are calculated for other appearance words “yen”, “nicotine”,. Is calculated.

  Subsequently, also for another analysis target sentence (A004), the total value β “7829” is calculated using the weighted value and the arithmetic expression for each word. Then, each of the total values is compared with a predetermined value “6000” which is a criterion for similarity determination. As a result, it is determined that the analysis target sentence A001 is not similar because the total value α is equal to or less than the predetermined value, that is, is not a harmful site. On the other hand, the analysis target sentence A004 is similar because the total value β is equal to or greater than a predetermined value, that is, it is determined that there is a high possibility of being a harmful site.

  The “similar sentence accumulating unit” (0207) accumulates the identification information of the analysis target sentence as a sentence similar to the sample sentence when the calculation result in the similarity calculation unit satisfies the predetermined condition. It has a function, and can be realized by, for example, the above-described various storage devices or similar sentence storage programs.

  For the website related to the analysis target sentence that is determined to have a high similarity to the sample sentence, that is, the possibility of being a harmful website, based on the total weighted result product with the sample sentence as described above. Identification information such as the URL is automatically accumulated. Therefore, the database update for the filtering process can be automatically updated at an effective level.

  The filtering device of the present embodiment having the above-described configuration is incorporated in a search server device on the network and responds to the determination result according to a search request from a client terminal, as will be described later in a second embodiment. You may comprise so that a filtering process may be performed. Alternatively, a configuration may be adopted in which the above-described determination processing is performed on a Web page that is incorporated in a client terminal device and is to be accessed by a browser or the like, and filtering processing is performed according to the determination result.

    <Hardware configuration>

  FIG. 6 is a schematic diagram illustrating an example of the configuration of the filtering device when the functional components described above are realized as hardware. The operation of each hardware component in the sentence similarity determination process will be described with reference to FIG.

  As shown in this figure, the filtering device is a morpheme analysis processing unit, an analysis target weighted result acquisition unit, a sample weighted result acquisition unit, and a similarity calculation unit, and a “CPU” ( 0601) and “main memory” (0602). In addition, a “flash memory” (0603) that is an arithmetic expression storage unit, an “I / O” (0604) that is a crawler unit, an “HDD” (0605) that is a similar sentence storage unit, and the like are also provided. They are connected to each other via a data communication path such as a “system bus” to transmit / receive information and process information.

  A program is read into the “main memory”, and the “CPU” executes various arithmetic processes by referring to and interpreting the read program. In addition, a plurality of addresses are assigned to each of the “main memory” and “flash memory”, and in the calculation processing of the “CPU”, the addresses are specified and accessed to store the data. It is possible to perform arithmetic processing using.

  Here, the “CPU” interprets the crawler program read to the “main memory”, and the filtering device of the present embodiment refers to the URL list (not shown) from “I / O” according to the program. Get a known website. Then, the document data of the unknown website is acquired by following the link destinations of the website one after another. Then, the sentence data acquired in this way is stored at address 1 of the “main memory” as an analysis target sentence.

  Subsequently, the “CPU” interprets the morphological analysis processing program, and executes processing for extracting words from the analysis target sentence stored in the address 1 of the “main memory” according to the interpretation result. Specifically, for example, the “CPU” refers to the word dictionary stored at the address B of the “flash memory”, and uses the longest match method to search for “words in the dictionary” and “words in the sentence”. Perform pattern matching processing. For the extracted word, the “CPU” refers to the grammar dictionary in the “flash memory” and executes a process of determining whether the word is correctly extracted from the word part-of-speech used in the grammar dictionary and the connection relationship. As for the extracted word determined to be incorrect as a result of the determination, another delimiter is found by the re-pattern matching process, and the word in the sentence is extracted in a grammatically correct form. Here, the “CPU” may refer to a synonym dictionary stored in the “flash memory” in accordance with the extraction program and perform a process of collecting synonyms out of the extracted words into one word. Then, the extracted words are stored as keywords at addresses 2,... Of “main memory”.

  Subsequently, the “CPU” interprets the analysis target weighted result acquisition program, and executes the weighting process for the word stored at the addresses 2... Of the “main memory” according to the interpretation result. Specifically, for example, the number of appearances of all words extracted from one sentence is counted. Subsequently, the number of occurrences of the word (keyword 1) to be weighted is counted, and the appearance ratio (%) in the sentence of the keyword 1 is calculated by the calculation process of “CPU”. The calculated value is stored as a weighted value for keyword 1 at address 3 of “main memory”. And the same calculation process is performed also about the other word (keyword 2, keyword 3, ...) which appears in a sentence, and the weight value for every word is calculated. The weight values calculated here are preferably stored in a flash memory or the like as a data table in association with the sentence ID as shown in FIG.

  Subsequently, the “CPU” acquires a sample weighted result relating to a sample sentence registered in advance as described above according to the sample weighted result acquisition program. As described above, this sample weighted result may be programmed so that only the weighted result is held in the flash memory or the like as table data as shown in FIG. 3B. . Alternatively, each time the similarity comparison determination process is performed, a sample sentence may be acquired from “I / O” with reference to a sample URL list or the like, and processing such as morphological analysis or calculation of a weighted result may be performed. .

  Subsequently, the “CPU” interprets the similarity calculation program and, according to the interpretation result, flashes the analysis target weighted result and the sample weighted result stored in the “flash memory” and “main memory” as described above. A process of substituting into the arithmetic expression stored at the address D of the memory and performing the calculation is executed. Specifically, for example, as shown in FIG. 4, a product of weight values for each appearance word is calculated by a calculation process of “CPU”, and a value obtained by adding the products is calculated. Then, the total value of the calculated weighted result products is stored in the address 4 of the “main memory” as a comparison index value indicating the content similarity between the analysis target sentence and the sample sentence.

  Then, for example, a magnitude comparison process between the comparison index value and a predetermined value (not shown) stored in the flash memory is executed by a calculation process of the “CPU”. As a result, if the index value is lower than the predetermined value, it is determined that the analysis target sentence has a low similarity with the sample sentence, that is, the Web site related to the analysis target sentence is not a harmful site.

  On the other hand, if the index value is less than the predetermined value as a result of the comparison processing between the comparison index value and the predetermined value, the analysis target sentence has a high similarity to the sample sentence, that is, the website related to the analysis target sentence is harmful. Judge that the site is likely to be. Then, according to the similar sentence recording program, the “CPU” records the identification information (URL, etc.) of the website determined to be similar to the sample sentence in the HDD, and performs filtering processing on the recorded identification information. It is used as a list for use.

  Further, the website itself determined to be similar to the sample text may be recorded in an HDD or the like and used as a sample text for the next time or later.

    <Process flow>

  FIG. 7 is a flowchart illustrating an example of a processing flow in the filtering device of the present embodiment. The steps shown below may be steps executed by each hardware configuration of the computer as described above, or may be processing steps that constitute a program for controlling the computer recorded on a medium. I do not care.

  As shown in this figure, first, an analysis target sentence is acquired using a crawler (step S0701), and the acquired analysis target sentence is subjected to morphological analysis (step S0702). Then, an analysis target weighting result obtained based on the keyword obtained by the morphological analysis is acquired (step S0703). Subsequently, a sample weighting result obtained based on the keyword obtained by the morphological analysis of the sample sentence is acquired (step S0704).

  Then, a predetermined arithmetic expression as shown in FIG. 4 using the analysis target weighted result and the sample weighted result as substitution values is acquired from the storage device (step S0705), and the analysis target weighted result and the sample weighted result are used as the arithmetic expressions. Input and execute the calculation (step S0706). Then, the degree of similarity between the analysis target sentence and the sample sentence is determined based on whether the calculation result satisfies a predetermined condition (for example, a magnitude condition between the calculation result value and a predetermined value that is a criterion for determining the similarity) (step S0707). . When the calculation result satisfies a predetermined condition, that is, when it is determined that the analysis target sentence is similar to the sample sentence, identification information of the analysis target sentence is recorded in the similar sentence storage unit (step S0708).

  Then, the subsequent filtering process can be executed using the recorded identification information. Further, in step S0708, a process of accumulating the analysis target sentence itself determined to be similar may be executed. In this case, the accumulated analysis target sentences are used as sample sentences for the next and subsequent times.

    <Brief description of effect>

  As described above, the filtering device of this embodiment can determine the similarity of sentences according to the words extracted by the morphological analysis process. Then, a list for filtering based on the similarity of the text content itself can be automatically generated according to the determination result.

  << Example 2 >>

    <Overview>

  FIG. 8 is a conceptual diagram illustrating an example of filtering processing in the filtering device of this embodiment. As shown in FIG. 8 (a), for example, in response to a search request “tobacco” from a client terminal device, the search server device performs “cigarette purchase IC card” site, “my smoking cessation method” site, “cigarette” In addition to the “hazardous” site, the “overseas tobacco import” site is also included in the search results and returned to the client terminal device, as indicated by α in the figure.

  On the other hand, in the search server device in which the filtering device of this embodiment is incorporated, as shown in FIG. 8B, an “overseas tobacco agency import” site determined to be a harmful site is converted into a search result by filtering processing. It is processed so that it is not reflected.

  As described above, the filtering device of this embodiment is incorporated in a search system such as a search server device, and executes a filtering process using the data automatically stored in the first embodiment on the search result. It is the filtering device characterized by doing.

    <Functional configuration>

  FIG. 9 is a diagram illustrating an example of functional blocks in the filtering device of the present embodiment. As shown in this figure, the “filtering device” (0900) of the present embodiment is based on the first embodiment, and “crawler unit” (0901), “morpheme analysis processing unit” (0902), and “analysis” “Target weighted result acquisition unit” (0903), “Sample weighted result acquisition unit” (0904), “Calculation expression storage unit” (0905), “Similarity calculation unit” (0906), “Similar sentence storage unit” (0907). Since these structural requirements have already been described in the first embodiment, description thereof will be omitted.

  And the feature point of the filtering apparatus of a present Example is a point which further has a "search text identification information acquisition part" (0908) and a "comparison part" (0909).

  The “search text identification information acquisition unit” (0908) has a function of acquiring search text identification information indicating a search result, and can be realized by, for example, “I / O” or a search text identification information acquisition program. Specifically, for example, a search process corresponding to the search query is executed in a search server or the like. And the sentence containing a search query is specified by the search process, and the identification information is acquired as search sentence identification information. In addition, such acquisition of search sentence identification information is good to set it as the structure acquired from a search server via a network, for example, when a filtering apparatus is integrated in the apparatus separate from a search server. In addition, if the filtering device is incorporated in the search server, it may be configured to obtain it internally from the device itself.

  The “comparison unit” (0909) has a function of comparing the search text identification information acquired by the search text identification information acquisition unit with the identification information stored in the similar text storage unit. Or a main memory, a comparison program, or the like. Specifically, the comparison unit refers to a URL (identification information) list of harmful Web sites stored in a similar text storage unit as shown in FIG. 10, for example. Then, by a logical operation process of the CPU, it is determined whether or not a URL that matches the URL in the search result indicated by the search sentence identification information is in the URL (identification information) list of the harmful website. .

  When a comparison result indicating that there is matching identification information is output, for example, filtering processing is performed by a filtering processing unit (not shown) so that the Web site indicated by the URL is not displayed in the search result. It is.

    <Process flow>

  FIG. 11 is a flowchart illustrating an example of a processing flow in the filtering device of the present embodiment. The steps shown below may be steps executed by each hardware configuration of the computer as described above, or may be processing steps that constitute a program for controlling the computer recorded on a medium. I do not care.

  Prior to the processing shown in this figure, the processing described in the first embodiment is executed, the identification information of similar sentences is stored in the storage section, and a list of URLs of harmful Web sites and the like is generated. Then, as shown in this figure, first, a search process for a sentence is executed in response to an input of a search query (step S1101), and search sentence identification information indicating the search result is acquired (step S1102). Then, the search text identification information is compared with the identification information stored in the similar text storage unit (step S1103), and if there is matching identification information, a filtering process such as not including it in the search result is executed.

    <Brief description of effect>

  As described above, the filtering device according to the present embodiment can perform effective filtering on search results when performing a search using a search server or a terminal.

  Example 3

    <Overview>

  FIG. 12 is a diagram illustrating an example of a filtering screen by the filtering device of the present embodiment. As shown in FIG. 12A, a terminal device on the network is about to access a Web site via a browser application. Here, for example, the filtering device of the present embodiment is incorporated in the terminal device, and the processing for determining the similarity between the access destination website and the sample text is executed in real time.

  If it is determined that they are similar to each other, as shown in FIG. 12 (b), a message such as “Browsing this website is prohibited” is displayed, and the website is displayed. It is not.

    <Functional configuration>

  FIG. 13 is a diagram illustrating an example of functional blocks in the filtering device of the present embodiment. As shown in this figure, the “filtering device” (1300) of the present embodiment is based on the first embodiment, and includes a “crawler section” (1301), a “morpheme analysis processing section” (1302), and an “analysis”. “Target weighted result acquisition unit” (1303), “Sample weighted result acquisition unit” (1304), “Calculation expression storage unit” (1305), “Similarity calculation unit” (1306), “Similar sentence storage unit” (1307). Further, on the basis of the second embodiment, a “search text identification information acquisition unit” or a “comparison unit” (not shown) may be included. Since these configuration requirements have already been described in the first and second embodiments, description thereof will be omitted.

  And the feature point of the filtering apparatus of a present Example has further a "URL receiving part" (1308), an "analysis object sentence acquisition part" (1309), and an "identification information output part" (1310). It is.

  The “URL receiving unit” (1308) has a function of receiving a URL, and can be realized by, for example, “I / O” or a URL receiving program. Specifically, for example, a plug-in (toolbar or the like) incorporated in the browser application of the client terminal device acquires the URL input to the browser and outputs / transmits the URL to the filtering device. Is received. The reception of the URL is an example of reception when the filtering device is incorporated in a device separate from the client terminal device, for example. On the other hand, if the filtering device is incorporated in the client terminal device, the URL may be received internally from the device itself.

  The “analysis target sentence acquisition unit” (1309) has a function of acquiring the text of the web page identified by the received URL as the analysis target sentence, and is realized by “I / O” or an analysis target sentence acquisition program, for example. be able to. Specifically, a normal HTTP request destined for the received URL is transmitted, and the text of the Web page is acquired by the response. In addition, it is preferable that the acquired Web page is not displayed on the display until the similarity determination process with the sample sentence is completed because of the nature of the filtering process.

  The “identification information output unit” (1310) has a function of outputting the fact to the URL transmission source when the calculation result in the similarity calculation unit satisfies the predetermined condition, for example, It can be realized by an arithmetic unit such as a CPU, an I / O, an identification information output program, or the like. The “predetermined condition” is a condition that the above-described index value indicating the degree of similarity is equal to or less than a threshold value, that is, a condition that both sentences (the sample sentence and the Web page sentence indicated by the received URL) are not similar. Also good. If the condition is satisfied, the Web page indicated by the received URL is returned to the transmission source as it is, for example, as a healthy site. Further, when the calculation result in the similarity calculation unit does not satisfy the condition, the Web page indicated by the URL is assumed to be a harmful site, for example, and the URL from the transmission source is not returned to the transmission source. For example, to prevent access.

  On the other hand, if the condition is that the index value indicating the degree of similarity exceeds a threshold value, that is, the condition that both sentences are similar, the calculation result is returned to the transmission source. Then, it is preferable to perform access permission / blocking processing to the Web site indicated by the URL according to the calculation result by processing at the client terminal device that is the transmission source. Further, a pop-up display of a block warning may be performed before the access block process.

  The information indicating “to that effect” output from the identification information output unit to the URL transmission source is information indicating that the calculation result in the similarity calculation unit satisfies a predetermined condition. Any information may be used. For example, if the program is set so that the reply of the identification information itself of the analysis target sentence such as the URL is processed as information indicating “to that effect”, the identification information output unit displays the identification information of the analysis target sentence. It may be output to the URL sender.

  In addition, the function of the identification information output unit is configured such that when the filtering device of the present embodiment is incorporated in the client terminal device, it outputs to the browser of the client device in which it is incorporated via the system bus or the like. Good.

  In addition, when the filtering device of this embodiment is incorporated in a server device on a network, it may be configured to output via the network network with the destination IP address of the URL indicated by the HTTP request or the like as the destination.

  When the client terminal device is used as the transmission source of the URL and access permission / blocking is executed by the client terminal device, the client terminal device may have the following configuration. That is, the client terminal device includes a URL transmission unit that transmits a URL to the filtering device, and a predetermined information reception unit that receives information indicating that the calculation result satisfies a predetermined condition in the identification information output unit of the filtering device. And an access control unit that controls access to the URL according to the received information indicating the calculation result.

  In this way, using the filtering device of this embodiment, it is possible to execute filtering processing in real time for a Web site to be accessed by a client terminal device.

    <Process flow>

  FIG. 14 is a flowchart illustrating an example of a process flow in the filtering device of the present embodiment. The steps shown below may be steps executed by each hardware configuration of the computer as described above, or may be processing steps that constitute a program for controlling the computer recorded on a medium. I do not care.

  As shown in this figure, first, for example, when a URL included in an HTTP request or the like is received (step S1401), the text of the Web page identified by the received URL is acquired as an analysis target text (step S1402). Then, the analysis target sentence is subjected to morphological analysis (step S1403). Subsequently, the processing described in the first embodiment with reference to FIG. 7 is executed, and as a result, the similarity between the analysis target sentence and the sample sentence is determined based on the calculation result (step S1404). If the determination result satisfies a predetermined condition, that is, if it is determined that the analysis target sentence is similar to the sample sentence, the determination result is output to the URL transmission source (step S1405Y). Further, when the determination result does not satisfy the predetermined condition, that is, when it is determined that the analysis target sentence is not similar to the sample sentence, the determination result that the similarity is not similar is sent to the URL transmission source. Output (step S1405N).

  Then, for example, a client terminal device that is a transmission source of the URL that receives the output determination result executes processing as shown in FIG. That is, a URL is transmitted in step S1501, and information indicating that the calculation result satisfying a predetermined condition regarding the URL is returned from the filtering device by the processing described in FIG. 14 is received (step S1502). Then, on the client terminal device side (transmission source), for example, when the received information is information indicating that the calculation results are similar, access to the Web site indicated by the URL is blocked. On the other hand, when the calculation result is information indicating that the results are not similar, the client terminal device is allowed to access the Web site indicated by the URL and execute the display of the Web site. A process for controlling access to the URL is executed in accordance with the calculation result received (step S1503).

    <Brief description of effect>

  As described above, the filtering apparatus according to the present embodiment can execute the filtering process in real time for the Web site to be accessed by the client terminal apparatus.

Conceptual diagram for describing an example of filtering target Web site determination processing by the filtering device according to the first embodiment. The figure showing an example of the functional block in the filtering apparatus of Example 1. The figure showing an example of the data table of the analysis object weighting result and sample weighting result which are accumulate | stored in the filtering apparatus of Example 1. FIG. The figure showing an example of the comparison index value table in the filtering apparatus of Example 1. FIG. 5 is a conceptual diagram illustrating an example of calculation processing for similarity determination in the similarity determination unit of the filtering device according to the first embodiment. 1 is a diagram illustrating an example of a hardware configuration in a filtering device according to a first embodiment. The flowchart showing an example of the flow of a process in the filtering apparatus of Example 1. Conceptual diagram illustrating an example of filtering processing in the filtering device according to the second embodiment. The figure showing an example of the functional block in the filtering apparatus of Example 2. The figure showing an example of the identification information list | wrist of the harmful website accumulate | stored in the similar text storage part of the filtering apparatus of Example 2. FIG. The flowchart showing an example of the flow of the process in the filtering apparatus of Example 2. The figure showing an example of the filtering screen by the filtering apparatus of Example 2. The figure showing an example of the functional block in the filtering apparatus of Example 3. A flowchart showing an example of the flow of processing in a filtering device of Example 3. 10 is a flowchart illustrating an example of a process flow in a client terminal device according to the third embodiment.

Explanation of symbols

0200 Filtering device 0201 Crawler unit 0202 Morphological analysis processing unit 0203 Analysis target weighted result acquisition unit 0204 Sample weighted result acquisition unit 0205 Calculation expression storage unit 0206 Similarity determination unit 0207 Similar sentence accumulation unit

Claims (12)

A crawler unit for obtaining the analysis target sentence;
A morphological analysis processing unit that performs a morphological analysis process on the analysis target sentence obtained in the crawler unit;
An analysis target weighted result acquisition unit for acquiring an analysis target weighted result that is a result of weighting a keyword obtained by morphological analysis processing based on an appearance frequency in an analysis target sentence;
A sample weighted result acquisition unit for acquiring a sample weighted result that is a result of weighting a keyword obtained by morphological analysis of a sample sentence based on an appearance frequency in the sample sentence;
An arithmetic expression storage unit for storing a predetermined arithmetic expression using the analysis target weighted result and the sample weighted result as substitution values;
A similarity calculation unit that calculates the similarity between the analysis target sentence and the sample sentence by calculating according to the analysis target weighted result, the sample weighted result, and the calculation formula;
When the calculation result in the similarity calculation unit is a calculation result that satisfies a predetermined condition, a similar sentence storage unit that stores the identification information of the analysis target sentence as a sentence similar to the sample sentence;
A filtering device. A search sentence identification information acquisition unit for acquiring search sentence identification information indicating a search result;
The filtering device according to claim 1, further comprising: a comparison unit that compares the search text identification information acquired by the search text identification information acquisition unit with the identification information stored in the similar text storage unit.   When the comparison result in the comparison unit is a comparison result that the identification information stored in the similar text storage unit matches the search text identification information, it is identified by the search text identification information. The filtering device according to claim 2, further comprising a filtering processing unit that executes a filtering process so that the search text is not included in the search result. A URL receiver for receiving a URL;
An analysis target sentence acquisition unit that acquires a sentence of a web page identified by the received URL as an analysis target sentence;
When the calculation result in the similarity calculation unit is a calculation result that satisfies a predetermined condition, an identification information output unit that outputs the fact to the URL transmission source;
The filtering device according to claim 1, comprising: The filtering apparatus according to any one of claims 1 to 4, wherein the expression stored in the arithmetic expression storage unit is an expression of the following formula (1).

A filtering system comprising: the filtering device according to claim 4 dependent on claim 4 or claim 4; and a client terminal device,
The client terminal device
A URL transmission unit that transmits the URL to the filtering device;
A predetermined information receiving unit that receives information indicating that the calculation result satisfies a predetermined condition in the identification information output unit of the filtering device;
A filtering system comprising: an access control unit that controls access to the URL according to information indicating a received calculation result. A crawler step to obtain the sentence to be analyzed;
A morphological analysis processing step for performing a morphological analysis process on the analysis target sentence obtained in the crawler step;
An analysis target weighted result acquisition step for acquiring an analysis target weighted result that is a result of weighting the keyword obtained by the morphological analysis processing based on the appearance frequency in the analysis target sentence;
A sample weighted result acquisition step of acquiring a sample weighted result that is a result of weighting a keyword obtained by morphological analysis of a sample sentence based on an appearance frequency in the sample sentence;
An arithmetic expression acquisition step for acquiring a predetermined arithmetic expression using the analysis target weighted result and the sample weighted result as substitution values from the arithmetic expression storage unit;
A similarity calculation step for calculating the similarity between the analysis target sentence and the sample sentence by calculating according to the analysis target weighted result, the sample weighted result, and the calculation formula;
Similar sentence recording for recording the identification information of the analysis target sentence as a sentence similar to the sample sentence in the similar sentence storage unit when the calculation result in the similarity calculation step satisfies the predetermined condition Steps,
Filtering method that causes a computer to execute. A search sentence identification information acquisition step for acquiring search sentence identification information indicating a search result;
The filtering method according to claim 7, further causing the computer to execute a comparison step of comparing the search text identification information acquired in the search text identification information acquisition step with the identification information stored in the similar text storage unit.   If the comparison result in the comparison step is a comparison result indicating that there is a match with the search text identification information in the identification information stored in the similar text storage unit, the search text identification information is identified. The filtering method according to claim 8, further causing a computer to execute a filtering process step of executing a filtering process so that a search sentence is not included in a search result. A URL receiving step for receiving a URL;
An analysis target sentence acquisition step of acquiring the sentence of the web page identified by the received URL as the analysis target sentence;
If the calculation result in the similarity calculation step is a calculation result that satisfies a predetermined condition, an identification information output step that outputs the fact to the URL transmission source;
The filtering method according to any one of claims 7 to 9, which causes a computer to execute. The filtering method according to any one of claims 7 to 10, wherein the expression acquired in the operation expression acquiring step is an expression of the following equation (1).

further,
A URL transmission step for transmitting a URL corresponding to the URL reception step;
A predetermined information receiving step for receiving information indicating that the calculation result satisfies a predetermined condition in the identification information output step;
An access control step for controlling access to the URL according to information indicating the received calculation result;
The filtering method according to claim 10, which is dependent on claim 10 or 11.

Images