JP2013101415A - Commodity web page analyzer, commodity web page analysis method, and program for commodity web page analyzer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To calculate similarity of commodities between commodity web pages to improve convenience of users.SOLUTION: A commodity web page analysis method includes: acquiring a first commodity web page and a second commodity web page (S1); extracting text data described on each commodity web page (S2); subjecting the text data to language analysis and extracting feature word candidates of each commodity web page (S3); creating a retrieval query using the feature word candidates as retrieval keywords and acquiring a retrieval result based on the retrieval query (S4); determining whether or not a category related word exists in snippets of the retrieval result; determining the feature word candidates as feature words when the category related word exists in the snippets (S6); calculating similarity between the first and second commodity web pages on the basis of the feature words (S7); and determining that the first and second commodity web pages handle similar commodities when the similarity is equal to or more than a prescribed value (S8).

Description

  The present invention relates to a technical field of a product web page analysis device, a product web page analysis method, and a program for a product web page analysis device for analyzing a web page related to a product on a shopping site on the Internet.

  There are many product web pages dealing with products on shopping sites on the Internet, and users browse the product web pages and purchase products. When browsing and purchasing, a user usually performs a product search. In this search technology, in order to accurately search for a product desired by a user from many product web pages, search accuracy is improved. For example, in Patent Document 1, a plurality of stores are provided for each of various product categories in order to improve the accuracy of product search by accurately identifying the seed or noise of product description information registered in a product. A category frequency calculation unit that calculates, for each product description information, the number of product categories to which product description information, which is information registered for each product provided, belongs, and a store frequency that includes the number of stores to which the product description information belongs. The store frequency calculation unit calculated for each explanation information, the machine learning unit that generates classification rule information from the ratio of the calculated category frequency to the store frequency, and the product description information is noise information based on the generated classification rule information Alternatively, a product information classification device including a determination unit that identifies and classifies product identification information is disclosed.

JP 2009-129087 A

  By the way, in a shopping site where various stores such as Cyber Mall participate, if each merchant (store owner) opening a shopping site independently creates each product web page, the same product May be different, or the notation method may not be unified. Further, in order to appeal to the user rather than other stores, information unique to the store is given to the product web page, or the heading portion of the product web page is filled with various information.

  For this reason, it is difficult to find a desired product web page by simply searching, and the user has to browse various product web pages that are considered to be the same product and determine a site to purchase. It is difficult to solve the above problem by simply increasing the search accuracy as in Patent Document 1.

  The present invention has been made in view of such a problem, and an example of the problem is a product web page analysis apparatus that can improve the convenience of the user by obtaining the similarity of the products on the product web page. An object of the present invention is to provide a product web page analysis method and a product web page analysis apparatus program.

  In order to solve the above-mentioned problem, the invention described in claim 1 includes category-related word storage means for storing category-related words related to a category for classifying products, a first product web page, and a second product web. Product web page acquisition means for acquiring a page, text data extraction means for extracting text data described in each product web page, and linguistic analysis of the extracted text data, Feature word candidate extraction means for extracting feature word candidates, search result acquisition means for creating a search query using the feature word candidates as search keywords, and acquiring search results based on the search queries, and the category related word storage means A search result determination means for determining whether or not the category related word is present in the snippet of the search result, Feature word determination means for determining the feature word candidate when the category-related word is present in the snippet as the feature word of each product web page, and based on the feature word, the first product web page and the second product word Similarity calculation means for calculating the similarity between the product web page and the first product web page and the second product web page handle similar products when the similarity is a predetermined value or more. Similar product determination means for determining that the product is a product web page.

  According to a second aspect of the present invention, in the product web page analyzing apparatus according to the first aspect, the similarity calculation unit uses the common feature word in the first product web page and the second product web page. Based on the above, the similarity is calculated.

  According to a third aspect of the present invention, in the product web page analyzing apparatus according to the second aspect of the invention, the similarity calculation unit is configured to obtain the feature word common to the first product web page and the second product web page. The similarity is calculated based on the reverse appearance frequency.

  The invention according to claim 4 is the product web page analysis apparatus according to any one of claims 1 to 3, further comprising domain name storage means for storing a domain name of a shopping site, and the search result The determining means refers to the category related word storage means and the domain name storage means to determine whether or not the category related words are present in a snippet of a shopping site in the search result. .

  According to a fifth aspect of the present invention, in the product web page analyzing apparatus according to any one of the first to fourth aspects, the feature word candidate extracting unit analyzes the language of the text data by morphological analysis. It has a noise removal means for removing a noise morpheme from the obtained morpheme.

  The invention according to claim 6 is a category related word storing step for storing a category related word related to a category for classifying a product, and a product web page acquisition for acquiring a first product web page and a second product web page. Means, a text data extraction step for extracting text data described in each product web page, and a linguistic analysis of the extracted text data to extract feature word candidates for each product web page A candidate extraction step, a search result acquisition step of creating a search query using the feature word candidate as a search keyword, and acquiring a search result based on the search query, and the category related word exists in the snippet of the search result A search result determination step for determining whether or not the category-related word exists in the snippet A feature word determination step of determining a combination of the feature word candidates as a feature word of each product web page, and a similarity between the first product web page and the second product web page based on the feature word Similarity product to be calculated, and when the similarity is equal to or greater than a predetermined value, the first product webpage and the second product webpage are determined to be product webpages that handle similar products. And a determining step.

  The invention according to claim 7 is a product web for acquiring category related word storage means, a first product web page, and a second product web page for storing category related words related to a category for classifying products. Page acquisition means, text data extraction means for extracting text data described in each product web page, feature words for extracting feature word candidates of each product web page by performing language analysis on the extracted text data A candidate extraction unit, a search query that uses the feature word candidate as a search keyword, a search result acquisition unit that acquires a search result based on the search query, and a snippet of the search result by referring to the category-related word storage unit Search result judging means for judging whether or not the category-related word exists in the snippet; Feature word determining means for determining the feature word candidate when a word related word exists as a feature word of each product web page, and based on the feature word, the first product web page and the second product web page A similarity calculation means for calculating the similarity between the first product web page and the second product web page when the similarity is equal to or greater than a predetermined value. It is made to function as a similar goods determination means determined to be present.

  According to the present invention, category related words related to a category for classifying products are stored, the first product web page and the second product web page are acquired, and the text data described in each product web page Linguistic analysis to extract feature word candidates for each product web page, create a search query using the feature word candidates as search keywords, obtain search results based on the search query, and use category-related terms in the search result snippet Is determined, a feature word candidate when a category-related word is present in the snippet is determined as a feature word of each product web page, and based on the feature word, the first product web page and the second product word are determined. If the degree of similarity between the product web page is calculated and the similarity is greater than or equal to a predetermined value, the first product web page and the second product web page are products that handle similar products. By determining that the web page is a product page, it is possible to obtain the product similarity of the product web page and, for example, to collect product web pages that handle the same or similar products, thereby improving user convenience. Can do.

It is a mimetic diagram showing an example of outline composition of a goods web page analysis system concerning one embodiment of the present invention. It is a block diagram which shows an example of schematic structure of the goods web page analysis server of FIG. It is a block diagram which shows an example of schematic structure of the shopping server of FIG. It is a block diagram which shows an example of schematic structure of the search server of FIG. It is a flowchart which shows the operation example which analyzes a goods web page in the goods web page analysis server of FIG. It is explanatory drawing which shows an example of the goods web page of the shopping server of FIG. It is explanatory drawing which shows an example of the source code of the goods web page of FIG. It is a schematic diagram which shows an example of the analysis process of the text data of the goods web page of FIG. It is a schematic diagram which shows an example of the search result of the search server of FIG. It is a schematic diagram which shows an example of the domain name of the shopping site memorize | stored in the domain name database of FIG. It is a schematic diagram which shows an example of the category related word memorize | stored in the category related word database of FIG. It is a flowchart which shows an example of the subroutine of similarity calculation in the goods web page analysis server of FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, embodiment described below is embodiment at the time of applying this invention with respect to a goods web page analysis system.

[1. Product Web Page Analysis System Configuration and Function Overview]

  First, the structure and outline | summary function of the goods web page analysis system which concern on one Embodiment of this invention are demonstrated using FIG.

  FIG. 1 is a schematic diagram illustrating a schematic configuration example of a product web page analysis system 1 according to the present embodiment.

  As shown in FIG. 1, a product web page analysis system 1 includes a product web page analysis server (an example of a product web page analysis device) 10 for analyzing a product web page, and a shopping server installed to operate a shopping site. 20, a search server 30 that provides a web search function for a web page on the Internet, a store owner terminal 40 of a store owner opening a store on a shopping site, and a user terminal 45 of a user who purchases a product on the shopping site And.

  The product web page analysis server 10, the shopping server 20, and the search server 30 are connected by a local area network and the like, and can transmit and receive data to each other, and constitute a server system 5. The server system 5, the store owner terminal 40, and the user terminal 45 are connected by the network 3, and can exchange data with each other using, for example, TCP / IP as a communication protocol. The network 3 is constructed by, for example, the Internet, a dedicated communication line (for example, CATV (Community Antenna Television) line), a mobile communication network (including a base station), a gateway, and the like.

  The product web page analysis server 10 analyzes the product web page registered in the shopping server 20, organizes the product web page on the shopping server 20 based on the analysis result, or stores the product web page registered The analysis result is notified to the store owner terminal 40, the user terminal 45, and the like.

  The shopping server 20 functions as a web server, a database server, or the like for selling products on a shopping site, and performs various processes such as acceptance of product web page registration, user registration, and purchase procedures for products. Moreover, the shopping server 20 has the goods web page classified for every category of goods.

  The search server 30 functions as a search engine on the Internet or the like, and provides a search API (Application Programming Interface) function to the product web page analysis server 10 or based on a received search query including a search keyword from a terminal. Search a search target such as a web page and return search results. Further, the search server 30 collects web page information from the shopping server 20 or a website on the Internet (not shown) in order to construct a search database, and analyzes the web page information in advance to perform a search. Create snippet of index information and search results. Here, the snippet is a text such as an explanatory text of each search result, such as a summary of a web page hit by the search or a text including a search keyword described in the web page.

  The store owner terminal 40 used by the store owner is a portable terminal such as a personal computer, a portable wireless telephone, or a PDA (Personal Digital Assistant). The store owner uses the store owner terminal 40 to register or update the product web page in the shopping server 20.

  The user terminal 45 used by the user is a portable terminal such as a personal computer, a portable wireless telephone, or a PDA. The user uses the user terminal 45 to search for or purchase a product.

[2. Configuration and function of each server]
(2.1 Configuration and Function of Product Web Page Analysis Server 10)
Next, the configuration and function of the product web page analysis server 10 will be described with reference to FIG.

  FIG. 2 is a block diagram illustrating an example of a schematic configuration of the product web page analysis server 10.

  As shown in FIG. 2, the product web page analysis server 10 that functions as a computer includes a communication unit 11, a storage unit 12, an input / output interface unit 13, and a system control unit 14. The system control unit 14 and the input / output interface unit 13 are connected via a system bus 15.

  The communication unit 11 is connected to the network 3 to control the communication state with the user terminal 45 or the like, or connected to the local area network to transmit / receive data to / from other servers such as the shopping server 20 and the search server 30. Or

  The storage unit 12 includes, for example, a hard disk drive and stores various programs such as an operating system and a server program, data, and the like. Note that the various programs may be acquired from, for example, another server device via the network 3, or may be recorded on a recording medium and read via a drive device.

  Further, the storage unit 12 includes a category related word database 12a (hereinafter referred to as “category related word DB”) as an example of a category related word storage unit, and a domain name database (hereinafter referred to as “category related word DB”) as an example of a domain name storage unit. Domain name DB ”) 12b, a feature word database (hereinafter referred to as“ feature word DB ”) 12c, and the like.

  In the category related term DB 12a, terms are stored for each category of products on the shopping site. For example, category related words such as “wine” and “alcohol” are included in the category of “wine”, and category related words such as “golf”, “driver” and “club” are included in the category of “golf driver”. Is recorded in advance. The categories have a hierarchical structure. For example, the category of “wine” belongs to the category of “western sake”. These category-related words are stored in association with identifiers corresponding to the respective categories. As described above, the category-related word DB 12a functions as an example of a category-related word storage unit that stores category-related words related to a category for classifying products.

  In the domain name DB 12b, domain names of various shopping sites on the Internet are stored in advance. Thus, the domain name DB 12b functions as an example of a domain name storage unit that stores the domain name of the shopping site.

  In the feature word DB 12c, feature words of the product web page extracted from the product web page are stored as an example of the analysis result by the product web page analysis server 10. The feature word is stored in association with the URL (Uniform Resource Locator) of the product web page.

  Next, the input / output interface unit 13 performs interface processing between the communication unit 11 and the storage unit 12 and the system control unit 14.

  The system control unit 14 includes a CPU (Central Processing Unit) 14a, a ROM (Read Only Memory) 14b, a RAM (Random Access Memory) 14c, and the like. In the system control unit 14, the CPU 14 a reads out and executes various programs stored in the ROM 14 b and the storage unit 12, thereby extracting feature word candidates for each product web page from the text data of the product web page. A feature word is determined from word candidates, and a similarity between product web pages is calculated based on the feature word.

(2.2 Configuration and function of shopping server 20)
Next, the structure and function of the shopping server 20 will be described with reference to the drawings.
FIG. 3 is a block diagram illustrating an example of a schematic configuration of the shopping server 20.

  As shown in FIG. 3, the shopping server 20 includes a communication unit 21, a storage unit 22, an input / output interface unit 23, and a system control unit 24, and the system control unit 24 and the input / output interface unit 23 are Are connected via a system bus 25. Since the configuration and function of the shopping server 20 are substantially the same as the configuration and function of the product web page analysis server 10, the configuration and function of the product web page analysis server 10 will be mainly described.

  The communication unit 21 controls the communication state with the store owner terminal 40, the user terminal 45, the product web page analysis server 10, and the like through the network 3, the local area network, and the like.

  In the storage unit 22, a product database (hereinafter referred to as "product DB") 22a, a member database (hereinafter referred to as "member DB") 22b, and the like are constructed.

  In the product DB 22a, product information such as a product name, type, product image, specification, and summary of product introduction, advertisement information, and the like are stored in association with a product ID that is an identifier for identifying the product. ing. The product DB 22a stores a product web page file described in a markup language such as HTML (HyperText Markup Language) or XML (Extensible Markup Language).

  In the member DB 22b, user information such as user ID, name, address, telephone number, and mail address of a registered user (user of Internet shop) is registered. Such user information can be determined for each user by the user ID. In the member DB 22b, a user ID, a login ID, and a password necessary for a user to log in to the Internet shop site from the user terminal 45 are registered. Here, the login ID and the password are login information used for login processing (user authentication processing).

  The system control unit 24 includes a CPU 24a, a ROM 24b, a RAM 24c, and the like. In the system control unit 24, the CPU 24a reads out and executes various programs stored in the ROM 24b and the storage unit 22, thereby registering or updating the product web page by the store owner, product purchase processing by the user, The purchase history is recorded for each user ID.

(2.3 Configuration and Function of Search Server 30)
Next, the configuration and function of the search server 30 will be described with reference to the drawings.
FIG. 4 is a block diagram illustrating an example of a schematic configuration of the search server 30.

  As shown in FIG. 4, the search server 30 includes a communication unit 31, a storage unit 32, an input / output interface unit 33, and a system control unit 34, and the system control unit 34 and the input / output interface unit 33 are Are connected via a system bus 35. Note that the configuration and function of the search server 30 are substantially the same as the configuration and function of the product web page analysis server 10, and therefore, the configuration and function of the product web page analysis server 10 will be mainly described.

  The communication unit 31 controls the communication state with the user terminal 45, the product web page analysis server 10 and the like through the network 3 and the local area network.

  In the storage unit 32, a search database (hereinafter referred to as “search DB”) 32a and the like are constructed.

  The search DB 32a stores index information generated by the indexer, snippets, and the like.

  The system control unit 34 includes a CPU 34a, a ROM 34b, a RAM 34c, and the like. In the system control unit 34, the CPU 34a reads and executes various programs stored in the ROM 34b and the storage unit 32 to collect web page information on the Internet, analyze the web page information in advance, A snippet of information or a search result is created, a search is performed based on a search query from the product web page analysis server 10 or the user terminal 45, and a search result is transmitted.

[3. Operation of product web page analysis system]
Next, operation | movement of the goods web page analysis system 1 which concerns on one Embodiment of this invention is demonstrated based on FIGS.

  FIG. 5 is a flowchart showing an operation example of analyzing a product web page in the product web page analysis server 10. FIG. 6 is an explanatory diagram illustrating an example of a product web page of the shopping server 20. FIG. 7 is an explanatory diagram showing an example of the source code of the product web page. FIG. 8 is a schematic diagram illustrating an example of an analysis process of text data of a product web page. FIG. 9 is a schematic diagram illustrating an example of a search result of the search server 30. FIG. 10 is a schematic diagram illustrating an example of a domain name of a shopping site stored in the domain name DB 12b. FIG. 11 is a schematic diagram illustrating an example of category-related words stored in the category-related word DB 12a. FIG. 12 is a flowchart showing an example of a similarity calculation subroutine in the product web page analysis server 10.

(3.1. Flow of product web page analysis)
First, the flow of analysis of a product web page will be described with reference to FIG.

  As shown in FIG. 5, the product web page analysis server 10 acquires two product web pages (step S1). Specifically, the system control unit 14 of the product web page analysis server 10 relates to a product belonging to the same category (for example, category “wine”) in the product DB 22a, and the first product web page and the second product web page. The product web page is acquired from the product DB 22 a through the communication unit 11. These product web pages are product web pages registered by the store owner or the like through the store owner terminal 40. As described above, the system control unit 14 and the communication unit 11 of the product web page analysis server 10 function as an example of a product web page acquisition unit that acquires the first product web page and the second product web page.

  Next, the product web page analysis server 10 extracts text data from each product web page (step S2). Specifically, the system control unit 14 of the product web page analysis server 10 extracts text data described in each product web page. More specifically, as shown in FIG. 6, text data of text portions 51, 52, 53, 54 is extracted from the product web page 50. Further, as shown in FIG. 7, the data in the source code described in a markup language such as HTML is also used. For example, text data of the title element portion of the HTML tag is also extracted.

  As described above, the system control unit 14 of the product web page analysis server 10 functions as an example of a text data extraction unit that extracts text data described in each product web page.

  Next, the product web page analysis server 10 extracts feature word candidates from the extracted text data by morphological analysis, which is an example of language analysis (step S3). Specifically, the system control unit 14 of the product web page analysis server 10 uses a morpheme analysis program to decompose the extracted text data into morphemes and obtain the part of speech of each phrase. As shown in FIG. 8, for example, a phrase group 53a is obtained. And the system control part 14 of the goods web page analysis server 10 removes noise from the phrase group 53a, and extracts the feature word candidate group 53b. For example, a part enclosed in parentheses or symbols such as “[]”, a verb phrase, a phrase related to a verb phrase, and the like are removed from the phrase group 53a to obtain a feature word candidate group 53b. Here, a part enclosed in parentheses or symbols is often noise, and a verb phrase or a phrase related to a verb phrase is often not a phrase related to a product. Note that a general morphological analysis program may be used as the morphological analysis, and there may be a process of forming a compound word when performing the morphological analysis. Also, feature word candidates are extracted for the other product web page.

  As described above, the system control unit 14 of the product web page analysis server 10 functions as an example of a feature word candidate extraction unit that analyzes the extracted text data and extracts feature word candidates of each product web page. In addition, the system control unit 14 of the product web page analysis server 10 functions as an example of a noise removing unit that removes noise morphemes from morphemes obtained by language analysis of text data by morphological analysis.

  Next, the product web page analysis server 10 performs a web search using each feature word candidate (step S4). Specifically, first, the system control unit 14 of the product web page analysis server 10 creates each search query using each feature word candidate as a search keyword in order to use the function of the search API provided by the search server 30. To do. And the system control part 14 of the goods web page analysis server 10 transmits to the search server 30 through the communication part 11, and receives the search result for every feature word candidate from the search server 30. FIG. As shown in FIG. 9, a search result title part 61 and a snippet part 62 are included for each search result on the search result page 60. Note that if the title engine and the snippet part are not clearly divided depending on the search engine used, a part of the text data of the search result is a snippet. As described above, the system control unit 14 of the product web page analysis server 10 functions as an example of a search result acquisition unit that creates a search query using a feature word candidate as a search keyword and acquires a search result based on the search query.

  Next, the product web page analysis server 10 narrows down to the search result of the shopping site (step S5). Specifically, the system control unit 14 of the product web page analysis server 10 narrows down the search results including the domain name of the shopping site in the URL (Uniform Resource Locator) linked to the search results. More specifically, as shown in FIG. 10, the system control unit 14 of the product web page analysis server 10 refers to the domain name DB 12b, and among the upper (for example, 30th) search results, the HTML tag “< The search result includes the domain name of the shopping site in the information “a href =...>” and the URL 62a of the snippet part 62.

  Next, the product web page analysis server 10 determines a feature word candidate when the category-related word is included in the snippet as a feature word (step S6). Specifically, the system control unit 14 of the product web page analysis server 10 selects a feature word candidate when a category-related word is included in the snippet of the search result narrowed down in step S5, and this feature word Candidates are determined as feature words. More specifically, as shown in FIG. 11, when the category is “wine”, the system control unit 14 of the product web page analysis server 10 includes the category-related words “wine” or “liquor” in the search result snippet. Are selected as feature words, and the feature word candidates are determined as feature words. Then, as shown in FIG. 8, for example, a phrase group 53c is obtained and stored in the feature word DB 12c.

  As described above, the system control unit 14 of the product web page analysis server 10 refers to the category related word DB 12a as an example of the category related word storage unit, and determines whether or not a category related word exists in the snippet of the search result. Functions as an example of search result determination means. Further, the system control unit 14 of the product web page analysis server 10 functions as an example of a feature word determination unit that determines a feature word candidate when a category-related word exists in a snippet as a feature word of each product web page. Further, the system control unit 14 of the product web page analysis server 10 refers to the category-related word storage unit and the domain name storage unit to determine whether or not a category-related word exists in the snippet of the shopping site in the search result. It functions as an example of a search result determination means for determining.

  Next, the product web page analysis server 10 calculates the similarity based on the feature word (step S7). Specifically, the system control unit 14 of the product web page analysis server 10 calculates the similarity by a subroutine that calculates the similarity between the product web pages. As described above, the system control unit 14 of the product web page analysis server 10 is an example of a similarity calculation unit that calculates the similarity between the first product web page and the second product web page based on the feature word. Function.

  Next, the product web page analysis server 10 determines the similarity of the product web page based on the similarity (step S8). Specifically, the system control unit 14 of the product web page analysis server 10 determines that a product web page that handles similar products between the target product web pages when the similarity calculated by the similarity calculation subroutine is equal to or greater than a predetermined value. Judged to be a page. In addition, you may determine with it being a product web page which handles the same goods, when a similarity is still higher among the product web pages determined to be a product web page which handles similar products. As described above, the system control unit 14 of the product web page analysis server 10 determines that the first product web page and the second product web page are product web pages that handle similar products when the similarity is equal to or greater than a predetermined value. It functions as an example of similar product determination means that determines that there is a product.

  Using the determination of the similarity of the product web pages, product web pages that handle the same product or similar products are collected, and a product web page database is constructed so that the user can browse through the user terminal 45. For example, a GUI (Graphical User Interface) button or the like for a certain product is provided on the web page, and when the user clicks, a list of product web pages that handle the same product or similar products can be viewed.

(3.2. Calculation of similarity based on feature words)
Next, a subroutine for calculating similarity based on feature words will be described with reference to FIG.

As shown in FIG. 12, the product web page analysis server 10 calculates the IDF value of each feature word (step S10). Specifically, the system control unit 14 of the product web page analysis server 10 sets a value of IDF (Inverse Document Frequency) as shown in the following formula (1) for each feature word, calculate.
IDF _w = log (N / N _w ) (1)

Here, IDF _w is the IDF value of the feature word w, N is the total number of web pages of a certain shopping site, and N _w is the number of web pages in which the feature word w appears in the shopping site. As the value of the number of web pages N _w of the feature word w is smaller, the value of the IDF _w is larger, and the feature word w is more likely to be a phrase unique to a product web page that handles products. The web page range for calculating the IDF value may be the number of pages crawled by a search engine, a web page in a certain country, a web page of a plurality of shopping sites in a certain country, or a target category. It may be a web page of a higher category.

Next, the product web page analysis server 10 calculates a similarity based on the IDF value (step S11). Specifically, the system control unit 14 of the product web page analysis server 10 performs the first product web page and the second product based on the IDF value calculated in step S10 as shown in the following equation (2). and calculates the degree of similarity S ₁₂ of the web page.
S ₁₂ = [ΣcεC ₁₂ (IDFc)] / [ΣuεU ₁₂ (IDFu)] (2)

Here, the set U ₁₂ is a set in which the feature words of the first product web page and the feature words of the second product web page are combined. That is, it is the union of the set of feature words of the first product web page and the set of feature words of the second product web page. Set C ₁₂ is a set of common feature words of the first items web page and the second of the product web page. That is, a product set of a set of feature words of the first product web page and a set of feature words of the second product web page. Further, the denominator of Expression (2) is the sum of the IDF values of the feature words that are the elements u of the set U ₁₂ , and the numerator of Expression (2) is the IDF of the common feature words that are the elements c of the set C _12. It is the sum of values. As shown in the equation (2), the more common feature words are in the first product web page and the second product web page, the higher the similarity S ₁₂ is, and the IDF of the feature words the higher the value, the similarity _{S 12} is high.

  In this way, the system control unit 14 of the product web page analysis server 10 functions as an example of a similarity calculation unit that calculates the similarity based on the feature words common to the first product web page and the second product web page. To do. In addition, the system control unit 14 of the product web page analysis server 10 is an example of a similarity calculation unit that calculates the similarity based on the reverse appearance frequency of a common feature word in the first product web page and the second product web page. Function as.

  Based on the similarity of the formula (2), the product web page analysis server 10 determines the similarity of the product web page in step S8.

  As mentioned above, according to this embodiment, the category related word relevant to the category which classifies goods is memorized in category related word DB12a, the 1st goods web page and the 2nd goods web page are acquired, and each goods web page 50, linguistically analyze text data such as text portions 51, 52, 53 described in 50, extract feature word candidates of each product web page, create a search query using the feature word candidates as search keywords, and search The search result page 60 based on the query is acquired, it is determined whether or not a category related word exists in the snippet of the snippet part 62 of the search result page, and a feature word candidate when the category related word exists in the snippet is determined. Determine the feature word of each product web page, and calculate the similarity between the first product web page and the second product web page based on the feature word When the similarity is equal to or higher than a predetermined value, the similarity between the products on the product web page is determined by determining that the first product web page and the second product web page are product web pages that handle similar products. For example, since the product web pages of the same product can be collected together, the convenience of the user can be improved.

  Further, according to the present embodiment, when narrowing down feature word candidates, the use status of words on the Internet can be reflected by using a snippet obtained by web search, and a highly accurate feature word can be extracted. Furthermore, feature words can be obtained at high speed by limiting to snippets. Furthermore, by using a web search snippet, even if the trend of a product or the like changes, it can follow the trend, an appropriate feature word is obtained, and it can be appropriately determined that the product is a product web page.

  In addition, when the system control unit 14 of the product web page analysis server 10 calculates the similarity based on the common feature word in the first product web page and the second product web page, the common feature word is more accurately used. Similarity can be calculated.

  Further, when the system control unit 14 of the product web page analysis server 10 calculates the similarity based on the reverse appearance frequency of the feature words common to the first product web page and the second product web page, the reverse of the feature words Since the feature words can be ranked based on the appearance frequency, similarity can be easily determined.

  Further, the system control unit 14 of the product web page analysis server 10 refers to the category related word DB (category related word storage means) 12a, and in the search result, whether or not a category related word exists in the snippet of the shopping site. In the case of determining whether or not, a feature word that appropriately characterizes the product can be extracted and the accuracy of the feature word can be improved because the product is narrowed down to a shopping site having many product web pages in which the product is described in order to sell the product.

  In addition, when the system control unit 14 of the product web page analysis server 10 removes noise morphemes from morphemes obtained by morphological analysis of text data, accurate feature word candidates can be extracted. Becomes higher.

  The search server 30 may be a search server of another search site outside the server system 5. In this case, the product web page analysis server 10 transmits a search query and receives a search result through the network 3. Moreover, the web search range for obtaining a snippet may be limited to a web page in the shopping server 20 or may be limited to a web page in a certain country or region.

  Furthermore, the present invention is not limited to the above embodiments. Each of the above embodiments is an exemplification, and any configuration that has substantially the same configuration as the technical idea described in the claims of the present invention and has the same operational effects can be used. It is included in the technical scope of the present invention.

3: Network 5: Server system 10: Product web page analysis server (product web page analysis device)
12a: Category related word DB (category related word storage means)
12b: Domain name DB
20: Shopping server 30: Search server

Claims (7)

Category-related word storage means for storing category-related words related to categories for classifying products;
Product web page acquisition means for acquiring a first product web page and a second product web page;
Text data extraction means for extracting text data described in each product web page;
Linguistic analysis of the extracted text data, and feature word candidate extraction means for extracting feature word candidates of each product web page;
A search result acquisition means for creating a search query using the feature word candidate as a search keyword and acquiring a search result based on the search query;
Search result determination means for referring to the category related word storage means to determine whether or not the category related word is present in the snippet of the search results;
Feature word determination means for determining the feature word candidate when the category-related word is present in the snippet as a feature word of each product web page;
Similarity calculation means for calculating the similarity between the first product web page and the second product web page based on the feature word;
When the similarity is equal to or greater than a predetermined value, the similar product determination unit that determines that the first product web page and the second product web page are product web pages that handle similar products;
A product web page analyzing apparatus comprising: The product web page analyzing apparatus according to claim 1,
The product web page analyzing apparatus, wherein the similarity calculating unit calculates the similarity based on the feature word common to the first product web page and the second product web page. The product web page analyzing apparatus according to claim 2,
The product web page analyzing apparatus, wherein the similarity calculating unit calculates the similarity based on a reverse appearance frequency of the feature word common to the first product web page and the second product web page. In the goods web page analysis device according to any one of claims 1 to 3,
A domain name storage means for storing the domain name of the shopping site;
The search result determining means refers to the category related word storage means and the domain name storage means to determine whether or not the category related word exists in a snippet of a shopping site in the search result. Product web page analysis device. In the goods web page analysis device according to any one of claims 1 to 4,
The product web page analyzing apparatus, wherein the feature word candidate extracting means includes noise removing means for removing noise morphemes from morphemes obtained by linguistic analysis of the text data by morphological analysis. A category-related word storage step for storing a category-related word related to a category for classifying products;
Product web page acquisition means for acquiring a first product web page and a second product web page;
A text data extraction step for extracting text data described in each of the product web pages;
A linguistic analysis of the extracted text data, and a feature word candidate extraction step of extracting feature word candidates of each product web page;
A search result acquisition step of creating a search query using the feature word candidate as a search keyword and acquiring a search result based on the search query;
A search result determination step for determining whether or not the category-related word is present in the search result snippet;
A feature word determination step of determining the feature word candidate when the category-related word is present in the snippet as a feature word of each product web page;
A similarity calculation step of calculating a similarity between the first product web page and the second product web page based on the feature word;
A similar product determination step for determining that the first product web page and the second product web page are product web pages that handle similar products when the similarity is equal to or greater than a predetermined value;
A method for analyzing a product web page, comprising: Computer
Category-related word storage means for storing category-related words related to categories for classifying products;
Product web page acquisition means for acquiring the first product web page and the second product web page;
Text data extraction means for extracting text data described in each product web page;
Linguistic analysis of the extracted text data and feature word candidate extraction means for extracting feature word candidates of each product web page;
A search result acquisition means for creating a search query using the feature word candidate as a search keyword and acquiring a search result based on the search query;
Search result determination means for determining whether or not the category related word exists in the snippet of the search result with reference to the category related word storage means,
Feature word determination means for determining the feature word candidate when the category-related word is present in the snippet as a feature word of each product web page;
Similarity calculating means for calculating the similarity between the first product web page and the second product web page based on the feature word; and
When the similarity is greater than or equal to a predetermined value, the first product web page and the second product web page function as a similar product determination unit that determines that the product web page handles similar products. A program for a product web page analyzer.

Images