KR20000049333A - Engine for comparatively searching product of internet shopping mole with intelligence type - Google Patents

Abstract

PURPOSE: An engine for comparatively searching a product of an internet shopping mole with an intelligence type is provided to increase a recognition rate and thereby reduce an error, by determines if information having uncertain type exists, analyzing and classifying a content of the information in order to obtain separate valuable data from the information. CONSTITUTION: A web searching system detects the change of a structure in a web page and a home page, updates data periodically and sets an update period to reduce a load. The web searching system includes a Meta Robot and Search Robot. The Meta Robot searches all web pages randomly, and distinguishes a searched web page from a web page that will be searched. The Meta Robot supports a distribution processing, and transfers data to the Search Robot if desired data is not included in the searched web page. The Search Robot searches only a web page containing the desired data. The Search Robot includes the first mode for checking the web page structure, the second mode for supporting a quick search and the third mode for detecting the change of the web page structure.

Description

Intelligent Internet Shopping Mall Product Comparison Search Engine {.}

Instead of requiring separate processing from shopping malls on the Internet, product information is automatically extracted and classified from documents provided in web form and provided to users as a directory type service.

Internet intelligent agent and search robot

In the past, it was simply to compare the words existing in the document to determine whether they matched. Therefore, if the contents can be properly provided, such as product information, such as product name, price, manufacturer, etc. I could not. Similar softwares already developed for the related fields also require a considerable amount of manual work in the registration and processing of shopping malls, product search, or have not been able to provide information smoothly due to the excessively low recognition rate and errors.

Increasing the low recognition rate and error of the existing intelligent agent software by determining whether there is an unclear form of information and by analyzing and classifying the content of the information to distinguish it as valuable separate information. It solved the problem of excessive management needs.

end. Web search system

(1) Introduction

The web search system is a system that automatically retrieves data from the web with specific information, updates data periodically, and catches new information. When the structure of the webpage changes or the structure of the homepage itself changes, it automatically checks for this and updates the data periodically, and includes a function to reduce the load by setting the update cycle automatically.

(2) Functional diagram

The web search system consists of two systems, Meta Robot using Robot technology and Search Robot.

end. Meta robot

Meta Robot searches randomly across the entire web. It constructs its own DB for the entire website, and distinguishes between searched and unsearched websites. In addition, it does not have a basic web site DB initially, and simply designates a specific web site as a starting site, and continuously searches using a web-like structure of the entire web.

As it is operated as a multi process by itself, distributed processing is possible and its multi performance can be determined.

While searching a website, it determines whether or not specific information exists in connection with an information analysis system, and delivers data to Search Robot when specific information exists.

If a keyword corresponding to specific information is clearly present, it may be higher than that of the entire web through the filtering function using existing search engines at home and abroad.

I. Search Robot

Search Robot searches only those sites where specific information exists among the websites determined by Meta Robot. The first mode to grasp how the specific information is displayed and the structure of the website, the second mode to quickly search based on the determined structure, and the third mode to identify when the structure of the website is changed or added / deleted. It consists of three parts.

(i) First Mode

First Mode is a mode used when searching for a specific site for the first time. The first mode is used to determine the structure of the website and the expression method of the specific information. Accurately separate and register only pages with specific information within the website. In addition, when there is a large amount of unnecessary information such as a bulletin board or a guest book, it is determined and excluded from the search to improve the search speed. By default it will search the entire page of your website. In addition, in order to prevent Search Robot from expanding the search target outside of a specific website, it sets itself a boundary. This boundary prevents the search target from being changed to another site.

(Ii) Fast Mode

Fase Mode searches based only on the information found in First Mode, so data can be updated quickly. That is, the search is performed only on the page where the information exists. The structure is also compared based on the data prepared in First Mode. When the data is newly added or deleted / changed, all of them can be detected, and the difference with the existing data can be detected, so the information is perfectly imported.

There is a separate Server program that runs Fast Mode periodically. It includes its own scheduling function, so you can set the interval to search periodically. Even if you initially set the frequency as the default, search through it several times to see how often the information on your website changes. Thus, the load on the other web server can be reduced to a minimum.

After importing the data of the web page where the information exists, only the target data is accurately imported by interlocking with the information analysis system. In parallel with this process, the other web server is automatically accessed at intervals. Fast Mode cannot detect when the structure of the website is changed. In this case, it is solved by calling Restart Mode, which is the third mode, through error handling.

(Ⅲ) Restart Mode

Restart Mode is a combination of First Mode and Fast Mode. By importing information only on pages where specific information exists, the entire structure of the website is identified to identify the new structure. The biggest difference from First Mode is that the administrator does not update the data manually and reflects the result of manual work. You can also update data through the fast speed used in Fast Mode.

(3) Implementation method

end. Meta robot

Meta Robot is divided into three parts: HTTP Client which can import HTML file or CGI, Parsing URL, and Cache part of URL.

HTTP Client solves through typical network programming method using C language or Java. The header is controlled by itself in order to operate on the virtual web hosting site, and the robots.txt file, a protocol for excluding the web robot, is processed.

Parsing the URL is done with two main arguments. The main purpose is to get an absolute URL based on the link information on the HTML TAG.

It saves the current URL in current_url and saves the current relative path as reference_url to make them one url. If a wrong link is made on a specific website, it is determined as an error. Basically, it is implemented in the same way that the absolute url is obtained from the web browser. Determining the relationship between absolute paths of two urls is implemented by using '/' to separate physical paths. This is described in detail later in the section called URL Parser.

The caching of URLs is to avoid re-searching the same website and to prevent the robot from performing infinite circulation on multiple websites. It distinguishes between the searched site and the site that has not been searched yet, and the site where specific information exists delivers the data for Search Robot to search. It uses DB and consists of three fields: URL, Contents, and SEARCH_TIME. The URL stores the absolute path, Contents contains the content, and SEARCH_TIME represents the search time.

Contents can have three values: NOT, END, and DATA. NOT indicates that the search has not been done yet, END indicates that the search has been completed, and DATA indicates the state where data is sent to the Search Robot because specific information exists. SEARCH_TIME saves the last searched time so that the website can be searched again periodically.

I. Search Robot

Search Robot will refer to two DB Tables, the structure of which will be explained first.

(i) DB

In order for the Search Robot to work, two pieces of information must be stored: one for storing the page where the information exists within the website, and one for storing the presentation of the information.

The page on which the information is present and the way it is displayed can be changed within the website, which can be detected in Restart Mode. However, in order to update information in fast mode at a high speed, it is necessary to store it in a cache form. This table consists of four fields: URL, C0NTENT, STATE, and TlME. URL stores absolute path and CONTENT stores what information. In other words, it is possible to search only the page containing the specific information in Fast Mode by determining whether the specific information is included or not.

The STATE field indicates the current state. It saves whether you are searching, whether the search is over, or not yet. TIME saves the latest time when the search is completed, and it is configured to be able to search again at a specific interval.

Storing the expression method is to identify and store the expression format of specific information in the information analysis system, which will be explained in the information analysis system.

(ii) First Mode

First Mode consists of the following steps.

ㆍ Search Target Website-Based on the information delivered by Meta Robot, find a website containing specific information that should be searched in First Mode.

• Check the current status-Stores the status that Search Robot is currently searching for a specific website. Do not search for duplicates while working as a Multi Process.

• Delete existing data-In case of operating in First Mode, existing data does not exist, but deletes information that may remain if the same site has different URL or terminates abnormally in existing First Mode. .

• STATE Modification-Adjust STATE to be the target to search all URLs corresponding to the website. If you are searching the website for the first time, this function is not necessary, but it may be abnormally terminated and retried in the existing First Mode, so change the STATE to NOT all.

ㆍ Determining URL to search-Recursively call unsearched URLs one by one. Whenever we search for a page, we check that the search is over, so if we simply call them one by one, we will actually search all the pages in order.

ㆍ Get Web Page-Get the page of the URL using HTTP Client.

ㆍ Determining the existence of information-Identify whether the imported web page contains specific information. If specific information exists, it is classified as a page with information. Otherwise, it is stored as a page without information.

ㆍ Information Storage-In the case of a page where specific information exists, the specific information is extracted and stored in association with the information analysis system. Instead of entering the DB directly, a temporary DB is used, which is used to recover data in case of error. After searching the entire website, copy the contents of temporary DB to main DB.

URL Parser-Using a program that parses URLs, it grasps all the links to other pages in the imported web page. In this way, you will search the entire website. The implementation of the URL Parser will be explained in detail later.

URL inspection-The parsed URL is deleted except for those belonging to the boundary of the website.

• Check the current status-Save the end of the search and change the last searched time.

DB Update-Moves data from the temporary DB that was used temporarily while searching to the actual DB. This completes the registration process.

(Ⅲ) Fast Mode

First Mode consists of the following steps.

ㆍ Search Target Website-Find a website to search in Fast Mode. The latest search time and the current time are compared to determine the search time.

• Check the current status-Stores the status that Search Robot is currently searching for a specific website. Do not search for duplicates while working as a Multi Process.

ㆍ Copy existing information-Since the function saves the changes based on the existing information, copy the contents from the main DB to the temporary DB without deleting the existing information.

• STATE Modification-Adjust STATE to be the target to search all URLs corresponding to the website.

ㆍ Determining URL to search-Recursively call unsearched URLs one by one. Whenever we search for a page, we check that the search is over, so if we simply call them one by one, we will actually search all the pages in order. Since it is Fast Mode, it searches only the pages where information exists in the existing URL DB.

ㆍ Get Web Page-Get the page of the URL using HTTP Client.

ㆍ Determining the existence of information-Identify whether the imported web page contains specific information. In the case of Fast Mode, only the information page is searched. However, if the structure of the website is changed, the information may not exist. In this case, it is determined that the structure of the website has been changed. Works.

ㆍ Information Storage-In the case of a page where specific information exists, the specific information is extracted and stored in association with the information analysis system. Instead of entering the DB directly, a temporary DB is used, which is used to recover data in case of error. After searching the entire website, copy the contents of temporary DB to main DB. If the same information already exists, only the change is identified and stored. In addition, when comparison with existing information is needed, the DB is configured to store the changes in a separate field.

• Check the current status-Save the end of the search and change the last searched time.

DB Update-Moves data from the temporary DB that was used temporarily while searching to the actual DB. This completes the registration process.

ㆍ Period setting-If the existing information and the newly searched information are very different, the search period is further reduced. Otherwise, if the existing information and the newly searched information match, the search period is further increased. As a rule of thumb, it can reduce the load of the web server by identifying the change cycle of the web site.

(Ⅳ) Restart Mode

Restart Mode consists of the following steps.

Target Web Site Search-Determines the website to perform Restart Mode based on the information delivered in Fast Mode. There are two cases in which the Restart Mode is executed, when a change in the structure of the website is detected in the Fast Mode, or when the time period for performing the Restart Mode has passed. The time period for performing Restart Mode is automatically set.

• Check the current status-Stores the status that Search Robot is currently searching for a specific website. Do not search for duplicates while working as a Multi Process.

ㆍ Copy existing information-Since the function saves the changes based on the existing information, copy the contents from the main DB to the temporary DB without deleting the existing information.

• STATE Modification-Adjust STATE to be the target to search all URLs corresponding to the website. If you are searching the website for the first time, this function is not necessary, but it may be abnormally terminated and retried in the existing First Mode, so change the STATE to NOT all.

ㆍ Determining URL to search-Recursively call unsearched URLs one by one. Whenever we search for a page, we check that the search is over, so if we simply call them one by one, we will actually search all the pages in order.

ㆍ Get Web Page-Get the page of the URL using HTTP Client.

ㆍ To grasp the structure of the web page-If there is a change in the information expression method of the web page previously identified, it is reconstructed and stored.

ㆍ Determining the existence of information-Identify whether the imported web page contains specific information. If specific information exists, it is classified as a page with information. Otherwise, it is stored as a page without information.

ㆍ Information Storage-In the case of a page where specific information exists, the specific information is extracted and stored in association with the information analysis system. Instead of entering the DB directly, a temporary DB is used, which is used to recover data in case of error. After searching the entire website, copy the contents of temporary DB to main DB.

URL Parser-Using a program that parses URLs, it grasps all the links to other pages in the imported web page. In this way, you will search the entire website. The implementation of the URL Parser will be explained in detail later.

URL inspection-The parsed URL is deleted except for those belonging to the boundary of the website.

• Check the current status-Save the end of the search and change the last searched time.

DB Update-Moves data from the temporary DB that was used temporarily while searching to the actual DB. This completes the registration process.

Period setting-If the existing website structure is different from the newly identified website structure, the restart search period is reduced. If the existing website structure and the newly identified website structure are the same, the restart search period is increased. As a rule of thumb, it can reduce the load of the web server by identifying the change cycle of the web site.

All. URL Parser

The URL Parser parses based on three arguments: the current url, the website name, and the HTML file name. Based on this, the work is done in the following order.

Link Tag Identification-Finds all tags in the HTML TAG that represent links to other pages. This is mainly the case using the <a href "tag and the javascript technology.

ㆍ Link string judgment-If link is in the text string, check this.

This is not necessary in the web search system, but when linked with the classification analysis system, the string itself may have a great meaning.

Determining the Current URL-Determining the current URL. This is to determine the depth of the current directory, mainly around the '/' character that divides the physical structure.

Link URL Analysis-Analyze link URL. The link URL may be an absolute path, a relative path, or only a file name. In case of absolute path, it can be used as it is. In case of relative path or file name, a new URL should be created by combining with the current URL.

I. Information Analysis System

(1) Introduction

The main purpose of the information analysis system is to read external document information and to analyze the contents contained in it to form a consistent information system. In the current system, an information analysis system is used to read HTML-formatted documents from external sites on the Internet and to extract product-related information existing in the document.

The information analysis system introduced human-centered thinking in reading and processing external information. That is, the information that humans rely most on analyzing information is image information, and the feature that distinguishes image information from other information is that location information exists. We have developed and applied a start-based information recognition method that distinguishes them from each other so that humans can obtain results similar to those when analyzing information. This concept is commonly applied to the whole system, and the parts that discriminate the structure of information by themselves or identify the structure of similar information are based on such human-centered thinking.

(2) Main function

The overall information analysis system has four main characteristics. These features are hard to find in similar types of analysis systems and are the biggest factors that define information analysis systems.

The main functions and explanations of the information analysis system are as follows.

A) Visually oriented information recognition

Currently, the entire system has collected information of various contents and formats from various sites based on the Internet environment. Therefore, the information gathered from the whole system is very diverse in its kind and format, making it difficult to find any consistent composition or concept. The current computer program technology has a common problem that all steps are logically operated, and thus batch data processing is impossible for illogical or overly diverse data types. Therefore, a more special type of technology is required to solve these problems. do.

In the information analysis system, the visual information recognition method is adopted to solve these problems. Visually-based information recognition method analyzes the information of a given document format and reconstructs it into a form that includes visual information such as up, down, left, and right together. To analyze information in such a way as to When this technique is applied, it helps to find exactly the desired information based on a wider variety of information by extending the kind of attributes that the information has from simple text-based information to visual information.

In case of using the visual-based information recognition method, it is possible to extract various information in addition to the extraction of product information like the one used in the entire system, and to grasp information on the priority and hierarchy of the information. It can have various advantages such as being applicable as an enemy.

B) Automatic extraction of key information

One of the most important problems in dealing with a variety of information that you don't define yourself is to find out what you want. In particular, if you want to clearly extract thousands or tens of thousands of similar information based on simple information such as the whole system, the importance becomes more important. Knowing exactly what is the information you want is helping to reduce the errors that can occur in information extraction. However, to this end, up to now, it usually uses simple dictionaries or relies entirely on human help. In this case, the proportion of manual labor is greatly increased, and thus the desired level of automation cannot be achieved.

However, the information analysis system has an automatic extraction function for key information, so that the desired information can be extracted even without human assistance. To this end, the information analysis system utilizes a separate analyzer that analyzes according to the type and format of information in addition to the analysis function by comparing word units, and analyzes the information. Therefore, much wider information can be placed on the analysis target than the conventional method, and the depth and accuracy of the information is also very high. And by automating the information acquisition process, it is possible to automate the entire stage of information analysis and greatly reduce the management area of the manager.

C) Intelligent information analysis

Even if you narrow down the scope of information you want, there is a significant difference from knowing exactly what you want. Even if the structural form that can represent the information is analyzed, it is still difficult to analyze all the similar information on the basis of fragmentary analysis information. In order to solve this problem, conventionally, a method of automatically extracting information only for a few cases of perfect matches is not available, so that all information cannot be obtained forever. From the perspective of search engines targeting an infinite number of pieces of information on the Internet, these fundamental constraints are a huge obstacle and must be overcome.

In order to solve this problem, the information analysis system developed an intelligent information analysis capability that infers even though the structure and contents of the information are not completely consistent, and judges that the information is different but identical. The whole information can never exist consistently in one structure, and such forms of assumptions should not be assumed by default. The information analysis system makes no assumption that there may be any form from the outset, and is conceptually completely separate from the way of considering the number of cases. The information analysis system knows only the basic rules on what information is key information and how it interacts with it, and does not have any kind of database related to the type of shopping mall. Even if they are outside of the box, they can analyze the information and solve the problem themselves for the unpredictable situation that may occur later. Even when the relevant information does not exist, it is possible to make a judgment that the related information does not exist, so that even when the information does not exist, it can be processed. Therefore, the information analysis system fundamentally solved the problems that could occur in obtaining the relevant information through the intelligent analysis ability of such information, and secured the permanence that can be deduced in the same way forever.

D) Convenient management structure

Although the information analysis system was developed independently of human management with the aim of being fully automatic, it does not mean that human management is unnecessary at all. Even with intelligent software, it is natural to be more accurate with human help, and considering the possibility of unpredictable errors, human management must be fully considered and prepared.

In the information analysis system, human management can only intervene in the early and late stages of information acquisition. The actual process of acquiring information completely excludes the participation of people, but only allows checking before and after the information. In this case, the manager only needs to help in advance the information related to the acquisition of the information in advance, and confirms it briefly afterwards, so that the manager is completely free from all kinds of excessive work occurring in the acquisition of the information. This can drastically reduce management costs by bringing down the number of people required for the management of the information analysis system to one level, and reduces errors caused by mistakes due to less manager intervention.

All. Classification Analysis System

(1) Introduction

The classification analysis system is a system that automatically analyzes and grasps classifications corresponding to external data when inputting external data into its own DB including classification information. Therefore, it can be usefully used in a system that automatically inputs external data into its own DB.

When the classification analysis system is applied to a website, it includes a function of determining the classification of the web page itself and a function of determining a separate classification for each information in the page. It also includes the ability to match external and self-classifications when the category is comprehensive or detailed rather than self-classification when other web sites are present.

(2) Functional diagram

It consists of two functions: a function of determining a classification of an information group including a plurality of information, and a function of determining a corresponding classification for each piece of information. In addition, classification judgment using these two functions is possible. Each function will be described in detail as follows.

(A) Classification judgment function for information group

It is a function to judge the whole group of information as one classification. In this case, the case where the information group is included in multiple classifications is excluded because it is implemented as a complex function described in (c).

Prior to determining the classification of the information group, the correlation between the various information groups is identified. The relationship between the information groups is reconstructed as a logical, not physical, to determine the top, bottom, parallel, and parallel relationships among the information groups to generate several groups of information in one classification.

The generated logical classification is matched with its own classification. In this case, it is divided into three cases as follows.

1. The classification of self-classification and external information is consistent

In this case, the classification itself is a perfect match, so the classification of external information can be used without any additional work.

2. The classification of external information is more detailed than its own classification.

When the classification of the external information is more detailed than the own classification, since a specific item of the own classification includes a plurality of external classification items, the classification is determined by combining detailed external classification information and matching it with the own classification.

3. Self classification is more detailed than classification of external information

When the self classification is more detailed than the classification of the external information, the classification of the external information group is a function of determining the classification for each information. In this case, the classification is determined by dividing the information group into several information groups again through the classification determination function of one piece of information described in (b).

(B) Classification judgment function for a single piece of information

This function determines the classification of each piece of information. The classification is determined according to the content of the information. In this case, a separate dictionary DB is required. The dictionary DB is not automatically inputted by the administrator, but is automatically generated based on the previously determined classification or internal information DATA. In order to prevent errors in classification judgment, the dictionary DB does not use a simple method of matching information and classification 1: 1, and connects the information and classification in each classification, and this method uses a recursive call method. Implemented regardless of the depth of classification.

(C) Complex classification judgment function

It is a function that judges by using two kinds of classification judgment functions described in (a) and (b). If the classification of external information is more comprehensive than its own classification, or if the information group itself contains information corresponding to multiple classifications, this function is used to determine the classification.

The classification described in (a) is first determined, but is not matched with the classification of the lowest sub item, and is determined as an upper classification including the sub classification. The classification is determined for each information by using the dictionary DB mentioned in (b). An error that does not include the classification of each information in the classification of the currently determined information group does not occur because the dictionary DB is recursively configured to apply to the information group.

(3) Implementation method

The interior of the classification analysis system is largely divided into two parts: classification judgment for information groups and classification judgment for each information. Since the matching method between the classification determined for the external information and the self classification is performed individually in each case, it is not composed of separate parts.

Each implementation method will be described based on the form of a shopping mall comparison search engine which is the current overall system. Therefore, information refers to a product being sold in a shopping mall, and the information group refers to a web page in which a plurality of products exist.

(A) Classification judgment function for information group

If there is already a classification of products for sale in a shopping mall, the program should determine the logical classification system by identifying these correlations. In this case, the characteristics of HTML TAG itself are understood and analyzed by using the characteristic of web page.

1. The ability to judge the relationship between each other

In order to determine the relationship between web pages, we basically grasp based on HTML TAG which can be a link. However, in this case, it is important to identify the web page as a top / bottom relationship because the web page does not have a physical structure in the upper / lower structure and is connected like a spider web. In order to grasp this structure, it is necessary to determine the core link among the multiple links in one page unit. The first link and the previous link are stored and reorganized. In this case, the exclusions are those connected in parallel. For example, a product page is divided into several parts and composed of 'previous' and 'next'. This parallel structure will be described again below.

In logically changing the structure of web pages connected like spider webs, we explained that we need to find the core link among several links. By reconfiguring the link, it is possible to find only the key link.

In the case of a link, it finds all the information deemed necessary and finds the core link based on it, which is processed recursively. That is, assuming that the classification of the upper stage is standing, and that the form is matched with its own classification structure, the classification corresponding to one product page is found.

The information needed in the connection of a general web page is a string to be connected, which is composed by comparing a separate dictionary. Called link_dictionary, this dictionary is configured to determine the classification corresponding to the link string, which basically has 1: 1 matching strings with the product classification, and has similar words with those for more detailed items. A separate similar dictionary is applied, which is identical to the similar dictionary used in a general search.

Even if there is no concatenated string, there is no problem in the classification judgment, because the implementation method based on the string is one of several structural identification methods.

Even if there is no information on one product page, and only a product list can be obtained, the product classification can be grasped, because there is a function to determine the classification of each product. By using this function, even if it is a 'hit product' page that includes products of several categories, the classification of products can be found accurately.

2. Ability to grasp parallel structure

In general, when pages with product information exist in parallel, they are configured as page links. This general information can be determined. However, even if it is not in the form of such a page link, it is possible to determine the classification of a page through the function of judging the product contained in each page. To judge. In addition, even if it is configured in a manner that is not substantially parallel structure, since it is determined based on the information itself regardless of the physical connection, it appears as a parallel structure in the reconstructed form.

(B) Classification judgment function for each information

In order to determine the classification corresponding to each product information, a separate ITEM_D dictionary is required. This dictionary is automatically generated based on the product information that you have. If a product with the same brand name exists in different classifications, it may cause an error. This problem is not composed of the ITEM_D DB structure that matches the product name and the classification information in a one-to-one manner, but includes a self-classification. It is possible to solve this problem.

la. Intelligent search system

(1) Introduction

Intelligent search system refers to a system for efficiently searching data in a database with a classification structure. It is a system that analyzes the meaning of search terms and finds correlations with classifications to find only the results that users want from a large number of data.

(2) Functional diagram

The intelligent search system is divided into three categories: one word search, two word search, and compound word search. In each case, it also has a function to grasp the correlation with the classification structure.

(A) Single word search

In the simplest case, a user searches for a word, it is divided into two categories. By referring to the dictionary of synonyms, both the representative word and the similar word for the search term are found. If there is no similar word, the search word itself becomes a representative word, and the search word itself is treated as a single similar word.

Based on the classification information, identify the classification corresponding to the representative word. In this case, the classification is divided into a case where there is a classification and a case where there is no classification. When a classification does not exist, a general similar word search is implemented. If the classification exists, the information in the classification is displayed. If the classification is a top category instead of the lowest category, the sub category is displayed, so that a more detailed classification can be selected when a comprehensive search term is entered. do.

(B) two-word search

In the case of two word search, the correlation between two words is determined. First, find representative words and similar words for each of the two words. Similar to single word search, the corresponding classification is searched based on the found representative word. The classification may or may not exist. Unlike single word search, each search word may or may not have a classification. It is divided into three rather than four because the order between two words is not important. In general, even if a word appears in detail after a comprehensive word, the correlation itself is judged again, so there is no meaning.

(Iii) no classification exists for both search terms

If both search terms do not have a classification, a general AND and OR search is performed. Since the previous work has already determined the representative words and similar words, it automatically includes the function for searching for similar words.

(Ii) if only one search term exists

If only one search term exists, a classification is automatically searched. In the corresponding category, information corresponding to the other search term is found. In this case, the similarity function determined above is included.

(Iii) a classification exists for both search terms

If the classification exists in both search terms, the upper and lower relations between the two classifications are determined. Therefore, it can be divided into several cases according to the relationship between the two classifications. The first is the case where the inclusion relation between the two categories is established, and the information corresponding to the subcategory is shown.

Second, there is no inclusion relationship between the two classifications. In this case, an intra-category search corresponding to the second search term is performed in the first category, and an intra-category search corresponding to the first search term is output in the second category.

(C) Compound word search

Compound word means three or more words. In the case of inputting a search word of three words or more, the search is basically performed in the same manner as the two word search. However, since it cannot be divided into three cases like two words, it is necessary to recursively reduce the search word. That is, when two or more search terms corresponding to the classification exist among the search terms and an inclusion relationship is established between the two, the search terms corresponding to the upper classification are removed. If the search term becomes less than 2 words through this process, the search function implemented in (A) and (B) is executed. If more than 3 words appear after reducing the search word, the result is output by AND and OR search. .

(3) Implementation method

Before we explain how to implement it, let me introduce the necessary DataBase to build an intelligent search system. The implementation of three or more compound words is the same as searching for two words, and there is a difference between using a recursive implementation internally. In addition, each implementation method will be described based on the shape of a shopping mall comparison search engine which is the current overall system. Thus, classification refers to a classification that includes a product, and information refers to product information.

(A) synonym dictionary DB

Since the analogy dictionary uses only a search word, its structure can be very simple. It is composed of a list of representative words and similar words corresponding to the representative word. In the case of a multiword word, since each word is judged separately, it is assumed that there is no representative word for the multiword word. Therefore, this process is performed by creating a separate function to remove spaces or other special characters.

Because the dictionary of synonyms is not fixed, but can change dynamically, you need a tool to register, modify, or delete it. In addition, in order to effectively register similar words, the user can easily register the similar words for the words frequently searched by the user by analyzing the log of the search word searched by the user.

(B) Single word search

For a single word search, the first thing to do is to find a representative word and a similar word. This process uses the synonym dictionary DB mentioned above, which can be easily solved using the SQL select statement. The processing of multiple similar words is implemented using the list tree structure.

Secondly, the product classification corresponding to the representative word is found. The reason for using only the representative word without using the similar word is that the similar word matching the product classification is registered as the representative word in the process of registering several similar words.

If the product classification exists, the product in the product classification is displayed as a result regardless of the search word. If the product string does not exist in the same category, the search is possible. The search order processed internally is as follows.

'mouse' search-> 'mouse' as a representative-> 'mouse' as a synonym-> search for a product category corresponding to 'mouse'-> print a product in the 'mouse' category-> name as 'Logitech Wheel' Searchable even if 'mouse' doesn't exist

(C) two-word search

In the case of searching for two words, the classification corresponding to each search word needs to be found.

The entire search term is separated into tokens using space as the token separator. The separated token removes the space again because there is no space in the synonym dictionary DB itself. The existence of a space already implies that there is more than one word in itself, so there is no space in the dictionary of words.

Each token that is separated is searched for a representative word and a similar word through a dictionary of similar words. This process is the same as for the single word described in (b).

In both words, a representative word must exist, because a search word that is not registered in a similar dictionary is used as the representative word.

The representative words are used to find the corresponding classification in the classification DB. In the classification dictionary, the classification name is connected with the representative words of the similar dictionary. This is because when a new dictionary of similar words is registered, a similar word corresponding to a classification among the similar words is registered as a representative word.

When the classification is found, it is divided into three cases.

(Iii) no classification exists for both search terms

If both search terms do not have a classification, a general AND and OR search is performed. In this case, the similarity function is included. The following is implemented using SQL Query.

DICTIONARY and 'keyword field' mean a field corresponding to a table of similar words and a representative word, respectively.

select * from [DICTIONARY] where [keywordField] = 'keyword'

SQL Query uses regular expressions by default, because there are many similar words. Multiple synonyms are all ORed, and many can be implemented recursively. Therefore, the entire query is implemented in the following form.

DBName and model indicate a field containing a randomly designated DB table name and a brand name.

select * from [DBName] where model regexp [prescript 1 OR synonym 1 OR synonym 2 OR ...] AND [prescript 1 'OR synonym 1' OR synonym 2 'OR ...]

In addition, in case of OR search between two search terms, SQL Query is implemented as follows.

select * from [DBName] where model regexp [prescript 1 OR synonym 1 OR synonym 2 OR ...] OR [prescript 1 'OR synonym 1' OR synonym 2 'OR ...]

(Ii) if only one search term exists

If only one of the two search terms has a classification, the search is performed within the classification and includes a similar word search function by itself.

The DB containing the classification information has a classification code. By using this classification code, a SQL Query is constructed as follows.

select * from [DBName] where [classification code Field] = 'classification code' AND model regexp [keyword1 OR synonym1 OR synonym2 OR ...]

Each product contains information on the corresponding classification, which is stored in the "Category Code Field" above. In addition, the "classification code" means a classification code extracted from the search word.

For example, this is the case of '15 inch monitor '. Since there is a classification corresponding to' monitor ', the string 15 inches is searched for in the monitor classification. In addition, since 15 inches and 15 "exist in the similar word dictionary, a product such as a" 15 "plane can be searched.

(Iii) a classification exists for both search terms

If a category exists in both search terms, it is divided again by the inclusion relationship between the two categories. The first case is when two classifications have inclusion relations with each other, and the second case is no inclusion relation with each other.

In the case of a search term such as 'computer mouse', a classification corresponding to 'computer' exists, a classification corresponding to 'mouse' exists, and the two classifications have an inclusion relationship with each other. In this case, the product in the 'mouse' category will be shown. Even if the string 'computer mouse' is not found in the product information, it is possible to search. Products with completely different product information such as 'Logitech Wheel' appear in the search results, so intelligent search is possible.

In the second case, there is a classification corresponding to 'computer', such as 'computer desk', and the classification corresponding to 'desk' exists in the 'furniture' category. In this case, a compound AND search is performed, which searches for 'desk' in the 'computer' category and at the same time searches for a product 'computer' in the 'desk' category. If there is a 'desk' in the 'computer' category and a 'computer desk' in the 'desk' category, both can be shown as search results.

A simple SQL query looks like this:

select * from [DBName] where ([Category code Field] = 'Category code 1' AND model regexp [Representative 1 OR Analogue 1 OR Analogue 2 OR ...]) OR ([Category code Field] = 'Category 2 'AND model regexp [keyword1' OR synonym1 'OR synonym2' OR ...])

Compared with the example presented above, the classification code 1 corresponds to the 'computer' and the representative word 1 corresponds to the 'desk'. Classification code 2 is a classification corresponding to 'desk', and the representative word 1 is 'computer'. In this case, the similar words can be searched through the similar word dictionary.

Almost all information on the Internet is randomly registered, unorganized or unrelated, which has already exceeded the amount of information an individual can digest, making it impossible to have meaning as a collection of valuable information. In order to overcome this situation, high-level software is required to understand the meaning of information, analyze it, and provide an appropriate level of information when necessary. The related technologies used in this software will mark important milestones in the development of Internet-related intelligent agent software by automating the three phases of the recognition, analysis and provision of this ambiguous information.

Claims (1)

Intelligent Information Retrieval and Analysis Agent