KR20200129782A - Searching service method using crawling - Google Patents

Abstract

Disclosed is a search agent service method using crawling. According to one aspect of the present invention, the search agent service method, which is performed in a search service device, comprises the following steps of: periodically crawling and collecting sales product pages handled in a member shopping mall at a preset point of time; deriving meta-information from the collected sales product pages in a preset manner and storing the same in an external database provided by itself; and when a search request including a search keyword is received from the member shopping mall, providing a search result according to the search keyword in the external database to the member shopping mall.

Description

Searching service method using crawling}

The present invention relates to a search agency service method using crawling.

In recent years, online shopping malls are active, and online product sellers operate their own online shopping malls or use large open markets such as Coupang, 11th Street, and Auction. In the case of open markets, the search function, etc., is well performed by a large number of developers because it is operated by a large company, whereas in the case of a shopping mall operated by a small company, various functions such as search function are somewhat insufficient due to development by outsourcing.

Recently, compared to online shopping using general personal PCs, online shopping using mobiles such as smartphones is becoming more dominant. In the case of mobile, it is difficult to show a large number of products due to limitations due to small screens, so a search-oriented service Has been provided. In addition, there are many companies that sell thousands or tens of thousands of products rather than a small number of companies. In a mobile environment, it is difficult to display a large number of items on one screen, so the search function is an important issue.

In the case of a self-contained shopping mall operated by a small company, there is a problem in that the product desired by the consumer is not accurately searched due to an insufficient search function, causing inconvenience to the consumer and thus not leading to purchase.

If, in order to reinforce the search function of the own shopping mall, the purchase of a search engine, development cost for applying the search engine to the shopping mall, and maintenance costs are consumed, and management is difficult, it is often not practical.

Republic of Korea Patent Registration No. 10-1745998 (Registration Date June 5, 2017) Linked search display device and method for online shopping mall product information

Accordingly, the present invention has been devised to solve the above-described problem, and is to provide a search agency service method using crawling that provides an enhanced search function to a shopping mall as a service type.

Other objects of the present invention will become more apparent through preferred embodiments described below.

According to an aspect of the present invention, there is provided a method for providing a search agency service performed by a search service device, comprising: periodically crawling and collecting sales form pages handled by a member shopping mall at a preset time; Deriving meta information from the collected sales product pages in a preset manner and storing it in an external database provided by itself; And when a search request including a search keyword is received from the member shopping mall, providing a search result according to the search keyword from the external database to the member shopping mall. A search agency service method using crawling is provided.

Here, when the crawling time point for the member shopping mall exceeds a preset threshold time point, further comprising performing accuracy verification on the search result, wherein the accuracy verification is performed by a predetermined number of contents according to the search result. It can be performed by attempting a direct connection and checking whether it is successful.

In addition, when the accuracy verification is less than the threshold value, the step of performing partial crawling of the member shopping mall may be further included.

In addition, when the accuracy verification is less than a second threshold value that is less than the threshold value, a full crawl is performed for the member shopping mall, and while the full crawl is performed, a search using a search engine provided in the member shopping mall itself is performed. You can do it.

In addition, if a page with an error is found by the above accuracy verification, the content of the error page is deleted and the search result is provided, and then all pages according to the search result are checked, so that all further error pages are removed. Information can be transmitted to the member shopping mall.

According to the present invention, it is possible to provide a reinforced search function in a shopping mall without installing a new engine and changing the DB by installing a new engine in the shopping mall itself by constructing a DB by crawling and performing a search externally.

1 is a block diagram schematically showing an entire system for a search service for a shopping mall according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a search service apparatus according to an embodiment of the present invention.
3 is a flow chart illustrating a search service process according to an embodiment of the present invention.
4 is an exemplary view showing a search user interface screen provided by a member shopping mall server according to an embodiment of the present invention.
5 is a flowchart illustrating a search agency process for performing accuracy verification in response to a crawl time point according to an embodiment of the present invention.
6 is a flowchart illustrating a processing procedure corresponding to an accuracy value by verifying accuracy according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. Should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component. For example, terms such as a first threshold value and a second threshold value to be described later may be pre-designated as threshold values that are substantially different or some of the same values, but when expressed in the same word as the threshold value, there is a confusion. There is room, so for the convenience of classification, terms such as first and second will be added together.

The terms used in the present specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

In addition, components of the embodiments described with reference to each drawing are not limited to the corresponding embodiments, and may be implemented to be included in other embodiments within the scope of maintaining the technical spirit of the present invention. Even if the description is omitted, it is natural that a plurality of embodiments may be implemented again as a unified embodiment.

In addition, in the description with reference to the accompanying drawings, the same or related reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the present invention, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

1 is a block diagram schematically showing an entire system for a search service for a shopping mall according to an embodiment of the present invention, and FIG. 2 is a block diagram showing a configuration of a search service apparatus according to an embodiment of the present invention. to be.

Referring to FIG. 1, the entire system according to this embodiment searches through member shopping mall servers 10-1, ..., 10-n, hereinafter referred to as 10) and member shopping mall servers 10 that operate a shopping mall. It includes a search service device 30 providing an agency service and an external DB 40 operated by the search service device 30.

The member shopping mall server 10 is a device for providing an online shopping mall to consumers, and the search service device 30 provides a search function to each member shopping mall registered as a member in a service format.

That is, the member shopping mall may have its own search function, but the search function uses the external search service device 30. In other words, a consumer who accesses the member shopping mall performs a search in the corresponding shopping mall, but the actual search is performed by the search service device 30, and the consumer can immediately check the search result in the shopping mall that has already been accessed.

Referring to FIG. 2 showing the configuration of the search service device, the search service device 30 includes a communication unit 210, a page collection unit 220, a meta information generation unit 230, a search unit 240, and a control unit 250. ).

The communication unit 210 is for communicating with the member shopping mall server 10 using a communication network such as an Internet network, and a more detailed description thereof will be omitted since communication using a communication network will be apparent to those skilled in the art.

The page collection unit 220 collects pages of products for sale that are handled by the member shopping mall. That is, the user page is collected according to each sale item provided in the member shopping mall. The user page is in the form of a webpage in the case of a web method, and a view page in the case of an application. It will be provided in the form of (view page). That is, the page collection unit 220 crawls and collects a sales product page, which is a user page for each sale product according to a web method or an app method, through a communication network.

In general, web crawling refers to a technology that collects web documents (webpages, etc.) that are distributed and stored in a myriad of computers and includes them as an index of the search target. However, crawling that collects documents (webpages, etc.) through communication networks is currently used by search engines. Also, since it is used in various fields, it will be apparent to those skilled in the art, so a more detailed description will be omitted.

The meta information generation unit 230 derives meta information from the collected sales product pages in a preset manner. For example, meta information such as category information, page title information, product serial number, and function information included in the content is derived. The search service apparatus may also provide an administrator interface in which a method of deriving meta information can be set by an administrator according to a situation.

The derived meta-information is stored in the external DB 40 and used for subsequent search.

The search unit 240 performs a search according to the search keyword in the external DB 40, and searches according to the search keyword in the external DB 40 to provide search results according to the search request including the search keyword from the member shopping mall. Perform. The results searched by the search unit 240 are provided to the member shopping mall server 10 that requested the search. Accordingly, the member shopping mall server 10 can provide a highly reliable search service to consumers by using an external search service device 30 with a more enhanced search function, so that sales increase due to product sales can be expected. . In other words, the search service device 30 according to the present embodiment builds a new DB based on the meta-information crawled in the external DB 40 provided separately from the DB operated by the member shopping mall server 10, It provides a search service using the newly built DB.

The control unit 250 performs overall control of the operation of the functions of each component. According to an example, the search history is stored in the external DB 40 (or a separate storage medium may be used), and a member using the search history You can set the collection point for each item handled by the shopping mall. That is, pages may be collected at the same time for all the items sold in the member shopping mall, but according to another example, the collection time of pages for each category or, in detail, for each product may be different. As a specific example, a page for sale items according to category A is crawled (collected) every 24 hours, and a page for items sold according to category B is crawled every 12 hours. For example, collection of sales product pages according to a category in which the number of searches is relatively large may be performed more often.

Accordingly, the control unit 250 may determine the collection time of each category of sales product pages by using the history of the searched category.

In addition, the control unit 250 may perform verification of the search result performed by the search unit 240, and the processing method and procedure according thereto will be described in detail later with reference to related drawings (Figs. 5 to 6). do.

3 is a flowchart illustrating a search service process according to an embodiment of the present invention, and FIG. 4 is an exemplary view showing a search user interface screen provided by a member shopping mall server according to an embodiment of the present invention.

Referring to FIG. 3, the search service device 30 collects a sales article page of the member shopping mall server 10 (S310), and derives and stores meta information from the page according to a preset criterion (S320).

Thereafter, when the member shopping mall server 10 performs a search request including a search keyword input from the user (S330), the search service device 30 performs a search using the search keyword based on the stored meta information (S340). . The search result information is provided to the member shopping mall server 10 (S350). The search result information is displayed in the user page provided by the member shopping mall server 10 (S360). In this case, the search service device 30 may provide the search result information in the form of a page or the member shopping mall server 10 It can also be provided as information using the following API. In other words, the search service device 30 may provide the search result as the page itself, or the member shopping mall server 10 may provide the information as the page.

Referring to FIG. 4 illustrating a user interface screen provided by the member shopping mall server 10 to the accessed user, when a search is requested using the search window 410 displayed on one side of the screen, a search keyword (climbing boots) ) Is provided to the search service device 30, and a search result by the search service device 30 is displayed at 420.

Hereinafter, a detailed description will be given of a search agency service process for performing accuracy verification in the search service device 30.

5 is a flowchart illustrating a search agency process for performing accuracy verification in response to a crawl time point according to an embodiment of the present invention.

Referring to FIG. 5, the search service device 30 receives a search proxy request (S510). That is, a request for a search agency service including a search keyword is received from an arbitrary member shopping mall server 10.

Then, the search service device 30 identifies a corresponding member shopping mall, and checks a time point at which the identified member shopping mall was last crawled (S520). For example, in the case of performing a full crawl for each member shopping mall basically every 24 hours (the execution time may be different), the time when the last crawl of the member shopping mall was performed (may be the start or end time) To confirm.

It is checked whether the crawl time exceeds a preset threshold (eg, after 12 hours) (S530). If the crawl time has passed too long, a difference may occur between the information stored in the DB of the member shopping mall server 10 and the information crawled and stored in the external DB 40 of the search service device 30, so check the crawl time. Is to do.

If the crawling time point has not exceeded the critical time point, a general search agency process is performed and the search result is provided to the member shopping mall server 10 that requested the search agency service (S540).

On the contrary, if the crawl time exceeds the critical time point, there may be a difference in information between the DB of the member shopping mall server 10 and the external DB 40, so that the accuracy verification is performed after performing a search using the external DB 40. It performs (S560).

The accuracy verification can be performed in various ways. According to an example, the connection success rate can be calculated as an accuracy verification value by directly accessing each page of the search result. For example, if a total of 100 webpages are searched as a search result for the search keyword [climbing clothes], and if 98 of each 100 webpages are directly accessed, the accuracy value can be calculated as 98%. .

According to another example, if the number of search results is large, trying to access all of them may be inefficient. Therefore, randomly, in the latest order, or in the oldest (for example, 10) are extracted and checked, and the results are viewed as a whole. A method of inferring as a result may be used.

Although not shown in the drawing, according to an example, if the accuracy verification result is less than a preset threshold (eg, 50%), the search agency is stopped and the member shopping mall server uses a search service using its own search engine. It can also be processed to do.

In contrast, when the accuracy verification is successful (for example, when the accuracy value is greater than or equal to a threshold value), the search agency service may be normally processed.

6 is a flowchart illustrating a processing procedure corresponding to an accuracy value by verifying accuracy according to an embodiment of the present invention.

Referring to FIG. 6, the search service device 30 performs a search corresponding to a search keyword requested as a search agent (of course, various search conditions such as gender, age group, production date, price order, etc. may be further added), A certain number of the search results is extracted (S610). As described above, all search results may be checked for accuracy verification, but accuracy verification may be performed by extracting several representatives. Here, among the search results, the extraction method and number of representative products for accuracy verification may vary according to the number of search results. For example, various extraction methods may be set, such as randomly extracting 10% of the number of search results, or extracting 5 items in the latest order regardless of the number of search results.

All of the extracted representative products are directly connected to each other (that is, access to the member shopping mall server DB through a communication network), and the accuracy value is calculated (S6230).

As a result of the calculation, it is checked whether the accuracy value is less than a first threshold value (eg, 90%, etc.) (S630). If an accuracy value equal to or greater than the first threshold is calculated, the search result (the checked error page may be removed) is provided to the member shopping mall server 10 (S640).

In contrast, if the accuracy value is less than the first threshold, it is further checked whether it is less than the second threshold (eg, 80%, etc.) (S650). If it is less than the second threshold, it is to prevent such a case because it may mean that the accuracy is too low to provide a proper search agency service.

When the accuracy value is greater than or equal to the second threshold and less than the first threshold, partial crawling corresponding to the search keyword is performed (S660). For example, before performing a periodic full crawl for the member shopping mall server 10, the external DB 40 is collected by crawling and collecting web pages by the member shopping mall server 10 corresponding to the current search keyword. It is to update. And, according to an example, in this case, the search result according to the search agency service is normally provided to the member shopping mall server. And, according to an example, if a page in which an error exists is found through accuracy verification, the content of the error page is deleted and the search result is provided to the member shopping mall server 10, and then all other pages according to the search result are further Related information (or new search results) may be transmitted to the member shopping mall so that all error pages found after checking are removed.

On the contrary, when the accuracy value is less than the second threshold, the next crawl time point is not reached, but the entire crawl is immediately performed (S670). In this case, while the entire crawl is being performed, a search can be performed using a search engine provided in the member shopping mall itself. That is, since the contents of the external DB 40 may not be accurate, the member shopping mall server performs a direct search using a direct search keyword, and performs a full crawl in the meantime.

Although the above has been described with reference to preferred embodiments of the present invention, those of ordinary skill in the relevant technical field can variously modify the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. It will be appreciated that it can be modified and changed.

10: Member shopping mall server
30: search service device
40: external DB

Claims (5)

In the search agency service providing method performed by a search service device,
Periodically crawling and collecting sales form pages handled by the member shopping mall at a preset time;
Deriving meta information from the collected sales product pages in a preset manner and storing it in an external database provided by itself; And
And when a search request including a search keyword is received from the member shopping mall, providing a search result according to the search keyword from the external database to the member shopping mall. The method according to claim 1,
When the crawling time point for the member shopping mall exceeds a preset threshold time point, further comprising: performing accuracy verification on the search result,
The accuracy verification is performed by confirming success by attempting direct access as many as a preset number of contents according to the search result, and the search agency service method using crawling. The method according to claim 2,
If the accuracy verification is less than the threshold value, further comprising the step of performing a partial crawl for the member shopping mall, search agency service method using crawling. The method of claim 3,
When the accuracy verification is less than a second threshold value that is less than the threshold value, a full crawl is performed on the member shopping mall, and a search using a search engine provided in the member shopping mall itself is performed while the full crawl is performed. , Search agency service method using crawling. The method of claim 3,
If a page with errors is found by the above accuracy verification, after deleting the contents of the error page and providing the search result,
A search agency service method using crawling, which checks all pages according to a search result and transmits related information to the member shopping mall so that all error pages found further are removed.

Images