KR102349624B1 - System and Method for Crawling News - Google Patents

Abstract

According to an aspect of the present invention, a news crawling system capable of exactly providing only news related to a search term included in a search query comprises: a URL collection unit for collecting URLs of news articles including a search term, when a search query including the search term is received from a user terminal; a news article collecting unit for collecting titles and texts of the news articles corresponding to the URLs for each news article; a first filtering unit for primary filtering of duplicate news among the news articles based on the collected titles of the news articles; and a second filtering unit for selecting related news articles by secondarily filtering the first filtered news articles based on the titles and texts of the firstly filtered news articles.

Description

News crawling system and news crawling method {System and Method for Crawling News}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a search system, and more particularly to news crawling.

News functions are provided online through portal services such as Naver or Google, or through newspapers. In order to provide news online, news article crawling is performed. Most news article crawling systems focus only on collecting a lot of news without a separate process for selecting accurate news, so the implementation is relatively difficult. Although simple, there is a problem that the accuracy is considerably lowered.

As a result, a large amount of duplicate news or articles other than the ones desired by the customer are often detected.

For example, if you enter 'POSCO ICT' as a search word to read the news of 'POSCO ICT' on Naver, as shown in FIG. , it can be seen that duplicate news with the same content is searched in large quantities.

Accordingly, in order to read only the news of the company that the user wants, there is a problem in that inconvenience increases because the user has to visually filter numerous duplicate news or news of an uninterested company in the search results.

Also, even if it is possible to remove duplicate news from within the search results, most of them can only remove the same news with the exact same title as duplicate news. There is a limitation in that it cannot be filtered as duplicate news.

SUMMARY OF THE INVENTION The present invention is to solve the above problems, and has a technical feature to provide a news crawling system and a news crawling method that can accurately provide only news related to a search term included in a search query.

Another technical feature of the present invention is to provide a news crawling system and a news crawling method that can accurately filter news of the same title as well as news of substantially the same title even if the titles do not completely match.

Another technical aspect of the present invention is to provide a news crawling system and a news crawling method capable of selecting news provided through a portal service based on the URLs of the news and performing duplicate news filtering on the selected news. characterized.

According to an aspect of the present invention, there is provided a news crawling system for achieving the above object, comprising: a URL collecting unit that collects URLs of news articles including a search term when a search query including a search term is received from a user terminal; a news article collecting unit that collects titles and texts of news articles corresponding to the URLs for each news article; a first filtering unit for primary filtering of duplicate news among the news articles based on the titles of the collected news articles; and a second filtering unit configured to select related news articles by secondarily filtering the firstly filtered news articles based on the titles and texts of the firstly filtered news articles.

According to an aspect of the present invention, there is provided a news crawling method for achieving the above object, comprising: when a search query including a search term is received from a user terminal, collecting URLs of news including the search term; selecting URLs including link information on a web page of a predetermined portal service from among the collected URLs; filtering duplicate news among the news articles based on the titles of the news articles corresponding to the selected URLs; selecting related news articles by secondarily filtering the firstly filtered news articles based on the titles and texts of the firstly filtered news articles; and providing the selected related news article as a news search result for the search query to the user terminal.

As described above, according to the present invention, duplicate news is primarily filtered based on the title of the news article, and related news articles are selected by secondary filtering the primary filtered result based on the title and body of the news article. By doing so, there is an effect that only news articles related to the search terms included in the search query can be accurately provided without duplicate news.

In addition, according to the present invention, duplicate news filtering and related news article selection are performed based on the syllables included in the title of the news article or the characters continuously expressed in the title and body of the news article in the text, so that the exact same title Of course, it has the effect of accurately filtering news articles with substantially the same title even if the titles do not completely match.

In addition, according to the present invention, since duplicate news filtering can be performed for only news provided through the portal service based on the URL of the news, the time required for filtering duplicate news can be shortened, as well as the corresponding news. There is no need to write a separate program code for filtering duplicate news for each website of the newspaper you created, and there is no need to modify the program code even if the structure of each newspaper website is changed. It has the effect of being able to prevent it in advance.

In addition, according to the present invention, the second filtering unit that performs duplicate news filtering is implemented as an AI model, and the AI model can be trained by extracting characters continuously expressed in a search word based on feedback from a user who reads the news. It has the effect of improving the accuracy of filtering duplicate news.

1 is a diagram illustrating an example of a result in which duplicate news or news not related to a specific company is included in a news search result.
2A is a block diagram schematically showing a network configuration to which a news crawling system according to an embodiment of the present invention is applied.
2B is a block diagram showing the configuration of a news crawling system according to an embodiment of the present invention.
3 is a view showing news articles including link information on a web page of a portal site among web search results.
4 is a view showing an example in which duplicate news is removed by the first filtering unit among news articles.
5 is a view showing an example in which related news is selected by the second filtering unit from among the firstly filtered news articles.
6 is a diagram illustrating an example of a storage unit in which related news articles are stored.
7 is a flowchart illustrating a news crawling method according to an embodiment of the present invention.

Like reference numerals refer to substantially identical elements throughout. In the following description, a detailed description of configurations and functions known in the art and cases not related to the core configuration of the present invention may be omitted. The meaning of the terms described in this specification should be understood as follows.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims.

The shapes, sizes, proportions, angles, numbers, etc. disclosed in the drawings for explaining the embodiments of the present invention are illustrative and the present invention is not limited to the illustrated matters. Like reference numerals refer to like elements throughout. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

When 'including', 'having', 'consisting', etc. mentioned in this specification are used, other parts may be added unless 'only' is used. When a component is expressed in the singular, cases including the plural are included unless otherwise explicitly stated.

In interpreting the components, it is construed as including an error range even if there is no separate explicit description.

In the case of a description of the positional relationship, for example, when the positional relationship of two parts is described as 'on', 'on', 'on', 'beside', etc., 'right' Alternatively, one or more other parts may be positioned between two parts unless 'directly' is used.

In the case of a description of a temporal relationship, for example, 'immediately' or 'directly' when a temporal relationship is described with 'after', 'following', 'after', 'before', etc. It may include cases that are not continuous unless this is used.

Although the first, second, etc. are used to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the spirit of the present invention.

The term “at least one” should be understood to include all possible combinations from one or more related items. For example, the meaning of “at least one of the first, second, and third items” means that each of the first, second, or third items as well as two of the first, second and third items are It may mean a combination of all items that can be presented from more than one.

Each feature of the various embodiments of the present invention may be partially or wholly combined or combined with each other, technically various interlocking and driving are possible, and each of the embodiments may be independently implemented with respect to each other or implemented together in a related relationship. may be

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

2A is a block diagram schematically showing a network configuration to which a news crawling system according to an embodiment of the present invention is applied. As shown in FIG. 2 , when a search query including a search term is received from the user terminal 205 , the news crawling system 200 according to an embodiment of the present invention displays related news articles related to the search term as a news search result. provided with In an embodiment, the search term may be a company name that the user wants to search for. However, the present invention is not limited thereto, and the search word may include various types such as the name of a person, a place name, or the name of a thing that the user wants to search.

A configuration of a news crawling system according to an embodiment of the present invention will be described in detail with reference to FIG. 2B. 2B is a block diagram showing the configuration of a news crawling system according to an embodiment of the present invention. As shown in FIG. 2B , the news crawling system 200 includes a URL collecting unit 210 , a news article collecting unit 220 , a first filtering unit 230 , a second filtering unit 240 , and a search result list. study 250 . In addition, the news crawling system 200 according to the present invention may further include a related news article refiner 260 .

When a search query including a search word is received from the user terminal 205 , the URL collection unit 210 collects Uniform Resource Locators (URLs) of news including the search word.

Specifically, when a search query is received from the user terminal 205 , the URL collection unit 210 requests a web search result including the corresponding search word as a web by using a search word included in the search query, Receive web search results from In an embodiment, the URL collection unit 210 may receive a web search result in a Hyper Text Markup Language (HTML) structure from the web.

According to this embodiment, the URL collection unit 210 converts the HTML-structured web search result received from the web into an XML format. The reason that the URL collection unit 210 according to the present invention converts the HTML-structured web search result into XML format is that the HTML-structured web search result is difficult to process, and the title position and body position of news within each search result. because it is difficult to specify

Accordingly, the URL collection unit 210 according to the present invention converts the HTML-structured web search result into XML format, so that the title position and body position of the news article can be easily specified from each search result.

In one embodiment, the URL collection unit 210 filters the URLs of the web search results converted into XML format, and includes link information on a web page of a predetermined portal service among the URLs of each web search result. Only URLs can be selected.

In the present invention, the reason that the URL collection unit 210 selects only URLs containing link information to a web page of a portal service from among the URLs of the web search results converted into XML format is that the newspaper company providing each news article This is because, since the structure of each website is different, if you want to use news articles provided by a newspaper as it is, you have to configure the code differently for each newspaper, which can be cumbersome. Another reason is that the code must be manually modified every time a newspaper website is renewed, and when the structure of a newspaper website is changed, it is difficult to predict the timing and contents of the structural change in advance. Failure to do so may result in errors.

On the other hand, since news articles provided through the portal service are provided through the website of the portal service having the same HTML structure, it is possible to prevent problems caused by using the newspaper's web page as described above. be able to

For example, as shown in FIG. 3 , the URL collection unit 210 according to the present invention selects news articles including link information 300 for Naver News Home from among the web search results, and selects the corresponding news. URL information on articles is provided to the news article collecting unit 220 .

The news article collection unit 220 collects titles and texts of news articles corresponding to URLs of each news article transmitted from the URL collection unit 210 from the web.

In one embodiment, the news article collection unit 220 may transmit the URLs of each news article as a search query to the web, and receive the news articles in the HTML structure as a result of the search. According to this embodiment, the news article collecting unit 220 converts the HTML-structured news articles received from the web into an XML format in the same way as the URL collecting unit 210 , and receives each news article from the XML format news articles. You can obtain the title and body of

The first filtering unit 230 first filters duplicate news among news articles based on the titles of each news article collected by the news article collecting unit 220 .

To this end, the first filtering unit 230 includes a title analyzing unit 232 and a first filter 234 as shown in FIG. 2B .

The title analyzing unit 232 analyzes the titles of each news article collected by the news article collecting unit 220 in units of at least one of a noun, a morpheme, and a syllable. The title analyzer 232 may obtain at least one of a noun, a morpheme, and a syllable from each news article by analyzing the title of each news article using a text mining technique.

In an embodiment, the title analyzer 232 may classify the title of each news article into at least one set of nouns, morphemes, and syllables using Hannanum of the Konlpy library, which is an open source Korean morpheme analyzer.

For example, if the headline of the news is "We contribute to customers' business by providing accurate corporate debt repayment ability by utilizing various companies' financial/non-financial data and AI/Big Data analysis technology. Improving corporate profitability by minimizing commerce risk If there is a news article titled "I want to become a companion of , "Analytical Technology", "Utilization", "Accurate", "Company", "Debt Repayment Capacity", "Provide", "Customer", "Business", "Commerce", "Minimization", "Enterprise", "Profitability" Nouns such as ", "enhance", and "companion" can be obtained.

As another example, the title analysis unit 232 analyzes the title as described above, such as "we", "is", "various", "corporate", "of", "financial", "/", "non", "Financial", "Data", "And", "AI", "/", "BigData", "Analytics", "A", "Utilization", "Ha", "Uh", "Accurate", " H", "b", "company", "of", "debt repayment capacity", "provided", "b", "through", "uh", "customer", "of", "business", " "on", "queer", "uh", "ha", "b", ".", "commerce", "risk", "minimize", "a", "through", "b", "enterprise" "," "of", "profitability", "improvement", "of", "companion", "to be", "to be", "to be", "to be", "to be", and "." It is possible to obtain morphemes such as

As another example, the title analysis unit 232 analyzes the titles as described above, such as "Wu", "Li", "A", "Da", "Yang", "Han", "Ki", "Up" , "of", "ash", "mu", "/", "b", "ash", "mu", "d", "a", "t", "a", "with", " A", "I", "/", "B", "I", "g", "D", "a", "t", "a", "Minute", "Seok", "Ki" , "Sake", "B", "Bow", "Dragon", "Ha", "W", "Jeong", "Hak", "Han", "Ki", "Up", "Of", " Chae", "Mu", "Sang", "Hwan", "Neung", "Power", "Je", "Gong", "B", "Tong", "Hae", "Go", "Gaek" , "of", "b", "z", "ni", "su", "e", "ki", "fe", "sum", "ni", "da", ".", "sang" ", "geo", "rae", "r", "i", "s", "k", "choi", "so", "wha", "a", "tong", "han", "Ki", "Up", "Ri", "Su", "Ri", "Sung", "Hyang", "Sang", "Ri", "Dong", "Ban", "Child", "Ga" It is possible to obtain syllables such as "," "do", "go", "ja", "hap", "ni", "da", ".", and the like.

The first filter 234 calculates a degree of similarity between the titles of news articles based on at least one of nouns, morphemes, and syllables analyzed by the title analyzer 232 , and news articles corresponding to titles having a similarity greater than or equal to a threshold. They are judged as duplicate news and filtered.

In an embodiment, the title analysis unit 232 calculates the similarity between the titles of the news articles based on syllables among nouns, morphemes, and syllables, and determines news articles corresponding to titles having a similarity greater than or equal to a threshold as duplicate news. can be filtered. The reason that the title analyzer 232 according to the present invention determines the degree of similarity between titles using syllables among nouns, morphemes, and syllables is that when determining the degree of similarity using a noun or morpheme, the same title is filtered even though it is the same title. This is because the non-detection rate may be relatively high.

The first filter 234 calculates a similarity index between the titles of each news based on at least one of a noun, a morpheme, and a syllable obtained from each title by the title analysis unit 232 , and the calculated similarity index is a threshold value News articles with more than one title may be determined as duplicate news. The first filter 234 may perform primary filtering on the news articles by selecting only one news article from among the news articles determined as duplicate news.

In an embodiment, the first filter 234 calculates a Jaccard Similarity index based on the syllables obtained from the title by the title analysis unit 232, and titles for which the calculated Jaccard Similarity index is greater than or equal to a threshold It is possible to determine the news article corresponding to the duplicate news. The Jacquard Similarity Index has a value between 0 and 1, has a value of 1 if two sets to be compared are identical, and has a value of 0 if there is no element in common between the two sets.

A method of calculating the jacquard similarity index can be expressed by Equation 1 below.

는 A제목과 B제목에 포함되어 있는 음절의 개수의 합을 나타내며,

는 A제목과 B제목에 공통으로 포함되어 있는 음절의 개수를 나타내며,

는 A제목에 포함되어 있는 음절의 개수를 나타내고,

는 B제목에 포함되어 있는 음절의 개수를 나타낸다.In Equation 1, J(A,B) represents the jacquard similarity index for the A title and the B title, which are to be compared,

represents the sum of the number of syllables in the A and B titles,

represents the number of syllables commonly included in the A and B titles,

represents the number of syllables included in the A title,

indicates the number of syllables included in the B title.

In the above-described embodiment, the first filter 234 may determine a news search result corresponding to a title having a jacquard similarity index greater than 0.3 obtained based on the syllable of each title as duplicate news.

Hereinafter, a method in which the first filtering unit 230 according to the present invention determines whether three news articles are duplicated based on syllables included in the titles of the three news articles will be described as an example. At this time, the title of news article A is “POSCO, Korea’s first establishment of a system for fairing terms and conditions using AI”, and the title of news article B is “POSCO filters unfair terms and conditions using AI. The first in Korea”, and the title of news article C Assume that "POSCO ICT last year's operating profit was 48.1 billion won, 14.2% IP from the previous year".

First, the title analysis unit 232 receives from the news article A, "Po", "S", "Nose", ",", "Guk", "My", "Choi", "Second", "A", " I", "Bow", "Dragon", "Han", "Yak", "Kwan", "Gong", "Jeong", "Hwa", "Si", "Shu", "Temp", "Gu" , and the syllable "axis". In addition, the title analysis unit 232 receives from the news article B, "Po", "S", "Nose", ",", "A", "I", "Bow", "Dragon", "Sun", " Fire", "Gong", "Jeong", "Yak", "Kwan", "Girl", "Ru", "Nan", "Da", "Guk", "My", "Choi", and "Second" " gets the syllable. In addition, the title analysis unit 232 receives the C news article from "Po", "S", "Nose", "I", "C", "T", "G", "I", "Hae", " Zero", "Up", "Revenue", "4", "8", "1", "Billion", "Before", "Year", "Rain", "1", "4", "." , "2", "%", "U", and "P" are obtained.

The first filter 234 calculates a first jacquard similarity index by using Equation 1 based on the syllables obtained from the news article A and the syllables obtained from the news article B. The first filter 234 is the number of syllables in the A news article, the number of syllables in the B news article, the number of syllables commonly included in the A news article and the B news article, and the syllables between the A news article and the B news article in Equation 1 The first jacquard similarity index of 0.567 is calculated by substituting the sum of the numbers into Equation 1.

Also, the first filter 234 calculates a second jacquard similarity index using Equation 1 based on the syllables obtained from the news article A and the syllables obtained from the news article C. The first filter 234 in Equation 1 shows the number of syllables in the news article A, the number of syllables in the news article C, the number of syllables commonly included in the news article A and the news article C, and the syllables between the news article A and the news article C. A second jacquard similarity index of 0.139 is calculated by substituting the sum of the numbers into Equation 1.

Since the first filter 234 has a first jacquard similarity index greater than 0.3, it is determined that news article A and news article B are duplicate news, and since the second jacquard similarity index is less than 0.3, news article A and news article C are It is judged not to be duplicate news. Accordingly, the first filter 234 may remove any one of the A news article and the B news article so that only one of the two news articles is included in the news search result.

As described above, since the jacquard similarity index is calculated based on the noun, morpheme, and syllable of the news article title according to the present invention, and news articles having the jacquard similarity index greater than or equal to the threshold can be determined as duplicate news, the title is completely identical Of course, even if the titles do not completely match, substantially the same news article can be removed as duplicate news, thereby improving the duplicate news removal rate.

An example in which duplicate news is removed by the first filtering unit 230 among the news articles collected by the news article collecting unit 220 is illustrated in FIG. 4 . Although there were 10 duplicate news of 17 news articles in FIG. 4A, as can be seen from FIG. 4B, all 10 duplicate news are removed by the first filtering unit 230 so that duplicate news does not exist in the news articles. it can be seen that

Referring back to FIG. 2B , the second filtering unit 240 performs secondary filtering of the firstly filtered news articles on the basis of the titles and texts of the firstly filtered news articles by the first filtering unit 230 to obtain related news. decide the article. The reason for additionally performing the second filtering operation using the second filtering unit 240 after the first filtering by the first filtering unit 230 in the present invention is that the first filtering unit 230 in news articles ) has been removed, but news articles unrelated to the search term included in the search query may still be included.

For example, it can be seen that, although duplicate news is removed by the first filtering unit 230 in FIG. 4B, a number of news articles related to “POSCO” are included, not news articles related to the company name “POSCO ICT” included in the search query. have.

Accordingly, the present invention removes news articles irrelevant to the search word included in the search query from among the firstly filtered news articles by additionally filtering the firstly filtered news articles using the second filtering unit 240 .

To this end, the second filtering unit 240 may include a second filter 242 , a third filter 244 , and a related news article selection unit 246 .

The second filter 242 counts the first frequency of appearance of the target phrase in which the search word and the predetermined character are continuously displayed in the firstly filtered titles of the news articles. The second filter 242 selects a news article corresponding to a title in which the counted first appearance frequency is equal to or greater than the first reference value as the first related news article.

The third filter 244 counts the second frequency of occurrence of the target phrase in the body of the firstly filtered news articles. The third filter 244 selects news articles corresponding to the main text having the counted second appearance frequency equal to or greater than the second reference value as second related news articles. In an embodiment, the second reference value may be set to have a larger value than the first reference value. This is because even if the target phrase is the same, the case where the target phrase is included in the title is much higher than the case where the target phrase is included in the body. For example, when the first reference value is set to 1, the second reference value may be set to 5.

In the above-described embodiment, the predetermined characters included in the target phrase are continuously displayed in the corresponding search word within the predetermined texts, and at least one of key searches, key words, and special characters whose frequency of appearance is equal to or greater than the reference value may include

For example, a key search is a search term that is continuously displayed within a specific text within a corresponding search term. It may include at least one of ", "and", and "and". In addition, the key word is a word that is continuously displayed on a corresponding search word in specific texts or appears as a pair with the corresponding search word, and may include at least one of "group", "family", and "do". In addition, special characters are symbols that are continuously displayed in the corresponding search word within specific texts, such as ",", """, "·", "(", "'", "]", "-", ".. ." may be included.

The related news article selection unit 246 determines the first related news article selected by the first filter 242 and the second related news article selected by the second filter 244 as related news articles. That is, when the related news article selection unit 246 is determined to be a related news article by either the first filter 242 or the second filter 244 , the related news article is finally determined as a related news article.

For example, when the company name included in the search query is “POSCO ICT”, the news articles first filtered by the first filtering unit 230 are as shown in FIG. 5A, 11 news articles shown in FIG. 5A Among them, it can be seen that there are only 4 news articles related to POSCO ICT. When the news articles shown in FIG. 5A are secondarily filtered using the second filtering unit 240, as shown in FIG. 5B, it can be seen that only 4 related news articles of POSCO ICT are displayed among 11 news articles. .

As described above, according to the present invention, since the second filtering unit 240 can select only the news articles related to the company from among the firstly filtered news articles, the accuracy of the search results can be improved, and thus the search results can be searched accordingly. Users who read the results do not need to visually select news related to the company they want to read from among the news search results, so user convenience is also increased.

Referring back to FIG. 2B , the search result providing unit 250 provides a news search result composed of related news articles determined by the second filtering unit 240 to the user terminal 205 as a search result for the search query. .

To this end, the search result providing unit 250 includes a storage 252 and a search result generating unit 254 .

The storage unit 252 stores the related news articles determined by the second filtering unit 240 . In one embodiment, the storage unit 252 includes a plurality of queues 600 for storing the relevant news article for each relevant news article as shown in FIG. 6 , and is The determined related news articles may be separated into the title, date, text, original link, and newspaper company information that generated the related news article of each relevant news article, and stored separately in a plurality of queues 600 .

The search result generating unit 254 generates a search result using the related news articles stored in the storage unit 252 according to a predetermined format, and provides the generated search result to the user terminal. In one embodiment, the search result generating unit 254 merges the title, date, and original link of the related news article to generate a search result and provide it to the user terminal 205 , and the corresponding news article by the user terminal 205 When the title of the news is clicked, the original text of the news article can be provided by using the original link of the news article.

In an embodiment, the search result generator 254 may output a search result including a related news article as a file in a predetermined format. For example, the search result generator 254 may output a search result including a related news article as an Excel file, a text file, and an HTML file.

In the above-described embodiment, the second filtering unit 240 may be implemented as an artificial intelligence (AI) model, and the news crawling system 200 according to the present invention provides training for training the second filtering unit 240 . A unit 260 may be further included.

The training unit 260 receives feedback from the user terminal 205 on the satisfaction with the news search results provided to the user terminal 205, and among the characters continuously displayed for the search word in the related news article with high feedback satisfaction. Characters having an appearance frequency greater than or equal to the reference value may be extracted again, and the second filtering unit 240 may be trained using the extracted characters. At this time, the training unit 260 newly adds the key research, key words, and special characters continuously displayed to the corresponding search word through the training of the second filtering unit 240, or sets the values of the first and second reference values. can be adjusted

In one embodiment, the training unit 260 may be configured to configure the training unit 260 to have a frequency of occurrence greater than or equal to a reference value among characters continuously displayed in a corresponding search word in related news articles for which “Like” is selected by the user terminal 205 from among the news search results. Characters may be extracted, and the second filtering unit 240 may be trained using the extracted characters.

In addition, the news crawling system 200 according to the present invention may further include a related news article refiner 270 for correcting time information at which a related news article is generated in a predetermined standard time format. In the present invention, the reason that the related news article generation time information is modified in a predetermined standard format through the related news article refining unit 270 is that the related news articles display the related news article creation time information as the date and time. This is because, although it may be displayed in the configured standard time format, some related news articles may be displayed in a format such as several hours or minutes before the current time when the related news article was created.

In this way, the news crawling system 200 according to the present invention can display all time information of the related news articles in a unified format through the related news article refiner 270, so that the user who reads the related news articles It is possible to easily check the creation time of news articles.

Hereinafter, a news crawling method according to the present invention will be described.

7 is a flowchart illustrating a news crawling method according to an embodiment of the present invention. The news crawling method shown in FIG. 7 may be performed by the news crawling system shown in FIGS. 2A and 2B .

First, the news crawling system receives a search query including a search term from the user terminal (S700). The news crawling system collects URLs of news related to the search word by using the search word included in the search query (S710). Thereafter, the news crawling system selects URLs including link information on a web page of a predetermined portal service from among the collected URLs ( S720 ).

In the present invention, the reason that the news crawling system selects only URLs that contain link information on the web page of the portal service from among the collected URLs is that the structure of the website is different for each newspaper that provides each news article. This is because, if you want to use the news articles provided by the website as they are, you have to configure the code differently for each newspaper company, which can be cumbersome. Another reason is that the code must be manually modified every time a newspaper website is renewed, and when the structure of a newspaper website is changed, it is difficult to predict the timing and contents of the structural change in advance. Failure to do so may result in errors.

On the other hand, since news articles provided through the portal service are provided through the website of the portal service having the same HTML structure, it is possible to prevent problems caused by using the newspaper's web page as described above. be able to

Thereafter, the news crawling system collects news articles corresponding to the news URLs selected in S720 (S730). In this case, the news crawling system may collect the title and body of the news article.

Thereafter, the news crawling system first filters duplicate news among the news articles based on the titles of the news articles (S820).

Specifically, the news crawling system acquires at least one of a noun, a morpheme, and a syllable from the title of each news by first analyzing the titles of the collected news articles in units of at least one of a noun, a morpheme, and a syllable. Hereinafter, for convenience of explanation, it is assumed that the news crawling system acquires syllables from the title.

The news crawling system calculates a similarity index between the titles of the collected news articles based on the acquired syllables, and determines news articles having the calculated similarity index greater than or equal to a threshold as duplicate news. In an embodiment, the news crawling system may calculate a jacquard similarity index based on the syllables of each news article title, and determine news articles corresponding to a title having a jacquard similarity index greater than or equal to a threshold as duplicate news. The news crawling system may complete the primary filtering by selecting any one news article from among the news articles determined to be duplicate news.

Thereafter, the news crawling system selects related news articles by performing secondary filtering on the primary filtered news articles based on the title and body of the news article (S750).

Specifically, the news crawling system converts a first related news article to a news article corresponding to a title in which a first frequency of occurrence of a target phrase in which predetermined characters are continuously displayed in a search word within the titles of the firstly filtered news articles is equal to or greater than a first reference value select with In addition, the news crawling system selects, as the second related news article, a news article corresponding to a text having a second appearance frequency of the target phrase in the body of the first filtered news articles equal to or greater than a second reference value greater than the first reference value.

In an embodiment, the predetermined characters included in the target phrase are continuously displayed in the search word within the predetermined texts, and the frequency of appearance may include at least one of a key search term, a key word, and a special character whose frequency of occurrence is equal to or greater than a reference value. . Since key search terms, key words, and special characters have been described in the second and third filter sections, detailed descriptions will be omitted.

When the first related news article and the second related news article are selected, the news crawling system selects the first related news article and the second related news article as final related news articles to be included in the news search result.

Thereafter, the news crawling system generates a news search result based on the related news articles selected in S750 (S760), and provides the generated news search result to the user terminal (S780).

Meanwhile, although not shown in FIG. 7 , the above-described second filtering process may be performed by an AI model, and the news crawling system receives feedback on the relevant news articles from the user terminal, Among the characters continuously displayed in the corresponding search word within the article, the characters having the frequency of appearance greater than or equal to the reference value can be extracted again, and the AI model can be trained using the extracted characters. At this time, the news crawling system may newly add key research, key words, and special characters continuously displayed in the corresponding search word through training of the AI model, or adjust the values of the first and second reference values.

Those skilled in the art to which the present invention pertains will understand that the above-described present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof.

Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

200: news crawling system 205: user terminal
210: URL collection unit 220: news article collection unit
230: first filtering unit 240: second filtering unit
250: search result providing unit 260: training unit
270: Refining Department of Related News Articles

Claims (15)

When a search query including a search term is received from the user terminal, a URL collection unit for collecting URLs of news articles including the search term;
a news article collecting unit that collects titles and texts of news articles corresponding to the URLs for each news article;
a first filtering unit for primary filtering of duplicate news among the news articles based on the titles of the collected news articles; and
and a second filtering unit for selecting related news articles by secondarily filtering the first filtered news articles based on the titles and texts of the firstly filtered news articles,
The second filtering unit,
Count the first frequency of occurrence of the target phrase in which a predetermined character is continuously displayed in the search word within the titles of the first filtered news articles, and a news article corresponding to a title in which the counted first frequency of occurrence is equal to or greater than a first reference value a second filter for selecting a first related news article;
Counting the second frequency of occurrence of the target phrase in the body of the first filtered news articles, and selecting a news article corresponding to the main text having the counted second frequency of occurrence equal to or greater than a second reference value as a second related news article a third filter; and
and a related news article selection unit for selecting the first related news article and the second related news article from among the first filtered news articles as the related news. According to claim 1,
The first filtering unit,
a title analysis unit that analyzes the titles of the collected news articles into at least one unit of a noun, a morpheme, and a syllable; and
The similarity between the titles of the collected news articles is calculated based on at least one of nouns, morphemes, and syllables analyzed by the title analysis unit, and news articles corresponding to titles having a similarity greater than or equal to a threshold are determined as duplicate news. News crawling system comprising a first filter for filtering. 3. The method of claim 2,
The first filter is
A Jaccard similarity index between the titles of each news article is calculated based on at least one of nouns, morphemes, and syllables obtained for each title of the collected news articles, and the calculated Jaccard similarity index is greater than or equal to the threshold. News crawling system, characterized in that for selecting any one of the news articles corresponding to the. delete The method of claim 1,
The predetermined character included in the target phrase is continuously displayed in the search word within predetermined texts, and the frequency of appearance includes at least one of a key word, a key word, and a special character. crawling system. 6. The method of claim 5,
The key research includes at least one of "is", "is", "this", "a", "a", "a", "of", "in", "and" The key word includes at least one of "group", "family", and "province", and the special character is ",", """, "·", "(", "'", "]" , "-", "..." News crawling system, characterized in that it includes at least one of. The method of claim 1,
The news crawling system, characterized in that the first reference value is set to have a smaller value than the second reference value. According to claim 1,
The second filtering unit is spherical in an AI (Artificial Intelligence) model, receives feedback on satisfaction with the related news article from the user terminal, and texts continuously displayed in the search word in related news articles selected according to the feedback satisfaction News crawling system, characterized in that it further comprises a training unit for training the AI model using the. According to claim 1,
The URL collection unit selects URLs including link information on a web page of a predetermined portal service from among the URLs, and provides the selected URLs to the news article collection unit. According to claim 1,
Further comprising a search result providing unit for providing the selected related news article to the user as a news search result for the search query,
The search result providing unit,
a storage unit including a plurality of queues and separately storing at least one of a title, a date, a text, an original link of the related news article, and information on a newspaper company generating a related news article in the plurality of queues; and
and a search result generator for generating the news search result by merging information stored in the queue according to a predetermined format, and providing the generated news search result to the user terminal. According to claim 1,
News crawling system, characterized in that it further comprises a related news article refiner for correcting the time information at which the related news article is generated in a predetermined standard time format. when a search query including a search term is received from the user terminal, collecting URLs of news including the search term;
selecting URLs including link information on a web page of a predetermined portal service from among the collected URLs;
filtering duplicate news among the news articles based on the titles of the news articles corresponding to the selected URLs;
selecting related news articles by secondarily filtering the firstly filtered news articles based on the titles and texts of the firstly filtered news articles; and
and providing the selected related news article to the user terminal as a news search result for the search query,
The secondary filtering step is
selecting, as a first related news article, a news article corresponding to a title in which a first frequency of occurrence of a target phrase in which predetermined characters are continuously displayed in the search word within the titles of the first filtered news articles is equal to or greater than a first reference value;
selecting, as a second related news article, a news article corresponding to a text in which a second frequency of occurrence of the target phrase is greater than or equal to a second reference value greater than the first reference value in the primary filtered news body; and
and selecting the first related news article and the second related news article from among the first filtered news articles as the related news article. 13. The method of claim 12,
The first filtering step is,
analyzing the titles of the news articles into units of at least one of a noun, a morpheme, and a syllable;
calculating a jacquard similarity index between the titles of the news articles based on at least one of the analyzed nouns, morphemes, and syllables; and
and selecting any one news article from among the news articles corresponding to a title having the jacquard similarity index equal to or greater than a threshold. delete 13. The method of claim 12,
The predetermined character included in the target phrase is continuously displayed in the search word within predetermined texts, and the frequency of appearance includes at least one of a key word, a key word, and a special character. How to crawl.

Images