KR20170071389A - New post notification system and method for analyzing and extracting new post of unspecified website - Google Patents

Abstract

A new method of notification and system is presented. The new news notification method proposed by the present invention includes the steps of registering the URLs of the web sites desired by the user on the URL registration page to use the news feed function, checking whether the new URLs are posted on the corresponding web sites of the registered URLs A step of defining metadata including a page name, a registered URL, a text extraction keyword, and a thumbnail of a web page of a news feed registered in a URL registration page when a new article is posted; And classifying new articles according to categories of news feeds on a web site of registered URLs and visualizing them through a list view.

Description

TECHNICAL FIELD [0001] The present invention relates to a new post notification system and method for analyzing new posts of an unspecified website,

The present invention relates to a new article notification system and method by analyzing a new article and extracting a body of an unspecified web site.

The service that feeds the existing new information searches the desired web site and registers the retrieved web site to receive the web site information or the web site itself.

The existing feed system is intended to provide limited service to specific web sites set by the provider providing the feed service. Or if the web site retrieved by the user does not provide the feed information or does not have the meta information, it is difficult to extract the information and use the service.

The present invention describes a system and a method for notifying a new article through new article analysis and extracting a body of an unspecified web site. More specifically, the present invention focuses on a service that can efficiently manage scattered data scattered in various websites .

According to an aspect of the present invention, there is provided a method of notifying a new user of a new message, comprising the steps of: registering URLs of a desired web site on a URL registration page to use a news feed function; A step of defining metadata including a page name of a web page of a news feed registered in a URL registration page, a registered URL, a text extraction keyword, and a thumbnail when a new article is posted, And classifying new articles according to the category of the news feed in the website of the registered URLs based on the defined metadata and visualizing them through the list view.

The user can register the news feed desired by the user in the bookmark and visit the web site through the registered news feed.

In the step of checking whether or not a new article is posted on the corresponding website of the registered URLs, the step of reading the HTML information from the registered URLs registers the HTML information in the database server, and searches for a new article by using the HTML information as a location . At this time, the crawler uses the data structure of the hash map (HasnMap) to store site-specific information using the URL as a key, updates the key of the hash map every crawling process, A hash map that stores crawled results before crawling to find information about new articles, and a hash map that stores sites that have proven to have a posting of new articles. Also, use the oldArticles member variable to determine whether a new post has been posted, extracting the new crawled list of new posts and the new, new ones from the new crawled list.

The step of defining metadata including a page name, a registered URL, a text extraction keyword, and a thumbnail of a web page of a news feed registered in the URL registration page in the case of the publication of the new article is performed by adding the new article to the addedArticles member variable And saves the html information of the bulletin board of the site by receiving information about the new article. In this case, if the header information is not extracted, the new article information for a plurality of sites is registered in the database, and the new article information for the plurality of sites is extracted based on the new article information . Also, a program for storing new article information for the plurality of sites is generated, and a database is accumulated through the program.

According to another aspect of the present invention, a new news notification system proposed by the present invention includes a URL registration unit for registering URLs of web sites desired by a user in a URL registration page to use a news feed function, If there is a new article check that confirms whether the article is posted, if there is a post of a new article, the metadata including the page name, the registered URL, the text extraction keyword, and the thumbnail of the web page of the news feed registered in the URL registration page And a news feed classification and notification unit for classifying new articles according to categories of news feeds and visualizing them through a list view on a web site of registered URLs based on the defined metadata.

The URL registering unit registers the news feed desired by the user in the bookmark and allows the user to visit the web site through the registered news feed.

The new article check unit reads HTML information from registered URLs, registers HTML information in a database server, and searches for new articles based on the HTML information. At this time, the crawler uses the data structure of the hash map (HasnMap) to store site-specific information using the URL as a key, updates the key of the hash map every crawling process, A hash map that stores crawled results before crawling to find information about new articles, and a hash map that stores sites that have proven to have a posting of new articles. Also, use the oldArticles member variable to determine whether a new post has been posted, extracting the new crawled list of new posts and the new, new ones from the new crawled list.

The metadata definition unit stores the new article in the addedArticles member variable, receives information about the new article, and stores the html information of the bulletin board of the corresponding site.

Wherein the metadata defining unit extracts header information for the extracted text keywords, and when the header information can not be extracted, new metadata information for a plurality of sites is registered in the database, . In addition, the metadata definition unit generates a program for storing new article information for the plurality of sites, and stores the database through the program.

There are numerous types of Web sites in a non-standard web environment. In order to manage these web sites integrally, it is limited to only those sites that provide existing feeds. This problem is solved and the information of all the websites desired by the user can be checked in real time, which is effective in managing the information scattered by the users.

FIG. 1 is a diagram illustrating an overall structure of a new text notification system according to an embodiment of the present invention.
2 is a flowchart illustrating a method of notifying a new text according to an embodiment of the present invention.
3 is a diagram for explaining a new text notification scenario according to an embodiment of the present invention.
4 is a diagram illustrating a new article extraction algorithm according to an embodiment of the present invention.
5 is a diagram illustrating a process of registering information on a new article according to an embodiment of the present invention.
6 is a diagram for explaining a process of registering a database according to an embodiment of the present invention.
7 is a diagram for explaining a structure of a new text notification system according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating an embodiment of a method for notifying a new text according to an embodiment of the present invention. Referring to FIG.

As the popularity of smartphones has increased the accessibility of the web, the data usage has exploded. In this trend, an application that can efficiently manage data to a user is required. Therefore, we propose a service that can efficiently manage sporadically scattered data from various websites. The proposed method and system can help a user to actively and easily set up a desired web site to manage data easily and conveniently.

It builds its own database and receives a choice about the area in which the user wants to be notified of new posts. It reads the HTML information from the selected area, registers the information in its own DB server, and creates an algorithm to search for new articles based on the information. If there is no meta information and meta information about the new article that the user wants to know, the user can extract the information of the relevant article in any case by applying the algorithm developed by himself / herself so that the user can use the feed service from the unspecified website Method and system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating an overall structure of a new text notification system according to an embodiment of the present invention.

A user terminal, that is, a client 110 selects an area to which a new notice is to be received, and the Web server 120 reads the HTML information from the area selected through the HTTP specification, Registers the information in the DB server and creates an algorithm to search for new articles based on the information.

When there is no meta information and meta information about a new article that the user wants to know, the crawler 230 can extract the information of the relevant article in any case by applying the algorithm developed by itself.

The Web server 120 and the crawler 230 store the collected information in the database server 240 using SQL. If there is no meta information and meta information about a new article that the user wants to know, the user can extract the information of the relevant article in any case by applying the algorithm developed by himself / herself so that the user can use the feed service from the unspecified website I suggest.

2 is a flowchart illustrating a method of notifying a new text according to an embodiment of the present invention.

In order to use the news feed function, the proposed new news notification method includes a step 210 of registering the URLs of the web sites desired by the user on the URL registration page (step 210), checking whether the new URLs are posted on the corresponding web sites of the registered URLs Step 230 of defining metadata including a page name, a registered URL, a text extraction keyword, and a thumbnail of a web page of a news feed registered in a URL registration page when a new article is posted; And sorting new articles in the web site of registered URLs based on the defined metadata according to categories of the news feeds and visualizing them through the list view (240).

In step 210, URLs of web sites desired by the user are registered on the URL registration page to use the news feed function. The user can register the news feed desired by the user in the bookmark and visit the web site through the registered news feed.

In step 220, it is confirmed whether a new article is posted on the corresponding website of the registered URLs. At this time, the HTML information is read from the registered URLs, the HTML information is registered in the database server, and a new article is searched based on the HTML information. The crawler uses the data structure of the hash map (HasnMap) to store site-specific information with the URL as the key. The hash map (HasnMap) updates the key of the hash map every crawling process, thereby providing information about the user receiving the new article of the site for each site, a hash map storing the crawling result before crawling to find a new article, Lt; RTI ID = 0.0 > of < / RTI > publications. When determining whether a new post has been posted, use the oldArticles member variable to retrieve the new crawled list of new posts and the new one from the new crawled list.

In step 230, metadata including the page name, the registered URL, the text extraction keyword, and the thumbnail of the web page of the news feed registered in the URL registration page is defined when there is a new article to be posted. The new article is stored in the addedArticles member variable, and information about the new article is stored in the html information of the bulletin board of the site. Extracts the header information for the extracted text keywords, and if the header information can not be extracted, registers the new text information for the plurality of sites in the database, and extracts the new text information based on the new text information for the plurality of sites. Then, a program for storing new article information about a plurality of sites is created, and the database is accumulated through the program.

In step 240, new articles are classified according to the category of the news feed in the website of registered URLs based on the defined metadata, and visualized through a list view.

3 is a diagram for explaining a new text notification scenario according to an embodiment of the present invention.

First, an application is executed (310) and a login screen is displayed. When the application is executed, the login (320) screen is displayed. When the user logs in by inputting the account information, the main frame of the application is moved.

Next, the URL to register a new article in the news feed is registered (330). In order to use the news feed function, which is a main service, the user registers the URL of the desired web site on the URL registration page.

Once the URL is registered, a news feed confirmation 340 and a body verification 350 can be performed. The user can view new posts on the registered web page in the news feed page. Posts from news feeds are dynamically executed and can be viewed in detail.

Then, it is possible to register a news feed to be frequently viewed and register it in the bookmark 360. Any news feed registered in the bookmark can visit the registered website at any time.

Management 370 such as registering a desired URL using an add button can be performed. If you enter the name and address of the site, a WebView is created. You can register your website by selecting the post you want to view as a news feed through WebView. In many news feed posts, you can register the keywords you want to see and select the posts you want.

4 is a diagram illustrating a new article extraction algorithm according to an embodiment of the present invention.

First, the URL list is confirmed (410), and the presence or absence of the posting of a new article by URL is confirmed (420). If there is a new post, the body of the new post is extracted (430). The push notification is transmitted 430 based on the user's push (PUSH) information.

5 is a diagram illustrating a process of registering information on a new article according to an embodiment of the present invention.

The crawler has site-specific information that uses a hash map (HashMap) data structure to make the URL a key. The site-specific information is as follows. By updating the key of this HashMap in every crawling process, the user registers and deletes the bulletin board deleted. The site information data structure can be represented as shown in FIG. 5A.

Each site has a hash map (HashMap) that stores the crawl results before crawling to find new posts, and a HashMap that stores the new named sites. When determining a new post, use the oldArticles member variable to retrieve the new post from the list of posts we crawled and the list of posts we just crawled. The new posts are stored in the addedArticles member variable. In order to store the information on the bulletin board, it receives the information of the posted bulletin from the client and stores the html information of the bulletin board for the site. The bulletin board information database can be represented as shown in FIG. 5B.

As described above, the tag name and the class name of the html that surrounds the area of the bulletin board by the site URL are extracted and stored in the database, and the links beneath the text are extracted and posted Let's write. This makes it possible to extract more accurate posts.

6 is a diagram for explaining a process of registering a database according to an embodiment of the present invention.

There are three ways to extract the body of a new bulletin board. The priority of the method is as follows. First, the text is extracted from the header information. Then, the text is extracted from the database information. Finally, use the Boiler Pipe library. When using Header information, extract information of meta-data area in head part of html. In this method, information can not be obtained if the site does not register information in the header. Therefore, if the header information is extracted first, the post information about various sites is directly registered in the database and extracted based on this information. We created a program to store the bulletin information, and accumulated the database by working directly. The contents of the database are shown in FIG.

According to the embodiment of the present invention, it is possible to accumulate information about news sites and bulletin boards of various non-community universities by setting rankings for each website based on traffic information. Through this, accuracy of extraction of various text html is improved. The program to register this database was created using Java Swing. In the case where the extraction is not possible by the above two methods, the Boiler pipe library known to have the highest extraction accuracy is used. Boiler pipe is known to provide about 73% accuracy in extracting text when tested with well-established news or blog post sites. These three text extraction methods can improve the accuracy of text extraction.

7 is a diagram for explaining a structure of a new text notification system according to an embodiment of the present invention.

The proposed new text notification system includes a URL registration unit 710, a new text verification unit 720, a metadata definition unit 730, and a news feed classification and notification unit 740.

The URL registration unit 710 registers URLs of web sites desired by the user on the URL registration page to use the news feed function. The user can register the news feed desired by the user in the bookmark and visit the web site through the registered news feed.

The new article confirming unit 720 confirms whether or not a new article is posted on the web site of the registered URLs. At this time, the HTML information is read from the registered URLs, the HTML information is registered in the database server, and a new article is searched based on the HTML information. The crawler uses the data structure of the hash map (HasnMap) to store site-specific information with the URL as the key. The hash map (HasnMap) updates the key of the hash map every crawling process, thereby providing information about the user receiving the new article of the site for each site, a hash map storing the crawling result before crawling to find a new article, Lt; RTI ID = 0.0 > of < / RTI > publications. When determining whether a new post has been posted, use the oldArticles member variable to retrieve the new crawled list of new posts and the new one from the new crawled list.

The metadata definition unit 730 defines metadata including a page name, a registered URL, a text extraction keyword, and a thumbnail of a web page of a news feed registered in a URL registration page when a new article is posted. The new article is stored in the addedArticles member variable, and information about the new article is stored in the html information of the bulletin board of the site. Extracts the header information for the extracted text keywords, and if the header information can not be extracted, registers the new text information for the plurality of sites in the database, and extracts the new text information based on the new text information for the plurality of sites. Then, a program for storing new article information about a plurality of sites is created, and the database is accumulated through the program.

The news feed classification and notification unit 740 classifies new articles in the web sites of registered URLs based on the defined metadata according to categories of news feeds and visualizes them through a list view.

FIG. 8 is a diagram illustrating an embodiment of a method for notifying a new text according to an embodiment of the present invention. Referring to FIG.

When the application is executed as shown in FIG. 8A, the first screen for login is displayed. If you enter your account information and log in, you will be taken to the application's mainframe.

The top menu uses a fragment to construct a tabbed menu. In addition, user defined names are categories, and categories can be dynamically selected by the user using a spinner.

As shown in FIG. 8B, the items of the news are composed of the name of the page defined by the user through the custom layout, the registered URL, the keyword extracted from the text, and the thumbnail. The corresponding individual list includes the title and summary Show. The categories are listed in the Admin tab, including the page name and URL entered when registering the URL. Each page shows the dynamically posted time, and you can press the reverse table link shortcut, add a bookmark, or remove the list from the list.

Referring to FIG. 8C, by clicking the magnifying glass at the top, the category of the news feed shown in the current list view can be confirmed. You can select a registered category to select a specified news feed. You can also extract newsfeeds that contain only the keywords that are entered through the keyword specified in the Administration tab.

When registering a bookmark on the news feed tab, a bookmark list can be found on the bookmark tab as shown in FIG. 8D. If you click on the registered bookmark, you can see the web of registered URL through WebView.

As shown in FIG. 8E, the management tab can register a URL to be received in the news feed. Press the Add button to enter the name and URL, and register the keyword as a # segment (eg # Football # weather). When you press the OK button, a WebView is created with a message to select a bulletin board posting to be registered through WebView. When you select a post you want to view, registration is completed successfully to view the news feed. When you select a post in WebView, the body div of the web post is extracted and the data for grasping the structure is stored in the DB. In the Administration tab, you can turn on / off the push alarm setting via the notification button, and modify or delete the registered information.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In the new method of notification,
Registering URLs of web sites desired by the user on the URL registration page to use the news feed function;
Confirming whether or not a new article is posted on the web site of the registered URLs;
Defining metadata including a page name, a registered URL, a text extraction keyword, and a thumbnail of a web page of a news feed registered in a URL registration page when a new article is posted; And
Categorizing new articles according to the category of the news feed in the website of registered URLs based on the defined metadata and visualizing them through a list view
How to notify a new post that contains. The method according to claim 1,
The user can register the news feed desired by the user in the bookmark and visit the web site through the registered news feed
How to notify new posts. The method according to claim 1,
Wherein the step of confirming whether or not a new article is posted on the corresponding website of the registered URLs,
The HTML information is read from the registered URLs, the HTML information is registered in the database server, and a new article is searched by using the HTML information as a location
How to notify new posts. The method of claim 3,
The crawler uses the hash map data structure to store site-specific information using a URL as a key, and updates the key of the hash map every crawling process, thereby obtaining information about a user And a hash map that stores the results of the crawling before crawling to find new articles, and a hash map that stores the sites that have been found to contain the publication of new articles
How to notify new posts. The method of claim 3,
Use the oldArticles member variable to determine whether there is a post for a new post, to extract new, new posts from the new crawled list of posts and the previously crawled list of new posts.
How to notify new posts. The method according to claim 1,
The step of defining the metadata including the page name, the registered URL, the text extraction keyword, and the thumbnail of the web page of the news feed registered in the URL registration page when the new article is posted,
Save the new article to the addedArticles member variable, receive information about the new article, and save the html information of the bulletin board of the site
How to notify new posts. The method according to claim 6,
Extracting header information for the main text extraction keyword and registering the new article information for a plurality of sites in the database when the header information is not extracted and extracting based on new article information for the plurality of sites
How to notify new posts. 8. The method of claim 7,
A program for storing new article information about the plurality of sites is created, and a database is accumulated through the program
How to notify new posts. In a new message notification system,
A URL registration unit for registering URLs of web sites desired by the user on the URL registration page to use the news feed function;
A new posting confirmation unit for confirming whether or not to post a new posting on the website of the registered URLs;
A metadata definition unit for defining metadata including a page name, a registered URL, a text extraction keyword, and a thumbnail of a web page of a news feed registered in a URL registration page when a new article is posted; And
A news feed classification and notification section for classifying new articles according to categories of news feeds in a web site of registered URLs based on the defined metadata and visualizing them through a list view
A new notification system that includes. 10. The method of claim 9,
The URL registering unit,
The user can register a news feed desired by the user in the bookmark and allow the user to visit the web site through the registered news feed
New post notification system. 10. The method of claim 9,
The new-
The HTML information is read from the registered URLs, the HTML information is registered in the database server, and a new article is searched by using the HTML information as a location
New post notification system. 12. The method of claim 11,
The crawler uses the hash map data structure to store site-specific information using a URL as a key, and updates the key of the hash map every crawling process, thereby obtaining information about a user And a hash map storing the results of the crawling before crawling to find new articles, and a hash map storing sites found to have the publication of new articles,
Use the oldArticles member variable to determine whether there is a post for a new post, to extract new, new posts from the new crawled list of posts and the previously crawled list of new posts.
New post notification system. 10. The method of claim 9,
Wherein the metadata definition unit comprises:
Save the new article to the addedArticles member variable, receive information about the new article, and save the html information of the bulletin board of the site
New post notification system. 14. The method of claim 13,
Wherein the metadata definition unit comprises:
Extracting header information for the main text extraction keyword and registering the new article information for a plurality of sites in the database when the header information is not extracted and extracting based on new article information for the plurality of sites
New post notification system. 15. The method of claim 14,
Wherein the metadata definition unit comprises:
A program for storing new article information about the plurality of sites is created, and a database is accumulated through the program
New post notification system.

Images