KR20080052097A - Harmful web site filtering method and apparatus using web structural information - Google Patents

Abstract

A method and an apparatus for filtering a harmful site by using web structure information are provided to effectively and definitely filter harmful sites like lewd sites and to prevent children or juveniles from accessing the harmful sites. An apparatus for filtering a harmful site includes a web document receiver(720), a basic filter(730) and a web structure information analyzer(740). The web document receiver receives web documents. The basic filter filters text and image from the received web documents first and sends the filtered web documents to the web structure information analyzer. The web structure information analyzer includes a harmful URL database comparator(760) and a learning model comparator(750). The web structure information analyzer determines whether the web documents are harmful by using the harmful URL database comparator which performs a learning model comparison or a link URL comparison, and if the web documents are determined to be harmful, makes a web document transmitter(770) transmit the web documents to a display unit(780). The harmful URL database comparator determines whether the web documents are filtered or not by using the link URL of the web documents. The learning model comparator(750) includes a web structure information extractor(752), a file creator(754), a learning model creator(756) and a file comparator(758). The web structure information extractor extracts the web structure information of a file transmitted from the web document receiver or the basic filter(730), and the file creator creates structure information, which is extracted from the web structure information extractor, in one file while performing the extraction of a harmful or harmless site.

Description

Harmful web site filtering method and apparatus using web structural information}

1 is a view showing a method for blocking a harmful site by the existing technology.

2 is a view showing a harmful site blocking method using web structure information according to the present invention in addition to the existing technology.

FIG. 3 is a diagram illustrating a method in which a filtering module according to an existing technology and a web structure information based filtering module according to the present invention interact with each other for an inbound packet and an outbound packet.

Figure 4 shows an embodiment of using a harmful URL database as a filtering based on the URL structure of the existing technology and the web structure information according to the present invention.

FIG. 5 shows an embodiment of using a URL filtering of the existing technology and a comparison based on a harmful URL database reference and a learning model as filtering based on web structure information according to the present invention.

6 shows a harmful site determination process using a learning model generation process and the generated learning model according to the present invention.

7 shows an apparatus for blocking harmful sites using web structure information according to the present invention.

The spread of the Internet, which connects the world, has brought a radical change in modern life. It is now important to go beyond the era of trying to get information, and how to select useful information from the vast amount of information in accessible databases such as the Internet. The sharing of information through the Internet is an undeniable fact that has made a remarkable contribution to the technological development and knowledge sharing of modern people, but its easy accessibility and rapid spread of information have many side effects.

The various side effects of the Internet include the leakage of personal information due to the lack of security, the accessibility of anyone, and the merging of pornography due to the advantages of using the Internet commercially. Indeed, the distribution of pornography via the Internet has become a big social problem, as there is a statistic that the most commercial success through the Internet is the pornography business.

Age groups who can easily use personal computers from young children to adults are indiscriminately exposed to these harmful sites. Many hardware devices and software programs have been developed to prevent these harmful sites from accessing computer terminals in the home or work. come. However, with the introduction of harmful site protection technologies, operators that distribute harmful sites have also developed and used methods to artfully disable these protection technologies for their economic benefit.

The present invention relates to a method and apparatus for blocking a site connected when a public network such as the Internet is a harmful site such as an obscene site in a network capable terminal such as a computer.

Conventionally, in case of outbound packet outgoing from network user to block malicious site, URL information input by user is used or in case of inbound packet accepted by network user, text or image information of web document contents It was determined whether or not harmful by using.

More specifically, when a network terminal user inputs specific URL information to access a certain site, the URL address is interpreted to determine whether or not the relevant site is related to the harmful site, thereby determining whether to load the corresponding site. In short, if the URL information contains words that appear mainly on harmful sites, for example, if it contains sex, hardcore, etc., it blocks the loading of the website by filtering them.

If this initial filtering fails, pre-checking is performed before printing the called web document on the screen. In other words, the text or image information of the web document content is analyzed by the blocking software in advance, and there is an expression containing harmful elements in the text content of the web document, or the image information, that is, the color information displayed on the screen, is excessively analyzed. If a lot of sensational colors-pink, red-etc. occupy a large portion of the site was determined to be a harmful site.

In the conventional method, the blocking method using URL information in the case of an outbound packet or the blocking method using a text and / or image information in the case of an inbound packet face various limitations. In other words, in the case of the blocking method using URL information, harmful sites that are not included in the already constructed URL information database can easily avoid this blocking method and disable the blocking method by automatically changing the URL address. You can also

In the case of analyzing the contents of the inbound web document, there is a possibility of overblocking or error. More specifically, in order to avoid blocking by the inbound web document analysis recently, text is processed as an image or image processing is performed. The reality is that web documents are organized using various methods that make it difficult.

In order to solve this technical problem, the present invention utilizes the link URL in the HTML source, the number and attributes of pop-up windows, title tags, fonts, frame information, etc. as the structure information of the web document, or is used alone. It is an object of the present invention to complement the above-mentioned limitations of the prior art and to provide a more effective and probable harmful site blocking method and apparatus.

More specifically, the present invention generates a learning model from the web structure information of a plurality of harmful sites and harmless sites, selectively performs URL filtering, text filtering, and / or image filtering on inbound web documents, and optionally And comparing the filtered web document with the generated learning model, and determining the blocking result to determine whether to block the harmful web site.

The present invention comprises the steps of generating a learning model from the web structure information of a plurality of harmful sites and harmless sites to achieve the above technical problem; Comparing an inbound web document with the generated learning model; And determining blocking based on a result of comparison with the learning model. The harmful site blocking method using the web structure information is provided.

Preferably, the method further includes generating URL learning, text filtering, and / or image filtering on the inbound web document after generating the learning model in the harmful site blocking method using the web structure information. It is characterized by.

Preferably, the generating of the learning model in the harmful site blocking method using the web structure information comprises: number of pop-up windows, background color, frame number, frame color, image from the web structure information of the plurality of harmful sites and harmless sites. Extracting values for the number and / or number of text images; Making the extracted value into a file; And generating a learning model by using the pattern classification technique on the generated file.

More preferably, the pattern classification technique is characterized by using a support vector machine (SVM), clustering (Slustering) or self organizing map (SOM).

In addition, the harmful site blocking method using the web structure information when accessing the web site for the purposes of the present invention comprises the steps of constructing a harmful URL database including a known harmful website and links; Comparing the link URL in the filtered web document with URL data in the generated harmful URL database; Determining the blocking of the web document according to the comparison result.

Preferably, the harmful site blocking method further includes the step of performing URL filtering, text filtering, and image filtering on the inbound web document after the building of the harmful URL database.

Preferably, the method may further include determining blocking by comparing the web document that has undergone URL filtering, text filtering, and image filtering with respect to the generated learning model for the inbound web document. Characterized in that it further comprises generating a learning model from the harmful web site and harmful structure of the web site information.

Preferably, generating the learning model extracts values for the number of pop-up windows, the background color, the number of frames, the frame color, the number of images, and / or the number of text images from the web structure information of the plurality of harmful and harmless sites. Doing; Making the extracted value into a file; And generating a learning model by using the pattern classification technique on the generated file.

More preferably, the pattern classification technique is characterized by using a support vector machine (SVM), clustering (Slustering) or a self organizing map (SOM).

In addition, the harmful site blocking device using the web structure information for the purposes of the present invention includes a web document receiving unit for receiving a web document transmitted from the outside; A web structure information analyzer for analyzing structure information of the web document transmitted from the outside; And a web document transmission unit for transmitting a web document that is found to be a harmful site through the web structure information analysis unit to a monitoring device that can be viewed by a user.

The apparatus may further include a basic filtering unit for performing URL filtering, text filtering, and image filtering on the received web document.

 Preferably, the web structure information analysis unit is characterized in that it comprises a learning model comparison unit and / or harmful URL database comparison unit.

More preferably, the learning model comparison unit comprises: a web structure information extracting unit for extracting values for the number of pop-up windows, the background color, the number of frames, the frame color, the number of images, and / or the number of text images in the web document; A file generator which makes the extracted web structure information into one file; And a learning model generator for generating a learning model using the generated file through the pattern classification technique. And a file comparator for comparing the learning model with a file generated by the file generator.

Hereinafter, a method and apparatus for blocking harmful sites using web structure information according to the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, when it is determined that detailed descriptions of related well-known technologies or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the present invention, which may vary according to a client's or operator's intention or custom. Therefore, the definition should be made based on the contents throughout the specification.

1 shows a method for blocking a harmful site by the existing technology. Once a particular web site is accessed, various web documents 110 enter the network terminal.

Existing methods analyze these web documents and filter them appropriately. The filtered web document is classified into a harmful web document 130 and a harmless web document 140. The filtering block 120 has various filtering units, and typically includes URL filtering 122, text filtering 124, and / or image filtering 126.

The URL filtering unit compares the website address with the URL of the database that stores the existing harmful site URL when the website address is input from the network terminal, and the website address contains the same or similar words as the harmful site URL information. Filter this to prevent access to the website. That is, strictly, the filtering is performed on the URL information input by the user, rather than the filtering on the web document 110 coming into the network terminal.

Usually, a web site is composed of a series of numbers, for example 244.128.72.2, but the combination of these numbers is not easy for the general public and is not easy to remember. This makes it easier for general users to recognize and remember, and to allow website operators to perform functions such as trademarks and trade names, or if the website operator is a business operator, to make it easier for the public to match their well-known business name. You will use a website address that you can. In other words, using a web site address such as www.carsale.co.kr, www.auction.com, etc. to make the function or content of the web site predictable by the end users of the web site or through the web It would be advantageous from the user's point of view. These web site addresses are converted to the numbers (addresses) mentioned above and the web site content is fetched from a particular server at that address.

In more detail, a URL (uniform resource locator) usually consists of http://www.msn.com/index.html or ftp://ftp.tb.ac.kr/pub/public.zip. In the above case, http and ftp are the protocols to use when connecting, and later www.msn.com and ftp.tb.ac.kr refer to the address and directory of the server where the user is searching. Immediately following index.html or pub / public.zip will be the path where the file is located.

In this case, if the URL information input by the user of the network terminal includes a word that is likely to be information representing a sexually explicit site, the URL filtering unit previously stores such information in a previously stored database (not shown), and then enters the user input URL. Blocking is performed by comparing with information.

In case of text filtering, unlike URL filtering, when outbound packets are not filtered by URL filtering, the web documents that are called are displayed on the network terminal based on URL information. The web documents of these inbound packets are displayed. It will filter out the text. In other words, text filtering analyzes text within a web document and compares it with words in a text database (not shown), depending on the frequency and selectivity of words when such texts commonly appear on harmful sites. The decision is made to block.

Image filtering, like text filtering, is also responsible for web document filtering of inbound packets. Image filtering uses image information (mainly color information) in a web document to determine whether it is pornographic or not.

However, the URL filtering, text filtering, and image filtering techniques shown in FIG. 1 have many possibilities of overblocking and error blocking. In addition, in recent years, obscene site operators have used automatic URL addressing to circumvent these traditional methods, and use text to image to avoid text filtering and to make image processing difficult. Due to the structure of the document, there is a limit to blocking harmful sites by the conventional method. As a result, there are fewer problems such as error and overblocking, and a new method of increasing the blocking property is needed despite various blocking methods of operators such as obscene sites.

2 is a diagram illustrating a harmful site blocking method using web structure information according to the present invention.

As shown in the figure, the present invention may provide more effective filtering when the filtering block 220 according to the existing technique shown in FIG. 1 is added. However, the filtering block 220 is not necessarily essential, and even when used alone as the filtering block 230 based on the structural information proposed in the present invention, the filtering of the harmful site may be quite effective.

As in FIG. 1, URL filtering 222 is performed on the initial outbound URL information, and the inbound web document 210 is primarily filtered by text filtering 224, image filtering 226, and the like. Secondly, through the web structure-based filtering 234 using the web document structure information proposed by the present invention, that is, comparing the inbound web document with the learning model 232 and referring to the harmful URL database 236. The web document is divided into a harmful site 240 and a harmless site 250.

Of course, the harmful site will be blocked and will not be displayed on a user display device (not shown) such as a monitor of a computer, but the harmless site 250 will be displayed on the user display device so that the user can perform a desired task.

The web structure-based filtering method will be described in more detail below.

FIG. 3 is a diagram illustrating a concrete manner in which a filtering module according to the existing technology and a web structure information based filtering module according to the present invention interact with each other for inbound packets and outbound packets.

First, a method of initially filtering using the URL-based filtering module 330 through the outbound path 370 will be described. The URL-based filtering module is a module that performs URL filtering in the basic filtering blocks 120 and 220 shown in FIGS. 1 and 2.

Once on a network terminal running a web application 310, a user enters access information to access a web application of his or her desire. Usually you will be able to do this through an Internet web browser such as Internet Explorer or Netscape.

This access information, the URL, is extracted during outbound to retrieve the corresponding web document and the information is compared with the entered URL by referring to a database (not shown) already built by the URL based filtering module. To determine whether to block. For example, if a user attempts to access a website such as www.sexygirl.com, the "sexygirl" may be compared with various harmful URL information in the existing database before connecting to the server where the web document provided by the website exists. If the word "sexy" exists in the database and the comparison results, the connection to the server itself is blocked.

If the suspicious information in the outbound packet is not filtered by the database by this method, the filtering is performed on the web document (inbound, 380) that has been transmitted from the server requesting the information.

As mentioned above, the filtering by the text and image-based filtering module 350 according to the existing technology is performed.

Text filtering refers to the text in the web document transmitted from the server. Similarly, if a word or combination thereof is found among the text contents, it is blocked. Because there are commonly used words that relate to the various indecent things that appear on harmful sites, you can determine the harmfulness of a web document in this way.

Similar to the above text filtering, image filtering can perform site blocking by determining how much color information commonly used on a harmful site is represented and included using a method such as using color information such as a frame. There will be.

However, this method alone raises the problem of blocking efficiency as mentioned above, and proposed filtering of inbound packets using the web structure based filtering module 360 in a more effective and efficient manner. The filtering method using the web structure-based filtering module will be described in more detail below.

Figure 4 shows an embodiment of using a harmful URL database as a filtering based on the URL structure of the existing technology and the web structure information according to the present invention.

For example, suppose that a user enters the web site URL 412 to access a harmful web site called www.newxxx.com, as shown in FIG. The www.newxxx.com is a new harmful site that does not exist in the harmful URL database 440, which cannot be blocked by existing URL filtering.

Thus, web documents corresponding to www.newxxx.com are inbound, for example, www.newxxx.com web documents can be used to navigate to another relevant harmful site, for example www.xxx.com Contains the link URL. This www.xxx.com URL exists in the harmful URL database, and the web structure-based filtering 430 detects this harmful link URL in the web document and determines that www.newxxx.com is also a harmful site. As a result, the web document is blocked from being displayed on the user display device 410.

This method shows that the blocking rate can be increased by searching all the link URLs in the web structure information and comparing them with the harmful URL database again. In practice, however, malicious sites in recent years show text that has been imaged to avoid text-based filtering. In other words, the user is text in view of the user display device, but the image is often an image of the actual web document.

To this end, text information extraction (TIE) technology research has recently been conducted, but it is not yet commercialized because it is still research level and its technology level and accuracy are very low. In addition, although many harmful images are included in the web document, the size of the web document is small. However, even if the image analysis technique is applied, the image is not filtered. And, if not only the web URL input by the user, but also the link URL does not exist in the harmful URL database, it will reveal a limit to blocking harmful sites.

FIG. 5 shows an embodiment of using a URL filtering of the existing technology and a comparison based on a harmful URL database reference and a learning model as filtering based on web structure information according to the present invention.

URL filtering 520 and web structure-based filtering 530 operate in the same manner as described with reference to FIG. 4 with reference to malicious URL database 540. Image filtering 560 and text filtering 570 are the same as described above.

In FIG. 5, the structure of the inbound web document is used to compare the structure information of the current www.newxxx.com web document with the learning model 550 that is already generated to determine whether to block. In addition to web structure-based filtering, filtering using a learning model is added to further enhance the efficiency and effectiveness of blocking harmful sites. Hereinafter, a specific example of blocking a harmful site by generating a learning model in FIG. 6 will be described in detail.

6 shows a harmful site determination process using a learning model generation process and the generated learning model according to the present invention.

First, the learning model generation process 600 will be described. The training model is previously site-specific HTML to generate a training model for different data sets, namely a large number of (e.g., here N) harmful sites 610 and another large (e.g., here M) harmless sites. For analysis 630. The analysis process extracts features common to harmful sites and extracts features characteristic of harmless sites to generate feature files (eg, feature.dat) (640).

At this time, the structure information such as the number of pop-up windows, background color, frame number, frame color, image number and / or text image number is utilized as web structure information. The feature file generated by the extracted values respectively displays the information about the site. As shown in 640, the information on the web structure information is displayed as a series of numbers, and the resulting harmful / harmless judgment value is determined by each. Each structural information content 642 of the feature file is indicated by 1 / -1.

The generated feature file is made into a learning model through machine learning 650 by a pattern classification technique. In FIG. 6, SVM (Support Vector Machine) is used as a pattern classification technique, and clustering and SOM ( Self Organizing Map) and the like, but without being limited thereto, all possible pattern classification techniques may be used to generate the learning model.

By the pattern classification technique, a model file such as model.dat is generated 660, thereby securing a comparison reference file through a learning model.

Next, when the harmful site determination process through the generated learning model is assumed, once the harmful site is inputted as 682, HTML analysis 684 identifies structural information about the inputted site. Features are extracted as shown in 686 through the structure information. In this case, no value (1 or -1) is assigned to the beginning of the feature file because no judgment has been made yet.

The extracted features will be filed (not shown) for ease of comparison as well as creating a learning model. The feature (file) extracted in this way and the model file by the learning model are compared with each other to determine whether there is a harm (688), and the result of blocking or not (690) is produced. By using various web structure information to block harmful sites, more accurate and effective blocking effect can be expected.

7 shows a harmful site blocking apparatus 700 using web structure information according to the present invention.

Once the web document 710 is transmitted to the network terminal, it is received by the web document receiver 720.

The web document receiving unit 720 sends the received web document 710 to the basic filtering unit 720 which primarily performs text filtering and / or image filtering. The basic filtering unit is optional. That is, although used together with the web structure information analysis unit 740 proposed in the present invention, it is possible to derive a more effective filtering result, but this basic filtering unit is not necessary. That is, in another embodiment, when the inbound web documents do not have the basic filtering unit, they are delivered to the web structure information analyzing unit.

When the web structure information analysis unit 740 determines whether the site is harmful or harmful through the harmful URL database comparison unit 760 that compares the learning model or the link URL according to the present invention, the web document transmission unit ( The 770 transmits the web document to the user display device 780 such as a monitoring device that can be viewed by the user. The user display device may include all kinds of display devices (LCD, CRT, built-in laptop computer monitor, PDA screen, mobile phone screen, etc.).

The web structure information analyzing unit 740 may be broadly divided into a learning model comparing unit 750 and a harmful URL database comparing unit 760. There may be two comparison units at the same time or only one of them.

First, the harmful URL database comparator 760 determines the blocking URL by determining the link URL in the inbound web document 710 as described above. Of course, in this case, as mentioned in detail above, it will refer to an already established harmful URL database (not shown).

The learning model comparison unit 750 includes a web structure information extraction unit 752, a file generation unit 754, a learning model generation unit 756, and a file comparison unit 758.

The web structure information extractor 752 extracts the web structure information of the file transmitted from the web document receiver 720 (or the basic filtering unit 730, if necessary).

Similarly, the file generator 754 generates the structure information extracted by the web structure information extractor 752 as a single file during an operation for site extraction, and compares the generated file with a learning model. The comparison unit 758 determines whether or not blocking is performed through comparison with the learning model.

The training model generator 756 is used to initially generate a training model. The training model generator 756 extracts a feature through the web structure information extractor 752, generates a file in the file generator 754, and then generates a training model. The model generates the model file using the pattern classification technique. The presence of this model file is the reference file for comparison. Of course, it is obvious that the web document added again after the comparison is performed by the file comparison unit 758 on the web document input each time can be used to generate the learning model.

As described above, the present invention has been described based on the preferred embodiments, but these embodiments are intended to illustrate the present invention, not to limit the present invention, so that those skilled in the art to which the present invention pertains can practice the above without departing from the technical spirit of the present invention. Various changes, modifications or adjustments to the example will be possible. Therefore, the protection scope of this invention will be limited only by the appended claims, and should be construed as including all changes, modifications or adjustments.

As described above, the present invention is applied to a terminal connected to a network, and when accessing an Internet web site from such a terminal, by using the structural information of the web site, it is possible to accurately determine the presence or absence of pornography to block harmful sites. Through the present invention, it is possible to more effectively and effectively block harmful sites such as unwanted sexually explicit sites, and it is possible to more easily control not only adults but also young children and adolescents who are vulnerable to sexually explicit sites. You will be able to.

Claims (13)

As a method of blocking harmful sites using web structure information, Generating a learning model from web structure information of a plurality of harmful sites and harmless sites; Comparing an inbound web document with the generated learning model; And determining blocking based on a result of comparison with the learning model. The method of claim 1, wherein after generating the learning model, And performing text filtering and / or image filtering on the inbound (inbound) web document and URL filtering on the URL information input by the user. The method of claim 1, wherein generating the learning model comprises: Extracting values for the number of pop-up windows, the background color, the number of frames, the frame color, the number of images, and / or the number of text images from the web structure information of the plurality of harmful and harmless sites; Making the extracted value into a file; And The harmful site blocking method using the web structure information, characterized in that it comprises the step of generating a learning model using the pattern classification technology of the generated file. The method of claim 3, wherein the pattern classification technique uses a support vector machine (SVM), clustering, or a self organizing map (SOM). As a method of blocking harmful sites using web structure information when accessing a website, Establishing a malicious URL database containing known harmful web sites and links; Comparing the link URL in the inbound web document with the URL data in the generated malicious URL database; And And determining the blocking of the web document according to the comparison result. The method of claim 5, wherein after the malicious URL database is established, And performing text filtering and image filtering on a web document which is URL-filtered and inbound with respect to URL information input by a user. The method of claim 5, After comparing the URL data in the malicious URL database, the method further comprises the step of determining blocking by comparing the inbound web document with an already generated learning model, wherein the malicious URL database building step is harmless to a plurality of harmful sites. The harmful site blocking method using the web structure information, characterized in that it further comprises generating a learning model from the web structure information of the site. The method of claim 7, wherein generating the learning model, Extracting values for the number of pop-up windows, the background color, the number of frames, the frame color, the number of images, and / or the number of text images from the web structure information of the plurality of harmful and harmless sites; Making the extracted value into a file; And The harmful site blocking method using the web structure information, characterized in that it comprises the step of generating a learning model using the pattern classification technology of the generated file. The method of claim 8, wherein the pattern classification technique uses a support vector machine (SVM), clustering, or a self organizing map (SOM). As a harmful site blocking device using web structure information, A web document receiving unit which receives a web document transmitted from the outside; A web structure information analyzer for analyzing structure information of the web document transmitted from the outside; And And a web document transmission unit for transmitting a web document, which is found to be a non-hazardous site through the web structure information analysis unit, to a monitoring device that can be viewed by a user. The web site of claim 10, further comprising: a basic filtering unit configured to perform URL filtering on the URL information input by the user, text filtering, and image filtering on the received web document. Blocking device. The method of claim 10, And the web structure information analyzing unit includes a learning model comparing unit and / or a harmful URL database comparing unit. The method of claim 12, wherein the learning model comparison unit A web structure information extraction unit for extracting values for the number of pop-up windows, the background color, the number of frames, the number of frames, the number of images, and / or the number of text images in the web document; A file generator which makes the extracted web structure information into one file; A learning model generation unit which generates a learning model using the generated file as a pattern classification technique; And The harmful site blocking device using the web structure information, characterized in that it comprises a file comparison unit for comparing the learning model and the file generated by the file generation unit.

Images