JP2008204425A - Processing omission decision program for similarity analysis of url - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that when a PC on which Web filtering by a URL list is mounted performs access to a page using Ajax or the like, huge URL access permission requests are generated, and the display of a page is extremely delayed. <P>SOLUTION: When large amounts of URL access permission requests are made from one Web page whose access has been already permitted, the next similarity decision is made, and the number of items of access requests is reduced to quickly achieve page display processing. Then, URL request transmission circumstances are monitored, and when it is detected that a large amount of requests have been transmitted is detected, the similarity of URL requested in the past from the same access source page with the current page is decided. Access permission decision logic is bypassed to the URL decided as the similar URL. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a processing omission determination program and apparatus based on URL similarity analysis.

Some web pages on the Internet constitute the entire page by automatically accessing and reading many other URLs. In particular, recently, Web applications that actively use a technology called Ajax (Asychronous JavaScript (registered trademark) + XML) (see, for example, Non-Patent Document 1) are becoming popular, and synchronization between the transmission side and the reception side is becoming popular. There is a tendency to increase the number of Web pages that send a large number of URL access requests at once without requesting them.
“A New Approach to Web Applications”, [online], Jesse James Garrett, February 18, 2005, [Search January 25, 2007], Internet <URL http: // www. adaptablepath. com / publications / essays / archives / 000385. php>

  In the above case, when Internet filtering is performed using the URL list, if an access permission request is transmitted to the filtering determination server for each URL and the determination result is received, the page is displayed on the browser. The display state in which a part of the data is missing from the whole continues for a long time, or the processing load increases on the filtering determination server side and the client PC side, resulting in a decrease in performance.

  In addition, when storing Internet access history for information security and other purposes, many unnecessary URLs are output to the log, which increases log file storage space and increases log analysis efficiency. It becomes a factor of decreasing.

  The present invention provides the following solutions in view of the above problems.

(1) A computer program that processes determination of similarity between URLs in response to a URL access request to a Web page transmitted from a user terminal,
On the computer,
Determining whether a plurality of URL access requests are transmitted; determining a similarity between the plurality of URLs when it is determined that the plurality of URL access requests are transmitted;
A step of omitting the similar URL from the processing target of the subsequent program when it is determined that it is similar by the determining step;
A computer program that runs

(2) The computer program according to (1), wherein, in the determining step, when there are a predetermined number or more of sockets in a transmission standby state, it is determined that the plurality of URL access requests are transmitted.

(3) In the determining step, it is determined that the plurality of URL access requests are transmitted when a predetermined number or more of HTTP requests and HTTP responses are detected in a predetermined time. (1) Or the computer program as described in (2).

(4) The step of determining the similarity includes:
A query character string is extracted from the HTTP request header data between the plurality of URLs, and when the patterns of the query character strings match, it is determined that the plurality of URLs are similar (1) Thru | or the computer program as described in (3).

(5) The step of determining the similarity includes:
Determining that the plurality of URLs are similar when the identifiers of the parameters included in the query character string, the order of the identifiers, and the data types of the identifiers are partially or completely matched. ).

(6) The step of determining the similarity further includes a step of referring to page link source reference information in the HTTP request header and verifying that the request source page is a display-permitted page. A computer program according to any one of (1) to (5).

(7) The method further comprises a step of accessing a URL corresponding to the URL access request.
The omitting step includes
The computer program according to any one of (1) to (6), wherein the access to the URL determined to be similar is omitted.

(8) The method further includes a step of storing an access history to the Internet accessed in response to the URL access request,
The omitting step includes
The computer program according to any one of (1) to (7), wherein storing of an access history for a URL determined to be similar is omitted.

(9) A similarity determination apparatus that processes determination of similarity of a URL in response to a URL access request to a Web page transmitted from a user terminal,
Means for determining whether a plurality of URL access requests are transmitted;
Means for determining similarity between the plurality of URLs when it is determined that the plurality of URL access requests are transmitted;
Means for omitting the similar URL from the processing target of the program when it is determined by the determining means to be similar;
A similarity determination apparatus comprising:

That is, according to the above solution, when a plurality of URL access permission requests are issued from one Web page that is already permitted to access, the following determinations a) and b) are made to the server. Page display processing can be speeded up by reducing the number of access requests.
a) The transmission status of URL requests is monitored, and when it is detected that a large number of requests are being transmitted, the similarity between the requested access destination URL and a previously requested URL is determined.
b) The access permission determination logic is bypassed for URLs determined to be similar URLs.

Here, in monitoring the URL request transmission status, the socket monitoring module monitors one of the following conditions c) and d), and confirms that “a plurality of URL access requests are being transmitted” in a short time. to decide.
c) The number of sockets waiting to be sent is a predetermined number or more d) The number of requests and responses is a predetermined number or more

  When it is determined that a plurality of access requests are transmitted, the character string of the connection destination URL is sent from the same access source page to the module that performs similarity determination of the current page with respect to the URL requested in the past, and the similarity Judgment is made. Further, a character string in a URL query string (query character string) when there are a plurality of access requests is patterned, and similarity determination is performed by pattern matching. Then, the page link reference information (for example, the reference information described by the identifier “Referer”), which is a data item in the header of the HTTP request, is checked to verify that the request source page is a display-permitted page. Check. Finally, when it is determined that the URL is similar, it is omitted from the processing target of the application software incorporating this program.

  According to the present invention, the following operational effects can be obtained.

  (1) In the case of a Web page using technology such as Ajax, there is a tendency for URL requests to occur frequently. When URL filtering is performed on such a page, the server is inquired for each URL, so a delay occurs in loading the page and performance is degraded. Therefore, a logic for analyzing the similarity of the URL is applied, and the similarity determination module analyzes whether or not there is a similarity relationship with the URL inquired in the past. For URLs that are determined to be similar based on the analysis results, the harmfulness of URLs that are in a similar relationship can be applied to greatly reduce the number of inquiries and the time required to load the page. In addition, the processing capability can be improved by reducing the number of inquiries.

  (2) There are cases where a large amount of unnecessary URLs are stored in an apparatus that counts access logs of Web pages. By applying the URL similarity determination module to such a situation, it is possible to greatly reduce the storage capacity of the log file and improve the log analysis efficiency.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. In the following embodiments, the similarity determination apparatus described above is described as a “harmful determination server” realized on a server.

[System overall configuration diagram]
FIG. 1 is a diagram showing an overall configuration of a system 1 according to an example of a preferred embodiment of the present invention.

  The system 1 constituting the present invention is a system for determining whether or not to omit a URL based on URL similarity analysis from a processing target. In this system, a WWW (World Wide Web) server 10 and a harmfulness determination server 40 are connected to a user terminal 20 via a communication line 30. Although the server 10 and the harmfulness determination server 40 are illustrated as if they are different servers in FIG. 1, they may be configured by the same server. Moreover, there is no restriction | limiting in the number of hardware of a server, You may comprise by 1 or multiple as needed.

  The WWW server 10 stores information such as documents and images (also referred to as Web pages), and responds to software requests such as a Web browser of the user terminal 20 through a communication line 30, for example, a network such as the Internet. A function of transmitting such information is provided. A web page group such as a personal or company home page or a place on the Internet where a web page group is placed is called a web site. There may be one or more Web sites on the WWW server 10.

  The harmfulness determination server 40 is a list (hereinafter referred to as a blacklist 41) of URLs of websites that are determined to be harmful by an administrator who monitors information on the Internet by acquiring information held by each WWW server 10. Remember. When receiving a request for determining whether or not the URL of the Web page is harmful from the user terminal 20, the blacklist 41 is referred to and the determination result (for example, the site of the URL for which the determination request is received in the list is harmful) A function of transmitting to the user terminal 20 a determination result of being a website). The criterion for determining the harmfulness is, for example, whether or not the web page is related to adults, violence, dating, anti-social acts, etc. that the child does not want to show. Note that the URLs of websites that are determined not to be harmful may be stored in another list (hereinafter referred to as white list 42) and referred to together with the black list 41. Access permission is transmitted to the user terminal 20 for the Web site in the white list 42.

  The communication line 30 is generally the Internet and is not limited to being realized by wire, but may be various as long as it meets the technical idea of the present invention, such as realized by a wireless LAN via an access point. Realized by communication technology.

  The user terminal 20 may be a communication terminal other than a so-called computer such as a PDA (Personal Data Assistant) in addition to a PC (Personal Computer).

[Function blocks of terminals and servers]
FIG. 2 is a functional block diagram of the user terminal 20 and the harmfulness determination server 40 according to an example of the preferred embodiment of the present invention.

  The user terminal 20 includes a transmission / reception unit 22 that has a communication I / F (communication interface), receives a Web page from the WWW server 10, and transmits a request by a Web browser. In addition, the user terminal 20 includes a display device 24 that has a display device such as a CRT or a liquid crystal display and displays a Web page or the like. Further, the user terminal 20 has input means such as a keyboard and a mouse, and includes an input unit 21 for inputting a URL in a Web browser and clicking a link displayed on the Web page.

  In addition, the user terminal 20 has a control unit 23, a socket monitoring module 201 that detects that a URL access request is transmitted, and makes it impossible to browse inappropriate information on a website or web page, or only useful information. A filtering module 202 that enables browsing of URLs, a URL similarity determination module 203 that analyzes and determines the similarity of URLs, and a cache module 204 that stores harmful determination results. The harmful determination result is a list of harmful websites and URLs of websites that are permitted to be accessed. Typically, the modules of the control unit 23 are realized by a computer program executed by a CPU (Central Processing Unit).

  The harmfulness determination server 40 includes a control unit 45 and includes a determination unit 401 that receives a determination request for a URL and determines whether a website having the URL is harmful and whether access is permitted. The determination unit 401 includes a black list 41 and a white list 42. It should be noted that only the black list 41 may be provided so that access is permitted except for websites in the black list 41. In addition, the harmfulness determination server 40 may include a transmission / reception unit, and may further include an input unit and a display unit, but the illustration is omitted here.

[URL filtering process flow]
FIG. 3 is a flowchart of URL filtering processing according to an example of the preferred embodiment of the present invention.

  First, in step S101, the control unit 23 of the user terminal 20 uses the input unit 21 to input a URL from the user and click a link (connection destination URL) displayed on the Web page on the Web browser. Accept input. When the new Web page of the movement destination uses a technology such as Ajax, the socket monitoring module 201 receives an HTTP (Hyper Text Transfer Protocol) request including a URL access request for automatically accessing a plurality of URLs. Here, the HTTP request is data transmitted from the terminal (client) to the server.

  Note that a URL access request that automatically accesses a plurality of URLs is to request a transmission of a Web page by accessing a plurality of URLs transmitted from the WWW server 10, for example, the user terminal 20 When a map is received and displayed from a website that provides a map, a plurality of URLs are transmitted using a technique such as Ajax, and a plurality of maps adjacent to the displayed map are downloaded from the website in advance. That is. By doing this, when the user moves to a place other than where the map is displayed, if it is not downloaded in advance, the display will be delayed by the download time. The map to be displayed can be displayed and the map can be changed smoothly.

  Next, in step S102, the socket monitoring module 201 monitors whether a large number of URL access requests are transmitted within a predetermined time. As a monitoring method, there are a method of monitoring the number of sockets in a transmission standby state and a method of monitoring the count of requests and responses. Each method will be specifically described later with reference to FIG. A socket is a network address that combines an IP address and a plurality of port numbers as subaddresses for simultaneous communication with a plurality of computers.

  Next, in step S103, when the socket monitoring module 201 detects that a large number of URL access requests are transmitted (S103: YES), the URL similarity determination module 203 performs processing. Further, when the transmission of a large number of URL access requests is not detected (S103: NO), a request for determining the harmfulness of the URL of the HTTP request destination (hereinafter referred to as the request destination URL) is transmitted to the harmfulness determination server 40.

  Next, in step S104, the URL similarity determination module 203 analyzes the similarity between the request destination URLs. The similarity analysis is performed by patterning a character string in a URL query string and performing an analysis based on pattern matching. Specifically, it will be described later with reference to FIG.

  Next, in step S105, if the cache module 204 does not have an access log of a URL having a similar relationship to the request destination URL (S105: NO), the filtering module 202 issues a determination request for the request destination URL. It transmits to the harmfulness determination server 40 via the transmission / reception unit 22 of the user terminal 20. Further, when there is an access log of a URL having a similar relationship with the request destination URL (S105: YES), the process proceeds to a harmfulness determination step.

  Next, in step S106, the filtering module 202 refers to the result of the harmful determination of the URL having a similar relationship with the cache module 204, and determines whether or not to permit access to the request destination URL. The harmful determination result also reflects the access setting of the user terminal 20 by the user. In other words, for example, a non-permitted access by a user such as a shopping site that a child does not want to use may be set / added to the Web site determined that the harmful determination server 40 is not harmful.

  Next, in step S107, the harmfulness determination server 40 receives the harmfulness determination request from the user terminal 20, and determines whether the website including the request destination URL is harmful using the blacklist 41. Further, the determination result (hazardous or not harmful) is transmitted to the user terminal 20. Then, the filtering module 202 of the user terminal 20 receives the determination result via the transmission / reception unit 22.

  Next, in step S108, when it is determined that the website including the request destination URL is harmful, and when the access is not permitted by the user although it is not harmful, the web page for the request destination URL is displayed. To block.

  Alternatively, if the Web site including the request destination URL is not harmful, and if it is not harmful and access permission, the filtering module 202 sends the HTTP request to the WWW server 10 by the transmission / reception unit 22 of the user terminal 20. Send. The WWW server 10 receives the HTTP request and transmits an HTTP response to the HTTP request (including Web page content data) to the user terminal 20. The cache module 204 receives an HTTP response (including Web page content data) via the transmission / reception unit 22, and displays the Web page on the display unit 24 of the user terminal 20. The HTTP response is data returned from the server to the terminal (client) in response to the HTTP request. Further, a cache (access log) is stored in the cache module 204. Since the cache here is a cache of information for which URL determination has been made in the past, the HTTP response of the WWW server 10 is not cached, but the determination result of the harmfulness of the URL of the WWW server 10 remains as a cache.

[System Overview of URL Filtering Processing]
FIG. 4 is a diagram showing a system outline of a conventional URL filtering process.

  In the conventional system, URL filtering processing is performed on the HTTP request from the Web browser by the filtering module 202 provided in the control unit 23 of the user terminal 20. The filtering module 202 refers to the determination result cache (access log) of the cache module 204 for the same URL as the URL included in the HTTP request. If there is no cache, a URL determination request is transmitted to the harmful determination server 40. Then, the harmfulness determination server 40 determines the harmfulness of the URL, and transmits the determination result of the URL to the filtering module 202 of the user terminal 20.

  Here, when there is an HTTP request including a large number of URL access requests and URL filtering processing is performed, the cache module 204 is referred to for each URL, and the same URL as the URL included in the HTTP request is further stored in the cache module 204. If it is not, the harmfulness determination request for each URL is transmitted to the harmfulness determination server 40.

  However, in the conventional system as described above, making a request for determining the harmfulness of each URL causes factors such as an increase in processing load on the harmful determination server 40 and the user terminal 20 and a decrease in performance. On the Web browser, a display state in which some data is missing from the entire Web page may continue. In addition, when the cache (access log) is saved by the cache module 204, many unnecessary URLs are output to the log, thereby increasing the log file storage space and further reducing the log reference efficiency. become.

  FIG. 5 is a diagram showing a system outline of URL filtering processing in which the conventional system of FIG. 4 is provided with a URL similarity determination module 203 according to an example of a preferred embodiment of the present invention.

  In this system, an HTTP request including a plurality of URL access requests is analyzed by the URL similarity determination module 203, and the cache module 204 is referred to for URLs having a similar relationship. In this way, since the harmfulness is determined with reference to the cache of the cache module 204 for each URL, the inquiry to the harmfulness determination server 40 is omitted. If the cache module 204 does not have a URL having a similar relationship, a request for determining the harmfulness of the URL having a similar relationship is transmitted to the harmful determination server 40. Next, the harmfulness determination server 40 determines the harmfulness of the URL requested for determination, and transmits the determination result to the user terminal 20.

  By doing so, it is possible to perform the processing accompanying the determination request more efficiently than the conventional system in which the processing is performed for each individual URL. Further, when the cache (access log) is stored by the cache module 204, only the URL having a similar relationship is output to the log, so that the log file storage space can be reduced and the log reference efficiency can be further increased. The access log will be described later with reference to FIG.

  FIG. 6 is a diagram showing a system outline of URL filtering processing in which another server is provided with the URL similarity determination module 203 according to another example of the preferred embodiment of the present invention.

  For example, a user terminal 20 connected to a company network cannot directly connect to a network such as the Internet, and therefore always passes through a server in an in-house network (a server in an intranet). Therefore, URL filtering processing can be performed by a server in the in-house network. This eliminates the need for URL filtering processing for each user terminal 20. It should be noted that even if it is not an in-house network but a school or a home, it can be performed in the same manner as long as it is an external access network via a server.

[Monitoring a large number of URL access requests by the socket monitoring module]
Next, a method of monitoring “a large number of URL access requests” by the socket monitoring module 201 will be described. The socket monitoring module 201 detects that a large number of URL access requests are transmitted at a time. The detection method includes (1) a method of monitoring the number of sockets waiting for transmission using WSPSelect (described later), and (2) a method of monitoring the count of requests and responses. A specific example in which a large number of URL access requests are transmitted at once is a Web page using a technique such as Ajax. For this reason, a case where a large number of URL access requests are transmitted is also referred to as an Ajax-compatible filtering mode.

  Ajax can be applied to Web search, for example, to search in real time by performing a search in the background while the user inputs a key, which was previously performed after input confirmation. To. In this way, there is a tendency to increase the number of Ajax specification Web pages that transmit a large number of URL access requests at a time without requiring synchronization between the transmission side and the reception side.

  A method of monitoring the number of sockets in a transmission standby state using WSPSelect (1) will be described. FIG. 7 is a flowchart for monitoring the number of sockets in a transmission standby state using WSPSelect of the socket monitoring module 201 according to an example of the preferred embodiment of the present invention. WSPSelect is one of the functions of the WinSock API in the Microsoft Windows (registered trademark) OS, and is a module for examining a socket. For details, refer to “WSPSelect”, [online], [Search February 2, 2007], Internet <URL http: // msdn2. Microsoft. com / en-us / library / ms742289. See aspx>. As described above, the socket is a network address that combines an IP address and a plurality of port numbers as subaddresses for simultaneous communication with a plurality of computers.

  First, in step S201, the socket monitoring module 201 uses WSPSelect to obtain the number of sockets in a transmission standby state.

  Next, in step S202, the socket monitoring module 201 determines that it is a normal URL access request when the obtained number of sockets in a transmission standby state is less than a predetermined number (for example, 2), and performs URL filtering processing. Do.

  Next, in step S203, when the number of sockets in the transmission standby state is equal to or larger than a predetermined number (for example, 2) in step S202, an HTTP request by a web page using a technique such as Ajax, that is, a large amount of It is determined that a URL access request has been transmitted (Ajax-compatible filtering mode). Then, URL filtering processing including the URL similarity determination module 203 is performed for each socket.

  A method of monitoring the request and response count in (2) will be described. For sockets using the same IP address, the number and order of requests and responses are counted, and if a request is continuously received within a certain time without receiving a response, a large amount The URL access request is regarded as being transmitted (Ajax-compatible filtering mode), and the URL filtering process provided in the URL similarity determination module 203 is performed. By doing so, it is possible to provide an auxiliary method to the method of monitoring the number of sockets in the transmission standby state using the above WSPSelect. As a main service to which this method is applied, there is an input assist function for supplementing user input that estimates and displays character string candidates when a user inputs a search character string.

[URL similarity analysis example]
FIG. 8 is a diagram showing an example of URL similarity analysis according to the preferred embodiment of the present invention.

  First, in step S301, from the HTTP request, the URL access request part “GET / mt? N = 404 & v = w2.12 & x = 58202 & y = 25812 & zoom = 1 HTTP / 1.1” is used, and the query character string “? N = 404 & v = w2.12 & x = 58202 & y = 25812 & zoom = 1 ”is extracted.

  Next, in step S302, each parameter of the query character string is divided by “? N =”, “& v =”, or the like. Here, “? N =”, “& v =”, etc. are parameter identifiers (id).

  Next, in step S303, the type of character string following each identifier is determined. Here, the type (Type) is defined as f: numerical value, c: character string, and e: “other”. For example, in “? N = 404”, 404 is a numerical value, so “? N =” is defined as “Type = f”. In “& v = w2.12”, w2.12 is a character string and a numerical value, and therefore “other”, and “& v =” is defined as “Type = e”.

  Next, in step S304, the query character string analysis results, such as the identifier and character string type (Type) as parameters (id), and their appearance order, are cached in the cache module 204 as URL similarity determination criteria. And used for subsequent similarity determination.

Note that the determination that there is similarity is limited to the case where both of the following conditions are satisfied among a plurality of URLs.
(1) URLs that do not contain character strings after “?” Are completely matched (2) Similarity criteria (id, Type, appearance order) are completely matched

  FIG. 9 is a diagram showing an HTTP request according to an example of the preferred embodiment of the present invention. If the above-mentioned URL similarity analysis is performed, it can be determined that (a), (b), and (c) are in a similar relationship. Specifically, when a predetermined number (for example, 2) or more sockets exist when the request (a) occurs, the similarity determination criterion obtained by analyzing the query character string by the URL similarity determination module 203 Is generated.

  Next, at the time of the request (b), since it is already determined that the request is a large number of URL access requests, the query character string is analyzed and compared with the similarity determination criterion generated previously in (a). Here, since the types of the query character strings of (a) and (b) are the same, it is determined that they are similar to the request of (a). The same applies to (c). Specifically, in (a), (b), and (c), the numerical values following “& x =” are different, but all are numerical values, and it can be determined that the Type is the same, and it is determined that there is a similarity relationship. The

  Next, when the request (d) occurs, if it is the same IP address as (a), (b), and (c), it is determined that the URL has no similar relationship. If it is a different IP address, the query character string is analyzed only when it is determined that the request is a large number of URL access requests, and a new similarity criterion is generated.

  10 and 11 are diagrams showing examples of HTTP requests including URL access requests having a similar relationship. In FIG. 10, the URLs from the first to third lines are in a similar relationship. In addition, the 8th and 9th lines are in a similar relationship. In FIG. 11, the first line, the fifth line, and the sixth line are in a similar relationship.

[Check Referer]
FIG. 12 is a flowchart including a Referer check in the URL filtering process including the URL similarity determination module 203 according to an example of the preferred embodiment of the present invention. Referer is an item included in the header of the HTTP request, and is data including the URL of the link source Web page when the link of the Web page is selected and moved to another Web page.

  First, Steps S 401 to S 405 are the same as the large number of URL access request monitoring by the socket monitoring module 201 and the URL filtering processing of the URL similarity determination module 203. In step S406, it is referred whether there is a Referer in the header of the HTTP request.

  Next, in Step S407, when the Referrer is included in the header of the HTTP request (S406: YES), the harmful determination of the URL included in the Referer is performed. By doing so, it is possible to discover whether data (URL) acquired in the background using a technology such as Ajax in an HTTP request is linked to a harmful website.

  Next, in step S408, when there is no Referer in the header of the HTTP request (S406: NO), the result of analyzing the query character string (similarity criterion) is determined to be harmful.

[Access log log file counting method]
FIG. 13 is a flowchart in the case where the URL similarity determination module 203 is applied to an apparatus that counts access logs according to an example of the preferred embodiment of the present invention. Since various methods are conceivable for installing the access log totaling device, here, an application example of the URL similarity determination module 203 will be described, focusing on only the totaling method.

  First, in step S501, the control unit 23 of the user terminal 20 acquires an access log (URL) of a Web page. In the Web browser, an access log (URL) of a plurality of data that has been moved to a new Web page and acquired in the background using a technique such as Ajax is acquired one by one.

  Next, in step S502, the URL similarity determination module 203 refers to the cache module 204 and performs the similarity analysis of the URL of the acquired access log. Here, URL similarity analysis between a plurality of acquired access logs may be performed.

  Next, in step S <b> 503, the URL similarity determination module 203 refers to whether or not the cache module 204 has a URL having a similarity relationship with the acquired access log URL.

  Next, in step S504, if there is a URL similar to the URL of the acquired access log in the cache module 204 (S503: YES), the acquired access log is not saved.

  In step S505, if there is no URL similar to the acquired access log URL in the cache module 204 (S503: NO), the URL obtained by similarity analysis of the acquired access log URL is stored. . As described above, when it is determined that there is a similarity with a URL that has already been saved, the URL is not newly saved as an access log, and is saved when the similarity is not recognized. An example of specific access log storage will be described with reference to FIG.

  In the example of FIG. 14, the upper part is an access log (URL) of a newly acquired Web page. Here, there is no access log in the cache module 204. When the URL similarity determination module 203 is applied to the acquired access log (URL), only similar URLs can be stored in the log as shown in the lower part of FIG. In this way, the log file capacity can be greatly reduced and the log file can be simplified. For example, since the URLs of the second, third, and fourth lines from the newly acquired line in FIG. 14 are in a similar relationship, only the second line URL is stored in the log. It's okay.

[Hardware configuration of server and terminal]
FIG. 15 is a diagram illustrating a hardware configuration of the WWW server 10, the harmfulness determination server 40 (hereinafter referred to as a server), and the user terminal 20 according to an example of the preferred embodiment of the present invention. Hereinafter, a server will be described as an example, but the same applies to the user terminal 20 that is a client.

  The server includes a CPU 110 constituting the control unit 101 (a plurality of CPUs such as the CPU 120 may be added in a multiprocessor configuration), a bus line 105, a communication I / F 140, a main memory 150, a BIOS (Basic Input Output System) 160, It includes a USB port 190, an I / O controller 170, input means such as a keyboard and mouse 180, and a display device 122.

  A storage unit 102 such as a tape drive 172, a hard disk 174, an optical disk drive 176, or a semiconductor memory 178 can be connected to the I / O controller 170.

  The BIOS 160 stores a boot program executed by the CPU 110 when the server is started, a program depending on the server hardware, and the like.

  The hard disk 174 stores various programs for functioning as a server and programs for executing the functions of the present invention.

  As the optical disk drive 176, for example, a DVD-ROM drive, a CD-ROM drive, a DVD-RAM drive, or a CD-RAM drive can be used. In this case, the optical disk 177 corresponding to each drive is used. A program or data may be read from the optical disk 177 by the optical disk drive 176 and provided to the main memory 150 or the hard disk 174 via the I / O controller 170. Similarly, the tape medium 171 corresponding to the tape drive 172 can be used mainly for backup.

  The program provided to the server is provided by being stored in a recording medium such as the hard disk 174, the optical disk 177, or a memory card. This program may be installed in the server and executed by being read from the recording medium via the I / O controller 170 or downloaded via the communication I / F 140.

  The above program may be stored in an internal or external storage medium. Here, in addition to the hard disk 174, the optical disk 177, or the memory card, a magneto-optical recording medium such as an MD, or a tape medium 171 can be used as the storage medium. Further, a storage device such as a hard disk 174 or an optical disk library provided in a server system connected to a communication line such as a dedicated communication line or the Internet may be used as a recording medium, and a program may be provided to the server via the communication line 30. Good.

  Here, the display device 122 displays a screen for accepting input of data by the user, displays a screen of calculation processing results by the WWW server 10 and the harmfulness determination server 40, and is a cathode ray tube display device (CRT). And a display device such as a liquid crystal display (LCD).

  Here, the input means accepts input by the user, and may be configured by a keyboard, a mouse 180, and the like.

  The communication I / F 140 is a network adapter that enables the server to be connected to a terminal via a dedicated network or a public network. The communication I / F 140 may include a modem, a cable modem, and an Ethernet (registered trademark) adapter. The user terminal 20 may have the same configuration as the server, but it goes without saying that the user terminal 20 may have a minimum hardware configuration necessary for the terminal.

  In the above example, the hardware configuration of the server has been mainly described. However, the functions described above can also be realized by installing a program in a computer and operating the computer as a server. Therefore, the functions realized by the server described as an embodiment in the present invention can be realized by executing the above method by the computer, or by introducing the above program to the computer and executing it. It is.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

1 is a diagram illustrating an overall configuration of a system 1 according to an example of a preferred embodiment of the present invention. It is a functional block diagram of the user terminal 20 which concerns on an example of suitable embodiment of this invention, and the harmfulness determination server 40. It is a flowchart of the URL filtering process which concerns on an example of suitable embodiment of this invention. It is a figure which shows the system outline | summary of the conventional URL filtering process. It is a figure which shows the system outline | summary of the URL filtering process provided with the URL similarity determination module 203 which concerns on an example of suitable embodiment of this invention. It is a figure which shows the system outline | summary of the URL filtering process provided with the URL similarity determination module 203 which concerns on another example of suitable embodiment of this invention. 10 is a flowchart of monitoring a large number of URL access requests using WSPSelect of the socket monitoring module 201 according to an example of the preferred embodiment of the present invention. It is a figure which shows an example of the similarity analysis of URL which concerns on suitable embodiment of this invention. It is a figure which shows the HTTP request which concerns on an example of suitable embodiment of this invention. It is a figure which shows the HTTP request | requirement containing the URL access request which has a similar relationship based on an example of suitable embodiment of this invention. It is a figure which shows the HTTP request containing the URL access request | requirement in the similar relationship which concerns on another example of suitable embodiment of this invention. 10 is a flowchart including a Referer check in URL filtering processing including a URL similarity determination module 203 according to an example of a preferred embodiment of the present invention. It is a flowchart in the case of applying the URL similarity determination module 203 to the apparatus which totals the access log which concerns on an example of suitable embodiment of this invention. It is a figure which shows the concrete access log preservation | save based on an example of suitable embodiment of this invention. It is a figure which shows the hardware constitutions of the WWW server 10, the harmfulness determination server 40, and the user terminal 20 which concern on an example of suitable embodiment of this invention.

Explanation of symbols

1 System 10 WWW Server 20 User Terminal 30 Communication Line 40 Hazard Determination Server 201 Socket Monitoring Module 202 Filtering Module 203 URL Similarity Determination Module 204 Cache Module

Claims (9)

A computer program that processes determination of similarity between URLs in response to a URL access request to a Web page transmitted from a user terminal,
On the computer,
Determining whether a plurality of URL access requests are transmitted;
Determining the similarity between the plurality of URLs when it is determined that the plurality of URL access requests are transmitted;
A step of omitting the similar URL from the processing target of the subsequent program when it is determined that it is similar by the determining step;
A computer program that runs   The computer program according to claim 1, wherein, in the determining step, when there are a predetermined number or more of sockets in a transmission standby state, it is determined that the plurality of URL access requests are transmitted.   The determination step determines that the plurality of URL access requests are transmitted when a predetermined number or more of HTTP requests and HTTP responses are detected in a predetermined time. The computer program described. The step of determining the similarity includes:
2. The query character strings are respectively extracted from HTTP request header data between the plurality of URLs, and when the patterns of the query character strings match, it is determined that the plurality of URLs are similar. The computer program according to claim 3. The step of determining the similarity includes:
The method includes a step of determining that the plurality of URLs are similar when the identifiers of the parameters included in the query string, the order of the identifiers, and the data types of the identifiers partially or completely match. 5. The computer program according to 4.   The step of determining the similarity further includes a step of referring to page link source reference information in the HTTP request header and verifying that the request source page is a display-permitted page. The computer program according to any one of claims 1 to 5. Further comprising accessing a URL corresponding to the URL access request;
The omitting step includes
The computer program according to claim 1, wherein the access to the URL determined to be similar is omitted. A step of storing an access history to the Internet accessed in response to the URL access request;
The omitting step includes
8. The computer program according to claim 1, wherein storing of an access history for the URL determined to be similar is omitted. A similarity determination apparatus that processes determination of similarity of a URL in response to a URL access request to a Web page transmitted from a user terminal,
Means for determining whether a plurality of URL access requests are transmitted;
Means for determining similarity between the plurality of URLs when it is determined that the plurality of URL access requests are transmitted;
Means for omitting the similar URL from the processing target of the program when it is determined by the determining means to be similar;
A similarity determination apparatus comprising:

Images