JP4507206B2 - Internet information collecting apparatus, program and method - Google Patents

Description

The present invention relates to an Internet information collection apparatus, program, and method for collecting link destination web information from a web page that has been developed on a screen, and more particularly to collecting Internet information collection by analyzing a tag sentence of a web page and collecting link destination web information. The present invention relates to an apparatus, a program, and a method.

  In recent years, it has been planned to construct a web library system that saves web pages that have been published only on the Internet and that are updated and deleted in a short time, and that is open to the public. A technology called web archiving that collects and stores information resources is required.

  In conventional web archiving, Internet websites have link information to other contents in web pages as hyperlinks, so transition from one page to the next is based on this link information. Judgment is made and information on related pages is collected.

Conventionally, a web robot has been known as an apparatus that automatically collects contents on the Internet. The web robot collects link information by analyzing an HTML document of a web page and hierarchically transitions web pages. Content is collected, and the user can search and browse web page information published on the Internet in the past.
JP 2002-073609 A JP 2003-345826 A

  By the way, recent Internet contents are dynamically extended externally by embedding a script in an HTML document as an extended HTML specification that allows a user to interact with a web page known as dynamic HTML. The things that are increasing are increasing.

  For example, the selection menu displayed on the web page indicates options “1” to “3” to the user, generates URLs as different link destination information according to the options selected by the user, and generates the web page of the generated URL Transition to.

  However, in the conventional method of analyzing the HTML document of the web page and collecting the link information, it is difficult to detect the link information generated by the user's operation in the interactive content. There is a problem that the collection of information is large.

  For example, in a web page in which a code for generating a link destination URL according to an option selected by the user from the selection menu is described, the transition destination URL cannot be determined only from the code described in the web page. There is a problem that web information cannot be collected.

  Of course, it is possible for the operator to detect the link information by operating the operation button or menu selection while the web page is open, but it takes time and effort because it requires an artificial operation. There is too much problem.

An object of the present invention is to provide an Internet information collection apparatus, program, and method for automatically collecting link information generated by user operations without omission.

The present invention provides an Internet information collecting apparatus. The Internet information collection device of the present invention is
A page browsing part (browser) that acquires web pages on the Internet and expands the screen,
A page analysis unit that analyzes a web page expanded in the page browsing unit and extracts an event operation tag sentence that dynamically generates link information according to an event generated by a user operation;
An event generation unit that generates an event for the event operation tag sentence extracted by the page analysis unit;
A link information detection unit that detects and stores link destination web information from page transitions by link information generated by an event generated by the event generation unit;
It is provided with.

  Here, the link information detection unit further detects and stores link destination web information from the proxy server accessed by the page browsing unit.

  The page analysis part extracts the input sentence from the range specified by the form sentence in the tag sentence that constructs the web page, and the event generation part sequentially generates all the events defined for the input sentence. The link information is generated by the valid event.

  In addition, the page analysis unit extracts select statements that indicate options to the user and input statements that require user operation from the range specified in the form statement in the tag statement that constructs the web page, and the event generation unit An event is generated for the input sentence while changing the choice of the select sentence.

  Specifically, the page analysis unit selects an input tag that is a child tag of the form tag and a selection list that is a sibling tag of the input tag from the range specified by the form tag in the tag sentence that constructs the web page. A plurality of option tags indicating the contents of a selection list that is a child tag of the tag and the select tag are extracted, and the event generation unit generates an event of the input tag while changing the plurality of option tags in the select tag.

  Also in this case, the event generation unit sequentially generates all the events defined for the input tag, and generates link information based on the valid events therein.

  The link information detection unit detects and saves all link information of the web page that transitions when an event occurs for the event operation tag statement of the currently deployed web page, and then expands the other web page to the event operation tag. Repeat the process of acquiring and saving the link information of the web page that transitions when an event occurs for the sentence.

  The link information detection unit detects link information of a web page that transitions without page transition from communication event information notified before communication to a link destination.

The present invention provides an Internet information collection program. The Internet information collection program of the present invention is stored in a computer.
A page browsing step to acquire web pages on the Internet and expand the screen,
A page analysis step that analyzes the web page expanded in the page browsing step and extracts an event operation tag sentence that dynamically generates link information according to an event generated by a user operation,
An event generation step for generating an event for the event operation tag sentence extracted in the page analysis step;
A link information detection step for detecting and storing linked web information from page transitions by link information generated by an event generated by the event generation step;
Is executed.

The present invention provides an Internet information collection method. The Internet information collection method of the present invention includes:
A page browsing step to acquire web pages on the Internet and expand the screen,
A page analysis step that analyzes the web page expanded in the page browsing step and extracts an event operation tag sentence that dynamically generates link information according to an event generated by a user operation,
An event generation step for generating an event for the event operation tag sentence extracted in the page analysis step;
A link information detection step for detecting and storing linked web information from page transitions by link information generated by an event generated by the event generation step;
It is provided with.

The details of the Internet information collection program and method of the present invention are basically the same as those of the Internet information collection apparatus of the present invention.

  According to the present invention, a script sentence or the like can be dynamically executed according to an event generated by a user operation requiring a mouse operation or a keyboard operation on a web page button or selection list displayed on the page browsing unit. The page transition by the URL generated in the above is realized by a pseudo operation by generating an event in the application, and the link information that is transitioned by the user's operation that could not be detected by the analysis of the HTML document can be detected. It is possible to collect web information on the Internet without any omissions.

  Similarly, for linked content, it is possible to collect all information published on the Internet by detecting link information by a pseudo operation caused by the occurrence of an event by an application and repeating this. .

Furthermore, for events that cannot be generated by a pseudo operation, such as when the mouse passes, the URL information of the link destination is stored in the proxy server accessed by the browser, so the URL is acquired as link information from the proxy server. Thus, web information on the Internet can be collected without omission from the deployed web page.

Block diagram of an Internet information collecting apparatus according to the present invention. 1 is a block diagram of a hardware environment of a computer in which the Internet information collecting apparatus of FIG. Explanatory drawing of a web page in which form parts that are subject to event generation are arranged in the present invention Explanatory drawing of the HTML source sentence that constructs the web page of FIG. Explanatory drawing of the DOM tree obtained by the DOM parsing analysis of the HTML source sentence of FIG. Explanatory drawing of the event list for the event that corresponds to the A tag HTML explanatory diagram describing the event that triggers the script Explanatory drawing of the event management table of FIG. Explanatory drawing of HTML source sentence of file and event which is event execution method provided by Internet Explorer (R) Explanatory drawing of the web page which arranged the selection list and operation button which are the event generation targets in the present invention Explanatory drawing of the HTML source sentence that constructs the web page of FIG. Explanatory drawing of the DOM tree obtained by the DOM parsing analysis of the HTML source sentence in FIG. Explanatory drawing of before navigate which is event information including link destination URL notified before communication by Internet Explorer (R) Explanatory drawing of the link list table of FIG. Flowchart of Internet information collection processing according to the present invention Flowchart of link information detection process of FIG. Flowchart of link information detection processing following FIG. The block diagram of other embodiment of the internet information collection device by the present invention. Explanatory drawing of URL which cannot be extracted by the Internet information collection device of FIG. Explanatory drawing of the processing operation of the embodiment of FIG. 18 that collects URLs from proxy server files Flowchart of Internet information collection processing in the embodiment of FIG.

  FIG. 1 is a block diagram showing an embodiment of a functional configuration of an Internet information collecting apparatus according to the present invention. In FIG. 1, an Internet information collecting apparatus 10 of the present invention is configured by a computer, for example, and is connected to websites 14-1, 14-2, 14-3 as information collecting destinations via the Internet 12. Can do.

  The Internet information collecting apparatus 10 is provided with a communication control unit 16 and an application execution environment 18. The communication control unit 16 performs communication control for browsing and browsing web pages with the websites 14-1 to 14-3 via the Internet 12.

  The application execution environment 18 is realized by execution of a program by a computer, and includes a browser 20, a page analysis unit 22, an event generation unit 24, a link information detection unit 26, an event management table 28, a link list table 30, and a content acquisition unit 32. I have.

  The browser 20 provided in the application execution environment 18 of the Internet information collecting apparatus 10 functions as a page browsing unit, acquires a web page of a website, for example, the website 14-1 via the Internet 12, and develops the screen.

  The page analysis unit 22 analyzes a web page developed by the browser 22 functioning as a page browsing unit, and extracts an event operation tag sentence that dynamically generates link information according to an event generated by a user operation. .

  The event operation tag sentence is a tag sentence that constructs a radio button or a selection list that requires mouse operation or keyboard operation arranged in an HTML source sentence for constructing a web page. Specifically, <FORM > The form sentence indicated by the tag is extracted.

  The event generation unit 24 generates an event for executing a script statement that dynamically generates a link destination LRU in accordance with a user operation, on the event operation tag sentence extracted by the page analysis unit 22. In the event management table 28, a list of events that occur in the event generation unit 24 is stored in correspondence with the tags that are subject to event generation.

  The occurrence of an event for the event operation tag sentence by the event generation unit 24 is the same as when a user operates a form part such as a button or selection list arranged on a web page with a mouse or a keyboard. A pseudo operation is performed.

  The link information detection unit 26 detects the link destination web page information, that is, the link destination URL from the page transition generated by the execution of the script sentence by the event generated by the event generation unit 24, and stores it in the link list table 30.

  When the collection of link destination URLs is completed, the content acquisition unit 32 sequentially extracts URLs from the link list table 30, connects to the link destination website, acquires the web page of the website, and stores it in the database. .

  The Internet information collecting apparatus 10 of the present invention in FIG. 1 is realized by, for example, hardware resources of a computer as shown in FIG. In the computer shown in FIG. 2, a bus 101 of the CPU 100 includes a RAM 102, a hard disk controller (software) 104, a floppy disk driver (software) 110, a CD-ROM driver (software) 114, a mouse controller 118, a keyboard controller 122, and a display controller 126. The communication board 130 is connected.

  The hard disk controller 104 is connected to the hard disk drive 106 and is loaded with the Internet information collection program of the present invention. A necessary program is called from the hard disk drive 106 when the computer is started up, developed on the RAM 102, and executed by the CPU 100.

  A floppy disk drive (hardware) 112 is connected to the floppy disk driver 110 and can read and write to the floppy disk (R). A CD drive (hardware) 116 is connected to the CD-ROM driver 114, and data and programs stored on the CD can be read.

  The mouse controller 118 transmits an input operation of the mouse 120 to the CPU 100. The keyboard controller 122 transmits the input operation of the keyboard 124 to the CPU 100. The display controller 126 performs display on the display unit 128. The communication board 130 uses a communication line 132 including radio, and communicates with a website server via a network such as the Internet.

  FIG. 3 is an explanatory diagram of a web page on which form parts to be event generation targets are arranged according to the present invention. In the web page 34 of FIG. 3, a link URL 36 is arranged, and an operation button 38 and an operation button 40 are arranged below it.

  For example, when the user clicks the link URL 36 in the web page 34 with a mouse, a transition is made to the web page “a.html”. When the user presses down the operation button 38, a transition is made to the “b.html” web page, and when the user presses the operation button 40, the transition is made to the “c.html” web page.

  FIG. 4 is an explanatory diagram of an HTML source sentence for constructing the web page 34 of FIG. In the HTML source sentence 42 in FIG. 4, the link URL 36 in the web page 34 in FIG. 3 jumps to “a.html” by the function of the A tag on the 11th line. The link destination “a.html” by the A tag on the 11th line of the HTML source sentence 42 can be directly detected by analyzing the HTML source sentence 42 as in the prior art.

  The operation buttons 38 and 40 of the web page 34 in FIG. 3 are constructed by a form sentence in a range surrounded by <FORM> tags on the 12th to 15th lines in the HTML source sentence 42 in FIG. In this form sentence, for example, when the user presses down the operation button 38 on the web page 34 in FIG. 3, an “onclick” event in the 13th line of the HTML source sentence 42 is generated. The defined "" jump () "" function is called.

  In this jump function, a page transition is performed by creating a link destination URL using the id attribute value of the INPUT tag and changing the location object for the script statements on the 3rd to 8th lines.

  As described above, a tag sentence that dynamically generates a link destination URL by a script sentence when an event occurs in response to a user operation in the form sentence is a link destination URL even if the HTML source sentence 42 itself is analyzed. “B.html” and “c.html” cannot be detected.

  Therefore, in the present invention, the page analysis unit 22 provided in the application execution environment 18 of the Internet information collecting apparatus 10 in FIG. 1 analyzes the HTML source sentence 42 shown in FIG. The DOM tree 44 shown in FIG. 5 that can be operated by the unit 24 is constructed, the event generating unit 24 generates an event onclick directly from the event generating unit 24 for the INPUT tag, and the execution of the script sentence causes the link destination URL “ The page transition of “b.html” and “c.html” is performed, and the URL of the link destination is detected as the link destination information accompanying this page transition.

  Here, the page analysis unit 22 shown in FIG. 1 includes an SDK (Software Development Kit) for the browser 20, and the SDK uses an application programming interface (hereinafter referred to as “API”) as software. Is a tool to build.

  More specifically, a DOM parser that analyzes an HTML source sentence 42 of a web page developed by the browser 20 is analyzed. The HTML source sentence is analyzed by the DOM parser, and a document object having a DOM tree 44 shown in FIG. Generate a model DOM. The document object model DOM indicated by the DOM tree 44 is an API for accessing an HTML tag sentence as a collection of node objects having a tree structure.

  By generating the document object model as the DOM tree 44 shown in FIG. 5, an onclick event is directly generated from the event generating unit 24 as a program for the INPUT tag in the form tag, and the script statement is executed. It is possible to change the page by generating the URL of the link destination.

  In other words, the onclick event of the 13th and 14th line INPUT tags in the form sentence in the HTML source sentence 42 of FIG. 4 is basically a button down operation by operating the push buttons 38 and 40 shown in the web page 34 of FIG. By doing so, an event occurs, and the JavaScript function in the script statement on the 3rd to 8th lines is called.

  On the other hand, in the present invention, a DOM parser (DOM analysis means) in the software development kit SDK provided in the page analysis unit 22 is used to make a tree structure as shown in the DOM tree 44 of FIG. By constructing a document object model DOM, which is an API for accessing a set of node objects having “”, an event “onclick” is generated directly from the event generating unit 24 as a program, and a script statement is executed. “html” and “c.html” can be generated and page transitions can be made. This means that the program simulates a button pressing operation by the user.

  By the way, in this embodiment, “onclick” is generated as a valid event as an event generated for the INPUT tag on the 13th and 14th lines in FIG. There are various types according to user operations.

  FIG. 6 is an explanatory diagram of the A tag occurrence event list 46 showing the types of events defined corresponding to the A tag used for the link setting on the 11th line in FIG.

  As shown in the A tag event occurrence list 46, 17 types of events are generated only by the A tag. The types of occurrence events are defined in substantially the same manner for the INPUT tags shown in the 13th and 14th lines of FIG.

  When the A tag occurrence event list 46 is described as shown in the script activation HTML source statement 48 of FIG. 7, the script statement can be activated by generating an event onclick. For other events, When an event occurs, it will be immediately discarded.

  Such an unnecessary event in an HTML tag sentence is automatically discarded, and in the present invention, the tag is defined for an event occurrence target tag extracted from a form sentence. All events in the list are generated, and only the events defined as shown in FIG. 7 are executed.

  In this way, by generating all events in the event list defined corresponding to the event generation target tag and knowing the event that actually executed the script statement, without being aware of the specific valid event Script statements can be executed when an event occurs.

  For valid events recognized by generating all events and executing script statements, the valid events are registered in the event management table 28 corresponding to the tag names as shown in FIG. As shown in FIG. 8, the valid event corresponding to the tag name registered in the event management table 28 can be used as statistical information for event generation for subsequent tags. Processing to generate all corresponding events for the tag is performed.

  Here, when Internet Explorer® is used as the browser 20 of FIG. 1, a method called “fire event” is prepared as a method for generating an event directly by a program as shown in FIG.

  As shown in the fire event HTML source statement 50 of FIG. 9, for example, the fire event method is “on focus” which is the focus setting as shown in the third and fourth lines for all tags, and is canceled. By performing “onduller”, an event can be directly issued to all the tags. As a result, a link destination URL is generated by executing a script sentence similar to that operated by the user in a pseudo manner, and the page Transitions can be made.

  FIG. 10 is an explanatory diagram of a web page in which a selection list to be an event generation target and operation buttons are arranged in the present invention. In FIG. 10, a map display button 54 is arranged on the web page 52. A selection list 56 is provided corresponding to the map display button 54, and the selection list 56 has three options of “Tokyo”, “Kanagawa Prefecture”, and “Shizuoka Prefecture”.

  FIG. 11 is an explanatory diagram of an HTML source sentence 58 for constructing the web page 52 of FIG. In the web page 52 of FIG. 10, the jump destination when the map display button 54 is pressed is changed depending on the selection location of the selection list 56.

  That is, when the map display button 54 is pressed in a state where “Tokyo” in the selection list 56 is selected, a jump is made to “Tokyo.html” as a link destination. When the map display button 54 is pressed while “Kanagawa Prefecture” is selected in the selection list 56, a jump is made to “Kanagawa Prefecture.html” as a link destination. Further, when the map display button 54 is pressed in a state where “Shizuoka Prefecture” is selected in the selection list 56, a jump is made to “Shizuoka Prefecture.html” as a link destination.

  In the HTML source sentence 58 of FIG. 11 for constructing such a link page 52, form parts such as the map display button 54 and the selection list 56 are basically enclosed by <FORM> tags on the 13th to 20th lines. It is made by the form sentence. This form sentence includes the <SELECT> tag on the 14th line and the <INPUT> tag on the 19th line, and these tags are positioned as child tags of the <FORM> tag.

  In this example, when the map display button 54 arranged with the <INPUT> tag on the 19th line is pressed, there are three selection states in the select statement of the <SELECT> tag on the 14th to 18th lines that become sibling tags. Existing.

  For this reason, when the <INPUT> tag indicating the map display button 54 is detected, three <OPTION> tags on the 15th to 17th lines indicating three patterns of <SELECT> tags that are sibling tags are obtained. It can be analyzed that there is a selection pattern.

  Therefore, in order to generate an event artificially in the <INPUT> tag that is the map display button 54, iterative processing is performed three times, and each time the selection state of the <SELECT> tag by the <OPTION> tag is changed, An event may be generated in the <INPUT> tag.

  FIG. 12 is an explanatory diagram of the DOM tree 60 obtained by the analysis by the DOM parsing in the page analysis unit 22 shown in FIG. 1 of the HTML source sentence of FIG. 11, and the <INPUT> tag has a sibling relationship <INPUT. > Tag and <SELECT> tag exist, and the selection list 56 is constructed. Below the <SELECT> tag, the <OPTION> tag corresponds to three choices, which are selection contents “Tokyo” and “Kanagawa”. It is arranged corresponding to “Shizuoka Prefecture”.

That is, the process is basically the following procedure.
Manipulate all tags in the HTML source sentence 58 of FIG.
-Determine the <FORM> tag.
Check all child tags within the range of the <FORM> tag, such as <INPUT><SELECT>, etc., and check the status of the sibling tag selection pattern.
・ After changing the status of sibling tags according to the number of patterns obtained for <SELECT>, issue an event to <INPUT>, the current child tag, in the script statement on lines 3-10 The link destination URL is generated by execution.

  FIG. 13 shows event information including a link destination URL notified before communication when an Internet explorer (R) used in the link information detection unit 26 of FIG. 1 starts communication access to an arbitrary web page. It is explanatory drawing of the before-navigation 62 which is.

  That is, in the case of Internet Explorer (R), when browsing a web page by designating a certain URL, before navigate (Before Navigate) is known as an event notified to the website before starting communication.

  In this before navigate 62, as shown in FIG. 13, the URL of the link destination is set in the argument “url” on the third line. In the link information detection unit 26 of the present invention, the URL of the link destination is detected from the argument “url” in the event information of the before navigate 62.

  Further, if the before navigation 62 is notified, if it is left as it is, the transition to the link page is performed, and since the URL of the link destination has already been detected, the eight lines of the before navigation 62 in FIG. Communication is canceled by setting “True” to “Cancel” which is the final parameter shown in the eye. Thereby, it is possible to detect and acquire only the link destination URL without page transition.

  FIG. 14 is an explanatory diagram of the link list table 30 of FIG. 1, in which the URL of the link destination detected by the link information detection unit 26 is stored.

  Here, the collection of the link information in the present invention is performed by developing a web page by the browser 20 using a certain URL, and arranging the web page by the page analysis unit 22, the event generation unit 24, and the link information detection unit 26. For all form parts that require user operation, if a pseudo-operation is performed by an event occurrence to cause a transition to a web page and a link destination URL is acquired, refer to the link list table after that Then, the newly acquired link destination web page is expanded, and the acquisition of the link destination URL due to the occurrence of an event for the form part that requires the user's operation arranged on the web page is repeated.

  That is, in the present invention, when a link destination URL is detected from a page transition caused by an event occurrence for a form part existing in a currently developed web page, the newly detected link destination URL web page is opened and the page is opened. The collection of link information in the hierarchical direction such as acquisition of the URL of the next link destination due to the occurrence of the event of the form part is not performed, and the collection of the URL of the next link destination is repeated for each web page. If the link information in the hierarchical direction is collected, after reaching the last web page, it is necessary to return to the original hierarchy, and the processing becomes complicated.

  FIG. 15 is a flowchart of Internet information collection processing according to the present invention. In FIG. 15, after obtaining a list of URLs collected by a conventional web robot or the like in step S1, one URL is selected from them in step S2, and the browser 20 is activated in step S3 to open a web page. To do.

  The opening of the web page by the browser does not require actual screen development, and is performed as a background process in the operation of the computer as the Internet information collecting apparatus 10.

  Next, in step S4, the page is analyzed by a DOM parser or the like, and a document object model DOM having a DOM tree in which an API capable of generating an event is constructed by the event generating unit 24 is constructed. A link information detection process is executed by a pseudo operation.

  In step S6, the URL list read in step S1 is checked to see if there is an unprocessed URL. If there is an unprocessed URL, the process returns to step S2 and the same processing is repeated. When processing for all URLs is completed in step S6, the process proceeds to step S7, where a list of newly detected link destination URLs is acquired, and link information detection from step S2 is performed until there are no unprocessed URLs in step S8. Repeat the process for.

  16 and 17 are flowcharts of the link information detection process according to the present invention corresponding to step S5 of FIG.

  In FIG. 16, in the link information detection process, the tag in the HTML tag sentence is operated in step S1, and it is checked whether or not it is a non-event occurrence tag in step S2. Non-event occurrence tags include the <A> tag, <IMG> tag, <LINK> tag, etc., shown in the eleventh line of FIG. If it is a non-event occurrence tag, the process proceeds to step S3, where the link destination URL is directly detected and stored.

  On the other hand, if the tag is not a non-event occurrence tag in step S2, the process proceeds to step S4 to determine whether it is a <FORM> tag. If it is a <FORM> tag, the process proceeds to step S5 to check whether the form part is an operation button.

  If it is an operation button, the process proceeds to step S6, where it is checked whether it is an <INPUT> tag, and if it is an <INPUT> tag, the occurrence event list prepared in advance in step S7 is sequentially selected. By selecting and issuing one event, the URL of the link destination is generated and the page is changed by executing the corresponding script sentence.

  Subsequently, whether or not there is a page transition is checked in step S8. If there is a page transition, the link destination URL is acquired and stored in step S9. Note that the page transition in step S8 is, as shown in FIG. 13, in the case of Internet Explorer (R), whether or not the before navigate 62, which is event information acquired before communication, is acquired. In this case, the link destination URL is detected and stored therein.

  The processing from step S7 is repeated until all event generations are completed in step S10. With respect to all the event occurrences, only events that are actually defined in the <INPUT> tag of the HTML statement function as valid events, and link destination URLs are generated by executing the script statement.

  Next, proceeding to step 11 in FIG. 17, it is checked whether or not the form part is a selection list. If it is a selection list, the process proceeds to step S12, and all child tags <INPUT> <SELECT> etc. within the range of the <FORM> tag are operated.

  In step S13, the selection pattern of the <SELECT> tag that is a sibling of the <INPUT> tag is analyzed. In the case of FIGS. 10 to 12, there are three types of selection patterns. In step S14, the state of the sibling tag <SELECT> is changed according to the selection pattern.

  Subsequently, in step S15, one event is selected and issued for the current child tag <INPUT>, and in step S16, whether or not a page transition has occurred is checked. If there is a page transition, the link destination URL is detected and stored in step S17. Subsequently, in step S18, it is checked whether or not all events have occurred, and the processing from step S15 is repeated until all events have ended.

  Next, in step S19, it is checked whether or not all selection patterns have ended. If the selection pattern has not ended, the process returns to step S14 to change the state of the sibling tag <SELECT> to the next selection pattern. The process of S18 is repeated.

  When the process is completed for all the selection patterns in step S19, the process proceeds to step S20, where it is checked whether or not the process is complete for all tags. If not completed, the process returns to step S1 in FIG. Processing is performed for the tags, and thereafter, the processing of steps S1 to S20 is repeated until the processing is completed for all the tags.

  The present invention also provides an Internet information collecting program executed by the Internet information collecting apparatus 10 constituted by a computer, and this program has processing procedures according to the flowcharts of FIGS. It is built as a program.

  FIG. 18 is a block diagram of another embodiment of the Internet information collecting apparatus according to the present invention. In this embodiment, as a function of the link information detection unit 26 provided in the application execution environment 18, web information is obtained from page transitions based on link information generated by an event generated by the event generation unit 24 in the embodiment of FIG. In addition to the function of detecting and storing, the web information of the link destination is further detected and stored from the proxy server 64 accessed by the browser 20 functioning as a page browsing unit.

  This solves the problem that there is a URL that cannot be extracted by the function of the Internet information collecting apparatus 10 of FIG.

Here, the following URLs cannot be extracted in the embodiment of FIG.
(1) When a static link is updated by a user operation.
(2) When performing HTTP communication independently by a Java program using a Java applet or the like.
(3) When an original program such as Active X Component performs HTTP communication independently.
(4) In a case where the software development kit (SDK) is operated on a platform where the before-navigation function as shown in FIG. 13 does not exist in a Unix (R) environment or the like.

  FIG. 19 is an explanatory diagram of URLs that cannot be extracted in the embodiment of FIG. 1 in which the user updates a static link in (1). In FIG. 19, an HTML source sentence 65 describes a script portion 66 on the 3rd to 5th lines and a script sentence 67 on the 6th to 8th lines.

  The script sentence 66 performs an operation of changing the image file to “over.gif” when the cursor passes over the image by a mouse operation or the like. The “over.gif” of the script sentence 68 is acquired from the website for the first time by the user's mouse operation. However, since it is not a page transition operation, it is not generated by the before-navigated event. Therefore, in the embodiment of FIG. 1, it is not possible to detect the URL of a website having the file name “over.gif” in the full path.

  The next script sentence 67 performs the operation of returning the image file to “out.gif” when the cursor moves over or away from the image. This “out.gif” is also acquired from the website for the first time by the user's mouse operation, and since it is not a page transition operation, the before navigate event in the embodiment of FIG. 1 does not occur, and the URL cannot be acquired.

  Therefore, in the present invention, as shown in FIG. 18, when the Internet information collection device 10 uses the browser 20, the websites 14-1 to 14-3 are always accessed via the proxy server 64. In this case, in the proxy server 64, the problem is solved by focusing on the fact that the access information is stored on the file in accordance with the HTTP request from the website and the HTTP response from the website.

  That is, in the present invention, after the link information detection unit 26 completes the process of detecting and storing the link destination URL from the occurrence of the page transition by the before-navigation function, the proxy server 64 is accessed, The transition destination URL is acquired from the file information stored in the full path and stored in the link list table 30.

  FIG. 20 is an explanatory diagram of the processing operation of the embodiment of FIG. 18 for detecting and collecting URLs from the proxy server file. In FIG. 20, when a fire event 68 is generated by the cursor movement on the image based on the script sentence 68 shown in FIG. 19 in the Internet information collecting apparatus 10, an HTTP request 72 is sent from the browser 20 to the website 14 via the proxy server 64. Is sent.

  In the web site 14 that has received this HTTP request 72, the web page 74 with the file name “over.gif” is returned as an HTTP response 78 to the browser 20 via the proxy server 64.

  Here, the proxy server 64 stores the access information 76 in the file 85 when sending the HTTP request 72 to the website 14, and the access information 80 when sending the HTTP response 78 from the website 14 to the browser 20. Is stored in the file 85.

  The first line of the access information 76 saved with the HTTP request 72 stores “over.gif” as the file name, and the third line stores the domain name “domain” of the website 14. .

  Accordingly, the link information detecting unit 20 provided in the Internet information collecting apparatus 10 shown in FIG. 18 refers to the file 85 of the proxy server 64 and starts from “HTTP: //” to the file name “over.gif”. “Http: //domain/over.gif” is detected as the URL 84 of the link destination of the full path shown, and is stored as shown in the record 82 of the link list table 30.

  FIG. 21 is a flowchart of the Internet information collecting apparatus in the embodiment of FIG. In FIG. 21, the processing of steps S1 to S8 is the same as the processing according to the embodiment of FIG. 1 shown in FIG. In the embodiment of FIG. 18, after the processing of steps S1 to S8 is completed, processing for acquiring a full path URL from the proxy server 64 and registering it in the link list table is executed in step S9.

  For events that occur with, for example, a mouse that cannot be generated by a temporary operation in this way, the URL is acquired as link information from a proxy server accessed by the browser, so that it can be used on the web page developed to collect Internet information. Web information on the Internet can be collected.

  The present invention includes appropriate modifications that do not impair the object and advantages thereof, and is not limited by the numerical values shown in the above embodiments.

Claims (5)

A page browsing part that acquires web pages on the Internet and expands the screen,
A page analysis unit that analyzes the web page expanded on the page browsing unit and extracts an event operation tag sentence that dynamically generates link information according to an event generated by a user operation;
An event generation unit that generates the event for the event operation tag sentence extracted by the page analysis unit;
A link information detection unit that detects and stores link destination web information from page transitions by link information generated by an event generated by the event generation unit;
An Internet information collecting apparatus comprising:   2. The Internet information collecting apparatus according to claim 1, wherein the link information detection unit further detects and stores link destination web information from a proxy server accessed by the page browsing unit. Collection device. In the internet information collection device according to claim 1,
The page analysis unit extracts an input sentence from a range defined by a form sentence in a tag sentence that constructs the web page,
The Internet information collection device, wherein the event generation unit sequentially generates all events defined for the input sentence, and generates link information based on valid events therein. On the computer,
A page browsing step to acquire web pages on the Internet and expand the screen,
Analyzing the web page expanded in the page browsing step, a page analysis step for extracting an event operation tag sentence that dynamically generates link information according to an event generated by a user operation;
An event generation step for generating the event with respect to the event operation tag sentence extracted in the page analysis step;
A link information detection step of detecting and storing link destination web information from page transitions by link information generated in the event generated by the event generation step;
Internet information collection program characterized by causing A page browsing step to acquire web pages on the Internet and expand the screen,
Analyzing the web page expanded in the page browsing step, a page analysis step for extracting an event operation tag sentence that dynamically generates link information according to an event generated by a user operation;
An event generation step for generating the event with respect to the event operation tag sentence extracted in the page analysis step;
A link information detection step of detecting and storing link destination web information from page transitions by link information generated in the event generated by the event generation step;
A method for collecting Internet information, comprising:

Images