JP5026068B2 - Information collection system - Google Patents

Description

  The present invention relates to a technique for comprehensively collecting and storing WEB pages via a network.

In an effort to comprehensively collect and store information (hereinafter referred to as WEB pages) published on the Internet and leave it for future generations (WEB archive), it is important to collect WEB pages without omission.
Conventionally, WEB pages are collected by a crawler, which is a robot that collects WEB pages by following links. However, the crawler cannot reach the web page displayed by the operation of the program or script triggered by a mouse event such as a mouse cursor movement or a mouse click on the web page, and cannot be collected. In addition, it is not efficient to collect countless WEB pages manually, and there is a high possibility of leakage.

Therefore, after collecting the WEB page by crawler or manually, the collected WEB page is displayed on the browsing software (WEB browser), the mouse cursor and mouse click are automatically controlled, and the WEB page is comprehensively clicked. When page transition is performed, an attempt has been made to improve completeness by recollecting the WEB page after the transition (for example, Patent Document 1).
JP 2005-149136 A

  However, in the above-described patent document, a base point is set on the WEB page, and mouse click control is performed by moving the mouse cursor up and down and left and right at regular intervals from this base point. For this reason, the mouse cursor movement and mouse click control are also performed on page portions that are not related to script operations and links, so it cannot be said that an efficient collection method can be performed.

In addition, the collection method in the above-mentioned patent document targets only WEB pages using HTML (HyperText Markup Language), and does not consider application documents such as document files and image files published on the Internet. It was. In other words, simply press the print button displayed on the application or the button to edit the document, etc. just by displaying the acquired WEB page on the browsing software, moving the mouse cursor exhaustively and controlling the mouse click Resulting in. As a result, there is a problem that collection may be interrupted by starting up the print screen or starting unnecessary video playback, and the contents of the document may change, making it impossible to collect reliably. is there.
The present invention has been made in view of such circumstances, and does not move the mouse cursor over the WEB page exhaustively, but accurately detects only points that generate events related to WEB page collection, The purpose is to collect WEB pages efficiently.

The invention according to claim 1 is an information collection system for receiving and storing network information provided via a network, and a collection network for storing collection network information location information, which is location information of network information Information location storage means, network information reception means for sending network information request signal and receiving network information with collected network information location position information as destination, network information displayed on screen by reading network information Event configuration that detects the position data indicating the coordinate position on the screen of the component having the feature that may generate the event by analyzing the image of the image, extracting the feature of the pixel included in the image and the feature of the neighboring region an element detecting means, based on the position data of the components, position data Move the mouse cursor to the coordinates position indicated by activates an event with corresponding components by entering the same signal as subjected to mouse clicks at coordinates, the new network information is another network information as a result of events An information collection system comprising: new location information extraction means for extracting location information of the network information when an information request is made to the network.

  The invention according to claim 2 is the information collection system according to claim 1, wherein the location information extracted by the new location information extraction means is additionally stored in the collection network information location storage means An information collection system further comprising a network information location adding means.

  The invention according to claim 3 is the information collection system according to claim 1, further comprising a read network information location storage unit that stores location information of the network information received by the network information reception unit. When the information receiving means requests network information based on the collected network information location information, if the same information as the collected network information location information is present in the already-read network information location storage means, an information request is made. An information collecting system further characterized by not performing.

The invention according to claim 4 is an information collection method for receiving and storing network information provided via a network, and a collection network for storing collection network information location information which is location information of network information An information location storing step, a network information receiving step for transmitting a network information request signal using the collected network information location information as a transmission destination, and receiving the network information, and a network displayed on the screen after reading the network information An event that detects the position data indicating the coordinate position on the screen of the component that has the characteristic that the event may occur by analyzing the image of the information, extracting the feature of the pixel included in the image and the feature of the neighboring area a component detection step, based on the position data of the components Move the mouse cursor to the coordinate position indicated by the position data, to activate the event marked with corresponding components by entering the same signal as subjected to mouse clicks at coordinates, is another network information as a result of the new event An information collection method comprising: a new location information extraction step for extracting location information of network information when an information request for the network information is made.

The invention according to claim 5 is an information collection program for receiving and storing network information provided via a network, and a collection network for storing collection network information location information that is location information of network information An information location storing step, a network information receiving step for transmitting a network information request signal using the collected network information location location information as a transmission destination and receiving the network information, and a network information displayed on the screen by reading the network information Event configuration that detects the position data indicating the coordinate position on the screen of the component having the feature that may generate the event by analyzing the image of the image, extracting the feature of the pixel included in the image and the feature of the neighboring region and element detection step, based on the position data of the components Dzu Te, move the mouse cursor to the coordinate position indicated by the position data, to activate the event marked with corresponding components by entering the same signal as subjected to mouse clicks at coordinates, a different network information as a result of events An information collection program comprising: a new location information extraction step for extracting location information of network information when an information request is made for certain new network information.

  As described above, according to the present invention, the WEB page is analyzed, and only the points that may cause the event related to the collection of the WEB page are detected and operated. Can be collected.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing the configuration of the system according to the present embodiment.
The information collection system according to the present invention includes an information transmission device 10, a control unit 20, and a database unit 30.

  The information transmitting apparatus 10 is connected to the control unit 20 via a network and has a WEB page 11. The information transmitting apparatus 10 transmits the WEB page 11 in response to the WEB page request from the control unit 20. The WEB page 11 is data in HTML format, XHTML (eXtensible HyperText Markup Language) format, XML (eXtensible Markup Language) format, and may be image, text, and other application data. Further, only one information transmitting apparatus 10 and one WEB page 11 are shown in FIG. 1, but a plurality of information transmitting apparatuses 10 and one WEB page 11 may be provided. Assuming that

The control unit 20 includes a page collection control unit 21, a WEB browser 22, a page analysis unit 23, and a mouse cursor control unit 24.
The page collection control unit 21 controls operation cooperation of each functional unit in order to perform page collection processing.
The WEB browser 22 is application software that performs Internet communication based on the IP (Internet Protocol) protocol, renders a WEB page so that the user can view it, and displays it on the screen. In the present embodiment, the WEB browser is used. However, if the application has each function performed by the WEB browser as described above, a separate data collection program may be used.

The page analysis unit 23 analyzes the data structure of the acquired WEB page, and detects position data (hereinafter referred to as element position data) of the event occurrence location. Detailed operation will be described later.
The mouse cursor control unit 24 performs mouse cursor control such as operating the mouse cursor in accordance with the element position data detected by the page analysis unit 23.

The database unit 30 is an information storage unit and includes a collection URL table 31, a read URL table 32, and a component position table 33.
The collection URL table 31 stores URL data that is a target of WEB page collection. A data example of the collection URL table 31 is shown in FIG.
The already-read URL table 32 stores URL data for which WEB page collection has already been performed. An example of data in the read URL table 32 is shown in FIG.

  The component position table 33 stores element position data of components detected by analyzing the WEB page data corresponding to the URL stored in the collection URL table 31. An example of data in the component position table 33 is shown in FIG. The component position table 33 has component number and element position data. For example, the component number is data that uniquely identifies the component, and the element position data is the position where the upper left end on the screen is (0, 0), the X pixel moved right, and the Y pixel moved downward (X, Y) is a coordinate value. For example, (10, 48) indicates a point moved 10 pixels right and 48 pixels down from the upper left corner of the screen.

Next, the WEB page collection operation will be described with reference to the flowcharts of FIGS. 2, 3, 4, and 5.
The user stores one or more pieces of URL data in the collection URL table 31 in advance.
The page collection control unit 21 refers to the collection URL table 31 and determines whether URL data exists in the collection URL table 31 (step S100). The page collection control unit 21 executes the processing up to step S400 until it is determined that no data exists in the collection URL table 31.

  If URL data exists in the collection URL table 31, the page collection control unit 21 performs a WEB page acquisition process (step S200). First, the page collection control unit 21 acquires one URL data stored in the collection URL table 31 (step S201). At this time, the page collection control unit 21 detects whether or not the same data as the acquired URL data exists in the already-read URL table 32, and if there is, it is collected data. Without performing step S100.

  Then, the page collection control unit 21 transmits URL data to the WEB browser 22. The WEB browser 22 receives the URL data and transmits a WEB page request to the corresponding URL. The information transmitting apparatus 10 receives the WEB page request and transmits the WEB page 11 to the WEB browser 22. The WEB browser 22 receives the WEB page 11 (step S202). The page collection control unit 21 stores the received WEB page 11 as a collection page. In addition, the URL obtained in the above is deleted from the collection URL table 31 and stored in the read URL table 32.

  Next, the page analysis unit 23 performs an analysis process on the received WEB page data (step S300). For example, the page analysis unit 23 first reads the extension of the received WEB page data file and determines the file format of the WEB page data. Then, the page analysis unit 23 parses the WEB page data into a DOM (Document Object Model) when the WEB page data to be analyzed is in one of the HTML format, the XHTML format, and the XML format (step S301).

  DOM is an interface function for making data described in a tag structure hierarchically structured so that it can be controlled from a program or script. Parsing to DOM means that data in HTML format, XHTML format, or XML format is hierarchically structured, and a data access structure (DOM object) for handling as a tree structure is generated from a program. For example, when the HTML data is parsed into DOM, the HTML data can be handled as tree-structured data having HTML components (tags) as nodes. Then, the page analysis unit 23 performs rendering to calculate a position for displaying the components of the DOM object, and displays the WEB page data on the WEB browser 22 (step S302).

  Next, the page analysis unit 23 detects a constituent element that may generate an event from the DOM object based on a rule set in advance by the user (step S303), and configures the position where the element is displayed. The element position table 33 is stored (step S304). For example, the page analysis unit 23 refers to the DOM object, detects an <A> tag, a button tag, an image tag, and the like, and stores the position where the element is displayed in the component position table. That is, the page analysis unit 23 refers to the DOM object, detects the node that is the <A> tag, and the page analysis unit 23 uses the position where the component calculated in step S302 is displayed as the element position data. It is stored in the component position table 33. The page analysis unit 23 similarly detects element position data of a component that generates an event even when the WEB page is application data such as a document file or an image file.

  Here, even when the WEB page is application data such as a document file or an image file, the page analysis unit 23 similarly detects a component that may generate an event, and displays its display position in the component position table. Remember me. For example, in the case of a PDF (Portable Document Format) file, an Anchor object included in a Document object is detected as a component that may generate an event. In the case of a WORD file, ActiveXObject. The Hyperlinks object included in the Documents is detected as a component that may generate an event. In the case of a Flash file, a Buttons object or the like is detected as a component that may generate an event. The position information of each detected element is detected and stored in the component position table.

  When detecting a component that may cause an event, the page analysis unit 23 may detect the element position data by performing image analysis, for example. In this case, after rendering the WEB page data, the page analysis unit 23 performs image analysis of the entire page, and extracts the features of each pixel and the features of the neighboring region to extract components that may generate an event. To detect. In other words, areas that are continuously filled only within a certain range on the screen, areas where text is displayed, areas with different colors, underlined areas, etc., can be events. Extract.

Next, the mouse cursor control unit 24 performs URL extraction processing by mouse control (step S400). First, the mouse cursor control unit 24 refers to the component position table 33 and determines whether or not element position data exists in the component position table 33 (step S401). The mouse cursor control unit 24 continues the processing up to S404 until it determines that the element position data does not exist in the component position table 33.
If element position data exists in the component position table 33, the mouse cursor controller 24 reads one piece of element position data from the component position table 33 (step S402).
The mouse cursor control unit 24 moves the mouse cursor to the coordinate position indicated by the read element position data, and inputs the same signal as when the mouse click is performed at the position (step S403).

  When the mouse click signal is detected, the WEB browser 22 performs an operation process when the mouse is clicked at the position. When an event is generated by a mouse click at the position and a screen transition is performed, the WEB browser 22 generates WEB page request data. When the WEB browser 22 generates the WEB page request data, the page collection control unit 21 acquires the WEB page request data, acquires the request destination URL data from the WEB page request data, and stores the request destination URL data in the collection URL table 31. Store (step S404). When the page collection control unit 21 acquires the request destination URL data, the WEB browser 22 cancels the WEB page request. In addition, the mouse cursor control unit 24 deletes the read element position data from the component position table 33.

According to this configuration, as shown in FIG. 9, when there are a plurality of components on the screen, it is possible to avoid the text portion and move the mouse cursor only to a portion where an event may occur. .
Further, a configuration for generating an event is provided by storing a feature of a component that is determined to generate an event when analyzing a WEB page, and storing whether or not the event has actually occurred when the mouse is clicked. A learning function for analyzing element characteristic trends may be provided.

  According to the present invention, it is possible to collect a WEB page with a minimum mouse cursor control without comprehensively moving the mouse cursor and controlling a mouse button in collecting the WEB page. That is, the time required for collecting WEB pages can be greatly reduced.

For example, if the WEB page to be collected is an HTML page of 800 pixels in width and 600 pixels in length, in the conventional method of comprehensively moving on the page, if the mouse is moved 10 pixels vertically and horizontally, 4800 times Need to move. On the other hand, according to the method of detecting a portion requiring mouse click in advance according to the present invention, since the average number of links per page is said to be around 10, it can be moved about 10 times on average. Therefore, the control time of the mouse cursor can be shortened to about 1/480.
In the present invention, collection by mouse cursor control can also be performed on application documents other than the HTML format.

  Note that the WEB page collection is performed by recording a program for realizing the function shown in FIG. 1 in the present invention on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. May be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows one Embodiment of this invention. It is a flowchart which shows one Embodiment of this invention. It is a flowchart which shows the WEB page acquisition process by this invention. It is a flowchart which shows the WEB page analysis process by this invention. It is a flowchart which shows URL extraction processing by this invention. It is a figure which shows the data structure of the component position table by this invention. It is a figure which shows the data structure of the collection URL table by this invention. It is a figure which shows the data structure of the read URL table by this invention. It is a figure which shows the mouse | mouth control screen by this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Information transmitter 11 Web page 20 Control part 21 Page collection control part (New location information extraction means, Collection network information location addition means)
22 WEB browser (network information receiving means)
23 Page analysis part (event component detection means)
24 mouse cursor control unit 30 database unit 31 collection URL table (collection network information location storage means)
32 Read URL table (read network information location storage means)
33 Component position table

Claims (5)

An information collection system for receiving and storing network information provided via a network,
Collection network information location storage means for storing collection network information location information that is location information of network information;
Network information receiving means for transmitting a request signal of network information using the collected network information location information as a transmission destination and receiving the network information;
A configuration having a feature in which an event may occur by reading the network information, analyzing the network information displayed on the screen, extracting a feature of a pixel and a feature of a neighboring region included in the image Event component detection means for detecting position data indicating the coordinate position of the element on the screen;
Based on the position data of the component , the mouse cursor is moved to the coordinate position indicated by the position data, and an event corresponding to the component is activated by inputting the same signal as when the mouse click is performed at the coordinate position And a new location information extracting means for extracting the location information of the network information when an information request is made to the new network information that is another network information as a result of the event. Collection system. The information collection system according to claim 1,
An information collection system further comprising collection network information location addition means for adding the location information extracted by the new location information extraction means to the collection network information location storage means for storage. The information collection system according to claim 1,
Read network information location storage means for storing location information of network information received by the network information receiving means,
When the network information receiving means requests the network information based on the collected network information location information, the same information as the collected network information location information is present in the read network information location storage means. In some cases, the information collection system is further characterized by not requesting information. An information collection method for receiving and storing network information provided via a network,
A collection network information location location storage step for storing collection network information location location information which is location location information of the network information;
A network information receiving step of transmitting a network information request signal using the collected network information location information as a transmission destination and receiving the network information;
A configuration having a feature in which an event may occur by reading the network information, analyzing the network information displayed on the screen, extracting a feature of a pixel and a feature of a neighboring region included in the image An event component detection step for detecting position data indicating the coordinate position of the element on the screen;
Based on the position data of the component , the mouse cursor is moved to the coordinate position indicated by the position data, and an event corresponding to the component is activated by inputting the same signal as when the mouse click is performed at the coordinate position And a new location information extraction step for extracting the location information of the network information when an information request is made to the new network information that is another network information as a result of the event. Collection method. An information collection program for receiving and storing network information provided via a network,
A collection network information location location storage step for storing collection network information location location information which is location location information of the network information;
A network information receiving step of transmitting a network information request signal using the collected network information location information as a transmission destination and receiving the network information;
A configuration having a feature in which an event may occur by reading the network information, analyzing the network information displayed on the screen, extracting a feature of a pixel and a feature of a neighboring region included in the image An event component detection step for detecting position data indicating the coordinate position of the element on the screen;
Based on the position data of the component , the mouse cursor is moved to the coordinate position indicated by the position data, and an event corresponding to the component is activated by inputting the same signal as when the mouse click is performed at the coordinate position And a new location information extraction step for extracting the location information of the network information when an information request is made to the new network information that is another network information as a result of the event. Collection program.

Images