JP4496929B2 - Parallel playback apparatus and program for multiple web browsing operations, and computer-readable recording medium - Google Patents

Description

The present invention, software for efficient browsing work on Web, relates to a parallel playback apparatus and a program and a computer-readable recording medium of plural Web browsing operation about the browser, in particular, a plurality of operation target (Web page) on parallel playback apparatus and a program and a computer-readable recording medium of plural Web browsing operation is performed to the efficiency of browsing operations to repeat multiple operations against.

  Browsing work on the Web, in particular, searching for store information, travel information, auction information, etc. on a certain portal site, and the same for Web pages indicated by each search result starting from the page listing the search results It often happens that the above operation is repeated. At this time, the user generally follows various links displayed in each Web page and acquires various information.

  For example, if you search for a restaurant near Yokohama on a certain search site and want to obtain a map and coupon for each store, access the introduction page of each store from the list of search results, and then first access the page displaying the map image. . Then, return to the introduction page, and then access the page displaying the coupon.

  Here, when there are a plurality of browsing objects (introduction pages of each store) and it is desired to acquire similar information (map and coupon) from each of them, the same operation is sequentially repeated for each browsing object.

  Even today, macro tools exist as software for recording and playing back user operations. Many existing macro tools record and play back operations on the user's desktop (for example, Patent Document 1). reference).

Further, there is a meta search system as a system that realizes batch execution of search operations using the same keyword for a plurality of search engines. At this time, in the meta search system, the search keyword is processed on the system side in a format determined by each search engine, and the search operation to each search engine is executed in parallel. (For example, refer to Patent Document 2).
JP-A-7-306769 Japanese Patent Laid-Open No. 11-3203

  However, in the above macro tool, when there are a plurality of operation contents to be executed or a plurality of operation objects, it is necessary to appropriately call a macro corresponding to each operation contents and execute them individually for each operation object.

  In other words, the conventional macro tool cannot execute different operations on a certain operation target in parallel or perform similar operations on a plurality of operation targets at once.

  Further, the meta search system is a system specialized for search operations, and a plurality of Web page browsing operations as described above cannot be executed in parallel. Also, in many existing meta search systems, search engines that can be used are determined in advance, and it is not possible for the user to set the search engine arbitrarily by a simple method, for example, by recording a search operation by the user. Can not.

The present invention has been made in view of the above points, and records a web browsing operation on a window (web page) of a user and converts it into a macro, and also creates different macros for a plurality of windows displayed on the desktop. Can be set. Then, starting from the macro playback operation from the user, a message describing the operation content is exchanged between the windows, and each window processes the message based on the macro set in itself, thereby enabling a plurality of Web and to provide a parallel playback apparatus and a program and a computer-readable recording medium of plural Web browsing operation for realizing the parallel reproduction of the browsing operation.

  FIG. 1 is an explanatory diagram of the principle of the present invention.

The present invention relates to a parallel playback method of a plurality of Web browsing operations for realizing recording / playback of a plurality of Web browsing operations.
In the Web browsing operation recording means, event information generated in response to the Web browsing operation in the user input device is received (step 1), and the contents of the operation performed by the user and the information of the execution destination object are received from the input device. Acquiring and storing a series of web browsing operation information as macros in the macro storage means (step 2) a web browsing operation recording step;
In the Web browsing operation reproduction means, the window on the user's display device and the Web browsing operation information set for each window are written in the window management table (step 3), and the information recorded in the window management table is recorded. In accordance with the web browsing operation of the user and the content of the web page being displayed on the display device, the macro stored in the macro storage means is executed (step 4) a web browsing operation reproduction step,
In the web browsing operation playback step,
A message receiving step for receiving a message created for each window for reproducing each Web browsing operation and transmitted from the Web browsing operation storage means;
Refers to the window management table, reads a macro that is set in each window from the macro storage means, read macro operations within collates the operation content information described in the message received and contents shown, the Web browsing operation A macro matching step for determining whether or not to play, analyzing the web page displayed in the window, and searching for an object that matches the execution target of the read macro operation ;
Based on the operation content information described in the received message when a macro operation that matches the operation content information described in the message and an object that matches the execution target of the macro operation are derived in the macro collation step. A macro execution step for executing the operation content indicated by the macro on the matching object derived in the macro collation step.
In the macro matching step,
When the web page displayed in each window is analyzed and the macro storage means is searched for an object that matches the execution target of each macro operation, the web page is analyzed and the object described in the operation content indicated by the macro is analyzed . A step of acquiring a list of objects in which a DOM (Document Object Model) representing an object and a tag type used in HTML match,
The similarity between the object described in the operation content indicated by the macro and each object in the object list is calculated based on the similarity of the display position of the object, the similarity on the DOM structure, and the similarity of the link destination URL. Steps,
Based on the calculated similarity, the step of making the object having the maximum similarity the matching object is performed.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 1) is a parallel playback device for a plurality of Web browsing operations for realizing recording and playback of Web browsing operations,
Web browsing operation recording means 11 for receiving event information generated by the user's Web browsing operation from the user's input device, and acquiring information on the operation performed by the user and information on the execution destination object;
A macro storage unit 14 for storing a series of Web browsing operations acquired by the Web browsing operation recording unit 11 as a macro;
Web browsing operation reproduction means 12 for executing a macro stored in the macro storage means 14 in accordance with the user's browsing operation and the contents of the displayed Web page;
Display control means 13 for managing windows on the display device 30 used during recording and reproduction of Web browsing operations;
A window management table 15 provided on a window on the display device 30 and a storage unit configured to manage a macro set in each window;
Have
Web browsing operation playback means 12
Message receiving means for receiving messages transmitted from the web browsing operation recording means 11 and created for each window for reproducing each web browsing operation;
Window refers to the management table 15, it reads the macros that are set in each window from the macro storage unit 14, collates the operation Description and received operation content information described in the message indicated by the read macro, Web browsing Macro collation means for determining whether or not to reproduce the operation, analyzing the web page displayed in the window, and searching for an object that matches the execution target of the read macro operation ;
Based on the operation content information described in the received message when a macro operation that matches the operation content information described in the message and an object that matches the execution target of the macro operation are derived by the macro matching means. Macro execution means for executing the operation content indicated by the macro for the matching object derived by the macro matching means,
Macro matching means
When the Web page displayed in each window is analyzed and an object suitable for the execution target of each macro operation is searched from the macro storage unit 14, the Web page is analyzed and described with the operation content indicated by the macro. A means for obtaining a list of objects having the same DOM (Document Object Model) representing the object and the types of tags used in HTML ;
The similarity between the object described in the operation content indicated by the macro and each object in the object list is calculated based on the similarity of the display position of the object, the similarity on the DOM structure, and the similarity of the link destination URL. Means,
And a means for adapting an object having the highest similarity among the calculated objects ,
When realizing parallel playback of the same Web browsing operation for multiple objects,
A selection object management list provided on storage means for storing information on a plurality of objects selected by the user;
The display control device
In response to the number of objects selected by the user, a new window for reproducing the web browsing operation is displayed, and the web browsing operation for the window already displayed on the display device is displayed. As a web browsing operation, the web browsing operation reproduction means also operates on a new window.
When applying web browsing operations,
Based on the received event information, describes the operation performed by the user, the information of the object operated by the user, and the information of the Web page displayed in the window operated by the user Event analysis means for creating a message, and creating a message in which the object information of the message is changed to the information of each object in the selected object management list;
With reference to the window management table, there is message transmission means for transmitting each message created by the event analysis means to the Web browsing operation reproduction means corresponding to each window on the display device.

The present invention (Claim 2 ) is a parallel reproduction program for a plurality of Web browsing operations for causing a computer to function as each means constituting the parallel reproduction apparatus for a plurality of Web browsing operations according to Claim 1 .

The present invention (Claim 3 ) is a computer-readable recording medium in which the parallel reproduction program for a plurality of Web browsing operations according to Claim 2 is stored.

  As described above, according to the present invention, as a browsing operation on the Web, a plurality of Web browsing operations are repeatedly executed for a plurality of Web pages, particularly a plurality of Web pages having similar layout and display contents. In such a case, it becomes possible to collectively execute a plurality of Web browsing operations on a certain Web page.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, definitions of terms used in the embodiment of the present invention are shown below.

  Macro: A function that describes and automates a specific operation procedure as a program, or a program thereof.

  Macro window: A window in which recording / reproduction of a web browsing operation is executed. One of the windows that a general Web browser opens.

  Macro controller: A window or operation screen used to instruct the system to record / replay a web browsing operation.

  • Dialog: A window or operation screen used for saving / reading files / displaying error messages.

  Object: Each element constituting a Web page such as an anchor (link), image, table, text document, etc. in the Web page.

  -Tag type-Object type: Tag types used in HTML (Hyper Text Markup Language) such as <A>, <LINK>, <TABLE>, <BODY>.

  -DOM (Document Object Model): A standard interface for handling each element described in HTML or XML. By processing the Web page with a tool called a DOM parser, the Web page is converted into a DOM structure, and as a result, each element (object) of the Web page can be easily handled.

  -Various attributes of objects (innerText, href, alt, src, id, name, title): <A href=”aa, id=”bb”, name=”cc> DD </A>, <IMG src = ” ee ”, alt =” ff ”> etc. Attribute information for each tag. After converting the Web page to the DOM structure, the information of the corresponding attribute can be easily obtained by accessing A. href and the like. The innerText attribute is a character string enclosed by anchor tags <A> </A>. In this case, it is “DD” and can be acquired by accessing A.innerText.

  ・ Comment: It is described in the HTML document in the form of <!-AAA-> and ignored when displayed by the browser, but it is described as an annotation.

  Form: An input field displayed on the Web page. It is described using the <FORM> tag, and there are various types in lists, text boxes, radio buttons, etc.

  Event: A signal generated when a user operation or a display state change occurs, and information on the phenomenon that has occurred.

  -Event type: Operation performed by the user. Here, click, double click, memu selection, etc.

  Event execution destination object: means an object operated by the user, for example, clicked anchor object, clicked image object, etc.

・ Portal site: Various information such as “goo” and “Yahoo”. A site that aggregates links to various information.

First, the configuration of the multiple Web browsing operation recording / reproducing apparatus will be described.

  FIG. 3 is a diagram showing a configuration of the Web browsing operation recording / reproducing apparatus according to the embodiment of the present invention.

  An input device 20 and a display device 30 are connected to the Web browsing operation recording / reproducing apparatus 10 shown in the figure, and are connected to a Web server 50 via a network 40.

  The web browsing operation recording / reproducing apparatus 10 includes a web browsing operation recording unit 11, a web browsing operation reproducing unit 12, a display control unit 13, a macro storage unit 14, and a window management table 15.

  The web browsing operation recording unit 11 includes an operation receiving unit 111, an event analysis unit 112, a macro generation unit 113, a browsing operation unit 114, and a message transmission unit 115.

  The web browsing operation reproduction unit 12 includes a message reception unit 121, a macro collation unit 122, and a macro execution unit 123.

  The display control unit 13 is connected to a user display device 30 such as a display, and displays a browsing window 301 and a plurality of macro windows 302.

  Here, the Web browsing operation reproduction unit 12 exists for each macro window 302 on the display device 30.

  In addition, a user input device 20 such as a keyboard or a mouse is connected to the operation reception unit 111, and the browsing operation unit 114 and the macro execution unit 123 can communicate with the Web server 50 via the network 40.

  In the present invention, the macro storage unit 14 is provided in a storage unit such as a memory or a hard disk in the Web browsing operation recording / reproducing apparatus 10.

  Next, processing in the Web browsing operation recording / reproducing apparatus 10 will be described.

  FIG. 4 is a diagram showing a presentation screen in one embodiment of the present invention.

  In the figure, the macro controller 310 includes a cursor 201, a browsing window 301, a macro window A 302, a macro window B 303, a record button 311, a playback button 312, and a file button 313.

  Note that the browsing window 301, the macro window A302, and the macro window B303 are Web browsers.

  Hereinafter, the operation of the web browsing operation recording / reproducing apparatus 10 is as follows: (1) recording of browsing operations, (2) adding / deleting macro windows, (3) setting of macros in macro windows, (4) playback of web browsing operations. , (5) Search for a conforming object, and (4 ′) Web browsing operation reproduction (continued) will be described.

(1) Web browsing operation recording:
FIG. 5 is a diagram showing the display of the browsing window in the embodiment of the present invention, and FIG. 6 is a flowchart of the recording process of the web browsing operation in the embodiment of the present invention.

  Hereinafter, with reference to FIGS. 3, 4, and 5, processing in the recording of the web browsing operation will be described using the flowchart of FIG. 6.

  First, an arbitrary Web page that is the starting point of the browsing operation is displayed on the browsing window 301 for the user. Thereafter, as shown in FIG. 5A, the recording of the browsing operation is started by pressing the recording button 311 of the macro controller 310 (step 401).

  Here, only the macro window A302 is displayed as the macro window, and the macro window B303 is newly displayed by the process described in (2) Addition / deletion of macro window described later.

  At this time, the operation reception unit 111 receives the recording start of the browsing operation, and displays the web page being displayed in the browsing window 301 in the macro window 302 as shown in FIG. 5B (step 402).

  Further, the macro generation flag is turned ON and the operation counter is initialized to 0 (step 403). Although not shown in FIG. 3, the macro generation flag and the operation counter are managed by a storage unit such as a memory in the Web browsing operation recording / reproducing apparatus 10.

  Then, the mode shifts to a mode for monitoring the user's browsing operation (step 404).

  After that, when event information generated on the macro window A in response to the user's browsing operation is received by the operation receiving unit 111 (step 405, Yes), the event analysis unit 112, based on the received event information, Information on the operation (event type) performed by the user and the event execution destination object is acquired and output as a macro by the macro generation unit 113 (step 406).

  Here, the macro is managed by the macro storage unit 14, and one operation is represented by one line as shown in FIG. The macro No. The value of the operation counter is entered in the part, and each browsing operation is managed in time series.

  Here, browsing operations performed by the user include a click operation (OnClick), a double click operation (OnDbClick) on a web page in the macro window, and a selection of a form (list, text box, etc.) in the web page (OnSelect, OnChange), input (OnKeyUP), send (OnSubmit) operations, copy (onEvent (WM_COPY)) / paste (, onEvent (WM_PASTE)) operations on the macro window, keyboard operations (onEvent (WM_KEYUP)), browser There are menu and right-click menu selection operations (onEvent (WM_MENUSELECT)).

  The operation receiving unit 111 receives an event function described in parentheses in accordance with the browsing operation by the user.

  For example, when the user clicks an anchor in a Web page displayed in the macro window A (see FIG. 5C), the operation receiving unit 111 sets OnClick (document.a [10]) as an event function. As received.

  The event analysis unit 112 acquires the event type “Click” of the operation performed by the user from the function name of the received event function, and acquires the event execution destination object “A [10]” from the argument of the function. .

  Here, the event execution destination object is represented by DOM (Document Object Model), and by accessing various attributes of the DOM, various information of the event execution destination object is acquired and output as a macro.

  For example, the text information of the anchor object “A” is acquired by accessing the innerText, id, name, and title attributes, the position information is the x and y attributes, and the link destination URL information is accessed by accessing the href attribute (No. 0 in FIG. 7). Line reference).

  In addition, for example, when the user clicks on an image in the Web page (see FIG. 5D), the operation reception unit 111 receives OnClick (document.img [3]) as an event function.

  The event analysis unit 112 acquires the event type “Click” of the operation performed by the user from the function name of the received event function, and acquires the event execution destination object “IMG [3]” from the argument of the function. .

  At this time, the text information of the image object “IMG” is acquired by accessing the alt, id, and name attributes, and the position information is accessed by accessing the x and y attributes.

  The link destination URL information accesses the parent object “IMG [3] .parentNode” of the image object, and when the parent object is the anchor object “A” or the link object “LINK”, the link destination of the parent object URL information (href attribute) is acquired.

  In the case of the image object “IMG”, the URL information of the image file is acquired by accessing the src attribute and described in the macro (see the No. 1 line in FIG. 7).

  Furthermore, as the event execution destination object information, the parent object is searched from the acquired event execution destination object to the hierarchy to be set separately, and when the comment “<!-->” Is found, the comment information is macro (See FIG. 7).

  As described above, the processing at the time of macro generation and the contents described in the macro differ depending on the received event function, that is, the browsing operation of the user, and the contents described in the macro are also bound to the items shown in FIG. It is possible to make various changes without adding new items or deleting existing items.

  For example, it is conceivable to add information about an object displayed around the event execution destination object.

  Next, the browsing operation unit 114 executes the browsing operation of the user received by the operation receiving unit 111 (step 407).

  Thereafter, the operation receiving unit 111 receives an update / load event (onLoad ()) of the Web page in the macro window A302 (step 408, Yes), increments the operation counter by 1 (step 409), and the macro generation flag is set. If it is ON (step 410, Yes), the process returns to step 404 to continue monitoring the user's browsing operation.

  On the other hand, when the macro generation flag is OFF (step 410, No), the recording of the browsing operation is ended.

  As shown in FIG. 5E, the macro generation flag is turned OFF when the user presses the recording button 311 of the macro controller 310 again.

  Here, the user can save the generated macro as a file using a dialog displayed by pressing the file button 313 of the macro controller 310.

(2) Add / delete macro window:
Next, a processing procedure for adding / deleting a macro window will be described.

  First, when a macro window addition operation occurs, for example, when “add macro window” is selected from a menu opened by right-clicking, the display control unit 13 newly displays a macro window on the display device 30. A new item is added to the window management table 15. In addition, a web browsing operation playback unit 12 corresponding to a new macro window is newly generated in the web browsing operation recording / playback apparatus 10.

  Here, as shown in FIG. 8, in the window name column of the window management table 15, information for specifying the browsing window, each macro window, and the Web browsing operation reproducing unit 12 corresponding to each macro window, for example, The window name, window handle, and the like are described. In the macro column, the macro information set in each macro window, for example, the file name is described, and sorted in the order of adding the windows.

  On the other hand, when a macro window deletion operation occurs, for example, when an arbitrary macro window is selected and “Delete Macro Window” is selected from the menu opened by right-clicking, the display control unit 13 displays the display device 30. The macro window selected from above is deleted, and the corresponding item is deleted from the window management table 15 using the window name of the selected macro window as a key. Further, the Web browsing operation playback unit 12 corresponding to the selected macro window is deleted from the Web browsing operation recording / playback device 10.

(3) Macro settings for the macro window:
Next, a processing procedure in macro setting for the macro window will be described.

  First, an arbitrary macro window on the display device is selected by the user, for example, a click operation is executed in the macro window. When the file button 313 of the macro controller 310 shown in FIG. 4 is pressed, an arbitrary (macro) file is called using the displayed dialog.

  At this time, as shown in FIG. 8, the display control unit 13 refers to the window management table 15 and enters the file name of the called (macro) file in the macro field of the selected macro window.

  Here, in addition to manually setting the macro in the macro window according to the above procedure, at the end of the recording of the above-described (1) Web browsing operation, the macro of the recorded Web browsing operation is automatically displayed in the macro window. It is also possible to set it to. At this time, the recorded macro of the web browsing operation is automatically saved in a temporary file.

(4) Web browsing operation playback:
FIG. 9 is a diagram showing the display of the browsing window at the time of reproduction of the web browsing operation according to the embodiment of the present invention, and FIG. 10 is a flowchart of the reproduction processing of the web browsing operation according to the embodiment of the present invention. is there.

  First, the processing procedure in the Web browsing operation recording unit 11 in the reproduction of the Web browsing operation will be described based on the flowchart in FIG. 10 with reference to FIGS.

  Here, in addition to the macro window A302, a macro window B303 is newly displayed on the display device 30 of the user in accordance with the procedure shown in the above (2), and in the above (3). In accordance with the procedure shown, a macro that acquires a map image (see Find_map.mac in FIG. 8) in the macro window A302 and a macro that acquires a coupon (see Find_coupon.mac in FIG. 8) are set in the macro window B303. And

  Each macro is created in advance by recording a user's Web browsing operation according to the procedure shown in (1) above.

  First, as shown in FIG. 9A, the user presses the playback button 312 of the macro controller 310 to start playback of the web browsing operation (step 801).

  At this time, the operation receiving unit 111 receives a reproduction start request for the Web browsing operation and turns on the macro execution flag (step 802). Although not shown in FIG. 3, the macro execution flag is managed by a storage unit such as a memory in the Web browsing operation recording / reproducing apparatus 10.

  Then, the mode shifts to a mode for monitoring a user's browsing operation on the browsing window 301 (step 803).

  After that, when event information generated on the viewing window is received by the operation receiving unit 111 in accordance with the user's browsing operation (Yes in step 804), the event analysis unit 112 uses the received event information based on the received event information. Information on the operation (event type) performed by the user and the event execution destination object is acquired (step 805). The process executed in step 805 is the same as the process executed in step 406 in the above-described (1) Web browsing operation recording process.

  For example, when the user clicks on an anchor in a Web page displayed in the browsing window (an anchor different from the anchor clicked when recording the Web browsing operation) (see FIG. 9B), the operation is received. The unit 111 receives OnClick (document.a [11]) as an event function.

  The event analysis unit 112 acquires the event type “Click” of the operation performed by the user from the function name of the received event function, and acquires the event execution destination object “A [11]” from the argument of the function. .

  Further, the URL of the Web page displayed in the browsing window 301 is acquired by accessing, for example, the location attribute of the browsing window 301 (step 806).

  Then, the event type (Click) of the operation performed by the user, information on the event execution destination object (DOM of A [11], text, display position, link destination URL, URL, comment information) acquired in step 805, In addition, a message describing the URL of the Web page being displayed in the browsing window acquired in step 806 is created (step 807).

  Next, the message transmission unit 115 refers to the window management table 15 and converts the message created in step 807 to all macro windows (here, macro window A and macro window B) on the display device 30 accurately. The message is transmitted to the message receiving unit 121 in the Web browsing operation reproducing unit 12 corresponding to the macro window (step 808).

  Also, the browsing operation unit 114 invalidates the user's browsing operation (received in step 804) performed on the browsing window 301 (step 809).

  Thereafter, when the macro execution flag is ON (step 810, Yes), the operation receiving unit 111 returns to step 803 and continues to monitor the browsing operation of the user on the browsing window 301.

  On the other hand, when the macro execution flag is OFF (step 810, No), the reproduction of the web browsing operation is terminated.

  As shown in FIG. 9E, when the playback button 312 of the macro controller 310 is pressed again, the operation reception unit 111 receives this operation and turns off the macro execution flag.

  Next, the processing procedure in each Web browsing operation playback unit 12 in the playback of the Web browsing operation will be described with reference to FIGS. 3, 7, and 9 and using the flowchart shown in FIG.

  First, each message reception unit 121 receives a message transmitted from the message transmission unit 115 (step 901), and displays a Web page indicating a URL described in the message, that is, a Web page being displayed in the browsing window 301. It is displayed in the macro window (step 902).

  Next, each macro matching unit 122 reads the macro (file) set in each macro window with reference to the window management table 15. Also, an operation counter managed by each Web browsing operation reproduction unit 12 is initialized to 0 (step 903).

  Then, the macro No. read by the macro storage unit 14 is read. The operation indicated by 0 (see the row No. 0 in FIGS. 7 and 12) and the operation described in the message, that is, the content of the operation performed by the user on the browsing window 301 are collated (step 904).

  For example, it is conceivable to collate the event types of both operations, and the DOM information of the event execution destination object, particularly the DOM object type.

  Here, the event type of the operation indicated by No. 0 of each macro is “Click” for both, and the DOM of the event execution destination object is also “A [10]”. On the other hand, as described in step 807 of FIG. 10, the event type of the operation described in the message is “Click”, and the DOM of the event execution destination object is “A [11]”.

  In other words, in this case, the operation indicated by No. 0 of each macro matches the event type (Click) of the operation described in the message and the object type (A object) of the DOM, and the collation result is “Yes” (step) 905, Yes).

  Thereafter, when the collation result is “Yes” (step 905, Yes), each macro execution unit 123 performs the operation described in the message, that is, the operation performed by the user on the browsing window 301. It executes on the macro window (step 906).

  Here, the web page being displayed in the browsing window is displayed in each macro window, and the operation performed by the user on the browsing window is reproduced as it is on each macro window.

  In this case, as shown in FIG. 9C, each macro window displays a Web page (store introduction page) indicated by the anchor (A [11]) clicked by the user on the browsing window 301. The

  Next, the macro collation unit 122 receives an update / read event of a Web page accompanying execution of an operation on each macro window (step 907, Yes), and then increments the operation counter by 1 and reads it in step 903. With reference to the macro, the operation indicated by the No. that matches the operation counter is acquired (step 908).

  Then, an object that matches the event execution destination object indicated by the operation acquired in step 908 is searched for in the web page displayed in each macro window (step 909).

  In the following, the procedure of the above search process will be described using the flowchart shown in FIG. 13 as an example when the operation counter is 1 (the operation indicated by the No. 1 line in FIGS. 7 and 12).

(5) Search for conforming objects:
FIG. 13 is a flowchart of the matching object search process according to the embodiment of the present invention.

  First, each Web page displayed in each macro window is analyzed, and a list of objects (DOM [n], where n is the number of elements) whose object type matches the DOM information indicated by each operation acquired in step 908 is acquired. (Step 1101).

  In this case, a list of image objects (IMG [n], see FIG. 7) is obtained in the macro window A, and a list of anchor objects (A [n], see FIG. 12) is obtained in the macro window B.

  Next, the text information indicated by each operation acquired in step 903 is collated with the text information of each object in the list acquired in step 1101 (step 1102). If there are a plurality of matched objects (step 1103, “multiple”). ), The object that did not match is deleted from the list (step 1104), and if there is no matching object (step 1103, “None”), the list is left as it is and the next processing is performed. In the case of one (step 1103, “one”), the object is set as a matching object, and the process is terminated.

  Note that the text information of each object in the list is acquired by the process (the process of the event analysis unit 112) executed in the above-described (1) Web browsing operation recording. In this case, in the macro window A, the text information ( The image object having “map” as the “alt attribute” and the anchor object having “IMG tag” as the text information and particularly “coupon” as the alt attribute are extracted in the macro window B (FIGS. 7 and 12). reference).

  In addition, the comment information indicated by each operation acquired in step 908 is collated with the comment information of each object in the list extracted in the process of step 1102 (step 1105), and when there are a plurality of matching objects (step 1106, “ Multiple "), the unmatched object is deleted from the list (step 1107), and if the matched object does not exist (step 1106," none "), the list remains as it is, and the process proceeds to the next process. If there is one object (step 1106, “one”), that object is regarded as a matching object, and the process is terminated.

  Note that the comment information of each object in the list is obtained by the above-described process (1) executed by recording the Web browsing operation (searching the parent object from the acquired event execution destination object and describing the comment information in a macro). In this case, the macro window A has an image object having “<-▼ MAP-> <!-▲ MAP->” as comment information, and the macro window B has “<!- Anchor objects having ▼ Coupon-> <!-▲ Coupon-> "are extracted (see FIGS. 7 and 12).

  Next, the similarity between the object indicated by each operation acquired in step 908 and each object remaining in the list is calculated by the following formula (step 1108), and the object having the maximum similarity is determined as the matching object. (Step 1109), the process is terminated.

Ｍ：ステップ９０８で取得した操作が示すオブジェクト；
Ｌ_ｎ：リスト中の各オブジェクト
第１項：表示位置の類似度（Ｍ_ｘ，Ｍ_ｙ，Ｌ_ｎ，ｘ，Ｌ_ｎｙ：各オブジェクトの表示位置のｘ成分、ｙ成分）；
第２項：DOM構造上の類似度（Ｍ_ｎｕｍ，Ｌ_ｎ，ｎｕｍ：各オブジェクトのDOM[n]のｎ部分）；
第３項：リンク先ＵＲＬ情報の類似度（Ｍ_linked、Ｌ_ｎ，linked:各オブジェクトのリンク先のＵＲＬの文字列）；
第４項は：ＵＲＬ情報の類似度（Ｍ_url、Ｌ_ｎ，url：各オブジェクトのＵＲＬの文字列）を表す。

M: an object indicated by the operation acquired in step 908;
L _n : each object in the list 1st term: similarity of display position (M _x , M _y , L _n , x, L _n y: x component, y component of display position of each object);
2nd term: similarity on DOM structure (M _num , L _n , num: n part of DOM [n] of each object);
Item 3: Similarity of _linked URL information (M _linked , L _n , linked: character string of linked URL of each object);
The fourth term represents the similarity of URL information (M _url , L _n , url: character string of URL of each object).

  Note that a, b, c, and d are constants that determine the priority of each similarity and are defined separately.

Also, ft (M, L _n ) is a function for calculating the similarity of URLs. For example, each URL is divided into “/” as a delimiter and the ratio of matching parts is calculated. It is conceivable that only the end part of the URL (the cc part of http: //aa/bb/cc.html) is collated and calculated by 1/0.

(4 ′) Web browsing operation playback (continued):
As a result of the processing in step 909, when an object that matches the event execution destination object indicated by each operation acquired in step 908 is derived (step 910, Yes), the macro execution unit 112 performs the operation on the derived matching object. Then, the operation indicated by each operation event type acquired in step 908 is executed (step 911).

  In this case, in the macro window A, an image object having “map” is derived as text information (alt attribute) in the store introduction page, and a click operation is executed on the image object (No. in FIG. 7). (See event type in row 1). In the macro window B, an anchor object having “IMG tag”, particularly “coupon” as the alt attribute, is derived as text information in the store introduction page, and a click operation is performed on the anchor object. (Refer to the event type in FIG. 10, No. 1 row).

  As a result, as shown in FIG. 9D, the macro window A includes a page displaying a map of the store corresponding to the anchor (A [11]) clicked by the user on the browsing window 301, and the macro window B. The web page displaying the coupon of the store corresponding to the anchor (A [11]) clicked by the user on the browsing window 301 is displayed.

  Thereafter, the macro collation unit 122 refers to the macro read in step 909, and when the next operation exists (step 912, No), returns to step 907, increments the operation counter by 1, and matches the operation counter. The operation indicated by No., that is, the next operation is acquired (step 908).

  Then, using the newly acquired operation, the processes shown in steps 909 to 911 are repeated.

  On the other hand, if the next operation does not exist in step 912 (step 912, Yes), that is, if all the operations indicated by the macro read in step 903 have been executed, the user browsing on the browsing window 301 is performed. The reproduction of a series of web browsing operations starting from the operation is completed.

  Here, for example, as shown in FIG. 9F, when another anchor in the Web page displayed in the browsing window 301 is clicked in a state where the reproduction of the Web browsing operation is started, Steps 405 to 408 are performed. In addition, the processing shown in steps 901 to 912 is executed, the macro window A corresponds to the Web page displaying the map of the store corresponding to the newly clicked anchor, and the macro window B corresponds to the newly clicked anchor. Web pages displaying coupons of shops to be displayed are respectively displayed, and parallel reproduction of a plurality of continuous Web browsing operations is realized.

[First embodiment]
In the above embodiment, different web browsing operations are reproduced on a plurality of macro windows, starting from a user browsing operation on the browsing window 301, for example, a click operation of an anchor.

  Here, as one embodiment, when a plurality of objects on the browsing window 301, for example, a plurality of anchor objects are selected, and a user performs a browsing operation on one of the objects, It is conceivable that a new macro window is displayed corresponding to the number, and the same web browsing operation is reproduced in parallel on each macro window starting from the browsing operation of the user.

(6) Object selection / cancellation:
First, a processing procedure in selecting a plurality of objects will be described with reference to FIGS. 3, 14, 16, and 17 and a flowchart shown in FIG.

  FIG. 14 shows a browsing window, FIG. 16 shows an example of data in the window management table 15, and FIG. 17 shows an example of data in the macro storage unit 14.

When a selection operation of a plurality of objects occurs, for example, as shown in FIG. 14A, when an area including an object to be selected is dragged (step 1301),
The operation receiving unit 111 receives event information generated in accordance with the operation, and turns on the multiple selection flag (step 1302).

  Although not shown in FIG. 3, the multiple selection flag is managed by a storage unit such as a memory in the Web browsing operation recording / reproducing apparatus 10.

  Next, as shown in FIG. 14B, the display control unit 13 displays a new macro window on the display device 30 corresponding to the number of selected objects (number of objects-1), A Web browsing operation playback unit 12 corresponding to a new macro window is newly generated in the Web browsing operation recording / playback apparatus 10 (step 1303).

  Further, as shown in FIG. 14, an item for a new macro window is added to the window management table 15 (step 1304).

  Here, in the column of the ID of the macro window newly added to the window management table 15, the child number of the ID of the macro window already registered in the window management table 15 (in this case, the ID of the macro window A = 1) is displayed. Are assigned in order, and the macro of the macro window already registered in the window management table 15 (in this case, the macro of the macro window A / .Firnd_map.mac) is copied to the macro column.

  In the window name column, as described in (2) Addition / deletion of macro window, information specifying each macro window and the Web browsing operation reproducing unit 12 corresponding to each macro window, for example, Describe the window name, window handle, etc.

  Further, the event analysis unit 112 acquires information on a plurality of objects selected by the user based on the received event information, and fills it in the selected object management list (see FIG. 17) (step 1305).

  Here, the information about the object is the same as the information (see FIG. 7) of the event execution destination object output as a macro in the above-mentioned (1) Web browsing operation record, and the DOM, text, display position, link of each object. The destination URL, URL, comment information, etc. are output to the selected object management list in the order of selection.

  Although not shown in FIG. 3, the selected object management list is managed by storage means such as a memory in the Web browsing operation recording / reproducing apparatus 10.

  Here, as an operation for selecting a plurality of objects, in addition to the drag operation described above, for example, an arbitrary object may be continuously clicked while a shift keyword is pressed.

  In FIG. 14A, the selection operation of a plurality of objects is an operation in the reproduction start state of the web browsing operation, but it is not necessarily in the reproduction state.

  On the other hand, when a release operation of a plurality of objects occurs, for example, when an area where nothing is displayed on the browsing window 301 is dragged, the operation reception unit 111 receives the operation and turns off the multiple selection flag.

  Next, the display control unit 13 refers to the window management table 15, specifies the macro window newly displayed by the selection operation of a plurality of objects using the ID of each macro window as a keyword, and displays the specified macro window. While deleting from the apparatus 30, the Web browsing operation reproduction | regeneration part 12 corresponding to each macro window is deleted. Then, the corresponding item is deleted from the window management table 15.

(7) Parallel playback of the same web browsing operation:
A processing procedure in the Web browsing operation recording unit 11 in the reproduction of the Web browsing operation when a plurality of objects are selected is described with reference to FIGS. 3, 14, 16, and 17 and a flowchart shown in FIG. It explains using.

  FIG. 18 is a flowchart of parallel reproduction processing of the same Web browsing operation according to the first embodiment of this invention.

  In the figure, the processing procedure when a plurality of objects are selected is assigned the same step number as that in FIG. 10 to the same part as the processing procedure described in the above (4) Web browsing operation reproduction. However, the only difference is that the message transmission process (step 808) is different.

  First, the operation receiving unit 111 receives a reproduction start request for a web browsing operation from the user (steps 801 and 802), and then shifts to a mode for monitoring the browsing operation of the user on the browsing window 301 (step 803). ).

  After that, for example, as shown in FIG. 14B, when the user clicks one of the plurality of objects previously selected (here, the anchor [5]), the operation reception unit 111 sets OnClick (document .a [15]) is received as an event function (step 804, Yes).

  Then, the event analysis unit 112 acquires the event type “Click” and the event execution destination object “A [15]” of the operation performed by the user from the function name of the received event function (step 805).

  Further, the URL of the Web page displayed in the browsing window 301 is acquired (step 806).

  Next, the event type (Click) of the operation performed by the user acquired in step 805, information on the event execution destination object (A [15]) information), and the browsing window 301 acquired in step 807 are displayed. A message describing the URL of the Web page being displayed is created (step 807).

  If the multiple selection flag is ON (step 1501, Yes), the message transmission unit 115 is displaying the event type “Click” performed by the user in the browsing window 301 based on the message created in step 806. The Web page URL is the same, and a message is created in which the event execution destination object information is changed to the information of each object in the selected object management list (see FIG. 17) (step 1502).

  Here, four messages changed to information of A [11], A [12], A [13], and A [14] are newly created as information of the event execution destination object.

  Next, in the message sending unit 115, the message receiving unit in the Web browsing / playing unit 12 corresponding to each macro window (macro windows A, B, C, D, E, F) of the window management table 15 (see FIG. 16). Each message created in step 806 and step 1502 is transmitted to 121 (step 1503).

  At this time, which message is to be transmitted to which macro window is arbitrary. For example, as shown in FIG. 14B, the object clicked by the user (A [15], [5 in FIG. 14A). ]) Is transmitted to the macro window A, and the remaining objects (A [11], A [12], A [13], A [14], [1] in FIG. 14A) are transmitted. , [2], [3], and [4] may be transmitted to the macro windows C, D, E, and F sequentially from the top of the window management table 15 (see FIG. 16).

  In addition, it is conceivable that each message transmission destination is determined so that the arrangement relationship of the selected objects in the Web page corresponds to the arrangement relationship of each macro window on the display device 30.

  Thereafter, the browsing operation unit 114 invalidates the browsing operation of the user performed on the browsing window 301 (step 809), and when the macro execution flag is ON (step 810, Yes), the process returns to step 803 to return to the browsing window. The user's browsing operation on 301 is continuously monitored. On the other hand, when the macro execution flag is OFF (step 810, No), the reproduction of the web browsing operation is terminated.

  Next, a processing procedure in the web browsing operation playback unit 12 in the playback of the web browsing operation when a plurality of objects are selected will be described with reference to FIGS. 3, 14, 16, and 17. At this time, the processing procedure is the same as the processing procedure (the flowchart shown in FIG. 11) in the reproduction of the above-described (4) Web browsing operation.

  First, each message reception unit 121 receives each message addressed to each macro window transmitted from the message transmission unit 115 (step 901), and displays a Web page indicated by the URL described in the message in each macro window. (Step 902). Here, the URL described in each message is the same, and the web page being displayed in the browsing window 301 is displayed in each macro window.

  Next, each macro matching unit 122 refers to the window management table 15 and refers to each macro window management table 15 to read a macro (file) set in each macro window (step 903).

  Here, as shown in FIG. 16, the macros (files) set in each macro window are the same, and the same Web browsing operation is reproduced in each macro window.

  Then, the operation indicated by the macro No. 0 read from the macro storage unit 14 (see the No. 0 line in FIG. 7) and the operation described in each message, specifically, the event type and event execution of both operations The DOM object type of the previous object is collated (step 904).

  Here, the event type of the operation described in each message is common to “Click”, and the event execution destination objects (A [11], A [12], A [13], A [14], A [15 ]) Is the same object type “A”. On the other hand, the event type of the operation indicated by the macro No. 0 of the macro storage unit 14 is “Click”, the DOM object type of the event execution destination object is “A”, and the collation result is “ "Yes" (step 905, Yes).

  Thereafter, each macro execution unit 123 executes the operation content described in each message on each macro window (step 906).

  Here, the click operation is executed for the event execution destination object (A [11], A [12], A [13], A [14], A [15]) described in each message, and the result Each macro window displays a store introduction page indicated by each anchor object (see FIG. 17C).

  Next, the macro collation unit 122 receives an update / read event of a Web page accompanying execution of an operation on each macro window (step 907, Yes), and then increments the operation counter by 1 and reads it in step 903. With reference to the macro, the operation indicated by the No. that matches the operation counter is acquired (step 908).

  Then, an object that matches the event execution destination object indicated by the acquired operation is searched for in the Web page displayed in each macro window (step 909).

  Here, as shown in FIG. 16, the macros set in each macro window are the same, and “map” is provided as text information (alt attribute) from the introduction page of each store displayed in each macro window. Each of the image objects is derived (step 910), and a click operation is performed on the image objects (see the row No. 1 in FIG. 7).

  As a result, as shown in FIG. 14D, each macro window displays a web page displaying a map of each store (step 911).

  Thus, by using the form shown in the first embodiment, the same Web browsing operation can be reproduced in parallel for a plurality of objects arbitrarily selected by the user.

  Here, in FIG. 14 (d), the window size of the newly displayed macro window is reduced. This window size can be freely enlarged / reduced by the user by means determined separately.

  In addition, when a new macro window is displayed on the display device 30 corresponding to the number of selected objects, a plurality of macro windows are displayed using tabs as shown in FIG. As shown in FIG. 14 (f), there are various presentation forms such as dividing an existing macro window into frames (the frame is not displayed) and displaying a list of web browsing operations in each macro window.

[Second Embodiment]
In the first embodiment, the system configuration and the processing flow when a parallel playback system for a plurality of Web browsing operations is constructed on a certain computer have been described.

  Here, as described in the above (4) Reproduction of Web browsing operation, in this system, at the time of reproduction of the Web browsing operation, messages are exchanged between the Web browsing operation recording unit and the plurality of Web browsing operation reproducing units. Realizes parallel playback of multiple web browsing operations.

  Therefore, as one example, as shown in FIG. 19, a parallel playback system for a plurality of Web browsing operations is mounted on a plurality of computers (clients), respectively, and when a Web browsing operation is played back, via the network 40, It is conceivable to exchange messages between parallel playback systems for Web browsing operations implemented on different clients. In FIG. 19, the same components as those in FIG. 3 are denoted by the same reference numerals, and the description thereof is omitted.

  Specifically, when the message transmitting unit 115 transmits a message to each message receiving unit 121 (step 808) in the reproduction of the above-described (4) Web browsing operation, first, a plurality of devices mounted on the clients 1 and 2 are installed. The client transmission list of each client described in the client management list 60 is referred to by referring to the client management list 60 that describes information for identifying the parallel playback system of the Web browsing operation of the client, for example, the IP address of each of the clients 1 and 2 A message is transmitted to 115.

  Next, after receiving the message, the message sending unit 115 of each client refers to the window management table 15 in the Web browsing operation recording / reproducing apparatus 10 installed in each of the clients 1 and 2, and then sends each client 1, 2. The message is transmitted to all the macro windows on the display device 30, more precisely, to the message receiving unit 121 in the Web browsing / playing unit 12 corresponding to each macro window (the processing described in step 808).

  Here, the client management list 60 is stored in a storage means such as a hard disk in the Web browsing operation recording / reproducing apparatus 10, and the client management list 60 is added or deleted manually between the Web browsing operation recording / reproducing apparatuses. It can be managed by exchanging update messages.

  As described above, by using the second embodiment, it is possible to reproduce the web browsing operation in parallel between a plurality of clients. For example, the web browsing operation is reproduced in parallel on a computer owned by a plurality of users. Response to remote collaboration work is also possible.

  Further, the operation of the above-described parallel playback device for browsing multiple Webs can be constructed as a program and installed in a computer used as the parallel playback device, or distributed via a network.

  It is also possible to install the constructed program on a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM connected to a computer used as a parallel playback device.

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to an apparatus for recording / reproducing a web browsing operation.

It is a principle explanatory view of the present invention. It is a principle block diagram of this invention. It is a block diagram of the Web browsing operation recording / reproducing apparatus in one embodiment of this invention. It is a figure which shows the presentation screen in one embodiment of this invention. It is a figure which shows the display of the browsing window at the time of recording of Web browsing operation in one embodiment of this invention. It is a flowchart of the recording process of Web browsing operation in one embodiment of this invention. It is a data example (the 1) of the macro storage part in one embodiment of this invention. It is an example of the window management table in one embodiment of this invention. It is a figure which shows the display of the browsing window at the time of reproduction | regeneration of Web browsing operation in one embodiment of this invention. It is a flowchart of the process of the browsing operation recording part at the time of reproduction | regeneration of Web browsing operation in one embodiment of this invention. It is a flowchart of a process of the web browsing operation production | generation part at the time of reproduction | regeneration of web browsing operation in one embodiment of this invention. It is a data example (the 2) of the macro storage part in one embodiment of this invention. It is a flowchart of the search process of the matching object in one embodiment of this invention. It is a figure which shows the display of the browsing window at the time of the object selection / cancellation of 1st Example of this invention. It is a flowchart of the selection / release operation | movement of the several object in 1st Example of this invention. It is an example of data of the window management table of 1st Example of this invention. It is an example of the selection object management list in 1st Example of this invention. It is a flowchart of the parallel reproduction | regeneration processing of the same web browsing operation of 1st Example of this invention. It is a block diagram of the parallel reproduction | regeneration system of several web browsing operation of 2nd Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 2 Client apparatus 10 Parallel reproduction | regeneration apparatus 11 of web browsing operation 11 Web browsing operation recording means, Web browsing operation recording part 12 Web browsing operation reproducing means, Web browsing operation reproducing part 13 Display control means, Display control part 14 Macro recording means , Macro storage unit 15 window management table, window management table 20 input device 30 display device 40 network 50 Web server 60 client management list 111 operation reception unit 112 event analysis unit 113 macro generation unit 114 browsing operation unit 115 message transmission unit 116 message reception Unit 121 message receiving unit 122 macro checking unit 123 macro executing unit 201 cursor 301 viewing window 302 macro window A
303 Macro window B
310 Macro Controller 311 Record Button 312 Play Button 313 File Button

Claims (3)

A parallel playback device for a plurality of Web browsing operations for realizing recording and playback of Web browsing operations,
Web browsing operation recording means for receiving event information generated by a user's Web browsing operation from the user's input device, and acquiring information on an operation performed by the user and information on an execution destination object;
Macro storage means for storing a series of web browsing operations acquired by the web browsing operation recording means as macros;
Web browsing operation reproduction means for executing a macro stored in the macro storage means in accordance with the browsing operation of the user and the content of the Web page being displayed;
Display control means for managing a window on the display device used when recording and reproducing a web browsing operation;
A window management table provided on a window on the display device, and storage means for managing a macro set in each window;
Have
The web browsing operation reproduction means includes:
Message receiving means for receiving a message created for each window for reproducing each web browsing operation and transmitted from the web browsing operation recording means;
The window refers to the management table, read from the macro memory means the macro that is set in each window, and compares the read operation Description and received operation content information described in the message macro shown, Web Macro collation means for determining whether or not to reproduce the browsing operation, analyzing a web page displayed in the window, and searching for an object suitable for the execution target of the read macro operation ;
When the macro matching means derives a macro operation that matches the operation content information described in the message and an object that matches the execution target of the macro operation, the operation content information described in the received message Macro execution means for executing the operation content indicated by the macro on the conforming object derived by the macro collating means,
The macro matching means includes
When the Web page displayed in each window is analyzed and the macro storage means is searched for an object that matches the execution target of each macro operation, the Web page is analyzed and described in the operation content indicated by the macro. Means for obtaining a list of objects having the same type of tag used in HTML and DOM (Document Object Model) representing the object,
The similarity between the object described in the operation content indicated by the macro and each object in the list of objects is based on the similarity of the display position of the object, the similarity on the DOM structure, and the similarity of the link destination URL. Means for calculating;
A means for adapting an object having the highest similarity among the objects whose similarity is calculated ,
When realizing parallel playback of the same Web browsing operation for multiple objects,
A selection object management list provided on storage means for storing information on a plurality of objects selected by the user;
The display control device includes:
In response to the number of objects selected by the user, a new window for reproducing the Web browsing operation is displayed, and the Web browsing operation for the window already displayed on the display device is displayed. Copying to the window management table as a web browsing operation of the window, the web browsing operation reproduction means also operates for a new window,
When applying the web browsing operation,
Based on the received event information, the operation performed by the user, the information of the object operated by the user, and the information of the Web page displayed in the window operated by the user Event analysis means for creating a message in which the information of the object of the message is changed to the information of each object in the selected object management list,
A message transmission unit that refers to the window management table and transmits each message created by the event analysis unit to a Web browsing operation reproduction unit corresponding to each window on the display device. A parallel playback device for a plurality of Web browsing operations. A parallel playback program for a plurality of Web browsing operations for causing a computer to function as each means constituting the parallel playback apparatus for a plurality of Web browsing operations according to claim 1 . A computer-readable recording medium storing the parallel reproduction program for a plurality of Web browsing operations according to claim 2 .

Images