JP2011159281A - System and method for recording and reproducing web-based mobile task - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and a device for recording and reproducing a web-based task. <P>SOLUTION: This method or the like is constituted to record the web-based task executed during a session carried out just before by a user without giving expressly a sign about start of the record session, and includes automatic finding of the start of the session without involvement of the user, reception of a request for reproducing the recorded web-based task, and the reproduction of the recorded web-based task. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

(priority)
[0001] This patent application is a priority over the corresponding provisional patent application 61 / 286,724, filed December 15, 2009, entitled "System and methods for record & replay of web-based mobile tasks". Claims and incorporates this provisional patent application by reference.

(Field of Invention)
[0002] Embodiments of the present invention relate to the field of recording and playback of tasks performed on a device by a user. More particularly, embodiments of the present invention relate to recording and playback of web-based tasks for mobile handset users.

(Background of the Invention)
[0003] Mobile Broadband and Advanced Mobile Platforms (such as iPhone (R) from Apple (R) computers in Cupertino, California, Android (R) from Google (R) in Mountain View, California, and WebOS platforms ) Greatly accelerated the spread of mobile device use for everyday web-related tasks. Increasingly, consumers use their smartphones to shop online, write blogs and comments, and write specific online information that changes dynamically (like a match from a favorite team) Information, and the current status of auction items that are the earliest expired items being collected).

  [0004] Nevertheless, performing web-related tasks from a mobile device still has some usability concerns, compared to personal computers (PCs) with fixed-line high-speed connections, and these concerns It originates from two characteristics inherent in such mobile devices: 1) weaker and less stable connections, and 2) smaller form factors.

  [0005] Despite recent technological advances, cellular network connections are still relatively relative to their fixed line connections, especially when devices are near the edge of the cell and / or when there is severe interference. Slow and inferior in stability. This results in longer and less predictable network latencies and user latencies, particularly in the case of web sessions that include user actions and Internet request / response sequences, and therefore the user experience is inferior. The small mobile device form factor also makes it much more difficult to interact with mobile devices. On a small screen, it is difficult to read what is displayed, or it is difficult to click / tap the correct position. The lack of a full-size keyboard and stable arm support makes it difficult to enter text or to transfer data from one application to another.

  [0006] With this in mind, a user viewing technology-related news is gizmodo. Imagine you find an interesting article at com (see Figure 1a) and want to cite this article on your blog and make a comment. To do this, the user highlights and copies an interesting paragraph of the article, then blogger. com and log in by typing in a username and password (see FIG. 1b). The user then clicks “New Post” to create a new blog entry (see FIG. 1c), where gizmodo. The past copied paragraph (possibly including the URL of the article) from the com article is pasted (see FIG. 1d). This seemingly simple task is much less simple on mobile devices. The reason is not only because it is difficult for the user to type and copy / paste, but also because the connectivity of the device can be variable and sometimes bad, the user has to wait for multiple pages to load slowly. It is because there is.

  [0007] There are several prior art solutions to these problems. WebVCR provides a VCR type user interface that allows a user to record and play back browsing steps. WebVCR is intended to be used to create shortcuts for accessing web data that requires multiple navigation steps, and is also used as a basis for web applications such as online tutorials and e-commerce You can also WebVCR has some limitations. First, WebVCR does not specifically target mobile web-based tasks. WebVCR assumes that the user is controlling session creation. That is, the user explicitly starts and ends session recording in the WebVCR. As a result, WebVCR does not address the need for subsequent automatic session tagging or the need for efficient session playback.

  [0008] Smart Bookmarks introduces the concept of retrospective recording for web navigation. The user can ask the system to bookmark the current page, and the system will retroactively search for a bookmarkable start page and a series of commands to go from the start page to the current page. become. Smart Bookmarks also has limitations. For example, retrospective recording is part of Smart Bookmarks, but automatic recording in Smart Bookmarks stops at the bookmarkable start page, which makes it impossible to automatically identify sessions related to multiple websites It becomes. In addition, similar to WebVCR, Smart Bookmarks also does not address efficient session playback.

  [0009] CoScripter is also a system that can record and autoplay user interaction steps for web browser based tasks. The user can explicitly request to record his interaction with the browser and save the recorded information as a script. Such scripts can be shared with other users and can be automatically played back later. However, CoScripter is similar to WebVCR in that it requires proactive recording. Like the above two systems, CoScripter does not specifically address the problem of adapting to the user's interaction context during playback, and does not place importance on efficiency in such playback.

  [0010] Highlight builds a re-authoring environment that allows mobile users to manually customize web-based applications that were originally designed for desktop users. Highlight uses a programming-by-demonition-based approach similar to CoScripter, allowing the user to directly specify what should be included on the mobile page. Another exemplary programming-based system for the mobile web is PageTailor, which records user customizations to web pages and automatically applies such recorded customizations to similar web pages. . Note that both Highlight and PageTailor are directed to individual web pages rather than interactive sequences.

  [0011] Thus, existing solutions have not fully addressed the above problems. Although there is conventional work on recording and playing back web-based user interaction sequences, such existing work is not specifically targeted at the mobile web, and varies to some extent in one or more of the following aspects: Still limited.

  1) Automated task recording without preparation. Rather than requiring the user to manually signal the start of a recording session, the user should be able to ask the system to record the dull session that has just been completed. In addition, the system should be able to automatically resolve the interaction events that make up the session to be recorded.

  2) Adaptive playback of recorded sessions. The recorded session should be usable for similar but slightly different tasks and should be able to respond appropriately to changes in web content.

  3) Efficient playback of recorded sessions. For mobile users, long latencies are as cumbersome as boring conversations. Playback of a recorded session should provide the user with a much better experience than when the user previously recorded the session, and this better experience should involve a much shorter task completion time .

(Summary of Invention)
[0012] A method and apparatus for recording and playing back web-based tasks is disclosed herein. In one embodiment, the method is to record web-based tasks performed during a session that has just been performed by a user without explicitly signaling the start of the recording session, Including automatically finding the start of a web page, receiving a request to play a recorded web-based task, and playing a recorded web-based task.

  Embodiments of the present invention relate to the field of recording and playback of tasks performed on a device by a user. More particularly, embodiments of the present invention relate to recording and playback of web-based tasks for mobile handset users.

a to d: Screenshots of browsers regarding examples of comment entry on a blog / message board. a and b: shows a user interface for tagging a session and playing a session. It is a figure which shows the illustration of the event sequence regarding a session production | generation, and the event is located in order with generation | occurrence | production. FIG. 4 illustrates pseudo code for one embodiment of a context recognition algorithm. It is a figure which shows the context recognition in session reproduction | regeneration. It is a figure which shows the list | wrist of the example of the generalization plan arranged in order with a high priority. FIG. 6 illustrates pseudo code for one embodiment of “smart” event playback. FIG. 6 illustrates remote execution of a tagged session. a to d: Browser screenshots for the “Expired eBay Auction” example. It is a figure which shows the example of the recorded session. a to d: Screenshots of browsers for an “Express Checkout” example. 1 is a block diagram of one embodiment of a computer system.

  [0013] The invention will be more fully understood from the detailed description provided below and the accompanying drawings of various embodiments of the invention. However, the drawings should not be construed as limiting the invention to the specific embodiments, but are for explanation and understanding only.

  [0014] A system and associated methods for recording (or tagging) and playing back tasks are described. In one embodiment, this task is a mobile web-based task. In one embodiment, the technique for recording and playing back tasks has three distinct features. First, these techniques include post-mortem automatic session tagging, which allows a user to tag a session that has just been completed without first signaling the start of recording, without any user intervention. Automatically detect the start of the session and remove the dialog noise. Second, these techniques include “smart” session playback, where recorded sessions adapt to contextual differences between when the user tags the session and when the user plays the session. , Essentially allowing parameterization of tagged sessions. In one embodiment, “smart” session playback techniques also adapt to web page changes to withstand web dynamics and accommodate user task changes. Third, these techniques involve remote session execution, and instead of re-executing the event from the mobile device, a large portion of the tagged session is played at the proxy server. The proxy server may have a fixed line high speed connection. With such remote execution and other optimizations, the time to replay the session is reduced to the time required to load the last page in the session.

  [0015] In one embodiment, session recording and playback techniques are implemented using front-end and back-end components, which have the distinguishing features described above (and discussed in detail in the following sections). Implemented, the front-end component handles user interaction.

  [0016] Specifically, the front-end component monitors the user's interaction state and interaction context, which is defined herein by a number of member variables that are currently open in the browser. Active page, highlighted text on the page currently open in the browser, clipboard content, form input made to the page currently open in the browser, and resource identifier (eg URL) Contains the current value of the input bar (if any) and one or more of them.

  [0017] An interaction context is defined herein as the user's most recent interaction state sequence. A transition from one dialog state to another dialog state corresponds to a dialog event. The front-end component intercepts (intercepts) interaction events from within the browser and closely monitors the state of user interaction. In one embodiment, there are the following types of interaction events: That is, a page load event (clicking on a hyperlink, pressing the enter key in the URL bar, or selecting from a drop-down list presented by the browser based on the contents of the URL bar, form submit button As a result of a user action, such as clicking, or clicking a particular browser toolbar button, or selecting some browser menu item (eg, a “Home” or “Back / Forward” button, and a bookmark menu) URL input event (which corresponds to the user typing in the URL bar), text highlight event (which corresponds to the user highlighting a text string on the current page) To display Corresponding), text copy event (which corresponds to the user copying the highlighted text to the clipboard), form input event (which the user enters in the input field of the form on the current page) Corresponding to (form input events can be triggered by the user typing text into the input field or pasting text into the input field) A special form of input event, which corresponds to the user copying the clipboard contents to the input field of the form on the current page), as well as a URL copy event (which copies the URL of the current page to the clipboard). Corresponding to).

  [0018] In one embodiment, all types of events have their own unique attributes as well as common attributes that indicate their start and end times.

  [0019] Similar to bookmarking a page after viewing the page, the user records or “tags” a task that has just been completed, which may not be easy to perform but is likely to be repeated in the future. You can ask them to “attach”. In one embodiment, the user can also assign a name to this recorded session (see, eg, FIG. 2 (a)). This type of session recording is referred to herein as “post-event session tagging”. The reason is that “recording” is performed without the user signaling the back-end component to begin recording the session and is performed after the session is completed.

  [0020] In the background, the back-end component obtains the user's current conversation context, from which it determines which events and states represent the session that the user wants to record and tag, Such events and states are stored with the assigned name.

  [0021] Later, when the user attempts to play a tagged session via the front-end user interface (see, eg, FIG. 2 (b)), the back-end component is in an interactive state corresponding to the tagged session. And obtain an event and then attempt to play back the recorded event of the session. However, in one embodiment, such playback is usually not an exact “carbon copy” re-execution of the original event. This is because the current user interaction context (and hence the user's intended task) may be different than when the user created the session tag. Another reason is that the web page that the user visited when the session was tagged has since made no more verbatim playback of some events (eg highlighting missing phrases). It may have changed before display. In one embodiment, the backend component therefore requires “smartness” in event playback to adapt to changes in user context and web content.

  [0022] In one embodiment, the backend component does not target a faithful imitation of manual user interaction in session playback. Rather, it takes advantage of various approaches, including remote execution of most of the sessions, for more efficient (and therefore more user-friendly) playback of tagged sessions.

  [0023] In the following description, numerous details are set forth to provide a more thorough explanation of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

  [0024] Some portions of the detailed descriptions that follow are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their tasks to others skilled in the art. The algorithm is here and generally considered to be a consistent series of steps leading to the desired result. These steps are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

  [0025] However, it should be borne in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As will be apparent from the discussion below, terms such as “process,” “calculate,” “calculate,” “determine,” or “display” are used throughout this description unless otherwise specified. The discussion utilized manipulates data represented as physical (electronic) quantities in computer system registers and memory to store, transmit, or display in computer system memory or registers, or other such information. It should be understood that it refers to the actions and processes of a computer system or similar electronic computing device that translates into other data that is also represented as physical quantities within the device.

  [0026] The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer-readable storage medium, and the computer-readable storage medium is not limited to the following, but includes a floppy (registered trademark) disk, an optical disk, a CD-ROM, and a magneto-optical disk. Any type of disk, read only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical card, or any type of medium suitable for storing electronic instructions, etc. Each coupled to a computer system bus.

  [0027] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it may be advantageous to build a more specialized apparatus for performing the required method steps. Sometimes it turns out. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

  [0028] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (eg, a computer). For example, machine-readable media include read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, and the like.

(Overview)
[0029] A system and associated methods for recording (or tagging) and playing back tasks are described. In one embodiment, the method is to record web-based tasks performed during a session that has just been performed by a user without explicitly signaling the start of the recording session, without user involvement. Including automatically finding the start of a session, receiving a request to play a recorded web-based task, and playing a recorded web-based task.

  [0030] In one embodiment, recording a web-based task is to monitor a user's interaction state and interaction context, wherein the interaction context is the most recent interaction state sequence, and a browser Intercepting dialogue events from within, each dialogue event corresponding to a transition from one dialogue state to another. In one embodiment, recording a web-based task segments the event stream, removes events that have no effect on the task from the event stream, and produces a de-noised event stream Determining dependencies between events in a segmented denoised event stream, and de-interventing events as part of a replayable session, traversing the dependency chain back from the last segment of the event Identifying from the generated event stream.

  [0031] In one embodiment, segmenting the event stream is performed based on time latency between events in the event stream. In one embodiment, segmenting the event stream is performed based on explicit user input actions.

  [0032] In one embodiment, removing an event from the event stream is performed without user involvement. In one embodiment, removing events from the stream includes removing events that occur as a result of user navigation that is not critical to the task.

  [0033] In one embodiment, traversing the dependency chain comprises one or more pairs of actions selected from the group consisting of highlight and copy operations, copy and paste operations, and input and submit operations. Including identifying.

  [0034] In one embodiment, the method further includes generalizing the session representing the task so that the session is applicable to a different context than the context used when the session was recorded. In one embodiment, generalizing the session includes abstracting the session.

  [0035] In one embodiment, playing the recorded web-based task includes a recorded session context that represents the recorded web-based task, the first associated with the user when the session was recorded. Replaying a recorded session using the user's current conversation context, adapted from the user's current conversation context. In one embodiment, adapting the context includes determining which state in the recorded session matches the user's context at the time the task playback request was made. In one embodiment, playing the recorded web-based task is adapted to adapt to changes in at least one web page that is part of the recorded web-based task. Adapting a recorded session representing.

  [0036] In one embodiment, playing the recorded web-based task includes playing the recorded session representing the recorded web-based task remotely using a proxy server. In one embodiment, playing a recorded session that represents a recorded web-based task remotely using a proxy server sends the recorded session to the proxy server along with the user's latest interaction state. Replay events with a proxy server using the state from the user's interactive state, including sending one or more requests to one or more servers and receiving responses from one or more servers Sending the last page corresponding to the last page load event in the session, along with the current interaction state on the proxy server, and loading the resource specified by the last page; Including completing the playback of one or more events that terminate the session.

(Session generation)
[0037] In one embodiment, session generation occurs in a post-hoc manner. That is, the user does not need to signal the system to begin recording before performing the sequence of interactions. Rather, the user can ask the system to mark the session that the user has performed immediately after ending the session. This has at least two benefits. First, it is believed that the user is more willing to create a session tag immediately after performing such a session than immediately before the start of a tedious web session. This is because at the start of a session, the user may not yet be aware of the boredom or reusability of the session, or may not have concentrated on completing the task without interruption. Second, post-session generation is important because many such sessions are not suitable for being recorded in advance. This means that repeating such a session can result in unwanted side effects (such as buying small items from an online store or leaving comments in public forums) or too boring for the user ( Because it requires some typing and / or needs to wait for a slow loading of the web page).

  When a user requests the system to tag a session that has just been performed, the system needs to identify which user action is a component of the task of the user that has just ended. This includes two sub-targets. The first sub-goal, referred to herein as “start state recognition”, is to identify the state corresponding to the start of the intended task from the sequence of monitored interaction events. The second sub-goal is to eliminate noise events between the intended task start state and the intended task end state, which is when the user requests creation of a session tag. Is currently in a conversational state.

  [0038] FIG. 3 shows an event sequence for session generation. Referring to FIG. 3, a session tag creation request 301 is generated after the intended task end state. It also shows several states in which events are arranged in order of occurrence, including noise events.

  [0039] In one embodiment, start state recognition and noise event removal are performed using an algorithm that includes the following high-level ordered steps:

  1) The event sequence is divided into a plurality of segments (division).

  2) Mark events that directly determine each member variable as relevant (mark).

  3) Remove events unrelated to the task from each segment (removal).

  4) Extend the relationship from the related segment to the non-related segment (expansion).

  5) Optimize the resulting event by removing trivial navigation sequences (optimization).

  [0040] After the algorithm is complete, all related segments (removed noise events, respectively) form the session state and event tagged, and the first state of the earliest related segment is the session start state It is.

(Split)
[0041] The split operation divides user interaction events into small groups or segments. Events within the same segment are closely related to each other, and there is usually a causal relationship between any two adjacent events. For example, a segment may be blogger. com page home page followed by a form input event in the user name field on this page.

  [0042] In one embodiment, the split operation is performed by identifying a split event. In one embodiment, each split event separates the events before and after it (including itself) into different segments. In one embodiment, there are two types of split events. The first type of split event is a URL submit event, which is caused by the user explicitly typing in a URL address, or by the user selecting a bookmark item or clicking the “Home” button. It is. The second type of split event is an event in which the time interval between the start time and the end time of the previous event is abnormally large. In one embodiment, the determination of whether the time gap is abnormally large is such that the time gap is marked as abnormally high (e.g., 2x, 3x, etc.) that is somewhat larger than the normal time gap. It can be based on the relative length of time that normally occurs between events. The reason for splitting in these two types of events is that these events are likely to indicate that the user may have switched from one task to another.

(mark)
[0043] The mark operation marks events that directly determine each member variable as relevant. For example, a page load event that loads a final state document is marked as relevant. Other marked events are a text copy event that sets the clipboard contents of the final state of the last event session, and a URL input event that sets the URL bar contents of the final state of the last event of the session.

(Removal)
[0044] The remove operation causes events that are not relevant to the current task to be excluded from each segment derived from the aforementioned split operation. Such unrelated events are primarily assumed to be the result of noise or mistakes in the user interaction. The removal operation is performed after the dividing operation. Otherwise, it may adversely affect segmentation.

  [0045] In one embodiment, the session generation algorithm covers two types of unrelated events: noise events and unimportant navigation. In one embodiment, the removal operation first removes all noise events and then removes all events related to any non-critical navigation. In one embodiment, an event is defined as a noise event if it has not yet been marked as relevant and meets one of the following conditions:

  An event is a form input event or a paste input event that has no corresponding page load event in any segment. As used herein, “corresponding page load event” means a page load caused by a form submission that includes the target input field of the input event. Such a noise event may be caused by user input to the form being interrupted.

  The event is a text copy event or URL copy event, and there is no paste input event (not noise) between this event and the next copy event. Such an event is considered noise because the text value copied to the clipboard is replaced before it is used for pasting.

  The event is a text highlight event, and there is no text copy event (no noise) between this event and the next text highlight event on the same page.

  The event is a URL input event, and there is no corresponding page load event or (non-noise) URL copy event between this event and the next URL input event. Such a noise event is caused by a user input in the URL input bar of the browser being interrupted.

  [0046] Note that in one embodiment, noise event cancellation is performed in four ordered paths, each corresponding to one of the above conditions.

(Extended)
[0047] The extend operation extends the relevance from events that are marked as relevant during the mark operation to other events. First, for a given related event, the expansion operation marks all other events in the same segment as also relevant, then iteratively relevance from the already marked as related to other segments Extend to The iteration stops when no new related segment is found. In one embodiment, the relevance is determined if any of the following pairs of events exists across two segments (any of the pair of events occurring in the segment is already relevant): Can be extended from one segment to another. That is, a text highlighting and text copy event pair, a text copy and paste input event pair, a URL copy and paste input event pair, a URL input and page load event pair, and a URL input and URL copy event pair. . This essentially extends the data dependency both forward and backward across segments.

(optimisation)
[0048] The optimization operation removes trivial navigation sequences among related events. A trivial navigation sequence is an event sequence that can be removed without affecting the firing of the immediately following event or affecting the state after triggering that particular event, and 2) the same event. A sequence is defined as not including any page load events corresponding to form submissions.

  [0049] An example of a trivial navigation sequence is a simple navigation cycle. The user clicks on the hyperlink on the page and then presses the browser's “back” button. Another example is a trivial page load just before navigating to another location, for example by selecting a bookmarked item.

  [0050] The result of the session creation (tagging) algorithm is a sequence of interaction events (and corresponding outcome states) that are considered relevant to the task that the user wishes to tag. Although the tagging algorithm is relatively simple and can be applied during execution, in one embodiment, other post-processing may take a significant amount of time, and such post-processing is offline. Implemented outside the mobile device. Part of such post-processing is referred to herein as generalization of tagged sessions. Simply put, generalization is a technique used to make a tagged session applicable to other similar tasks in a context that is slightly different from the context in which the user created the session tag. .

  [0051] Since generalization is the key to understanding that a tagged session will be replayed later, generalization will be discussed in detail below before discussing session replay.

(Generalization)
[0052] When a user generates a request to play a previously tagged session, many things may have changed since then, so that each event in a tagged session is 100% faithful. It is impossible or meaningless to reproduce. For example, eBay. com's homepage changes frequently. In addition, the content of the page containing the item that expires earliest in a particular auction category, as well as the URL, may have changed.

  [0053] Generalization is the process of abstracting the state of a tagged session. The resulting abstract state is referred to herein as a generalized state, and the original state is referred to herein as a concrete state. Therefore, there are concrete events and generalized events.

  [0054] In one embodiment, the state consists of five member variables (document, URL input, form input, highlighted text, and clipboard content), so the generalization of the state is ultimately a member variable for each state. Is a generalization of Each member variable is generalized by its own generalized method, and the five generalized methods in the conversation state constitute the generalized plan for that state.

  [0055] One of the functions of the generalized method is to take a specific event and the context of the specific event and convert the specific event into a generalized event. Each generalized method also has an inverse function that takes a context (consisting of a concrete state and an event) and a generalized event and converts the generalized event into a concrete event. This inverse function is referred to herein as instantiation.

  [0056] Generalized methods generate an abstract state with the state as input. The state can include a web document as part of it.

  [0057] The following is a list of examples of generalized methods:

  NOGEN: This is a special generalized method that can be applied to all state member variables. This method does not do any generalization. That is, nothing is changed when general events are generalized or when generalized events are instantiated.

  ANY: This is also a special generalized method that can be applied to all state member variables. This method considers any value of a member variable to be equivalent to each other (even when no value is set).

  BIN: This is also a special generalized method that can be applied to all state member variables. This method only distinguishes between binary states whether or not the value of the state member variable is set. Thus, this function returns true or false when applied to a state.

  CLK_A: This is a generalized method for document variables based on a subsequent click on a hyperlink, form button, or list item in the page. In the case of a hyperlink or list item x, the document is abstracted to "contains a link to URLx", and in the case of a form submit button, the document is abstracted to "contains a form with action x and input item name y". It becomes.

  CLK_T: This is similar to the above, except that in the case of hyperlinks, the document is abstracted to “include hyperlink with anchor text x”. In the case of a form button, the document is abstracted to “include a form with action x and submit button value y (or image URIz)”. In the case of a list item, the document is abstracted to “include list item with embedded URL and text label x”.

  CLK_P: Same as above, with the exception that hyperlinks, form buttons, and list items are represented by their xpath in the document, which is well known in HTML document technology.

  HL_PATH: This is a generalized method for highlight variables based on the xpath of the text node containing the highlighted text and the offset of the highlighted text within the text node.

  HL_TREE: This is similar to the above, except that xpath starts at the root of the document subtree, not the root of the document itself.

  FI_1: This is a generalized method for form input variables. The form input destination is abstracted as “form having action x and input field name y”.

  URL_1: This is a generalized method for URL input variables. A complete URL in the form “http: //www2.xyz.com/path1/path2 / ...” is abstracted to “xyz.com/path1”.

  [0058] Other generalization methods may also be used, particularly generalization methods based on document content. For example, in one embodiment, a generalization method generalizes two documents containing the same map address or containing multiple images together into the same generalized document.

  [0059] FIG. 6 shows a list of sample generalization plans using the generalization methods described above. There may be several different methods for each state member variable, but since many method combinations are not feasible or redundant, the number of practical generalized plans is not exponentially large. Please keep in mind. Referring to FIG. 6, the generalized plans in the table are ordered by the “fidelity” (relative to the original concrete state) of the generalized states they produce.

  [0060] Once a session is tagged, its state is generalized using a similar generalized plan list (see, eg, FIG. 5). For each original concrete state, the result is a list of generalized states in order of increasing fidelity to the original state. Note that the generalized plan may fail to generalize one or more states (“x” in the chart of FIG. 5).

(Session playback)
[0061] As discussed above, when a user requests playback of a tagged session, the user typically does not want to repeat the exact same interaction sequence from the beginning of the tagged session. In fact, what the user thinks is the start of his session may be different from what the system inferred through the tagging algorithm. Therefore, it is important that the session playback logic automatically determines the user's context at the time the user requests playback and which state in the tagged session matches that context. . The matching state becomes the starting point for playing subsequent events in the tagged session one by one.

  [0062] Within the recorded session, a context is defined as a sequence of interaction states (as well as corresponding events) representing the most recent interaction with the browser by the user. The length of the sequence can be customized, but is usually short (eg 8 in one run). The most important information in the context is the last (ie latest) interaction state in the sequence.

  [0063] Since the latest state of the context C is S1, the i-th state of C is Si, the list of generalized plans arranged in descending order of priority is GPL, and the generalized plan gp is applied to the state Si. Let the generalized state derived be Sigp.

  [0064] The pseudo code in FIG. 4 describes one embodiment of an algorithm for context recognition. In essence, the algorithm takes each supported generalized plan, from most preferred to least preferred, and applies it to the current state of the context to produce a generalized state. If such a generalized state (Slgp) exists, then such generalized state is derived from applying the same plan to one of the states of the tagged session; Check for equality. If there is at least one such state, the first (earliest) state of such states is returned as a match state. If no generalized plan can produce a generalized state match, the algorithm returns the first state of the tagged session as the match state. This indicates that the tagged session needs to be played from the beginning.

  [0065] Note that in one embodiment, the generalization of state Sl takes both state itself and context C as inputs. This is because some generalization methods need to use additional information when generalizing state member variables. The same is true when the state during a tagged session is generalized.

  [0066] FIG. 5 is a more descriptive depiction of the context recognition process. When a session is tagged, its state is abstracted by a generalized plan. Generalized states derived from some plans maintain high fidelity to the original concrete state, while other generalized states are much more abstract and many distinct features of the concrete state Lose. When a user requests session replay, the user's final state is retrieved and an effort is made to find a generalized state match that maintains the highest fidelity to the final state.

  [0067] In addition to determining the starting point for session playback, context awareness can also be used to help identify the tagged session that the user wishes to play. This means that when the user opens a list of all tagged sessions, the backend system identifies a set of tagged sessions each with a high fidelity match to the user's context. Because you can. This set of tagged sessions can then be displayed more prominently (eg, highlighted or promoted to the top of the list) to help the user locate their location. This can improve the usability of the system when there are dozens or more of tagged sessions created.

  [0068] Once a match state within a tagged session is identified, the remaining session playback process starts from the user's interaction context and plays each event following the match state in the tagged session. .

  [0069] FIG. 7 is an example of pseudo code for an algorithm for “smart” event playback. Referring to FIG. 7, of the two while loops of the algorithm, the outer loop iterates through the tagged session events (starting with a match) and replays that event for each event (Et). Try that. If event playback fails, the entire process is aborted and an exception is thrown.

  [0070] Much of the complexity and the source of the “smartness” of the session replay algorithm is in the inner while loop. The inner loop checks each generalized plan (gp) that successfully generalized the next state (Su) in the tagged session, in order of priority. From gp, get the generalized method (gm) as well as the generalized event (Eg) based on the type of the original event Et in the tagged session. It then asks gm to instantiate Eg using the current concrete state Sc. If such instantiation is successful, the obtained concrete event (Ec) is used to derive the updated current concrete state.

  [0071] Events in a tagged session can be played one by one from the mobile device, but such playback is a proxy server with a fixed line high-speed connection to the Internet and higher computational capacity It may be better to run in Remote playback of the tagged session at such a proxy server reduces the number of requests and the amount of data passing through the slower, longer-latency wireless access network and in some cases minimized. Processing related to “smart” reproduction of each event is also accelerated. As a result, playing a session tagged with such remote execution is much shorter and improves the user experience.

  [0072] FIG. 8 illustrates operations related to remote execution of a tagged session. Referring to FIG. 8, a tagged session is first sent from the mobile handset 801 to the proxy server 802 along with the user's latest interaction state (1). The proxy server 802 then replays the event that follows the matching condition using the procedure described above. During this process, proxy server 802 sends requests to one or more web servers 803 and receives responses from such server (s) (2). At the last page load event, proxy server 802 gets the page from one of web servers 803 and then passes the page to mobile handset 801 along with the current interaction state on proxy server 802 (3). The mobile handset 801 then loads the last page of resources and completes playing the remaining events. Note that in step (2), because of the page load event, proxy server 802 need only download the HTML text of the page, and does not need to download resources such as images or scripts.

  [0073] Compared to a user performing a tagged session manually, session playback by remote execution is more efficient for the following reasons. This means that the page renders less, only the last page needs to be rendered on the mobile device, and less data is downloaded from the web server, with the exception of the last page, the download embedded in the page There are no resources to do, and the number of requests and the total amount of data going through the slower and longer latency wireless access network is lower. The last page plus a small amount of system overhead (tagged session and interaction state) is sent between the mobile device and the proxy server.

  [0074] FIG. 10 below shows a recorded session (for an example of an expired eBay auction described below) stored in text format. Referring to FIG. 10, a session has 10 states (including an initial state (in this case, state 0 which is the default start state)) and 9 events.

(Sample use case)
(Voluntary blog entry)
[0075] Voluntary blog entry was described above. In this use case, when the user requests session tagging, the system will prompt the user for gizmod. It is likely that the session will have been started since the com home page was opened. But then, when the user goes to another website, highlights some text in the article, and requests the system to play a voluntary blogging session, the system says "Gen Plan 6 (see Figure 6)" It recognizes that this is a generalized plan that abstracts the current state of the user and the state of the user after highlighting the Gizmodo article into the same generalized state. In this case, the system starts from here and plays the remaining events.

(Expired ebay auction)
[0076] In this example, whenever a user is interested in a particular collection (eg antique seal) and has one to two minutes free time, he wants to check what is currently auctioned on eBay. I think that it is. The user can go to the home page of eBay, type “Antique Seal”, and search on eBay (FIG. 9A). After the search results are loaded on a new page (Fig. 9 (b)), the user sorts the auction items by expiration time (Fig. 9 (c)) and finally clicks on the page for the item that expires the earliest (FIG. 9D).

  [0077] In this use case, the system prompts the user to start an ebay. com as the correct time to go. If the user is currently on a random website and requests to play a session, the system starts from the first state of the tagged session. The reason is that no generalized plan can abstract any other state and the user's current state in exactly the same way.

(Express Checkout)
[0078] In this example, consider that a user sometimes purchases an accessory using an online shopping site (FIG. 11 (a)). The user goes to the shopping cart, finds the desired product (FIG. 11B), and clicks the checkout button. As a result, the user is led to a form for inputting delivery and payment information (FIG. 11C), and finally goes to a page for final review of his / her order (FIG. 11D). Note that on the screen (FIG. 11 (c)), some information, such as customer name and delivery information, has already been automatically entered, but the user still needs to enter payment related information. The system is Amazon. For sites that do not support something similar to “Express Checkout” at sites such as com for various reasons, the user is given the option of creating it.

(Example of computer system)
FIG. 12 is a block diagram illustrating an example computer system that can perform one or more of the operations described herein. With reference to FIG. 12, the computer system 1200 may comprise an exemplary client or server computer system. Computer system 1200 includes a communication mechanism or bus 1211 for communicating information, and a processor 1212 coupled with bus 1211 for processing information. The processor 1212 includes a microprocessor, but is not limited to a microprocessor such as Pentium (registered trademark), PowerPC (registered trademark), Alpha (registered trademark), or the like.

  System 1200 further comprises random access memory (RAM) or other dynamic storage device 1204 (shown as main memory) coupled to bus 1211 for storing information and instructions to be executed by processor 1212. . Main memory 1204 may also be used to store temporary variables or other intermediate information during execution of instructions by processor 1212.

  Computer system 1200 also includes a read only memory (ROM) and / or other static storage device 1206 coupled to bus 1211 for storing static information and instructions for processor 1212, and magnetic or optical disks. And a data storage device 1207 such as a disk drive corresponding thereto. Data storage device 1207 is coupled to bus 1211 for storing information and instructions.

  Computer system 1200 may be further coupled to a display device 1221, such as a cathode ray tube (CRT) or liquid crystal display (LCD), coupled to bus 1211 for displaying information to a computer user. A alphanumeric input device 1222 comprising alphanumeric keys and other keys can also be coupled to the bus 1211 for communicating information and command selections to the processor 1212. Additional user input devices include a mouse, trackball, trackpad, stylus, coupled to bus 1211 to communicate direction information and command selections to processor 1212 and to control cursor movement on display device 1221. Or a cursor control 1223 such as a cursor direction key.

  Another device that can be coupled to the bus 1211 is a hardcopy device 1224, which may be used to store information on paper, film, or similar types of media. Another device that can be coupled to the bus 1211 is a wired / wireless communication function 1225 for communicating with a telephone or handheld palm device.

  Note that any or all components of system 1200 and associated hardware may be used in the present invention. However, it can be appreciated that other configurations of the computer system may include some or all of these devices. It should also be noted that some or all of these components can be included in a mobile device (eg, a handset) that performs the functions described herein.

  Many modifications and variations of the present invention will become apparent to those skilled in the art after reading the foregoing description, but any particular embodiment illustrated and described by way of illustration is not intended to be limiting. It should be understood that it is never intended to be considered. Accordingly, references to details of various embodiments are not intended to limit the scope of the claims, which enumerate only features that are considered essential to the invention.

  Embodiments of the present invention relate to the field of recording and playback of tasks performed on a device by a user. More particularly, embodiments of the present invention relate to recording and playback of web-based tasks for mobile handset users.

Claims (20)

Recording a web-based task performed during the session performed by a user without explicitly signaling the start of the recording session, the automatic start of the session without user involvement Including steps to find,
Receiving a request to play the recorded web-based task;
Playing back the recorded web-based task;
Including methods. Recording the web-based task comprises:
Monitoring a user's interaction state and interaction context, wherein the interaction context is a most recent interaction state sequence;
Intercepting conversation events from within the browser, and
The method of claim 1, wherein each interaction event corresponds to a transition from one interaction state to another interaction state. At least one of the interaction states is currently displayed by the browser, the currently open page in the browser, the highlighted text on the page currently displayed by the browser, the contents of the clipboard, and the browser. Including an interactive state selected from the group consisting of form input made to the page and the current value of the resource identifier in the input bar,
Furthermore, at least one of the dialog states includes a page load event, a resource identifier input event, a text highlight event, a text copy event, a form input event, a paste input event, and a resource identifier copy event. The method of claim 2, comprising an interaction event selected from: Recording the web-based task comprises:
A substep to segment the event stream;
A sub-step of removing events from the event stream that have no effect on the task to produce a de-noised event stream;
Determining a dependency between events in the segmented denoised event stream;
Identifying the event from the denoised event stream as part of a replayable session, traversing the dependency chain back from the last segment of the event;
The method of claim 1 comprising:   The method of claim 4, wherein the sub-step of segmenting the event stream is performed based on a time latency between events in the event stream.   The method of claim 4, wherein the sub-step of segmenting the event stream is performed based on an explicit user input action.   The method of claim 4, wherein the substep of removing an event from the event stream is performed without user involvement.   The method of claim 4, wherein the sub-step of removing events from the stream comprises removing events that result from user navigation that is not important to the task.   Substeps traversing the dependency chain include identifying one or more pairs of actions selected from the group consisting of highlight and copy operations, copy and paste operations, and input and submit operations. The method according to claim 4.   The method of claim 1, further comprising generalizing a session representing the task to make the session applicable to a context different from the context used when the session was recorded.   The method of claim 10, wherein generalizing the session includes substeps of abstracting the session.   Replaying the recorded web-based task includes a recorded session context representing the recorded web-based task from a first interaction context associated with the user when the session was recorded. The method of claim 1, comprising substeps of playing a recorded session using the user's current interaction context, adapted to the user's current interaction context.   The sub-step of adapting the context includes determining which state in the recorded session matches the user's context at the time the task playback request was made. Method.   Replaying the recorded web-based task represents the recorded web-based task by adapting to changes in at least one web page that is part of the recorded web-based task. The method of claim 1, comprising the substep of adapting a recorded session.   The replaying of the recorded web-based task comprises the sub-step of replaying a recorded session representing the recorded web-based task remotely using a proxy server. Method. The sub-step of playing the recorded session representing the recorded web-based task remotely using a proxy server comprises:
Sending the recorded session to the proxy server along with the user's latest interaction state;
Sending an event to the proxy server using a state from the user interaction state, including sending one or more requests to one or more servers and receiving a response from the one or more servers Playing by,
Sending by the proxy server the last page corresponding to the last page load event during the session, along with the current interaction state on the proxy server; and
16. The method of claim 15, comprising loading a resource specified by the last page and completing playback of one or more events that terminate the session. An article of manufacture having one or more non-transitory computer readable storage media having instructions stored thereon, wherein the instructions are executed by the system.
Recording a web-based task performed during the session performed by a user without explicitly signaling the start of the recording session, the automatic start of the session without user involvement Including steps to find,
Receiving a request to play the recorded web-based task;
Playing back the recorded web-based task;
An article of manufacture that causes the system to perform a method comprising: Recording the web-based task comprises:
Monitoring a user's interaction state and interaction context, wherein the interaction context is a most recent interaction state sequence;
A substep to intercept conversation events from within the browser;
18. The article of manufacture of claim 17, wherein each interaction event corresponds to a transition from one interaction state to another interaction state. Recording the web-based task comprises:
A substep to segment the event stream;
A sub-step of removing events from the event stream that have no effect on the task to produce a de-noised event stream;
Determining a dependency between events in the segmented denoised event stream;
Identifying the event from the denoised event stream as part of a replayable session, traversing the dependency chain back from the last segment of the event;
The article of manufacture of claim 17 comprising:   Replaying the recorded web-based task includes a recorded session context representing the recorded web-based task from a first interaction context associated with the user when the session was recorded. 18. The article of manufacture of claim 17, comprising the substep of adapting to a user's current interaction context to play a recorded session using the user's current interaction context.

Images