EP3227785A1 - Playback and automatic execution of a process to control a computer system - Google Patents
Playback and automatic execution of a process to control a computer systemInfo
- Publication number
- EP3227785A1 EP3227785A1 EP15817646.1A EP15817646A EP3227785A1 EP 3227785 A1 EP3227785 A1 EP 3227785A1 EP 15817646 A EP15817646 A EP 15817646A EP 3227785 A1 EP3227785 A1 EP 3227785A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- playback
- user
- component
- task
- recording
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04847—Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/323—Visualisation of programs or trace data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3414—Workload generation, e.g. scripts, playback
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3438—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04842—Selection of displayed objects or displayed text elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
Definitions
- Computer systems can be used to control a wide variety of things.
- Computer systems can be used to control automotive vehicles, aircraft, industrial processing systems, and even the operation of organizations.
- Such computer systems can be deployed, for instance, in vehicles or aircraft as control systems. They can be deployed at industrial processing facilities as industrial control applications. They can also be deployed in organizations as enterprise resource planning systems, customer relations management systems, etc.
- Such computer systems are often controlled by various processes. Some processes can include human involvement.
- a vehicle control computing system can sense human interactions with the system and control a vehicle based on those interactions.
- a vehicle may sense that a user has entered the vehicle and actuated the ignition switch. These sensed interactions can be used to control the vehicle.
- FIG. 1 is a block diagram of one example of a recording system architecture.
- FIG. 2 is a more detailed block diagram of one example of a task recording system.
- Figs. 2A-2C show a flow diagram illustrating one example of the operation of the task recording system shown in Fig. 2.
- Figs 2D-2F show examples of user interface displays.
- FIG. 3 is a more detailed block diagram of one example of a playback system.
- Figs. 3A-3C show a flow diagram illustrating one example of the operation of the playback system shown in Fig. 3.
- Fig. 3D shows an example of a user interface display.
- FIG. 4 is a more detailed block diagram of one example of a test generation system.
- Fig. 4A is a flow diagram showing one example of the operation of the test generation system shown in Fig. 4.
- FIG. 5 is a more detailed block diagram of one example of a test system.
- Fig. 5A is a flow diagram illustrating one example of the operation of the test system showing in Fig. 5.
- FIG. 6 is a block diagram of one example of the architecture shown in Fig. 1 , deployed in a cloud computing architecture.
- Figs. 7-9 show various examples of mobile devices.
- Fig. 10 is a block diagram of one example of a computing environment that can be deployed in the architecture illustrated in Fig. 1.
- FIG. 1 is a block diagram of one example of a recording system architecture 100.
- Architecture 100 illustratively includes computing system 102 that generates user interface displays 104 with user input mechanisms 106 for interaction by user 108.
- User 108 illustratively interacts with user input mechanisms 106 in order to control and manipulate computing system 102.
- Architecture 100 also illustratively includes task recording system 110, task recording store 112 that stores task recordings 114, playback system 116, test generation system 118, test store 120 that stores tests 122, testing system 124, test result store 126 and development environment 128.
- Architecture 100 can include other systems 130 as well.
- computing system 102 illustratively includes processors or servers 132, user interface component 134, application component 136, test application programming interface (API) generator 138 that generates test API(s) 140, and it can include other items 158 as well.
- Data store 142 illustratively includes one or more applications 144, processes 146, work flows 148, tasks 150, entities 152, forms 154, and it can include other items 156.
- Computing system 102 can be a wide variety of different types of computing systems. In one example, it senses user interactions with computing system 102, by user 108, in performing a task or process relative to computing system 102, and controls its operation based upon the sensed inputs. As an example, where computing system 102 is a computing system that controls a piece of machinery, it senses user interactions by user 108 and controls the machinery based upon those interactions. Where it is an industrial control application, it controls the operation of an industrial processing facility based upon sensed user interactions, as well as various other sensed parameters. Where it is a customer relations management or enterprise resource planning system, it controls operations of an organization or a computing system deployed by an organization based upon the sensed user interactions by user 108.
- Application component 136 illustratively runs one or more applications 144 that, themselves, can implement processes, workflows, tasks, or other items.
- the applications 144 can also operate on entities 152 or other data records 156. They can surface information for user 108, or other systems 130, or for any of the other systems shown in architecture 100, using forms 154 or other user interface displays or surfacing mechanisms.
- Test API generator 136 illustratively generates test APIs 140 that can be used to interact with forms 154 or other user interface display mechanisms in the various applications 144 of computing system 102.
- the test APIs 140 are strongly typed APIs so that static type checking can be performed on them during compile time.
- user 108 can invoke task recording system 110 in order to record a process or tasks performed within computing system 102.
- System 110 illustratively records the various steps in the process or task, along with the consequences of those steps within computing system 102, and stores them as a task recording 114.
- task recording system 110 not only records that the user has interacted with that user input mechanism, but it also records what happens as a result of that user interaction. This type of causal information is stored as part of the task recording 114 for the task being recorded.
- the task recording 114 can then be accessed by playback system 116.
- Playback system 116 can play back the recorded task against API(s) 140 to actually control computing system 102 to perform the task automatically. It can also play back the recording in a tutorial mode which can be viewed by user 108 so that user 108 can more quickly learn the tasks that were performed as part of the recorded process. It can further play back the task recording 114 in a user driven mode where the playback guides the user to take the steps in the process or task, but the user actually takes those steps instead of playback system 116 automatically performing the steps.
- Test generation system 118 illustratively accesses task recordings 114 and automatically generates tests 122 that can be stored in test store 120.
- Testing system 124 can access the tests 122 and run them against computing system 102 to determine whether computing system 102 is performing properly.
- Testing system 124 illustratively stores the test results in store 126 for later use by other systems 130, user 108, a developer that uses development environment 128, or otherwise.
- testing system 124 illustratively interacts with the user input mechanisms 106 through test APIs 140 that are generated for forms 154 or other user interface displays.
- Development environment 128 can also deploy test generation system 118. It can be used by a developer to further refine the tests that were automatically generated, to add comments to the tests, to modify the task recordings 114, or to perform other development activities.
- Fig. 2 is a more detailed block diagram of one example of task recording system 110.
- task recording system 110 illustratively includes one or more processors or servers 160, task recording store 112 (although this can be separate from task recording system 110 as well), user interaction detector component 162, record preview component 164, scope generator system 166, recording edit system 168, user interface component 170, rollback component 171 and it can include other items 172.
- task recording store 112 includes a set of task recording structures 174-176.
- the set of task recording structures 174-176 in task recording store 112 are each illustratively formed of a set of groups 178-180.
- Each group includes a plurality of different steps.
- group 178 illustratively includes steps 182-184.
- Group 180 illustratively includes steps 186-188.
- Each step illustratively represents a sensed user interaction with computing system 102.
- Each step in task recording structure 174 thus stores a variety of different types of information that identify the particular sensed user interaction that it represents. It can illustratively include, for instance, a step identifier 190, step content information 192, step context information 194, scope (causation) information 196, and it can include other information 198.
- Step identifier 190 illustratively identifies the particular step that was detected.
- the step identifier can indicate that the user interacted with a particular input mechanism.
- Step content 192 illustratively identifies a variety of different types of content corresponding to the step. For instance, if the user actuated a user input mechanism to open a form, step content 192 may illustratively identify the particular form that was opened. Content 192 can also identify information that was input by the user during the interaction, among other things.
- Step context 194 illustratively identifies the context of the application (or other context information) surrounding the detected step.
- Scope information 196 illustratively identifies various actions or other things that happened within the application, within the scope of the current step. This is described in greater detail below with respect to scope generator system 166.
- the step information 182 is illustratively generated for each step, and the steps 182-184 are arranged in the sequence in which they were taken (or detected).
- Task recording structure 174 can also be a hierarchical structure. For instance, within a given group 178, a particular step 182 may open a given form for the user. Then, within that form, the user may actuate a certain user input mechanism. In that case, the action that was detected to open the form will be entered into group 178 as a parent node in a hierarchical structure. The step that was detected to actuate the user input mechanism within the given form can be stored as a child node, that depends from the parent node. In this way, certain contextual information can be identified, simply based upon the hierarchical structure. If a node is a child of a parent node, then the detected action corresponding to the child node can be identified as being within the context of the parent node. Context information can be identified in other ways as well.
- Scope generator 166 illustratively includes scope open component 200, scope action detector 202, action recording component 204, scope close component 206, scope trimming component 208, and it can include other items 210.
- Scope generation system 166 illustratively opens a scope corresponding to each detected or sensed user interaction. It records everything that happens within the application (or within computing system 102) or both, during that scope. When processing of the sensed user interaction is completed, then the scope is closed. The scope information recorded can illustratively be interpreted as causation information.
- Scope open component 200 illustratively opens a scope 196 for a given step 182, once that step is detected.
- Scope action detector 202 detects the various actions which the application or computing system take while the scope is open.
- Action recording component 204 records all of the detected actions, and scope close component 206 closes the scope once processing for the detected action has been completed. It may be that some of the actions or things recorded by action recording component 204, during a given scope, are unwanted. In that case, scope trimming component 208 can trim unwanted items from the scope, after it is closed.
- Recording edit system 168 illustratively includes grouping component 212, task identifier component 214, note component 216, attachment component 218, examples/required component 220, and it can include other items 222.
- Edit system 168 can be used by a given user in order to edit or provide other authoring inputs either as a task recording is being generated, or after it has been generated. It may be, for instance, that the user wishes to group certain user interactions into groups corresponding to a given task or a set of steps.
- task identifier component 214 can be used to provide identifying information to identify that group of steps
- grouping component 212 illustratively generates user input mechanisms that can be actuated in order to group detected steps or user interactions into groups.
- Note component 216 illustratively displays user input mechanisms that allow the user to enter notes or additional explanatory material corresponding to a step, group or task.
- Attachment component 218 illustratively displays user input mechanisms that can be actuated to allow the user to attach documents, videos, or other attachments, to a recorded step, in line in the task recording. When the task recording is later played back, the notes and attachments are indicated so that the user viewing the playback can access them.
- Example/required component 220 illustratively allows the user to indicate whether a particular value entered during a step is required or is simply an example.
- a given detected user interaction is to actuate a user input mechanism that displays a dropdown menu, and a next user interaction is to select a value from the dropdown menu
- the author of the task recording wishes subsequent users to select that same item from the dropdown menu, every time.
- the author of the task recording simply wants the subsequent user to select any value from the dropdown menu, and the selected value may not necessarily need to be the same one that was selected during the task recording.
- component 220 illustratively generates user input mechanisms that can be actuated by the user to indicate that the detected action is only an example.
- the user input mechanisms can be actuated to indicate that the detected user interaction is mandatory or required.
- this is only one example of the component 220.
- Figs. 2A-2C show a flow diagram illustrating one example of the operation of task recording system 110, shown in Fig. 2. It is first assumed that user 108 (or another user) provides an input to launch an application 144 that is used to perform a process or otherwise control computing system 102. Sensing that user interaction input is indicated by block 230 in Fig. 2A. Application component 136 then launches the application, as indicated by block 232. [0042] At some point, user 108 will illustratively interact with the application through user input mechanisms 106. One such user input interaction will be an indication that the user wishes task recording system 110 to record a set of steps (e.g., a task or process). Sensing such a user interaction input is indicated by block 234.
- Sensing an input to record a task is indicated by block 236. Sensing an input to record a process is indicated by block 238. Sensing an input to record another item is indicated by block 240. For purposes of the present discussion, it will be assumed that computing system 102 has sensed that the user has provided an input to record a task or process.
- Computing system 102 then turns on recording system 110. This is indicated by block 242.
- task recording system 110 When task recording system 110 is turned on, in one example, it illustratively displays a recording pane that has user actuatable input mechanisms that can be actuated in order to control the recording, such as to stop, pause, start, etc. This is indicated by block 244.
- preview component 164 provides a visual indication to the user indicating which steps or user interactions will be recorded by system 1 10.
- the application 144 that is running, and for which the user wishes to record a task, may display a user interface display that has certain user input mechanisms that can be actuated in order to take actions within the application.
- preview component 164 provides a visual indication indicating whether user actuation of the corresponding input mechanism will be recorded as a step. For instance, it may be that component 164 highlights the user input mechanism (by holding it, shading it, etc.) to indicate whether it will be recorded as a step in the task or process being recorded. This is indicated by block 246. It will of course be noted that task recording system 110 or preview component 164 can perform other actions, once it is turned on, and this is indicated by block 248.
- the recording pane displayed by task recording system 110 illustratively includes user input mechanisms that allow the user to provide inputs identifying the task or process to be recorded.
- the user input mechanisms can include a text box that allow the user to type text to give the task recording or process recording a name. The user can illustratively provide other identifying information as well.
- Receiving user inputs identifying a task or process to be recorded is indicated by block 250 in Fig. 2A.
- task recording system 110 illustratively opens a task recording structure 174 (shown in Fig. 2) for the task recording about to be generated. Opening the task recording structure is indicated by block 252 in Fig. 2A. Task recording system 110 is then ready to record a task or process.
- Fig. 2D shows an example of a user interface display 271.
- Display 271 illustratively includes an application pane 273 and a task recorder pane 275.
- Application pane 273 displays information and controls for an open application, and for which a set of user interactions are to be recorded as a task.
- Task recorder pane 275 illustratively includes a set of user input mechanisms that can be actuated to control task recording system 110.
- the user input mechanisms can include a create mechanism 277 that can be actuated to create a new task recording. When it is actuated, it navigates the user through a set of user interface displays that allow the user to create a new recording.
- the user input mechanisms can also include a resume mechanism 279.
- the mechanisms can also include a playback mechanism 281 that can be actuated to invoke the playback system 116. This is described in greater detail below with respect to Figs. 3-3D.
- the mechanisms on pane 275 can also include rollback mechanism 283 and collect traces mechanism 286.
- mechanism 283 When mechanism 283 is turned on, then the rollback component 171 rolls back all changes made to the application during recording of the task, once the recording is completed. This can be done, for instance, by creating a database transaction that is aborted when the user stops the recording. It can be done in other ways as well. Rolling back the changes can be useful in a variety of ways. For instance, when the user is recording multiple recordings in succession, the subsequent recordings may be undesirably impacted by the changes made in the earlier recordings. This may result in confusion when the recordings are played back.
- mechanism 285 When mechanism 285 is turned on, the system illustratively collects event traces during the recording. These can be made available with the recording, or separately.
- Fig. 2E shows an example of user interface display 271 after the user has actuated mechanism 277 to create a new task recording. It can be seen that task recorder pane 275 now includes a name user input mechanism 287 that can be actuated by the user to enter a name for the task to be recorded. The user can actuate start mechanism 289 to begin the task recording, cancel mechanism 291 to cancel it and stop mechanism 293 to end the task recording. [0053] At some point, user 108 will interact with a user input mechanism 106 generated by the application for which the task or process is to be recorded. When that happens, user interaction detector component 162 (shown in Fig.
- User interaction detector component 162 can detect this in a variety of different ways. For instance, it can register to receive notifications 256 that indicate when the particular user input mechanisms have been actuated or otherwise interacted with. It can also identify other user entry points into the application 144, for which detector component 162 wishes to be notified. This is indicated by block 258. It can also simply detect user interaction with a control, as indicated by block 260, or it can detect any user interaction (or step) in other ways as well, and this is indicated by block 262.
- component 162 also records the detected user interaction (or step) in the corresponding task recording structure 174 that is open for this task, and in sequence with any other user interactions (or steps) that have already been detected. This is indicated by block 264 in Fig. 2A. By way of example, it can record the identity of the particular application component (in application 144) that was interacted with based on the user interaction. This is indicated by block 266. It can record the type of step or interaction that was taken (such as open a form, actuate a button, click a link, etc). Identifying the type of step is indicated by block 268. It can also take a screenshot of a browser and store that as well.
- scope open component 200 opens a scope, in memory, corresponding to the interaction (or step) that was just detected. This is indicated by block 278.
- Scope action detector 202 detects any actions taken by application 144, computing system 102, or any other processing components or other components in computing system 102, in response to the user interaction detected. Detecting these actions is indicated by block 280.
- Action recording component 204 then records all of those detected actions, in sequence, in the scope corresponding to this interaction (or step). This is indicated by block 282.
- Scope generator 166 continues to detect actions and record those actions, within the present scope, until the application processing for the detected user interaction (or step) has been completed. This is indicated by block 284. Once that processing has been completed, then scope close component 206 detects this and closes the scope. This is indicated by block 286.
- scope trimming component 208 can then trim any undesired information from the scope that has just been closed. For instance, it may be that a given user who is authoring a task recording only wishes certain information to be recorded in the scope, and regards other actions as obfuscating the desired information. Thus, the user can configure scope trimming component 208 to trim all information, other than that desired, from the scope. Of course, component 208 can be preconfigured to do this as well. In any case, if component 208 is to trim items from the scope, it removes them from the corresponding scope before the task recording is finalized. This is indicated by blocks 288 and 290 in Fig. 2B.
- causal information indicates actions which were caused in response to the detected user interaction (or step).
- user interface component 170 in task recording system 110 illustratively displays, on a display pane or other user interface display, an indication of the task recording structure 174 being formed for the current task recording.
- that indication includes a list of steps that have been detected, in sequence.
- Recording edit system 168 can also illustratively display user input mechanisms that allow the author of the task recording to edit the task recording in certain ways. For instance, recording edit system 168 can sense user interactions with those user input mechanisms to perform various edits on the task recording. Displaying the indication of the recording structure and sensing edit inputs are indicated by blocks 292 and 294 in
- recording edit system 168 illustratively processes the recording structure to reflect those recording edit inputs. This is indicated by block 296.
- task identifier component 214 can display a user input mechanism that allows the user to name or otherwise identify the task recording being generated.
- Grouping component 212 illustratively generates user input mechanisms that allow the user to group steps (or detected user interactions) into groups. This can be helpful because some processes have hundreds or thousands of steps. By allowing the author of the task recording to group those steps into groups, and then to name the groups, this can provide significant efficiencies both to the user viewing a playback, and to the playback system, itself.
- note component 216 illustratively generates user input mechanisms that allow the user to add notes to a step, or other parts of the task recording structure. This is indicated by block 298.
- Attachment component 218 illustratively generates user input mechanisms that allow the user to add attachments, links or other attachments, etc. This is indicated by block 300.
- Examples/required component 220 as discussed above, illustratively allows the user to mark an entered value or an action as required or simply as an example. This is indicated by block 302.
- Grouping component 212 allows the user to group steps into groups. This is indicated by block 304.
- Fig. 2F shows another example of user interface display 271 after the user has actuated user input mechanism 289 to start the task recording.
- Task recorder pane 275 now includes a set of control mechanisms 295 that allow the user to continue a paused recording, pause a running recording and conclude or stop a recording. When the pause mechanism is actuated, the user can take steps in the application without those steps being recorded.
- Pane 275 can also include a row selection user input mechanism. It can be turned on to allow a user to perform advanced row selection or filtering in a grid.
- Pane 275 can also include mechanism 299. When it is turned on, then screen shots are captured during the recording as well as the other information discussed herein.
- Pane 275 can also include mechanisms 301 and 303. They can be actuated to group steps in a task into groups. For instance, mechanism 301 can be actuated to start a new group of steps, and mechanism 303 can be actuated to end that group.
- Validation mechanism 305 can be actuated to add a validation. This is discussed below with respect to Fig. 3C. It can be done in the development environment or in another environment where a task recording is accessible for editing.
- Step display portion 307 illustratively displays the steps that have been recorded thus far for the current task recording, grouped based on user actuation of the grouping mechanisms 301 and 303. In the example shown in Fig. 2F, no steps have yet been recorded.
- task recording system 1 10 simply waits to receive an input indicating that the task recording is to be concluded. This is indicated by block 308.
- task recording is illustratively output for later use. This is indicated by block 310. For instance, it can be stored in task recording store 1 12. This is indicated by block 312. It can be output to a tutorial system or another playback system 1 16. This is indicated by block 314. It can be output to test generation system 1 18. This is indicated by block 316. It can be output to development environment 128, as indicated by block 318, or it can be output in other ways, as indicated by block 320.
- gestures can be recorded as well. For instance, when the user uses a copy/paste gesture to set a value for a control, both the copy input and the paste input are recorded as steps. This can be advantageous in some scenarios. For instance, it can be advantageous in test code generation (which is discussed in more detail below).
- the recording system need not create a parameterized input variable for the control's value, since it is set based on another control's value. In scenarios where a record is created, and an identifier for the record is frequently entered during the recording, the user can repeatedly paste the record identifier, and the task recording system will generate only a single parameterized input variable to represent the record identifier. Thus, changing the input data for a generated test can be made much easier.
- task recording system 1 10 illustratively provides an automatic recording of application-specific steps that are sensed as user interactions with the computing system. Because this is performed by task recording system 1 10, the recording is generated without substantially adding any computing or memory overhead to computing system 102. Also, because user interaction detector component 162 can be configured to detect a wide variety of user interactions, the system is extensible to detect interactions with application components, as new components are added. It also illustratively identifies causation information, automatically, by opening a scope corresponding to each sensed user interaction and recording all processing actions that occur within that scope. The task recording structure thus records how each recorded interaction relates to the steps of the underlying process or task being recorded.
- Fig. 3 shows a more detailed block diagram of one example of playback system 116.
- playback system 116 illustratively accesses task recordings 114 in task recording store 112 and plays them back. It can playback task recordings 114 in a variety of different contexts. For instance, system 116 can play them back in a tutorial context in which the recorded steps and corresponding data are simply played back for a user viewing the recording, to teach the user how to perform the steps. System 116 can also play the recording back in a runtime environment in which playback system 116 plays back the recording against a runtime instance of the application and automatically performs the steps in the runtime instance.
- system 116 can play back the recordings 114 in a development environment in which the recording 114 can be edited, extended, etc.
- playback system 116 illustratively includes a processor or server 322, user interface component 324, interpreter component 326, tutorial playback component 328, runtime playback component 330 (which, itself, includes user driven component 332 and automatic component 334), and development environment playback component 336.
- System 116 can include other items 338 as well.
- Figs. 3A and 3B show one example of a flow diagram illustrating the operation of playback system 116. It is first assumed that a user is operating playback system 116 (or it is being operated automatically by another processing system) to receive or import a task recording structure 174 representing a recorded process or task. This is indicated by block 340 in Fig. 3 A.
- system 116 can import a task recording 114 from recording store 112. Playback system 116 then receives an input indicating that it is to play back the task recording in a given mode.
- Interpreter component 326 then parses the task recording structure to identify the various steps recorded therein. This is indicated by block 342. It can also identify a wide variety of other things in the task recording structure, such as the step sequence 344, the step types corresponding to the steps, as indicated by block 346, and any user added annotations, as indicated by block 348. Further, interpreter component 326 can identify the step context 350, any data inputs or outputs and/or causation data. For instance, it can be directed to play back the task recording in a tutorial mode, in a runtime execution mode, or in a development environment mode. When directed to play back the task recording in a tutorial mode, tutorial playback component 328 generates the playback.
- it plays back the process or recorded steps visually, without actually executing those steps in a runtime instance of the application.
- This is indicated by block 358.
- This can visually guide the user by highlighting the various user input mechanisms that are actuated, in sequence.
- This is indicated by block 360.
- It can display any annotations and descriptions that have been entered by the author, as indicated by block 362.
- It can provide an indicator indicating that certain steps or interactions have corresponding attachments, such as documentation, video attachments, etc.
- This can also play back the recording in other ways, as indicated by block 366.
- user interface component 324 in playback system 116 illustratively also displays a playback control pane that includes user input mechanisms that sense user control inputs that are provided to control the playback. Sensing the user control inputs is indicated by block 366. For instance, the user can provide control inputs to indicate that the system should play back the recording one step at a time, either pausing or stopping between each step so that the user can view additional material. This is indicated by block 368. It can provide input mechanisms that allow the user to pause the recording as is indicated by block 370, to rewind the recording as indicated by block 372, to view contextual information, or other information, as indicated by block 374, or to control the playback in other ways, as well, as indicated by block 376.
- tutorial playback component 328 controls the playback based on the user control inputs. This is indicated by block 378.
- runtime playback component 330 accesses the task recording and plays it back using either user driven component 332 or automatic component 334.
- runtime playback component 330 illustratively identifies the APIs exposed by the application, and that are needed to take the steps in the task recording. This is indicated by block 380 in Fig. 3 A.
- the APIs can be for navigation between user interface displays (such as forms, etc). This is indicated by block 382. They can be APIs used for actuating or entering data into controls. This is indicated by block 384. They can be APIs for executing application logic in the corresponding application. This is indicated by block 386. They can be other APIs as well, as indicated by block 388.
- Runtime playback component 330 then executes actions in the application, based upon the steps in the recorded step sequence in the task recording. This is indicated by block 390 in Fig. 3B.
- automatic component 334 automatically executes the actions against the APIs, in order to actually perform the steps within a runtime instance of the application. This is indicated by block 392. In an example, it can do this while visually showing or displaying to the user the steps that are being taken. In one example, the user can also control the playback to pause between steps so that the user can review documentation, annotations, links to attachments, etc.
- automatic component 334 simply plays back the steps in the recording, against the runtime instance of the application, without input from the user.
- user driven component 332 illustratively prompts the user to take the actions against the runtime instance of the application. It can do this by highlighting controls to be actuated, by presenting explanatory notes indicating how and why they are to be actuated, data that is to be entered, etc. Playing back the task recording in a user driven way is indicated by block 394. Component 330 can execute actions in the application in other ways as well, and this is indicated by block 396.
- component 330 generates a visual display of the execution of each step, with control user input mechanisms.
- the visual indicator can show a playback indicator indicating that a task recording is currently being played back.
- block 400 can show the various actions that are being taken (e.g., the steps) as indicated by block 402, and it can be shown which steps have been taken thus far, and which steps are yet to be taken, in the sequence of steps. It can highlight controls before they are actuated. In this way, it can guide the user's eye to a given user input mechanism and highlight it, and then after a brief period, actuate that control, so that the user can more easily follow the playback.
- component 330 can display a control pane with a set of control user input mechanisms that allow the user to control the playback (such as to play back the steps in a step-by-step fashion, to pause between the steps, to rewind, to stop the playback, etc.). This is indicated by block 412. In that case, component 330 controls the playback based upon the sensed control inputs. This is indicated by block 414.
- development environment playback component 336 opens playback system 116 in the development environment 128. This is indicated by block 416 in Fig. 3C. It illustratively displays an exploration window (or other location user interface display) that has user input mechanisms that allows a user of development environment 128 to find and select a process or task recording for playback. This is indicated by block 418.
- Development environment playback component 336 displays a development interface with development input mechanisms for playing and modifying the process recording (or task recording).
- the development interface can illustratively include user input mechanisms that can be actuated to add, modify, delete, or otherwise edit steps in the task recording.
- the steps are illustratively displayed in an editable way, so that they can be edited accordingly.
- Component 336 also illustratively displays group user input mechanisms that allow the user to group the steps into groups, and to name those groups, or otherwise identify or name the steps or groups. This is indicated by blocks 424 and 426 in Fig. 3C.
- Component 336 can also illustratively display a validation user input mechanism. This allows the user to insert a validation into the task recording.
- a validation user input mechanism This allows the user to insert a validation into the task recording.
- the user of development environment 128 can insert a validation that stops the task recording playback and displays a message such as "Ensure that you have selected the first item in the dropdown menu”.
- the validation can also display a "Confirm” user input mechanism which can be actuated by the subsequent user viewing the playback, in order to validate that the selection has been made, and to continue the playback. Entering a validation is indicated by block 428.
- Component 336 also illustratively displays user input mechanisms that can be actuated by the user in order to attach attachments to a step, or links to attachments.
- the attachments or links can be displayed, in line, in the task recording, and displayed in conjunction with the particular step to which they are attached.
- the user of development environment 128 may attach explanatory documentation, a video link, or other items that provide additional information about a given step or user interaction that is to be performed.
- These links or attachments can be displayed, in line, as that step is about to be performed, so that the user can view the attachments, links, etc.
- the user input mechanisms can allow the user to enter comments or descriptions. This is indicated by block 432.
- Component 336 can display the development interface that allows the user to perform other development activities as well, and this is indicated by block 434.
- Component 336 then plays the task recording in the development environment 128 and modifies it, based upon the sensed development inputs described above. This is indicated by block 436. Once the task recording has been played back and modified in development environment 128, it is output for further processing. This is indicated by block 438. For instance, it can be stored in task recording store 112. This is indicated by block 440. It can be output to test generation system 118, as indicated by block 442. It can be output in other ways as well, and this is indicated by block 444.
- Fig. 3D shows another example of a user interface display that can be generated by playback system 116, during playback.
- Display 271 shown in Fig. 3D is similar to that shown in Fig. 2F and similar items are similarly numbered.
- Fig. 3D shows that step display portion 307 now has one or more recorded steps displayed therein, and playback controls 309 and 311 are also displayed.
- the user can play back the recording in a step-wise mode or a continuous mode.
- the playback system 116 executes the next step in the recording, and then pauses.
- the play input mechanism 311 the system begins sequential execution of the remaining steps in the recording until the playback is paused or all steps are executed.
- mechanism 311 becomes a pause button.
- the task recording can be played back to automatically control computing system 102, or a corresponding application 144, or both, based upon the sensed interactions recorded in the task recording. It can also be played back in a development environment where it can be modified to control the computing system 102 or application 144 (or both) in different ways. It can be played back in a tutorial environment where it is displayed for tutorial purposes.
- Fig. 4 is a block diagram illustrating one example of test generation system 118, in more detail.
- test generation system 118 illustratively includes task recording import component 446, recording parser component 448, test code generator 450, one or more processors or servers 452, test configuration component 454, user interface component 456, and it can include other items 458.
- Fig. 4A is a flow diagram illustrating one example of the operation of test generation system 118, in automatically generating a test for the application 144 or computing system 102 (or both) based upon a given task recording.
- Task recording import component 446 first receives or imports a task recording 115 representative of the recorded task or process. This is indicated by block 460.
- the task recording as described above, can include a set of step sequences 462, step types 464, step content 466, step context 468, inputs/outputs and/or causation data 470, or other items 472.
- User interface component 456 can generate a user interface display with user input mechanisms that allow a user to input a context for which the test is to be generated. This is indicated by block 474. For instance, it may identify a project 476, a computing system model 478, or other items 480 against which the test will be run.
- Recording parser component 448 then parses the input to identify the different steps and their corresponding data. This is indicated by block 482.
- Test code generator 450 then generates code methods based upon the step sequences. This is indicated by block 484. For instance, it can generate methods with code execution statements that call the appropriate application test APIs 140 in computing system 102. Adding code execution statements to call the test APIs is indicated by block 486 in Fig. 4A.
- Code generator 450 also illustratively processes input/output data related to each step as well as user annotations that are provided to each step. This is indicated by block 488. For instance, it can identify the actual data entered by the user and convert that data into variables as indicated by block 490. Thus, the specific data driving the test can be changed independently of the test execution flow.
- test generation system 118 uses test configuration component 454 to generate a user interface display that allows a user to configure or modify the test. It processes any test configuration or modification or edit inputs. This is indicated by block 496.
- component 454 can allow a user to group the steps in the test into groups. This is indicated by block 498. It can allow the user to name the steps or groups, as indicated by block 500, and it can perform other actions as well, as indicated by block 502.
- test generation system 118 For use by a testing system 124. This is indicated by block 504. For instance, it can be output to test store 120 for later access by testing system 124, so testing system 124 can run it against an application or another portion of a computing system 102.
- Fig. 5 is a more detailed block diagram showing one example of testing system 124.
- testing system 124 illustratively includes test locator component 506, test running component 508, one or more servers or processors 510, user interface component 512, test editor component 514, and it can include other items 516.
- Fig. 5 A is a flow diagram illustrating one example of the operation of test system 124 in finding and running a test against an application or other part of computing system 102. Figs. 5 and 5A will now be described in conjunction with one another.
- Test locator component 506 illustratively displays a user input mechanism that can be actuated by a user in order to locate a test. This is indicated by block 518 in Fig. 5 A.
- test locator component 506 can be an explorer window that allows a user to search for and navigate to, and then select, a given test.
- User interface component 512 then displays the test location user input mechanism 520 which, as described above, can be an explorer or other search display 522, or another display 524. It then receives or senses a user input selecting a test to be run. This is indicated by block 526.
- Test editor component 514 illustratively displays user input mechanisms that can be invoked in order to edit the test, before it is run. Component 514 thus receives any added code and adds it to (or otherwise modifies) the selected test. This is indicated by block 528.
- Test running component 508 then runs the selected test, as indicated by block 530. In doing so, it illustratively calls the test APIs 140 to execute the steps against the forms or other user interface displays (and corresponding controls) of the application under test in computing system 102. This is indicated by block 532. It can run the selected test in other ways, as indicated by block 534.
- Component 508 then outputs test results. This is indicated by block 536.
- the results can be output to another system 130 for further processing. This is indicated by block 538. They can also be stored in test result store 126, as indicated by block 540. The test results can be surfaced on a user interface display for display to the user. This is indicated by block 542. The test results can be output in other ways as well, and this is indicated by block 544.
- processors and servers include computer processors with associated memory and timing circuitry, not separately shown. They are functional parts of the systems or devices to which they belong and are activated by, and facilitate the functionality of the other components or items in those systems.
- the user actuatable input mechanisms can be text boxes, check boxes, icons, links, drop-down menus, search boxes, etc. They can also be actuated in a wide variety of different ways. For instance, they can be actuated using a point and click device (such as a track ball or mouse). They can be actuated using hardware buttons, switches, a joystick or keyboard, thumb switches or thumb pads, etc. They can also be actuated using a virtual keyboard or other virtual actuators. In addition, where the screen on which they are displayed is a touch sensitive screen, they can be actuated using touch gestures. Also, where the device that displays them has speech recognition components, they can be actuated using speech commands.
- the figures show a number of blocks with functionality ascribed to each block. It will be noted that fewer blocks can be used so the functionality is performed by fewer components. Also, more blocks can be used with the functionality distributed among more components.
- Fig. 6 is a block diagram of architecture 100, shown in Fig. 1, except that its elements are disposed in a cloud computing architecture 500.
- Cloud computing provides computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services.
- cloud computing delivers the services over a wide area network, such as the internet, using appropriate protocols.
- cloud computing providers deliver applications over a wide area network and they can be accessed through a web browser or any other computing component.
- Software or components of architecture 100 as well as the corresponding data can be stored on servers at a remote location.
- the computing resources in a cloud computing environment can be consolidated at a remote data center location or they can be dispersed.
- Cloud computing infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user.
- the components and functions described herein can be provided from a service provider at a remote location using a cloud computing architecture.
- they can be provided from a conventional server, or they can be installed on client devices directly, or in other ways.
- Cloud computing both public and private
- Cloud computing provides substantially seamless pooling of resources, as well as a reduced need to manage and configure underlying hardware infrastructure.
- a public cloud is managed by a vendor and typically supports multiple consumers using the same infrastructure. Also, a public cloud, as opposed to a private cloud, can free up the end users from managing the hardware.
- a private cloud may be managed by the organization itself and the infrastructure is typically not shared with other organizations. The organization still maintains the hardware to some extent, such as installations and repairs, etc.
- FIG. 6 specifically shows that the items in architecture 100 can be located in cloud 502 (which can be public, private, or a combination where portions are public while others are private). Therefore, user 108 uses a user device 504 to access those systems through cloud 502.
- cloud 502 which can be public, private, or a combination where portions are public while others are private. Therefore, user 108 uses a user device 504 to access those systems through cloud 502.
- Fig. 6 also depicts another example of a cloud architecture.
- Fig. 6 shows that it is also contemplated that some elements of architecture 100 can be disposed in cloud 502 while others are not.
- data stores 112, 120, 126 and 142 can be disposed outside of cloud 502, and accessed through cloud 502.
- task recording system 110, or other items can also be outside of cloud 502. Regardless of where they are located, they can be accessed directly by device 504, through a network (either a wide area network or a local area network), they can be hosted at a remote site by a service, or they can be provided as a service through a cloud or accessed by a connection service that resides in the cloud. All of these architectures are contemplated herein.
- architecture 100 can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, etc.
- FIG. 7 is a simplified block diagram of one illustrative embodiment of a handheld or mobile computing device that can be used as a user's or client's hand held device 16, in which the present system (or parts of it) can be deployed.
- Figs. 8-9 are examples of handheld or mobile devices.
- Fig. 7 provides a general block diagram of the components of a client device 16 that can run components of architecture 100 or that interacts with architecture 100, or both.
- a communications link 13 is provided that allows the handheld device to communicate with other computing devices and under some embodiments provides a channel for receiving information automatically, such as by scanning.
- Examples of communications link 13 include an infrared port, a serial/USB port, a cable network port such as an Ethernet port, and a wireless network port allowing communication though one or more communication protocols including General Packet Radio Service (GPRS), LTE, HSPA, HSPA+ and other 3G and 4G radio protocols, lXrtt, and Short Message Service, which are wireless services used to provide cellular access to a network, as well as Wi-Fi protocols, and Bluetooth protocol, which provide local wireless connections to networks.
- GPRS General Packet Radio Service
- LTE Long Term Evolution
- HSPA High Speed Packet Access
- HSPA+ High Speed Packet Access Plus
- 3G and 4G radio protocols 3G and 4G radio protocols
- lXrtt Long Term Evolution
- Short Message Service Short Message Service
- SD card interface 15 and communication links 13 communicate with a processor 17 (which can also embody processors 132, 160, 322, 452 or 510 from previous Figures) along a bus 19 that is also connected to memory 21 and input/output (I/O) components 23, as well as clock 25 and location system 27.
- processor 17 which can also embody processors 132, 160, 322, 452 or 510 from previous Figures
- bus 19 that is also connected to memory 21 and input/output (I/O) components 23, as well as clock 25 and location system 27.
- I/O components 23 are provided to facilitate input and output operations.
- I/O components 23 for various embodiments of the device 16 can include input components such as buttons, touch sensors, multi-touch sensors, optical or video sensors, voice sensors, touch screens, proximity sensors, microphones, tilt sensors, and gravity switches and output components such as a display device, a speaker, and or a printer port.
- Other I/O components 23 can be used as well.
- Clock 25 illustratively comprises a real time clock component that outputs a time and date. It can also, illustratively, provide timing functions for processor 17.
- Location system 27 illustratively includes a component that outputs a current geographical location of device 16. This can include, for instance, a global positioning system (GPS) receiver, a LORAN system, a dead reckoning system, a cellular triangulation system, or other positioning system. It can also include, for example, mapping software or navigation software that generates desired maps, navigation routes and other geographic functions.
- GPS global positioning system
- Memory 21 stores operating system 29, network settings 31, applications
- Memory 21 can include all types of tangible volatile and non- volatile computer-readable memory devices. It can also include computer storage media (described below). Memory 21 stores computer readable instructions that, when executed by processor 17, cause the processor to perform computer-implemented steps or functions according to the instructions.
- Device 16 can have a client system 24 which can run various business applications or embody parts or all of architecture 100. Processor 17 can be activated by other components to facilitate their functionality as well.
- Examples of the network settings 31 include things such as proxy information, Internet connection information, and mappings.
- Application configuration settings 35 include settings that tailor the application for a specific enterprise or user.
- Communication configuration settings 41 provide parameters for communicating with other computers and include items such as GPRS parameters, SMS parameters, connection user names and passwords.
- Applications 33 can be applications that have previously been stored on the device 16 or applications that are installed during use, although these can be part of operating system 29, or hosted external to device 16, as well.
- Fig. 8 shows one example in which device 16 is a tablet computer 600.
- Computer 600 is shown with user interface display screen 602.
- Screen 602 can be a touch screen (so touch gestures from a user's finger can be used to interact with the application) or a pen-enabled interface that receives inputs from a pen or stylus. It can also use an on-screen virtual keyboard. Of course, it might also be attached to a keyboard or other user input device through a suitable attachment mechanism, such as a wireless link or USB port, for instance.
- Computer 600 can also illustratively receive voice inputs as well.
- Device 16 can be a feature phone, smart phone or mobile phone.
- the phone can include a set of keypads for dialing phone numbers, a display capable of displaying images including application images, icons, web pages, photographs, and video, and control buttons for selecting items shown on the display.
- the phone can include an antenna for receiving cellular phone signals such as General Packet Radio Service (GPRS) and lXrtt, and Short Message Service (SMS) signals.
- GPRS General Packet Radio Service
- lXrtt Long Term Evolution
- SMS Short Message Service
- the phone also includes a Secure Digital (SD) card slot that accepts a SD card.
- SD Secure Digital
- the mobile device can also be a personal digital assistant (PDA) or a multimedia player or a tablet computing device, etc. (hereinafter referred to as a PDA).
- PDA personal digital assistant
- the PDA can include an inductive screen that senses the position of a stylus (or other pointers, such as a user's finger) when the stylus is positioned over the screen. This allows the user to select, highlight, and move items on the screen as well as draw and write.
- the PDA can also include a number of user input keys or buttons which allow the user to scroll through menu options or other display options which are displayed on the display, and allow the user to change applications or select user input functions, without contacting the display.
- the PDA can include an internal antenna and an infrared transmitter/receiver that allow for wireless communication with other computers as well as connection ports that allow for hardware connections to other computing devices.
- Such hardware connections are typically made through a cradle that connects to the other computer through a serial or USB port. As such, these connections are non-network connections.
- Fig. 9 shows that the phone can be a smart phone 71.
- Smart phone 71 has a touch sensitive display 73 that displays icons or tiles or other user input mechanisms 75. Mechanisms 75 can be used by a user to run applications, make calls, perform data transfer operations, etc.
- smart phone 71 is built on a mobile operating system and offers more advanced computing capability and connectivity than a feature phone.
- Fig. 10 is one example of a computing environment in which architecture 100, or parts of it, (for example) can be deployed.
- an example system for implementing some embodiments includes a general-purpose computing device in the form of a computer 810.
- Components of computer 810 may include, but are not limited to, a processing unit 820 (which can comprise processor 132, 160, 322, 452 or 510), a system memory 830, and a system bus 821 that couples various system components including the system memory to the processing unit 820.
- the system bus 821 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
- ISA Industry Standard Architecture
- MCA Micro Channel Architecture
- EISA Enhanced ISA
- VESA Video Electronics Standards Association
- PCI Peripheral Component Interconnect
- Computer 810 typically includes a variety of computer readable media.
- Computer readable media can be any available media that can be accessed by computer 810 and includes both volatile and nonvolatile media, removable and non-removable media.
- Computer readable media may comprise computer storage media and communication media.
- Computer storage media is different from, and does not include, a modulated data signal or carrier wave. It includes hardware storage media including both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
- Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 810.
- Communication media typically embodies computer readable instructions, data structures, program modules or other data in a transport mechanism and includes any information delivery media.
- modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
- communication media includes wired media such as a wired network or direct- wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
- the system memory 830 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 831 and random access memory (RAM) 832.
- ROM read only memory
- RAM random access memory
- BIOS basic input/output system 833
- RAM 832 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 820.
- Fig. 10 illustrates operating system 834, application programs 835, other program modules 836, and program data 837.
- the computer 810 may also include other removable/non-removable volatile/nonvolatile computer storage media.
- Fig. 10 illustrates a hard disk drive 841 that reads from or writes to non-removable, nonvolatile magnetic media, and an optical disk drive 855 that reads from or writes to a removable, nonvolatile optical disk 856 such as a CD ROM or other optical media.
- a removable, nonvolatile optical disk 856 such as a CD ROM or other optical media.
- Other removable/nonremovable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like.
- the hard disk drive 841 is typically connected to the system bus 821 through a non-removable memory interface such as interface 840, and optical disk drive 855 are typically connected to the system bus 821 by a removable memory interface, such as interface 850.
- the functionality described herein can be performed, at least in part, by one or more hardware logic components.
- illustrative types of hardware logic components include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
- the drives and their associated computer storage media discussed above and illustrated in Fig. 10, provide storage of computer readable instructions, data structures, program modules and other data for the computer 810.
- hard disk drive 841 is illustrated as storing operating system 844, application programs 845, other program modules 846, and program data 847. Note that these components can either be the same as or different from operating system 834, application programs 835, other program modules 836, and program data 837.
- Operating system 844, application programs 845, other program modules 846, and program data 847 are given different numbers here to illustrate that, at a minimum, they are different copies.
- a user may enter commands and information into the computer 810 through input devices such as a keyboard 862, a microphone 863, and a pointing device 861, such as a mouse, trackball or touch pad.
- Other input devices may include a joystick, game pad, satellite dish, scanner, or the like.
- These and other input devices are often connected to the processing unit 820 through a user input interface 860 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB).
- a visual display 891 or other type of display device is also connected to the system bus 821 via an interface, such as a video interface 890.
- computers may also include other peripheral output devices such as speakers 897 and printer 896, which may be connected through an output peripheral interface 895.
- the computer 810 is operated in a networked environment using logical connections to one or more remote computers, such as a remote computer 880.
- the remote computer 880 may be a personal computer, a hand-held device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 810.
- the logical connections depicted in Fig. 10 include a local area network (LAN) 871 and a wide area network (WAN) 873, but may also include other networks.
- LAN local area network
- WAN wide area network
- Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
- the computer 810 When used in a LAN networking environment, the computer 810 is connected to the LAN 871 through a network interface or adapter 870. When used in a WAN networking environment, the computer 810 typically includes a modem 872 or other means for establishing communications over the WAN 873, such as the Internet.
- the modem 872 which may be internal or external, may be connected to the system bus 821 via the user input interface 860, or other appropriate mechanism.
- program modules depicted relative to the computer 810, or portions thereof may be stored in the remote memory storage device.
- Fig. 10 illustrates remote application programs 885 as residing on remote computer 880. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
- Example 1 is a computing system, comprising:
- an interpreter component that receives a task recording representing a recorded task performed by a user in an application to identify a set of steps, in the task recording, performed to perform the task;
- a playback system that detects a playback input indicative of a detected playback mode, of a plurality of different playback modes, in which to play back the task recording, the plurality of different playback modes including at least a runtime execution mode;
- a playback component that plays back the task recording in the detected playback mode and controls the user interface component to display a visual playback of the task recording in the detected playback mode.
- Example 2 is the computing system of any or all previous examples wherein the playback component comprises:
- a runtime playback component that, in response to the detected playback mode being the runtime execution mode, executes the identified set of steps against the application.
- Example 3 is the computing system of any or all previous examples wherein the runtime playback component identifies application programming interfaces (APIs) that are called to perform the identified set of steps and calls the identified APIs.
- APIs application programming interfaces
- Example 4 is the computing system of any or all previous examples wherein the runtime playback component comprises:
- a user driven component that controls the user interface component to display a playback control user interface display with control user input mechanisms and detects user actuation of the control user input mechanism to control the execution of the identified set of steps in a user driven playback that executes the steps incrementally based on user control inputs.
- Example 5 is the computing system of any or all previous examples wherein the runtime playback component controls the user interface component to display a playback user interface display visually highlighting a next control, generated by the application, to be actuated during the execution of the steps against the application.
- Example 6 is the computing system of any or all previous examples wherein the runtime playback component controls the user interface component to display a playback user interface display visually displaying descriptive text corresponding to the steps executed against the application.
- Example 7 is the computing system of any or all previous examples wherein the runtime playback component controls the user interface component to display a playback user interface display with a notes/links indicator visually indicating whether any notes or links have been inserted in the task recording, corresponding to the steps being executed against the application, during recording of the task recording.
- Example 8 is the computing system of any or all previous examples wherein the playback component comprises:
- a development environment playback component that, in response to the detected playback mode being a development environment mode, opens the playback system in a development environment.
- Example 9 is the computing system of any or all previous examples wherein the development environment playback component controls the user interface component to display a development interface display with development input mechanisms that are actuated to play and modify the task recording.
- Example 10 is the computing system of any or all previous examples wherein the development input mechanisms comprise:
- Example 11 is the computing system of any or all previous examples wherein the development input mechanisms comprise:
- a validation input mechanism that is actuated to insert a validation for a corresponding recorded step in the task recording.
- Example 12 is the computing system of any or all previous examples wherein the development input mechanisms comprise:
- an attachment/link input mechanism that is actuated to insert an attachment or link for a corresponding recorded step in the task recording.
- Example 12 is the computing system of any or all previous examples wherein the development input mechanisms comprise:
- a description input mechanism that is actuated to insert a textual description or comment for a corresponding recorded step in the task recording.
- Example 13 is the computing system of any or all previous examples wherein the playback component comprises:
- a tutorial playback component that, in response to the detected playback mode being a tutorial mode, plays back the task recording without executing the steps against the application.
- Example 14 is s computer implemented method, comprising:
- controlling a user interface component to display a visual playback of the task recording in the detected playback mode
- Example 15 is the computer implemented method of any or all previous examples wherein playing back the task recording comprises:
- Example 16 is the computer implemented method of any or all previous examples wherein displaying the visual playback of the task recording comprises:
- Example 17 is the computer implemented method of any or all previous examples wherein playing back the task recording comprises:
- Example 18 is the computer implemented method of any or all previous examples wherein playing back the task recording comprises:
- Example 19 is a computing system, comprising:
- a playback system that detects a playback input indicative of a detected playback mode, of a plurality of different playback modes, in which to play back the task recording, the plurality of different playback modes including at least a runtime execution mode and a development environment mode;
- a runtime playback component that, in response to the detected playback mode being the runtime execution mode, executes the identified set of steps against the application and controls the user interface component to display a visual playback of the task recording in the runtime execution mode;
- a development environment playback component that, in response to the detected playback mode being the development environment mode, opens the playback system in a development environment and controls the user interface component to display a development interface display with development input mechanisms that are actuated to play and modify the task recording.
- Example 20 is the computing system of any or all previous examples and further comprising:
- a tutorial playback component that, in response to the detected playback mode being a tutorial mode, plays back the task recording without executing the steps against the application.
Abstract
Description
Claims
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201462088469P | 2014-12-05 | 2014-12-05 | |
US14/695,624 US20160162167A1 (en) | 2014-12-05 | 2015-04-24 | Playback and automatic execution of a process to control a computer system |
PCT/US2015/063845 WO2016090195A1 (en) | 2014-12-05 | 2015-12-04 | Playback and automatic execution of a process to control a computer system |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3227785A1 true EP3227785A1 (en) | 2017-10-11 |
EP3227785B1 EP3227785B1 (en) | 2019-01-30 |
Family
ID=55027809
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15817646.1A Not-in-force EP3227785B1 (en) | 2014-12-05 | 2015-12-04 | Playback and automatic execution of a process to control a computer system |
Country Status (4)
Country | Link |
---|---|
US (1) | US20160162167A1 (en) |
EP (1) | EP3227785B1 (en) |
CN (1) | CN107003925A (en) |
WO (1) | WO2016090195A1 (en) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11461090B2 (en) | 2020-06-26 | 2022-10-04 | Whatfix Private Limited | Element detection |
US11372661B2 (en) | 2020-06-26 | 2022-06-28 | Whatfix Private Limited | System and method for automatic segmentation of digital guidance content |
CN112214836B (en) * | 2020-10-09 | 2022-04-19 | 中国航空工业集团公司成都飞机设计研究所 | Parameterized flight dynamic playback method |
CN112383757B (en) * | 2020-11-16 | 2022-07-05 | 四川九洲空管科技有限责任公司 | Unmanned aerial vehicle comprehensive monitoring data multi-mode playback method and system |
US11704232B2 (en) * | 2021-04-19 | 2023-07-18 | Whatfix Private Limited | System and method for automatic testing of digital guidance content |
US11669353B1 (en) | 2021-12-10 | 2023-06-06 | Whatfix Private Limited | System and method for personalizing digital guidance content |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5220658A (en) * | 1986-03-10 | 1993-06-15 | International Business Machines Corporation | System for testing a performance of user interactive-commands using an emulator-overlay for determining the progress of the user timing response |
JPS62214447A (en) * | 1986-03-10 | 1987-09-21 | インタ−ナショナル ビジネス マシ−ンズ コ−ポレ−ション | Performance testing system for interactive system program |
US5335342A (en) * | 1991-05-31 | 1994-08-02 | Tiburon Systems, Inc. | Automated software testing system |
US5432932A (en) * | 1992-10-23 | 1995-07-11 | International Business Machines Corporation | System and method for dynamically controlling remote processes from a performance monitor |
US5786814A (en) * | 1995-11-03 | 1998-07-28 | Xerox Corporation | Computer controlled display system activities using correlated graphical and timeline interfaces for controlling replay of temporal data representing collaborative activities |
EP0869433A3 (en) * | 1997-03-31 | 1999-10-06 | Siemens Corporate Research, Inc. | A test development system and method for software with a graphical user interface |
US6286030B1 (en) * | 1998-07-10 | 2001-09-04 | Sap Aktiengesellschaft | Systems and methods for recording and visually recreating sessions in a client-server environment |
US7565607B2 (en) * | 2003-01-07 | 2009-07-21 | Microsoft Corporation | Automatic image capture for generating content |
US20050060719A1 (en) * | 2003-09-12 | 2005-03-17 | Useractive, Inc. | Capturing and processing user events on a computer system for recording and playback |
US8386928B1 (en) * | 2004-03-18 | 2013-02-26 | Adobe Systems Incorporated | Method and system for automatically captioning actions in a recorded electronic demonstration |
US7627821B2 (en) * | 2004-06-15 | 2009-12-01 | Microsoft Corporation | Recording/playback tools for UI-based applications |
US7653896B2 (en) * | 2004-06-30 | 2010-01-26 | Microsoft Corporation | Smart UI recording and playback framework |
US20080104528A1 (en) * | 2006-10-30 | 2008-05-01 | Logan Bryan M | Capturing a GUI Display Environment |
US8281234B2 (en) * | 2007-03-20 | 2012-10-02 | Microsoft Corporation | Definable application assistant |
US20120131456A1 (en) * | 2010-11-22 | 2012-05-24 | Microsoft Corporation | Capture and Playback for GUI-Based Tasks |
US9507698B2 (en) * | 2012-06-06 | 2016-11-29 | Oracle International Corporation | Test code generation based on test documentation |
US20140344683A1 (en) * | 2013-04-17 | 2014-11-20 | Candas Urunga | Methods, system and computer program product for user guidance |
US9047147B2 (en) * | 2013-06-18 | 2015-06-02 | Yahoo! Inc. | Method and system for recording and replaying user sessions in browser based applications |
-
2015
- 2015-04-24 US US14/695,624 patent/US20160162167A1/en not_active Abandoned
- 2015-12-04 WO PCT/US2015/063845 patent/WO2016090195A1/en active Application Filing
- 2015-12-04 EP EP15817646.1A patent/EP3227785B1/en not_active Not-in-force
- 2015-12-04 CN CN201580065915.3A patent/CN107003925A/en not_active Withdrawn
Also Published As
Publication number | Publication date |
---|---|
WO2016090195A1 (en) | 2016-06-09 |
US20160162167A1 (en) | 2016-06-09 |
EP3227785B1 (en) | 2019-01-30 |
CN107003925A (en) | 2017-08-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3227785B1 (en) | Playback and automatic execution of a process to control a computer system | |
US10642604B2 (en) | Workflow generation and editing | |
US9665359B2 (en) | Automatically resolving conflicts after installation of selected updates in a computer system | |
US9626176B2 (en) | Update installer with technical impact analysis | |
CN105229678B (en) | Process modeling and interface | |
US9395890B2 (en) | Automatic discovery of system behavior | |
US9965464B2 (en) | Automatic process guidance | |
US9317257B2 (en) | Folded views in development environment | |
US9892015B2 (en) | Integration and automation of build and test in a customized computing system | |
US20160162398A1 (en) | Automated test generation and execution for testing a process to control a computer system | |
EP3044671A1 (en) | Automatic installation of selected updates in multiple environments | |
US20150106723A1 (en) | Tools for locating, curating, editing, and using content of an online library | |
US10152308B2 (en) | User interface display testing system | |
US20150012329A1 (en) | Process flow infrastructure and configuration interface | |
US20160162168A1 (en) | Interaction sensing and recording of a process to control a computer system | |
US9736032B2 (en) | Pattern-based validation, constraint and generation of hierarchical metadata | |
WO2015187516A1 (en) | Form preview in a development environment | |
WO2017039974A1 (en) | Automatic process guidance | |
US20160274871A1 (en) | Isolating components using method detouring | |
Li et al. | Tools for Rapidly Prototyping Mobile Interactions |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20170523 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20180611 |
|
INTG | Intention to grant announced |
Effective date: 20180622 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1093812 Country of ref document: AT Kind code of ref document: T Effective date: 20190215 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015024160 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: FP |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190430 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190530 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1093812 Country of ref document: AT Kind code of ref document: T Effective date: 20190130 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190530 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190501 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015024160 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
26N | No opposition filed |
Effective date: 20191031 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602015024160 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MM Effective date: 20200101 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20191231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20191204 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200101 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200701 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191204 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191204 Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191204 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20151204 Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190130 |