JP2008547128A - Dynamic user experience with semantic rich objects - Google Patents

Abstract

  Semantic rich object for dynamic user experience. One or more actions in a process are represented by a semantic rich object (SRO) for exposing the functionality of the process. Each SRO includes metadata associated with the action and the execution environment of the action. In response to an interactive gesture from a user selecting at least one of the SROs to perform a desired action in the process, metadata is extracted from the selected SRO and the desired action is executed in the process's execution environment. To determine if it is available.

Description

  Users often exchange information with computer application programs through a series of forms-based operations and actions to access the functions of the application programs. For example, selecting an icon object on a computer display using an input device such as a mouse may present a menu of options to the user. Then, the user selects one of the options and uses the function of the application program to exchange information with the selected icon object (for example, “copy” of the highlighted text or selected For example, "sending" a document to an email recipient). For this reason, the set of menus is the entry point of the user experience to the application program. Typically, currently available user interface designs have the disadvantage of limiting the user to only the displayed options, as applications typically have other functions that are not presented to the user. As a result, with current user interface designs, the user cannot see what the form designer / developer has not coded or presented. To display all or most of the functions of an application, user interface designers or developers currently need to hard code those functions.

  For example, business applications use forms, navigation, query interfaces, and other specialized user interface (UI) artifacts to create the required user experience. The UI artifact design determines the user's entry point into the application state and behavior. However, UI artifact design also limits such entry points. Continuing with this example, the workflow application typically exposes the task to the user in a form dedicated to a particular task instance (eg, a form for a task “Approve this order”). As a result, the user may not be able to fully know what he / she wants to know about a particular task (eg, “What happens if I skip this task? Are there other unprocessed tasks associated with this task? As a result, it becomes difficult to link and share information.

  Some existing systems enhance the standard or default option menu by providing a list of shortcut keys and hints that allow the user to further interact with the application program. However, such augmented options are usually presented as a separate list, which is only a static extension of the default option, while forcing the user to remember more shortcut keys that are not easy to use.

  Currently marketed workflow applications provide limited default options with their own dedicated task UI. Some other application specific systems and UIs provide access to other functions of the application through custom code or customization. In the workflow example, it is possible to custom code additional information for a dedicated task UI, but any possibility that the UI designer may want to consider in connection with the dedicated task UI It is first impossible to predict the correct combination or direction. Therefore, in addition to being overwhelming at times, these custom codes and customizations typically do not cover all available features and are inherently static, limiting the use of customizations And limited.

  Presenting sufficient functionality to the user is a small form factor that is difficult to accommodate the traditional "verbose" form (a form in which the header and frame contain a relatively large amount of spatial and textual information). This is particularly a problem with devices such as mobile devices. The detail form includes, for example, a button, a navigation bar, a parent / child table (the child table moves according to the selection in the parent table), and the like. For this reason, it is difficult to notify the user of both the nature of the information exchanged by the user and the behavior that can be used from the underlying business application in relation to the information. The importance of the ability to handle this information using voice commands is increasing.

  With voice commands, the user can handle a greater number of possibilities (i.e. sentences) than traditional visual forms. A voice interface system, such as a voice recognition system, provides a basic set of functions, but the user does not necessarily know what functions are available. For example, in a conventional automated customer service system, a user may receive a set of options via a pre-recorded set of human voices. The user is required to speak to the handset the type of customer service that they want to use. As an illustration, suppose a user wants to talk to a customer representative about an invoice. First, the user selects which service area to call from among “Billing”, “Add service”, “Technical support”, or “Other” service by the automated voice system. Prompted. In this example, the user can respond to the voice prompt by describing the desired service. After this initial selection, the user receives a further option or a cascaded set of options that are presented to the user before being instructed to speak directly with a human customer representative. In many cases, existing voice interface systems lead to human customer service, skipping the options presented by the user saying “agent”, “person”, or “speak to agent” Allow that. However, this “hidden” shortcut is typically not found by the user, even if the voice interface system has the capability to perform such actions.

  Embodiments of the present invention overcome the shortcomings of conventional systems by using metadata through Semantic Rich Objects (SRO) to expose user-available functions from underlying data sources and applications. In addition to being available to the user through one SRO, the metadata includes information that allows the user to determine that it is available through various combinations of SROs exposed by the same or different applications and data sources. Alternative embodiments also allow the user to determine how to use the SRO in combination. The publicly available action functions can be combined and used by the user in a convenient manner that further exposes the actions and functions of the application program. According to one aspect of the present invention, in addition to individual single actions (eg, copy, paste, etc.), actions can be presented to the user according to semantic rules (eg, “This document is called John Share it and put the document into John's schedule at 10:00 am on July 1, 2005 "), etc.) to extract the metadata in the SRO and combine it with the metadata of another SRO.

  Advantageously, through published SRO metadata, embodiments of the present invention provide the user with application program actions in plain text so that processes within the application program are transparent to the user. . Embodiments of the present invention also expose instances of SROs with metadata so that application programs can determine appropriate and available actions without relying on standard or static user interface (UI) configurations. .

  That is, aspects of the present invention completely solve the problem of custom logic in the UI that determines and limits the user's ability to fully utilize the rich behavior and insights that can be utilized from business applications and the like. Aspects of the present invention also reduce the amount of code required in traditional form-based UI designs by increasing the semantic richness of actions presented by the underlying application and data source.

  Alternatively, the present invention can include various other methods and devices.

  Other features are partly obvious and some are pointed out below.

  This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary does not identify key or essential features of the claimed subject matter, nor should it be used as an aid in determining the scope of the claimed subject matter.

  Corresponding reference characters indicate corresponding parts throughout the drawings.

  Referring initially to FIG. 1, a block diagram illustrates a system 100 that uses a semantic rich object (SRO) to provide a dynamic user experience in the process according to one embodiment of the invention. The system 100 can be, for example, a computing system such as the computer 130 of FIG. 7, a computing device, a computer server, a plurality of computer servers, or computer-executable instructions, applications, application software, application programs 112, computer-executable routines or code. Can be other computing devices capable of executing In another embodiment, the system 100 includes computing devices in a distributed system in which multiple computing devices are connected by a common communication network such as an intranet, the Internet, or the like.

  With continued reference to FIG. 1, system 100 includes a processor 102 associated with an interface 104. The processor 102 processes instructions or commands from the user 106 via the interface 104 and executes computer-executable instructions, routines, applications, application programs, and the like. The interface 104 receives interactive gestures from the user 106. The interface 104 is a display such as the display shown in FIG. 7, for example, and the user 106 selects one or more objects displayed on the display using an input device such as a mouse 182 or a keyboard 180. Can do. In another embodiment, the interface 104 is a voice device coupled to a voice recognition system and provides the user 106 with an interactive voice dialog so that the user 106 can respond to voice prompts with voice.

  The system 100 also includes a memory 108 that is volatile, non-volatile, a common communication network such as the system memory 134 or non-volatile memory interface 166 of FIG. 7, or the Internet, an intranet, or other type of communication network. Other computer-readable media for storing data that may be transmitted over the Internet.

  In one embodiment, the memory 108 stores a plurality of semantic rich objects (SROs) 110 that include metadata that presents actions and data states in the application program 112. For example, the application program 112 can include actions, functions, operations, etc. for performing a task or processing an event, and data relating to the content of the task (see the description below with reference to FIG. 2) ( For example, a certain task related to the processing of a specific order of a customer may also be held. By way of illustration, a workflow management application program handles workflow processes that involve task assignment, task delegation, and the like. The communication software may include actions such as facilitating communication between the two parties (eg, sending an email, sharing documents between parties, etc.) or providing a messaging service. As another example, multimedia software may include actions such as organizing media objects (eg, audio files, still image files, video files, etc.), exchanging media objects with another person, etc. .

  Currently, a set of static form-based user interfaces presents actions to the user in such application programs or processes. For example, many well-known UIs include a standard set of high-level menu options such as “File, Edit, Tool, Help” for the user to select. Each of these higher level menu options provides additional options. Continuing with this example, the higher level menu option “File” may include “Open, Save, Exit, Print” or other lower options. In general, the user needs to learn such various individual menu options in order to become familiar with or interact with the application program 112 or process. This learning experience is similar to learning a new language. That is, before exchanging information with the application program 112, vocabulary and grammatical rules or logical rules (for example, the “Open” option is a submenu of “File” instead of “Edit”) must be learned.

  Advantageously, embodiments of the present invention provide within application program 112 or process so that a set of available or appropriate actions can be presented to the user in response to the user's choice and in plain text or semantic form. Use metadata that represents the action. In alternative embodiments, a set of available or appropriate actions is selectively provided or presented to the user based on the context or execution environment of the application program 112. Referring again to FIG. 1, an SRO is an item or object that is provided to a user to operate, verify, or share directly within an execution environment such as system 100. Alternatively, the SRO may be provided to the user through a medium such as a form that can constrain or enhance the behavior available from that SRO. In one embodiment, the SRO may be specific to the context or execution environment that the user 106 is in direct control. For example, a computer network system administrator can be presented with a complete set of available functions to manage the computer network. On the other hand, workstation user A may be presented with only a small set of options for the role of workstation user A, workstation user.

  It should be understood that a form based UI design may be used in conjunction with or in addition to the SRO. For example, a UI developer can provide a form with options for the user that receives a wealth of information in the SRO so that the UI developer can code all or part of the options presented in that form There is no need to do. Alternatively, embodiments of the present invention using SRO reduce the amount of logical coding in forms that UI developers have traditionally provided to present functions to users and enable the functions.

  In an alternative embodiment, the system 100 stores data including performance history of execution of the process of the application program 112, exchange of information between the process execution and the user's past, and data related to the process execution model in the memory 108; As system 100 extracts metadata from SRO 110, it allows users 106 to be presented with habitual or frequently used sets of actions.

  Referring now to FIG. 2, a data structure 202 including metadata according to one embodiment of the present invention is shown in a block diagram. The first data field 204 includes metadata that represents an action associated with the process. For example, the first data field 204 includes metadata 206 related to the available action code, metadata 208 related to the action execution code, metadata 210 related to the gesture process identification number (PIN), and related to the extensibility method. Metadata 212 associated with the object model and metadata 214 associated with the object model.

  The data structure 202 also includes a second data field 216 that includes metadata representing the execution environment of the process, the first data field being associated with the second data field and used in the execution environment of the process. Define a set of possible actions. The second data field 216 can include, for example, metadata 218 associated with the application context and metadata 220 associated with the peripheral service definition. Peripheral service definition 220 includes metadata associated with data that, for example, matches an interactive gesture to the execution of a process or application program 112 action. For example, assume that the context of an application program 112 or some SRO is a web application service that executes computer-executable instructions in assembly programming language. In this example, the peripheral service definition 220 can provide metadata associated with a given interactive gesture and corresponding assembly language executed by the web application service.

  In another embodiment, the application context 218 also includes data related to the context or execution environment of the application program that the SRO attempts to present its action. For example, the data contained in the application context 218 may include constraints on user name, user type, user execution environment (computer identification, operating system version, etc.), or availability of one or more options for the user. Restrictions can be included.

  In one embodiment, the data structure 202 further includes a third data field 222 that includes metadata representing a set of semantic rules, where the semantic rules include the metadata of the first data field and the metadata of the second data field. Define one or more interactive gestures associated with the data. For example, the third data field 222 can include metadata 224 associated with the gesture map, metadata 226 associated with the speech act map, and metadata 228 associated with the set of context rules. Illustratively, the user 106 can interact with the SRO 110 of the system 100 via interactive gestures, such as selecting and dragging icons on the computer desktop using a mouse; This includes responding to voice prompts with voice, or other things (eg, pressing a key on a telephone keypad or keyboard). Such interactive gestures allow the system 100 to associate metadata extracted from the selected SRO with a set of actions that can be used by the user to interact with processes such as workflows.

  In one embodiment, the set of semantic rules is interpreted or evaluated based on metadata and data associated with the execution environment or context of the application program that is included in the application context 218. For example, when an SRO is presented from an application / process / data source, the SRO includes some initial data as a contest. A UI environment (a UI that exposes an SRO while included in the SRO, or a UI that provides an additional layer such as a form in addition to the SRO) can assert or retract data into the context. For example, the set of semantic rules evaluates the username / user type data before applying the set of semantic rules, asserts that data into the UI, interprets the user's interactive gestures, and selects the selected SRO. A set of actions available to the user can be extracted from. In another example, a set of semantic rules may evaluate an action and determine that the action can only be used once. Therefore, the UI does not provide that action to the user.

  The data structure 202 includes data fields having the types of metadata illustrated here, but other metadata types or process executions or processes within the application program 112 without departing from the scope of the present invention. Data having information related to the execution environment may be included. Also, although embodiments of the present invention use metadata as an exemplary data type, any element managed so that recompilation of data is not required prior to execution without departing from the scope of the present invention. And / or other data structures or data organizations that describe any element that is executed when the data is executed may be used.

  To further describe the SRO, FIG. 3 is a block diagram illustrating functions represented by SRO metadata, according to an embodiment of the present invention. In the illustrated embodiment, for example, processes such as aggregation application 302, web service application 304, taskflow application 306, context space application 308, and document application 310 include one or more actions. Also, one or more SROs 312, 314, 316, 318, and 320 are used so that the features and actions of these application programs are exposed. Advantageously, aspects of the present invention expose functionality between various applications through the use of SRO. For example, the task flow application 306 can send a task event to the SRO 318 via arrow 322, which can be represented by the user creating a task. When a user creates a task event in SRO 318, assigns the task to an individual and executes the task, SRO 318 generates a roll event for SRO 320 via arrow 324, and SRO 320 passes via arrow 326. An action event is generated for the SRO 312. Thus, according to aspects of the present invention, each SRO is multifunctional and can be used by a variety of different application programs to respond to various events and actions.

  With continued reference to FIG. 3, the arrows between the application and the SRO shown in FIG. 3 represent an exemplary semantic relationship between the SRO and the application. For example, an arrow 328 from the web service application 304 to the SRO 312 can represent that the web service application 304 exposes an action or event to the user (eg, the user 106) through the SRO 312. In another example, an arrow 324 from SRO 318 to SRO 320 can indicate that a task has been assigned to a regional representative at a store. Thus, by exposing one or more actions in the process through the type of SRO to expose the functionality of the application program process, the system 100 provides the user 106 with a dynamic user experience.

  Illustrating a relatively simple dynamic user experience, FIG. 4 is a block diagram illustrating extraction of metadata from an SRO according to one embodiment of the present invention. For example, as shown in FIG. 4, the map type SRO 402 includes metadata regarding a place. In this example, the map type SRO 402 represents a map of Seattle. With continued reference to FIG. 4, the weather type SRO 404 includes time related metadata. As shown, the weather type SRO 404 indicates that the weather in Seattle on June 3, 2005 at 10:00 am is cloudy and the temperature is 67 degrees Fahrenheit.

  When user 106 selects or obtains a photo of Space Needle, a popular tourist destination in Seattle, the Photo of Space Needle is represented by photo type SRO406, which includes metadata related to photo quality, time, and location. Is done. Thus, the metadata is used to represent the photo type SRO406, and by extracting the metadata from the photo type SRO406, the map type SRO402, and the weather type SRO404, the location of Space Needle from Seattle, GPS (Global Positioning) Presenting the user 106 with a box or window 408 containing a map of Space Needle from Seattle by satellite, current weather conditions around Space Needle, and / or a forecast of weather conditions around Space Needle. Can do. In an alternative embodiment, the use of SRO allows the user to take additional actions related to Space Needle, such as requesting travel details or purchasing a Space Needle map.

  Referring now to FIG. 5, a flowchart illustrates a method for providing a dynamic user experience in the process using SRO (Semantic Rich Objects) according to one embodiment of the present invention. Specifically, FIG. 5 illustrates an example of an interactive gesture by a user using a keyboard. Initially at 502, a note type SRO 506 represents one or more actions in the process, including metadata associated with the actions and the execution environment of those actions. At 504, the interface 104 receives an interactive gesture from the user 106 that selects at least one of the SROs to perform a desired action in the process. In the example illustrated here, the user 106 inputs the text “Jennifer“ Finding Novel ”” from the keyboard for the note type SRO506. The action requested by the user 106 may be to see “Finding Neverland” with Jennifer. At 508, the system 100 extracts metadata from the note type SRO 506 in response to the received interactive gesture to determine if the requested action is available in the execution environment of the process.

  In this determination, the processor 102 may first define a set of semantic rules at 510 to evaluate or interpret the interactive gesture “Jennifer“ Finding Neverland ””. As mentioned above, each SRO has some data (such as in the application context 218 in the second data field 216) that is used to evaluate a set of semantic rules. In one example, some of the data in the application context 218 can be presented by the application program, and some can be asserted / retracted by the UI environment (eg, in a form hosting the SRO). The defined semantic rule may determine that “Jennifer” refers to a role type SRO, while “Finding Nerland” may refer to a location type SRO. If it is determined that “Finding Nerland” does not refer to a place, the defined semantic rule can further determine that “Finding Nerland” is the name of an object such as a document, the title of a file, and the like. In this way, the system 100 searches for available files stored in the memory 108 or accesses data from an application program, and the semantic meaning of “Finding Novel” is a text or media object type SRO. It can be determined that there is. The system 100 provides a set of actions available at 516 to perform the required action as it continues to resolve the “Jennifer” and “Finding Neverland” using semantic rules at 514. In one embodiment, the SRO includes code such as available action code 206, action execution code 208, routines, or computer-executable instructions in the first data field 204. For example, these codes can be used to perform actions. In another example, this code can be complex code, and if the action is provided by a web service, the action execution code may need to know how to communicate with the web service .

  In this example, the system 100 indicates that Jennifer is an instance of a role type SRO in the user's personal contact list (address book, email contact list, or other communication contact list), and “Finding Nerland” Is determined to be the title of a movie. Therefore, at 518, the system 100 calls "342-403-3323 to contact Jennifer and rents the movie" Finding Neverland "from the video rental store" or "Movie A Theater" Finding Newland "movie. And according to the user's schedule and the screening times of Theater A, the user will have a set of available actions, such as “Saturday and Sunday are convenient from 4 to 10 pm”. That is, let the user 106 select Jennifer from the user's phone book and check the phone number, or browse the Internet to determine if the title “Finding Neverland” is provided, or check the location of the A theater Instead of having a convenient viewing time determined, the system 100 provides the user 106 with a dynamic, contextual and interesting user experience. Thus, the user 106 is not limited to standard actions provided from application programs, or even custom-coded actions. Instead, the SRO uses metadata to expose process functions and actions within the application program, and provides actions to the user 106 in simple and plain language.

  In one embodiment, the user's interactive gestures including speech activity (see Appendix A for more details) are mapped to SRO metadata and the system 100 responds to a combination of visual and audio interactive gestures. The metadata can then be extracted from the selected SRO to determine the required action.

  In another alternative embodiment, the user generates one or more events via interactive gestures such as mouse clicks, mouse movements on the display. The system 100 receives or intercepts an event from the user and maps the event to an action exposed through SRO metadata according to a set of semantic rules. When the metadata is extracted from the SRO by the system 100, the user is presented with a set of available actions in response to the interactive gesture. In some cases, the system 100 may perform or perform available actions.

  FIGS. 6A-6C further illustrate providing a set of actions available in response to an interactive gesture from a user using a pointing device (eg, a mouse), according to one embodiment of the invention. FIG. In this example, the interactive gestures through the pointing device can include:

• Click to launch something • Drag / drop one object onto another to do something • Right to know more about something or get more options Click “Frame enclosure” to combine multiple objects or perform common operations on all of them
Search for something through a non-hierarchical metaphor and scroll through the results, possibly moving to the results.

  FIG. 6A shows a display area 602 and an operational space 604 for providing the user 106 with a set of available actions. An icon or event object 606 represents an SRO. In this example, the event object 606 represents “order 123”. The display area 602 also includes search input fields 608 and 610 that can be queried when the user 106 presses the “Search” button 612.

  Referring now to FIG. 6B, the user 106 first selects the event object 606 using a pointing device and drags the event object 606 (the dotted event object 614 represents the drag movement by the user 106). An event object 606 is dropped as an event object 616 in the space 604. In one embodiment, such drag-and-drop interactive gestures can be associated with SRO metadata according to the following rules:

Rule 1: <actor, UI gesture, object, environment, default action set>
Under this rule, interactive gestures (eg, “drag / drop”) are tied to an actor SRO (eg, a business report) and an object SRO (eg, a colleague) within an execution environment (eg, the action space 604).

  Alternatively, the user 106 drag and drop interactive gesture may be associated with SRO metadata according to the following rules:

Rule 2: <Actor, Act, Speech Act>
In this example, the actions available to the actor SRO are associated with a speech act such as a request, notification, or command.

  In another alternative embodiment, the user 106 associates a SRO with one or more other SROs using interactive gestures such as drag and drop mouse actions, and the one SRO and one or more. A set of available actions can be created as semantic text containing actions or behaviors represented by other SROs. For example, the action or behavior disclosed in the metadata of the user 106 is, for example, “Assigning task A to user B, and sending the notification to the immediate supervisor of user B when the task is completed by user B”. SRO X can be selected, such as in response to user 106 interactive gestures, so that it can be created.

  In FIG. 6C, after receiving a drag and drop interactive gesture from user 106, a set of available actions 618 is displayed or provided to user 106. In this example where the workflow is relevant, the set of available actions 618 includes a “process history” (down arrow 620 indicates when the user 106 moves the pointing device over the down arrow 620 or uses the pointing device. Indicating that additional actions are available when the down arrow 620 is selected). When the system 100 extracts metadata from the “order 123” and determines whether the information contained in the “process history” is likely to be available to the user 106, within the displayed “process history” , A set of information related to “Order 123” is presented to the user 106.

  For example, the process history shows the purchase order confirmation information of “Order 123” of user 1 in Seattle on March 4, 2005. Note that user 106 did not initially select an action such as “display process history” or “display purchase order confirmation” in FIG. 6A. Instead, the user 106 simply drags and drops the event object 606 onto the action space 604 and the system 100 responds to the drag-and-drop interactive gesture for such action space 604 and places “Order 123”. Metadata is extracted from the representing event object 606 to provide the user 106 with a set of available actions. Thus, embodiments advantageously provide a multi-functional SRO enhanced user experience without requiring the user to memorize shortcuts or options that are in form-based menus.

  In another embodiment, system 100 provides a list of available actions after extracting metadata from source SRO and target SRO according to Rule 1. If a default action set is available, the system 100 changes the previous list according to performance history, past user exchanges, and so on. Alternatively, the system 100 may present the user 106 with a complete list of actions available from the actor SRO and the object SRO.

  In yet another embodiment according to Rule 2, the system 100 interprets available actions according to a set of SpeechActs. In this example, a SRO representing “user 1”, another SRO representing “user 2” (colleague), and a third object representing a business object related to “sales volume in the western region filtered by business unit”. , There are three SROs. After selecting the business object, the user 1 drags the business type SRO onto the SRO representing the user 2 and drops the business type SRO onto the SRO representing the user 2. Upon receiving such a visual interactive gesture, the system 100 extracts metadata from the metadata from the three SROs and maps it to a voice interactive gesture such as speech activity, and the user 106: Can be offered.

  A. The business object is shared with the user 2.

B. Communicate in relation to the business object, such as asking the user 2 about the validity of the business object. Or C.I. Make user 2 an administrator / owner of the business object (or give user 2 certain rights to access the business object).

  In one embodiment, user 106 may also combine drag and drop interactive gestures and text queries by entering search terms in search input fields 608 and 610. For example, if the user 106 frequently uses a particular utterance action for a particular instance of a particular actor / object pair, the system 100 may provide for all instances of that actor / object pair. A combination of these can be proposed as new speech actions that can be used. As described above, the metadata related to the utterance action can be updated with the new utterance action based on the past or the history of information exchange by the user. As a result, embodiments of the present invention eliminate the need to design customized code or customized action sequences for the user.

  In a further alternative embodiment, process execution and user interaction in the past, process execution performance history, and process execution model are represented as SROs with metadata relating to the actions of those processes in the application program. Can do. For this reason, there are cases where actions such as search can be used in accordance with SRO which is information exchange performed by the past user, and the set of actions available to the user 106 can be managed, inquired, and expanded / reduced. Become.

  FIG. 7 illustrates an example of a general purpose computing device in the form of a computer 130. In one embodiment of the invention, a computer, such as computer 130, is suitable for use in the other figures shown and described herein. The computer 130 has one or more processors or processing units 132 and a system memory 134. In the illustrated embodiment, system bus 136 couples various system components, including system memory 134, with processor 132. Bus 136 represents one or more of several bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor bus or a local bus using any of a variety of bus architectures. . By way of example and not limitation, such architectures include an ISA (Industry Standard Architecture) bus, an MCA (Micro Channel Architecture) bus, an EISA (Enhanced ISA) bus, a VESA (Video Electronics Standards bus, and a Video Electronics Standards bus). There is a PCI (Peripheral Component Interconnect) bus.

  Computer 130 typically has at least some form of computer readable media. Computer readable media can be any available media that can be accessed by computer 130 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media comprises computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in methods or techniques for storing information such as computer readable instructions, data structures, program modules, or other data. included. For example, computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or others Or any other medium that can be used to store desired information and that can be accessed by computer 130. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Those skilled in the art are familiar with the modulated data signal, which has one or more of its characteristics set or changed in such a manner as to encode information in the signal. Wired media such as wired networks and direct wire connections, and wireless media such as acoustic, RF, infrared, and other wireless media are examples of communication media. Combinations of the above media are also included within the scope of computer-readable media.

  The system memory 134 includes computer storage media in the form of removable and / or non-removable, volatile and / or nonvolatile memory. In the illustrated embodiment, system memory 134 includes read only memory (ROM) 138 and random access memory (RAM) 140. A basic input / output system 142 (BIOS) that includes basic routines that assist in transferring information between elements within the computer 130, such as during startup, is typically stored in the ROM 138. RAM 140 typically holds data and / or program modules that are immediately accessible from and / or currently being manipulated by processing unit 132. By way of example and not limitation, FIG. 7 illustrates an operating system 144, application programs 146, other program modules 148, and program data 150.

  The computer 130 may also include other removable / non-removable, volatile / nonvolatile computer storage media. For example, FIG. 7 shows a hard disk drive 154 that reads from and writes to a non-removable, non-volatile magnetic medium. FIG. 7 also shows a removable magnetic disk drive 156 that reads from and writes to a non-volatile magnetic disk 158 and an optical disk drive 160 that reads from and writes to a removable non-volatile optical disk 162 such as a CD-ROM or other optical media. Show. Other removable / non-removable, volatile / nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, and digital versatile discs. Digital video tape, solid-state element RAM, solid-state element ROM, and the like. The hard disk drive 154, magnetic disk drive 156, and optical disk drive 160 are typically connected to the system bus 136 by a non-volatile memory interface such as an interface 166.

  The drive or other mass storage device and associated computer storage medium described above and illustrated in FIG. 7 provides storage of computer readable instructions, data structures, program modules, and other data for computer 130. In FIG. 7, for example, the hard disk 154 stores an operating system 170, application programs 172, other program modules 174, and program data 176. Note that these components can either be the same as or different from operating system 144, application programs 146, other program modules 148, and program data 150. Here, the operating system 170, the application program 172, the other program module 174, and the program data 176 are given different reference numerals to indicate that they are at least different copies.

  A user may enter commands and information into the computer 130 through input devices such as a keyboard 180 and a pointing device 182 (such as a mouse, trackball, pen, touch pad) or a user interface selection device. Other input devices (not shown) include a microphone, joystick, game pad, satellite dish, scanner, and the like. These and other input devices are connected to the processing unit 132 through a user input interface 184 coupled to the system bus 136, but with other interfaces and bus structures such as parallel ports, game ports, USB (Universal Serial Bus), etc. You may connect. A monitor 188 or other type of display device is also connected to the system bus 136 via an interface, such as a video interface 190. In addition to the monitor 188, the computer often includes other peripheral output devices (not shown) such as printers and speakers, which can be connected through an output peripheral interface (not shown). .

  Computer 130 may operate in a network environment that uses logical connections with one or more remote computers, such as remote computer 194. Remote computer 194 is a personal computer, server, router, network PC, peer device, or other common network node, and typically includes many or all of the elements described above in connection with computer 130. The logical connections shown in FIG. 7 include a local area network (LAN) 196 and a wide area network (WAN) 198, but may include other networks. LAN 136 and / or WAN 138 is a wired network, a wireless network, a combination thereof, or the like. Such network environments are common in offices, enterprise-wide computer networks, intranets, and global computer networks (eg, the Internet).

  When used in a local area network environment, the computer 130 is connected to the LAN 196 through a network interface or adapter 186. When used in a wide area network environment, the computer 130 typically includes a modem 178 or other means for establishing communications over the WAN 198 such as the Internet. The modem 178 may be internal or external and is connected to the system bus 136 via a user input interface 184 or other suitable mechanism. In a network environment, the program modules illustrated in connection with computer 130, or portions thereof, may be stored in a remote memory storage device (not shown). By way of example and not limitation, in FIG. 7, the remote application program 192 resides in a memory device. The network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

  In general, the data processor of computer 130 is programmed with instructions stored at different times on the various computer-readable storage media of the computer. The program and the operating system are usually distributed, for example, on a floppy (registered trademark) disk or a CD-ROM. From there, it is installed or loaded into the auxiliary memory of the computer. At runtime, at least a portion is loaded into the main electronic memory of the computer. The invention described herein is directed to such and various other computers where such media retains instructions or programs for carrying out the steps described below in conjunction with a microprocessor or other data processor. Includes a readable storage medium. The invention also includes the computer itself when programmed according to the methods and techniques described herein.

  For purposes of explanation, the program and other executable program components such as an operating system are illustrated here as separate blocks. However, it will be appreciated that such programs and components exist at different times in the various storage components of the computer and are executed by the computer's data processor.

  Although described in connection with an exemplary computing system environment, including computer 130, the invention is operational with numerous other general purpose or special purpose computing system environments or configurations. This computing system environment is not intended to suggest any limitation as to the scope of use or functionality of the invention. In addition, this computing system environment should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the present invention include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems. And microprocessor based systems, set-top boxes, programmable consumer electronics, mobile phones, network PCs, minicomputers, mainframe computers, distributed computing environments including the above systems or devices, and the like.

  The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers or other devices. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

  An interface in the context of a software architecture includes a sequence of software modules, components, code portions, or other computer-executable instructions. The interface includes, for example, a first module that accesses a second module and performs computing tasks for the first module. The first and second modules include, by way of example, an application programming interface (API) provided by the operating system, a component object model (COM) interface (eg, for peer-to-peer application communication), and an extensible markup language meta A data interchange format (XMI) interface (eg, for communication between web services) is included.

  This interface can be a tightly coupled, synchronous implementation, as in the example of J2EE (Java® 2 Platform Enterprise Edition), COM, or Distributed COM (DCOM). Alternatively or in addition, the interface may be a loosely coupled, asynchronous implementation such as a web service (eg, using a simple object access protocol). In general, an interface includes any combination of the characteristics of tightly coupled, loosely coupled, synchronous, and asynchronous. Further, the interface can conform to a standard protocol, a proprietary protocol, or a combination of standard and proprietary protocols.

  The interfaces described herein can all be part of a single interface, or can be implemented as separate interfaces or combinations thereof. The interface can be executed locally or remotely to provide functionality. Further, an interface may include more or less functions than those illustrated or described herein.

  In operation, computer 130 executes computer-executable instructions as shown in FIG. For example, the computer 130 represents one or more actions in the process with a semantic rich object (SRO) to expose the functionality of the process. Each SRO includes metadata associated with the action and the execution environment for that action. The computer 130 also receives an interactive gesture from the user for selecting at least one of the SROs to perform a desired action in the process. Computer 130 further extracts metadata from the selected SRO to determine whether the desired action is available in the execution environment of the process.

  The order in which the methods illustrated and described herein are performed or performed is not critical unless otherwise specified. That is, the elements of the methods may be performed in any order unless otherwise specified, and the method may include more or fewer elements than disclosed herein. For example, it is contemplated that executing or performing a particular element before, contemporaneously with, or after another element is within the scope of the invention.

  In describing an element of the invention or an embodiment thereof, the articles “a”, “an”, “the”, “said” shall mean that there is one or more of the element. The terms “comprising”, “including”, and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.

  From the foregoing description, it will be appreciated that the several objects of the invention are achieved and other advantageous results are obtained.

  Since various modifications can be made to the system and method described above without departing from the scope of the invention, all content contained in the above description and illustrated in the accompanying drawings is in a limiting sense. Rather, it should be construed as exemplary.

  Appendix A lists examples of speech acts according to one embodiment of the present invention.

Appendix A
Examples of utterance actions An utterance action has several characteristics and features. Formally speaking, the act of speaking involves the act of communicating between the speaker, the listener, and both. For example, J. et al. L. Austin and John R. Seale has developed a fairly rigorous specialized vocabulary for explaining speech act theory (SAT). This entails distinguishing the three phases of speech act as follows (e.g., Seale, John R., Speech Acts, An Essay in the Philosophy of Language, Cambridge University, 19:19).

A. Speech act (what was literally said)
B. Acts within speech (eg actions taken, such as requests)
C. Spoken mediation (acted by performing the act; social impact)
Using semantic rules and gesture algebra operators, person A can click on the SRO representing person B and be given a list of more meaningful but general utterances, such as: .

A. Request person B to do something. Ask person B a question. Make a promise / pledge to person B against person B. Notify person B about something. Or E. Share resources / objects with person B

1 is an exemplary diagram illustrating a system for providing a dynamic user experience in a process using semantic rich objects (SRO) according to one embodiment of the present invention. FIG. FIG. 5 is an exemplary block diagram illustrating a data structure describing an SRO including metadata according to an embodiment of the present invention. FIG. 6 is an exemplary block diagram illustrating functions represented by SRO metadata according to an exemplary embodiment of the present invention. FIG. 6 is an exemplary block diagram illustrating a dynamic user experience by extracting metadata from an SRO according to an embodiment of the present invention. 6 is an exemplary flow diagram illustrating a method for providing a dynamic user experience in a process using a semantic rich object (SRO) according to an embodiment of the present invention. FIG. 6 is a block diagram illustrating providing a set of available actions in response to an interactive gesture from a user using a pointing device (eg, a mouse) according to one embodiment of the present invention. FIG. 6 is a block diagram illustrating providing a set of available actions in response to an interactive gesture from a user using a pointing device (eg, a mouse) according to one embodiment of the present invention. FIG. 6 is a block diagram illustrating providing a set of available actions in response to an interactive gesture from a user using a pointing device (eg, a mouse) according to one embodiment of the present invention. FIG. 7 is a block diagram illustrating an example of a suitable computing system environment in which the present invention may be implemented.

Claims (20)

A computerized method for providing a dynamic user experience within a process,
Representing one or more actions in the process by means of a semantic rich object (SRO) to expose the functionality of the process, each SRO comprising metadata associated with the action and the action Including data related to the execution environment for
Receiving from the user an interactive gesture for selecting at least one of the SROs to perform a desired action within the process;
In response to the received interactive gesture, extract the metadata from the selected SRO to determine whether the desired action is available in the execution environment of the process Comprising the step of:   Each of the SROs further includes code for performing the action, and extracting the metadata is responsive to the received gesture and the desired action is available in the execution environment of the process. The computerized method of claim 1, comprising executing the included code to determine whether it is present.   Providing a set of actions that can be utilized by the user to interact with the process based on the extracted metadata in response to the received interactive gesture; The computerized method of claim 1, comprising:   Providing the set of available actions includes displaying the set of available actions exposed through the SRO in a form-based user interface, and allowing the user to interact with the process. At least one of generating a user interface that selectively displays the set of available actions to perform, the process comprising one or more of an application program, application software, and a workflow processing application The computerized method of claim 3, comprising:   Performing the desired action based on the extracted metadata and the received interactive gesture further comprising defining a set of semantic rules for interpreting the received interactive gesture from the user The computerized method of claim 1, further comprising the step of:   Providing a plurality of SROs in response to a query from a user, and extracting the extracted metadata from the execution history of the process, exchanging information between the execution of the process and the user's past, and a process The computerized method of claim 1, further comprising associating with one or more of the execution models.   Associating the interactive gesture with the metadata of each of the SROs, further comprising an action available as a semantic sentence including the action represented by the selected SRO and the one or more other SROs The computer of claim 1, further comprising associating the extracted metadata of the selected SRO with metadata of one or more other SROs to create a set of data. Method.   The computerized method of claim 1, wherein the one or more computer readable media includes computer executable instructions for performing the computerized method of claim 1. A system that actively generates behavior for users to interact with processes,
A memory that stores one or more semantic rich objects (SROs), each of the SROs including metadata corresponding to a behavior and an execution environment of the behavior, each of the SROs executing the behavior Memory that also contains relevant data,
An interface for receiving a selection of one or more SROs from the user;
A processor executing computer-executable instructions for generating a set of behaviors available to the user by associating the received selection from the interface with stored metadata in the selected SRO. Prepared,
The system, wherein the interface provides the user with the set of generated behaviors for interacting with the process in the execution environment.   The system of claim 9, further comprising a user interface having a user selection device for receiving a selection of the one or more SROs from the user.   The system of claim 9, wherein the processor is further configured to associate the received selection from the interface with the stored metadata of each of the SROs.   The system of claim 11, further comprising a database associating a set of speech actions with the stored metadata of each of the SROs.   The interface receives a selection of one SRO from the user and associates it with one or more other SROs, and the processor selects the received selection, the one SRO and the one or more other Configured to generate a set of behaviors available to the user based on the stored metadata of the SRO and associate the one SRO with the one or more other SROs The system according to claim 9.   The processor is further configured to execute the stored data associated with execution of the behavior to generate the set of behaviors based on the stored metadata and the received selection from the interface. The system of claim 9, wherein the system is configured. The interface provides one or more SROs in response to a query from the user, selectively provides the set of generated behaviors of the SRO based on the execution environment of the process, and forms Configured to receive a selection of one or more SROs from the user via a user interface of the scheme;
The processor is further configured to correlate the stored metadata with one or more of an execution history of the process, an execution history of the process and an exchange of information in the user's past, and a process execution model. The system of claim 9, wherein:   The processor is further configured to define a set of semantic rules for interpreting the received selection from the user via the interface, the processor based on the defined set of semantic rules. 10. The system of claim 9, wherein the set of available behaviors is generated by determining whether the desired behavior set is available in an execution environment of the process. . A computer-readable medium storing a data structure of a semantic rich object,
A first data field that includes metadata representing one or more actions associated with the process;
A second data field containing metadata representing an execution environment of the process, the first data field being associated with the second data field and available in the execution environment of the process A computer-readable medium comprising: a second data field defining a set of actions.   The method further comprises a third data field including metadata representing a set of semantic rules, the semantic rule being associated with the metadata of the first data field and the metadata of the second data field. The computer-readable medium of claim 17, wherein one or more interactive gestures are defined.   The metadata of the third data field includes metadata representing a set of interactive gestures and a set of speech actions, wherein the set of interactive gestures and the set of speech actions are the said data set of the first data field. The computer-readable medium of claim 18, wherein the computer-readable medium is associated with metadata and the metadata of the second data field.   A set of process peripheral service definitions, an interactive gesture PIN, a process extensibility method, a process object model, an execution history of the process execution, an exchange of information between the execution of the process and the user's past, execution of the behavior, and The computer-readable medium of claim 17, further comprising one or more data fields that include metadata associated with one or more of the process execution models.

Images