EP3111392A1 - Présentation unifiée d'informations contextuellement connectées afin d'améliorer l'efficacité et la performance d'interaction de l'utilisateur - Google Patents

Présentation unifiée d'informations contextuellement connectées afin d'améliorer l'efficacité et la performance d'interaction de l'utilisateur

Info

Publication number
EP3111392A1
EP3111392A1 EP15710630.3A EP15710630A EP3111392A1 EP 3111392 A1 EP3111392 A1 EP 3111392A1 EP 15710630 A EP15710630 A EP 15710630A EP 3111392 A1 EP3111392 A1 EP 3111392A1
Authority
EP
European Patent Office
Prior art keywords
data
user
user interface
distinct visual
distinct
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP15710630.3A
Other languages
German (de)
English (en)
Inventor
Qi Lu
Derrick Leslie CONNELL
Darren Shakib
William H. GATES
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Priority claimed from PCT/US2015/017167 external-priority patent/WO2015127404A1/fr
Publication of EP3111392A1 publication Critical patent/EP3111392A1/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces

Definitions

  • a unified experience environment can support mechanisms that collect and utilize contextual metadata to associate information in accordance with its relevance to a user's current context.
  • the unified experience environment can support an ambient data collector that can obtain contextual and activity information coincident with a user's creation, editing or consumption of data and can associate it with such data as contextual metadata.
  • the unified experience environment can further support a context generator that can utilize contextual metadata to identify
  • Such a proactive presentation can include a context panel that can be alternatively displayed and hidden in an application-independent manner and a unified activity feed that can comprise correlated data groupings identified by one or more correlation engines, including a universal cross-application correlation engine and individual, application-specific correlation engines that can exchange information with one another through data correlation interfaces.
  • the context panel and unified activity feed either individually or together, enable a user to more conveniently and more efficiently access data that is relevant to the user's current context and increase the user's interaction performance with a computing device presenting such user interfaces by enabling the user to operate within a task-centric paradigm instead of the existing application-specific or device-specific paradigms described above.
  • Figure 1 is a block diagram of an exemplary system for providing unified presentation of contextually connected information
  • Figures 2a and 2b are block diagrams of exemplary components for acquiring contextual metadata and using acquitted contextual metadata, respectively, to support a unified presentation of contextually connected information;
  • Figure 3 is a diagram of an exemplary user interface providing unified
  • Figures 4a and 4b are block diagrams of exemplary components for providing a unified presentation of contextually connected information
  • Figure 5 is a diagram of another exemplary user interface providing unified presentation of contextually connected information
  • Figure 6 is a block diagram of an exemplary computing device.
  • a unified experience environment can operate and execute on multiple different types of computing device hardware and can span the multiple types of computing devices that a user would utilize.
  • the unified experience environment is capable of knowing the users, their data, their documents, and is able to provide contextually relevant insights for the user and give the user a way to interact with the device in a contextual manner.
  • a device-independent service layer that can be constructed around the user and can provide mechanisms by which a user can interact with a computing device, and receive assistance from the computing device, in the performance of tasks.
  • the exemplary system 100 illustrated therein provides context for the descriptions below directed to a unified experience environment, such as the exemplary unified experience environment 140 shown in the exemplary system 100 of Figure 1.
  • the computer-readable data with which users interact can be compartmentalized into data relevant to the users themselves, data relevant to groups or organizations to which the user belongs, and other data that the user may, at some point in time, desire to consume or interact with.
  • Graphically, such division of data is illustrated in the form of the circles 111, 112 and 113, where the innermost circle 111 can represent a "digital self, a surrounding circle 112 can represent a "digital organization", and an outermost circle 113 can represent a "digital universe”.
  • the "digital self, such as that conceptually encompassed by the circle 111, can be an amalgamation of the data that can be managed on behalf of a user, such as data retained by a service that the user can access, such as via network communications, from multiple computing devices.
  • data includes data generated by the user, which can be stored in a private and secure manner such that it is only accessible by the user, or others to whom the user has given permission. Additionally, such data includes data explicitly or implicitly defined by the user, such as user settings, preferences, locations and the like.
  • Such data can be stored in a user store 121, which can represent various storage devices that can be communicationally coupled to computing devices being utilized by the user, in particular storage devices that the user can access through network communications, such as storage devices maintained by a data storage service being utilized by the user to store their data.
  • the "digital organization" such as that conceptually encompassed by the circle 112, can be an amalgamation of the data that can be managed, or maintained, on behalf of a defined group of users, such as a corporation, club, organization, or other like group of users, where access to such data is typically limited to only those users within the defined group.
  • such data can be retained by a service that members of the group access comma such as via network communications, for multiple computing devices.
  • Such data can include data generated by coworkers within a corporate or business environment, a family members within a family environment, peers within a social networking environment, and the like.
  • the data from which the "digital organization" is comprised can be stored in a productivity store 122, which can be any storage device or service from which the user would access such "digital organization" data.
  • the productivity store 122 can be a networked document sharing service, a centralized document management system, a distributed document synchronization system, or combinations thereof.
  • Publically available data can be represented by the "digital universe" that is conceptually encompassed by the circle 113 of Figure 1. As illustrated, such data is typically accessed through search engines and, as such, can be indexed as part of a search store 123, an entity store 124, or other like database or index. While such data is typically presented through hypermedia documents, it is not so limited and can be any data, sourced externally from the user or groups to which the user belongs, which can be presented to or consumed by the user.
  • the unified experience environment 140 can facilitate the performance of digital activities 131 that utilize, generate, or consume the data conceptualized by the circles 111, 112 and 113. More specifically, in one embodiment, the unified experience environment 140 can collate and export, or otherwise provide access to, functionality including
  • contracts/actions functionality 141 context functionality 142, natural interaction functionality 143, data/activity connections functionality 144 and support functionality 145.
  • the contracts/actions functionality 141 can support the various capabilities of application programs, and other like "experience endpoints" through which a user utilizes a computing device, to perform useful actions.
  • the term "contract” can refer to an explicit, or implicit, agreement between a user and an application program or service, namely to define the functionality or features that the application program or service provides to the user.
  • the contracts/actions functionality 141 can include interfaces, data structures, and computer-executable instructions that can be utilized by experience endpoints to perform actions on behalf of the user, or in response to the user's requests.
  • the context functionality 142 can represent mechanisms by which the context of a user and the tasks that they are performing is understood in such a manner that the user can be more accurately and more helpfully assisted in the performance of such tasks.
  • the context functionality 142 can include capabilities directed to the capturing of contextual information, including environmental information, and the subsequent utilization of such contextual information.
  • Contextual information that can be captured, and subsequently utilized, can include activities that the user was performing concurrently with the current task, or during a period of time that includes the current task, a time of day that a task was performed, the manner in which the task was performed, and other like contextual information.
  • Environmental information can also provide contextual information and can be part of the context functionality 142.
  • environmental information can include a geographic location of the user at a particular point in time, other users, or other computing devices, that were found to be proximate to the user at that point in time, the hardware status of various computing devices at that point in time, and other like environmental information.
  • the natural interaction functionality 143 can represent capabilities by which a user's interactions with a computing device can be more human-centric. More specifically, the mechanisms by which users currently interact with computing devices typically comprise interfaces that the user does not use to interact with any other person or thing.
  • the natural interaction functionality 143 can include capabilities to enable users to interact with computing devices in a more human-centric manner, such as by speaking commands, having information spoken back to them, "showing" the computing device images to convey meaning, and others like more human-centric interfaces.
  • the natural interaction functionality 143 can comprise natural language recognition capabilities, as well as the capabilities to communicate back to a user utilizing natural language functionality, optical character recognition functionality by which the content of digital images can be interpreted, and other like capabilities that can facilitate more natural interactions with a computing device.
  • the data/activity connections functionality 144 can represent those aspects of the user's interaction with the computing device that are transient and not typically retained. More specifically, user interaction with computing devices often includes substantial back- and- forth communications where, typically, it is only the result of such communications that is retained.
  • the data/activity connections functionality 144 can represent capabilities based upon, and directed to, the retention of such back-and-forth interactions between a user and a computing device.
  • the data/activity connections functionality 144 can include the capture and utilization of information such as a user's search history, an ordered listing of the files a user accessed, created or modified, words that the user had auto-corrected, folders or data stores that the user had open for an extended period of time, the user's email viewing history and other like data indicative of transient aspects of the user's interaction with one or more computing devices, including patterns of interaction with different computing devices including, for example, locations where specific computing devices utilized, times of day the specific computing devices utilized, and other like patterns of interaction.
  • information such as a user's search history, an ordered listing of the files a user accessed, created or modified, words that the user had auto-corrected, folders or data stores that the user had open for an extended period of time
  • the user's email viewing history and other like data indicative of transient aspects of the user's interaction with one or more computing devices, including patterns of interaction with different computing devices including, for example, locations where specific computing devices utilized, times of day the specific computing devices utilized, and other like
  • the support functionality 145 can represent logistical functionality including automated backup functionality, error correction and redundancy functionality, performance optimization functionality, hardware support and driver functionality, and other like functionality. As illustrated in the system 100 of Figure 1, the unified experience
  • the unified experience environment 140 can be a layer that can execute on one or more operating systems or other like execution support platforms that can facilitate the execution of computer-executable instructions on computing device hardware, and, as such, the unified experience environment 140 can provide the support functionality 145 by amalgamating existing support functionality in the particular operating system on which the unified experience environment 140 is layered with support functionality that can be externally and independently provided.
  • the unified experience environment 140 can be a layer between an operating system/platform layer 152 and an experience endpoints layer 151 that can comprise application programs, hypermedia documents and other like mechanisms with which a user directly interacts.
  • the unified experience environment 140 is operating system and platform independent and is designed to execute on any computing device hardware.
  • the computing device layer 153 can include various different types of computing devices, such as laptop computing devices, desktop computing devices, smartphone computing devices, tablet computing devices, and smart wearable devices, such as smart watches, smart glasses, smart sensors, and the like, and can include both dedicated-use computing devices, and general- purpose computing devices.
  • the operating system/platform layer 152 providing support for the unified experience environment 140, can include any operating system and platform providing the relevant execution support, including multiple ones of the operating systems and platforms that are most commonly utilized.
  • Illustrative, but strictly exemplary, aspects of the functionality of the unified experience environment 140 can be described by reference to exemplary common tasks performed by users with computing devices. For example, a user performing note taking tasks on their computing device can be aided in such tasks by the mechanisms supported by the unified experience environment 140.
  • the natural interaction functionality 143 can enable the user to take a photograph and have information from the photograph recognized on the user's behalf.
  • the user could simply take a photograph of such a printed advertisement, and the natural interaction functionality 143 could recognize relevant aspects of such a printed advertisement, such as, for example, that the printed advertisement contained text specifying a particular date and time and a particular place for the event.
  • the contextual functionality 142 could recognize that the printed advertisement was for an event, as opposed to, for example, for a product or a service. Consequently, by simply taking a photograph of the printed advertisement, the user could have relevant information automatically entered into their computing device, such as a calendar event for a particular date and time and specifying a particular location.
  • a user performing content creation tasks on their computing device can be aided in such tasks by the mechanisms supported by the unified experience environment 140.
  • the data/activity connections 144 can include recent searches on a particular topic, which, in combination with the contextual functionality 142 can be utilized to identify the specific project on which the user is currently focused without requiring the user to explicitly identify such a project, or even ever explicitly delineate such a project as a category or construct by which to organize or categorize information.
  • contextual information automatically derived, mechanisms can be automatically invoked to aid the user in the relevant content creation. For example, when a user attempts to create a new word processing document, a template for documents associated with the specific project can be automatically invoked based upon the recognition that the user was likely currently focused on such a project.
  • the unified experience environment 140 can enable users of computing devices to focus on task completion, as opposed to application program interfaces, and can, thereby, enable such users to be more productive and complete more tasks more quickly.
  • the scope of assistance that can be provided by the unified experience environment 140 can be far-reaching.
  • a user can receive an email from a colleague notifying the user that a part in a device being designed by the user's company is failing.
  • the unified experience environment 140 can support mechanisms by which the language of such a supplier can be automatically recognized and computer- translation can be automatically applied such that, during the audio or video communication, each party can speak in their native language and have their communications automatically translated.
  • the unified experience environment 140 can support mechanisms by which the identity of the project and, thereby, the identity of the relevant team and its membership, is automatically recognized, an available conference room at the time referenced by the user is automatically identified, and the meeting invitation is automatically generated on the user's behalf.
  • the unified experience environment 140 can support mechanisms by which the presence of video presentation equipment can be automatically recognized and communicationally coupled with the user's computing device such that presentation can be commenced immediately.
  • the unified experience environment 140 can support mechanisms by which the presence of the user's coworkers can be detected and, based upon keywords in the meeting invitation, and the coworkers present, the relevant documentation can be made available to the user without the user having to explicitly search for it.
  • the unified experience environment 140 can support a more natural, task- focused interface between a user and any computing device the user seeks to utilize.
  • the unified experience environment 140 can support the automatic presentation of information that is relevant to a task that the user is currently performing, including automatically selecting and presenting, to the user, various suggestions, documents, events, communications, entities, other users, collections, and any other data that can aid the user in the performance of the task they are currently focused on.
  • unified experience environment 140 can support the automatic organization, collation and presentation of information to a user, thereby automatically organizing all of the user's digital life, including documents, events, communications, entities, other users, collections, and any other data generated by, collected by, utilized by, or otherwise associated with the user.
  • the unified experience environment 140 can abstract away conventional hierarchical data organization interfaces, enable the user to access the relevant data and the relevant functionality of the user's computing device, without explicitly having to organize and then subsequently find such data, and without having to explicitly install, manage, and select one or more application programs comprising computer-executable instructions for implementing such functionality.
  • the exemplary system 201 illustrates one embodiment by which contextual information can be retained for subsequent utilization, such as to facilitate the above described mechanisms and functionality.
  • the exemplary system 201 of Figure 2a can comprise an ambient data collector component 210 that can collect and retain ambient data, such as exemplary ambient data 231.
  • Ambient data is data regarding the environment within which a data-generating, or data-modifying event occurs.
  • ambient data includes both data regarding the physical environment, such as a current time, geographic location, users, devices, or entities that are physically present and other like physical environmental attributes, as well as data regarding the computing environment, such as activities that were being performed concurrently on the computing device, related actions by the user, such as searches the user performed, documents the user opened, text messages the user exchanged, and other like computing environment aspects.
  • An application program such as exemplary application 220, can generate data, such as exemplary data 242, and can store such data on a data store, such as exemplary data store 250, as illustrated by the arrow 232.
  • the exemplary application 220 can be any data-creating and/or data-consuming application, including content creating applications, such as word processors and spreadsheet applications, as well as digital image editing applications, social networking applications, email or personal information management applications, relational database application, and any other data-creating or data-organizing application.
  • the exemplary application 220 can make available, to a user, capabilities that are supported by the contracts/actions functionality 141 and the natural interaction functionality 143 of the unified experience environment that was described in detail above.
  • the data store 250 which can represent any hardware data storage, or software-based data storage service, can be supported by the support functionality 145 of the unified experience environment.
  • an ambient data collector component such as the exemplary ambient data collector 210, can, pending the user's approval, generate ambient data and can store such ambient data, as illustrated by the arrow 231 , with the data 242, on the data store 250, in the form of associated contextual metadata 241. More specifically, in one
  • an ambient data collector component such as exemplary ambient data collector 210 can interface with, or be a part of, the context functionality 142 of the unified experience environment described previously. Similarly in one embodiment, either in addition to the context functionality 142, or as an alternative, the ambient data collector 210 can also interface with, or be a part of, the data/activity connections functionality 144.
  • the ambient data collector 210 can have access to ambient data such as, for example, the time at which the application 220 is generating the data 242, the geographic location of the computing device on which the application 220 is executing, or the computing device through which the user is interacting with the application 220, if the application 220 is executing remotely from the user, to cause it to generate the data 242, one or more users that may be proximate to the user at the time the data 242 is being generated by the application 220, devices that may be proximate to the user at that time, any activities that the user may have been performing concurrently, or near in time, such as searches or other data consumption or creation actions, data sources that the user, or the application 220, may have accessed as part of the generation of the data 242, and other like ambient data.
  • ambient data such as, for example, the time at which the application 220 is generating the data 242, the geographic location of the computing device on which the application 220 is executing, or the computing device through which the user is interacting with the application 220,
  • the ambient data collector 210 can, as part of the data/activity connections functionality 144, have access to ambient data such as searches that the user may have performed in connection with the application 220 generating the data 242, data or
  • any data sources from which portions of the data 242 may have been copied or pasted communications, such as chats, text messages, audio communications, video communications and the like that the user may have engaged in concurrently with the generation of the data 242, and other like ambient data.
  • the ambient data collector 210 can obtain data from other computing devices that are also being utilized, either concurrently or at differing times, by the same user.
  • the context functionality 142 can provide indicia of the fact that, for example, the user's tablet computing device is proximately located to the user's laptop computing device, and the data/activity connections 144 can further provide indicia indicating that while the user was, for example, editing a presentation with the application 220 on the user's laptop computing device, the user was simultaneously engaging in, for example, a video chat on the user's tablet computing device.
  • information regarding such a video chat on the user's tablet computing device can be part of the contextual metadata 241 that can be stored by the ambient data collector 210 executing on, for example, the user's laptop computing device.
  • the ambient data collector 210 can be executing on a networked computing device hosting a service through which the user was performing one or both of: editing the presentation on the user's laptop computing device and/or engaging in the video chat on the user's tablet computing device.
  • a user can be provided with control regarding the sort of ambient data that the ambient data collector 210 can collect.
  • control can be generalized, such as alternatively allowing, or not allowing, ambient data to be collected and stored.
  • control can be specific, thereby enabling a user to delineate which types of ambient data can be collected, where such collected ambient data can be stored, and under what conditions such ambient data can be utilized, such as by the context generator 260, described in detail below.
  • the user can be provided with notifications when ambient data is being collected, or with notifications generally indicating that the user is agreeing to enable ambient data collection.
  • the ambient data collector 210 can filter the collected ambient data and can select a subset thereof to be retained as the contextual metadata 241 that is associated with the data 242.
  • the filtering of collected ambient data can be based on predetermined criteria that can be indicative of the relevance of collected ambient data. For example, if the data 242 is part of a word processing document, then searches that the user may have been performing concurrently with the storage of the data 242 may have greater relevance while, as another example, if the data 242 is part of a photograph that the user has taken, then the user's geographic location or other users or proximate to the user can be more relevant ambient data.
  • the ambient data collector 210 can retain all of the ambient data that it is able to collect without filtering such data prior to storage on the data store 250.
  • the ambient data collected by the ambient data collector component 210 can be stored as the contextual metadata 241 that can be associated with the data 242.
  • the contextual metadata 241 can be stored as part of the file containing the data 242.
  • the contextual metadata 241 can be retained in various forms and formats including, for example, name/value pairs, extensible markup language defined values, a tabular, or database structure, and other like forms.
  • the exemplary system 202 shown therein illustrates exemplary mechanisms by which the contextual metadata 241, generated and stored by the ambient data collector component 210, can be subsequently utilized, such as by the exemplary context generator component 260 shown.
  • the application 220 can read data, such as the exemplary data 242, from the data store 250, as illustrated by the arrow 272.
  • the application 220 can be any data-consuming and/or data-creating application and can be supported by the contracts/actions functionality 141 and the natural interaction functionality 143 of the unified experience environment.
  • the data store 250 can, as before, be supported by the support mechanisms 145 of the unified experience environment.
  • the reading of the data 242, by the application 220, as illustrated by the arrow 272 can trigger a contextual metadata consuming component, such as the exemplary context generator 260, shown in Figure 2b, to utilize the contextual metadata 241 that is associated with the data 242 that was read by the application 220, and identify relationships, as illustrated by the identify relationships action 281, between such contextual metadata 241 and one or more other collections of contextual metadata, such as the exemplary contextual metadata 292, 294 and 296.
  • the relationships between the contextual metadata 241 and other contextual metadata, such as the exemplary contextual metadata 292, 294 and 296, can be based on identified similarities, or equivalencies, such as shared aspects, between the contextual metadata 241 and the contextual metadata 292, 294 and 296.
  • each of the contextual metadata 241, 292, 294 and 296 can comprise contextual information indicating that the user was the same geographic location, such as the user's office, when the data corresponding to such contextual metadata was generated.
  • each of the contextual metadata 241, 292, 294 and 296 can identify that the same collection of other users were proximate to the user when the data corresponding to such contextual metadata was generated.
  • the relationships between the contextual metadata 241 and other contextual metadata can be based on heuristic analysis to identify commonalities or similarities between. For example, the user can have been scheduled in meetings directed to a common topic during the time when the data associated with the contextual metadata 292, 294 and 296 were generated.
  • reference to existing identified relationships can be utilized as part of the identify relationships action 281.
  • the identify relationships action 281, performed by the context generator component 260 can take such existing structure and correlations into account when determining which of the data associated with the contextual metadata 292, 294 and 296 is related to data 242 that is associated with the contextual metadata 241.
  • Another existing relationship that can inform the identification of relationships action 281 can be the relationship among computing devices based on the computing devices each having been utilized by a common user.
  • the user's actions on any one or more of those computing devices can serve as input from which contextual metadata, such as the contextual metadata 242, 292, 294 and 296 was generated.
  • the user's actions on any one or more of those computing devices can further server as the impetus behind the
  • Each of the contextual metadata 292, 294 and 296, which can have been identified to have a relationship with the contextual metadata 241, can be associated with one or more collections of data, such as exemplary data 291, 293 and 295, respectively. Therefore, in one embodiment, a context panel, such as the exemplary context panel 261, or other like user- facing functionality, can be invoked by the user and, in response to such an invocation, the data 291, 293 and 295 that is associated with the contextual metadata 292, 294 and 296 that was deemed related to the contextual metadata 241, can be provided to the context panel 261 by the context generator component 260, as illustrated by the arrow 276.
  • a context panel such as the exemplary context panel 261, or other like user- facing functionality
  • data determined to be contextually related to the data associated with the task the user is currently performing can be made available to aid the user in the performance of such a task.
  • the contextually relevant data that can be made available to the user can include documents, images, email messages, schedule events, individuals, entities, social media postings, collections and any other like data.
  • the exemplary user interface 300 shown therein illustrates exemplary embodiment by which contextually relevant data can be presented to a user to aid in the performance of tasks.
  • contextually relevant data can be presented within a context panel 310 which can be, alternatively, displayed and hidden by the user, such as via the user interface element 311.
  • the task being performed by a user can be performed as part of the primary content area 301.
  • the context panel 310 can then present contextually relevant information that can aid the user in, for example, editing the word processing document.
  • the context panel 310 could present documents, such as other textual documents, photographs, audio recordings, videos, emails or other like documents that are related to the textual document that user is currently editing, such as by being directed to the same topic.
  • the context panel 310 can also present other data deemed contextually relevant to the document the user is currently editing such as, for example, contact information for people related to the project or topic to which the document is directed, prior meetings at which the content of the document appears to have been discussed, email messages containing keywords or other indicia indicating their association with the document being edited.
  • the context panel 310 can be interactive. For example, the information presented therein can be acted upon by the user to perform other actions, such as generating a meeting invitation, sending a text message, or searching for an image. As another example, the user can add content to the context panel 310 such as, for example, by dragging such content from the primary content area 301 into the context panel 310.
  • the user can obtain content from the context panel 310 to include with the task the user's currently performing in the primary content area 301.
  • the context panel 310 contained a collection of documents relevant to the document currently being edited
  • the document currently being edited could be saved with, or at least associated with, that collection of documents by simply dragging from the primary content area 301 to the context panel 310.
  • the user could find such a presentation already presented to them in the context panel 310, such as the presentation represented by the tile 331 in the exemplary context panel 310 shown in Figure 3.
  • the context panel 310 can proactively provide, to the user, information and data contextually relevant to the task the user is performing, which the user can have immediately available to them, thereby enabling the user to perform their task more efficiently and remain task- focused, as opposed to being distracted by trying to find the information they desire.
  • a user entering a conference room to begin a meeting can open a document, such as in the primary content area 301, which the user desires to present as part of the meeting.
  • the user's location, the identity of other users proximate to such a user, the user's calendar, indicative of a meeting at the current time, and other like environmental data can correlate the document currently open in the primary content area 301 with other meeting-related, or project-related, collections of data and such documents and collections of data can be waiting for the user in the context panel 310, which, as indicated previously, the user can alternatively trigger the display of, or the hiding of, the at the exemplary user interface element 311.
  • the identification of documents or other related content can be in the form of the tiles, such as the exemplary document tiles 331 and 332 illustrated in Figure 3. Should the user desire to present one of the documents identified by the tiles 331 and 332 during the presentation, the user need not waste any time in finding such documents since, in one embodiment, access to such documents can be obtained directly through the context panel, such as by clicking on, touching, selecting or otherwise activating the appropriate tiles, or otherwise directing user action thereto.
  • a context panel such as the exemplary context panel 310, can comprise active suggestions to the user, including suggestions for the performance of other tasks.
  • the user can be presented with suggestions, such as via the suggestion tile 320, by which the user can be suggested to, for example, instruct the computing device to establish communicational connection with the projector in the meeting room in which the user is currently located.
  • Such suggestions can, in one embodiment, remain in the context panel 310, and can then be removed when is determined that they are no longer relevant.
  • the user can be provided with explicit selection or decline options, such as the exemplary options 331 and 332.
  • user action directed to such suggestions can be yet another source of information for tuning the mechanisms that select which suggestions to surface to the user.
  • a context panel such as the exemplary context panel 310
  • the exemplary context panel 310 is shown in Figure 3 as comprising presentation of individual people, such as illustrated by the exemplary tiles 300 and 349, that can have been determined to be related to the task being performed by the user in the primary content area 301.
  • the context panel 340 can be interactive and, as such, identifications of people, such as via the exemplary tiles 340 and 349 can include communicational options, such as the communication options 341, 342 and 343, selection of which can trigger a communication to be directed to the targeted person via the selected communication vector directly from the context panel 310.
  • a context panel such as the exemplary context panel 310, can provide the user with information, notifications and options that are deemed to be contextually relevant the task the user is currently performing, thereby enabling the user to perform such a task more efficiently.
  • contextual groupings of documents, notifications, emails, people, photos, meetings, social network feeds, text messages, entities, individuals, web content, services and other like data can act as a default interface through which a user interacts with their computing device and accesses the functionality provided by application programs accessible to the user through such a computing device. More specifically, mechanisms supported by the unified experience environment can proactively organize and continually update digital content for the user to enable the user to focus on tasks rather than the organization of content.
  • the exemplary system 401 shown therein illustrates an exemplary embodiment for organizing and presenting content in accordance with determined relevance and interrelationships.
  • content to be made available to the user can be stored across multiple, independent and disparate stores or repositories.
  • a social network store 411, document store 412 and personal information manager store 413 are illustrated by way of example.
  • the social network store 411 can represent the data and content, relevant to a user, that can be stored by, and accessed through, existing social network mechanisms, such as network-based social networking services.
  • the document store 412 can represent the documents created by, edited by or relevant to a user, that can be stored by, and accessed through, a collaborative document management and sharing platform
  • the personal information manager store 413 can represent the emails, meetings, calendar events, notes and other like data that can be created by, stored by and accessed through a personal information manager, such as an email program.
  • an abstraction layer such as the abstraction layer 420
  • an abstraction layer can be supported by the functionality made available via the unified experience environment, and can abstract data from multiple different sources into a single abstracted data 431 that can then be correlated together into groupings or collections for more elegant and efficient presentation to a user, thereby enabling a user to remain task- focused rather than application- focused.
  • an abstraction layer such as the exemplary abstraction layer 420, can interface with one or more data stores, such as the exemplary social network store 411, document store 412 and personal information manager store 413, and extract therefrom data in a native format, such as a format in which it is retained by such stores.
  • the exemplary abstraction layer 420 can obtain data 421 from the social network store 411, data 422 from the document store 412 and data 423 from the personal information manager store 413.
  • the abstraction layer 420 can also obtain data from the data/activity connections functionality 144 of the unified experience environment. As illustrated in Figure 4a, and as described previously, the data/activity connections functionality 144 can make available, to the abstraction layer component 420, additional information that can become part of the abstracted data 431 , such as, for example, prior searches that the user performed, documents, webpages or other like collections of data that the user viewed, transcripts of communicational exchanges, such as A/V chats, text messages the user center received, content the user copied or pasted, and other like data that is not typically saved as part of any other data store.
  • additional information that can become part of the abstracted data 431 , such as, for example, prior searches that the user performed, documents, webpages or other like collections of data that the user viewed, transcripts of communicational exchanges, such as A/V chats, text messages the user center received, content the user copied or pasted, and other like data that is not typically saved as part of any other data store.
  • the abstraction layer component 420 can then abstract the data it has obtained, such as the data 421, 422 and 423, into a common abstracted data 431, which can, in one embodiment, utilize a common format.
  • a common format can include, for example, data structures for retaining environmental or contextual metadata, such as that described in detail above. Additionally, such a common format can include data structures by which original data can be retained while generated metadata can also be included and retained therewith.
  • the data 421, from the social network store 411 can include pictures generated by the user or by other users that are associated with the user through the social network.
  • the abstraction layer component 420 can utilize the natural interaction functionality 143, provided by the unified experience environment, to perform optical character recognition or other image analysis to generate metadata from such pictures.
  • image analysis can be performed to identify the faces of the individuals shown in the pictures.
  • optical character recognition can be utilized to detect textual content in the pictures such as, for example, street signs, building names, and other like data that can be useful for subsequently correlating the pictures with other related data.
  • metadata which can be generated by the abstraction layer component 420, can be part of the abstracted data 431, as can the original pictures that can have been part of the data 421 that was obtained from the social network store 411.
  • the data 422, from the document store 412 can include textual documents, such as word processing documents, that were generated by the user or which are otherwise associated with the user.
  • the abstraction layer component 420 can utilize the natural interaction functionality 143 to perform heuristic analysis of such documents to identify documents directed to a common topic. For example, returning to the above examples where the user is designing a part, the heuristic analysis, such as that provided by the natural interaction functionality 143, can look for references to part design, part components, specific part nomenclature, and other like heuristic cues, by which metadata can be generated identifying the topics of one or more of the documents that can be part of the data 422, obtained from the document store 412, by the extraction layer component 420. The abstraction layer component 420 can then save such generated metadata, such as, for example, based on heuristic analysis, together with the documents themselves, as part of the abstracted data 431.
  • generation of the abstracted data 431 can comprise the reformatting data into a more generalized, or abstracted form.
  • a word processing document received as part of the data 422, from the document store 412, can be saved, as the abstracted data 431, in a format that includes the original word processing document, but also includes only the textual content of such a document, independent of any formatting, or application-specific annotations and information intertwined with such textual content in the word processing document itself.
  • the abstracted data 431 can be easier for a correlation engine, such as the exemplary correlation engine 430, to parse and correlate into data groupings.
  • a correlation engine component such as exemplary correlation engine component 430
  • the correlation engine component 430 can reference metadata, which is saved as part of the abstracted data 431 , to identify correlations among the abstracted data 431. For example, metadata indicating the presence of specific keywords in each of several word processing documents can be utilized by the correlation engine 430 to determine that each of such word processing documents is directed to a common topic and, consequently, that such word processing documents are to be part of a correlated data grouping, such as exemplary correlated data groupings 441, encompassing such a common topic.
  • Metadata indicating that a document was created during a meeting with another user can be utilized by the correlation engine 430 to generate a correlated data grouping comprising the document as well as the meeting notification and such other user's information, which can have been obtained, for example, as part of the data 423, from the personal information management store 413.
  • the correlated data groupings 441 can comprise any of the data, originally obtained by the abstraction layer 420, into a single grouping.
  • a grouping encompassing a trip the user took with their friends can include email messages exchanged regarding such a trip, photographs posted on social networking sites from such a trip, the itinerary of the trip, as can have been obtained from an online travel service, the contact information of the friends that accompanied to the user on the trip, a spreadsheet maintaining trip costs, searches the user performed in connection with the trip, and other like data.
  • the correlated data groupings are not limited to a specific type or format of data, but rather encompass all data that can be conceptualized as being part of a specific topic or task.
  • the correlated data groupings 441 can be more intuitive for users. More specifically, users typically associate content with the context within which such content was generated. Thus, to a human user, a photograph taken on a trip is associated with the itinerary of the trip. Conversely, to currently existing computing processes, such a photograph would the accessible through a digital image organization application program, while the itinerary of the trip would be accessible through an online travel service, such as that accessed through a web browser application program. As such, currently existing computing processes force a user to introduce artificial barriers between data that the user otherwise conceptualizes as being related. To that end, the correlation engine 430 seeks to generate correlated data groupings 441 that group together the data that a user would conceptualize as being related.
  • the correlation engine 430 can utilize context functionality 142, such as that provided as part of the unified experience environment described above.
  • context functionality 142 can enable the correlation engine 430 to obtain and reference information such as the time when one or more sets of data were created or modified, the geographic location of the user when such data was created or modified, other users or devices that were proximate to the user when such data was created or modified, other activities that the user may have been engaging in at approximately the same time when the data was created or modified, or other like environmental or contextual information that can further enable the correlation engine 430 to identify correlations among the data.
  • the correlation engine 430 can generate correlated data groupings 441 based on an initial pass through the abstracted data 431. A subsequent pass can then be utilized to determine whether the correlated data groupings can be subdivided further, or, conversely, whether two or more correlated data groupings are to be combined together.
  • the correlation engine 430 can utilize heuristic analysis to distinguish between, for example, a user's vacation to one foreign country versus that user's business trip to another foreign country, or to distinguish between the user's friends and the user's family.
  • the correlation engine can continually reevaluate its groupings, or collections of data, and can subdivide groupings, or combine multiple groupings together, as appropriate in light of newly received data.
  • the correlation engine 430 can be provided to a unified activity feed component 440.
  • the unified activity feed component 440 can determine an ordering or ranking of the correlated data groupings 441 and can generate presentations of such correlated data groupings 441, to the user, via the user interface hardware that is present on whatever computing device the user is currently utilizing. In an alternative embodiment, such ranking can be performed by the correlation engine component 430.
  • the unified activity feed component 440 can be implemented in a cross-platform manner such that a user can be presented with equivalent sets of correlated data groupings whether the user is utilizing, for example, their desktop computing device, their tablet computing device, their smartphone computing device, their integrated automobile computing device, or any other dedicated or general purpose computing device. Additionally, the unified activity feed component 440 can continuously reorder the presentation of the correlated data groupings 441 based upon changes to the underlying data such as, for example, new documents being saved in the document store 412, new email messages arriving in the personal information management store 413, new social media postings being saved in the social network store 411 and other like changes.
  • Unified activity feed component 440 can also reorder the presentation of the correlated data groupings 441 based upon the context of the user and the tasks in which the user is currently engaging. Thus, for example, if the user's context is determined to include their location as being in their office at work, then the unified activity feed 440 can present correlated data groupings related to work more highly then correlated data groupings related to leisure activities. Conversely, if the user's context is determined to include the user's location at home, than the unified activity feed 440 can present correlated data groupings related to leisure activities. As another example, if the user is at home, but is working on a presentation for their job, the unified activity feed component 440 can present correlated data groupings that comprise both work-related data groupings and leisure-related data groupings.
  • the unified activity feed component 440 can utilize, or be a part of, the contracts/actions functionality 141 of the unified experience environment.
  • the support functionality 145 the unified experience environment can be utilized to support the various data stores from which the abstraction layer component 420 can obtain data.
  • the exemplary system 402 shown therein illustrates an alternative implementation by which a user can be provided with a unified activity. More specifically, the exemplary system 402 comprises the data 421, from the social network store 411, being consumed by a dedicated social networking application, such as the exemplary social networking application 451. Similarly, the data 422, from the document store 412, can be consumed by a dedicated document creation/editing application 452, and the data 423 from the personal information manager store 413 can be consumed by a dedicated personal information manager application 453. Thus, to the extent that the data 421, 422 and 423 is being consumed by applications for 451, 452 and 453, respectively, that are dedicated applications directed to the consumption of such data, the exemplary system 402 represents an existing, fractured, data consumption paradigm.
  • one or more of the various applications can comprise data correlation interfaces, such as the exemplary data correlation interfaces 461, 462 and 463, which can enable those applications to obtain data from other applications and, thereby, independently generate correlated data groupings, such as exemplary correlated data groupings 481, 482 and 483.
  • individual applications such as the social network application 451, the document creation/editing application 452 and the personal information manager application 453 can initially correlate the data 421, 422 and 423, respectively, associated with such application programs, into one or more correlated data groupings, such as exemplary correlated data groupings 481, 482 and 483.
  • individual applications such as the social network application 451, the document creation/editing application 452 and the personal information manager application 453, can utilize data correlation interfaces to request additional data from others of the individual applications.
  • the social network application 451 can transmit a request to the document creation/editing application 452, via the data correlation interface 462 thereof, requesting that the document
  • creation/editing application 452 provide, to the social network application 451, via the data correlation interface 461 thereof, documents from the documents store 412 that are part of the correlated data groupings 481 generated by the social network application 451 from the data 421 from the social network store 411.
  • documents from the documents store 412 that are part of the correlated data groupings 481 generated by the social network application 451 from the data 421 from the social network store 411.
  • the document creation/editing application 452 can transmit the corresponding request, to the social network application 451 , via the data correlation interface 461 thereof, requesting that the social network application 451 provide, to the document creation/editing application 452, via the data correlation interface 462 thereof, social network information, such as postings, photos, messages and other like social network information that are part of the correlated data groupings 482 generated by the document creation/editing application 452 from the data 422 from the document store 412.
  • social network information such as postings, photos, messages and other like social network information that are part of the correlated data groupings 482 generated by the document creation/editing application 452 from the data 422 from the document store 412.
  • one or more of the individual applications can obtain, from the other applications, data that can be grouped with the correlated data groupings generated by an individual application program from data to which such an application program is natively directed.
  • individual application programs such as the social network application 451, the document creation/editing application 452 and the personal information manager application 453, can perform the correlation and data grouping functionality described above with reference to the correlation engine 430 shown in Figure 4a.
  • abstraction layer component 420 may not be necessary.
  • various individual application programs such as the social network application 451, the document creation/editing application 452 and the personal information manager application 453, can make use of the context functionality 142, the natural interaction functionality 143 and the data/activity connections functionality 144 provided by the unified experience environment, such as in the manner described in detail above.
  • each of the individual applications can provide to their correlated data groupings, such as the exemplary correlated data groupings 481, 482 and 483, to the unified activity feed component 440, whose operation was described in detail above.
  • unified activity feed component 440 can prioritize, organize and present and amalgamated set of the correlated data groupings from the individual correlated data groupings 481, 482 and 483 provided by the individual application programs.
  • such a prioritization and organization of correlated data groupings can be performed prior to the presentation thereof to the unified activity feed component 440. For example,
  • communications analogous to the communications 471, 472 and 473 can be exchanged among the individual applications to enable the individual applications to perform such prioritization and organization, and then provide a unified collection of correlated data groupings to the unified activity feed 440.
  • exemplary user interface 500 shown therein illustrates one exemplary embodiments by which a unified activity feed can be presented to a user.
  • various groupings or collections such as the collections 510, 530, 540, 550, 560 and 580, can be visually presented to a user such that each of the collections is presented as a separate, distinct visual element encompassing a defined visual area.
  • the most relevant collections to the user's current task or current context can be presented at the top of the exemplary user interface 500, while other collections can be presented lower down in the exemplary user interface 500, or, may not yet be visible and can require scrolling.
  • the exemplary scroll motion 590 can result in the visually oriented collections appearing to move up or down, as illustrated by the arrows 591 and 592.
  • the user can be presented with multiple collections of data through a "infinite scroll" paradigm where continually lower relevance collections require increasingly greater scrolling downward in order to make them visible within the exemplary user interface 500.
  • the user interface 500 can provide for selection mechanisms, including the displaying and selecting of additional content within a collection, or and the displaying and selection of additional collections themselves, that is amenable to computing devices having non-traditional user interfaces, such as television or media center computing devices, game console computing devices, in- car computing devices, and wearable computing devices, such as smartwatch computing devices and eyeglasses-based computing devices.
  • each individual collection such as, for example, the collection 550
  • the exemplary scroll motion 570 can result in the tiles 551, 552, 553, 554, 555 and 556 visually moving up or down so that other tiles either above or below exemplary tiles 551, 552, 553, 554, 555 and 556 can be displayed, as illustrated by the arrows 571 and 572.
  • the collections of data can represent associated and correlated data including, for example, documents, entities, meetings, emails, text messages, individuals, photographs, search histories, and any other data.
  • the previously described mechanisms can automatically generate titles for the collections of data, tags or labels for individual entries in the collections of data, or combinations thereof.
  • a user can be provided with the opportunity to enter or modify such titles, tags, labels or the like to better suit the user's needs.
  • the visual presentation of the individual data presented within a collection can, such as in the embodiment illustrated by the exemplary user interface 500, utilize a tile motif.
  • the exemplary collection 510 can include a tiles 511 and 512, which can be representative of meetings, tiles 513 and 514, which can be representative of documents, tile 515, which can be representative of an email, tile 516, which can be representative of an instant message communication session, tile 517, which can be representative of an individual, and other like tiles.
  • the user can direct a user action, such as an exemplary click or touch, selection, or other activation action 520 , to a tile representative of data the user desires to access, and an appropriate application program can be automatically selected and invoked to, for example, view, manipulate or consume the data.
  • a user action such as an exemplary click or touch, selection, or other activation action 520
  • an appropriate application program can be automatically selected and invoked to, for example, view, manipulate or consume the data.
  • the exemplary user interface 500 can be a primary user interface presented to a user, such as through a display device communicationally coupled to the computing device the user is currently using, because the user no longer has a need to maintain a hierarchical file organization, nor does the user have a need to directly access application programs.
  • the unified activity feed can be dynamic such that the ordering of collections, such as the exemplary collections 510, 530, 540, 550, 560 and 580, can be reordered and represented in different orientations and locations depending upon activity within those collections, as well as the user's current context.
  • tiles representative of data within a collection can, likewise, be reordered and presented in different orientations and locations depending upon the relevance to such a collection and the user's current context.
  • the collection can be dynamically resorted such that the documents the user is likely to select next can be visually presented near the top of the collection.
  • user scrolling can be minimized, since the goal can be to proactively present to the user an ordering that does not require the user to scroll often to find the content they seek.
  • a unified activity feed can also modify its visual appearance and presentation to the user in accordance with the user's context. For example, if the user's location is in a car, then a mobile computing device to which the user is presented with a unified activity feed can present a simplified unified activity feed that, for example, presents only one collection at a time, as opposed to the multiple collections presented within the exemplary user interface 500 of Figure 5. Distinction can also be made, in such an example, between a driver and a passenger in the car, and a driver can have a more appropriate interface vector selected for interfacing with the driver, such as, for example, exclusively utilizing a speech interface, as opposed to a visual interface that could distract the driver.
  • notifications can be presented via tiles in collections or other like notification paradigms, which can also take into account the context of the user and the ordering of the collections being presented in the unified activity feed. For example, if the unified activity feed is being presented in a manner consistent with the user's context at work, then work-based notifications, such as, for example, a notification that one of the elevators is not working, can be presented to the user. Predictive aspects of the user's context can also inform which notifications are presented to the user.
  • a notification can be proactively presented informing the user of such traffic conditions, and suggesting that the user leave very shortly in order to not be late for the family event.
  • the unified activity feed is being presented in a manner consistent with the user's context at home, then such work-based notifications need not be presented, and, instead, notifications relevance to, for example, the user's social life can be presented.
  • unified activity feed can provide a singular point of reference through which the user can access their digital data in a task- focused manner.
  • the user's computing devices can act as true "personal assistants”.
  • the exemplary computing device 600 can be any one or more of the computing devices referenced above.
  • the exemplary computing device 600 of Figure 6 can include, but is not limited to, one or more central processing units (CPUs) 620, a system memory 630, that can include RAM 632, and a system bus 621 that couples various system components including the system memory to the processing unit 620.
  • the system bus 621 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
  • the computing device 600 can optionally include graphics hardware, such as for the display of graphical user interfaces, including, but not limited to, a graphics hardware interface 690 and a physical display device 691, which can include physical display devices capable of receiving touch-based user input, such as a touch-sensitive, or multi-touch capable, display device.
  • graphics hardware such as for the display of graphical user interfaces, including, but not limited to, a graphics hardware interface 690 and a physical display device 691, which can include physical display devices capable of receiving touch-based user input, such as a touch-sensitive, or multi-touch capable, display device.
  • one or more of the CPUs 620, the system memory 630 and other components of the computing device 600 can be physically co-located, such as on a single chip.
  • some or all of the system bus 621 can be nothing more than silicon pathways within a single chip structure and its illustration in Figure 6 can be nothing more than notational convenience for the purpose of illustration.
  • the computing device 600 also typically includes computer readable media, which can include any available media that can be accessed by computing device 600 and includes both volatile and nonvolatile media and removable and non-removable media.
  • computer readable media may comprise computer storage media and communication media.
  • Computer storage media includes media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device 600.
  • Computer storage media does not include communication media.
  • Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
  • communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
  • the system memory 630 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 631 and the aforementioned RAM 632.
  • ROM read only memory
  • RAM 632 A basic input/output system 633 (BIOS), containing the basic routines that help to transfer information between elements within computing device 600, such as during startup, is typically stored in ROM 631.
  • BIOS basic input/output system
  • RAM 632 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 620.
  • Figure 6 illustrates the operating system 634 along with other program modules 635, and program data 636.
  • the computing device 600 may also include other removable/non-removable, volatile/nonvolatile computer storage media.
  • Figure 6 illustrates the hard disk drive 641 that reads from or writes to non-removable, nonvolatile media.
  • Other removable/non-removable, volatile/nonvolatile computer storage media that can be used with the exemplary computing device include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and other computer storage media, as defined and delineated above.
  • the hard disk drive 641 is typically connected to the system bus 621 through a non-removable memory interface such as interface 640.
  • the drives and their associated computer storage media discussed above and illustrated in Figure 6, provide storage of computer readable instructions, data structures, program modules and other data for the computing device 600.
  • hard disk drive 641 is illustrated as storing operating system 644, other program modules 645, and program data 646. Note that these components can either be the same as or different from operating system 634, other program modules 635 and program data 636.
  • Operating system 644, other program modules 645 and program data 646 are given different numbers hereto illustrate that, at a minimum, they are different copies.
  • the computing device 600 can operate in a networked environment using logical connections to one or more remote computers.
  • the computing device 600 is illustrated as being connected to the general network connection 661 through a network interface or adapter 660, which is, in turn, connected to the system bus 621.
  • program modules depicted relative to the computing device 600, or portions or peripherals thereof may be stored in the memory of one or more other computing devices that are communicatively coupled to the computing device 600 through the general network connection 661. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between computing devices may be used.
  • the exemplary computing device 600 can be a virtual computing device, in which case the functionality of the above-described physical components, such as the CPU 620, the system memory 630, the network interface 660, and other like components can be provided by computer-executable instructions.
  • Such computer-executable instructions can execute on a single physical computing device, or can be distributed across multiple physical computing devices, including being distributed across multiple physical computing devices in a dynamic manner such that the specific, physical computing devices hosting such computer-executable instructions can dynamically change over time depending upon need and availability.
  • the underlying physical computing devices hosting such a virtualized computing device can, themselves, comprise physical components analogous to those described above, and operating in a like manner.
  • virtual computing devices can be utilized in multiple layers with one virtual computing device executed within the construct of another virtual computing device.
  • the term "computing device”, therefore, as utilized herein, means either a physical computing device or a virtualized computing environment, including a virtual computing device, within which computer-executable instructions can be executed in a manner consistent with their execution by a physical computing device.
  • terms referring to physical components of the computing device as utilized herein, mean either those physical components or
  • a method for generating contextual metadata comprising: detecting the generation of a set of data for which a corresponding contextual metadata is to be generated; obtaining ambient data comprising at least one of: one or more users proximate to a computing device generating the set of data, one or more devices proximate to a computing device generating the set of data, and one or more concurrent activities being performed concurrently with the generating the set of data; generating at least a portion of the corresponding contextual metadata based upon the obtained ambient data.
  • a method for providing contextually relevant data comprising: detecting an accessing of a set of data having corresponding contextual metadata; identifying relationships between the corresponding contextual metadata and at least one other set of contextual metadata; obtaining at least one other set of data corresponding to the at least one other set of contextual metadata; and providing the obtained at least one other set of data in response to the detecting the accessing of the set of data.
  • a method for generating correlated data groupings comprising: obtaining data from multiple data stores, wherein each data store comprises an independent data format; abstracting the obtained data to comprise both the obtained data and additional data in a unified format; selecting a first one of the obtained data; determining, based on the first one of the obtained data and additional data in the unified format generated by abstracting the first one of the obtained data whether the selected first one of the obtained data is to be correlated with an existing data grouping; and correlating the selected first one of the obtained data with the existing data grouping or correlating the selected first one of the obtained data with a new data grouping based on the determining.
  • a method for generating correlated data groupings comprising: obtaining data from a first data store; selecting a first one of the obtained data; determining, based on the first one of the obtained data, whether the selected first one of the obtained data is to be correlated with an existing data grouping; correlating the selected first one of the obtained data with the existing data grouping or correlating the selected first one of the obtained data with a new data grouping based on the determining; obtaining, through an interface with an independently executing application program, additional data for at least one of: the existing data grouping or the new data grouping, the additional data being from a second data store different from, and independent of, the first data store.
  • the descriptions above include, as a first example, a system for improving a user's efficiency, and increasing the user's interaction performance, with the system, through the presentation of contextually connected information, the system comprising: one or more processing units; a display device communicationally coupled to the one or more processing units; and one or more computer-readable storage media communicationally coupled to the one or more processing units, the one or more computer-readable storage media comprising computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform steps comprising: detecting the user's accessing, via the system, of a first data that has a first contextual metadata corresponding to it, the first contextual metadata identifying prior contexts of the user that are associated with the first data;
  • identifying a relationship between the first contextual metadata and a second, different contextual metadata obtaining a second data to which the second contextual metadata corresponds, the second data being different than the first data; and physically generating, on the display device, a graphical user interface comprising: a primary content area comprising a display of at least a portion of the first data; and a context panel visually adjacent to the primary content area, the context panel comprising a first user interface element that is indicative of the obtained second data and that is user-selectable to access the obtained second data.
  • a second example is the system of the first example, wherein the graphical user interface further comprises a second user interface element, the second user interface element being displayed within the graphical user interface externally to, but visually connected, to the context panel; and wherein further the computer-readable media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: detecting user action directed to the second user interface element; and, in response to the detecting, variously displaying or hiding the context panel within the graphical user interface.
  • a third example is the system of the first example, wherein the first user interface element comprises a tile comprising a textual identification of the obtained second data and a graphical icon representative of the obtained second data.
  • a fourth example is the system of the first example, wherein content of the context panel is user-modifiable; and wherein further the computer-readable media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: detecting a user action comprising a drag action from the primary content area to the context panel; and updating the graphical user interface to comprise an updated context panel now comprising second user interface element that is indicative of the first data.
  • a fifth example is the system of the first example, wherein the second data is a document and the first user interface element is a graphical icon representative of the document; and wherein further the computer-readable media comprise further computer- executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: detecting a user action directed to the graphical icon; accessing, in response to the detected user action, the second data; and updating the graphical user interface to comprise an updated primary content area now comprising a display of at least a portion of the document.
  • a sixth example is the system of the first example, wherein the second data references a person and the first user interface element comprises graphical icons
  • the computer-readable media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: detecting a user action directed to one of the graphical icons; and initiating, in response to the detected user action, via the system, a communication with the person, the initiated communication corresponding to the one of the graphical icons to which the detected user action was directed.
  • a seventh example is the system of the first example, wherein the second data references an action performable by the system; and wherein further the computer-readable media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: detecting a user action directed to the first user interface element; and initiating, in response to the detected user action, the action performable by the system that is referenced by the second data.
  • An eighth example is the system of the first example, wherein the context panel is automatically displayed within the graphical user interface based on a current time or a current location of the user.
  • a ninth example is the system of the first example, wherein the relationship between the first contextual metadata and the second contextual metadata is identified based on equivalencies or similarities between the first contextual metadata and the second contextual metadata.
  • a tenth example is the system of the first example, wherein the first contextual metadata comprises ambient data of prior user accesses of the first data.
  • An eleventh example is the system of the first example, wherein the computer- readable media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: physically generating, on the display device, a second graphical user interface prior to the detecting, the second graphical user interface comprising: a unified activity feed comprising: a first distinct visual element representing a first grouping of correlated data, the first distinct visual element encompassing a defined visual area and being distinct from other visual elements representing other groupings of correlated data, the first distinct visual element having displayed within it elements comprising: a second user interface element that is indicative of a third data; and a third user interface element that is indicative of a fourth data, differing from the third data, the third user interface element being visually adjacent to the second user interface element within the first distinct visual element; wherein the third and fourth data are correlated with one another; and a second distinct visual element representing a second, different grouping of correlated data, the second distinct visual element also encompassing
  • the second distinct visual element having displayed within it elements comprising: a fourth user interface element that is indicative of a fifth data, differing from the third and fourth data; and a fifth user interface element that is indicative of a sixth data, differing from the third, fourth and fifth data, the fifth user interface element being visually adjacent to the fourth user interface element within the second distinct visual element; wherein the fourth and fifth data are correlated with one another; and wherein the first distinct visual element and the second distinct visual element are adjacent to one another within the unified activity feed.
  • a twelfth example is the system of the eleventh example, wherein the computer- readable media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: detecting user action directed to scrolling the unified activity feed; updating the second graphical user interface, in response to the detecting the user action directed to scrolling the unified activity feed, such that the first distinct visual element and the second distinct visual element together move in response to the scrolling of the unified activity feed; detecting user action directed to scrolling within the first distinct visual element; and updating the second graphical user interface, in response to the detecting the user action directed to scrolling within the first distinct visual element, such that the second user interface element and the third user interface element together move within the first distinct visual element, exclusive of movement of the distinct visual elements in the unified activity feed and exclusive of movement of individual user interface elements in other distinct visual elements, in response to the scrolling within the first distinct visual element.
  • a thirteenth example is the system of the eleventh example, wherein the computer-readable media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the system to perform further steps comprising: determining a current context of the user; ranking the third or fourth data as more relevant to the determined current context of the user than the fifth or sixth data; visually arranging, in response to the ranking, the first distinct visual element above the second distinct visual element within the unified activity feed; and repeating the determining the current context of the user, the ranking and the visually arranging as the current context of the user changes.
  • a fourteenth example is a graphical user interface for improving a user's efficiency, and increasing the user's interaction performance, with a computing device physically generating the graphical user interface on a physical display device
  • the graphical user interface comprising: a primary content area comprising a display of at least a portion of a first data accessed by the user with the computing device; and a context panel visually adjacent to the primary content area, the context panel comprising a first user interface element that is indicative of a second data and that is user-selectable to access the second data; wherein the second data has a second contextual metadata corresponding to it, the second contextual metadata having an identified relationship with a first contextual metadata that corresponds with the first data that was accessed by the user.
  • a fifteenth example is the graphical user interface of the fourteenth example, further comprising a second user interface element, the second user interface element being displayed within the graphical user interface externally to, but visually connected, to the context panel, the second user interface element being user actionable to trigger displaying or hiding the context panel within the graphical user interface.
  • a sixteenth example is the graphical user interface of the fourteenth example, wherein the first user interface element comprises a tile comprising a textual identification of the obtained second data and a graphical icon representative of the obtained second data.
  • a seventeenth example is the graphical user interface of the fourteenth example, wherein the second data is either a document or the second data references a person or an action performable by the computing device; and wherein further the context panel further comprises a second user interface element that is indicative of a third data that differs in type from the second data and also is either a document or references a person or an action performable by the computing device.
  • An eighteenth example is a graphical user interface for improving a user's efficiency, and increasing the user's interaction performance, with a computing device physically generating the graphical user interface on a physical display device
  • the graphical user interface comprising: a unified activity feed comprising: a first distinct visual element representing a first grouping of correlated data, the first distinct visual element encompassing a defined visual area and being distinct from other visual elements representing other groupings of correlated data, the first distinct visual element having displayed within it elements comprising: a first user interface element that is indicative of a first data; and a second user interface element that is indicative of a second data, differing from the first data, the second user interface element being visually adjacent to the first user interface element within the first distinct visual element; wherein the first and second data are correlated with one another; and a second distinct visual element representing a second, different grouping of correlated data, the second distinct visual element also encompassing a defined visual area and being distinct from other visual elements representing other groupings of correlated data, including the first distinct visual element, the second distinct visual element having displayed within it elements comprising: a third user interface element that is indicative of a third data, differing from the first and second data; and a fourth
  • a nineteenth example is the graphical user interface of the eighteenth example, wherein the unified activity feed is scrollable by a user of the computing device separately from scrolling by the user within the first distinct visual element or the second distinct visual element; wherein further scrolling the activity feed causes visual movement, within the graphical user interface, of both the first and second distinct visual elements together, while scrolling within the first distinct visual element causes the first and second user interface elements to visually move together within the first distinct visual element, exclusive of movement of the distinct visual elements in the unified activity feed and exclusive of movement of individual user interface elements in other distinct visual elements.
  • a twentieth example is the graphical user interface of the eighteenth example, wherein a visual arrangement of distinct visual elements within the unified activity feed is in accordance with a determined relevance between a current context of the user and data indicated by user interface elements within those distinct visual elements, the visual arrangement being automatically updated as the current context of the user changes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Resources & Organizations (AREA)
  • Operations Research (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un environnement d'expérience unifié supportant des mécanismes qui collectent et utilisent des métadonnées contextuelles afin d'associer des informations en fonction de leur pertinence vis-à-vis d'un contexte actuel de l'utilisateur. Un collecteur de données ambiantes obtient des informations contextuelles et d'activité coïncidant avec la création, la modification ou la consommation de données par l'utilisateur et les associe à ces données en tant que métadonnées contextuelles. Un générateur de contexte qui utilise des métadonnées contextuelles pour identifier des relations entre des données et activer la présentation proactive de données pertinentes pour un contexte actuel de l'utilisateur. La présentation proactive comprend un panneau de contexte qui peut être affiché et caché alternativement indépendamment de l'application et un fil d'activité unifié qui comprend des regroupements de données corrélées identifiés par des moteurs de corrélation, incluant un moteur de corrélation inter-application universel et des moteurs de corrélation spécifiques à l'application individuels qui échangent des informations par l'intermédiaire d'interfaces de corrélation de données. Le panneau de contexte et le fil d'activité unifié permettent à des utilisateurs d'accéder plus efficacement à des données et d'augmenter leur performance d'interaction avec un dispositif informatique.
EP15710630.3A 2014-02-24 2015-02-23 Présentation unifiée d'informations contextuellement connectées afin d'améliorer l'efficacité et la performance d'interaction de l'utilisateur Withdrawn EP3111392A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201461944033P 2014-02-24 2014-02-24
PCT/US2015/017167 WO2015127404A1 (fr) 2014-02-24 2015-02-23 Présentation unifiée d'informations contextuellement connectées afin d'améliorer l'efficacité et la performance d'interaction de l'utilisateur

Publications (1)

Publication Number Publication Date
EP3111392A1 true EP3111392A1 (fr) 2017-01-04

Family

ID=57179418

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15710630.3A Withdrawn EP3111392A1 (fr) 2014-02-24 2015-02-23 Présentation unifiée d'informations contextuellement connectées afin d'améliorer l'efficacité et la performance d'interaction de l'utilisateur

Country Status (2)

Country Link
EP (1) EP3111392A1 (fr)
CN (1) CN106062790B (fr)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US8676904B2 (en) 2008-10-02 2014-03-18 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
DE212014000045U1 (de) 2013-02-07 2015-09-24 Apple Inc. Sprach-Trigger für einen digitalen Assistenten
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
DK201770427A1 (en) 2017-05-12 2018-12-20 Apple Inc. LOW-LATENCY INTELLIGENT AUTOMATED ASSISTANT
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
EP3428793A1 (fr) * 2017-07-10 2019-01-16 Siemens Aktiengesellschaft Procédé d'assistance d'un utilisateur lors d'une création d'une application logicielle et programme informatique comprenant une mise en uvre du procédé et interface programmable pouvant être utilisée dans le cadre d'un tel procédé
US10928918B2 (en) 2018-05-07 2021-02-23 Apple Inc. Raise to speak
DK180639B1 (en) 2018-06-01 2021-11-04 Apple Inc DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT
US11462215B2 (en) 2018-09-28 2022-10-04 Apple Inc. Multi-modal inputs for voice commands
US11348573B2 (en) 2019-03-18 2022-05-31 Apple Inc. Multimodality in digital assistant systems
US11227599B2 (en) 2019-06-01 2022-01-18 Apple Inc. Methods and user interfaces for voice-based control of electronic devices
US11061543B1 (en) 2020-05-11 2021-07-13 Apple Inc. Providing relevant data items based on context
WO2021231207A2 (fr) * 2020-05-11 2021-11-18 Apple Inc. Fourniture d'éléments de données pertinents sur la base d'un contexte
US11490204B2 (en) 2020-07-20 2022-11-01 Apple Inc. Multi-device audio adjustment coordination
US11438683B2 (en) 2020-07-21 2022-09-06 Apple Inc. User identification using headphones
CN114692195A (zh) * 2020-12-29 2022-07-01 腾讯科技（深圳）有限公司 一种文档处理方法、装置、设备及存储介质
CN117743470B (zh) * 2024-02-06 2024-05-07 中科云谷科技有限公司 用于异构大数据的处理系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070300185A1 (en) * 2006-06-27 2007-12-27 Microsoft Corporation Activity-centric adaptive user interface
US20090055739A1 (en) * 2007-08-23 2009-02-26 Microsoft Corporation Context-aware adaptive user interface
US20120011477A1 (en) * 2010-07-12 2012-01-12 Nokia Corporation User interfaces
CN102591657B (zh) * 2011-12-29 2014-06-25 东南大学 一种基于cpu及dsp协同机制的gui系统实现方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
None *
See also references of WO2015127404A1 *

Also Published As

Publication number Publication date
CN106062790A (zh) 2016-10-26
CN106062790B (zh) 2020-03-03

Similar Documents

Publication Publication Date Title
US10691292B2 (en) Unified presentation of contextually connected information to improve user efficiency and interaction performance
CN106062790B (zh) 统一呈现根据上下文连接的信息以改善用户的效率和交互绩效
CN109154935B (zh) 一种用于分析用于任务完成的捕获的信息的方法、系统及可读存储设备
US9275376B2 (en) Method and apparatus for providing soft reminders
US10671245B2 (en) Collection and control of user activity set data and activity set user interface
US20100211535A1 (en) Methods and systems for management of data
US20150169285A1 (en) Intent-based user experience
JP2019537081A (ja) コンテンツ管理システムにおけるプロジェクトの作成
US20120143931A1 (en) Context-aware folders
WO2017083218A1 (fr) Présentation par carte à puce des données tabulaires provenant d'une base de données de collaboration
US20110099189A1 (en) Method and apparatus for exploring connections of a polymorphic note
US20130332378A1 (en) Methods and Systems for Integrated Social Media, Strategy, and Organizational Hierarchy
WO2013152101A1 (fr) Traitement de document intelligent avec données en ligne et flux d'action associés
CA2810683A1 (fr) Dispositif electronique et procede de classification d'objets de donnees de communication
US20120144315A1 (en) Ad-hoc electronic file attribute definition
US11474843B2 (en) AI-driven human-computer interface for associating low-level content with high-level activities using topics as an abstraction
US20170192625A1 (en) Data managing and providing method and system for the same
US11244106B2 (en) Task templates and social task discovery
CN110462615B (zh) 增强的粘贴板使用的技术
KR101266132B1 (ko) 소셜 네트워크 기반의 디지털 컨텐츠 제작 방법과 공유 방법 및 이를 이용한 디지털 컨텐츠 제작 시스템과 공유 시스템
US20200004890A1 (en) Personalized artificial intelligence and natural language models based upon user-defined semantic context and activities
US11449764B2 (en) AI-synthesized application for presenting activity-specific UI of activity-specific content
US20200004388A1 (en) Framework and store for user-level customizable activity-based applications for handling and managing data from various sources
US8671123B2 (en) Contextual icon-oriented search tool
US11354581B2 (en) AI-driven human-computer interface for presenting activity-specific views of activity-specific content for multiple activities

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20160719

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20180802

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN WITHDRAWN

18W Application withdrawn

Effective date: 20190115