CN112905276A - System and method for actively identifying and displaying relevant content on a touch sensitive device - Google Patents

Abstract

Systems and methods for actively identifying and displaying relevant content on an electronic device with a touch-sensitive display are disclosed herein. In one aspect, the method includes executing an application on the electronic device in response to an instruction from a user of the electronic device. While executing the application, the method further includes collecting usage data. The usage data includes at least one or more actions performed by the user within the application. The method further comprises the following steps: automatically obtaining, without human intervention, at least one trigger condition based on the collected usage data, and associating the at least one trigger condition with a particular action of the one or more actions performed by the user within the application. Upon determining that the at least one trigger condition has been met, the method includes providing an indication to the user of: the particular action associated with the trigger condition is available.

Description

System and method for actively identifying and displaying relevant content on a touch sensitive device

The present application is a divisional application of the invention patent application having a filing date of 2016, 27/05, and having a filing number of 201680044146.3, entitled "system and method for actively identifying and displaying related content on a touch-sensitive device".

Technical Field

Embodiments disclosed herein relate generally to electronic devices with touch-sensitive displays, and more particularly, to systems and methods for actively identifying and displaying relevant content on an electronic device in communication with a display and a touch-sensitive surface.

Background

Handheld electronic devices with touch sensitive displays are ubiquitous. Users of these widely used handheld electronic devices can now install a large number of applications on their devices and use these applications to help them perform daily activities more efficiently. But to access these applications, the user typically must unlock their device, locate the desired application (e.g., by browsing the home screen to locate an icon associated with the desired application or by searching for the desired application within a search interface), and then locate the desired functionality within the desired application. As a result, users often spend a significant amount of time locating desired applications and desired functions within those applications without being able to immediately execute (e.g., through a single touch input) the desired applications and/or the desired functions.

Furthermore, the large number of installed applications overwhelms the user with a continuous stream of information, making it impossible to view this information thoroughly and immediately. Thus, users often wish to return at some later point in time to view some particular piece of information that they previously noted or to use some particular piece of information at some later point in time. But users often cannot locate or remember how to locate a particular piece of information.

It is therefore desirable to provide an intuitive and easy-to-use system and method for actively identifying and displaying relevant content (e.g., a particular piece of information) on an electronic device in communication with a display and a touch-sensitive surface.

Disclosure of Invention

Accordingly, there is a need for electronic devices having faster, more efficient methods and interfaces for quickly accessing applications and desired functionality within those applications. In addition, there is a need for electronic devices to help users manage the continuous stream of information they receive each day by actively identifying and providing relevant information (e.g., contacts, nearby locations, applications, news articles, addresses, and other information available on the device) before the user explicitly requests the information. Such methods and interfaces optionally complement or replace conventional methods of accessing applications. Such methods and interfaces produce a more efficient human-machine interface by requiring fewer inputs in order for a user to locate desired information. For battery-driven devices, this method and interface saves power and increases battery charging interval time (e.g., by requiring a smaller number of touch inputs to perform various functions). Moreover, such methods and interfaces help extend the life of a touch-sensitive display by requiring a smaller number of touch inputs (e.g., the methods and interfaces disclosed herein can actively provide a desired piece of information without requiring user input, rather than having to continuously and purposefully tap the touch-sensitive display to locate the piece of information).

The above-described deficiencies and other problems associated with user interfaces for electronic devices having touch-sensitive surfaces are addressed by the disclosed devices. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, tablet, or handheld device). In some embodiments, the device has a touch pad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touchscreen" or "touchscreen display"). In some embodiments, the device has a Graphical User Interface (GUI), one or more processors, memory, and one or more modules, programs or sets of instructions stored in the memory for performing a plurality of functions. In some embodiments, the user interacts with the GUI primarily through stylus and/or finger contacts and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, rendering, word processing, web page creation, disc editing, spreadsheet making, game playing, telephone answering, video conferencing, emailing, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playing, and/or digital video playing. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

(A1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured according to any of computing devices A-D of FIG. 1E) having a touch-sensitive display (touch screen 112, FIG. 1C). The method comprises the following steps: in response to an instruction from a user of the electronic device, an application is executed on the electronic device. When executing the application program, the method further comprises: usage data is collected. The usage data includes at least one or more actions (or types of actions) performed by the user within the application. The method further comprises the following steps: (i) automatically obtaining, without human intervention, at least one trigger condition based on the collected usage data, and (ii) correlating the at least one trigger condition with a particular action of the one or more actions performed by the user within the application. Upon determining that the at least one trigger condition has been met, the method comprises: an indication is provided to the user that a particular action associated with the trigger condition is available.

(A2) In some embodiments of the a1 method, obtaining the at least one trigger condition comprises: the usage data is sent to one or more servers located remotely from the electronic device and the at least one trigger condition is received from the one or more servers.

(A3) In some embodiments of the methods of any of a 1-a 2, providing the indication includes displaying a user interface object on the lock screen on the touch-sensitive display, the user interface object corresponding to a particular action associated with the trigger condition.

(A4) In some embodiments of the a3 method, the user interface object includes a description of a particular action associated with the trigger condition.

(A5) In some embodiments of the a4 method, the user interface object further includes an icon associated with the application.

(A6) In some embodiments of the method of any one of A3 to a5, the method further comprises: a first gesture is detected at a user interface object. In response to detecting the first gesture: (i) displaying an application on the touch-sensitive display, and (ii) while displaying the application, the method comprising: a particular action associated with the trigger condition is performed.

(A7) In some embodiments of the a6 method, the first gesture is a swipe gesture on the user interface object.

(A8) In some embodiments of the method of any one of A3 to a5, the method further comprises: a second gesture is detected at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display, a particular action associated with the trigger condition is performed.

(A9) In some embodiments of the A8 method, the second gesture is a single tap on a predefined area of the user interface object.

(A10) In some embodiments of the methods of any of A3-a 9, the user interface object is displayed in a predefined central portion of the lock screen.

(A11) In some embodiments of the a1 method, providing to the user an indication that a particular action associated with the trigger condition is available includes performing the particular action.

(A12) In some embodiments of the a3 method, the user interface object is an icon associated with the application, and the user interface object is displayed substantially on the touch-sensitive display locking a corner of the screen.

(A13) In some embodiments of the method of any one of a1 to a12, the method further comprises: an instruction is received from a user to unlock the electronic device. In response to receiving the instruction, the method includes: a home screen of the electronic device is displayed on the touch-sensitive display. The method further comprises the following steps: an indication is provided to the user on the home screen indicating that a particular action associated with the trigger condition is available.

(A14) In some embodiments of the a13 method, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device, and (ii) a second portion displayed adjacent to the first portion for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion, and providing an indication on the home screen includes displaying an indication on the second portion.

(A15) In some implementations of the a14 method, the second set of applications is different from and smaller than the first set of applications.

(A16) In some embodiments of the methods of any one of a1 to a15, determining that the at least one trigger condition has been met comprises: determining that the electronic device has been coupled with a second device different from the electronic device.

(A17) In some embodiments of the methods of any one of a1 to a16, determining that the at least one trigger condition has been met comprises: it is determined that the electronic device has reached a corresponding address of a home location or work location associated with the user.

(A18) In some embodiments of the a17 method, determining that the electronic device has reached a corresponding address of a home location or a work location associated with the user includes: motion data from an accelerometer of the electronic device is monitored, and a time for which the electronic device has not moved beyond a threshold amount is determined based on the monitored motion data.

(A19) In some embodiments of the method of any of a 1-a 18, the usage data further comprises verbal instructions from the user that are provided to the virtual assistant application while continuing to execute the application. The at least one trigger condition is further based on a verbal instruction provided to the virtual assistant application.

(A20) In some embodiments of the a19 method, the verbal instructions include a request to create a reminder that corresponds to a current state of the application, the current state corresponding to a state of the application at the time the verbal instructions were provided.

(A21) In some embodiments of the a20 method, the application state at the time the verbal instructions are provided is selected from the group consisting of: a page displayed within the application when the verbal instructions are provided, content played within the application when the verbal instructions are provided, a notification displayed within the application when the verbal instructions are provided, and an active portion of the page displayed within the application when the verbal instructions are provided.

(A22) In some embodiments of the a20 method, the verbal instructions include the word "this" in relation to the current state of the application.

(A23) In another aspect, the method is performed at one or more electronic devices (e.g., portable multifunction device 100 of FIG. 5, one or more servers 502 of FIG. 5). The method comprises the following steps: an application is executed on a first electronic device of the one or more electronic devices in response to an instruction from a user of the first electronic device. In the process of executing the application program, the method comprises the following steps: usage data is automatically collected without human intervention, the usage data including at least one or more actions (or types of actions) performed by a user within an application. The method further comprises the following steps: at least one trigger condition is automatically established based on the collected usage data without human intervention. The method further comprises: the at least one trigger condition is associated with a particular action of the one or more actions performed by the user within the application. Upon determining that the at least one trigger condition has been met, the method comprises: an indication is provided to the user that a particular action associated with the trigger condition is available.

(A24) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: a touch-sensitive display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of a 1-a 22.

(A25) In yet another aspect, an electronic device is provided, the electronic device including: a touch sensitive display and an apparatus for performing any of the methods of a 1-a 22.

(A26) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive display, will cause the electronic device to perform the method described in any of a 1-a 22.

(A27) In another aspect, a graphical user interface on an electronic device with a touch-sensitive display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of a 1-a 22.

(A28) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 4201 of fig. 42), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4203 of fig. 42), and a processing unit (e.g., processing unit 4205 of fig. 42). In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (i.e., computing devices a-D). For ease of illustration, fig. 42 shows the display unit 4201 and the touch-sensitive surface unit 4203 integrated with the electronic device 4200, however, in some embodiments, one or both of these units are in communication with the electronic device while these units remain physically separate from the electronic device. The processing unit is coupled with the touch-sensitive surface unit and the display unit. In some embodiments, the touch-sensitive surface unit and the display unit are integrated in a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes an execution unit (e.g., execution unit 4207 of fig. 42), a collection unit (e.g., collection unit 4209 of fig. 42), an acquisition unit (e.g., acquisition unit 4211 of fig. 42), an association unit (e.g., association unit 4213 of fig. 42), a providing unit (e.g., providing unit 4215 of fig. 42), a transmission unit (e.g., transmission unit 4217 of fig. 42), a reception unit (e.g., reception unit 4219 of fig. 42), a display unit (e.g., display unit 4221 of fig. 42), a detection unit (e.g., detection unit 4223 of fig. 42), an execution unit (e.g., execution unit 4225 of fig. 42), a determination unit (e.g., determination unit 4227 of fig. 42), and a monitoring unit (e.g., monitoring unit 4229 of fig. 42). The processing unit (or one or more components thereof, such as units 4207-4229) is configured to: executing an application on the electronic device (e.g., by execution unit 4207) in response to an instruction from a user of the electronic device; while executing the application, collecting usage data (e.g., usage collection unit 4209), the usage data including at least one or more actions performed by a user within the application; automatically acquiring, without human intervention, at least one trigger condition based on the collected usage data (e.g., by the acquisition unit 4211); associating (e.g., by associating unit 4213) the at least one trigger condition with a particular action of the one or more actions performed by the user within the application; and upon determining that the at least one trigger condition has been satisfied, provide an indication to the user (e.g., via providing unit 4215) indicating that a particular action associated with the trigger condition is available.

(A29) In some embodiments of the electronic device of a28, obtaining the at least one trigger spring comprises: the usage data is transmitted (e.g., by the transmitting unit 4217) to one or more servers located remotely from the electronic device, and the at least one trigger condition is received (e.g., by the receiving unit 4219) from the one or more servers.

(A30) In some embodiments of the electronic device of any of a 28-a 29, the providing the indication includes displaying (e.g., by display unit 4217 and/or display unit 4201) a user interface object on the lock screen on the touch-sensitive display unit that corresponds to the particular action associated with the trigger condition.

(A31) In some embodiments of the electronic device of a30, the user interface object includes a description of a particular action associated with the trigger condition.

(A32) In some embodiments of the electronic device of a31, the user interface object further includes an icon associated with the application.

(A33) In some embodiments of the electronic device of any of a 30-a 32, the processing unit is further configured to: a first gesture is detected (e.g., by detection unit 4223 and/or touch-sensitive surface unit 4203) at the user interface object. In response to detecting the first gesture: (i) display (e.g., via display unit 4217 and/or display unit 4201) an application on the touch-sensitive display unit, and (ii) perform (e.g., via execution unit 4225) a particular action associated with the touch spring while the application is displayed.

(A34) In some embodiments of the electronic device of a33, the first gesture is a swipe gesture on the user interface object.

(A35) In some embodiments of the electronic device of any of a 30-a 33, the processing unit is further configured to: a second gesture is detected (e.g., by detection unit 4223 and/or touch-sensitive surface unit 4203) at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display unit, the processing unit is configured to: a particular action associated with the trigger condition is executed (e.g., by execution unit 4225).

(A36) In some embodiments of the electronic device of a35, the second gesture is a single tap of a predefined area of the object at the user interface.

(A37) In some embodiments of the electronic device of any of a 30-a 36, the user interface object is displayed in a predefined central portion of the lock screen.

(A38) In some embodiments of the electronic device of a28, providing the user with an indication that the particular action associated with the trigger condition is available includes performing (e.g., by execution unit 4225) the particular action.

(A39) In some embodiments of the electronic device of a30, the user interface object is an icon associated with the application, and the user interface object is displayed substantially in a corner of the lock screen on the touch-sensitive display unit.

(A40) In some embodiments of the electronic device of any of a 28-a 39, the processing unit is further configured to: an instruction to unlock the electronic device is received (e.g., by the receiving unit 4219) from the user. In response to receiving the instruction, the processing unit is configured to: a home screen of the electronic device is displayed (e.g., via display unit 4217 and/or display unit 4201) on the touch-sensitive display unit. The processing unit is further configured to: an indication is provided to the user (e.g., via providing unit 4215) on the home screen indicating that a particular action associated with the trigger condition is available.

(A41) In some embodiments of the electronic device of a40, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device, and (ii) a second portion displayed adjacent to the first portion for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion, and providing the indication on the home screen includes displaying the indication on the second portion (e.g., via display unit 4217 and/or display unit 4201).

(A42) In some implementations of the electronic device of a41, the second set of applications is different from and smaller than the first set of applications.

(A43) In some implementations of the electronic device of any of a 28-a 42, determining that the at least one trigger condition has been met comprises: it is determined (e.g., by determining unit 4227) that the electronic device has been coupled with a second device different from the electronic device.

(A44) In some implementations of the electronic device of any of a 28-a 43, determining that the at least one trigger condition has been met comprises: it is determined (e.g., by determining unit 4227) that the electronic device has reached a corresponding address of a home location or work location associated with the user.

(A45) In some embodiments of the electronic device of a44, determining that the electronic device has reached a corresponding address of a home location or a work location associated with the user comprises: motion data from an accelerometer of the electronic device is monitored (e.g., by monitoring unit 4229) and a time for which the electronic device has not moved more than a threshold amount is determined (e.g., by determining unit 4227) based on the monitored motion data.

(A46) In some embodiments of the electronic device of any of a 28-a 45, the usage data further comprises verbal instructions from the user that are provided to the virtual assistant application while continuing to execute the application. The at least one trigger condition is further based on a verbal instruction provided to the virtual assistant application.

(A47) In some embodiments of the electronic device of a46, the verbal instruction includes a request to create a reminder that corresponds to a current state of the application, the current state corresponding to a state of the application at the time the verbal instruction was provided.

(A48) In some embodiments of the electronic device of a47, the application state at which the verbal instructions are provided is selected from the group consisting of: a page displayed within the application when the verbal instructions are provided, content played within the application when the verbal instructions are provided, a notification displayed within the application when the verbal instructions are provided, and an active portion of the page displayed within the application when the verbal instructions are provided.

(A49) In some embodiments of the electronic device of a46, the verbal instructions include the term "this" in relation to the current state of the application.

(B1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured according to any of computing devices A-D of FIG. 1E) having a touch-sensitive display (touch screen 112, FIG. 1C). The method comprises the following steps: at least one trigger condition is obtained, the at least one trigger condition being based on usage data associated with a user of the electronic device, the usage data including one or more actions (or types of actions) performed by the user within the application when the application is executed on the electronic device. The method further comprises the following steps: the at least one trigger condition is associated with a particular action of the one or more actions performed by the user within the application. Upon determining that the at least one trigger condition has been met, the method comprises: an indication is provided to the user that a particular action associated with the trigger condition is available.

(B2) In some embodiments of the B1 method, the method further comprises the method described in any one of a2 to a 22.

(B3) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: a touch-sensitive display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of B1-B2.

(B4) In yet another aspect, an electronic device is provided and includes: a touch sensitive display and means for performing any of the methods of B1-B2.

(B5) In another aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer readable storage medium is for storing executable instructions that, when executed by an electronic device with a touch-sensitive display, will cause the electronic device to perform the method described in any of B1-B2.

(B6) In another aspect, a graphical user interface on an electronic device with a touch-sensitive display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of B1-B2.

(B7) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 4201 of fig. 42), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4203 of fig. 42), and a processing unit (e.g., processing unit 4205 of fig. 42). In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (i.e., computing devices a-D). For ease of illustration, fig. 42 shows the display unit 4201 and the touch-sensitive surface unit 4203 integrated with the electronic device 4200, however, in some embodiments, one or both of these units are in communication with the electronic device while these units remain physically separate from the electronic device. The processing unit includes an execution unit (e.g., execution unit 4207 of fig. 42), a collection unit (e.g., collection unit 4209 of fig. 42), an acquisition unit (e.g., acquisition unit 4211 of fig. 42), a correlation unit (e.g., correlation unit 4213 of fig. 42), a providing unit (e.g., providing unit 4215 of fig. 42), a transmission unit (e.g., transmission unit 4217 of fig. 42), a reception unit (e.g., reception unit 4219 of fig. 42), a display unit (e.g., display unit 4221 of fig. 42), a detection unit (e.g., detection unit 4223 of fig. 42), an execution unit (e.g., execution unit 4225 of fig. 42), a determination unit (e.g., determination unit 4227 of fig. 42), and a monitoring unit (e.g., monitoring unit 4229 of fig. 42). The processing unit (or one or more components thereof, such as units 4207-4229) is configured to: obtaining (e.g., by obtaining unit 4211) at least one trigger condition, the at least one trigger condition based on usage data associated with a user of the electronic device, the usage data including one or more actions performed by the user within the application when the application is executed on the electronic device; associating (e.g., by associating unit 4213) the at least one trigger condition with a particular action of the one or more actions performed by the user within the application; and upon determining that the at least one trigger condition has been satisfied, provide an indication to the user (e.g., via providing unit 4215) indicating that a particular action associated with the trigger condition is available.

(B8) In some embodiments of the electronic device of B7, obtaining the at least one trigger condition comprises: the usage data is transmitted (e.g., by transmitting unit 4217) to one or more servers located remotely from the electronic device, and the at least one trigger condition is received (e.g., by receiving unit 4219) from the one or more servers.

(B9) In some embodiments of the electronic device of any of B7-B8, providing the indication includes displaying (e.g., by display unit 4217 and/or display unit 4201) a user interface object on the touch-sensitive display that corresponds to the particular action associated with the trigger condition.

(B10) In some embodiments of the electronic device of B9, the user interface object includes a description of a particular action associated with the trigger condition.

(B11) In some embodiments of the electronic device of B10, the user interface object further includes an icon associated with the application.

(B12) In some embodiments of the electronic device of any of B9-B11, the processing unit is further configured to: a first gesture is detected (e.g., by detection unit 4223 and/or touch-sensitive surface unit 4203) at the user interface object. In response to detecting the first gesture: (i) display (e.g., via display unit 4217 and/or display unit 4201) the application on the touch-sensitive display, and (ii) perform (e.g., via execution unit 4225) the particular action associated with the trigger condition while the application is displayed.

(B13) In some embodiments of the electronic device of B12, the first gesture is a swipe gesture on the user interface object.

(B14) In some embodiments of the electronic device of any of B9-B12, the processing unit is further configured to: a second gesture is detected (e.g., by detection unit 4223 and/or touch-sensitive surface unit 4203) at the user interface object. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display, the processing unit is configured to: the particular action associated with the trigger condition is executed (e.g., by execution unit 4225).

(B15) In some embodiments of the electronic device of B14, the second gesture is a single tap of a predefined area of the object at the user interface.

(B16) In some embodiments of the electronic device of any of B9-B15, the user interface object is displayed in a predefined central portion of the lock screen.

(B17) In some embodiments of the electronic device of B7, providing the user with an indication that the particular action associated with the trigger condition is available includes performing (e.g., by execution unit 4225) the particular action.

(B18) In some embodiments of the electronic device of B9, the user interface object is an icon associated with the application, and the user interface object is displayed substantially in a corner of the lock screen on the touch-sensitive display.

(B19) In some embodiments of the electronic device of any of B7-B18, the processing unit is further configured to: an instruction to unlock the electronic device is received (e.g., by the receiving unit 4219) from the user. In response to receiving the instruction, the processing unit is configured to: a home screen of the electronic device is displayed (e.g., via display unit 4217 and/or display unit 4201) on the touch-sensitive display. The processing unit is further configured to: an indication is provided to the user (e.g., via providing unit 4215) on the home screen indicating that a particular action associated with the trigger condition is available.

(B20) In some embodiments of the electronic device of B19, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device, and (ii) a second portion displayed adjacent to the first portion for launching a second set of applications available on the electronic device. The second portion is displayed on all user interface pages included in the first portion, and providing the indication on the home screen includes displaying the indication on the second portion (e.g., via display unit 4217 and/or display unit 4201).

(B21) In some implementations of the electronic device of B20, the second set of applications is different from and smaller than the first set of applications.

(B22) In some embodiments of the electronic device of any of B7-B21, determining that the at least one trigger condition has been met comprises: it is determined (e.g., by determining unit 4227) that the electronic device has been coupled with a second device different from the electronic device.

(B23) In some embodiments of the electronic device of any of B7-B22, determining that the at least one trigger condition has been met comprises: it is determined (e.g., by determining unit 4227) that the electronic device has reached a corresponding address of a home location or work location associated with the user.

(B24) In some embodiments of the electronic device of B23, determining that the electronic device has reached a corresponding address of a home location or a work location associated with the user comprises: motion data from an accelerometer of the electronic device is monitored (e.g., by monitoring unit 4229) and a time for which the electronic device has not moved more than a threshold amount is determined (e.g., by determining unit 4227) based on the monitored motion data.

(B25) In some embodiments of the electronic device of any of B7-B24, the usage data further includes verbal instructions from the user that are provided to the virtual assistant application while continuing to execute the application. The at least one trigger condition is further based on a verbal instruction provided to the virtual assistant application.

(B26) In some embodiments of the electronic device of B25, the verbal instruction includes a request to create a reminder that corresponds to a current state of the application, the current state corresponding to a state of the application at the time the verbal instruction was provided.

(B27) In some embodiments of the electronic device of B26, the application state at which the verbal instructions are provided is selected from the group consisting of: a page displayed within the application when the verbal instructions are provided, content played within the application when the verbal instructions are provided, a notification displayed within the application when the verbal instructions are provided, and an active portion of the page displayed within the application when the verbal instructions are provided.

(B28) In some embodiments of the electronic device of B26, the verbal instructions include the term "this" in relation to the current state of the application.

(C1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured according to any of computing devices A-D of FIG. 1E) having a touch-sensitive display (touch screen 112, FIG. 1C). The method comprises the following steps: a search activation gesture from a user of the electronic device is detected on the touch-sensitive display. In response to detecting the search activation gesture, the method includes: displaying a search interface on a touch-sensitive display, the search interface comprising: (i) a search entry portion and (ii) a prediction portion displayed before any user input is received at the search entry portion. The prediction part is populated by one or more of: (a) at least one affordance for contacting a person of a plurality of previously contacted people, the person being automatically selected from the plurality of previously contacted people based at least in part on a current time, and (b) at least one affordance for performing a predicted action in an application of a plurality of applications available on an electronic device, the predicted action being automatically selected based at least in part on an application usage history associated with a user of the electronic device.

(C2) In some embodiments of the C1 method, the person is further selected based at least in part on location data corresponding to the electronic device.

(C3) In some embodiments of the method of any of C1-C2, the application usage history and the contact information for the person are retrieved from a memory of the electronic device.

(C4) In some embodiments of the method of any of C1-C2, the person's application usage history and contact information are retrieved from a memory located remotely from the electronic device.

(C5) In some embodiments of the methods of any of C1 through C4, the predictive part is further populated by executing at least one affordance of a predictive application that is automatically selected based at least in part on application usage history.

(C6) In some embodiments of the methods of any of C1-C5, the predicted portion is further populated with at least one affordance of a predicted category of places (or nearby places), and the predicted category of places is automatically selected based at least in part on one or more of current time and location data corresponding to the electronic device.

(C7) In some embodiments of the method of any one of C1 to C6, the method further comprises: user input is detected to scroll through the predicted portions. In response to detecting user input to scroll through the predicted portion, the method includes: the predictive part is scrolled according to the user input. In response to scrolling, the method includes: at least one affordance of a predicted news article is surfaced in the predictive portion (e.g., a predicted news article is an article predicted to be of interest to the user).

(C8) In some embodiments of the C7 method, the predicted news article is automatically selected based at least in part on location data corresponding to the electronic device.

(C9) In some embodiments of the method of any one of C1 to C8, the method further comprises: a selection of the at least one affordance to perform the predicted action within the application is detected. In response to detecting the selection, the method includes: the application is displayed on the touch-sensitive display and the predicted action is performed within the application.

(C10) In some embodiments of the method of any one of C3 to C4, the method further comprises: a selection of the at least one affordance to contact the person is detected. In response to detecting the selection, the method includes: the person is contacted using his contact information.

(C11) In some embodiments of the C5 method, the method further comprises: a selection to execute the at least one affordance of the prediction application is detected. In response to detecting the selection, the method includes: displaying the predicted application on the touch-sensitive display.

(C12) In some embodiments of the C6 method, the method further comprises: a selection of the at least one affordance of the category of predicted places is detected. In response to detecting the selection, the method further comprises: (i) receive data corresponding to at least one nearby location, and (ii) display the received data corresponding to the at least one nearby location on the touch-sensitive display.

(C13) In some embodiments of the C7 method, the method further comprises: a selection of the at least one affordance of a predicted news chapter is detected. In response to detecting the selection, the method includes: displaying the predicted news article on the touch-sensitive display.

(C14) In some embodiments of the methods of any of C1-C13, the search activation gesture is obtainable from at least two different user interfaces, and a first user interface of the at least two different user interfaces corresponds to displaying a respective home screen page of a sequence of home screen pages on the touch-sensitive display.

(C15) In some embodiments of the C14 method, when the corresponding home screen page is the first home screen page in the sequence of home screen pages, the search activation gesture includes one of: (i) a gesture that moves in a generally downward direction relative to the user of the electronic device, or (ii) a continuous gesture that moves in a generally left-to-right direction relative to the user and is generally perpendicular to the downward direction.

(C16) In some embodiments of the C15 method, the search activation gesture includes a continuous gesture that moves in a generally downward direction relative to the electronic device user when the corresponding home screen page is the second home screen page in the sequence of home screen pages.

(C17) In some embodiments of the C14 method, a second user interface of the at least two different user interfaces corresponds to displaying an application switching interface on the touch-sensitive display.

(C18) In some embodiments of the C17 method, the search activation gesture includes contacting a predefined search activation portion of the application switching interface on the touch-sensitive display.

(C19) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: a touch-sensitive display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of C1-C18.

(C20) In yet another aspect, an electronic device is provided and includes: a touch-sensitive display and means for performing any of the methods of C1-C18.

(C21) In another aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer readable storage medium is for storing executable instructions that, when executed by an electronic device with a touch-sensitive display, will cause the electronic device to perform the method described in any of C1-C18.

(C22) In another aspect, a graphical user interface on an electronic device with a touch-sensitive display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of C1-C18.

(C23) In another aspect, an electronic device is provided that includes a display unit (e.g., the display unit 4301 of fig. 43), a touch-sensitive surface unit (e.g., the touch-sensitive surface unit 4303 of fig. 43), and a processing unit (e.g., the processing unit 4305 of fig. 43). In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (i.e., computing devices a-D). For ease of illustration, fig. 43 shows the display unit 4301 and the touch-sensitive surface unit 4303 integrated with the electronic device 4300, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a display unit (e.g., the display unit 4309 of fig. 43), a detection unit (e.g., the detection unit 4307 of fig. 43), a retrieval unit (e.g., the retrieval unit 4311 of fig. 43), a fill unit (e.g., the fill unit 4313 of fig. 43), a scroll unit (e.g., the scroll unit 4315 of fig. 43), a presentation unit (e.g., the presentation unit 4317 of fig. 43), a selection unit (e.g., the selection unit 4319 of fig. 43), a contact unit (e.g., the contact unit 4321 of fig. 43), a reception unit (e.g., the reception unit 4323 of fig. 43), and an execution unit (e.g., the execution unit 4325 of fig. 43). The processing unit (or one or more components thereof, such as the units 4307-4225) is configured to: detect (e.g., by the detection unit 4307 and/or the touch-sensitive surface unit 4303) a search activation gesture from a user of the electronic device on the touch-sensitive display; in response to detecting the search activation gesture, a search interface is displayed (e.g., by display unit 4309 and/or display unit 4301) on the touch-sensitive display, the search interface including: (i) a search entry section; and (ii) a predicted portion displayed prior to receiving any user input at the search entry portion, the predicted portion populated by one or more of: (a) at least one affordance for contacting a person of a plurality of previously contacted people, automatically selecting (e.g., by selection unit 4319) the person from the plurality of previously contacted people based at least in part on a current time, and (b) at least one affordance for performing a predicted action in an application of a plurality of applications available on the electronic device, the predicted action automatically selected (e.g., by selection unit 4319) based at least in part on an application usage history associated with a user of the electronic device.

(C24) In some embodiments of the electronic device of C23, the person is further selected (e.g., by selection unit 4319) based at least in part on the location data corresponding to the electronic device.

(C25) In some embodiments of the electronic device of any of C23-C24, the application usage history and the contact information for the person are retrieved (e.g., by retrieving unit 4311) from a memory of the electronic device.

(C26) In some embodiments of the electronic device of any of C23-C24, the application usage history and the contact information for the person are retrieved (e.g., by retrieving unit 4311) from a server located remotely from the electronic device.

(C27) In some embodiments of the electronic device of any of C23-C26, the predicted portion is further populated (e.g., by populating unit 4313) by at least one affordance executing a predicted application that is automatically selected (e.g., by selecting unit 4319) based at least in part on application usage history.

(C28) In some embodiments of the electronic device of any of C23-C27, the predicted portion is further populated by at least one affordance of the predicted category of places (e.g., by populating unit 4313), and the predicted category of places is automatically selected (e.g., by selecting unit 4319) based at least in part on one or more of current time and location data corresponding to the electronic device.

(C29) In some embodiments of the electronic device of any of C23-C28, the processing unit is further configured to: user input is detected (e.g., by the detection unit 4307 and/or the touch-sensitive surface unit 4303) to scroll through the predicted portions. In response to detecting the user input to scroll the predicted portion, the processing unit is configured to: the predicted portion is scrolled (e.g., via scrolling unit 4319) according to the user input. In response to scrolling, the processing unit is configured to: at least one affordance of a predicted news article is surfaced (e.g., by surfacing unit 4317) in the predictive portion (e.g., the predicted news article is an article predicted to be of interest to the user).

(C30) In some embodiments of the electronic device of C7, the predicted news article is automatically selected (e.g., by selection unit 4319) based at least in part on the location data corresponding to the electronic device.

(C31) In some embodiments of the electronic device of any of C23-C30, the processing unit is further configured to: a selection of the at least one affordance to perform the predicted action within the application is detected (e.g., by the detection unit 4307 and/or the touch-sensitive surface unit 4303). In response to detecting the selection, the processing unit is configured to: the application is displayed (e.g., by display unit 4309) on a touch-sensitive display (e.g., display unit 4301) and the predicted action is performed (e.g., by execution unit 4325) within the application.

(C32) In some embodiments of the electronic device of any of C25-C26, the processing unit is further configured to: a selection of the at least one affordance to contact the person is detected (e.g., by the detection unit 4307 and/or the touch-sensitive surface unit 4303). In response to detecting the selection, the processing unit is configured to: the person is contacted (e.g., via contact element 4321) using his contact information.

(C33) In some embodiments of the electronic device of C27, the processing unit is further configured to: the selection of the at least one affordance to execute the prediction application is detected (e.g., by the detection unit 4307 and/or the touch-sensitive surface unit 4303). In response to detecting the selection, the processing unit is configured to: the predicted application is displayed (e.g., by display unit 4307) on a touch-sensitive display (e.g., display unit 4301).

(C34) In some embodiments of the electronic device of C28, the processing unit is further configured to: a selection of the at least one affordance for the predicted category of places is detected (e.g., by the detection unit 4307 and/or the touch-sensitive surface unit 4303). In response to detecting the selection, the processing unit is configured to: (i) receive (e.g., via receiving unit 4323) data corresponding to at least one nearby venue, and (ii) display (e.g., via display unit 4307) the received data corresponding to the at least one nearby venue on a touch-sensitive display (e.g., display unit 4301).

(C35) In some embodiments of the electronic device of C29, the processing unit is further configured to: a selection of the at least one affordance of the predictive news article is detected (e.g., by the detection unit 4307 and/or the touch-sensitive surface unit 4303). In response to detecting the selection, the processing unit is configured to: the predicted news article is displayed (e.g., by display unit 4307) on a touch-sensitive display (e.g., display unit 4301).

(C36) In some embodiments of the electronic device of any of C23-C35, the search activation gesture is obtainable from at least two different user interfaces, and a first user interface of the at least two different user interfaces corresponds to a respective home screen page that displays a sequence of home screen pages on the touch-sensitive display.

(C37) In some embodiments of the electronic device of C36, when the corresponding home screen page is the first home screen page in the sequence of home screen pages, the search activation gesture includes one of: (i) a gesture that moves in a generally downward direction relative to the user of the electronic device, or (ii) a continuous gesture that moves in a generally left-to-right direction relative to the user and is generally perpendicular to the downward direction.

(C38) In some embodiments of the electronic device of C37, the search activation gesture includes a continuous gesture that moves in a generally downward direction relative to the electronic device user when the corresponding home screen page is the second home screen page in the sequence of home screen pages.

(C39) In some implementations of the electronic device of C36, the second user interface of the at least two different user interfaces corresponds to displaying an application-switching interface on the touch-sensitive display.

(C40) In some embodiments of the electronic device of C39, the search activation gesture includes contacting a predefined search activation portion of the application switching interface on the touch-sensitive display.

Accordingly, electronic devices having a display, a touch-sensitive surface, and optionally one or more sensors for detecting intensity of contacts with the touch-sensitive surface are provided with faster, more efficient methods and interfaces for actively accessing applications and actively executing functions within applications, thereby improving the efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for accessing applications and functions associated therewith.

(D1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured in accordance with any of computing devices A-D of FIG. 1E) having a touch-sensitive surface (e.g., touch-sensitive surface 195, FIG. 1D) and a display (e.g., display 194, FIG. 1D). The method comprises the following steps: content associated with an application program being executed on an electronic device is displayed on a display. The method further comprises the following steps: a swipe gesture is detected through the touch-sensitive surface that, when detected, will cause the electronic device to enter a search mode different from the application. The method further comprises the following steps: in response to detecting the swipe gesture, a search mode is entered, the search mode including a search interface displayed on the display. In connection with entering a search mode, the method includes: at least one suggested search query is determined based at least in part on information associated with the content. Prior to receiving any user input at the search interface, the method includes: populating a displayed search interface with the at least one suggested search query. In this way, the device provides relevant suggestions to the user based on the application content viewed by the user, and the user need only select one of the suggestions without typing in any content, thereby eliminating the need for the user to remember and re-enter information in the search interface.

(D2) In some implementations of the D1 method, detecting the swipe gesture includes detecting the swipe gesture over at least a portion of the content currently displayed.

(D3) In some embodiments of the method of any one of D1 to D2, the method further comprises: detecting an input corresponding to a request to view a home screen of the electronic device prior to detecting the swipe gesture; and in response to detecting the input, ceasing to display the content associated with the application and displaying a corresponding page of the home screen of the electronic device. In some embodiments, the respective page is an initial page in a sequence of home screen pages, and the swipe gesture is detected while the initial page of the home screen is displayed on the display.

(D4) In some embodiments of the methods of any of D1-D3, the search interface is displayed as a semi-transparent overlay application.

(D5) In some embodiments of the method of any one of D1 to D4, the method further comprises: in accordance with a determination that the content includes textual content, the at least one suggested search query is determined based at least in part on the textual content.

(D6) In some embodiments of the D5 method, determining the at least one suggested search query based at least in part on textual content comprises: the textual content is analyzed to detect one or more predefined keywords for determining the at least one suggested search query.

(D7) In some embodiments of the methods of any one of D1-D6, determining the at least one suggested search query comprises determining a plurality of suggested search queries, and populating the search interface comprises populating the search interface with the plurality of suggested search queries.

(D8) In some embodiments of the method of any one of D1 to D7, the method further comprises: detecting, by the touch-sensitive surface, a new swipe gesture on the currently displayed new content; and in response to detecting the new swipe gesture, entering a search mode, the entering the search mode including displaying a search interface on the display; and in connection with entering the search mode and in accordance with a determination that the new content does not include textual content, populating a search interface with a suggested search query, the suggested search query based on a selected set of historical search queries from a user of the electronic device.

(D9) In some embodiments of the D8 method, the search interface displays the points of interest based on location information provided by a second application different from the application.

(D10) In some embodiments of the methods of any one of D8-D9, the search interface further comprises one or more suggested applications.

(D11) In some embodiments of the methods of any of D8-D10, the set of historical search queries is selected based at least in part on a frequency of recent search queries.

(D12) In some embodiments of the method of any one of D1 to D11, the method further comprises: in connection with entering the search mode, information associated with the content is obtained by using one or more auxiliary functions available on the electronic device.

(D13) In some embodiments of the D12 method, using the one or more auxiliary functions includes using the one or more auxiliary functions to generate information associated with the content by: (i) applying a natural language processing algorithm to textual content currently displayed in an application, and (ii) determining one or more keywords describing the content using data obtained from the natural language processing algorithm, and determining the at least one suggested search query based on the one or more keywords.

(D14) In some embodiments of the D13 method, determining one or more keywords describing the content further comprises: (i) retrieving metadata corresponding to non-textual content currently displayed in the application, and (ii) determining one or more keywords using the retrieved metadata in addition to data obtained from the natural language processing algorithm.

(D15) In some embodiments of the methods of any one of D1-D14, the search interface further comprises one or more trend queries.

(D16) In some embodiments of the D15 method, the search interface further includes one or more applications predicted to be of interest to the user of the electronic device.

(D17) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the touch-sensitive surface and the display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of D1-D16.

(D18) In yet another aspect, an electronic device is provided and includes: a touch-sensitive surface and a display, and means for performing any of the methods of D1-D16.

(D19) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of D1-D16.

(D20) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of D1-D16. In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of D1-D16.

(D21) In another aspect, an electronic device is provided that includes a display unit (e.g., the display unit 4401 of fig. 44), a touch-sensitive surface unit (e.g., the touch-sensitive surface unit 4403 of fig. 44), and a processing unit (e.g., the processing unit 4405 of fig. 44). The processing unit is coupled with the touch-sensitive surface unit and the display unit. In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (i.e., computing devices a-D). For ease of illustration, fig. 44 shows the display unit 4401 and the touch-sensitive surface unit 4403 integrated with the electronic device 4400, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated in a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a detection unit (e.g., the detection unit 4407 of fig. 44), a display unit (e.g., the display unit 4409 of fig. 44), a retrieval unit (e.g., the retrieval unit 4411 of fig. 44), a search pattern entry unit (e.g., the search pattern entry unit 4412 of fig. 44), a filling unit (e.g., the filling unit 4413 of fig. 44), an acquisition unit (e.g., the acquisition unit 4415 of fig. 44), a determination unit (e.g., the determination unit 4417 of fig. 44), and a selection unit (e.g., the selection unit 4419 of fig. 44). The processing unit (or one or more components thereof, such as units 1007-1029) is configured to: displaying (e.g., via the display unit 4407) content associated with an application that is executing on an electronic device on a display unit (e.g., the display unit 4407); detecting (e.g., via detection unit 4407) a swipe gesture through a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4403), which gesture, when detected, will cause the electronic device to enter a search mode different from the application; in response to detecting the swipe gesture, enter a search mode (e.g., via search mode entry unit 4412), the search mode including a search interface displayed on a display unit (e.g., display unit 4407); in connection with entering a search mode, determining (e.g., by determining unit 4417) at least one suggested search query based at least in part on information associated with the content; and populate (e.g., via population unit 4413) the displayed search interface with the at least one suggested search query prior to receiving any user input at the search interface.

(D22) In some embodiments of the electronic device of D21, detecting the swipe gesture includes detecting (e.g., by detecting element 4407) the swipe gesture over at least a portion of the content currently displayed.

(D23) In some embodiments of the electronic device of any of D21-D22, wherein the processing unit is further configured to: prior to detecting the swipe gesture, detecting (e.g., by detecting unit 4407) an input corresponding to a request to view the home screen of the electronic device; and in response to detecting (e.g., via detection unit 4407) the input, ceasing to display the content associated with the application and displaying a corresponding page of the home screen of the electronic device (e.g., via display unit 4409), wherein: the corresponding page is an initial page in a home screen page sequence; and a swipe gesture is detected (e.g., by detecting element 4407) while the initial page of the home screen is displayed on the display element.

(D24) In some embodiments of the electronic device of any of D21-D23, the search interface is displayed (e.g., by display unit 4409 and/or display unit 4401) as semi-transparent overlays the application.

(D25) In some embodiments of the electronic device of any of D21-D24, the processing unit is further configured to: in accordance with a determination that the content includes textual content, the at least one suggested search query is determined (e.g., by determination unit 4417) based at least in part on the textual content.

(D26) In some embodiments of the electronic device of D25, determining the at least one suggested search query based at least in part on the textual content comprises: the textual content is analyzed to detect (e.g., by detecting unit 4407) one or more predefined keywords for determining (e.g., by determining unit 4417) the at least one suggested search query.

(D27) In some embodiments of the electronic device of any of D21-D26, determining the at least one suggested search query comprises determining (e.g., by determining unit 4417) a plurality of suggested search queries, and populating the search interface comprises populating (e.g., by populating unit 4413) the search interface with the plurality of suggested search queries.

(D28) In some embodiments of the electronic device of any of D21-D27, the processing unit is further configured to: detecting (e.g., by detecting unit 4407), by a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4403), a new swipe gesture on new content currently displayed; and in response to detecting the new swipe gesture, enter a search mode (e.g., via search mode entry unit 4412), and entering the search mode includes displaying a search interface on the display unit (e.g., via display unit 4409); and in connection with entering the search mode and in accordance with a determination that the new content does not include textual content, populate (e.g., via population unit 4413) the search interface with suggested search queries, wherein the suggested search queries are based on a set of selected historical search queries from a user of the electronic device.

(D29) In some embodiments of the electronic device of D28, the search interface is displayed with a point of interest based on location information provided with a second application different from the application (e.g., display unit 4409).

(D30) In some embodiments of the electronic device of any of D28-D29, the search interface further comprises one or more suggested applications.

(D31) In some embodiments of the electronic device of any of D28-D30, the set of historical search queries is selected (e.g., by selection unit 4419) based at least in part on a frequency of recent search queries.

(D32) In some embodiments of the electronic device of any of D21-D31, the processing unit is further configured to: in connection with entering the search mode, information associated with the content is acquired (e.g., by acquisition unit 4415) using one or more auxiliary functions available on the electronic device.

(D33) In some embodiments of the electronic device of D32, using the one or more auxiliary functions includes using the one or more auxiliary functions to generate information associated with the content by: (i) apply a natural language processing algorithm to textual content currently displayed in the application, and (ii) determine (e.g., by determining unit 4417) one or more keywords describing the content using data obtained from the natural language processing algorithm (e.g., by obtaining unit 4415), wherein the at least one suggested search query is determined based on the one or more keywords (e.g., by determining unit 4417).

(D34) In some embodiments of the electronic device of D33, determining the one or more keywords describing the content further comprises: (i) retrieve (e.g., via retrieval unit 4411) metadata corresponding to non-textual content currently displayed in the application, and (ii) determine (e.g., via determination unit 4417) one or more keywords using the retrieved metadata in addition to data obtained from the natural language processing algorithm.

(D35) In some embodiments of the electronic device of any of D21-D34, the search interface further comprises one or more trend queries.

(D36) In some embodiments of the electronic device of D35, the search interface further includes one or more applications predicted to be of interest to the electronic device user.

(E1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured in accordance with any of computing devices A-D of FIG. 1E) having a touch-sensitive surface (e.g., touch-sensitive surface 195, FIG. 1D) and a display (e.g., display 194, FIG. 1D). The method comprises the following steps: a swipe gesture on the user interface is detected through the touch-sensitive surface, and when detected, the electronic device is caused to enter a search mode. The method further comprises the following steps: in response to detecting the swipe gesture, entering a search mode, and entering the search mode comprises: a search interface different from the user interface is populated with the first content item prior to receiving any user input within the search interface. In some embodiments, in accordance with a determination that the user interface includes content associated with an application, wherein the user interface is different from a home screen that includes selectable icons for invoking the application, populating the search interface with the first content item includes populating the search interface with the at least one suggested search query, wherein the search query is based at least in part on the content associated with the application; and in accordance with a determination that the user interface is associated with the page of the home screen, populating the search interface with the first content item includes populating the search interface with an affordance that includes a selectable description of at least one point of interest within a threshold distance of a current location of the electronic device.

(E2) In some embodiments of the E1 method, populating the search interface with the affordance includes displaying a search entry portion of the search interface on the touch-sensitive surface; and the method further comprises: detecting an input of a search entry portion; and in response to detecting input of the search entry portion, ceasing to display the affordance and displaying the at least one suggested search query within the search interface.

(E3) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the computer-readable medium may include a touch-sensitive surface, a display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of E1-E2.

(E4) In yet another aspect, an electronic device is provided and includes: a touch-sensitive surface, a display, and means for performing any of the methods of E1-E2.

(E5) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of E1-E2.

(E6) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any one of E1-E2. In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of E1-E2.

(E7) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 4501 of fig. 45), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4503 of fig. 45), and a processing unit (e.g., processing unit 4505 of fig. 45). The processing unit is coupled with the touch-sensitive surface unit and the display unit. In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (i.e., computing devices a-D). For ease of illustration, FIG. 45 shows display unit 4501 and touch-sensitive surface unit 4503 integrated with electronic device 4500, however, in some implementations, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated in a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a detection unit (e.g., the detection unit 4507 of fig. 45), a display unit (e.g., the display unit 4509 of fig. 45), a fill unit (e.g., the fill unit 4511 of fig. 45), and a search mode entry unit (e.g., the search mode entry unit 4513 of fig. 45). The processing unit (or one or more components thereof, such as units 4507-4513) is configured to: detecting (e.g., by detection unit 4507), by a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4503), a swipe gesture on the user interface, wherein when the swipe gesture is detected, the electronic device is to be brought into a search mode; and in response to detecting the swipe gesture, enter a search mode (e.g., enter unit 4513 via the search mode), wherein entering the search mode comprises: a search interface different from the user interface is populated (e.g., by populating unit 4511) with the first content item prior to receiving any user input within the search interface. In some embodiments, in accordance with a determination that the user interface includes content associated with an application, wherein the user interface is different from a home screen that includes selectable icons for invoking the application, populating the search interface with the first content item includes populating (e.g., via the populating unit 4511) the search interface with the at least one suggested search query, wherein the search query is based at least in part on the content associated with the application; and in accordance with a determination that the user interface is associated with the page of the home screen, populating the search interface with the first content item includes populating (e.g., by populating unit 4511) the search interface with affordances that include selectable descriptions of at least one point of interest within a threshold distance of a current location of the electronic device.

(E8) In some embodiments of the electronic device of E7, populating the search interface with affordances includes displaying (e.g., via display unit 4507 and/or display unit 4501) a search entry portion of the search interface; and the processing unit is further configured to: detect (e.g., by detection unit 4507) an input to the search entry portion; and in response to detecting an input at the search entry portion, ceasing to display (e.g., via display unit 4507 and/or display unit 4501) the affordance and display (e.g., via display unit 4507 and/or display unit 4501) the at least one suggested search query within the search interface.

(F1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, in accordance with any one of the configurations of computing devices A-D of FIG. 1E) having a position sensor, a touch-sensitive surface (e.g., touch-sensitive surface 195, FIG. 1D), and a display (e.g., display 194, FIG. 1D). The method comprises the following steps: automatically determining, without an instruction from a user, that a user of an electronic device is located in a vehicle that is parked at a geographic location; when it is determined that the user has left the vehicle at the geographic location, it is determined whether the positioning information retrieved from the location sensor for identifying the geographic location meets an accuracy criterion. The method further comprises the following steps: upon determining that the positioning information does not meet the accuracy criteria, a prompt is provided to the user to enter information regarding the geographic location. The method further comprises: in response to providing the prompt, information regarding the geographic location is received from the user and stored as vehicle location information.

(F2) In some embodiments of the method of claim F1, the method further comprising: upon determining that the positioning information meets the accuracy criteria, the positioning information is automatically stored as vehicle location information without instructions from the user.

(F3) In some embodiments of the method of claim F2, the method further comprising: in accordance with a determination that the user is heading to the geographic location, a user interface object including vehicle location information is displayed.

(F4) In some embodiments of the method of claim F3, the user interface object is a map object that includes a user current location identifier and a separate identifier of the geographic location.

(F5) In some embodiments of the method of claim F4, the user interface object is displayed on a lock screen of the electronic device.

(F6) In some embodiments of the method of claim F4, the user interface object is displayed in response to a swipe gesture that causes the electronic device to enter a search mode.

(F7) In some embodiments of the method of claim F6, in response to receiving the swipe gesture, performing a determination of whether the user is heading for the geographic location.

(F8) In some embodiments of the method of any of F1 through F7, the prompt is an audio prompt provided by a virtual assistant usable with the electronic device, receiving information from the user includes receiving a verbal description from the user identifying the geographic location, displaying the user interface object includes displaying a selectable affordance that, when selected, will cause the device to play back the verbal description.

(F9) In some embodiments of the method of any of F1 through F7, the prompting is displayed on a display of the electronic device, the receiving information from the user includes receiving a textual description from the user identifying the geographic location, and the displaying the user interface object includes displaying the textual description from the user.

(F10) In some embodiments of the methods of any of F1 through F7, determining whether the user is heading for the geographic location comprises: the electronic device is determined to be moving to the geographic location using the new positioning information received from the location sensor.

(F11) In some embodiments of the method of F10, determining whether the user is heading for the geographic location comprises: (i) determine that the electronic device is staying at another geographic location for more than a threshold period of time, and (ii) determine that the new positioning information indicates that the electronic device is leaving the other geographic location and moving toward the geographic location.

(F12) In some embodiments of the methods of any of F1 through F11, determining that the user is in a vehicle parked at the geographic location comprises: (i) determining that the user is located within the vehicle by determining that the electronic device is traveling above a threshold speed (ii) determining that the vehicle has stayed at the geographic location by one or more of: (a) determining that the electronic device has stayed at the geographic location for more than a certain threshold period of time, (b) determining that a communication link between the electronic device and the vehicle has been broken, and (c) determining that the geographic location corresponds to a certain location within the parking lot.

(F13) In some embodiments of the method of claim F12, determining that the vehicle has stayed at the geographic location comprises determining that the electronic device has stayed at the geographic location for more than a certain threshold period of time.

(F14) In some embodiments of the method of any of F12-F13, determining that the vehicle has been parked at the geographic location includes determining that a communication link between the electronic device and the vehicle has been broken.

(F15) In some embodiments of the methods of any of F12 through F14, determining that the vehicle has been parked at the geographic location comprises determining that the geographic location corresponds to a location within a parking lot.

(F16) In some implementations of the method of any of F1-F15, the accuracy criteria include a criterion that is met when a GPS reading associated with the positioning information is accurate above a threshold level of accuracy.

(F17) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the apparatus may include a touch-sensitive surface, a display, a position sensor, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of F1-F16.

(F18) In yet another aspect, an electronic device is provided and includes: a touch-sensitive surface, a display, a position sensor, and means for performing any of the methods of F1-F16.

(F19) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface, a display, and a position sensor, will cause the electronic device to perform the method described in any of F1-F16.

(F20) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface, a display, and a position sensor is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of F1-F16. In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface, a display, and a position sensor is provided. The information processing apparatus includes: means for performing the method described in any of F1 to F16.

(F21) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 4601 of fig. 46), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4603 of fig. 46), a position sensor unit (e.g., position sensor unit 4607 of fig. 46), and a processing unit (e.g., processing unit 4605 of fig. 46). The processing unit is coupled with the touch-sensitive surface unit, the display unit, and the position sensor unit. In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (i.e., computing devices a-D). For ease of illustration, fig. 46 shows the display unit 4601 and the touch-sensitive surface unit 4603 integrated with the electronic device 4600, however, in some embodiments, one or both of these units communicate with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated in a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a display unit (e.g., the display unit 4609 of fig. 46), a retrieval unit (e.g., the retrieval unit 4611 of fig. 46), a determination unit (e.g., the determination unit 4613 of fig. 46), a storage unit (e.g., the storage unit 4615 of fig. 46), a recognition unit (e.g., the recognition unit 4617 of fig. 46), a selection unit (e.g., the selection unit 4619 of fig. 46), a reception unit (e.g., the reception unit 4621 of fig. 46), a provision unit (e.g., the provision unit 4623 of fig. 46), and a playback unit (e.g., the playback unit 4625 of fig. 46). The processing unit (or one or more components thereof, such as the units 4607 to 4625) is configured to: automatically determine (e.g., via determination unit 4613) that the user of the electronic device is located within a vehicle that is parked at a geographic location without instructions from the user; upon determining that the user has left the vehicle at the geographic location, determining (e.g., via determining unit 4613) whether positioning information retrieved (e.g., via retrieving unit 4611) from a location sensor unit (e.g., location sensor unit 4607) for identifying (e.g., via identifying unit 4617) the geographic location satisfies an accuracy criterion; upon determining (e.g., via determining unit 4613)) that the positioning information does not meet the accuracy criteria, providing (e.g., via providing unit 4623) a prompt to the user to enter information regarding the geographic location; and, in response to providing the prompt, receive (e.g., via receiving unit 4621) information from the user regarding the geographic location and store (e.g., via storage unit 4615) the information as vehicle location information.

(F22) In some embodiments of the electronic device of F21, the processing unit is further configured to: upon determining that the positioning information meets the accuracy criteria, the positioning information is automatically stored (e.g., via storage unit 4615) as vehicle location information without instruction from the user.

(F23) In some embodiments of the electronic device of F22, the processing unit is further configured to: in accordance with a determination that the user is heading to the geographic location, a user interface object including vehicle location information is displayed (e.g., via the display unit 4609 in conjunction with the display unit 4601).

(F24) In some embodiments of the electronic device of F23, the user interface object is a map object that includes a user current location identifier and a separate identifier of a geographic location.

(F25) In some embodiments of the electronic device of F24, the user interface object is displayed (e.g., by the display unit 4609 in conjunction with the display unit 4601) on a lock screen of the electronic device.

(F26) In some embodiments of the electronic device of F24, the user interface object is displayed (e.g., by display unit 4609 in conjunction with display unit 4601) in response to a swipe gesture that causes the electronic device to enter a search mode.

(F27) In some embodiments of the electronic device of F26, in response to receiving the swipe gesture, a determination is made as to whether the user is heading for the geographic location.

(F28) In some embodiments of the electronic device of any of F21-F27, the prompt is an audio prompt provided by a virtual assistant usable with the electronic device, receiving information from the user includes receiving (e.g., via the receiving unit 4621) a verbal description from the user identifying the geographic location, displaying the user interface object includes displaying (e.g., via the display unit 4609 in conjunction with the display unit 4601) a selectable affordance that, when selected (e.g., via the selecting unit 4619), will cause the device to playback (e.g., via the playback unit 4625) the verbal description.

(F29) In some embodiments of the electronic device of any of F21-F27, the prompt is displayed (e.g., by the display unit 4609 in conjunction with the display unit 4601) on a display of the electronic device, receiving information from the user includes receiving (e.g., by the receiving unit 4621) a text description from the user identifying the geographic location, and displaying the user interface object includes displaying the text description from the user.

(F30) In some embodiments of the electronic device of any of F21-F27, determining whether the user is heading for the geographic location comprises: the new positioning information received (e.g., by the receiving unit 4621) from the location sensor unit (e.g., the location sensor unit 4607) is used to determine (e.g., by the determining unit 4613) that the electronic device is moving to the geographic location.

(F31) In some embodiments of the electronic device of F30, determining whether the user is heading for the geographic location comprises: (i) determine (e.g., via determination unit 4613) that the electronic device is staying at another geographic location for more than a threshold period of time, and (ii) determine (e.g., via determination unit 4613) that the new positioning information indicates that the electronic device is leaving and moving to the other geographic location.

(F32) In some embodiments of the electronic device of any of F21-F31, determining that the user is in a vehicle parked at the geographic location comprises: (i) determine that the user is located within the vehicle by determining (e.g., by determining unit 4613) that the electronic device is traveling above a threshold speed (ii) that the vehicle has stayed at the geographic location by one or more of: (a) determine (e.g., via determination unit 4613) that the electronic device has stayed at the geographic location for more than a certain threshold period of time, (b) determine (e.g., via determination unit 4613) that a communication link between the electronic device and the vehicle has been disconnected, and (c) determine (e.g., via determination unit 4613) that the geographic location corresponds to a certain location within the parking lot.

(F33) In some embodiments of the electronic device of F32, determining that the vehicle has stayed at the geographic location includes determining (e.g., via determining unit 4613) that the electronic device has stayed at the geographic location for more than a certain threshold period of time.

(F34) In some embodiments of the electronic device of any of F32-F33, determining that the vehicle has stayed at the geographic location includes determining (e.g., by determining unit 4613) that a communication link between the electronic device and the vehicle has been disconnected.

(F35) In some embodiments of the electronic device of any of F32-F34, determining that the vehicle has stayed at the geographic location includes determining (e.g., by determining unit 4613) that the geographic location corresponds to a certain location within the parking lot.

(F36) In some implementations of the electronic device of any of F21-F35, the accuracy criterion includes a criterion that is met when a GPS reading accuracy associated with the positioning information is above a threshold level of accuracy.

(G1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, in accordance with any one of the configurations of computing devices A-D of FIG. 1E) having a position sensor, a touch-sensitive surface (e.g., touch-sensitive surface 195, FIG. 1D), and a display (e.g., display 194, FIG. 1D). The method comprises the following steps: a geographic location of the electronic device is monitored using the location sensor. The method further comprises the following steps: determining that the electronic device is within a threshold distance of a predetermined type of point of interest based on the monitored geographic location. The method further comprises the following steps: in accordance with a determination that the electronic device is within a threshold distance of the point of interest: identifying at least one activity that is currently popular at the point of interest and retrieving information about the point of interest, including retrieving information about the at least one activity that is currently popular at the point of interest. The method further comprises the following steps: detecting a first input through the touch-sensitive surface, the first input, when detected, causing the electronic device to enter a search mode; and in response to detecting the first input, entering a search mode, wherein entering the search mode comprises, prior to receiving any user input at the search interface, presenting, via the display, an affordance comprising: (i) information about the at least one activity, and (ii) an indication that the at least one activity has been determined to be currently popular at the point of interest.

(G2) In some embodiments of the method of claim G1, the method comprising: detecting a second input; and in response to detecting the second input, updating the affordance to include available information about a current activity at a second point of interest that is different from the point of interest, the point of interest being within a threshold distance of the electronic device.

(G3) In some embodiments of the methods of any of G1 to G2, the affordance further includes a selectable point of interest category, and the method further includes: detecting selection of a respective selectable category; and in response to detecting the selection, updating the affordance to include information about additional points of interest, wherein the additional points of interest are within a second threshold distance of the device.

(G4) In some embodiments of the methods of any of G1-G3, the point of interest is an amusement park and the retrieved information includes a current wait time for a ride of the amusement park.

(G5) In some embodiments of the method of claim G4, the retrieved information includes information about a wait time for a ride that is within a predefined distance of the electronic device.

(G6) In some embodiments of the method of any of G1-G3, the point of interest is a restaurant and the retrieved information includes information about a popular menu item in the restaurant.

(G7) In some embodiments of the method of claim G6, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

(G8) In some embodiments of the methods of any of G1-G3, the point of interest is a movie theater, and the retrieved information includes information about show times for movie theaters.

(G9) In some embodiments of the method of claim G8, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

(G10) In some embodiments of the methods of any of G1-G9, the affordance is available in response to a swipe in a generally horizontal direction on an initial page of a home screen of the electronic device after unlocking the electronic device.

(G11) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the computer-readable medium may include a touch-sensitive surface, a display, a position sensor, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of G1-G10.

(G12) In yet another aspect, an electronic device is provided and includes: a touch sensitive surface, a display, a position sensor, and means for performing any of the methods of G1-G10.

(G13) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface, a display, and a position sensor, will cause the electronic device to perform the method described in any of G1-G10.

(G14) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface, a display, and a position sensor is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of G1 to G10. In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface, a display, and a position sensor is provided. The information processing apparatus includes: means for performing the method described in any of G1 to G10.

(G15) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 4701 of fig. 47), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4703 of fig. 47), a position sensor unit 4707, and a processing unit (e.g., processing unit 4705 of fig. 47). The processing unit is coupled with the touch-sensitive surface unit, the display unit, and the position sensor unit. In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (i.e., computing devices a-D). For ease of illustration, fig. 47 shows display unit 4701 and touch-sensitive surface unit 4703 integrated with electronic device 4700, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the touch-sensitive surface unit and the display unit are integrated in a single touch-sensitive display unit (also referred to herein as a touch-sensitive display). The processing unit includes a detection unit (e.g., the detection unit 4709 of fig. 47), a display unit (e.g., the display unit 4711 of fig. 47), a retrieval unit (e.g., the retrieval unit 4713 of fig. 47), a determination unit (e.g., the determination unit 4715 of fig. 47), an identification unit (e.g., the identification unit 4717 of fig. 47), an unlocking unit (e.g., the unlocking unit 4719 of fig. 47), and a search pattern entering unit (e.g., the search pattern entering unit 4721 of fig. 47). The processing unit (or one or more components thereof, such as units 4709-4721) is configured to: without receiving any instruction from the user of the electronic device: monitoring a geographic location of the electronic device using a location sensor unit (e.g., location sensor unit 4707); determining (e.g., by the determining unit 4715) that the electronic device is within a threshold distance of a predetermined type of point of interest based on the monitored geographic location; in accordance with a determination that the electronic device is within a threshold distance of the point of interest: identifying (e.g., by the identifying means 4717) at least one activity currently prevalent at the point of interest; retrieving (e.g., by retrieving unit 4713) information about the point of interest, including retrieving information about at least one activity that is currently popular at the point of interest; detecting (e.g., via the detection unit 4709) a first input via a touch-sensitive surface unit (e.g., the touch-sensitive surface unit 4703), the first input, when detected, causing the electronic device to enter a search mode; and in response to detecting the first input, entering a search mode (e.g., via search mode entry unit 4721), and entering the search mode includes, prior to receiving any user input at the search interface, presenting (e.g., via display unit 4711) an affordance via a display unit (e.g., display unit 4701) including: (i) information about the at least one activity, and (ii) an indication that the at least one activity has been determined to be an activity that is currently popular at the point of interest.

(G16) In some embodiments of the electronic device of G15, the processing unit is further configured to: detecting (e.g., by detecting unit 4709) a second input; and in response to detecting the second input, updating (e.g., via display unit 4711) the affordance to include available information about a current activity at a second point of interest that is different from the point of interest, and the point of interest is within a threshold distance of the electronic device.

(G17) In some embodiments of the electronic device of any of G15-G16, the affordance further includes a selectable point of interest category, and the processing unit is further configured to: detecting (e.g., by detecting unit 4709) a selection of a respective selectable category; and in response to detecting the selection, updating (e.g., via display unit 4711) the affordance to include information about additional points of interest that are within a second threshold distance of the device.

(G18) In some embodiments of the electronic device of any of G15-G17, the point of interest is an amusement park and the retrieved information includes a current wait time for a ride of the amusement park.

(G19) In some embodiments of the electronic device of G18, the retrieved information includes information about a wait time for a ride that is within a predefined distance of the electronic device.

(G20) In some embodiments of the electronic device of any of G15-G17, the point of interest is a restaurant and the retrieved information includes information regarding a popular menu item in the restaurant.

(G21) In some embodiments of the electronic device of G20, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

(G22) In some embodiments of the electronic device of any of G15-G17, the point of interest is a movie theater and the retrieved information includes information about the show time of the movie theater.

(G23) In some embodiments of the electronic device of G22, the retrieved information is information retrieved from a social network associated with the user of the electronic device.

(G24) In some embodiments of the electronic device of any of G15-G23, upon unlocking (e.g., by unlocking unit 4719) the electronic device, the affordance is available in response to a swipe in a generally horizontal direction on an initial page of a home screen of the electronic device.

(H1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured according to any of computing devices A-D of FIG. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of FIG. 1C). The method comprises the following steps: at least a portion of the voice communication (e.g., a real-time telephone call of 10 seconds or less or a recorded voice mailbox) is received, the portion of the voice communication including voice provided by a remote user of a remote device, the remote user being different from a user of the electronic device. The method further comprises the following steps: the content item is extracted based at least in part on speech provided by a remote user of the remote device. The method further comprises the following steps: it is determined whether the content item is currently available on the electronic device. In accordance with a determination that the content item is not currently available on the electronic device, the method includes: (i) identify an application associated with the content item, and (ii) display a selectable depiction of the content item on the display. In response to detecting selection of the alternative description, the method includes: the content item is stored for presentation by the identified application (e.g., the content item is stored so that it can be presented by the identified application). In this way, the user is able to store content items mentioned or discussed in the voice communication without having to remember all the details of the discussion and then enter them to create the appropriate content item. Instead, the electronic device is able to detect and extract content items based on speech describing various respective content items, and then provide selectable descriptions of the content items that are selectable by the user for storing the respective content items on the electronic device.

(H2) In some embodiments of the method of H1, the content item is some new event.

(H3) In some embodiments of the method of H1, the content item is a new event detail for an event currently associated with a calendar application on the electronic device.

(H4) In some embodiments of the method of H1, the content item is a new contact.

(H5) In some embodiments of the method of H1, the content item is new contact information for an existing contact associated with a phone application on the electronic device.

(H6) In some embodiments of the method of any one of H1-H5, the voice communication is a real-time telephone call.

(H7) In some embodiments of the methods of any of H1-H5, the voice communication is a real-time FaceTime call.

(H8) In some embodiments of the method of any of H1-H5, the voice communication is a recorded voicemail.

(H9) In some embodiments of the methods of any of H1-H8, displaying the selectable description includes displaying the selectable description within a user interface, wherein the user interface includes recent calls made using a telephone application. In this way, the user can easily and conveniently access the extracted content items (e.g., the content items extracted during the corresponding voice communication) directly through the user interface including the most recent call.

(H10) In some embodiments of the method of H9, the optional description is shown with the following indication: the content item is associated with a voice communication.

(H11) In some embodiments of the method of H9, the detecting the selection comprises: the selection is received while displaying the user interface including the most recent call.

(H12) In some embodiments of the method of any one of H1 to H11, the method further comprises: in conjunction with displaying the selectable description of the content item, feedback (e.g., tactile feedback generated by the electronic device or a user interface object presented on the second device such that the user does not have to remove the phone from his ear during the phone call) that the content item has been detected is provided to the electronic device user. In this way, the user is provided with a concise indication that the content item was detected/retrieved during the voice communication, and can then decide whether to store the content item.

(H13) In some embodiments of the method of H12, providing feedback includes sending information related to detection of the content item to another electronic device (e.g., a laptop, a television monitor, a smart watch, etc.) in proximity to the electronic device. In this way, the user does not have to interrupt the voice communication, but can still view details related to the detected/extracted content item on another device.

(H14) In some embodiments of the method of any one of H1 to H13, the method further comprises: determining that the voice communication includes information about the first physical location (e.g., an address mentioned during the telephone call or a restaurant name discussed during the telephone call, etc., more details are provided below). The method further comprises the following steps: an input is detected, and in response to detecting the input, an application capable of accepting location data is opened and populated with information about the first physical location. Thus, in addition to detecting and extracting events and contact information, the electronic device is also capable of extracting location information discussed during voice communications and providing the location information to the user in an appropriate application (e.g., to protect the user from being distracted by particular location details discussed on a memory phone, particularly new details that the user may be unfamiliar with, the device extracting these location details and displaying for use by the user).

(H15) In some embodiments of the method of H14, the application is a mapping application, and populating the mapping application with information related to the first physical location includes: a map displayed within the mapping application is populated with a location identifier corresponding to the first physical location. In this way, the user can conveniently use the newly extracted location details to go to a new destination, see how far away a particular location is, and use other functionality provided by the mapping application.

(H16) In some embodiments of the method of any one of H1 to H13, the method further comprises: it is determined that the voice communication includes information related to the first physical location. The method further comprises the following steps: an input (e.g., a search activation gesture, such as a swipe gesture discussed in detail below) is detected, and in response to detecting the input, a search interface is populated with information about the first physical location. In this way, in addition to (or instead of) providing location information for a particular application to the user, the electronic device can also provide location information for use in a search interface (e.g., for searching for relevant points of interest or for searching for more details about the first physical location, such as a phone number, menu, etc.).

(H17) In some embodiments of the method of any of H1-H16, extracting the content item includes analyzing the portion of the voice communication to detect a predetermined type of content, and performing the analysis while outputting the voice communication through an audio system in communication with the electronic device (e.g., analyzing the voice communication in real-time while outputting the voice communication to a user of the electronic device).

(H18) In some embodiments of the method of H17, analyzing the voice communication comprises: (i) converting speech provided by a remote user of a remote device to text; (ii) applying a natural language processing algorithm to the text to determine whether the text includes one or more predefined keywords; and (iii) in accordance with a determination that the text includes the respective predefined keyword, determining that the voice communication includes speech describing the content item.

(H19) In some embodiments of the methods of any of H1-H18, receiving at least the portion of the voice communication includes receiving an indication from a user of the electronic device that the portion of the voice communication should be analyzed.

(H20) In some embodiments of the method of H19, the indication corresponds to a selection of a hardware button (e.g., a user selecting a hardware button while the audio system is outputting a voice communication indicating a number of seconds (e.g., the previous 10, 15, or 20 seconds) of the predetermined voice communication that should be analyzed). In some embodiments, the button may also be a button presented on the electronic device display for selection by the user (e.g., a button that displays "tap here to analyze the voice communication to obtain new content" during the voice communication).

(H21) In some embodiments of the method of H19, the indication corresponds to a command from a user of the electronic device, including the word "hey, Siri". Thus, the user can easily instruct the electronic device to begin analyzing the portion of the voice communication to detect content items (such as events, contact information, and information about the physical location) discussed in the voice communication.

(H22) In some embodiments of the method of any one of H1 to H21, the method further comprises: a second portion of the voice communication is received, the second portion including voice provided by a remote user of the remote device and voice provided by a user of the electronic device (e.g., the voice communication is a real-time telephone call, and the second portion includes discussions between the user and the remote user). The method further comprises the following steps: the second content item is extracted based at least in part on speech provided by a remote user of the remote device and speech provided by a user of the electronic device. In accordance with a determination that the second content item is not currently available on the electronic device, the method includes: (i) identify a second application associated with the second content item, and (ii) display a second selectable description of the second content item on the display. In response to detecting selection of the second selectable description, the method includes: the second content item is stored for presentation by the identified second application.

(H23) In some embodiments of the method of H22, the selectable description and the second selectable description are displayed in a user interface, wherein the user interface includes recent calls made using a telephone application. In this way, a single interface is provided to the user, which conveniently includes content items detected over multiple voice communications (e.g., multiple phone calls, voice mailboxes, or both).

(H24) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the touch-sensitive surface, the display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of H1-H23.

(H25) In yet another aspect, an electronic device is provided, the electronic device including: a touch-sensitive surface, a display, and means for performing any of the methods of H1-H23.

(H26) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of H1-H23.

(H27) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of H1-H23.

(H28) In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of H1-H23.

(H29) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 4801 of fig. 48), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4803 of fig. 48), and a processing unit (e.g., processing unit 4805 of fig. 48). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 48 shows the display unit 4801 and touch-sensitive surface unit 4803 integrated with the electronic device 4800, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a voice communication receiving unit (e.g., the voice communication receiving unit 4807 of fig. 48), a content item extracting unit (e.g., the content item extracting unit 4809 of fig. 48), an availability determining unit (e.g., the availability determining unit 4811 of fig. 48), an application identifying unit (e.g., the application identifying unit 4813 of fig. 48), a display unit (e.g., the display unit 4815 of fig. 48), a content item storage unit (e.g., the content item storage unit 4817 of fig. 48), a feedback providing unit (e.g., the feedback providing unit 4819 of fig. 48), an input detecting unit (e.g., the input detecting unit 4821 of fig. 48), an application opening unit (e.g., the application opening unit 4823 of fig. 48), a filling unit (e.g., the filling unit 4825 of fig. 48), and a voice communication analyzing unit (e.g., the voice communication analyzing unit 4827 of fig. 48). The processing unit (or one or more components thereof, such as units 4807-4827) is configured to: at least a portion of the voice communication is received (e.g., by voice communication receiving unit 4807), the portion of the voice communication including voice provided by a remote user of a remote device different from the user of the electronic device. The processing unit is further configured to: the content item is extracted based at least in part on the speech provided by the remote user of the remote device (e.g., by content item extraction unit 4809), and it is determined whether the content item is currently available on the electronic device (e.g., by availability determination unit 4811). In accordance with a determination that the content item is not currently available on the electronic device, the processing unit is further configured to: (i) identify an application associated with the content item (e.g., by application identification unit 4813), and (ii) display an optional description of the content item on the display (e.g., by display unit 4815 and/or display unit 4801). In response to detecting selection of the selectable description (e.g., by the input detection unit 4821 and/or the touch-sensitive surface unit 4803), the processing unit is configured to: the content item is stored for rendering by the identified application (e.g., by content item storage unit 4817).

(H30) In some embodiments of the electronic device of H29, the processing unit (or one or more components thereof, such as units 4907 to 4927) is further configured to perform the method described in any of H2 to H23.

(I1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured according to any of computing devices A-D of FIG. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of FIG. 1C). The method comprises the following steps: at least a portion of the voice communication is received, the portion of the voice communication (e.g., a real-time telephone call, a recorded voice mailbox) including voice provided by a remote user of the remote device, the remote user being different from a user of the electronic device. The method further comprises the following steps: determining the voice communication includes recognizing voice at the first physical location. In response to determining that the voice communication includes voice recognition of the physical location, the method includes: providing an indication (e.g., providing tactile feedback to a user, displaying a user interface object with information about the physical location, or sending information about the physical location to a nearby device for display on the nearby device), indicating that information about the physical location has been detected. The method further comprises: the input is detected through the touch sensitive surface. In response to detecting the input, the method includes: (i) opening an application program for receiving the geographic position data; and (ii) populating the application with information about the physical location. In this way, the user is able to store information about the physical location mentioned or discussed in the voice communication without having to remember all of the details discussed and then enter them in the appropriate application. Instead, the electronic device can detect and extract information about a physical location based on speech describing the physical location (e.g., a description of a restaurant, a driving route for a particular physical location, etc.), and then provide an indication that information related to the corresponding physical location has been detected.

(I2) In some embodiments of the method of I1, the voice communication is a real-time telephone call.

(I3) In some embodiments of the method of I1, the voice communication is a real-time FaceTime call.

(I4) In some embodiments of the method of I1, the voice communication is a recorded voicemail.

(I5) In some embodiments of the methods of any of I1-I4, providing the indication includes displaying a selectable description of the physical location within the user interface, wherein the user interface includes recent calls made using the phone application.

(I6) In some embodiments of the method of I5, the optional description indicates that the content item is associated with a voice communication.

(I7) In some embodiments of the methods of any one of I5 to I6, detecting the input comprises: an input is detected on the selectable depiction while the user interface including the most recent call is displayed.

(I8) In some embodiments of the methods of any of I1-I7, providing the indication comprises providing tactile feedback to a user of the electronic device.

(I9) In some embodiments of the methods of any of I1-I8, providing the indication comprises sending information about the physical location to another electronic device in proximity to the electronic device.

(I10) In some embodiments of the method of any of I1-I9, determining that the voice communication includes voice describing a physical location includes: the portion of the voice communication is analyzed to detect information about the physical location, and the analysis is performed while the voice communication is output through an audio system in communication with the electronic device.

(I11) In some embodiments of the methods of any of I1-I10, receiving at least the portion of the voice communication includes receiving instructions from a user of the electronic device that the portion of the voice communication should be analyzed.

(I12) In some embodiments of the method of I11, the instruction corresponds to a selection of a hardware button. In some embodiments, the button may also be a button presented on the electronic device display for selection by the user (e.g., a button that displays "tap here to analyze the voice communication for new content" during the voice communication).

(I13) In some embodiments of the method of I11, the instruction corresponds to a command from a user of the electronic device, including the word "hey, Siri".

(I14) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the touch-sensitive surface, the display, the one or more processors, and the memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of I1-I13.

(I15) In yet another aspect, an electronic device is provided, the electronic device including: a touch-sensitive surface, a display, and means for performing any of the methods of I1-I13.

(I16) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of I1-I13.

(I17) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of I1-I13.

(I18) In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of I1-I13.

(I19) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 4901 of fig. 49), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4903 of fig. 49), and a processing unit (e.g., processing unit 4905 of fig. 49). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 49 shows display unit 4901 and touch-sensitive surface unit 4903 integrated with electronic device 4900, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a voice communication receiving unit (e.g., a voice communication receiving unit 4907 of fig. 49), a content item extracting unit (e.g., a content item extracting unit 4909 of fig. 49), an indication providing unit (e.g., an indication providing unit 4911 of fig. 49), an input detecting unit (e.g., an input detecting unit 4913 of fig. 49), an application opening unit (e.g., an application opening unit 4915 of fig. 49), an application filling unit (e.g., an application filling unit 4917 of fig. 49), a feedback providing unit (e.g., a feedback providing unit 4919 of fig. 49), and a voice communication analyzing unit (e.g., a voice communication analyzing unit 4921 of fig. 49). The processing unit (or one or more components thereof, such as units 4907 to 4921) is configured to: at least a portion of the voice communication is received (e.g., by the voice communication receiving unit 4907), the portion of the voice communication including voice provided by a remote user of a remote device different from the user of the electronic device. The processing unit is further configured to: determining the voice communication includes recognizing the voice of the physical location (e.g., by the content item extraction unit 4909). In response to determining that the voice communication includes voice identifying the physical location, the processing unit is configured to: an indication is provided (e.g., by the content item extraction unit 4909) that information regarding the physical location has been detected. The processing unit is further configured to: an input is detected by the touch-sensitive surface unit (e.g., by the input detection unit 4911). In response to detecting the input, the processing unit is configured to: the application that accepts the geographic location data is (i) opened (e.g., by application opening unit 4913), and (ii) populated with information about the physical location (e.g., by application populating unit 4915).

(I20) In some embodiments of the electronic device of I19, the processing unit (or one or more components thereof, e.g., units 4907 to 4921) is further configured to perform the method described in any of I2 to I13.

(J1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured according to any of computing devices A-D of FIG. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of FIG. 1C). The method comprises the following steps: the text entry field and the copy of the session are presented in the instant messaging application on the display. The method further comprises the following steps: when the instant messaging application is presented on the display, it is determined that the next possible input from the user of the electronic device is information about the physical location. The method further comprises the following steps: content associated with the text entry field and the session copy is analyzed, and a suggested physical location is determined based at least in part on a portion of the analyzed content. The method further comprises: presenting a selectable user interface element identifying the suggested physical location within the instant messaging application on the display, and receiving a selection of the selectable user interface element. In response to receiving the selection, the method includes: a representation of the suggested physical location is presented in the text entry field. In this way, the user of the electronic device may be conveniently provided with the desired content without having to type in any content and without having to search for the content (e.g., the user simply selects a selectable user interface element to enter their current address without having to access the map application to determine their exact location and then switch back to the instant messaging application and provide explicit input to send the location information).

(J2) In some embodiments of the method of J1, the instant messaging application includes a virtual keyboard and the selectable user interface element is displayed in a suggestion portion adjacent to and above the virtual keyboard.

(J2) In some embodiments of the methods of any one of J1-J2, determining that the next possible input from the user of the electronic device is information about the physical location comprises: content associated with the text entry field and the transcript of the session is processed to detect that the transcript of the session includes a question about the current location of the user. In this way, the user is provided with a suggested physical location that is directly related to the discussion in the session copy (e.g., in response to the second user's question "where do you are", the user is presented with a user interface object that, when selected, will cause the device to send information to the second user about the user's current location).

(J4) In some embodiments of the method of J3, processing the content includes applying a natural language processing algorithm to detect one or more predefined keywords that constitute the question (e.g., "where do you.

(J5) In some embodiments of the methods of any one of J3-J4, the question is included in a message received from a second user different from the user.

(J6) In some embodiments of the methods of any one of J1-J5, determining that the next possible input from the user of the electronic device is information about the physical location comprises: a text input portion of an instant messaging application is monitored for a typed input received from a user.

(J7) In some embodiments of the method of any one of J1 to J6, the method further comprises: in accordance with a determination that the user is typing and has not selected the selectable user interface element, ceasing to present the selectable user interface element. Thus, if it can be determined that the user is not interested in the selection object, the device will not continue to present the selectable user interface object.

(J8) In some embodiments of the method of any one of J1 to J7, the method further comprises: in accordance with a determination that the user has provided additional input indicating that the user will not select the selectable user interface element, presentation of the selectable user interface element is ceased. Thus, if it can be determined that the user is not interested in the selection object, the device will not continue to present the selectable user interface object.

(J9) In some embodiments of the methods of any one of J1-J5, the representation of the proposed physical location includes information identifying a current geographic location of the electronic device.

(J10) In some embodiments of the methods of any one of J1 to J9, the representation of the proposed physical location is an address.

(J11) In some embodiments of the methods of any one of J1-J9, the suggested physical location is a map object that includes an identifier of the suggested physical location.

(J12) In some embodiments of the methods of any one of J1-J11, the suggested physical location corresponds to a location that the user recently viewed in an application other than the instant messaging application.

(J13) In some embodiments of the methods of any one of J1-J12, the instant messaging application is an email application.

(J14) In some embodiments of the methods of any one of J1-J12, the instant messaging application is a text messaging application.

(J15) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the touch-sensitive surface, the display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of J1-J14.

(J16) In yet another aspect, an electronic device is provided, the electronic device including: a touch-sensitive surface, a display, and means for performing any of the methods of J1-J14.

(J17) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of J1-J14.

(J18) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any one of J1 to J14.

(J19) In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: apparatus for performing the method described in any of J1 to J14.

(J20) In another aspect, an electronic device is provided that includes a display unit (e.g., display unit 5001 of fig. 50), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 5003 of fig. 50), and a processing unit (e.g., processing unit 5005 of fig. 50). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 50 shows display unit 5001 and touch-sensitive surface unit 5003 integrated with electronic device 5000, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing units include a presentation unit (e.g., the presentation unit 5007 of fig. 50), a next input determination unit (e.g., the next input determination unit 5009 of fig. 50), a content analysis unit (e.g., the content analysis unit 5011 of fig. 50), a selection reception unit (e.g., the selection reception unit 5013 of fig. 50), a key input monitoring unit (e.g., the key input monitoring unit 5015 of fig. 50), and a presentation stop unit (e.g., the presentation stop unit 5017 of fig. 50). The processing unit (or one or more components thereof, e.g., units 5007-5017) is configured to: the text entry field and the copy of the session are presented in the instant messaging application on the display (e.g., via presentation unit 5007 and/or display unit 5001). When presenting the instant messaging application on the display, the processing unit is further configured to: determining that the next possible input from the electronic device user is information about the physical location (e.g., by the next input determination unit 5009) the processing unit is further configured to: analyzing content associated with the text entry field and the session copy, determining a suggested physical location based at least in part on a portion of the analyzed content (e.g., by the content analysis unit 5011); presenting a selectable user interface element identifying the suggested physical location within the instant messaging application on the display (e.g., via the presenting unit 5007); receiving a selection of a selectable user interface element (e.g., by selecting the receiving unit 5013 and/or the touch-sensitive surface unit 5003); and in response to receiving the selection, a representation of the suggested physical location is presented in the text entry field (e.g., by presentation element 5007).

(J21) In some embodiments of the electronic device of J20, the processing unit (or one or more components thereof, e.g., units 5007-5017) is further configured to perform the method described in any one of J2-J14.

(K1) According to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of FIG. 1A, configured according to any of computing devices A-D of FIG. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of FIG. 1C). The method comprises the following steps: while the first application is displayed, information identifying a first physical location viewed by the user in the first application is obtained (e.g., restaurants that the user has searched in an application that allows searching for local merchants). The method further comprises the following steps: the first application is exited, and after exiting the first application, a request is received from the user to open a second application different from the first application. In some embodiments, the request is received without receiving any input in the first application (e.g., the request does not include clicking on a link or button within the first application). In response to receiving the request and in accordance with a determination that the second application is capable of accepting the geographic location information, the method includes: presenting the second application, and presenting the second application includes populating the second application with information based at least in part on the first physical location identifying information. In this way, the user does not need to manually transfer information between two different applications. Instead, the device intelligently determines that the second application is able to accept the geographic location information and then populates the second application directly with information about the physical location viewed in the first application (e.g., populating a map object in the second application to include an identifier of the physical location).

(K2) In some embodiments of the method of K1, receiving the request to open the second application comprises: after exiting the first application, an input is detected on an affordance of a second application. In other words, the request does not correspond to clicking on a link within the first application, rather, the user explicitly and directly requests to open the second application, and the device then decides to populate the second application with information about the previously viewed physical location (previously viewed in a different first application) so that the user can further study or investigate the previously viewed physical location in the second application.

(K3) In some embodiments of the method of K2, the affordance for the second application is an icon displayed within a home screen of the electronic device. In some embodiments, the home screen is a system level component of the operating system, including icons for invoking applications available on the electronic device.

(K4) In some embodiments of the method of K2, detecting the input comprises: (i) detecting a double-click on the physical Home button, (ii) displaying an application-switching user interface in response to detecting the double-click, and (iii) detecting selection of an affordance from within the application-switching user interface.

(K5) In some embodiments of the methods of any of K1-K4, populating the second application includes displaying a user interface object that includes information that is based at least in part on the first physical location identifying information.

(K6) In some embodiments of the method of K5, the user interface object includes a textual description informing the user that the first physical location was recently viewed in the first application.

(K7) In some embodiments of the method of K6, the user interface object is a map displayed in the second application, and populating the second application includes populating the map to include the identifier of the first physical location.

(K8) In some embodiments of the method of any of K6-K7, the second application presents a virtual keyboard and the user interface object is displayed above the virtual keyboard.

(K9) In some embodiments of the methods of any of K6 through K8, obtaining the information includes obtaining information about the second physical location, and displaying the user interface object includes displaying the user interface object with information about the second physical location.

(K10) In some embodiments of the methods of any one of K1 to K9, determining that the second application is capable of accepting geographic location information comprises one or more of: (i) determining that the second application includes an input-receiving field capable of accepting and processing the geographic location data; (ii) determining that the second application is capable of displaying the geographic location information on the map; (iii) determining that the second application is capable of using the geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using the geographic location information.

(K11) In some embodiments of the method of K10, determining that the second application is capable of accepting geographic location information comprises: it is determined that the second application includes an input-receiving field capable of accepting and processing the geographic location data, and that the input-receiving field is a search box that allows searching within a map displayed in the second application.

(K12) In some embodiments of the method of any one of K1 to K11, the method further comprises: in response to receiving the request, it is determined whether the second application is associated with the first application based on the application usage history of the user (e.g., a threshold number of times after the first application is opened).

(K13) In some embodiments of the method of K12, the method further comprises: the second application is provided with access to the information identifying the first physical location prior to rendering the second application, and the second application cannot access the information identifying the first physical location prior to providing the access. In this way, the second application can receive information about the actions performed by the user in the first application, thereby providing the user with a way to use that information in the second application (e.g., search for more information about the first physical location, or use the first physical location for some service available through the second application, such as windmilling).

(K14) In another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the computer-readable medium may include a touch-sensitive surface, a display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of K1-K13.

(K15) In yet another aspect, an electronic device is provided, the electronic device including: a touch-sensitive surface, a display, and means for performing any of the methods of K1-K13.

(K16) In another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of K1-K13.

(K17) In another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of K1-K13.

(K18) In another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of K1 to K13.

(K19) In another aspect, an electronic device is provided that includes a display unit (e.g., the display unit 5101 of fig. 51), a touch-sensitive surface unit (e.g., the touch-sensitive surface unit 5103 of fig. 51), and a processing unit (e.g., the processing unit 5105 of fig. 51). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 51 shows the display unit 5101 and touch-sensitive surface unit 5103 integrated with the electronic device 5100, however, in some embodiments, while these units remain physically separate from the electronic device, one or both of these units are in communication with the electronic device. The processing units include an information acquisition unit (e.g., the information acquisition unit 5107 of fig. 51), an application exit unit (e.g., the application exit unit 5109 of fig. 51), a request reception unit (e.g., the request reception unit 5111 of fig. 51), an application capability determination unit (e.g., the application capability determination unit 5113 of fig. 51), an application presentation unit (e.g., the application presentation unit 5115 of fig. 51), an application filling unit (e.g., the application filling unit 5117 of fig. 51), an input detection unit (e.g., the input detection unit 5119 of fig. 51), an application switching user interface display unit (e.g., the application switching user interface display unit 5121 of fig. 51), an application association determination unit (e.g., the application association determination unit 5123 of fig. 51), and an access providing unit (e.g., the access providing unit 5125 of fig. 51). The processing unit (or one or more components thereof, e.g., units 5107-5125) is configured to: while the first application is displayed, information identifying the first physical location viewed by the user in the first application is acquired (e.g., by information acquisition unit 5107). The processing unit is further configured to: the first application is exited (e.g., by the application exit unit 5109), and upon exiting the first application, a request is received from the user to open a second application different from the first application (e.g., by the request receiving unit 5111). In response to receiving the request and in accordance with a determination that the second application is able to accept the geographic location information (e.g., a determination processed or performed by the application capability determination unit 5113), the second application is presented (e.g., by the application presentation unit 5115), and presenting the second application includes populating the second application with information based at least in part on the first physical location identification information (e.g., by the application population unit 5117).

(K20) In some embodiments of the electronic device of K19, the processing unit (or one or more components thereof, such as units 5107-5125) is further configured to perform the method described in any of K2-K13.

(L1) according to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of fig. 1A, any one of the configurations of computing devices a-D according to fig. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of fig. 1C). The method comprises the following steps: information identifying a first physical location viewed by a user in a first application is obtained and a first input is detected. In response to detecting the first input, the method includes: (i) identifying a second application that is capable of accepting geographic location information, and (ii) presenting an affordance different from the first application on at least a portion of the display and suggesting opening of the second application having information about the first physical location. The method further comprises the following steps: a second input is detected at the affordance. In response to detecting the second input at the affordance: (i) open the second application, and (ii) populate the second application to include information based at least in part on the first physical location identifying information.

In contrast to the operations associated with K1 described above, the operations associated with L1 do not receive a request from the user to open the second application before providing a suggestion to the user to open the second application with the first physical location-related information. Thus, by providing the operations associated with K1 and L1 described above (and using a combination of some processing steps from each of these methods), the electronic device can provide an efficient user experience, allowing a user to predictively use location data before or after opening an application that is capable of accepting geographic location information. Additionally, for L1, determining that the second application is capable of accepting geographic location information is performed prior to opening the second application, and in this manner, in embodiments of L1, where the input corresponds to a request to open an application-switching user interface, the application-switching user interface displays the request requesting opening of an application (e.g., the second application) having the first physical location-related information only if the application is known to be able to accept location data.

(L2) in some embodiments of the method of L1, the first input corresponds to a request to open an application-switching user interface (e.g., the first input is a double-click on a physical Home button of the electronic device)

(L3) in some embodiments of the method of L2, the affordance is presented within an application-switching user interface.

(L4) in some embodiments of the method of L3, presenting the affordance includes: presenting, in conjunction with presenting the affordance, a representation of an application executing on the electronic device within the application-switching user interface (e.g., an application content snapshot of the application); and presenting the affordance in an area of the display that is below the application representation.

(L5) in some embodiments of the method of L1, the first input corresponds to a request to open a Home screen of the electronic device (e.g., the first input is a single tap on a physical Home button of the electronic device).

(L6) in some embodiments of the method of L5, the affordance is presented over a portion of the home screen.

(L7) in some embodiments of the method of any one of L1-L6, the suggestion includes a textual description that is specific to a type associated with the second application.

(L8) in some embodiments of the method of any one of L1 to L7, populating the second application includes displaying a user interface object that includes information that is based at least in part on the first physical location identifying information.

(L9) in some embodiments of the method of L8, the user interface object includes a textual description informing the user that the first physical location was recently viewed in the first application.

(L10) in some embodiments of the method of L9, the user interface object is a map displayed in a second application, and populating the second application includes populating the map to include an identifier of the first physical location.

(L11) in some embodiments of the method of any one of L9 to L10, the second application presents a virtual keyboard and the user interface object is displayed above the virtual keyboard.

(L12) in some embodiments of the method of any one of L1 to L11, identifying that the second application is capable of accepting geographic location information includes one or more of: (i) determining that the second application includes an input-receiving field capable of accepting and processing the geographic location data; (ii) determining that the second application is capable of displaying the geographic location information on the map; (iii) determining that the second application is capable of using the geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using the geographic location information.

(L13) in some embodiments of the method of L12, identifying that the second application is able to accept geographic location information comprises: it is determined that the second application includes an input-receiving field capable of accepting and processing the geographic location data, and that the input-receiving field is a search box that allows searching within a map displayed in the second application.

(L14) in another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the touch-sensitive surface, the display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of L1-L13.

(L15) in yet another aspect, there is provided an electronic device comprising: a touch-sensitive surface, a display, and means for performing any of the methods of L1-L13.

(L16) in another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of L1-L13.

(L17) in another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of L1 to L13.

(L18) in another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of L1-L13.

(L19) in another aspect, an electronic device is provided that includes a display unit (e.g., display unit 5201 of fig. 52), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 5203 of fig. 52), and a processing unit (e.g., processing unit 5205 of fig. 52). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 52 shows the display unit 5201 and the touch-sensitive surface unit 5203 integrated with the electronic device 5200, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing units include an information acquisition unit (e.g., the information acquisition unit 5207 of fig. 52), an input detection unit (e.g., the input detection unit 5209 of fig. 52), an application identification unit (e.g., the application identification unit 5211 of fig. 52), an affordance presentation unit (e.g., the affordance presentation unit 5213 of fig. 52), an application opening unit (e.g., the application opening unit 5215 of fig. 52), an application filling unit (e.g., the application filling unit 5217 of fig. 52), an application switching user interface presentation unit (e.g., the application switching user interface presentation unit 5219 of fig. 52), and an application capability determination unit (e.g., the application capability determination unit 5221 of fig. 52). The processing unit (or one or more components thereof, such as the units 5207 to 5221) is configured to: information for identifying a first physical location viewed by a user in a first application is acquired (e.g., by the information acquisition unit 5207) and a first input is detected (e.g., by the input detection unit 5209). In response to detecting the first input, the processing unit is configured to: (i) identify a second application that is capable of accepting geographic location information (e.g., via the application identification unit 5209), and (ii) present an affordance different from the first application on at least a portion of the display and suggest opening the second application having information about the first physical location (e.g., via the affordance presentation unit 5213). The processing unit is further configured to: a second input is detected (e.g., by input detection unit 5209) at the affordance. In response to detecting the second input at the affordance, the processing unit is configured to: (i) open the second application (e.g., by the application opening unit 5215), and (ii) populate the second application to include information based at least in part on the first physical location identifying information (e.g., by the application populating unit 5217).

(L20) in some embodiments of the electronic device of L19, the processing unit (or one or more components thereof, such as the units 5207 to 5221) is further configured to perform the method described in any one of L2 to L13.

(M1) according to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of fig. 1A, any one of the configurations of computing devices a-D according to fig. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of fig. 1C). The method comprises the following steps: information identifying a first physical location is obtained, wherein the first physical location is a location viewed by a user in a first application executing on an electronic device. The method further comprises the following steps: it is determined that the user has entered the vehicle. In response to determining that the user has entered the vehicle, providing a prompt to the user prompting the user to use the first physical location as a destination for route guidance. In response to providing the prompt, the method includes: an instruction is received from a user instructing the user to use the first physical location as a destination for route guidance. The method further comprises the following steps: route guidance to a first physical location is facilitated. In this way, it may be convenient to provide suggestions to the user that the user plan a route to the destination based on the physical location they previously viewed in the application on the electronic device.

(M2) in some embodiments of the method of M1, the method further comprises: detecting that the electronic device has received the message, including detecting that the message includes information identifying the second physical location; and in response to the detection, providing a new prompt to the user prompting the user to use the second physical location as a new destination for route guidance. In this way, the user may also dynamically add waypoints or add new destinations for route guidance based on information included in the message (e.g., short message, email, voice mailbox, etc.).

(M3) in some embodiments of the method of M2, the method further comprises: route guidance to the second physical location is facilitated in response to receiving an instruction from the user to use the second physical location as a new destination.

(M4) in some embodiments of the method of any one of M2-M3, detecting that the message contains information identifying the second physical location includes: the detection is performed when a virtual assistant available on the electronic device is reading a message to the user through an audio system in communication with the electronic device. In this way, when the user is listening to a message read by the audio system (e.g., via a personal assistant available via the electronic device), the electronic device detects information identifying the second physical location and uses the detected information to propose the second physical location as a new destination. Thus, the user does not have to divert his attention from the road while driving, but is still able to dynamically adjust the route guidance settings and the destination.

(M5) in some embodiments of the method of any one of M2-M4, determining that the user has entered the vehicle includes detecting that the electronic device has established a communication link with the vehicle.

(M6) in some embodiments of the method of any one of M2-M5, facilitating route guidance comprises providing route guidance through a display of an electronic device.

(M7) in some embodiments of the method of any one of M2-M5, facilitating route guidance includes sending information identifying the first physical location to the vehicle.

(M8) in some embodiments of the method of any one of M2-M7, facilitating route guidance includes providing route guidance through an audio system in communication with the electronic device (e.g., a car speaker or an internal speaker of the device itself).

(M9) in another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the computer-readable medium may include, but is not limited to, a touch-sensitive surface, a display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of M1-M8.

(M10) in yet another aspect, there is provided an electronic device comprising: a touch-sensitive surface, a display, and means for performing any of the methods of M1-M8.

(M11) in another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of M1-M8.

(M12) in another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of M1-M8.

(M13) in another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of M1-M8.

(M14) in another aspect, an electronic device is provided that includes a display unit (e.g., display unit 5301 of fig. 53), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 5303 of fig. 53), and a processing unit (e.g., processing unit 5305 of fig. 53). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 53 shows the display unit 5301 and the touch-sensitive surface unit 5303 integrated with the electronic device 5300, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing units include an information acquisition unit (e.g., the information acquisition unit 5307 of fig. 53), a vehicle entry determination unit (e.g., the vehicle entry determination unit 5309 of fig. 53), a presentation providing unit (e.g., the presentation providing unit 5311 of fig. 53), an instruction receiving unit (e.g., the instruction receiving unit 5313 of fig. 53), a route guidance promoting unit (e.g., the route guidance promoting unit 5315 of fig. 53), and a message detection unit (e.g., the message detection unit 5317 of fig. 53). The processing unit (or one or more components thereof, such as units 5307-5317) is configured to: information identifying a first physical location viewed by a user in the first application executing on the electronic device is obtained (e.g., by information obtaining unit 5307). The processing unit is further configured to: it is determined that the user has entered the vehicle (e.g., by the vehicle entry determination unit 5309). In response to determining that the user has entered the vehicle, the processing unit is configured to: a prompt is provided to the user prompting the user to use the first physical location as a destination for route guidance (e.g., by prompt providing unit 5311). In response to providing the prompt, an instruction is received from the user instructing the user to use the first physical location as a destination for route guidance (e.g., by instruction receiving unit 5313). The processing unit is further configured to: route guidance to the first physical location is facilitated (e.g., by route guidance facilitation unit 5307).

(M15) in some embodiments of the electronic device of M14, the processing unit (or one or more components thereof, such as units 5307-5317) is further configured to perform the method described in any one of M2-M8.

(N1) according to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of fig. 1A, any one of the configurations of computing devices a-D according to fig. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of fig. 1C). The method comprises the following steps: content is presented in a first application. The method further comprises the following steps: a request from a user to open a second application different from the first application is received, the second application including an input-receiving field. In response to receiving the request, the method includes: a second application having an input-receiving field is presented. Prior to receiving any user input at the input receiving field, the method includes: a selectable user interface object is provided to allow a user to paste at least a portion of the content into the input-receiving field. In response to detecting selection of the selectable user interface object, the method includes: pasting the portion of content into the input-receiving field. In this way, the user is provided with an active paste action in the second application-based content previously viewed in the first action (e.g., this enables the user to paste content into the second application without having to re-open the first application, perform an explicit copy operation, re-open the second application, and then explicitly request that the copied content be pasted into the second application).

(N2) according to some embodiments of the method of N1, before providing the selectable user interface object, the method includes: the input-receiving field is identified as a field that is capable of accepting the portion of the content.

(N3) according to some embodiments of the method of N2, in response to detecting selection of the input-receiving field, identifying the input-receiving field as a field capable of accepting the portion of the content.

(N4) according to some embodiments of the method of any one of N1 to N3, the portion of content corresponds to an image.

(N5) the portion of content corresponds to textual content according to some embodiments of the method of any one of N1 to N3.

(N6) the portion of content corresponds to text content and an image according to some embodiments of the method of any one of N1 to N3.

(N7) according to some embodiments of the method of any one of N1-N6, the first application is a web browsing application and the second application is a messaging application.

(N8) according to some embodiments of the method of any one of N1-N6, the first application is a photo browsing application and the second application is a messaging application.

(N9) some embodiments of the method according to any one of N1 to N8, the method comprising: receiving a request to copy at least the portion of the content prior to receiving a request to open the second application.

(N10) according to some embodiments of the method of any one of N1-N9, the selectable user interface object is displayed with an indication that the portion of content was recently viewed in the first application. In this way, the user is provided with a clear indication as to why the paste suggestion was made.

(N11) in another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the touch-sensitive surface, the display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of N1-N10.

(N12) in yet another aspect, there is provided an electronic device comprising: a touch-sensitive surface, a display, and means for performing any of the methods of N1-N10.

(N13) in another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of N1-N10.

(N14) in another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of N1 to N10.

(N15) in another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of N1 to N10.

(N16) in another aspect, an electronic device is provided that includes a display unit (e.g., display unit 5401 of fig. 54), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 5403 of fig. 54), and a processing unit (e.g., processing unit 5405 of fig. 54). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 54 shows the display unit 5401 and touch-sensitive surface unit 5403 integrated with the electronic device 5400, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a presentation unit (e.g., the presentation unit 5407 of fig. 54), a request reception unit (e.g., the request reception unit 5409 of fig. 54), a user interface object providing unit (e.g., the user interface object providing unit 5411 of fig. 54), an active pasting unit (e.g., the active pasting unit 5413 of fig. 54), and a capability determination unit (e.g., the capability determination unit 5415 of fig. 54). The processing unit (or one or more components thereof, such as units 5407-5415) is configured to: presenting content in the first application (e.g., via presentation unit 5407 and/or display unit 5401); receiving, from a user, a request to open a second application different from the first application (e.g., via the request-receiving element and/or the touch-sensitive surface element 5403), the second application including an input-receiving field; in response to receiving the request, presenting the second application with the input-receiving field (e.g., via presentation unit 5407 and/or display unit 5401); prior to receiving any user input at the input-receiving field, providing a selectable user interface object to allow a user to paste at least a portion of the content into the input-receiving field (e.g., via the user interface object providing unit 5411 and/or the display unit 5401); and paste the portion of content into the input-receiving field (e.g., via the active paste unit 5413) in response to detecting selection of the selectable user interface object.

(N17) in some embodiments of the electronic device of N16, the processing unit (or one or more components thereof, such as units 5407-5415) is further configured to perform the method described in any of N1-N10.

(O1) according to some embodiments, the method is performed on an electronic device (e.g., portable multifunction device 100 of fig. 1A, any one of the configurations of computing devices a-D according to fig. 1E) having a touch-sensitive surface and a display (in some embodiments, the touch-sensitive surface and the display are integrated together, as shown in touch screen 112 of fig. 1C). The method comprises the following steps: text content associated with the application is presented on the display. The method further comprises the following steps: determining that a portion of the textual content is relevant to: (i) a location, (ii) a contact, or (iii) an event. Upon determining that the portion of the textual content is associated with a location, the method includes: location information is obtained from a location sensor on the electronic device and the obtained location information is prepared for display as a predicted content item. Upon determining that the portion of the textual content is relevant to the contact, the method includes: a search is conducted on the electronic device, contact information related to the portion of the textual content is searched, and information associated with at least one contact retrieved by the search is prepared for display as a predicted content item. Upon determining that the portion of the textual content is relevant to the event, the method includes: a new search is conducted on the electronic device, event information related to the portion of the textual content is searched, and information based at least in part on at least one event (retrieved by the new search) is prepared for display as a predicted content item. The method further comprises the following steps: displaying, within an application, an affordance including a predicted content item; detecting selection of an affordance through a touch-sensitive surface; and in response to detecting the selection, displaying information associated with the predicted content item on a display adjacent to the textual content. In this way, the user may be conveniently provided with predicted content items that may be used to complete a presentation (or in response to other users, for example, questions posed in an instant messaging application), without having to type in any content, and without having to browse information available on the electronic device to find the desired information. For example, the electronic device provides a phone number, a current location, scheduling functionality for new events, details associated with existing events, all without requiring any explicit request or additional work by the user, thereby saving time while still ensuring that the required information is effectively provided to the user.

(O2) according to some embodiments of the method of O1, the portion of the textual content corresponds to textual content that was recently presented in the application.

(O3) according to some embodiments of the method of any one of O1-O2, the application is an instant messaging application and the portion of the text content is a question received in the instant messaging application from a remote user of a remote device other than the electronic device.

(O4) according to some embodiments of the method of any one of O1 to O2, the portion of the textual content is input provided by an input-receiving field within the application by a user of the electronic device.

(O5) according to some embodiments of the method of O1, the portion of the textual content is identified in response to a user input selecting a user interface object that includes the portion of the textual content.

(O6) according to some embodiments of the method of O5, the application is an instant messaging application and the user interface object is a message bubble in a conversation displayed within the instant messaging application.

(O7) according to some embodiments of the method of any one of O5 to O6, the method further comprises: detecting selection of a second user interface object; in response to detecting the selection, (i) ceasing to display the affordance with the predicted content item and determining that the textual content associated with the second user interface object relates to a location, contact, or event; and, in accordance with the determination, displaying the new predicted content item within the application. In this way, the user can easily return to the instant messaging conversation to select a previously received message and still be provided with the appropriate predicted content item.

(O8) the affordance is displayed in an input-receiving field within the application adjacent to the virtual keyboard, according to some embodiments of the method of any one of O1-O7.

(O9) according to some embodiments of the method of any one of O8, the input reception field is a field displaying a typing input received on a virtual keyboard.

(O10) according to some embodiments of the methods of any one of O1 to O9, the determining comprises: the textual content is parsed as it is received by the application to detect stored patterns that are known to be related to contacts, events, and/or locations.

(O11) in another aspect, an electronic device is provided. In some embodiments, the electronic device comprises: the computer-readable medium may include, but is not limited to, a touch-sensitive surface, a display, one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method described in any of O1-O10.

(O12) in yet another aspect, there is provided an electronic device comprising: a touch-sensitive surface, a display, and means for performing any of the methods of O1-O10.

(O13) in another aspect, a non-transitory computer-readable storage medium is provided. A non-transitory computer readable storage medium is used to store executable instructions that, when executed by an electronic device with a touch-sensitive surface and a display, will cause the electronic device to perform the method described in any of O1-O10.

(O14) in another aspect, a graphical user interface on an electronic device with a touch-sensitive surface and a display is provided. In some embodiments, the graphical user interface comprises a user interface displayed according to the method described in any of O1 to O10.

(O15) in another aspect, an information processing apparatus for use in an electronic device that includes a touch-sensitive surface and a display is provided. The information processing apparatus includes: means for performing the method described in any of O1 to O10.

(O16) in another aspect, an electronic device is provided that includes a display unit (e.g., display unit 5501 of fig. 55), a touch-sensitive surface unit (e.g., touch-sensitive surface unit 5503 of fig. 55), and a processing unit (e.g., processing unit 5505 of fig. 55). In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 55 shows display unit 5501 and touch-sensitive surface unit 5503 integrated with electronic device 5500, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a presentation unit (e.g., presentation unit 5507 of fig. 55), a determination unit (e.g., determination unit 5509 of fig. 55), an acquisition unit (e.g., acquisition unit 5511 of fig. 55), a search execution unit (e.g., search execution unit 5513 of fig. 55), an information preparation unit (e.g., information preparation unit 5515 of fig. 55), an affordance display unit (e.g., affordance display unit 5517 of fig. 55), and a detection unit (e.g., detection unit 5519 of fig. 55). The processing unit (or one or more components thereof, such as units 5507-5519) is configured to: present textual content associated with the application on the display (e.g., via presentation unit 5507 and/or display unit 5501); determining that a portion of the textual content is relevant to: (i) location, (ii) contact, or (iii) event (e.g., by determining unit 5509); upon determining that the portion of the text content is location-related, location information is acquired from a location sensor on the electronic device (e.g., by acquisition unit 5511), and the acquired location information is prepared for display as a predicted content item (e.g., by information preparation unit 5515); upon determining that the portion of the text content is related to a contact, a search is conducted on the electronic device, contact information related to the portion of the text content is searched (e.g., by search execution unit 5513), and information associated with at least one contact retrieved by the search is prepared for display as a predicted content item (e.g., by information preparation unit 5515); upon determining that the portion of the textual content is related to an event, performing a new search on the electronic device, searching for event information related to the portion of the textual content (e.g., by search execution unit 5513), and preparing information based at least in part on at least one event retrieved by the search for display as a predicted content item (e.g., by information preparation unit 5515); displaying an affordance including a predicted content item in an application (e.g., via affordance display unit 5517 and/or display unit 5501); detecting selection of an affordance through the touch-sensitive surface (e.g., through detection unit 5519); and in response to detecting the selection, information associated with the predicted content item is displayed on the display adjacent to the textual content (e.g., by presentation unit 5507 and/or display unit 5501).

(O17) in some embodiments of the electronic device of O16, the processing unit (or one or more components thereof, such as the units 5507-5519) is further configured to perform the method described in any one of O1-O10.

As mentioned above (and described in greater detail below), one aspect of the present technology is to collect and use data from a variety of sources (e.g., based on voice provided during voice communications) to improve the delivery of content to a user that may be of interest to the user. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or may be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, home addresses, or any other identifying information.

The present disclosure recognizes that the use of such personal information data in the present technology may be useful to benefit the user. For example, the personal information data may be used to deliver target content that is of greater interest to the user. Thus, the use of such personal information data enables planned control of delivered content. In addition, the present disclosure contemplates other uses for which personal information data is beneficial to a user.

The present disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with established privacy policies and/or privacy practices. In particular, such entities should enforce and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining privacy and security of personal information data. For example, personal information from a user should be collected for legitimate and legitimate uses by an entity and not shared or sold outside of these legitimate uses. In addition, such collection should only be done after the user has informed consent. Furthermore, such entities should take any required steps to secure and protect access to such personal information data, and to ensure that others who have access to the personal information data comply with their privacy policies and procedures. Additionally, such entities may subject themselves to third party evaluations to prove compliance with widely accepted privacy policies and practices.

Regardless of the foregoing, the present disclosure also contemplates embodiments in which a user selectively prevents use or access to personal information data. That is, the present disclosure contemplates that hardware elements and/or software elements may be provided to prevent or block access to such personal information data. For example, in terms of monitoring voice communications or monitoring actions performed by a user in an application, the techniques of the present invention may be configured to allow a user to opt-in to "join" or "opt-out of participating in the collection of personal information data during registration services. As another example, the user may choose not to provide location information for the targeted content delivery service. As another example, the user may choose not to provide accurate location information, but to permit transmission of location area information.

Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present technology do not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences based on non-personal information data or an absolute minimum of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

It is noted that the various embodiments described above may be combined with any of the other embodiments described herein. The features and advantages described in this specification are not all-inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.

Drawings

For a better understanding of various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals refer to corresponding parts throughout.

FIG. 1A is a high-level block diagram of a computing device with a touch-sensitive display, according to some embodiments.

Fig. 1B is a block diagram of exemplary components for event processing, according to some embodiments.

Figure 1C is a schematic diagram of a portable multifunction device with a touch-sensitive display, in accordance with some embodiments.

FIG. 1D is a schematic diagram illustrating a computing device having a touch-sensitive surface separate from a display, according to some embodiments.

FIG. 1E illustrates an example electronic device in communication with a display and a touch-sensitive surface, according to some embodiments.

FIG. 2 is a schematic diagram of a touch screen for showing a user interface for a menu of an application, according to some embodiments.

Fig. 3A-3B are block diagrams illustrating data structures for storing application usage data, according to some embodiments.

Fig. 4A-4B are block diagrams illustrating data structures for storing trigger conditions, according to some embodiments.

Fig. 5 is a block diagram illustrating an example of a trigger condition establishment system according to some embodiments.

Fig. 6A-6B are flow diagram representations of methods for actively identifying and displaying (e.g., displaying for user selection) related content on an electronic device with a touch-sensitive display, according to some embodiments.

Fig. 7A-7B are schematic diagrams of a touch-sensitive display showing a user interface intended to actively identify and display relevant content, according to some embodiments.

Fig. 8A-8B are flow diagram representations of methods for actively identifying and displaying (e.g., displaying for user selection) related content on an electronic device with a touch-sensitive display, according to some embodiments.

Fig. 9A-9D are schematic diagrams of a touch-sensitive display showing a user interface intended to actively identify and display relevant content, according to some embodiments.

10A-10C are a flowchart representation of a method of proactively suggesting search queries based on content currently displayed on an electronic device, according to some embodiments.

11A-11J are schematic diagrams of a touch-sensitive display illustrating a user interface for actively suggesting search queries based on content currently displayed on the touch-sensitive display, according to some embodiments.

Fig. 12 is a flowchart representation of a method of entering a search mode based on a heuristic process, according to some embodiments.

Fig. 13A-13B are schematic diagrams of a touch sensitive display to illustrate a user interface for entering a search mode based on a heuristic process, according to some embodiments.

Fig. 14 is a flowchart representation of a method of actively providing vehicle location on an electronic device with a touch-sensitive display, according to some embodiments.

15A-15B are schematic diagrams of touch sensitive displays for illustrating a user interface intended to actively provide vehicle location, according to some embodiments.

Fig. 16A-16B are a flowchart representation of a method of proactively providing nearby point of interest (POI) information for a search query, according to some embodiments.

17A-17E are schematic diagrams of a touch-sensitive display to illustrate a user interface intended to actively provide nearby point of interest (POI) information for a search query, according to some embodiments.

Fig. 18A-18B are a flowchart representation of a method of extracting and interacting with content items from voice communications, according to some embodiments.

19A-19F are schematic diagrams of a touch-sensitive display illustrating a user interface intended to display and interact with content items extracted from voice communications, according to some embodiments.

Fig. 20 is a flowchart representation of a method of determining voice communications including recognizing voice of a physical location and populating an application with information about the physical location, according to some embodiments.

21A-21B are schematic diagrams of a touch-sensitive display to illustrate an application's user interface intended to determine that voice communication includes recognizing voice of a physical location and populating the application with information about the physical location, according to some embodiments.

Fig. 22A-22B are a flowchart representation of a method of proactively suggesting physical locations for use in an instant messaging application, according to some embodiments.

Fig. 22C is a flowchart representation of a method of proactively suggesting information related to a location, event, or contact, according to some embodiments.

23A-23O are schematic diagrams of a touch-sensitive display used to illustrate a user interface intended to actively suggest information related to a location, event, or contact (e.g., to facilitate user selection and inclusion in an instant messaging application), according to some embodiments.

Fig. 24A-24B are a flowchart representation of a method of actively populating an application with information that a user previously viewed in another application, according to some embodiments.

Fig. 25A-25J are schematic diagrams of a touch sensitive display to illustrate an intent to actively populate a user interface of an application with information that a user previously viewed in another application (e.g., populate a windmill application with information about location that the user viewed in an inspection application), according to some embodiments.

Fig. 26A-26B are flow diagram representations of proactively suggesting that suggestion information previously viewed by a user in a first application be used in a second application, according to some embodiments.

Fig. 27 is a flowchart representation of proactively suggesting a physical location for use as a destination for route guidance in a vehicle, according to some embodiments.

Fig. 28 is a schematic diagram of a touch-sensitive display showing a user interface intended to actively suggest a physical location for use as a destination for route guidance in a vehicle, according to some embodiments.

Fig. 29 is a flowchart representation of a method of proactively suggesting a paste action, according to some embodiments.

Fig. 30A-30D are schematic diagrams of a touch sensitive display for illustrating a user interface intended to actively suggest a paste action, according to some embodiments.

Fig. 31_1 illustrates a mobile device configured to perform dynamic adjustment of the mobile device, in accordance with some embodiments.

Fig. 31_2 illustrates an exemplary process for invoking a heuristic process, according to some embodiments.

Fig. 31_3 illustrates a process for adjusting mobile device settings using a heuristic process, according to some embodiments.

Fig. 31_4 illustrates an example system for performing background fetch application updates, according to some embodiments.

Fig. 31_5 illustrates a peer-to-peer forecast for determining user invocation probability of an application on a mobile device 100, according to some embodiments.

Fig. 31_6 is a flow diagram of an exemplary process for predictively launching an application to perform background updates, according to some embodiments.

Fig. 31_7 is a flow diagram of an exemplary process for determining when to launch an application on a mobile device, according to some embodiments.

Fig. 31_8 is a flow diagram illustrating state transitions of entries in a trend table, according to some embodiments.

Fig. 31_9 is a block diagram illustrating a system for providing push notifications to a mobile device, according to some embodiments.

Fig. 31_10 is a flow diagram of an exemplary process for performing a non-wakeup push at a push notification server, according to some embodiments.

Fig. 31_11 is a flow diagram of an exemplary process for performing an application background update in response to a low priority push notification, according to some embodiments.

Fig. 31_12 is a flow diagram of an exemplary process for performing an application background update in response to a high priority push notification, according to some embodiments.

Fig. 31_13 is a block diagram of an exemplary system for performing background downloads and/or data uploads on a mobile device, according to some embodiments.

Fig. 31_14 is a flow diagram of an exemplary process for performing background downloads and uploads, according to some embodiments.

Fig. 31_15 illustrates an exemplary Graphical User Interface (GUI) for enabling and/or disabling application background updates on a mobile device, according to some embodiments.

Fig. 31_16 illustrates an example system for sharing data between peer devices according to some embodiments.

Fig. 31_17 illustrates an example process for sharing data between peer devices, according to some embodiments.

Fig. 32_1 is a block diagram of one embodiment of a system for returning search results based on an input query prefix, according to some embodiments.

Figure 32_2 is a flow diagram of one embodiment of a process for determining query completion terms and associated results based on an input query prefix, according to some embodiments.

Fig. 32_3 is a block diagram of one embodiment of an aggregator and multiple search domains, according to some embodiments.

Fig. 32_4 is an illustration of an embodiment of a query completion term search domain, according to some embodiments.

Fig. 32_5 is an illustration of an embodiment of a map search domain. Fig. 32_6 is a flow diagram of one embodiment of a process for determining query completion terms from multiple search domains, according to some embodiments.

Figure 32_7 is a flow diagram of one embodiment of a process of determining relevant results from multiple search domains that determine query completion terms, according to some embodiments.

Fig. 32_8 is a block diagram of one embodiment of a system to incorporate user feedback into a feedback search index, according to some embodiments.

Fig. 32_9 is a flow diagram of one embodiment of a process for incorporating user feedback into a reference search index, according to some embodiments.

Fig. 32_10 is a flow diagram of one embodiment of a process of collecting user feedback during a user search session, according to some embodiments.

Fig. 32_11 is a flow diagram of one embodiment of a process of incorporating user feedback into a feedback index, according to some embodiments.

Fig. 32_12 is a flow diagram of one embodiment of a process for updating a results cache using user feedback, according to some embodiments.

Fig. 32_13 is a block diagram of one embodiment of a federator that performs multi-domain searches using token query completion terms, according to some embodiments.

Figure 32_14 is a flow diagram of one embodiment of a process for determining a correlation result using vocabulary services, according to some embodiments.

Figure 32_15 is a flow diagram of one embodiment of a process for characterizing query completion terms, according to some embodiments.

Fig. 32_16 is a block diagram of one embodiment of a completion term module for determining query completion terms from multiple search domains, according to some embodiments.

Fig. 32_17 is a block diagram of one embodiment of a results module for determining relevant results across multiple search domains from determining query completion terms, according to some embodiments.

Fig. 32_18 is a block diagram of one embodiment of a collect feedback module for collecting user feedback during a user search session, according to some embodiments.

Fig. 32_19 is a block diagram of one embodiment of a process feedback module for incorporating user feedback into a feedback index, according to some embodiments.

Fig. 32_20 is a block diagram of one embodiment of an update query results module for updating a results cache using user feedback, according to some embodiments.

Fig. 32_21 is a block diagram of one embodiment of a process feedback module for incorporating user feedback into a feedback index, according to some embodiments.

Fig. 32_22 is a block diagram of one embodiment of an update query results module for updating a results cache using user feedback, according to some embodiments.

FIG. 33_1 illustrates, in block diagram form, a local search subsystem and a remote search subsystem on a computing device as is known in the art.

FIG. 33_2 illustrates, in block diagram form, a local search subsystem having local learning capabilities that can be used to improve results returned from a remote search application on a computing device, in accordance with some embodiments.

Fig. 33_3 illustrates, in block diagram form, a method of learning query features locally using local search queries, local results, and local feedback based on local results, in accordance with some embodiments.

Fig. 33_4 illustrates, in block diagram form, a method of learning query features locally with search results returned from both local and remote search queries and local feedback of both local and remote search query results, in accordance with some embodiments.

Fig. 33_5 illustrates, in block diagram form, a method of learning query features that are passed by a remote service to a local service in response to a query sent to the remote service, in accordance with some embodiments.

Fig. 33_6 illustrates, in block diagram form, a method of receiving or determining a new feature, training a feature locally, and utilizing a feature in accordance with some embodiments.

Fig. 33_7 illustrates an exemplary embodiment of a software stack that may be used in some embodiments of the invention according to some embodiments.

Fig. 34_5A illustrates a block diagram of an exemplary data architecture for suggesting contacts, according to some embodiments.

Fig. 34_5B illustrates a block diagram of an exemplary data architecture for suggesting calendar events, according to some embodiments.

34_ 6A-34 _6G illustrate exemplary user interfaces for providing suggested contacts and calendar events, according to some embodiments. 1A-1B, 2, and 3 describe exemplary devices for performing the techniques described in this section for suggesting contact and event information. Fig. 34_6A to 34_6G show exemplary user interfaces for suggesting contact and event information, and the user interfaces in these figures are also used to explain the processes described below, including the processes in fig. 34_7A to 34_ 13.

34_7A and 34_7B illustrate a flow diagram of an exemplary process for generating suggested contacts, according to some embodiments.

34_8A and 34_8B illustrate a flow diagram of an exemplary process for updating an existing contact using suggested contact information items, according to some embodiments.

34_9A and 34_9B illustrate a flow diagram of an exemplary process for displaying contacts with suggested contact information, according to some embodiments.

Fig. 34_10 illustrates a flow diagram of an exemplary process for displaying suggested contact information with a message, according to some embodiments.

34_11A and 34_11B illustrate a flow diagram of an exemplary process for generating suggested calendar events, according to some embodiments.

Fig. 34_12 illustrates a flow diagram of an exemplary process for displaying suggested event information with a message, according to some embodiments.

Fig. 34_13 illustrates a flow diagram of an exemplary process for displaying multiple suggested contact information or event information with a message, according to some embodiments.

Fig. 34_14 is a functional block diagram of an electronic device according to some embodiments.

Fig. 34_15 is a functional block diagram of an electronic device according to some embodiments.

Fig. 35_1 is a flow diagram of a method 35_100 for suggesting applications based on detected events, according to some embodiments.

Fig. 35_2 illustrates a segmentation process 35_200 according to some embodiments.

Fig. 35_3 illustrates a decision tree 35_300 that may be generated according to some embodiments.

Fig. 35_4 is a flow diagram of a method 35_400 for suggesting applications to a user of a computing device based on events, according to some embodiments.

Fig. 35_ 5A-35 _5D illustrate graphs of exemplary binomial distributions for various correct and incorrect numbers according to some embodiments.

Fig. 35_6A and 35_6B illustrate parent and child models generated by segmentation, according to some embodiments.

Fig. 35_7 illustrates an exemplary architecture 35_700 for providing a user interface for interaction with one or more applications to a user, according to some embodiments.

Fig. 36_1 is a flow diagram of a method of identifying an application based on a triggering event, according to some embodiments.

Fig. 36_2 illustrates a block diagram of a system for determining a trigger event, according to some embodiments.

Fig. 36_3 illustrates a block diagram of a system that identifies an application for a user based on a triggering event, according to some embodiments.

Fig. 36_4 illustrates a block diagram of a system for identifying an application with multiple predictive models, according to some embodiments.

Fig. 36_5 is a flow diagram of a method of identifying an application by a device based on a triggering event, according to some embodiments.

Fig. 36_6 is a simplified diagram of a device with a music application user interface according to some embodiments.

Fig. 36_7A and 36_7B are flow diagrams of methods for removing an identified application from a user interface, according to some embodiments.

Fig. 37_1 is a flow diagram of a method 100 for suggesting recipients to contact based on detected events, according to some embodiments.

Fig. 37_2 illustrates a block diagram of a system for determining a triggering event, according to some embodiments.

Fig. 37_3 illustrates a block diagram of a system that identifies a recipient to contact based on a triggering event, according to some embodiments.

Fig. 37_4 illustrates an example of suggesting a recipient to contact in a user interface of an email application according to some embodiments.

Fig. 37_5 illustrates an example of suggesting recipients to contact in a user interface of a search application according to some embodiments.

Fig. 37_6 is a flow diagram of a method 37_600 for suggesting recipients to a user of a computing device based on an event, according to some embodiments.

Fig. 37_7 illustrates an exemplary data flow for suggesting recipients to contact, according to some embodiments.

Fig. 37_8 illustrates a block diagram of an interaction module, according to some embodiments.

Fig. 37_9 illustrates an exemplary architecture 37_900 for providing a user interface to a user to suggest recipients to contact, according to some embodiments.

Fig. 38_1 illustrates a block diagram of different components of a mobile computing device configured to implement various techniques described herein, according to some embodiments.

FIG. 38_2 illustrates a method performed by the application prediction engine of FIG. 38_1 according to some embodiments.

FIG. 38_3 illustrates a method performed by the search application of FIG. 1 according to some embodiments.

Fig. 38_4 illustrates a conceptual diagram of an exemplary user interface of the search application of fig. 38_1, according to some embodiments.

Fig. 39_1 illustrates a block diagram of different components of a mobile computing device configured to implement various techniques described herein, according to some embodiments.

Fig. 39_2 illustrates a block diagram of a more detailed view of certain components of the mobile computing device illustrated in fig. 39_1 (or fig. 1A), according to some embodiments.

Fig. 39_3A illustrates a method for advanced initialization and operation of a prediction engine, according to some embodiments.

Fig. 39_3B illustrates a method for synchronously providing predictions at a prediction engine, according to some embodiments.

Fig. 39_3C illustrates a method for asynchronously providing predictions at a prediction engine, according to some embodiments.

Fig. 39_4A illustrates a method for a consumer application to request synchronous receipt of a prediction, according to some embodiments.

Fig. 39_4B illustrates a method of a consumer application registering to asynchronously receive predictions, according to some embodiments.

Fig. 39_5A illustrates a method for managing prediction engine registrations at a prediction engine center, according to some embodiments.

Fig. 39_5B illustrates a method for providing predictions to consumer application synchronization in a prediction engine center, according to some embodiments.

Fig. 39_5C illustrates a method for asynchronously providing predictions to a consumer application in a prediction engine center, according to some embodiments.

Fig. 40_1 is a block diagram of an exemplary system for monitoring, predicting, and notifying a contextual client computing device of a change in a current context, according to some embodiments.

FIG. 40_2A illustrates an example of context items that may constitute a current context, according to some embodiments.

Fig. 40_2B illustrates an example of a new context item added to a current context, according to some embodiments.

FIG. 40_3 illustrates an exemplary callback predicate database, according to some embodiments.

Fig. 40_4 is a diagram illustrating an exemplary state associated with a context item as a function of time, according to some embodiments.

Fig. 40_5 is a diagram illustrating an exemplary event stream associated with a context item, according to some embodiments.

Fig. 40_6 illustrates an exemplary historical event stream database, according to some embodiments.

Fig. 40_7 is a block diagram of an exemplary system for providing context callback notifications to a requesting client, according to some embodiments.

Fig. 40_8A is a block diagram of an exemplary system showing a requesting client whose restart has been terminated, according to some embodiments.

Fig. 40_8B is a block diagram of an exemplary system showing a requesting client whose restart has been terminated, according to some embodiments.

Fig. 40_9A is a block diagram of an exemplary system showing a context daemon that a restart has terminated, according to some embodiments.

Fig. 40_9B is a block diagram of an exemplary system showing a context daemon that a restart has terminated, according to some embodiments.

Fig. 40_10A is a block diagram of an exemplary system showing a context daemon and a requesting client that have restarted that have been terminated, according to some embodiments.

Fig. 40_10B is a block diagram of an exemplary system showing a context daemon and a requesting client that have restarted that have been terminated, according to some embodiments.

Fig. 40_11 is a block diagram of an exemplary system configured to restart a client and/or context daemon based on device state information received by a launch daemon, according to some embodiments.

Fig. 40_12A is a block diagram of an exemplary system illustrating restarting a context daemon using a restart daemon, according to some embodiments.

Fig. 40_12B is a block diagram of an exemplary system illustrating restarting a context daemon using a restart daemon, according to some embodiments.

Fig. 40_13A is a block diagram of an exemplary system illustrating restarting a requesting client using a restart daemon, according to some embodiments.

Fig. 40_13B is a block diagram of an exemplary system illustrating restarting a requesting client using a restart daemon, according to some embodiments.

Fig. 40_14 is a diagram illustrating an example of a slot-wise averaging for predicting future events according to some embodiments.

Fig. 40_15 depicts an exemplary diagram showing slot weighting according to some embodiments.

Fig. 40_16A is a diagram illustrating an exemplary method for predicting future context according to some embodiments.

Fig. 40_16B is a diagram illustrating an exemplary method of converting a slot-wise probability to a probability curve according to some embodiments.

Fig. 40_17 illustrates an exemplary event stream including predicted future events, according to some embodiments.

Fig. 40_18 is a flow diagram of an exemplary process for notifying a change in context on a client computing device, according to some embodiments.

Fig. 40_19 is a flow diagram of an exemplary process for restarting a context daemon to service callback requests, according to some embodiments.

Fig. 40_20 is a flow diagram of an exemplary process for restarting a callback client to receive callback notifications, according to some embodiments.

Fig. 40_21 is a flow diagram of an exemplary process for predicting future events based on historical context information, according to some embodiments.

Fig. 40_22 is a flow diagram of an exemplary process for servicing a sleep context callback request, according to some embodiments.

FIG. 41_1 is a block diagram of one embodiment of a system to index application states for use in a local device search index.

FIG. 41_2 is a block diagram of one embodiment of a system for searching an application state using an on-device application state search index.

FIG. 41_3 is a block diagram of an embodiment of a user interface displaying application state query results among other query results.

FIG. 41_4A is a flow diagram of one embodiment of a process for indexing application state received from multiple different applications on a device.

FIG. 41_4B is a flow diagram of one embodiment of a process for determining query results for a query using an application state index.

FIG. 41_5 is a flow diagram of one embodiment of a process for receiving and presenting application state as part of a query result.

FIG. 41_6 is a block diagram of one embodiment of a system to index application states for use in remote search indexing.

Fig. 41_7 is a block diagram of one embodiment of a system for searching application states using a remote application state search index.

FIG. 41_8 is a flow diagram of one embodiment of a process for adding application state to an application state index.

Figure 41_9 is a flow diagram of one embodiment of a process to export application state to an application state index service.

FIG. 41_10 is a flow diagram of one embodiment of a process for performing a query search using an application state index.

FIG. 41_11 is a flow diagram of one embodiment of a process for receiving and presenting application state as part of a query result.

41_12 is a block diagram of one embodiment of a system to index application state views for use in remote search indexing.

Fig. 41_13 is a block diagram of one embodiment of an application view.

FIG. 41_14 is a flow diagram for one embodiment of a process for generating an application state view using application states.

FIG. 41_15 is a flow diagram of one embodiment of a process for receiving and presenting application state (including application state views) as part of a query result.

Fig. 42-55 are functional block diagrams of electronic devices according to some embodiments.

Detailed Description

As mentioned above and described in more detail below, there is a need for an electronic device having faster, more efficient methods and interfaces for quickly accessing applications and desired functionality within those applications. In particular, there is a need for a device that can help a user avoid repetitive tasks and identify and provide relevant information to provide proactive assistance before the user explicitly requests the information. In addition, there is a need for quick access to applications and desired functionality within applications for a particular period of time (e.g., waking up each morning followed by accessing a calendar application), a particular location (e.g., accessing a music application at a gym), and so forth. Novel methods and interfaces are disclosed herein for addressing these needs and providing users with a means to quickly access these applications and functions within the applications at these particular locations, time periods, etc. Such methods and interfaces optionally supplement or replace conventional methods of accessing applications. Such methods and interfaces reduce the cognitive burden placed on the user and result in a more efficient human machine interface. For battery-driven devices, such methods and interfaces may conserve power and increase the time between battery charges. Moreover, such methods and interfaces help extend the life of the touch-sensitive display by requiring a smaller number of touch inputs (e.g., the methods and interfaces disclosed herein can actively provide a desired piece of information without requiring user input, rather than having to continuously and purposefully tap the touch-sensitive display to locate the piece of information).

Fig. 1A-1E and 2 provide a description of exemplary devices, below. Fig. 10 and 11 provide functional block diagrams of exemplary electronic devices. Fig. 3A through 3B and 4A through 4B are block diagrams of exemplary data structures for actively identifying and displaying relevant content (which are used in the methods described with reference to fig. 6A through 6B and the methods described with reference to fig. 8A through 8B). Fig. 5 is a block diagram illustrating an exemplary system intended to establish trigger conditions for actively identifying and displaying relevant content (which exemplary system is used in the method described with reference to fig. 6A-6B and the method described with reference to fig. 8A-8B). Fig. 6A-6B are flow diagrams depicting a method of actively identifying and displaying relevant content. Fig. 7A-7B are schematic diagrams of a touch-sensitive display illustrating exemplary user interfaces and gestures intended to actively recognize and display related content. Fig. 8A-8B are flow diagrams depicting a method for proactively identifying and displaying relevant content. Fig. 9A to 9D are schematic diagrams of a touch sensitive display illustrating additional user interfaces intended to actively identify and display relevant content. Fig. 3A to 3B, 4A to 4B, 5 and 7A to 7B are used to illustrate the method and/or process in fig. 6A to 6B. Fig. 3A-3B, 4A-4B, 5 and 9A-9D are used to illustrate the methods and/or processes of fig. 8A-8B.

10A-10C are flow diagrams depicting a method for proactively suggesting search queries based on content currently displayed on an electronic device with a touch-sensitive display. 11A-11J are schematic diagrams of a touch-sensitive display illustrating a user interface for actively suggesting search queries based on content currently displayed on the touch-sensitive display. Fig. 11A-11J are used to illustrate the method and/or process of fig. 10A-10C.

FIG. 12 is a flowchart representation of a method of entering a search mode based on a heuristic procedure. Fig. 13A-13B are schematic diagrams of a touch sensitive display for illustrating a user interface for entering a search mode based on a heuristic process. Fig. 13A-13B are used to illustrate the method and/or process of fig. 12.

Fig. 14 is a flowchart representation of a method of actively providing vehicle location on an electronic device with a touch-sensitive display, according to some embodiments. 15A-15B are schematic diagrams of touch sensitive displays for illustrating a user interface intended to actively provide vehicle location, according to some embodiments. Fig. 15A-15B are used to illustrate the method and/or process of fig. 14.

Fig. 16A-16B are a flowchart representation of a method of proactively providing nearby point of interest (POI) information for a search query, according to some embodiments. 17A-17E are schematic diagrams of touch sensitive displays for illustrating user interfaces intended to actively provide nearby point of interest (POI) information for a search query, according to some embodiments. Fig. 16A-16B are used to illustrate the method and/or process of fig. 17A-17E.

Fig. 18A-18B are a flowchart representation of a method of extracting and interacting with content items from voice communications, according to some embodiments. 19A-19F are schematic diagrams of a touch-sensitive display illustrating a user interface intended to display and interact with content items extracted from voice communications, according to some embodiments. Fig. 19A-19F are used to illustrate the method and/or process of fig. 18A-18B.

Fig. 20 is a flowchart representation of a method of determining voice communications including recognizing voice of a physical location and populating an application with information about the physical location, according to some embodiments. 21A-21B are schematic diagrams of a touch-sensitive display to illustrate an intent to determine voice communication includes recognizing speech at a physical location and populating a user interface of an application with information about the physical location, according to some embodiments. Fig. 19A-19F and 21A-21B are used to illustrate the method and/or process of fig. 20.

Fig. 22A-22B are a flowchart representation of a method of proactively suggesting physical locations for use in an instant messaging application, according to some embodiments. 23A-23O are schematic diagrams of a touch-sensitive display used to illustrate a user interface intended to actively suggest information related to a location, event, or contact (e.g., to facilitate user selection and inclusion in an instant messaging application), according to some embodiments. Fig. 23A to 23O are used to illustrate the method and/or process of fig. 22A to 22B.

Fig. 22C is a flowchart representation of a method of proactively suggesting information related to a location, event, or contact, according to some embodiments. Fig. 23A-23O are used to illustrate the method and/or process in fig. 22C.

Fig. 24A-24B are a flowchart representation of a method of actively populating an application with information that a user previously viewed in another application, according to some embodiments. Fig. 25A-25J are schematic diagrams of a touch sensitive display to illustrate an intent to actively populate a user interface of an application with information that a user previously viewed in another application (e.g., populate a downwind application with information about a location that the user viewed in an inspection application), according to some embodiments. Fig. 25A-25J are used to illustrate the method and/or process in fig. 24A-24B.

Fig. 26A-26B are flow diagram representations of proactively suggesting that suggestion information previously viewed by a user in a first application be used in a second application, according to some embodiments. Fig. 25A-25J are used to illustrate the method and/or process of fig. 26A-26B.

Fig. 27 is a flowchart representation of proactively suggesting a physical location for use as a destination for route guidance in a vehicle, according to some embodiments. Fig. 28 is a schematic diagram of a touch-sensitive display, for illustrating a user interface intended to actively suggest a physical location for use as a destination for route guidance in a vehicle, according to some embodiments. FIG. 28 is used to illustrate the method and/or process of FIG. 27.

Fig. 29 is a flowchart representation of a method of proactively suggesting a paste action, according to some embodiments. Fig. 30A-30D are schematic diagrams of a touch sensitive display to illustrate a user interface intended to actively suggest a paste action, according to some embodiments. Fig. 30A-30D are used to illustrate the method and/or process of fig. 29.

Sections 1 to 11 of the "additional description of embodiments" describe more details to supplement the details provided with reference to fig. 1A to 30D.

Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of various described embodiments. It will be apparent, however, to one skilled in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc. may be used herein to describe various elements in some cases, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact can be termed a second contact, and, similarly, a second contact can be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises," "comprising," "includes" and/or "including," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term "if" is optionally to be interpreted to mean "when … …" or "at … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrase "if determined … …" or "if [ stated condition or event ] is detected" is optionally to be construed to mean "upon determination … …" or "in response to determination … …" or "upon detection of [ stated condition or event ] or" in response to detection of [ stated condition or event ] ", depending on the context.

The disclosure herein relates interchangeably to detecting touch input on, at, above, on top of, or substantially within a particular user interface element or a particular portion of a touch-sensitive display. As used herein, a touch input detected "at" a particular user interface element can also be detected "on," "above," "top," or "substantially within" the same user interface element. In some embodiments and as described in more detail below, the desired sensitivity level for detecting touch input is configured by a user of the electronic device (e.g., the user may decide (and configure the electronic device to operate) that the touch input should only be detected when the touch input is entirely within the user interface element).

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also contains other functions, such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, those from Apple Inc (Cupertino, California)

IPOD

And

an apparatus. Other portable electronic devices are optionally used, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and/or touch pads). It should also be understood that in some embodiments, the device is not a portable communication device, but is a desktop computer with a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).

In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and/or joystick.

The device typically supports various applications, such as one or more of the following: a mapping application, a rendering application, a word processing application, a website creation application, a disc editing application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, a health/fitness application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

Various applications executing on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and/or varied for different applications and/or within respective applications. In this way, the general physical architecture of the device (such as a touch-sensitive surface) optionally supports various applications with a user interface that is intuitive and clear to the user.

Attention is now directed to embodiments of portable electronic devices having touch-sensitive displays. FIG. 1A is a block diagram illustrating a portable multifunction device 100 (also interchangeably referred to herein as electronic device 100 or device 100) with a touch-sensitive display 112 in accordance with some embodiments. Touch sensitive display 112 is sometimes referred to as a "touch screen" for convenience and is sometimes referred to or called a touch sensitive display system. Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a controller 120, one or more processing units (CPUs) 122, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, input/output (I/O) subsystems 106, other input or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 (e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100) for detecting the intensity of contacts on device 100. Device 100 optionally includes one or more tactile output generators 167 for generating tactile outputs on device 100 (e.g., generating tactile outputs on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or a trackpad of device 100). These components optionally communicate over one or more communication buses or signal lines 103.

As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a substitute (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of the contact has a range of values that includes at least four different values and more typically includes hundreds of different values (e.g., at least 256). The intensity of the contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure forces at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and/or changes thereto, the capacitance of the touch-sensitive surface in the vicinity of the contact and/or changes thereto, and/or the resistance of the touch-sensitive surface in the vicinity of the contact and/or changes thereto are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the surrogate measurement of contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurement). In some implementations, the substitute measurement of contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure).

As used in this specification and claims, the term "haptic output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to a center of mass of the device that is to be detected by a user using the user's sense of touch. For example, where the device or component of the device is in contact with a surface of the user that is sensitive to touch (e.g., a finger, palm, or other portion of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a haptic sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by a user as a "down click" or "up click" of a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement is not moving. As another example, movement of the touch sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch sensitive surface even when there is no change in the smoothness of the touch sensitive surface. While such interpretation of touch by a user will be limited by the user's individualized sensory perception, many sensory perceptions of the presence of touch are common to most users. Thus, when a haptic output is described as corresponding to a particular sensory perception of a user (e.g., "up click," "down click," "roughness"), unless otherwise stated, the generated haptic output corresponds to a physical displacement of the device or a component thereof that would generate the described sensory perception of a typical (or ordinary) user.

It should be understood that device 100 is only one embodiment of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of these components. The various components shown in fig. 1A are implemented in hardware, software, or a combination of software and hardware, including one or more signal processing and/or application specific integrated circuits.

The memory 102 optionally includes high-speed random access memory (e.g., DRAM, SRAM, DDR RAM or other random access solid state memory devices), and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid state memory devices. Memory 102 optionally includes one or more storage devices located remotely from the one or more processors 122. Access to memory 102 by other components of device 100, such as CPU 122 and peripherals interface 118, is optionally controlled by controller 120.

Peripheral interface 118 may be used to couple the input and output peripherals of the device to CPU 122 and memory 102. The one or more processors 122 run or execute various software programs and/or sets of instructions stored in the memory 102 to perform various functions of the device 100 and process data.

In some embodiments, peripherals interface 118, CPU 122, and controller 120 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

RF (radio frequency) circuitry 108 receives and transmits RF signals, also called electromagnetic signals. The RF circuitry 108 converts electrical signals to/from electromagnetic signals and communicates with communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a Subscriber Identity Module (SIM) card, memory, and so forth. RF circuitry 108 optionally communicates with networks, such as the internet (also known as a World Wide Web (WWW)), intranets, and/or wireless networks, such as cellular telephone networks, wireless Local Area Networks (LANs), and/or Metropolitan Area Networks (MANs), as well as other devices via wireless communications. The wireless communication optionally uses any of a number of communication standards, protocols, and technologies, including, but not limited to, global system for mobile communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), evolution, pure data (EV-DO), HSPA +, dual cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), wideband code division multiple access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), bluetooth low power, wireless fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and/or IEEE 802.11 n).

Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. The audio circuitry 110 receives audio data from the peripheral interface 118, converts the audio data to electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to a human. The audio circuit 110 also receives electrical signals converted by the microphone 113 from sound waves. The audio circuit 110 converts the electrical signals to audio data and transmits the audio data to the peripheral interface 118 for processing. Audio data is optionally retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripheral interface 118. In some embodiments, audio circuitry 110 also includes a headset jack. The headset jack provides an interface between the audio circuitry 110 and a removable audio input/output peripheral such as an output-only headset or a headset having both an output (e.g., a monaural or binaural headset) and an input (e.g., a microphone).

The I/O subsystem 106 connects input/output peripheral devices on the device 100, such as the touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I/O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive/transmit electrical signals from/to other input control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, and the like. In some alternative embodiments, one or more input controllers 160 are optionally coupled to (or not coupled to) any of: a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. The one or more buttons may optionally include an up/down button for volume control of the speaker 111 and/or microphone 113. The one or more buttons optionally include a push button.

Touch-sensitive display 112 provides an input interface and an output interface between the device and the user. Display controller 156 receives and/or transmits electrical signals to and/or from touch screen 112. Touch screen 112 displays visual output to a user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object.

Touch screen 112 has a touch-sensitive surface, sensor, or group of sensors that accept input from a user based on tactile sensation and/or tactile contact. Touch screen 112 and display controller 156 (along with any associated modules and/or sets of instructions in memory 102) detect contact (and any movement or breaking of the contact) on touch screen 112 and convert the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touch screen 112. In an exemplary embodiment, the point of contact between touch screen 112 and the user corresponds to the area under the user's finger.

Touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although other display technologies are used in other embodiments. Touch screen 112 and display controller 156 optionally detect contact and any movement or breaking thereof using any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 112. In one exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in Apple Inc. (Cupertino, California)

IPOD

And

the technique of (1).

Touch screen 112 optionally has a video resolution in excess of 400 dpi. In some embodiments, touch screen 112 has a video resolution of at least 600 dpi. In other embodiments, touch screen 112 has a video resolution of at least 1000 dpi. The user optionally makes contact with touch screen 112 using any suitable object or finger, such as a stylus or finger, and the like. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures. In some embodiments, the device translates the finger-based input into a precise pointer/cursor position or command for performing the action desired by the user.

In some embodiments, in addition to a touch screen, device 100 optionally includes a touch pad (not shown) for activating or deactivating particular functions. In some embodiments, the trackpad is a touch-sensitive area of the device that, unlike a touchscreen, does not display visual output. The trackpad is optionally a touch-sensitive surface separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

The device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, Alternating Current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a Light Emitting Diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

The device 100 optionally further includes one or more optical sensors 164. FIG. 1A shows an optical sensor coupled to an optical sensor controller 158 in the I/O subsystem 106. The optical sensor 164 optionally includes a Charge Coupled Device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with imaging module 143 (also called a camera module), optical sensor 164 optionally captures still images or video. In some embodiments, an optical sensor is located on the back of device 100, opposite touch screen 112 on the front of the device, enabling the touch-sensitive display to be used as a viewfinder for still and/or video image capture. In some embodiments, another optical sensor is located on the front of the device so that the user optionally obtains an image of the user for the video conference while viewing other video conference participants on the touch sensitive display.

Device 100 optionally further comprises one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 in the I/O subsystem 106. Contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor 165 receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some implementations, at least one contact intensity sensor is collocated with or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100 opposite touch screen 112 located on the front of device 100.

The device 100 optionally further includes one or more proximity sensors 166. Fig. 1A shows a proximity sensor 166 coupled to the peripheral interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I/O subsystem 106. In some embodiments, the proximity sensor is turned off and the touch screen 112 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

Device 100 optionally further comprises one or more tactile output generators 167. FIG. 1A shows a haptic output generator coupled to a haptic feedback controller 161 in the I/O subsystem 106. Tactile output generator 167 optionally includes one or more electro-acoustic devices such as speakers or other audio components; and/or an electromechanical device for converting energy into linear motion such as a motor, solenoid, electroactive aggregator, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). Contact intensity sensor 165 receives haptic feedback generation instructions from haptic feedback module 133 and generates haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112), and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., into/out of the surface of device 100) or laterally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100 opposite touch-sensitive display 112 located on the front of device 100.

Device 100 optionally also includes one or more accelerometers 168. Fig. 1A shows accelerometer 168 coupled to peripheral interface 118. Alternatively, accelerometer 168 is optionally coupled to input controller 160 in I/O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch-sensitive display based on an analysis of data received from the one or more accelerometers. Device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of device 100.

In some embodiments, the software components stored in memory 102 include an operating system 126, an active module 163 (optionally including one or more application usage data tables 335, trigger condition tables 402, trigger establishment module 163-1, and/or usage data collection module 163-2), a communication module (or set of instructions) 128, a contact/motion module (or set of instructions) 130, a graphics module (or set of instructions) 132, a text input module (or set of instructions) 134, a Global Positioning System (GPS) module (or set of instructions) 135, and applications (or set of instructions) 136. Further, in some embodiments, memory 102 stores device/global internal state 157, as shown in fig. 1A. Device/global internal state 157 includes one or more of: an active application state to indicate which applications (if any) are currently active; a display state to indicate what applications, views, or other information occupy various areas of touch-sensitive display 112; sensor states including information obtained from the various sensors of the device and the input control device 116; and location information regarding device location and/or pose (e.g., device orientation).

An operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between the various hardware and software components.

The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuitry 108 and/or the external ports 124. External port 124 (e.g., Universal Serial Bus (USB), firewire, etc.) is adapted to couple directly to other devices or indirectly through a network (e.g., the internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to, and/or compatible with, a 30-pin connector used on some embodiments of an IPOD device from APPLE inc. In other embodiments, the external port is a multi-pin (e.g., 8-pin) connector that is the same as, or similar to and/or compatible with, the 8-pin connector used in the light tning connector from APPLE inc.

Contact/motion module 130 optionally detects contact with touch screen 112 (in conjunction with display controller 156) and other touch sensitive devices (e.g., a trackpad or physical click wheel). The contact/motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger-down event), determining contact intensity (e.g., force or pressure of contact, or a substitute for force or pressure of contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining whether contact has ceased (e.g., detecting a finger-up event or contact disconnection). The contact/motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the point of contact optionally includes determining velocity (magnitude), velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the point of contact, the movement of the point of contact being represented by a series of contact data. These operations are optionally applied to a single contact (e.g., one finger contact) or multiple simultaneous contacts (e.g., "multi-touch"/multiple finger contacts). In some embodiments, the contact/motion module 130 and the display controller 156 detect contact on a touch pad.

In some embodiments, the contact/motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (e.g., determine whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of particular physical actuators and may be adjusted without changing the physical hardware of the device 100). For example, without changing the trackpad or touch-sensitive display hardware, the mouse "click" threshold of the trackpad or touch-sensitive display can be set to any one of a wide range of predefined thresholds. Additionally, in some implementations, a user of the device is provided with software settings for adjusting one or more intensity thresholds of a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and/or by adjusting multiple intensity thresholds at once with a system-level click on an "intensity" parameter).

The contact/motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and/or intensities of detected contacts). Thus, the gesture is optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event, and then detecting a finger-up (lift-off) event at the same location (or substantially the same location) as the finger-down event (e.g., at the location of the icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event, then detecting one or more finger-dragging events, and in some embodiments, also subsequently detecting a finger-up (lift-off) event.

Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphic" includes any object that may be displayed to a user, including without limitation text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations and the like.

In some embodiments, graphics module 132 stores data to be used to represent graphics. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives one or more codes for specifying a graphic to be displayed from an application program or the like, and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to output to the display controller 156.

Haptic feedback module 133 includes various software components for generating instructions for use by haptic output generator 167 in producing haptic outputs at one or more locations on device 100 in response to user interaction with device 100.

The text input module 134, which is optionally a component of the graphics module 132, provides a softkey pad for entering text in various applications, such as the contacts module 137, the email client module 140, the IM module 141, the browser module 147, and any other application that requires text entry.

The GPS module 135 determines the location of the device and provides such information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture/video metadata; and to applications that provide location-based services such as weather desktop widgets, local yellow pages desktop widgets, and map/navigation desktop widgets).

Application programs ("applications") 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof:

contacts module 137 (sometimes called an address book or contact list);

a phone module 138;

a video conferencing module 139;

an email client module 140;

an Instant Messaging (IM) module 141;

a health module 142;

a camera module 143 for still and/or video images;

an image management module 144;

a browser module 147;

A calendar module 148;

desktop applet module 149, optionally including one or more of: an weather desktop applet 149-1, a stock market desktop applet 149-2, a calculator desktop applet 149-3, an alarm desktop applet 149-4, a dictionary desktop applet 149-5 and other desktop applets obtained by the user, and a user created desktop applet 149-6;

a search module 151;

a video and music player module 152, optionally consisting of a video player module and a music player module;

a notepad module 153;

a map module 154; and/or

Online video module 155.

Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, website creation applications, disk editing applications, spreadsheet applications, JAVA enabled applications, encryption, digital rights management, voice recognition, a desktop applet creator module 149-6 for making user-created desktop applets, and voice replication.

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, contacts module 137 is optionally used to manage an address book or contact list (e.g., stored in contacts module 137 in memory 102 or memory 370), including: adding one or more names to an address book; deleting one or more names from the address book; associating one or more telephone numbers, one or more email addresses, one or more physical addresses, or other information with a name; associating the image with a name; classifying and classifying names; providing a telephone number or email address to initiate and/or facilitate communication through the telephone module 138, the video conference module 139, the email client module 140, or the IM module 141; and so on.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in address book 137, modify an entered telephone number, dial a corresponding telephone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As noted above, the wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conference module 139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants according to user instructions.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with the image management module 144, the e-mail client module 140 makes it very easy to create and send an e-mail having a still image or a video image photographed by the camera module 143.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant message module 141 includes executable instructions for entering a sequence of characters corresponding to an instant message, modifying previously entered characters, sending a corresponding instant message (e.g., using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messages or using XMPP, SIMPLE, or IMPS for internet-based instant messages), receiving an instant message, and viewing the received instant message. In some embodiments, the transmitted and/or received instant messages optionally include graphics, photos, audio files, video files, and/or MMS and/or other attachments supported in an Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both telephony-based messages (e.g., messages transmitted using SMS or MMS) and internet-based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 146, health module 142 includes executable instructions for: creating fitness (e.g., with time, distance, and/or calorie burning goals); communicating with fitness sensors (sports devices such as watches or pedometers); receiving fitness sensor data; calibrating a sensor for monitoring fitness; selecting body-building music and playing; and displaying, storing and transmitting fitness data.

In conjunction with touch screen 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for: capturing still images or video (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or video, or deleting the still images or video from the memory 102.

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for: arrange, modify (e.g., edit), or otherwise manipulate, tag, delete, present (e.g., in a digital slide show or album), and store still and/or video images.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions to browse the internet (including searching for, linking to, receiving and displaying web pages or portions thereof, and attachments and other files linked to web pages) according to user instructions.

In conjunction with the RF circuitry 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140, and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and data associated with calendars (e.g., calendar entries, to-do items, etc.) according to user instructions.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, the desktop applet module 149 is a mini-application (e.g., weather desktop applet 149-1, stock desktop applet 149-2, calculator desktop applet 149-3, alarm desktop applet 149-4, and dictionary desktop applet 149-5) or a mini-application created by a user (e.g., user created desktop applet 149-6) that is optionally downloaded and used by the user. In some embodiments, the desktop applet includes an HTML (hypertext markup language) file, a CSS (cascading style sheet) file, and a JavaScript file. In some embodiments, the desktop applet includes an XML (extensible markup language) file and a JavaScript file (e.g., Yahoo! desktop applet).

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, a desktop applet creator module (not shown) is optionally used by the user to create a desktop applet (e.g., to transfer a user-specified portion of a web page into the desktop applet).

In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions for searching memory 102 for text, music, sound, images, videos, and/or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions. In some embodiments, the search module 151 also includes executable instructions for displaying a search entry portion and a prediction portion (e.g., the search entry portion 920 and the prediction portion 930 of fig. 9B, and discussed in more detail below with reference to fig. 6A-9C). In some embodiments, the search module 151, in conjunction with the proactive module 163, also populates the predictive portion with affordances of suggested or predicted people, actions within the application, applications, nearby places, and/or news articles (as discussed in more detail below with reference to fig. 3A-9C) before the search entry portion receives any user input.

In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speakers 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and playback recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions that display, present, or otherwise play back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player, such as an IPOD from APPLE inc.

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, notepad module 153 includes executable instructions to create and manage notepads, backlogs, etc. according to user instructions.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions; data related to stores and other points of interest at or near a particular location; and other location-based data) according to user instructions.

In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions that allow a user to access, browse, receive (e.g., by streaming and/or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send emails with links to particular online videos, and otherwise manage online videos in one or more file formats, such as h.264. In some embodiments, the instant message module 141, rather than the email client module 140, is used to send a link to a particular online video.

As shown in fig. 1A, portable multifunction device 100 also includes an active module 163 for actively identifying and displaying relevant content (e.g., displaying user interface objects corresponding to lock screen actions within an application or actions within a search interface (e.g., UI objects for playing playlists within a music application)). The active module 163 optionally includes the following modules (or sets of instructions), or a subset or superset thereof:

Application usage table 335;

trigger condition table 402;

trigger setup module 163-1;

usage data collection module 163-2;

active advice module 163-3; and

a (voice communication) content extraction module 163-4.

In conjunction with application 136, GPS module 135, operating system 126, I/O subsystem 106, RF circuitry 108, external portion 124, proximity sensor 166, audio circuitry 110, accelerometer 168, speaker 111, microphone 113, and peripherals interface 118, application usage table 335, and usage data collection module 163-2 receive (e.g., from the above-identified components of device 100, FIG. 1A) and/or store application usage data. In some embodiments, application usage is reported to the usage data collection module 163-2 and then stored in the application usage table 335. In some embodiments, the application usage data includes all (or most important, relevant, or predicted) context usage information corresponding to the user's use of a particular application 136. In some embodiments, each particular application stores usage data while the user interacts with the application, and then reports that usage data to the application usage data table 335 for storage (e.g., the usage data 193 for the particular application 136-1 shown in FIG. 3B includes all sensor readings, actions performed within the application, device coupling information, etc., and sends this usage data 193 to the application usage table 335 for storage as a record within the table). For example, as the user interacts with the browser module 147, the application usage data receives and stores all contextual usage information, including current GPS coordinates of the device 100 (e.g., as determined by the GPS module 135), motion data (e.g., as determined by the accelerometer 168), ambient light data (e.g., as determined by the optical sensor 164), and intra-application actions performed by the user within the browser module 147 (e.g., URLs visited, time spent visiting each page), as well as other sensor data and other contextual usage information received and stored by the application usage table 335. Additional information regarding application usage table 335 is provided below with reference to fig. 3A-3B. As discussed below with reference to fig. 5, in some embodiments, the application usage data is stored remotely (e.g., in one or more servers 502 shown in fig. 5).

The trigger condition table 402 and trigger setup module 163-1 receive and/or store trigger conditions that are established based on the usage data stored in the application usage table 335. In some embodiments, the trigger setup module 163-1 searches and analyzes the data stored in the application usage table 335 to identify patterns. For example, if the application usage data indicates that the user always launches a music application between 3:00 pm and 4:00 pm each day, trigger setup module 163-1 creates and stores trigger conditions in trigger conditions table 402 that, when satisfied (e.g., the current time is within 3:00 pm to 4:00 pm), will cause device 100 to launch the music application or at least provide an indication to the user that the music application is available (e.g., display a UI object on the lock screen that allows the user easy access to the music application). Additional information regarding the trigger condition table 402 is provided below with reference to fig. 4A-4B. As discussed below with reference to fig. 5, in some embodiments, a pattern is identified and a trigger condition (e.g., one or more servers 502 shown in fig. 5) is established at a remote server based on the identified pattern.

The proactive suggestion module 163-3 works in conjunction with other components of the device 100 to proactively provide content to the user for the various different applications available on the electronic device. For example, the active suggestion module 163-3 provides suggested search queries and other suggested content for inclusion in a search interface (e.g., as discussed below with reference to fig. 10A-10C), provides information for assisting a user in locating their parked vehicle (e.g., as discussed below with reference to fig. 14), provides information about nearby points of interest (e.g., as discussed below with reference to fig. 16A-16B), provides content items extracted from speech provided during voice communications (e.g., as discussed below with reference to fig. 18A-18B), and assists in providing a host of other suggestions (e.g., as discussed below with reference to fig. 20, 21A-21B, 24A-24B, 26A-26B, 27, and 29), which are intended to assist a user in efficiently locating desired content with a minimum number of inputs (e.g., without having to search for the content, the active suggestion module 163-3 helps ensure that the content is provided for selection by the user at the appropriate time).

The (voice communication) content extraction module 163-4 works in conjunction with other components of the device 100 to recognize the voice associated with the new content item and extract the new content item (e.g., contact information, information about the event, and information about the location, as discussed in more detail below with reference to fig. 18A-18B and 20) from the voice communication.

Each of the above identified modules and applications corresponds to a set of executable instructions for performing one or more of the functions described above as well as the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures described above. Further, memory 102 optionally stores additional modules and data structures not described above.

In some embodiments, device 100 is a device in which operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a trackpad. The number of physical input control devices (such as push buttons, dials, and the like) on device 100 is optionally reduced by using a touch screen and/or touch pad as the primary input control device for operation of device 100.

The predefined set of functions performed exclusively through the touch screen and/or trackpad optionally includes navigating between user interfaces. In some embodiments, the trackpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to a main, home, or root menu. In such embodiments, a "menu button" is implemented using a touch pad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

Fig. 1B is a block diagram illustrating exemplary components for event processing, according to some embodiments. In some implementations, memory 102 (in FIG. 1A) includes event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 selected from among applications 136 of portable multifunction device 100 (FIG. 1A) (e.g., any of the aforementioned applications stored in memory 102 with applications 136).

Event classifier 170 receives the event information and determines application 136-1 to which the event information is to be delivered and application view 191 of application 136-1. The event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views that are displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device/global internal state 157 is used by event classifier 170 to determine which application(s) are currently active, and application internal state 192 is used by event classifier 170 to determine application view 191 to which event information is to be delivered.

In some embodiments, the application internal state 192 includes additional information, such as one or more of: resume information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by the application 136-1 or information that is ready for display by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat/undo queue of previous actions taken by the user.

Event monitor 171 receives event information from peripheral interface 118. The event information includes information about a sub-event (e.g., a user touch on touch-sensitive display 112, as part of a multi-touch gesture). Peripherals interface 118 transmits information that it receives from I/O subsystem 106 or sensors, such as proximity sensor 166, one or more accelerometers 168, and/or microphone 113 (via audio circuitry 110). Information received by peripheral interface 118 from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to peripheral interface 118 at predetermined intervals. In response, peripheral interface 118 transmits the event information. In other embodiments, peripheral interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and/or receiving more than a predetermined duration).

In some embodiments, event classifier 170 further includes hit view determination module 172 and/or active event recognizer determination module 173.

When touch-sensitive display 112 displays more than one view, hit view determination module 172 provides a software process for determining where within one or more views a sub-event has occurred. The view consists of controls and other elements that the user can see on the display.

Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the respective application) in which the touch is detected optionally corresponds to a programmatic level within a programmatic or view hierarchy of applications. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events considered as correct inputs is optionally determined based at least in part on the hit view of the initial touch that initiated the touch-based gesture.

Hit view determination module 172 receives information related to sub-events of the contact-based gesture. When the application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should handle the sub-event. In most cases, the hit view is the lowest level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

The active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some implementations, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and thus determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the area associated with a particular view, the higher views in the hierarchy will remain actively participating views.

The event dispatcher module 174 dispatches the event information to an event recognizer (e.g., event recognizer 180). In embodiments that include an active event recognizer determination module 173, the event dispatcher module 174 delivers the event information to the event recognizer determined by the active event recognizer determination module 173. In some embodiments, the event dispatcher module 174 stores event information in an event queue, which is retrieved by the respective event receiver 182.

In some embodiments, the operating system 126 includes an event classifier 170. Alternatively, application 136-1 includes event classifier 170. In another embodiment, the event classifier 170 is a stand-alone module or is part of another module stored in the memory 102 (such as the contact/motion module 130).

In some embodiments, the application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for processing touch events that occur within a respective view of the application's user interface. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, the respective application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher level object from which the application 136-1 inherits methods and other properties. In some embodiments, the respective event handlers 190 comprise one or more of: data updater 176, object updater 177, GUI updater 178, and/or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in a respective application view 191.

The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and recognizes events from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 further includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

The event receiver 182 receives event information from the event sorter 170. The event information includes information about a sub-event such as a touch or touch movement. According to the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event relates to motion of a touch, the event information optionally also includes the velocity and direction of the sub-event. In some embodiments, the event comprises rotation of the device from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa) and the event information comprises corresponding information about the current orientation of the device (also referred to as the device pose).

Event comparator 184 compares the event information to predefined event or sub-event definitions and, based on the comparison, determines an event or sub-event or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definitions 186. Event definition 186 contains definitions of events (e.g., predefined sub-event sequences), such as event 1(187-1), event 2(187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1(187-1) is a double click on the displayed object. For example, a double tap includes a first touch (touch start) on the displayed object for a predetermined length of time, a first lift-off (touch end) for a predetermined length of time, a second touch (touch start) on the displayed object for a predetermined length of time, and a second lift-off (touch end) for a predetermined length of time. In another example, the definition of event 2(187-2) is a drag on the displayed object. For example, dragging includes a touch (or contact) on a displayed object for a predetermined length of time, a movement of the touch on touch-sensitive display 112, and a lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

In some embodiments, event definition 186 includes definitions of events for respective user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view that displays three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the results of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

In some embodiments, the definition of the respective event 187 further includes a delay action that delays the delivery of the event information until it has been determined that the sequence of sub-events does or does not correspond to the event type of the event identifier.

When the respective event recognizer 180 determines that the sub-event string does not match any event in the event definition 186, the respective event recognizer 180 enters an event not possible, event failed, or event end state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, the other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the sustained touch-based gesture.

In some embodiments, the respective event recognizer 180 includes metadata 183 with configurable attributes, flags, and/or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and/or lists that indicate how or how the event recognizers interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate whether a sub-event is delivered to different levels in the view or programmatic hierarchy.

In some embodiments, when one or more particular sub-events of an event are identified, the respective event identifier 180 activates the event handler 190 associated with the event. In some embodiments, the respective event identifier 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending sub-events (and deferred sending) to corresponding hit views. In some embodiments, the event recognizer 180 throws a marker associated with the recognized event, and the event handler 190 associated with the marker retrieves the marker and performs a predefined process.

In some embodiments, the event delivery instructions 188 include sub-event delivery instructions that deliver event information about sub-events without activating an event handler. Instead, the sub-event delivery instructions deliver event information to event handlers associated with the series of sub-events or to actively participating views. Event handlers associated with the series of sub-events or with actively participating views receive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, the data updater 176 updates a phone number used in the contacts module 137 or stores a video file used in the video or music player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 176 creates a new user interface object or updates the location of a user interface object. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends it to graphics module 132 for display on a touch-sensitive display.

In some embodiments, one or more event handlers 190 include or have access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of a respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

In some embodiments, each particular application 136-1 stores usage data while the user is interacting with the application and then reports that usage data to the application usage data table 335 for storage (e.g., the usage data 193 for the particular application 136-1 shown in FIG. 3B includes all sensor readings, actions performed within the application, device coupling information, etc., and sends this usage data 193 to the application-specific application's application usage table 335 for storage as a record within the table). In some embodiments, the usage data 193 stores data reported by the usage data collection module 163-2 while the particular application 136-1 is in use (e.g., the user actively interacts with the particular application 136-1).

It should be understood that the above discussion of event processing with respect to user touches on a touch sensitive display is also applicable to other forms of user input utilizing an input device to operate multifunction device 100, not all of which are initiated on a touch screen. For example, optionally with mouse movements and mouse button presses, optionally in combination with single or multiple keyboard presses or holds; contact movements on the touchpad, such as tapping, dragging, scrolling, etc.; inputting by a stylus; movement of the device; verbal instructions; a detected eye movement; a biometric input; and/or any combination thereof, as inputs corresponding to the sub-events, defining the event to be identified.

Fig. 1C is a schematic diagram of a portable multifunction device (e.g., portable multifunction device 100) with a touch-sensitive display (e.g., touch screen 112) in accordance with some embodiments. In this embodiment, as well as other embodiments described below, a user may select one or more of these graphics by making a gesture on the screen, for example, with one or more fingers or with one or more styli. In some embodiments, selection of one or more graphics will occur when the user breaks contact with the one or more graphics (e.g., by lifting a finger off the screen). In some embodiments, the gesture optionally includes one or more tap gestures (e.g., a series of touches to the screen after liftoff), one or more swipe gestures (continuous contact during the gesture along the screen surface, e.g., left to right, right to left, up, and/or down), and/or a scrolling of a finger (e.g., right to left, left to right, up, and/or down) that has made contact with device 100. In some implementations, or in some cases, inadvertent contact with a graphic does not select the graphic. For example, when the gesture used to launch an application is a tap gesture, a swipe gesture that swipes over an application affordance (e.g., an icon) optionally does not launch (e.g., open) the corresponding application.

Device 100 optionally also includes one or more physical buttons, such as a "home" button or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu buttons are implemented as soft keys in a GUI displayed on touch screen 112.

In one embodiment, device 100 includes touch screen 112, menu button 204, push button 206 for powering the device on and off and locking the device, volume adjustment button 208, Subscriber Identity Module (SIM) card slot 210, headset jack 212, and docking/charging external port 124. The push button 206 is optionally used to: powering on/off the device by pressing and maintaining the button in a depressed state for a predetermined time interval; locking the device by pressing the button and releasing the button before a predetermined time interval has elapsed; and/or unlocking the device or initiating an unlocking process. In an alternative embodiment, device 100 also accepts voice input through microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contacts on touch screen 112, and/or one or more tactile output generators 167 for generating tactile outputs for a user of device 100.

FIG. 1D is an illustration for showing a user interface on a device (e.g., device 100 of FIG. 1A) having a touch-sensitive surface 195 (e.g., a tablet or trackpad) separate from a display 194 (e.g., touch screen 112). In some embodiments, the touch-sensitive surface 195 includes one or more contact intensity sensors (e.g., one or more of contact intensity sensors 359) for detecting intensity of contacts on the touch-sensitive surface 195, and/or one or more tactile output generators 357 for generating tactile outputs for a user of the touch-sensitive surface 195.

Although some of the examples that follow will be given with reference to input on touch screen 112 (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, as shown in FIG. 1D. In some implementations, the touch-sensitive surface (e.g., 195 in fig. 1D) has a primary axis (e.g., 199 in fig. 1D) that corresponds to a primary axis (e.g., 198 in fig. 1D) on the display (e.g., 194). According to these embodiments, the device detects contacts (e.g., 197-1 and 197-2 in FIG. 1D) with the touch-sensitive surface 195 at locations corresponding to respective locations on the display (e.g., 197-1 corresponds to 196-1 and 197-2 corresponds to 196-2 in FIG. 1D). In this way, user inputs (e.g., contact 197-1 and contact 197-2 and movements thereof) detected by the device on the touch-sensitive surface (e.g., 195 in FIG. 1D) are used by the device to manipulate the user interface on the display (e.g., 194 in FIG. 1D) of the multifunction device when the touch-sensitive surface is separated from the display. It should be understood that similar methods are optionally used for the other user interfaces described herein.

Additionally, while the following examples are given primarily with reference to finger inputs (e.g., finger contact, single-finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of these finger inputs are replaced by inputs from another input device (e.g., mouse-based inputs or stylus inputs). For example, the swipe gesture is optionally replaced by a mouse click (e.g., rather than a contact) followed by movement of the cursor along the path of the swipe (e.g., rather than movement of the contact). As another example, a flick gesture is optionally replaced by a mouse click (e.g., rather than detection of a contact, and then ceasing to detect a contact) while the cursor is over the location of the flick gesture. Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and multiple finger contacts are optionally used simultaneously.

As used herein, the term "focus selector" refers to an input element that is used to indicate the current portion of the user interface with which the user is interacting. In some particular implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touch-sensitive surface 195 in fig. 1D (in some embodiments, touch-sensitive surface 195 is a trackpad)) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted in accordance with the detected input. In some implementations that include a touch screen display (e.g., touch-sensitive display system 112 or touch screen 112 in fig. 1A) that enables direct interaction with user interface elements on the touch screen display, a contact detected on the touch screen serves as a "focus selector" such that when an input (e.g., a press input by the contact) is detected on the touch screen display at a location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted in accordance with the detected input. In some implementations, focus is moved from one area of the user interface to another area of the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using tab keys or arrow keys to move focus from one button to another); in these implementations, the focus selector moves according to focus movement between different regions of the user interface. Regardless of the particular form taken by the focus selector, the focus selector is typically a user interface element (or contact on a touchscreen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device the elements of the user interface with which the user desires to interact). For example, upon detection of a press input on a touch-sensitive surface (e.g., a trackpad or touch-sensitive display), the location of a focus selector (e.g., a cursor, contact, or selection box) over a respective button will indicate that the user desires to activate the respective button (as opposed to other user interface elements shown on the device display).

FIG. 1E illustrates an exemplary electronic device in communication with a display 194 and a touch-sensitive surface 195. For at least a subset of the electronic devices, the display 194 and/or the touch-sensitive surface 195 are integrated in the electronic device according to some embodiments. Although examples are described in more detail below with reference to touch-sensitive surface 195 and display 194 in communication with an electronic device (e.g., portable multifunction device 100 in fig. 1A-1B), it should be understood that according to some embodiments, the touch-sensitive surface and/or display is integrated with the electronic device, while in other embodiments, one or more of the touch-sensitive surface and display are separate from the electronic device. Additionally, in some embodiments, the electronic device has an integrated display and/or an integrated touch-sensitive surface, and communicates with one or more additional displays and/or touch-sensitive surfaces that are independent of the electronic device.

In some embodiments, all operations described below with reference to fig. 6A-6B, 7A-7B, 8A-8B, 9A-9D, 10A-10C, 11A-11J, 12, 13A-13B, 14, 15A-15B, 16A-16B, 17A-17E, 18A-18B, 19A-19F, 20, 21A-21B, 22A-22C, 23A-23O, 24A-24B, 25A-25J, 26A-26B, 27, 28, 29, 30A-30D are performed on a single electronic device having user interface navigation logic 480 (e.g., computing device a described below with reference to fig. 1E). It should be understood, however, a plurality of different electronic devices are often connected together to perform the operations described below with reference to fig. 6A-6B, 7A-7B, 8A-8B, 9A-9D, 10A-10C, 11A-11J, 12, 13A-13B, 14, 15A-15B, 16A-16B, 17A-17E, 18A-18B, 19A-19F, 20, 21A-21B, 22A-22C, 23A-230, 24A-24B, 25A-25J, 26A-26B, 27, 28, 29, 30A-30D (e.g., an electronic device with user interface navigation logic 480 and a separate electronic device with display 194 and/or a separate electronic device with touch-sensitive surface 195 communication). In any of these embodiments, the electronic device described below with reference to fig. 6A to 6B, 7A to 7B, 8A to 8B, 9A to 9D, 10A to 10C, 11A to 11J, 12, 13A to 13B, 14, 15A to 15B, 16A to 16B, 17A to 17E, 18A to 18B, 19A to 19F, 20, 21A to 21B, 22A to 22C, 23A to 23O, 24A to 24B, 25A to 25J, 26A to 26B, 27, 28, 29, 30A to 30D is an electronic device (or device) including user interface navigation logic 480. Additionally, it should be understood that in various embodiments, user interface navigation logic 480 may be divided among a plurality of different modules or electronic devices; for purposes of the description herein, however, user interface navigation logic 480 will be primarily referred to as being located in a single electronic device, so as not to unnecessarily obscure other aspects of the embodiments.

In some embodiments, user interface navigation logic 480 includes one or more modules (e.g., one or more event handlers 190, including one or more object updaters 177 and one or more GUI updaters 178 as described in more detail above with reference to fig. 1C) that receive interpretation inputs, and in response to these interpretation inputs, generate instructions for updating the graphical user interface in accordance with the interpretation inputs, which are then used to update the graphical user interface on the display. In some embodiments, the interpretation input is an input that has been detected (e.g., by contact movement 130 in fig. 1A-1B and 3), identified (e.g., by event recognizer 180 in fig. 1C), and/or prioritized (e.g., by event classifier 170 in fig. 1C). In some implementations, the interpretation input is generated by a module at the electronic device (e.g., the electronic device receives raw contact input data to identify a gesture from the raw contact input data). In some embodiments, some or all of the interpretation input is received by the electronic device as the interpretation input (e.g., the electronic device including the touch-sensitive surface 195 processes the raw contact input data to identify gestures from the raw contact input data and sends information indicative of the gestures to the electronic device including the user interface navigation logic 480).

In some implementations, both the display 194 and the touch-sensitive surface 195 are integrated with an electronic device (e.g., computing device a in fig. 1E) that includes user interface navigation logic 480. For example, the electronic device may be a desktop computer or a laptop computer with an integrated display (e.g., 340 in fig. 3) and a trackpad (e.g., 355 in fig. 3). As another example, the electronic device may be a portable multifunction device 100 (e.g., a smart phone, a PDA, a tablet computer, etc.) with a touch screen (e.g., 112 in fig. 2).

In some implementations, the touch-sensitive surface 195 is integrated with an electronic device, while the display 194 is not integrated with an electronic device (e.g., computing device B in fig. 1E) that includes user interface navigation logic 480. For example, the electronic device may be a device (e.g., a desktop computer or laptop computer, etc.) having an integrated touchpad (e.g., 355 in fig. 3) that is connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, television, etc.). As another example, the electronic device can be a portable multifunction device 100 (e.g., a smartphone, a PDA, a tablet computer, etc.) with a touch screen (e.g., 112 in fig. 2) that is connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, a television, etc.).

In some implementations, display 194 is integrated with an electronic device, while touch-sensitive surface 195 is not integrated with an electronic device (e.g., computing device C in fig. 1E) that includes user interface navigation logic 480. For example, the electronic device may be a device (e.g., desktop computer, laptop computer, television with integrated set-top box) with an integrated display (e.g., 340 in fig. 3) connected (via wired or wireless connection) to a separate touch-sensitive surface (e.g., remote touchpad, portable multifunction device, etc.). As another example, the electronic device can be a portable multifunction device 100 (e.g., a smartphone, a PDA, a tablet, etc.) with a touch screen (e.g., 112 in fig. 2) connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote trackpad, another portable multifunction device in which the touch screen is used as a remote trackpad, etc.).

In some implementations, neither the display 194 nor the touch-sensitive surface 195 are integrated with an electronic device (e.g., computing device D in fig. 1E) that includes user interface navigation logic 480. For example, the electronic device may be a standalone electronic device (e.g., a desktop computer, laptop computer, console, set-top box, etc.) connected (by wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, portable multifunction device, etc.) and a separate display (e.g., a computer monitor, television, etc.). As another example, the electronic device can be a portable multifunction device 100 (e.g., a smartphone, a PDA, a tablet, etc.) with a touch screen (e.g., 112 in fig. 2) connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote trackpad, a portable multifunction device with another touch screen used as a remote trackpad, etc.).

In some embodiments, a computing device has an integrated audio system. In some embodiments, the computing device communicates with an audio system that is separate from the computing device. In some embodiments, an audio system (e.g., an audio system integrated in a television unit) is integrated with a separate display 194. In some embodiments, the audio system (e.g., stereo system) is a separate system from the computing device and display 194.

Attention is now directed to user interface ("UI") implementations and associated processes that may be implemented on an electronic device, such as device 100, having a display and a touch-sensitive surface.

FIG. 2 is a schematic diagram of a touch screen for showing a user interface for a menu of an application, according to some embodiments. A similar user interface is optionally implemented on device 100 (fig. 1A). In some embodiments, the user interface displayed on touch screen 112 includes the following elements, or a subset or superset thereof:

one or more signal strength indicators 202 for one or more wireless communications (such as cellular signals and Wi-Fi signals);

time 203;

a bluetooth indicator 205;

a battery status indicator 206;

Tray 209 with icons for common applications, such as:

an icon 216 of the phone module 138 labeled "phone," optionally including an indicator 214 of the number of missed calls or voice messages;

an icon 218 of the email client module 140, labeled "mail", optionally including an indicator 210 of the number of unread emails;

icon 220 of browser module 147, labeled "browser"; and

an icon 222 labeled "IPOD" for the video and music player module 152 (also known as IPOD (trade mark of APPLE inc.) module 152); and

icons for other applications, such as:

icon 224 of IM module 141, labeled "message";

icon 226 of calendar module 148 labeled "calendar";

icon 228 labeled "photo" for image management module 144;

icon 230 of camera module 143 labeled "camera";

icon 232 of online video module 155 labeled "online video";

an icon 234 of the stock market desktop applet 149-2 labeled "stock market";

icon 236 of map module 154 labeled "map";

Icon 238 of weather desktop applet 149-1 labeled "weather";

icon 240 labeled "clock" for alarm clock desktop applet 149-4;

an "healthy" labeled icon 242 of health module 142;

icon 244 of omicron notepad module 153 labeled "notepad";

an icon 246 that sets applications or modules, which provides access to the settings of the device 100 and its various applications; and

other applications such as app stores, iTunes, voice memos, and other icons of utilities.

It should be noted that the icon labels shown in fig. 2 are merely exemplary. Other tags are optionally used for various application icons. For example, the icon 242 of the fitness module 142 is alternatively labeled as "fitness support," exercise support, "" sports support, "or" fitness. In some embodiments, the label of the respective application icon includes a name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

Fig. 3A-3B are block diagrams illustrating data structures for storing application usage data, according to some embodiments. As shown in FIG. 3A, the application usage data table 335 comprises a collection of data structures 335, optionally implemented as a collection of tables for each application installed on the device 100, each application usage data table storing usage data associated with a respective corresponding application installed on the electronic device (e.g., application 1 usage data table 335-1 stores usage data for application 1 and application usage data table 335-2 stores usage data for application 2). In some embodiments, each table (e.g., table 335-1, table 335-2, table 335-3.. table 335-N) in the set of application usage data tables stores usage data for more than one application installed on the electronic device (e.g., table 335-1 stores usage data for each of the related applications provided by a common application developer or application vendor in order to effectively store the potentially related data).

In some embodiments, one or more application usage data tables 335 (e.g., application 1 usage data table 335-1) are used to store usage data associated with applications installed on device 100. As shown in FIG. 3B, application 1 usage data table 335-1 contains a plurality of usage entries. In some embodiments, the usage entries are stored in separate records 340-1 through 340-z, and optionally in a header 340-0. In some embodiments, header 340-0 contains a brief description of each information field (e.g., each field associated with each record) stored within the table. For example, header 340-0 indicates that each record 340-1 through 340-z includes an entry ID for uniquely identifying the entry in use. In some embodiments, in addition to the entry ID field, the application 1 usage data table 335-1 includes additional fields, such as a timestamp field for identifying when the usage entry was created and/or stored in the table 335-1, and a related usage entry field for identifying related usage entries that may be stored in the other application usage data table 335.

In some embodiments, each record in application 1 usage data table 335-1 contains one or more usage entries containing usage data collected as the user interacts with application 1 (e.g., each time the user launches application 1, a new usage entry is created to store the collected usage data). In some embodiments, each usage entry in the table stores the following information and data structures, or a subset or superset thereof:

information identifying in-application actions performed by the user within the application (e.g., performed in-application actions 340-1(a)) (in some embodiments, these actions are reported by the application to the device), e.g., the application reports to the usage data collection module 163-2 that the user played a particular song in a particular playlist;

information used to identify other actions performed by the user (e.g., other actions performed 340-1(b)) within other applications (e.g., system level applications), such as providing verbal instructions to a virtual assistant application or performing a search, searching for information items within the application (e.g., search module 151 of FIG. 1A);

sensor data (e.g., usage data 340-1(c)), including data collected by sensors on device 100 while the user is interacting with an application associated with the usage entry, optionally including:

Time of day (e.g., time of day 340-1(d)) information;

position data (e.g., position data 340-1(e)) for identifying a current location when the user launches the application and other locations (e.g., reported by GPS module 132) visited by the user when executing the application;

other sensor data (e.g., other sensor data 340-1(f)), sensor data collected while the user is interacting with the application (such as ambient light data, altitude data, pressure readings, motion data, etc.);

device coupling information (e.g., device coupling information 340-1(g)) identifying an external device coupled to the device 100 when the user interacts with the application (e.g., an exemplary external device may be a pair of headphones interfaced with the headphones; or another exemplary device may be a device connected via bluetooth (e.g., a speaker in a motor vehicle or a hands-free system associated with a motor vehicle)); and

other information (e.g., other information 340-1(h)), information collected while the user is interacting with the application (e.g., information about completed transactions, such as information about the user's use of the APPLE PAY)

In some embodiments, each usage entry of the application further includes information identifying the type of action performed by the user, while in other embodiments, the information identifying the action performed within the application is used to determine or derive the type of action.

In some embodiments, the application usage data table 335 also stores information about privacy settings associated with a user of the device 100. For example, a user of device 100 can configure privacy settings associated with a set of usage data for each application. In some embodiments, the user is able to control the data collection settings for all information contained within each usage entry (e.g., performed in-application actions, performed other actions, sensor data, device coupling information, and other information). For example, the user may configure the privacy settings such that the device 100 (or components thereof, such as the usage data collection module 163-2) does not collect location data, but rather collects information about in-application actions performed with respect to the browser module 147. As another example, the user may configure the privacy settings such that the device 100 does not collect information about the performed in-application actions, but rather collects location data for the online video module 155. In this way, the user is able to control the collection of usage data on the device 100 and configure the appropriate privacy settings for each application available on the device 100 based on their personal preferences regarding usage data collection.

Fig. 4A-4B are block diagrams illustrating data structures for storing trigger conditions, according to some embodiments. As shown in fig. 4A, the active trigger condition table 402 includes a set of data structures 402, optionally implemented as a set of tables for each respective application installed on the device 100, each active trigger condition table storing trigger conditions associated with the respective application (e.g., application 1 trigger condition table 402-1 stores trigger conditions associated with application 1 (e.g., trigger conditions that will cause the device 100 to start or use application 1 once satisfied)). In some embodiments, the application uses each table (e.g., table 335-1, table 335-2, table 335-3.. table 335-N) in the set of data tables to store trigger conditions associated with more than one application installed on the electronic device (e.g., table 335-1 stores trigger conditions for each of the related applications provided by a common application developer or application provider in order to effectively store potentially related data).

In some embodiments, one or more active trigger condition tables 402 (e.g., application 1 trigger condition table 402-1) are used to store trigger conditions associated with applications installed on device 100. For example, as shown in FIG. 4B, application 1 trigger condition table 402-1 contains information identifying a plurality of prerequisites and associated actions for each trigger condition associated with application 1. As shown in FIG. 4B, application 1 trigger condition table 402-1 contains records 414-1 through 414-z, and optionally includes a header 414-0. In some embodiments, the header 414-0 contains a brief description of each information field (e.g., each field associated with each record) stored within the table. Each record (e.g., record 414-1) includes information that allows device 100 to determine prerequisites for satisfying each trigger condition. In some embodiments, prerequisite 1 of record 414-1 contains or identifies multiple prerequisites (e.g., sensor readings) that, when detected, will cause device 100 to perform an associated action (e.g., action 4).

As a specific example, prerequisite 1 may indicate whether the time of day is between 4:00 PM and 4:30 PM; the location data (e.g., reported by the GPS module 135) shows that the user is still near their office (e.g., within a predetermined distance from their work site); and accelerometer data shows that the user is moving (e.g., reported by accelerometer 168), device 100 should detect the trigger condition associated with prerequisite 1 and perform action 4 (e.g., action 4 is associated with instant message module 141 and causes module 141 to send a message to the user's spouse (or present a dialog box asking if the user wants to send a message), indicating that he/she is returning home from work). In some embodiments, prerequisites are identified based on patterns of user behavior identified by trigger establishment module 163-1 (FIG. 1A). In some embodiments, the trigger establishment module 163-1, in conjunction with the usage data collection module 163-2 and the application usage data table 335, searches data stored in the application usage data table to identify patterns of user behavior. Continuing with the previous example, after observing that the user sent a message to his spouse between 4:00 pm and 4:30 pm on three different dates (and while the user was within a predetermined distance from his work place and the user was moving), then when a prerequisite is observed, trigger setup module 163-1 creates a corresponding trigger to automatically send the message (or solicit permission from the user to automatically send the message). In some embodiments, the trigger setup module 163-1 analyzes or searches the application usage data table 335 at predefined intervals (e.g., hourly, every four hours, daily, or when the device is connected to an external power source) and creates trigger conditions only at these predefined intervals. In some embodiments, the user confirms that the trigger condition should be created (e.g., device 100 presents the user with a dialog box describing the prerequisite and associated action, and then the user confirms or rejects creation of the trigger condition). For example, an exemplary dialog box contains the text "i notice that you always send a message to your wife at this time of the day telling her that you are on her way home. You want to send her a short message telling her that: is i on the way home? "

Fig. 5 is a block diagram illustrating an exemplary trigger condition establishment system, according to some embodiments. As shown in fig. 5, the trigger condition setup system 500 includes the portable multifunction device 100, and also includes one or more servers 502. Portable multifunction device 100 communicates with one or more servers 502 over one or more networks. The one or more networks (e.g., one or more networks 520) communicatively connect each component of the trigger condition establishment system 500 with other components of the trigger condition establishment system 500. In some embodiments, the one or more networks 520 include a public communication network, a private communication network, or a combination of public and private communication networks. For example, the one or more networks 520 may be any network (or combination of networks) such as the internet, other Wide Area Networks (WANs), Local Area Networks (LANs), Virtual Private Networks (VPNs), Metropolitan Area Networks (MANs), peer-to-peer networks, and/or ad hoc connections.

In some embodiments, one or more active trigger condition tables 402 are stored on the portable multifunction device 100 and one or more other active trigger condition tables 402 are stored on one or more servers 502. In some embodiments, portable multifunction device 100 stores an active trigger condition table 402; while in other embodiments, the one or more servers 502 store the master trigger spring table 402. Similarly, in some embodiments, one or more application usage data tables 335 are stored on portable multifunction device 100, and one or more other application usage data tables 335 are stored on the one or more servers 502. In some embodiments, portable multifunction device 100 stores application usage data table 335; in yet other embodiments, the one or more servers 502 store an application usage data table 335.

In embodiments where one or more active trigger condition tables 402 or one or more application usage data tables 335 are stored at the one or more servers 502, some of the functions performed by the trigger establishment module 163-1 and the usage data collection module 163-2, respectively, are performed on the one or more servers 502. In these embodiments, information is exchanged between the one or more servers 502 and the device 100 over the network 520. For example, if the one or more servers 502 store the active trigger condition table 402 for the online video module 155, in some embodiments, the device 100 will send one or more usage entries corresponding to the online video module 155 to the one or more servers 502. In some embodiments, the one or more servers 502 then search the received usage data to identify usage patterns and create trigger conditions (as discussed above with reference to fig. 4A-4B), and send the created trigger conditions to the device 100. In some embodiments, upon receiving data (e.g., data of one or more video streams) associated with the online video module 155, the device 100 and the one or more servers 502 exchange usage data and trigger conditions. In some embodiments, the one or more servers 502 are also capable of detecting the created trigger condition (e.g., based on usage data received during data exchange of one or more video streams, the server may determine that the trigger condition has been met) such that the trigger condition need not be sent to the device 100 at all. In some embodiments, the range of usage data sent to the one or more servers 502 is limited such that it contains only information about the user's use of the online video module 155 (as described above, the user must also configure privacy settings related to usage data collection, which in some embodiments also allow the user to configure exchanges of usage data with the one or more servers 502 (e.g., configure the type of data that should be sent and the type of data that should not be sent)).

In some embodiments, the data structures discussed below with reference to sections 1 through 11 are also used to facilitate implementing and/or improving any of the methods discussed herein. For example, the prediction engine discussed below with reference to fig. 1-11 is used to help establish trigger conditions, and/or other techniques discussed in sections 1-11 are also used to help monitor application usage history.

Fig. 6A-6B illustrate a flowchart representation of a method 600 for actively identifying and displaying relevant content, according to some embodiments. Fig. 3A-3B, 4A-4B, 5 and 7A-7B are used to illustrate the methods and/or processes of fig. 6A-6B. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 600 is performed by an electronic device (e.g., portable multifunction device 100 of fig. 1A) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 600 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 600 performed by the apparatus 100 is described below. In some embodiments, with reference to fig. 1A, the operations of method 600 are performed at least in part by or using: an active module (e.g., active module 163), an application usage data table (e.g., application usage data table 335), a trigger event table (e.g., trigger condition table 402), a trigger setup module (e.g., trigger setup module 163-1), a usage data collection module (e.g., usage data collection module 163-2), an active suggestion module (e.g., active suggestion module 163-3), a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), one or more contact intensity sensors (e.g., contact intensity sensor 165), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 600 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 600 provides an intuitive way to actively identify and display relevant content on an electronic device with a touch-sensitive display. The method performs various functions by requiring fewer touch inputs to create a more efficient human-machine interface. For battery-driven electronic devices, identifying and displaying relevant content faster and more efficiently conserves power and increases the time between battery charges.

As shown in fig. 6A, the device executes an application on the electronic device (602) in response to an instruction from a user of the electronic device. In some implementations, the instruction from the user is a touch input over an icon associated with the application or a voice command received from the user instructing a virtual assistant application (e.g., a virtual assistant application managed by operating system 126 in fig. 1A) to execute the application. While executing the application, the device (or a component thereof, such as usage data collection module 163-2) collects (604) usage data that includes one or more actions performed by the user within the application. In some embodiments, the usage data includes information identifying a type of action associated with each of the one or more actions in addition to or instead of including the one or more actions. For example, the usage data includes information identifying the following actions when interacting with the music player module 152: the user searches for the first playlist, navigates within the first playlist, selects a first track in the first playlist, and then searches for the second playlist (e.g., the usage data includes each of the one or more actions performed by the user within the music player module 152). As such, the usage data includes information about each of the various actions performed (e.g., the user searching for and playing the first track of the first playlist), and also includes information identifying the type of action (search, navigation, selection, etc.). In some embodiments, the one or more actions are collected using the data collection module 163-2, and then the trigger establishment module 163-1 later assigns an action type to each of the one or more actions.

In some embodiments, the collected usage data is stored in a usage entry in an application usage data table associated with the application (as described above with reference to fig. 3A-3B). In some embodiments, the collected usage data includes in-application actions performed by the user, other actions performed by the user (e.g., interactions with the virtual assistant application, interactions with the search interface (e.g., search module 151), and other interactions with the application managed by operating system 126), information associated with calendar events, and additional data obtained from sensors on device 100 (as explained above with reference to fig. 3B).

In some embodiments, the usage data includes (618) verbal instructions from the user that are provided to the virtual assistant application while continuing to execute the application, and the at least one trigger condition is further based on the verbal instructions provided to the virtual assistant application. In some embodiments, the verbal instructions include a request to create a reminder that corresponds to (e.g., references or needs to be reconstructed/re-executed) the current state of the application that corresponds to the state of the application at the time the verbal instructions were provided (e.g., one or more application views 191 of fig. 1B). In some embodiments, the application state at the time the verbal instructions are provided is selected from the group consisting of: a page displayed within the application when the verbal instructions are provided, content played within the application when the verbal instructions are provided (e.g., a currently playing audio track), a notification displayed within the application when the verbal instructions are provided (e.g., a notification from instant messaging module 141 displayed when the user interacts with browser module 147), and an active portion of the page displayed within the application when the verbal instructions are provided (e.g., video content is currently playing in a web page). As an additional example, the current state of the application may also correspond to: (i) an identifier of a particular page currently viewed by the user within the application when the verbal instructions are provided (e.g., a URL of a currently displayed web page), or a history of actions performed by the user prior to navigating to the current page in the application (e.g., a URL visited by the user prior to the currently displayed web page).

In some embodiments, the verbal instructions include the terms "this" or "that" with respect to the current state of the application. For example, upon displaying the notification from instant messaging module 141, the user provides an instruction to the virtual assistant application to "remind me 'this'", and in response, the virtual assistant application causes device 100 to create a reminder that corresponds to the content displayed within the notification. As another example, when a user views particular video content in the online video module 155, the user provides an instruction to the virtual assistant application to "remind me to 'this'", and in response, the virtual assistant application causes the device 100 to create a reminder corresponding to the particular video content. In some embodiments, device 100 receives information about the current state of the application when verbal instructions are provided from the application itself (e.g., continuing with the previous example, online video module 155 reports its current state back to device 100, or to a component thereof, such as active module 163, and in this manner, active module 163 receives information identifying the particular video content)

The device automatically acquires (606), without human intervention, at least one trigger condition based on the collected usage data. In some embodiments, the at least one trigger spring is established on the device, while in other embodiments, the trigger condition is obtained (612) from a server (e.g., one or more servers 502 shown in fig. 5) that establishes the trigger condition based on usage data sent from the device to the one or more servers 502 (as explained above with reference to fig. 5). In some embodiments, when the at least one trigger condition is met, the device (or components thereof, such as the active module 163) will be caused to allow the user to easily perform (e.g., without any input or only a single touch or verbal input from the user) the action associated with the at least one trigger condition. For example, one exemplary trigger may indicate that between 2:00 pm and 2:30 pm, when accelerometer data (e.g., reported by accelerometer 168) indicates that the user has walked between previously visited GPS coordinates (e.g., between two buildings that the user often visited near work), the device should automatically (and without any input from the user) open a music application (e.g., music player 152 of fig. 1A) and begin playing a particular playlist. In some embodiments, in collecting the usage data and determining that the collected usage data associated with the music player 152 indicates: between 2:00 pm and 2:30 pm on weekdays, the exemplary trigger is established (by one or more servers 502 or by device 100) after the user turns on music player 152 and plays a particular playlist as the user walks between the previously visited GPS coordinates. In this way, the device (or server) identifies and recognizes patterns based on the collected usage data. By automatically performing this action for the user (e.g., playing a particular playlist), the user need not waste any time unlocking the device, searching for the music player 152, searching for a particular playlist, and then playing the particular playlist.

In some embodiments, the method further comprises: the privacy settings associated with the device user are checked prior to establishing or obtaining the trigger conditions in order to confirm that the user has allowed the device to collect certain usage data and/or to verify that the user allows the device to establish the trigger conditions (e.g., the user may configure the settings to prohibit the device from establishing a trigger condition that causes the device to automatically send a short message).

The device (or components thereof, such as trigger establishment module 163-1) also associates (608) the at least one trigger condition with a particular action (or with a particular action type corresponding to the particular action) of the one or more actions performed by the user within the application (e.g., by storing prerequisites for meeting the trigger condition with the particular action in the active trigger condition table 402 shown in fig. 4A-4B). Upon determining that the at least one trigger condition has been met, the device provides (610) an indication to the user that a particular action (or a particular action type) associated with the trigger condition is available. In some embodiments, providing the indication to the user comprises: displaying a user interface object (e.g., UI object 702, fig. 7A) for initiating a particular action (or for performing an action corresponding to a particular action type), displaying an icon associated with an application performing the particular action (e.g., application icon 710, as shown in the lower left corner of touch screen 112 of fig. 7A), or simply performing the particular action (as described in the above example for the particular playlist). In some embodiments, the device displays user interface objects and/or icons while also simply performing (automatically without human intervention) a particular action (or an action of the same particular action type as the particular action).

In some embodiments, obtaining the at least one trigger condition comprises (612): the usage data is sent to one or more servers located remotely from the electronic device (e.g., server 502 of fig. 5) and the at least one trigger condition is received from the one or more servers. For example, consistent with these embodiments, the electronic device sends one or more usage entries (e.g., usage entry 1 of fig. 3B) to the server 502 (over the network 520), and based on the usage data, the server 502 establishes the at least one trigger condition. Continuing with the example, server 502 then sends (using network 520) the at least one trigger condition (e.g., the prerequisite and associated actions stored in active trigger condition table 402-1) to device 100.

In some embodiments, providing the indication includes (614) displaying a user interface object on the lock screen on the touch-sensitive display, the user interface object corresponding to a particular action associated with the trigger condition. In some embodiments, the user interface object is displayed in a predefined central portion of the lock screen (e.g., as shown in fig. 7A, the UI object 702 is displayed substantially in the middle of the lock screen). For example, the device provides an indication by displaying a UI object 702 on the lock screen (fig. 7A). As shown in fig. 7A, UI object 702 includes predicted action 706. In some embodiments, the predicted action 706 is a description of an action associated with the at least one trigger condition (in other words, the user interface object includes a description of a particular action associated with the trigger condition (616), such as "swipe to play track 2" of the walking playlist). In some embodiments, the UI object 702 also optionally includes additional information 704 for providing the user with information of the reason for displaying the UI object 702. In some embodiments, additional information 704 includes a description of usage data used to detect a trigger condition (e.g., sensor data 340-1(c)) and/or a description of a prerequisite of the at least one trigger condition (e.g., prerequisite 1 of record 414-1 of fig. 4B). For example, the additional information 704 indicates that the predicted action 706 is displayed because the user often listens to the walking playlist while the user walks at a particular time of day. In some embodiments, selecting the additional information 704 (e.g., tapping on the top of the additional information 704) causes the device 100 to display a user interface that allows the user to change privacy settings associated with the collection of usage data and the creation of trigger conditions.

In some embodiments, UI object 702 also optionally includes (616) an application icon 710 associated with predicted action 706. For example, the application icon 710 is an icon of the music player 152 (shown in FIG. 7A). In some embodiments, the UI object 702 also includes an affordance 708 that, when selected, will cause the device to perform the predicted action (e.g., cause the device to begin playing track 2 of the walking playlist). In some embodiments, the user interface object (e.g., user interface object 702) includes a description of a particular action (e.g., predicted action 706 as explained above) associated with the trigger condition. In some embodiments, user interface object 702 also includes an icon associated with an application (e.g., application icon 710 displayed within UI object 702). In some embodiments, the user interface object 702 also includes a pause button that, when selected, will cause the device to stop displaying the UI object 702 and to redisplay the UI object 702 after a user selected or preconfigured period of time. For example, the user selects to pause the UI object 702 for two hours, after which the device redisplays the UI object 702. As another example, the user selects to pause the UI object 702 until it is available, and in some embodiments, the device 100 searches the calendar module 148 to identify the next open time slot in the user's schedule and redisplays the UI object 702 during the identified next open time slot.

In some embodiments, the device detects (622) a first gesture at the user interface object. In response to detecting the first gesture, the device displays (624) the application on the touch-sensitive display and performs a particular action associated with the trigger condition while the application is displayed. In some embodiments, the first gesture is a swipe gesture on the user interface object. In some embodiments, in response to detecting the swipe gesture on the user interface object, the device also unlocks itself before displaying the application (in other embodiments, the application is displayed just on the lock screen). In some embodiments, the first gesture is indicated by text displayed within UI object 702 (e.g., the text within predicted action 706 includes a description of the first gesture, e.g., "swipe."). Referring to FIG. 7A for example, the user contacts the touch-sensitive surface on top of the UI object 702 and moves the contact finger across the UI object 702 in a substantially horizontal direction without breaking contact with the touch-sensitive surface. In response to detecting this swipe gesture from the user on UI object 702, the device displays music player 152 and begins playing track 2 of the walking playlist.

Alternatively, in some embodiments, the device detects (626) a second gesture (e.g., a gesture different from the first gesture discussed above, such as a single tap at a predefined area of the user interface object (e.g., a play button, such as affordance 708) at the user interface object instead of detecting the first gesture. In response to detecting the second gesture and while continuing to display the lock screen on the touch-sensitive display, the device performs (628) a particular action associated with the trigger condition. In other words, the device performs a particular action from the lock screen and continues to display the lock screen without displaying the application.

In some embodiments, the first gesture and the second gesture discussed above with reference to operations 622 through 628 are the same gesture, but they are performed on different objects displayed within the UI object 702. For example, the first gesture is a swipe gesture over predicted action 706, and the second gesture is a swipe gesture over affordance 708. As another example, the first gesture is a single tap on predicted action 706 and the second gesture is a single tap on affordance 708.

In some embodiments, providing the user with an indication that the particular action is available includes letting the user know that the particular action is available for execution. In some embodiments, providing the user with an indication that a particular action associated with the trigger condition is available includes performing the particular action. In some embodiments, an indication is provided to the user by way of performance of a particular action (e.g., the user hears that the desired playlist is playing). In some embodiments, the UI object 702 is displayed on the lock screen and performs a particular action (such as the first gesture and the second gesture discussed above) without receiving any user input.

In some embodiments, instead of (or in addition to) displaying UI object 702, the device displays icons associated with applications at locations that are physically located at the corners of the lock screen (e.g., application icon 710 is displayed substantially in the lower left corner of touch screen 112, as shown in fig. 7A).

In some embodiments, the device receives an instruction from the user to unlock the electronic device (e.g., after prolonged contact with the Home button 204, the user's fingerprint is identified as valid). In response to receiving the instruction (e.g., after unlocking the device and ceasing to display the lock screen), the device displays (620) a home screen of the device on the touch-sensitive display and provides an indication to the user on the home screen that a particular action associated with the touch spring is available. As shown in fig. 7B, upon receiving an instruction to unlock the device, the UI object 702 is displayed to cover the springboard portion of the home screen (or the application launcher). In some embodiments, instead of or in addition to displaying the UI object 702 at the top of the home screen, the device will also display an application icon 710 at the bottom of the taskbar portion that overlays the home screen. In some embodiments, the home screen includes: (i) a first portion including one or more user interface pages for launching a first set of applications available on the electronic device (e.g., the first portion contains all of the individual pages of the springboard portion of the home screen), and (ii) a second portion displayed adjacent to (e.g., below) the first portion for launching a second set of applications available on the electronic device, the second portion displayed on all of the user interface pages included in the first portion (e.g., the second portion is a taskbar portion). In some embodiments, providing the indication on the home screen includes displaying the indication on the second portion (e.g., the bottom portion including the application icon 710 is displayed on the taskbar portion as shown in FIG. 7B). In some embodiments, the second set of applications is different from and smaller than the first set of applications (e.g., the second set of applications displayed within the taskbar portion is a selected set of icons corresponding to the applications that the user has favorite).

In some embodiments, determining that the at least one trigger condition has been met comprises: determining that the electronic device has been coupled with a second device different from the electronic device. For example, the second device is a pair of headphones coupled to the device through the headset jack 212, and the at least one trigger condition includes a prerequisite to indicate that the pair of headphones are coupled to the device (e.g., prior to performing a particular action, the particular action includes launching a podcast of the user's collection in a podcast application, wherein the podcast is a podcast that is frequently launched by the user after connecting the headphones). As another example, the second device is a bluetooth speaker or other hands-free device associated with a motor vehicle of the user, and the at least one trigger condition includes a prerequisite indicating that the bluetooth speaker of the motor vehicle has been coupled to the device (e.g., prior to performing a particular action, the action includes calling the user's mother when the time of day and the user's location meet the prerequisite to call the user's mother's particular action). More details regarding the coupling of external devices and the performance of actions in response to the coupling are provided in section 6 below (e.g., see FIG. 36_1 of section 6).

In some embodiments, determining that the at least one trigger condition has been met comprises: it is determined that the electronic device has reached a corresponding location of a home location or a work location associated with the user. In some embodiments, the device monitors locations that the user frequently visits (e.g., specific GPS coordinates or street addresses associated with the location), and uses this information to determine a home or work location associated with the user. In some embodiments, the device determines the addresses of these locations based on information received from the user or information input by the user (such as stored contacts). In some embodiments, determining that the electronic device has reached a corresponding address of a home location or a work location associated with the user comprises: motion data from an accelerometer of the electronic device is monitored, and a determination is made based on the monitored motion data that the electronic device has not moved more than a threshold amount of time (e.g., the user is at home and has not moved for 10 minutes). In this way, for example, the device ensures that the particular action associated with the at least one trigger condition has been performed when the user has actually settled down at their residence, not just when the user arrives at the lane of their residence.

In some embodiments of the method 600 described above, the method begins with the obtaining operation 606 and optionally includes performing operation 602 and the collecting operation 604. In other words, in these embodiments, the method 600 comprises: obtaining at least one trigger condition, the at least one trigger condition based on usage data associated with a user of the electronic device, the usage data including one or more actions performed by the user within the application when the application is executed on the electronic device; associating the at least one trigger condition with a particular action of the one or more actions performed by the user within the application; and upon determining that the at least one trigger condition has been met, providing an indication to a user that a particular action associated with the trigger spring is available.

It should be understood that the particular order in which operations have been described in fig. 6A-6B is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with reference to other methods described herein (e.g., method 800) also apply in a similar manner to method 600 described above with reference to fig. 6A-6B. For example, the user interface object described above with reference to method 600 optionally has one or more of the features of the user interface object described herein with reference to other methods (e.g., method 800) described herein. In some embodiments, any of the relevant details of sections 1 through 11 may be used in conjunction with method 600 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 8A-8B illustrate a flowchart representation of a method 800 for actively identifying and displaying relevant content, according to some embodiments. Fig. 3A-3B, 4A-4B, 5 and 9A-9D are used to illustrate the methods and/or processes of fig. 8A-8B. In some embodiments, the user interfaces shown in fig. 9A-9D are referred to as zero keyword searches. A zero-keyword search is a search that is conducted without any input from the user (e.g., the search entry box remains blank), and allows the user to, for example, view people, applications, actions within the application, nearby places, and/or news articles that the user may next (or predict) be searched. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 800 is performed by an electronic device (e.g., portable multifunction device 100 of fig. 1A) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 800 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 800 performed by the apparatus 100 is described below. In some embodiments, with reference to fig. 1A, the operations of method 800 are performed at least in part by or using: an active module (e.g., active module 163), an application usage data table (e.g., application usage data table 335), a trigger condition table (e.g., trigger condition table 402), a trigger setup module (e.g., trigger setup module 163-1), a usage data collection module (e.g., usage data collection module 163-2), an active suggestion module (e.g., active suggestion module 163-3), a search module (e.g., search module 151), a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), one or more contact intensity sensors (e.g., contact intensity sensor 165), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 800 are optionally combined, and/or the order of some operations is optionally changed.

As described below, method 800 provides an automated method for actively identifying and displaying relevant content on an electronic device with a touch-sensitive display (before the user explicitly asks for the relevant content, e.g., before the user enters any text in a search entry portion of a search interface). The method reduces the cognitive burden on the user when accessing the application, thereby creating a more efficient human-machine interface.

As shown in fig. 8A, the device detects (802) a search activation gesture on the touch-sensitive display. For example, as shown in FIG. 9A, the device detects a search activation gesture 902-1 (e.g., contacting a touch-sensitive display followed by continuously moving the contact finger in a substantially vertical direction (e.g., downward) on the display). As another example, as also shown in fig. 9A, the device detects a search activation gesture 902-2 (e.g., contacting the touch-sensitive surface, followed by a continuous movement of the contacting finger in a substantially horizontal direction (e.g., to the right) across the touch-sensitive surface). In some embodiments, the search activation gesture is retrievable from at least two different user interfaces, and a first user interface of the at least two different user interfaces corresponds to displaying a respective home screen page of a sequence of home screen pages on the touch-sensitive display.

In some embodiments, when the corresponding home screen page is the first home screen page in a sequence of home screen pages (e.g., as shown in fig. 9A), the search activation gesture includes one of: (i) a gesture that moves in a generally downward direction relative to the user of the electronic device (e.g., gesture 902-1), or (ii) a continuous gesture that moves in a generally left-to-right direction relative to the user and is generally perpendicular to the downward direction (e.g., gesture 902-2). In some embodiments, when the home screen page is the second home screen page in the sequence of home screen pages (in other words, not the first home screen page), the search activation gesture is a continuous gesture that moves in a generally downward direction relative to the electronic device user (in other words, only the search activation gesture 902-1 is available, and the gesture 902-2 is not available).

In some implementations, a second user interface of the at least two different user interfaces corresponds to displaying an application-switching interface on the touch-sensitive display (e.g., in response to the user double-clicking the Home button 204). In some embodiments, the search activation gesture includes contacting a predefined search activation portion of the application switching page on the touch-sensitive display (e.g., the application switching page includes a search entry portion that is a predefined search activation portion displayed at the top of the application switching page (similar to search entry portion 920 of fig. 9B)).

In response to detecting the search activation gesture, the device displays (804) a search interface on the touch-sensitive display, the search interface including (806): (a) a search entry portion (e.g., search entry portion 920 shown in fig. 9B for receiving input from a user, where the input is to be used as a search query), and (B) a prediction portion (e.g., prediction portion 930 shown in fig. 9B) that is displayed before any user input is received at the search entry portion. The prediction part is filled by one or more of: (a) at least one affordance for contacting a person of a plurality of previous contacts (e.g., the affordance displayed in the suggested person 940 portion of fig. 9B), and (B) at least one affordance for performing a predicted action within an application of a plurality of applications (e.g., a "deep link") available on the electronic device (e.g., the suggested actions 950 portion of fig. 9B). By "within" an application is meant that the at least one affordance performing the predicted action represents a link to a particular page, view, or state within the application (e.g., one of the application views 191 of FIG. 1B). In other words, when the at least one affordance for performing the predicted action is selected, not only is the application launched and default content (or content from previous interactions with the application) displayed, but a particular page, view, or state corresponding to the deep link is displayed.

In some embodiments, the person is automatically selected (e.g., by device 100 or active module 163) from a plurality of prior contacts based at least in part on the current time. For example, around 5:30 pm each day, while the user is still at work (determining the work location according to the explanation above with reference to fig. 6A-6B), the user sends a short message to their roommates indicating that they are returning home, so the prediction section includes an affordance associated with the roommates (e.g., P-1 indicates roommates).

In some embodiments, the predicted action (e.g., as provided by one or more of the application usage tables 335 shown in fig. 3A-3B) is automatically selected (e.g., by device 100 or active module 163) based at least in part on an application usage history associated with a user of the electronic device, the user opens the search interface (by providing a search activation gesture as described above) around 2:15 pm each day, searches for "music," selects a particular music application search result, and then plays a "walking playlist," so that the predicted portion includes an affordance for starting to play the playlist within the music application (e.g., as shown in the content displayed in the suggested action 950 portion of fig. 9B) before any user input is received by the search entry portion based on the application usage history). In some embodiments, the at least one affordance for performing the predicted action within the application is also selected (as an alternative or in addition to the application usage history) based at least in part on a current time (e.g., based on the user providing a search activation gesture around the time the user typically performed the predicted action). In some embodiments (as shown in fig. 9B), the at least one affordance for performing the predicted action corresponds to user interface object 702, and thus the details provided above with respect to user interface object 702 (fig. 6A-6B and 7A-7B) are equally applicable to suggested action portion 950 and the content displayed therein.

In some embodiments, people are further selected based at least in part on location data corresponding to the electronic device (e.g., important people that are frequently contacted when the user arrives at an address associated with their work in the morning). In some embodiments, the person's application usage history and contact information are retrieved from a memory of the electronic device (e.g., memory 102 of device 100 of fig. 1A). In some embodiments, the application usage history and contact information for the person are retrieved from a server located remotely from the electronic device (e.g., one or more servers 502 of fig. 5).

In some embodiments, the predicted portion (e.g., the suggested application 955 portion of fig. 9B) is further populated (808) with the at least one affordance of the execution prediction application. In some embodiments, the predicted application is automatically selected (by device 100) based at least in part on application usage history. For example, an application usage history (e.g., one or more records within one of the application usage data tables 335 shown in fig. 3A-3B) indicates that the calendar module 143 (fig. 1A) was opened about 9:00 a day in the morning when the user was at his home address. Thus, if the current time is around 9:00 a.m. and the location data indicates that the user is at his home address, the suggested applications 955 portion will include an affordance for the calendar module 143. As another example, the application usage history indicates that a weather application (e.g., weather desktop applet 149-1 of FIG. 1A) has been launched for three consecutive days around 5:15 in the morning, now 5:17 in the morning (e.g., the current time when the user launched spotlight using the search activation gesture is 5:17 in the morning), so the electronic device populates the search interface with the weather application based at least in part on the application usage history to make the weather application one of the forecast applications in the forecast section. In some embodiments, the predicted application and the predicted action are displayed in a single portion, with the predicted action displayed above the predicted application. As described in the previous example, in some embodiments, the selection to execute the at least one affordance of the predicted application is also based at least in part on a current time (e.g., based on a user providing a search activation gesture around a time that the predicted application is typically used).

In some embodiments, to populate the suggested application 955 portion, the device 100 (or a component thereof such as the active module 163) determines whether any prerequisites for a trigger (e.g., prerequisites stored in one of the trigger condition tables 402 of fig. 4A-4B) are satisfied, and in accordance with the determination that a particular trigger is satisfied, the device 100 populates the suggested application 955 portion accordingly (e.g., adds an affordance corresponding to the application associated with the trigger, such as the calendar module 143 or the weather desktop applet 149-1 in the foregoing example). In some embodiments, other portions within the search interface (e.g., portions 940,950,955,960 and 990) are populated using a similar determination process (for brevity, these details are not repeated here).

In some embodiments, the predicted portion (e.g., suggested location 960 portion of fig. 9B) is further populated (808) with a predicted nearby location category, and the predicted location (e.g., nearby location) category is automatically selected based at least in part on one or more items of current time and location data corresponding to the device. For example, the current time is around 7:30 a.m., the location data indicates that the device is near (within a predetermined distance of) an happy coffee shop (in some embodiments, the popularity of the coffee shop is determined by crowd-sourced usage data on a large number of devices 100 associated with a large number of different users), and thus, the affordance of the device 100 using "coffee shops" indicates that the suggested portion of the site 960 is filled. In some embodiments, the suggested place 960 portion is populated with information corresponding to a predictive search of nearby places based on the current time (in addition to or instead of the predictive place category). In other words, the device proactively predicts a search that the user is likely to do again based on a search previously performed by the user near the current time (e.g., a search performed within search module 151 or browser module 147). For example, based on the user searching for "coffee" between 7:20 am and 8:00 am on the previous four occasions (or some other threshold number of occasions), in response to detecting the search activation gesture, the device (e.g., trigger setup module 163-1) fills the suggested location 960 portion with the affordance of "coffee shop". In other embodiments, the suggested categories are based only on the current location of the device and not on time. For example, an affordance is displayed that links to a nearby coffee shop. In this way, the user need not manually search for "coffee" again, but may simply select the "coffee shop" or "dining" affordance and then quickly view a listing of the listed nearby coffee shops. In some embodiments, the previous search history along with one or more usage entries is stored as other information (e.g., other information 340-1(h) shown in FIG. 3B) and/or other actions performed (e.g., other actions 340-1(B) performed shown in FIG. 3B).

In some embodiments, the device detects a user input to scroll the predicted portion (e.g., the scroll gesture 970 shown in fig. 9B), and in response to detecting the user input to scroll the predicted portion, the device scrolls the predicted portion in accordance with the user input (e.g., scrolls the search interface down or only scrolls the predicted portion within the search interface). In response to the scrolling, the device renders at least one affordance of the predicted news article in a predicted portion (e.g., the suggested news article 990 portion shown in fig. 9C). In some embodiments, the predicted news article is automatically selected (via device 100) based at least in part on location data corresponding to the electronic device. In some embodiments, the suggested news article 990 portion is displayed without scrolling input. In some embodiments, the predicted news article is optionally selected (in addition to or instead of location data) based at least in part on: a current time (e.g., a time at which the user has read a similar or related article more than a threshold number of times (e.g., three times) near the current time (e.g., a time at which the user provided a search activation gesture that caused the device to display a search interface with the predictive portion 930)); previous search history corresponding to the user (e.g., the user has searched for articles similar to or related to predicted news articles more than a threshold number of times (e.g., three)); trend data associated with news stories searched on social media such as Twitter or Facebook by other users, friends of users, and the like.

In some embodiments, the particular order in which portions 940,950,955,960 and 990 are displayed in prediction portion 930 may be configurable, enabling a user to select a desired order for each portion. For example, the user may configure the sequence such that the first displays a suggested application 955 portion, the second displays a suggested person 940 portion, the third displays a suggested action 950 portion, the fourth displays a suggested news article 990 portion, and the last displays a suggested location 960 portion. In some embodiments, predicted portion 930 includes any two of portions 940,950,955,960 and 990. In other embodiments, predicted portion 930 includes any three of portions 940,950,955,960 and 990. In other embodiments of the present invention, the predicted portion 930 includes any four of portions 940,950,955,960, and 990. In another embodiment, predicted portion 930 includes all of portions 940,950,955,960 and 990. In some embodiments, the user configures preferences regarding how many and which of portions 940,950,955,960 and 990 should be displayed within prediction portion 930.

Additionally, in some embodiments, the user can configure the weights assigned to the data (e.g., current time, application usage history, location data, other sensor data, etc.) used to populate each of portions 940,950,955,960, and 990. For example, when determining the affordance displayed in the suggested people 940 portion of the prediction section 930, the user configures the preference so that the weight of the current time is higher than the weight of the location data.

Turning now to fig. 8B, in some embodiments, the affordances displayed within each of the aforementioned portions 940,950,955, 960, and 990 are optional, enabling a user to select one of: suggested actions, suggested applications, suggested locations, or suggested news articles (each discussed below in order).

With respect to the selection of an affordance displayed within the suggested person 940 portion, in some embodiments, the device detects (810) the selection of the at least one affordance to contact the person. In some embodiments, the device detects a single touch input on the at least one affordance (e.g., a single tap on the affordance corresponding to P-1 displayed within the suggested person 940 portion). In some embodiments, in response to detecting selection of the at least one affordance to contact the person, the device contacts the person (or suggests a different communication medium to contact the person, e.g., text message, email, phone, etc.) using contact information for the person (e.g., contact information retrieved from the device or one or more servers discussed above). For example, in response to detecting a single tap on the affordance corresponding to P-1, the device sends a short message to the user's roommate, the content of the short message being "on the way home". In some embodiments, the device automatically contacts P-1, while in other embodiments, the device displays the instant message module 141 and pre-populates the interface within the module 141 with a message (e.g., "go home"), then waits for a request from the user (e.g., a voice command or selection of a send button by the user), and resends the message upon receipt of the request. In this way, the device user can conveniently and quickly contact the person (e.g., P-1) and also send the relevant (or desired) message without having to enter any text in the search entry portion (thereby saving time and avoiding the embarrassment of the user having to enter text without locating the person).

With respect to selection of the affordance displayed within the suggested action 950 portion, in some embodiments, the device detects (812) selection of the at least one affordance to perform the predicted action. For example, the device detects a single touch input (e.g., a tap over an icon of the music player 152 or a tap on the text "tap to play track 2" of the walking playlist) within the suggested action 950 portion. In some embodiments, in response to detecting selection of the at least one affordance to perform the predicted action, the device displays the application on the touch-sensitive display and performs the predicted action within the displayed application. In other words, the device stops displaying the search interface (e.g., search module 151 having a search entry portion and a prediction portion), but starts and displays the application, and performs the predicted action within the displayed application. For example, in response to detecting a single tap on the text "tap to play track 2 of the walking playlist", the device displays the music player module 152 and performs the predicted action by playing track 2 of the walking playlist. In this way, the device user can conveniently and quickly access the relevant (or desired) application (e.g., music player) and still perform the desired function within the desired application without having to enter any text in the search entry portion (thereby saving time and avoiding the embarrassment that the user must not enter text and yet locate the music player module).

With regard to selection of the affordance displayed within the suggested applications 955 portion, in some embodiments, the device detects (814) selection of the at least one affordance to execute the predicted application. In some embodiments, the device detects a single touch input on the at least one affordance (e.g., a single tap on the affordance of the icon of the browser application 147). In some embodiments, in response to detecting selection of execution of the at least one affordance of the predicted application, the device displays the predicted application on the touch-sensitive display (e.g., the device ceases to display a search interface having a search entry portion and a predicted portion and opens and displays the predicted application on the touch-sensitive display). For example, in response to detecting a single tap on the affordance corresponding to the browser application 147 icon, the device displays the browser application 147 (e.g., the browser module 147 shown in fig. 1A). In this way, a device user can conveniently and quickly access a related (or desired) application (e.g., a browser application) without having to enter any text in the search entry portion (thereby saving time and avoiding the embarrassment of the user having to enter text without being able to locate the browser application).

With respect to selection of the affordances displayed within the suggested location 960 portion, in some embodiments, the device detects (816) selection of the at least one affordance of a category of predicted locations (e.g., nearby locations). In some embodiments, the device detects a single touch input on the at least one affordance (e.g., a single tap on the affordance for "coffee shop"). In some embodiments, in response to detecting selection of the at least one affordance of a category of places to perform prediction, the device: (i) receive data corresponding to at least one nearby place (e.g., address information or GPS coordinates of the at least one nearby place determined by the map module 154), and (ii) display the received data corresponding to the at least one nearby place on the touch-sensitive display (e.g., stop displaying the search interface, start the map module 154, display the map module 154, which includes user interface elements within the displayed map that correspond to the received data, such as dots representing the GPS coordinates of the at least one nearby place). In some implementations, the receiving and displaying steps are performed substantially in parallel. For example, in response to detecting a single tap on the affordance corresponding to "coffee shop," the device retrieves GPS coordinates of a nearby cafe offering coffee and concurrently displays the map module 154 and, after receiving the GPS coordinates, displays a dot representing the cafe GPS coordinates. In this way, the device user can conveniently and quickly locate a relevant (or desired) point of interest (e.g., the cafe discussed above) without having to enter any text in the search entry portion (thereby saving time and avoiding the embarrassment that the user has to enter text without being able to locate the cafe or any coffee shop). In some embodiments, the receiving data operation described above is performed (or at least partially performed) prior to receiving the selection of the at least one affordance of the predicted category of places. In this way, data corresponding to nearby places is preloaded and displayed quickly on the map after receiving a selection of the at least one affordance for the predicted category of places.

With respect to selection of an affordance displayed within the suggested news article 990 portion, in some embodiments, the device detects (818) selection of the at least one affordance of a predicted news article. In some embodiments, the device detects a single touch input on the at least one affordance (e.g., a single tap on the affordance of news 1 shown in fig. 9C). In some embodiments, in response to detecting selection of the at least one affordance of a predicted news article, the device displays the predicted news article on the touch-sensitive display (e.g., the device stops displaying the search interface having the search entry portion and the predicted portion and opens and displays the predicted news article within the browser module 147). For example, in response to detecting a single tap on the affordance corresponding to news 1, the device displays a news article corresponding to news 1 within the browser application 147 (e.g., the browser module 147 shown in fig. 1A). In this way, a device user can conveniently and quickly access a relevant (or desired) news article (e.g., a browser application) without having to enter any text in the search entry portion (thereby saving time and avoiding the embarrassment that the user would have to enter text without being able to locate a predicted news article).

In some embodiments, predicted/suggested content items included in the search interface are selected based on the techniques discussed below with reference to sections 1-11 (e.g., in conjunction with methods 600 and 800, or any other methods discussed herein).

It should be understood that the particular order in which operations have been described in fig. 8A-8B is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with reference to other methods described herein (e.g., method 600) also apply in a similar manner to method 800 described above with reference to fig. 8A-8B. For example, the user interface object described above with reference to method 800 optionally has one or more of the features of the user interface object described herein with reference to other methods described herein (e.g., method 600). In some embodiments, any of the relevant details of sections 1 through 11 may be used in conjunction with method 800 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 10A-10C illustrate a flowchart representation of a method 1000 of proactively suggesting search queries based on content currently displayed on an electronic device, according to some embodiments. Fig. 11A-11J are used to illustrate the method and/or process of fig. 10A-10C. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 1000 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 1000 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 1000 performed by the apparatus 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 1000 are performed at least in part by or using: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1000 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 1000 provides an intuitive way to actively suggest relevant content (e.g., suggested search queries) on an electronic device with a touch-sensitive display. This approach requires less touch input to perform a search on the electronic device (e.g., the user need only select a suggested search query, and not enter any text), thereby creating a more efficient human-machine interface and allowing the user to quickly perform related searches. By providing suggested search queries, the method 1000 also helps to ensure that the user is aware that active assistants on the device are available to help perform actions faster (thereby increasing user satisfaction with their devices). For battery-powered electronic devices, method 1000 both conserves power and increases the time interval between battery charges.

As shown in fig. 10A, the device displays (1002) content associated with an application program being executed on the electronic device on the display. For example, as shown in FIG. 11A, content associated with an email application executing on electronic device 100 is displayed on touch screen 112. The content includes at least a sender name and/or an email address (e.g., "sender: John appecore"), subject text (e.g., "where next destination. In some embodiments, the body of the email may include images 1108 and/or text 1110.

While displaying the application, the device detects (1004) a swipe gesture through the touch-sensitive surface, which when detected, causes the electronic device to enter a different search mode than the application. In some implementations, detecting the swipe gesture includes (1006) detecting the swipe gesture over at least a portion of the currently displayed content. In some embodiments, the swipe gesture is used to invoke a search interface on an application (e.g., such as shown in fig. 11B). In some embodiments, the swipe gesture is a first swipe gesture received on the application and the gesture is not received within any user interface fields included in content associated with the application (e.g., the first swipe gesture is not a tap within a search box that may be displayed in the application). In some embodiments, the first swipe gesture causes the electronic device to enter a search mode of the electronic device different from the application, the search mode including displaying a search interface (e.g., such as the search interfaces shown in fig. 11B and 11D, 11F-11J, and discussed in more detail below).

In some embodiments, a first swipe gesture (e.g., swipes down 1102-1 and 1102-3 shown in fig. 11A and 11E, respectively) may be provided at any time by swiping down (and moving at least a threshold distance (e.g., 2, 3, 4cm)) on the touch-sensitive display. In some embodiments, a swipe gesture (e.g., the first swipe gesture described above) is detected while the application is currently displayed on the touch-sensitive display, and the swipe gesture is detected at the top of the content currently displayed for the application. For example, in FIGS. 11A and 11E, the swipe down gestures 1102-1 and 1102-3 are detected at the top of the email content while the email application is currently displayed.

In some implementations, a second swipe gesture that also causes the device to enter the search mode is available at a later time (e.g., after exiting the application). In some embodiments, prior to detecting the swipe gesture, the device detects (1008) an input corresponding to a request to view the home screen of the electronic device, and in response to detecting the input, the device stops displaying content associated with the application and displays a corresponding page of the home screen of the electronic device. In some embodiments, the respective page is an initial page in a sequence of home screen pages (e.g., a first page in the sequence of home screen pages), and a swipe gesture (e.g., a second swipe gesture) is detected while the initial page of the home screen is displayed on the display.

For example, as shown in FIGS. 11A and 11E, while displaying the application, the user exits the application by tapping 1106 the physical Home button 204 of the device and switches to viewing the Home screen shown in FIG. 11C. In FIG. 11C, the first page of the home screen is indicated by highlighting the first point 1112-1 of the home screen page indicator and not highlighting the remaining points 1112-2 of the home screen. While viewing the first page of the home screen, the user is able to provide a second swipe gesture by swiping in a substantially horizontal direction (e.g., swipe gesture 1104-1 from left to right as shown in FIG. 11E). In response to receiving the second swipe gesture, the electronic device enters a search mode that includes displaying a search interface on the touch-sensitive display (as discussed in more detail below with reference to fig. 11D).

In response to detecting the swipe gesture, the device enters (1010) a search mode that includes a search interface displayed on the display. Exemplary search interfaces are shown in fig. 11B and 11D. In some embodiments, the search interface is displayed (1012) as semi-transparently overlaying an application, such as search interface 1115 in FIG. 11B. In some embodiments, the search interface is displayed to overlay the application (e.g., search interface 1115 shown in fig. 11B) semi-transparently. In some embodiments, the search interface 1115 is gradually displayed such that an animation of the search interface 1115 is played, such as fading in and/or switching from side to side. In FIG. 11B, search interface 1115 is displayed semi-transparently overlaid over the email application such that the email application portion is visibly located below search interface 1115 on touch-sensitive display 112. In some embodiments, in response to the second swipe gesture described above, the search interface is displayed as semi-transparently overlaying the home screen, as shown in fig. 11G-11J.

In some embodiments, the search interface further includes (1014) one or more trend queries, such as one or more trend terms executed by members of a social network associated with the user. In some embodiments, the one or more trend queries include trend terms based on: (i) trending news items, (ii) a current location of the electronic device (e.g., if the user visits a location outside of his residence, such as tokyo, and/or (iii) known items of interest to a visitor, etc.. for example, the trending search 1160 is shown in fig. 11B and 11D as being optional and the one or more trending terms include, for example, "balagania," "ecuador," "ruinshan," etc.

In connection with entering the search mode, the device determines (1018) the at least one suggested search query based at least in part on information associated with the content. In some embodiments, this determination is made as an animation of the search interface 1115 is played (e.g., as the search interface 1115 gradually appears). In other implementations, this determination is made prior to receiving the swipe gesture.

In some embodiments, in accordance with a determination that the content includes textual content, the device determines (1022) the at least one suggested search query based at least in part on the textual content. In some embodiments, determining the at least one suggested search query based at least in part on textual content comprises (1024): the textual content is analyzed to detect one or more predefined keywords for determining the at least one suggested search query. In some embodiments, the one or more predefined keywords are stored in one or more data structures stored on the electronic device, including a first data structure having at least 150,000 entries for the predefined keywords. In this way, the device includes a plurality of generic terms that can be quickly detected in the content and subsequently provided to the user as a suggested search query, and do so without requiring any input from the user at the search interface. In some embodiments, a second data structure of the one or more data structures is associated with a context toolkit that identifies a context of the content using the second data structure, and then identifies the at least one suggested search query based at least in part on the identified context of the content. In some embodiments, the second data structure is an on-device index (such as a wiki encyclopedia index specific to the electronic device). In some embodiments, both the first data structure and the second data structure are used to determine suggested search queries, which are then aggregated and presented to the user (e.g., within a search interface prior to receiving any user input). In some embodiments, utilizing both the first data structure and the second data structure also allows the electronic device to help distinguish between businesses that are the same name but different addresses/phones.

For example, in FIG. 11A, the content associated with the email application includes textual content, such as sender and/or recipient information, a subject line, and the text "I love guar!in the email body! "and the like. The device determines at least one suggested search query based at least in part on the textual content and displays the search results as shown in fig. 11B, e.g., ecuador, John appleore, tour guide services, cabere volcano, anisana volcano, and the like. The term "ecuador" may be a predefined key stored on the electronic device as part of an entry in a first data structure, while other entries may be identified based on the context of the content using a second data structure while utilizing the first data structure.

In some embodiments, determining the at least one suggested search query includes (1026) determining a plurality of suggested search queries, and populating the search interface includes populating the search interface with the plurality of suggested search queries. As shown in FIG. 11B, a suggested search query "Ecuador" is displayed in suggested search 1150. Optionally, as shown by the dashed lines in fig. 11B, in the suggested search 1150 section, in addition to "ecuador", a plurality of suggested search queries are displayed, such as "John appecore", "tour guide service", "canabie volcano", and "ancana volcano", among others.

In some embodiments, in connection with entering the search mode, the device obtains (1036) information associated with the content by using one or more auxiliary functions available on the electronic device (before and/or after displaying the search interface). In some embodiments, the operating system of the electronic device does not have direct access to (or knowledge of) what is currently displayed by some applications on the electronic device (e.g., third party applications developed by companies other than the operating system provider). Thus, the operating system obtains information about the content through the use of APIs (e.g., accessibility APIs) and other functionality available on the electronic device and allows the operating system to learn about the content displayed within third party applications.

In some embodiments, using one or more auxiliary functions includes (1038) using the one or more auxiliary functions to generate information associated with the content by: (i) applying a natural language processing algorithm to textual content currently displayed in an application; and (ii) determining one or more keywords describing the content using data obtained from a natural language processing algorithm, and wherein the at least one suggested search query is determined based on the one or more keywords. (e.g., for providing natural language processing algorithms such as VoiceOver, dictionary, and Speak Screen functions that may be used as the one or more auxiliary functions on the electronic device). In some embodiments, the information associated with the content includes information extracted from the content currently displayed in the application, including name, address, phone number, instant messaging, and email address (e.g., extracted using the natural language processing algorithm described above).

In some embodiments, determining the one or more keywords describing the content further comprises (1040): (i) retrieving metadata corresponding to non-textual content currently displayed in the application, and (ii) determining one or more keywords using the retrieved metadata in addition to data obtained from the natural language processing algorithm. Examples of non-textual content are images displayed within an application (e.g., image 1108 in FIG. 11A and one or more images 1114 in FIG. 11E). In some implementations, one or more informational tags (such as HTML tags, CSS descriptors, and other similar metadata) are associated with the image and may be used to help the one or more ancillary functions understand the image (e.g., one of the informational tags may describe the type of image and/or provide details about the content displayed in the image).

In some embodiments (particularly when only non-textual content is displayed in an application), the one or more keywords are determined using no natural language processing algorithm, but only the retrieved metadata. In some embodiments, input from the user previously provided in the application is also used to help determine the one or more keywords. For example, the user searches for a particular restaurant name to locate an address, and/or may also use the restaurant's phone number and name (e.g., i.e., have the restaurant name currently and displayed in the application and used only as an earlier input or search query) to help determine the one or more keywords describing the content.

Turning to FIG. 10C, prior to receiving any user input at the search interface, the device populates (1020) the displayed search interface with the at least one suggested search query. In some embodiments, the search interface includes a search entry portion (e.g., search entry portion 1120 located at the top of search interface 1115 of fig. 11B, 11D, and 11F-11J) and a search results portion (e.g., search results portion 1130 located directly below search entry portion 1120 of fig. 11B, 11D, and 11F-11J), and the at least one suggested search query is displayed within the search results portion. For example, in FIG. 11B, suggested search 1150 includes at least one suggested search query, such as "Ecuador", "John Aplecore", "tour guide service", "Kayangbeikon", "Andsasana volcano", and the at least one suggested query is displayed in search results portion 1130.

In some embodiments, the first swipe gesture described above is available while any page of the home screen is displayed. For example, in addition to being able to enter the search mode shown in FIGS. 11A and 11B on an application using the first swipe gesture 1102-1, the user can also enter the search mode using the first swipe gesture on any page of the home screen. In FIG. 11C, in response to the swipe 1104-2 in a generally vertical direction (e.g., downward), the device enters a search mode and displays a search interface 1105 as shown in FIG. 11D. Thus, at any time the user chooses to enter the search mode, the user is presented with related search queries related to the most recently viewed content in the application. Although FIG. 11C illustrates detection of the swipe gesture 1104-2 on the first page of the home screen (indicated by highlighting the first point 1112-1 of the home screen page indicator and not highlighting the remaining points 1112-2 of the home screen page indicator), the swipe gesture 1104-2 may be detected on any page of the home screen, e.g., a swipe gesture is detected on a page other than the initial page of the home screen page, wherein one of the remaining points 1112-2 is highlighted and the first point 1112-1 is not highlighted.

In some implementations, the device detects (1028), through the touch-sensitive surface, a swipe gesture over the new content currently displayed. In response to detecting the new swipe gesture, the device enters a search mode. In some embodiments, entering the search mode includes displaying a search interface on the display. In conjunction with entering a search mode and in accordance with a determination that the new content does not include textual content, in some embodiments, the device populates a search interface with suggested search queries, wherein the suggested search queries are based on a selected set of historical search queries from a user of the electronic device.

For example, after viewing the email content as shown in FIG. 11A and exiting the search interface, the user viewed a picture of Renile mountain 1114-2 (FIG. 11E) and a picture of Ecuador 1114-1 (FIG. 11E). Neither image includes textual content. Subsequently, as shown in FIG. 11E, a new swipe gesture 1102-3 is detected. In response to detecting the new swipe gesture 1102-3, the device enters a search mode and displays a search interface 1115, as shown in FIG. 11F, on the display. In FIG. 11F, "Rinieshan" is shown as a historical search query, and in the recent search 1155 section.

In some embodiments, the search interface is displayed with a point of interest (1130) based on location information provided by a second application different from the application. For example, continuing the above example, location information of the riemery mountain is obtained by a second application, such as an imaging application, based on the tags and/or metadata associated with the image. . As shown in FIG. 11F, in response to the new swipe gesture 1102-3 (FIG. 11E), the search interface 1115 displays the point of interest Renill mountain 1157-1 in the suggested places portion 1154, as shown in FIG. 11F.

In some implementations, the display of the point of interest is not only responsive to a new swipe gesture on the non-textual content. The points of interest may be displayed in response to a new swipe gesture on the text content. For example, in a scenario where the user is searching for restaurants in a first application (such as the YELP application), the user then switches to a text messaging application (e.g., an application), and then the user provides a swipe gesture on the text messaging application, in response, the device pre-populates the search interface to include and treat the points of interest (e.g., the delicious sushi 1157-2 shown in FIG. 11F) as suggested search queries based on the user's earlier interactions with the first application.

In some embodiments, the search interface further includes (1032) one or more suggested applications. The suggested applications are applications that are predicted to be of interest to the user of the electronic device based on the application usage history associated with the user (application usage history is discussed above with reference to fig. 3A and 3B). In some embodiments, the set of historical search queries is selected based at least in part on the frequency of recent search queries (e.g., based on the time and frequency with which each historical search query was made by the user). For example, as shown in FIG. 11D, health 242, book 242, map 236 applications are suggested to the user in the suggested applications 1162 section based on application usage history. These application suggestions may be selected based at least in part on the frequency of recent search queries. In some embodiments, the applications that are predicted to be of interest to the user are applications that have not been installed on the electronic device. The name of the not-yet-installed application 237 is displayed with other suggested applications and provides an installation link for the application.

In some embodiments, the application that displays the one or more suggestions is not only responsive to a new swipe on the non-textual content. For example, as shown in fig. 11D and 11E, in response to detecting the swipe gesture 1104-2 on the home screen (e.g., on any page of the home screen), the suggested application 1155 is optionally displayed in the search results portion 1130 of the search interface 1115.

Although FIGS. 11B, 11D, and 11F illustrate sorting suggested search results and displaying suggested searches in different portions of search interface 1115, other display forms are illustrated to the user. For example, suggested search results may be blended. As shown in FIG. 9D, the points of interest, suggested locations, recent search, and suggested applications are displayed together in "My location and recent browsing". In some embodiments, the blending of the suggested searches is performed according to a set of predefined rules. For example, there is a limit (e.g., 8) to the maximum number of search result discoveries from each of the proposed search sources. The order in which the search is suggested (e.g., relevance, history, and unloaded hero assets) is determined using a predetermined priority order. In another example, the predetermined set of rules includes: (i) for each type of suggested search result, it has a certain position and has a maximum number limit; (ii) for some types of suggested search results (e.g., applications that have not yet been installed), there is a maximum number limit (e.g., only 1 for each type) for such results in the blended result; (iii) or, for historical results, user dependent. For example, in some embodiments, the set of historical search queries is selected (1034) based at least in part on the frequency of recent search queries. (e.g., based on the time and frequency with which each historical search query was made by the user).

In some embodiments, only the one or more suggested applications predicted to be most interesting to the user are displayed in response to the search activation gesture. For example, as shown in fig. 11G-11J, in response to receiving a search activation gesture (e.g., swipe 1104 in fig. 11C), the device enters a search mode and displays a semi-transparent search interface on the touch-sensitive display. The search interface includes a search entry portion 1120 and a search results portion 1130. For example, as shown in FIG. 11G, the suggested application is predicted to be the application of most interest to the user. A number of applications are displayed in search results portion 1130.

In some embodiments, the suggested application uses the location information to suggest content that is predicted to be of most interest to the user. For example, in FIG. 11H, the "find My Car" application is predicted to be of most interest to the user. In search results section 1130, a user interface for the find My cars application is displayed. The application uses the user's location information to display a large head of the needle on the map and to display the user's relative location to the car indicated by the dots. In another example, based on the user's location and/or other information described above (e.g., usage data, textual content, and/or non-textual content, etc.), the application that displays nearby points of interest is predicted to be of most interest to the user. In FIG. 11I, search results section 1130 includes points of interest, such as restaurants in the "restaurants" category named "Go Japanese Fusion". The "dining" category is highlighted as a double circle as shown, and the nearby restaurant "Go Japanese Fusion" is located based on the user's location information and the restaurant's location. In another example, as shown in FIG. 11J, multiple points of interest in the "dining" category are predicted to be of most interest to the user, and these points of interest within the dining category are displayed, such as Caffe Macs, Out Steakhouse, and Chip Mexican Grill, while the "dining" category is highlighted.

It should be understood that the particular order in which the operations have been described in fig. 10A-10C is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., method 600 and method 800) also apply in a similar manner to method 1000 described above in connection with fig. 10A-10C. For example, the user interface object described above with reference to method 1000 (e.g., a user interface object displayed within a search interface) optionally has one or more of the features of the user interface object described herein with reference to other methods described herein (e.g., method 600 and method 800). In some embodiments, aspects of method 1000 are optionally interchanged or supplemented by aspects of method 1200 discussed below (and vice versa). In some embodiments, any of the relevant details of sections 1 through 11 may be used in conjunction with method 1000 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 12 illustrates a flow diagram representation of a method 1200 of entering a search mode, according to some embodiments. Fig. 13A-13B are used to illustrate the method and/or process of fig. 12. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 1200 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 1200 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 1200 performed by the apparatus 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 1200 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1200 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 1200 provides an intuitive way of actively suggesting relevant content (e.g., a suggested search query or affordance having content associated with a user's current location) on an electronic device in response to receiving a gesture. The method allows a user to efficiently identify and select desired content with a minimal number of user inputs, thereby creating a more efficient human-machine interface (e.g., the device provides suggested search queries and content for nearby points of interest, and the user need only select these suggested search queries and content, rather than having to search for and locate them). For battery-driven electronic devices, identifying and displaying relevant content faster and more efficiently conserves power and increases the time between battery charges.

As shown in fig. 12, the device detects (1202) a swipe gesture on the user interface through the touch-sensitive surface. In some implementations, the electronic device will be caused to enter a search mode when a swipe gesture is detected. In response to detecting the swipe gesture, the device enters a search mode. In some embodiments, entering the search mode comprises: prior to receiving any user input within the search interface (e.g., no text is entered in a search box within the search interface, no input is received within the search box (no tapping within the search box), etc.), a search interface different from the user interface is populated with the first content item.

In some embodiments, in accordance with a determination that the user interface includes content associated with an application, wherein the user interface is different from a home screen that includes selectable icons for invoking the application (and thus, a swipe gesture is detected on the application-specific content), the device populating the search interface with the first content item includes populating the search interface with at least one suggested search query, wherein the search query is based at least in part on the content associated with the application. For example, as described above with reference to fig. 11A-11B, the search interface 1115 is populated (fig. 11B) in response to a swipe gesture 1102 on an email application having content such as "John appelecore," ecuador images, and/or "i love ecuador" text (fig. 11A). The search interface 1115 includes at least one suggested search query, such as "ecuador", "John appecore", based at least in part on content associated with the email application. In another example, as described above with reference to fig. 11E-11F, the search interface 1115 is populated (fig. 11F) in response to a swipe gesture 1102 on an image application having content in ecuador and/or ruinean mountains (fig. 11E). The search interface 1115 includes at least one suggested search query, such as "ecuador", "ruinshan", based at least in part on the image content.

In some embodiments, in accordance with a determination that the user interface is associated with a page of the home screen (e.g., a swipe gesture on the initial home screen page shown in fig. 11C), populating the search interface with the first content item includes populating the search interface with an affordance that includes a selectable description of at least one point of interest within a threshold distance of a current location of the electronic device. For example, when the device approaches a mall with restaurants, information about those restaurants will be displayed rather than a suggested search query, because the information about restaurants is predicted to be of most interest to the user based on the proximity of the user to the mall. In the example explained above with reference to fig. 11I and 11J, in response to detecting the swipe gesture 1104 (fig. 11C) on the home screen, at least one nearby point of interest, e.g., "Go Japanese Fusion" restaurant (fig. 11I), "Caffe Macs," "Out Steakhouse," "Chip Mexican hill" (fig. 11J), is displayed in the search results portion 1130 of the search interface without displaying the suggested search query interface as shown in fig. 11D. In fig. 11I and 11J, each point of interest includes an affordance and includes a selectable description that, when selected, will provide more information about the point of interest, e.g., selecting an icon and/or a description of a point of interest will provide more description, price, menu, and/or distance information.

In some embodiments, the determination as to whether to use the suggested search query or the near point of interest representation to populate the search interface is additionally or alternatively based on whether a predetermined period of time has elapsed since the application content was displayed. For example, in accordance with a determination that (i) a swipe gesture is detected on the home screen page (e.g., a swipe gesture is detected on content), and (ii) a period of time since content associated with the application was displayed is below a threshold period of time, the search interface is still populated with the at least one suggested search query. Thus, in such embodiments, determining that a swipe gesture is not detected on the content comprises: determining that a period of time since the content was displayed meets or exceeds a threshold period of time (e.g., viewing the content 2 minutes, 3 minutes before a significant period of time), the device determines that the user is less likely to be interested in suggested search queries based on the content, and instead populates the search interface with an affordance that includes a selectable description of the at least one point of interest. Thus, if the device determines that content associated with the application has recently been displayed, the suggested search queries are still provided to the user.

In some embodiments, populating the search interface with affordances includes (1204) displaying a search entry portion of the search interface. In some embodiments, the device detects (1206) an input at a search entry portion; and in response to detecting an input (e.g., tapping therein) at the search entry portion, the electronic device ceases to display the affordance and displays the at least one suggested search query within the search interface. For example, as shown in FIG. 13A, the search interface includes a search entry portion 1120 and a search results portion 1130 with an affordance of at least one nearby point of interest (e.g., a nearby restaurant as shown in FIG. 13A). While displaying a search interface with nearby points of interest, an input 1302 is detected in search entry portion 1120, e.g., as shown in FIG. 13A, input 1302 is a user tap within a search box. In response to detecting input 1302, in FIG. 13B, the device stops displaying the at least one affordance associated with the nearby point of interest and displays a suggested search query, e.g., Ecuador, Leishan, delicious sushi, etc., in search results portion 1130. Thus, the device is able to quickly switch between suggested search queries and suggested points of interest (in this example, the user taps within the search box, indicating that they are not interested in suggested points of interest, and thus, the device attempts to provide different types of suggested content, e.g., suggested search queries based on content previously viewed in other applications).

Further details regarding optional description of the at least one point of interest are provided below with reference to fig. 16A-16B and 17A-17E. More details regarding populating a search interface with the at least one suggested search query are provided above with reference to fig. 10A-10C and 11A-11J.

It should be understood that the particular order of operations that have been described in FIG. 12 is merely exemplary and is not intended to suggest that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., method 600,800,1000) also apply in a similar manner to method 1200 described above in connection with fig. 12. For example, the user interface objects and/or operations described above with reference to method 1200 optionally have one or more of the features of the user interface objects and/or operations described herein with reference to other methods described herein (e.g., methods 600,800, and 1000). In some embodiments, any of the relevant details of sections 1 through 11 may be used in conjunction with method 1200 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 14 shows a flowchart representation of a method 1400 of actively providing vehicle location information on an electronic device with a touch-sensitive display, in accordance with some embodiments. Fig. 15A-15B are used to illustrate the method and/or process of fig. 14. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 1400 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of the electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 1400 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 1400 performed by the apparatus 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 1400 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), one or more position sensors (e.g., accelerometer 168, magnetometer, and/or GPS receiver), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1400 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 1400 provides an intuitive way to proactively provide location information when the user immediately needs the information. The method creates a more efficient human machine interface by actively providing vehicle location information without requiring the user to personally attempt to locate the information and providing the information upon determining that the user is returning to a parked vehicle. For battery-powered electronic devices, the method 1400 both conserves power and increases the time interval between battery charges.

As shown in fig. 14, without instruction from the user, the device automatically performs (1402) steps 1404 and 1406 described below. In step 1404, the device determines that the user of the electronic device is located in a vehicle that is parked at a geographic location.

In some embodiments, determining that the vehicle has stayed at a certain geographic location comprises: it is determined that the electronic device has been parked at the geographic location for more than a certain threshold period of time, for example, the device has been parked at a location for about 2 minutes after moving at a speed that exceeds a threshold speed, thus indicating that the vehicle is now parked. In some embodiments, determining that the vehicle has stayed at a certain geographic location comprises: determining that the communication link between the electronic device and the vehicle has been broken, e.g., the device loses bluetooth connection with the vehicle and/or a user removes a cable connecting the device and the vehicle, etc., thereby providing an indication that the vehicle has stopped and/or the vehicle engine has shut down. In some embodiments, determining that the vehicle has stayed at a certain geographic location comprises: it is determined that the geographic location corresponds to a location within the parking lot, e.g., current GPS coordinates are inserted (or transmitted) into a mapping application to determine whether the geographic location is within the parking lot.

In some embodiments, only one or more of the above determinations are made in order to determine whether the vehicle has stayed at the geographic location, in other embodiments two or more of the determinations are performed, and in other embodiments all three determinations are performed to assess whether the vehicle has stayed at the geographic location. For example, in some embodiments, determining that the user is in a vehicle parked at the geographic location comprises: (i) determining that the user is located within the vehicle by determining that the electronic device is moving at a speed that exceeds a threshold speed as described above (ii) determining that the vehicle has stayed at the geographic location by one or more of: (a) determining that the electronic device has stayed at the geographic location for more than a certain threshold period of time as described above, (b) determining that the communication link between the electronic device and the vehicle has been broken as described above, and (c) determining that the geographic location corresponds to a certain location within the parking lot as described above.

In step 1406, the device further determines whether the user has left the vehicle. In some embodiments, the device makes this determination by determining that its current location is beyond a threshold distance from the geographic location. In some embodiments, the device makes this determination by determining that the user has physically disconnected the device from the vehicle, or that the user has disconnected a wireless connection (e.g., bluetooth or WiFi based connection) between the device and the vehicle. More detail is provided below regarding determining that the user has left a vehicle (with a sufficiently high confidence level) located at the geographic location.

Upon determining that the user has left the vehicle at the geographic location, the device determines (1408) whether the positioning information retrieved from the location sensor to identify the geographic location meets an accuracy criterion. In some implementations, the accuracy criteria include a criterion that is met when the GPS reading accuracy associated with the positioning information is above a threshold level of accuracy (e.g., a circle error probability of 10 meters or less).

Upon determining that the positioning information does not satisfy the accuracy criteria (1408-no), the device provides (1410) a prompt to the user to enter information about the geographic location, and in response to providing the prompt, the device receives information about the geographic location from the user and stores the information as vehicle location information. In some embodiments, the prompt is an audio prompt provided by a virtual assistant that is usable by the electronic device. When the prompt is an audio prompt, receiving information from the user includes receiving a verbal description from the user identifying the geographic location. In some embodiments, the prompt from the virtual assistant instructs the user to take a picture of the vehicle located at the geographic location and/or to take a picture of an area surrounding the vehicle. In some embodiments, the user is instructed to provide a verbal description of the geographic location.

In some embodiments, upon determining that the location information meets the accuracy criteria (1408-yes), the device automatically stores (1412), without instruction from the user, the location information as vehicle location information. In some embodiments, if the positioning information is sufficiently accurate (e.g., meets accuracy criteria), no prompt is provided to the user. In other embodiments, even if the location information is accurate enough, the device prompts the user to provide additional details regarding the geographic location (verbally, textually, or by taking a picture as explained above with reference to operation 1410) to save these additional details and present them to the user when, for example, the user returns to the vehicle without a strong GPS signal by the device.

In some embodiments, the device further determines (1414) whether the user is heading for the geographic location. In some embodiments, determining whether the user is heading for the geographic location comprises: the new positioning information received from the location sensor is used to determine that the electronic device is moving to the geographic location. In some embodiments, determining whether the user is heading for the geographic location comprises: (i) determining that the electronic device stays at another geographic location (e.g., at a location/place associated with a shopping mall, restaurant, known user home address or work address, etc.) for more than a threshold period of time; and (ii) determining that the new positioning information indicates that the electronic device is leaving another geographic location and moving toward the geographic location. In some embodiments, additionally or alternatively, the device compares the picture taken of the geographic location with an image of the user's current location to determine whether the user is heading for the geographic location (e.g., by identifying common or overlapping visual elements in the image). In some embodiments, additionally or alternatively, the device detects that the user is accessing a settings user interface that allows the user to establish or search for a data connection with the vehicle, and in this way the device receives a prompt that the user is heading to the geographic location.

In some embodiments, in accordance with a determination that the user is heading to the geographic location, the device displays a user interface object that includes vehicle location information. In some embodiments, the user interface object is a map object that includes a user current location identifier and a separate identifier of a geographic location. For example, as shown in FIG. 15A, the search user interface includes a search entry section 1120 and a search results section 1130, the search results section being a map object that includes vehicle location information at a geographic location identified by a dot, a location tag "Infinite Loop 2", and a user's current location identified by a pin alone.

In some embodiments, the user interface object is displayed on a lock screen of the electronic device. For example, as shown in fig. 15B, a map object is displayed on the lock screen. Accordingly, without an instruction from the user, the device predicts that the search automobile is predicted to be of interest to the user based on the relatively accurate location information and provides a map indicating the location of the automobile without the user unlocking the electronic device.

In some embodiments, the user interface object is displayed in response to a swipe gesture that causes the electronic device to enter a search mode. In some embodiments, in response to receiving the same swipe gesture, a determination is made as to whether the user is heading for the geographic location. Thus, the same swipe gesture causes the device to determine whether the user is heading for the geographic location and display the user interface object based on relatively accurate location information.

In some embodiments, the search mode includes displaying a search interface that is pre-populated to include a user interface object (e.g., a map object that includes an identifier corresponding to the geographic location). In other words, prior to receiving any user input from the user within the search interface (e.g., prior to the user entering any search query), the search interface is populated to include the map object such that the user is provided with quick access to visual reminders regarding the geographic location of their stops (e.g., user interface object 1130 or user interface object 1525 or both shown in fig. 15A). In some embodiments, the swipe gesture is generally in a left-to-right direction, and the swipe gesture is provided by the user while the electronic device displays the initial page of the home screen (e.g., 1104-1 in fig. 11C). In some cases, the swipe gesture is generally in a downward direction and is provided by the user while viewing content associated with the application (e.g., 1102 in fig. 11A and 11E).

In some embodiments, in conjunction with determining that the user is heading to the geographic location (as discussed above with reference to operation 1414), the device also determines whether a current GPS signal associated with a location sensor of the electronic device is sufficiently strong to allow the device to provide an accurate route back to the geographic location, and upon determining that the GPS signal is not sufficiently strong, the device provides both positioning information and additional details from the user so that the user can rely on both of these pieces of information to help locate their parked vehicle.

In some embodiments, the prompt is an audio prompt (as discussed above with reference to operation 1410) provided by a virtual assistant that is usable with the electronic device, receiving information from the user includes receiving a verbal description from the user identifying the geographic location, and displaying the user interface object includes displaying a selectable affordance (e.g., affordance 1502 shown in fig. 15A-15B) that, when selected, causes the device to playback the verbal description. In some embodiments, the prompt from the virtual assistant instructs the user to take a picture of the vehicle located at the geographic location and/or to take one or more pictures/videos of the area surrounding the vehicle, and displaying the user interface object includes displaying a selectable affordance (e.g., affordance 1502 shown in fig. 15A-15B) that, when selected, will cause the device to play back recorded media. In some embodiments, the selectable affordance is displayed adjacent to the map object (as shown by affordance 1502), while in other embodiments, the selectable affordance is displayed separately (particularly where the positioning information does not meet accuracy, one example of such other display form is affordance 1535 shown in fig. 15A-15B). In some embodiments (depending on whether positioning information other than the user-provided location information has been provided), one or both of affordances 1130 and 1535 are displayed upon determining that the user is heading for a vehicle for which they are parked.

In some embodiments, the user interface object/affordance (e.g., 1130, 1535, or both) includes an estimated distance to the parked vehicle (e.g., user interface object 1130 includes "0.3 miles" in the upper right corner).

In some embodiments, the prompt is displayed on a display of the electronic device, receiving the information from the user includes receiving a textual description from the user identifying the geographic location, and displaying the user interface object includes displaying the textual description from the user. In other embodiments, a selectable affordance is displayed that allows a user to access the textual description. For example, in response to selecting the affordance 1535 (fig. 15A-15B), the device opens a notepad application that includes a textual description from the user.

It should be understood that the particular order of operations that have been described in FIG. 14 is merely exemplary and is not intended to suggest that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., methods 600,800,1000 and 1200) also apply in a similar manner to method 1400 described above in connection with fig. 14. For example, the user interface object described above with reference to method 1400 optionally has one or more of the features of the user interface object described herein with reference to other methods described herein (e.g., methods 600,800,1000 and 1200). Additionally, the details, operations, and data structures described below with reference to sections 1-11 may also be used in conjunction with the method 1400 (e.g., the details discussed with reference to section 6 may be used to help determine when to present a user interface object that includes the location of a user's parked vehicle, the details discussed with reference to section 5 may be used to help identify and learn user patterns related to when a user typically parks their vehicle and subsequent return times, and the details related to section 10 may be used to help refine vehicle location information by relying on contextual information). In some embodiments, any other relevant details of sections 1-11 can be used in conjunction with method 1400 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 16A-16B are a flowchart representation of a method 1600 of proactively providing information about nearby points of interest (POIs), according to some embodiments. Fig. 17A-17E are used to illustrate the method and/or process of fig. 16A-16B. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 1600 is performed by an electronic device (e.g., portable multifunction device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 1600 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the following describes method 1600 performed by device 100. In some embodiments, referring to fig. 1A, the operations of method 1600 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), one or more position sensors (e.g., accelerometer 168, magnetometer, and/or GPS receiver), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1600 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 1600 proactively provides point-of-interest information on an electronic device without requiring the user to search for and locate the information themselves (and then display the information when the user is within a particular distance of a particular POI). Thus, the method creates a more efficient human-machine interface by requiring less touch input to perform the required action (e.g., viewing information about nearby POIs). For battery-powered electronic devices, the method 1600 both saves power and increases the time interval between battery charges.

As shown in fig. 16A, the device monitors (1602) the geographic location of the electronic device using a location sensor without receiving any instructions from the user of the electronic device. Also without receiving any instructions from the electronic device user, the device determines, based on the monitored geographic location, that the electronic device is within a threshold distance of a predetermined type of point of interest (e.g., a point of interest for which an activity recommendation is available, such as a restaurant, amusement park, or movie theater).

In some embodiments, the predetermined types of points of interest are determined based on points of interest that are frequently visited by the user. In some embodiments, the points of interest also include points of interest predicted to be of interest to the user based on current text messages, email, and/or other data associated with the user's social network.

Still without receiving any instructions from the electronic device user, in accordance with a determination that the electronic device is within a threshold distance of the point of interest, the device identifies at least one activity that is currently popular at the point of interest and retrieves information about the point of interest, including retrieving information about the at least one activity that is currently popular at the point of interest (e.g., a currently popular ride, a currently popular menu item, a currently popular movie, etc.). In some embodiments, the popularity of an activity is assessed based on whether a threshold number (e.g., more than 5) or a threshold percentage (e.g., 5% or 10%) of individuals whose posted content is related to the at least one activity in the user's social network is reached. In some embodiments, the device maintains a list of a predetermined number (e.g., 5, 10, or 20) of points of interest that the user visits frequently (and/or points of interest determined to be of current interest based on text messages, email, or activity in the user's social network, as described above), and when the user is within a threshold distance of any of these points of interest (e.g., 1 mile, 1.5 miles, 2 miles), the device retrieves information about the current activity at these points of interest.

Still referring to FIG. 16A, after retrieving the information about the point of interest, the device detects (1616) a first input via the touch-sensitive surface that, when detected, will cause the electronic device to enter a search mode. In some embodiments, the search mode is a system level search mode, allowing searches to be conducted across the electronic device (e.g., across applications and content sources (both on the device and elsewhere), rather than just within a single application). In some implementations, the first input corresponds to a swipe gesture received on the touch-sensitive surface while the device displays an initial page of the home screen (e.g., swipe gesture 1104-1, shown in FIG. 11C, generally in a left-to-right direction).

In some embodiments, in accordance with a determination that the device is within the threshold distance of the point of interest, the device further displays an affordance on the lock screen indicating that information about a current activity at the point of interest is available. In some embodiments, the first input corresponds to a request to browse available information about a current activity at the point of interest. For example, as shown in fig. 17D, a restaurant information object is displayed on the lock screen. The icon and/or description of the restaurant is selectable and indicates that more information about the restaurant, such as menu information, is available. In response to a first input, such as a tap on the "view menu" link, the menu is displayed (e.g., directly on the lock screen or by unlocking the device and opening the appropriate application to view the menu). In some embodiments, any of the user interface objects/affordances shown in fig. 17A-17E (e.g., 1713 and 1715 and the content contained therein) may be presented within a search interface or a lock screen (or both).

Turning to FIG. 16B, in response to detecting the first input, the device enters (1618) a search mode. In some embodiments, entering the search mode comprises: before any user input is received by the search interface (e.g., no search terms have been entered in a search box within the search interface, and no input has been received), an affordance is presented, via the display, that includes: (i) information about the at least one activity, and (ii) an indication that the at least one activity has been identified as a popular activity at the current point of interest, such as popular menu items for nearby restaurants (e.g., affordances 1715 in fig. 17C-17D), ride waiting times for nearby amusement parks (e.g., affordances 1713 in fig. 17A-17B), current show times for nearby movie theaters, and so forth.

For example, as shown in fig. 17A, in some embodiments, the point of interest is (1604) an amusement park and the retrieved information includes a current wait time for a ride of the amusement park. In some embodiments shown in fig. 17A, the electronic device uses the retrieved information to present an average wait time (e.g., 1 hour) for all rides, and the user can select a link to view the wait time for each individual ride. As shown in fig. 17B, in some embodiments, the retrieved information portion includes (1606) information regarding the wait times of rides located within a predetermined distance of the electronic device, e.g., three rides/rides are within about 100 to 150 feet of the electronic device, and the wait time for each ride/ride is displayed (upon receiving input from the user requesting to view the ride wait time, such as input on the "view wait time" text shown in fig. 17A).

As another example, as shown in FIG. 17C, the point of interest is a restaurant (1608), and the retrieved information includes information regarding popular menu items in the restaurant. In some embodiments, the retrieved information is information retrieved (1610) from a social network associated with the electronic device user. For example, in FIG. 17C, the popular menu item "Yakiniku Koji" in the restaurant "Go Japanese Fusion" is displayed within the affordance 1715, and the popular menu item may be determined based on information retrieved from a social network associated with the electronic device user.

As another example, the point of interest may be (1612) a movie theater, and the retrieved information includes information about the show time of the movie theater. In some embodiments, the retrieved information about the show time is information retrieved (1614) from a social network associated with the electronic device user (e.g., based on information recently posted by the individual in the social network).

In some embodiments, the device detects (1620) a second input, e.g., selecting a display more link displayed near (e.g., above) the affordance (such as the display more links of affordances 1713 and 1715 in fig. 17A-17D), and in response to detecting the second input, the device updates the affordance to include available information for current activity at a second point of interest that is different from the point of interest. In some embodiments, the second point of interest is also located within a threshold distance of the electronic device. For example, in response to the user selecting the display of more links as shown in FIG. 17D, the device update affordance 1715 can include available information about a restaurant and a food offered by another restaurant "Out Steakhouse" within 1 mile of the electronic device, as shown in FIG. 17C. In other words, affordance 1715 initially presents only information about "Go Japanese Fusion," and in response to a second input, affordance 1715 is updated to include information about a second point of interest (e.g., information about "Out Steakhouse," as shown in fig. 17C in dashed lines). In some embodiments, in response to detecting the second input, more than one point of interest is displayed that is different from the point of interest, e.g., the device updates the restaurant information representation to include available information for two or more new restaurants other than the point of interest. In some embodiments, the affordances presented on the lock screen (e.g., affordance 1715 displayed on the lock screen shown in FIG. 17D) may also provide the same functionality (i.e., functionality that allows the user to view information about additional points of interest in response to selecting to display more links).

In some embodiments, since the user has not selected the closest point of interest, to capture points of interest that the user may be interested in, the affordance further includes (1622) a selectable category of points of interest, and the device detects (1624) selection of the respective selectable category and, in response to detecting the selection, updates the affordance to include information about additional points of interest located within a second threshold distance of the device (e.g., the second threshold distance is greater than the threshold distance). For example, the first threshold distance is 100 feet. When the electronic device is about 50 feet from the point of interest, the device will display "Go Japanese Fusion" as the point of interest, as shown in fig. 17C and 17D. In response to detecting the selection of the "dining" category, as shown in FIG. 17E, additional points of interest located 100 feet away and 1 mile away will be displayed, e.g., "Out Steakhouse" and "Chip Mexican Grill".

In some embodiments, upon unlocking the electronic device, the user interface object is available (1626) in response to a swipe in a generally horizontal direction (e.g., swipe 1104-1 from left to right as shown in fig. 11C) on the initial page of the home screen of the electronic device.

It should be understood that the particular order in which the operations have been described in fig. 16A-16B is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., methods 600,800, 1000, 1200, and 1400) also apply in a similar manner to method 1600 described above in connection with fig. 16. For example, the user interface objects and/or operations described above with reference to method 1600 optionally have one or more of the features of the user interface objects and/or operations described herein with reference to other methods described herein (e.g., methods 600,800, 1000, 1200, and 1400). In some embodiments, any of the relevant details of sections 1-11 can be used for any suitable purpose in connection with method 1600. For the sake of brevity, these details are not repeated here.

Fig. 18A-18B are a flowchart representation of a method 1800 of extracting and interacting with content items from voice communications, according to some embodiments. Fig. 19A-19F are used to illustrate the methods and/or processes in fig. 18A-18B. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 1800 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 1800 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 1800 performed by the apparatus 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 1800 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 1800 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 1800 provides an intuitive way to extract content items from voice communications and present them to a user on an electronic device with a touch-sensitive display. The method reduces the amount of user input required (e.g., the device automatically extracts relevant information for contacts, locations, and events and prepares the information for storage and use on the device), thereby creating a more efficient human-machine interface and assisting the user in adding new content based on the content discussed in the voice communication. For battery-operated electronic devices, this approach both helps to conserve power and increases the time interval between battery charges.

As shown in fig. 18A, the device receives at least a portion of the voice communication, the portion of the voice communication including voice provided by a remote user of the remote device, the remote user being different from a user of the electronic device (1801). In some embodiments, the voice communication is a real-time telephone call, a real-time video call (e.g., FaceTime call), or a recorded voice mailbox (1803). In some embodiments, the voice communication is a real-time telephone call (or FaceTime call) between the user and the remote user, and thus the voice communication includes voice provided by both the user and the remote user. In other embodiments, the voice communication is a recorded voice mailbox sent by the remote user to the user, the recorded voice mailbox is transmitted from the remote device to the electronic device over the telecommunications network, and the recorded voice mailbox is then stored on the electronic device for later playback.

In some embodiments, the portion of the voice communication is identified based on an instruction from a user of the electronic device (1805). For example, based on the user's selection of a hardware button, the electronic device user marks the portion for analysis (e.g., the user taps a hardware button such as a volume button, in response to which the device begins analyzing a predefined amount of voice communication (e.g., the previous 10,9,8, or 7 seconds) to detect/extract the content item). In some embodiments, the button may also be a button presented on the electronic device display for selection by the user (e.g., a button displayed on a user interface similar to that shown in fig. 21B during a voice communication that includes the text "tapping here to analyze the voice communication to obtain new content").

In some embodiments, the instruction from the user corresponds to a spoken command that includes the phrase "hey, Siri" (e.g., "hey, Siri, please save," or "hey, Siri, please remember," or "hey, Siri, please grab the event details just mentioned," etc.). In some embodiments, the verbal instruction from the user is any predefined phrase that causes the device to begin analyzing the voice communication to detect new content (e.g., the phrase may be in some other language than english, or the phrase may include a different word, such as "Siri, please analyze this call" or "Siri, please begin analyzing" or a phrase expressing a similar meaning).

In some embodiments, the device does not record or maintain any portion of the voice communication in persistent memory, rather, the device only analyzes that portion of the voice communication (e.g., 10 seconds at a time), then immediately deletes all recorded data, and only saves the content items extracted based on the analysis (as discussed in more detail below). In this way, the extracted content item is made available to the user without storing the actual content of the voice communication, thereby helping to protect the privacy of the user.

In some embodiments, the device analyzes the portion of the voice communication (e.g., the portion of the recorded voicemail tagged by the user, or the portion of the real-time telephone call between the user of the device tagged by the user and another remotely located user of the device, or the portion automatically recognized by the device as including new content for retrieval) to detect a predetermined type of content (1807). In some embodiments, analyzing the voice communication comprises: converting the speech provided by the remote user (and, if applicable, by the electronic device user) to text; applying a natural language processing algorithm to the text to determine whether the text includes one or more predefined keywords; and in accordance with a determination that the text includes the respective predefined keyword, determining that the voice communication includes speech describing the content item (1809).

In other words, the voice communication is passed through a speech-to-text processing algorithm, natural language processing is performed on the text produced by the speech-to-text processing, and the electronic device then determines whether the text includes any one or more predefined keywords. In some embodiments, automatic speech recognition algorithms are utilized (e.g., to assist in performing speech-to-text and natural language processing operations). In some embodiments, the one or more predefined keywords comprise a data detector for identifying key phrases/strings in the text and for providing a suggested output (e.g., the optional description introduced above). In some embodiments, this entire process is performed on the electronic device (converting speech to text and processing the text to detect new content) and does not use a server or any external device to assist in performing these operations, and in this way, the privacy of the user is maintained and protected. In some embodiments, a circular buffer (e.g., a small circular buffer comprising 10 seconds or less of voice communication) is used in analyzing voice communications, and the data in the circular buffer is used to store and transcribe voice, privacy is also protected as the entire conversation is not recorded, monitored, or stored. In this way, the device can quickly and efficiently process voice communications to detect new events, new contact information, and other new content items.

In some embodiments, for certain types of content that may be extracted from voice communications (e.g., telephone numbers), instead of or in addition to searching for the one or more predefined keywords, the device also checks whether the text generated by the natural language processing algorithm includes a predefined number of bits (e.g., 10 or 11 bits for U.S. telephone numbers). In some embodiments, both techniques are used simultaneously (e.g., the device searches for a predefined keyword such as "phone number" and then shortly thereafter searches for a predefined number of digits in the text in order to locate the referenced phone number).

In some embodiments, the analysis is performed while outputting a voice communication through an audio system in communication with the electronic device (e.g., operations 1807 and 1809). In some embodiments, the predetermined type of content includes information content discussed in voice communications and related to contacts, events, and/or location information (more details regarding detecting and extracting location information from voice communications are provided below). For example, analyzing the voice communication to detect the predetermined type of content includes analyzing to detect new contact information (including new contact information for contacts and existing contacts) and new events (or content related to modifying existing events). In some embodiments, the audio system is an internal speaker of the device, an external headset, or an external audio system, such as a speaker or a stereo system of the vehicle.

In some embodiments, the device extracts the content item (1811) based at least in part on speech provided by a remote user of the remote device (e.g., speech that identifies or describes the content item, such as details about the upcoming event (start time, end time, location, attendees, etc.), contact information (phone number, contact name, employer name, etc.), restaurant name, phone number, route to point of interest, and other details that may be used to extract the content item from the speech). In some embodiments, the content item is also extracted based at least in part on speech provided by the user of the electronic device (e.g., two users are discussing event details and the device extracts these event details based on speech provided by the two users) (1815).

In some embodiments, the content item is a new event detailing an event currently associated with a calendar application on the electronic device; the new contact, the new contact information detailing existing contacts associated with the phone application on the electronic device (1813).

In some embodiments, the electronic device determines whether the content item is currently available on the electronic device (1817).

Turning now to FIG. 18B, in accordance with a determination that the content item is not currently available on the electronic device, the electronic device identifies an application associated with the content item and displays an optional description of the content item on the display (1821). Fig. 19A illustrates an exemplary user interface in which an optional description 1902 is displayed when a user is currently engaged in a voice communication (e.g., a real-time telephone call). As shown in fig. 19A, the selectable descriptions 1902 include an icon of the identified associated application (e.g., an icon of a calendar application), a description of the content item (e.g., text indicating that a new event was found on this phone call), and details about the content item (e.g., event details associated with the new event).

In some embodiments, displaying the selectable description includes displaying the selectable description in a user interface, wherein the user interface includes recent calls made using the phone application (1823). In some embodiments, the user interface including the most recent call is displayed after the voice communication is completed (i.e., the user first displays the selectable description 1902 when the call is being made and then displays the user interface including the most recent call when the call is ended). For example, FIG. 19B illustrates an exemplary user interface including selectable depictions 1901,1903 and 1905 of content items extracted from voice communications. In particular, the optional description 1901 indicates that a new event is found on the first phone call, the optional description 1903 indicates that new contact information is found on the second phone call, and the optional description 1905 indicates that a location is found on the third phone call. As described above, the voice communication may also be a recorded voice mailbox, and thus, the user interface illustrated in FIG. 19B may also be displayed in the voice mailbox tab of the telephone application.

In some embodiments, the optional description is displayed with the following indication: the content item is associated with a voice communication. For example, each of the selectable descriptions 1901-1905 is displayed in proximity to the voice communication from which they were extracted, providing the user with a clear indication of the respective voice communication associated with each extracted content item.

In accordance with a determination that the content item is not currently available on the electronic device, the electronic device also provides feedback to the user that a new content item has been detected (1825). In some embodiments, feedback is provided in connection with displaying the selectable description (i.e., displaying and providing feedback in a substantially simultaneous manner, such that a user is able to receive haptic feedback that then guides them to view a display that displays the selectable description 1902 during the voice communication). In some embodiments, providing feedback includes sending information related to the detection of the content item to another electronic device in the vicinity of the electronic device (e.g., sending information to a nearby laptop or watch so that the user does not have to remove the phone from the ear to view detailed information about the new content item detected) (1827).

In some embodiments, in response to detecting selection of the selectable description (e.g., a user input provided at a user interface shown in either of fig. 19A or 19B), the electronic device stores the content item for presentation by the identified application (1829). The selectable description may be selected while the user is listening to the voice communication (e.g., by tapping on the selectable description 1902 shown in fig. 19A) or from a user interface that includes recent conversations (e.g., by tapping on the selectable description 1901 shown in fig. 19B) (1831). In response to the selection, the content item is stored by the identified application (e.g., a calendar application or a contacts application, depending on the type of content item extracted). For example, in response to selecting either of the selectable descriptions 1902 or 1901, the electronic device opens a create new event user interface and populates the create new event user interface with details extracted from the portion of the voice communication (e.g., populates the user interface shown in FIG. 19C to include a title, location, start time, end time, etc.).

As another example, in response to selecting the selectable description 1903, the electronic device opens a user interface of the contacts application (e.g., allowing new contacts to be created or new contact details to be added to existing contacts, as shown in fig. 19D-19E, respectively) and populates the user interface with details extracted from the portion of the voice communication (e.g., the user interface shown in fig. 19D includes a first name, a last name, a telephone number, an email address, etc., and the user interface shown in fig. 19E includes a new mobile phone number for an existing contact).

In some embodiments, the electronic device also detects/extracts information about the physical location mentioned or discussed during the voice communication. In particular and referring back to fig. 18B, the electronic device determines that the voice communication includes information about the first physical location (e.g., with reference to a geographic location or providing a route for reaching the first geographic location) (1833). The electronic device also detects an input (e.g., the input), and in response to detecting the input, the electronic device performs either of operation 1837 or operation 1839, depending on whether the input corresponds to a request to open an application that accepts geographic location data or corresponds to a request to search for content on the electronic device (e.g., any search activation gesture discussed herein) (1835).

In accordance with a determination that the input corresponds to a request to open an application that accepts geographic location data, the electronic device opens an application that is capable of accepting the location data and populates the application with information about the first physical location (which may be information included in or based on voice communications, such as a restaurant name discussed in a real-time telephone call or a telephone number queried by the electronic device using the restaurant name) (1839). For example, as shown in fig. 19F, the application is a mapping application, and populating the mapping application with information about the first physical location includes: the map displayed within the mapping application is populated with the location identifier corresponding to the first physical location (or, as shown in fig. 19F, multiple location identifiers for each physical location discussed/extracted during the voice communication).

In accordance with a determination that the input corresponds to a request to enter a search mode, the electronic device populates a search interface with information about the first physical location (which may be information included in or based on voice communications, such as a restaurant name discussed in a real-time phone call or a phone number queried by the electronic device using the restaurant name) (1837). For example, the search interface discussed above with reference to FIG. 13B can be populated to include information about the first physical location as one of the suggested searches 1150 (e.g., by receiving a request through a phone application).

In some embodiments, the voice communication may include speech (from a single user or multiple users that are each speaking during the voice communication) that describes multiple various content items (e.g., new contacts or new contact information for existing contacts, multiple physical locations, multiple details about new events or existing events, or a combination thereof), and the electronic device is configured to ensure that each of these content items is extracted from the voice communication. For example, method 1800 also includes causing the electronic device to receive a second portion of the voice communication (e.g., the second portion including voice provided by one or more of a remote user of the remote device and a user of the electronic device). In some embodiments, the electronic device extracts the second content item based at least in part on speech provided by a remote user of the remote device and speech provided by a user of the electronic device. In accordance with a determination that the second content item is not currently available on the electronic device, the electronic device identifies a second application associated with the second content item and displays a second selectable description of the second content item on the display (e.g., if multiple content items are extracted from each associated voice communication and if applicable, the user interface illustrated in fig. 19A may include more than one selectable description 1902, and/or the user interface illustrated in fig. 19B may include more than one selectable description 1901,1903 or 1905). In response to detecting selection of the second selectable description, the electronic device stores the second content item for presentation by the identified second application (as discussed above with reference to the first content item).

In some embodiments, upon selecting the selectable description or the second selectable description, the electronic device ceases to display the corresponding selectable description in the user interface including the most recent call. In some embodiments, each selectable description is also displayed with a delete affordance (e.g., "x") that, when selected, will cause the electronic device to cease displaying the corresponding selectable description (such as the selectable descriptions shown in fig. 19A and 19B).

It should be understood that the particular order in which the operations have been described in fig. 18A-18B is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with reference to other methods described herein (e.g., method 2000) also apply in a similar manner to method 1800 described above with reference to fig. 18A-18B. For example, the operations described above with reference to method 1800 are optionally implemented or incorporated herein with reference to the operations described herein with respect to other methods (e.g., method 2000). In addition, in section 4: the following details provided in the "structured suggestions" may also be used in conjunction with method 2000 (e.g., the details discussed in section 4 regarding detecting contact and event related information in a message may also be used to extract the same information from a voice communication). In some embodiments, any relevant details of sections 1 through 11 may be used in conjunction with method 1800 for any suitable purpose. For the sake of brevity, these details are not repeated here.

In some embodiments, the techniques described with reference to method 1800 above and method 2000 below are also used to detect other types of content that may be extracted from voice communications. For example, a telephone number may be extracted and presented to the user for storage as contact information (e.g., for a new contact or an existing contact) or for immediate use (e.g., the user makes a telephone call and hears an answer message that includes the new telephone number, and in response to detecting that the message includes this new telephone number, the device presents the telephone number on a user interface such as that shown in fig. 21B so that the user can quickly and easily dial the new telephone number).

In some embodiments of methods 1800 and 2000, tactile feedback is provided whenever a device detects new content (e.g., location, phone number, contact information, or any other content) to provide a user with an explicit indication that new content is available

Fig. 20 is a flowchart representation of a method of determining voice communications including recognizing voice of a physical location and populating an application with information about the physical location, according to some embodiments. Fig. 19A-19F and 21A-21B are used to illustrate the method and/or process of fig. 20. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 2000 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, the method 2000 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, method 2000 is described below as being performed by device 100. In some embodiments, referring to fig. 1A, the operations of method 2000 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 2000 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 2000 provides an intuitive way to extract content items from voice communications and present them to a user on an electronic device with a touch-sensitive display. The method reduces the amount of user input required (e.g., the device automatically extracts relevant information about the physical location and prepares the information for storage and use on the device), thereby creating a more efficient human-machine interface and assisting the user in invoking the information about the physical location based on what is discussed in the voice communication. For battery-powered electronic devices, this approach both helps to conserve power and increases the time interval between battery charges.

As shown in fig. 20, the device receives at least a portion of the voice communication, the portion of the voice communication including voice provided by a remote user of the remote device, the remote user being different from a user of the electronic device (2001). In some embodiments, the voice communication is a real-time telephone call, a real-time video call (e.g., FaceTime call), or a recorded voice mail box (2003). Further details regarding examples of voice communications (and their associated portions) are provided with reference to fig. 18A-18B. In some embodiments, the portion of the voice communication is identified based on an instruction received from a user of the electronic device (2005). More details regarding examples of instructions received from a user are provided above with reference to fig. 18A-18B (e.g., the instructions may correspond to a selection of a hardware button or a spoken command from a user).

In some embodiments, the device analyzes the portion of the voice communication to detect information about the physical location, and performs the analysis when the voice communication is output through an audio system in communication with the electronic device (2007). In some embodiments, the audio system may be an internal speaker of the device, an external headset, an external audio system, such as a speaker, or a stereo system of the vehicle. More information about this analysis operation 2007 and other speech-to-text processing examples is provided above, and these techniques are also applicable to detecting physical location.

In some embodiments, the electronic device determines that the voice communication includes recognizing voice of the physical location (2009). In some embodiments, identifying the voice of the physical location includes talking about a driving route to a particular point of interest, talking about a restaurant (or other point of interest) name, and so forth. In some embodiments, the physical location may correspond to any point of interest (such as a restaurant, a residence, an amusement park, etc.), and the speech used to identify the physical location may include speech that refers to a street address, speech that refers to location information (GPS coordinates, latitude/longitude, etc.) of the physical location, and other related speech that provides information that may be used to locate the physical location on a map (via a device). In some embodiments, a physical location is also referred to as a named location or a physically addressable location.

In some embodiments, in response to determining that the voice communication includes recognizing voice of the physical location, the electronic device provides an indication that information about the physical location has been detected (e.g., the device provides tactile feedback and/or displays a UI object for selection, such as user interface object 2101 or 2103 shown in fig. 21A and 21B, respectively) (2011). In some embodiments, providing the indication includes displaying a selectable description of the physical location within a user interface, wherein the user interface may be: a user interface that includes recent calls made using a phone application (e.g., the optional description 1905 shown in fig. 19B), a user interface associated with voice communications (e.g., the optional descriptions 2101 and 2103 shown in fig. 21A-21B, respectively), or both of such user interfaces (e.g., within a user interface associated with voice communications when voice communications are ongoing; within a user interface that includes recent calls when voice communications are over) (2013). In some embodiments, the selectable description indicates that the content item is associated with a voice communication, e.g., the selectable description is displayed below an identifier of the voice communication (as shown in fig. 19B), or the selectable description is displayed in a user interface associated with the voice communication (as shown in fig. 21A-21B).

In some embodiments, providing the indication includes providing haptic feedback to a user of the electronic device (2015).

In some embodiments, providing the indication includes sending information related to the physical location to another electronic device in the vicinity of the electronic device (e.g., sending information to be presented on a nearby laptop or watch, thereby eliminating the need for the user to remove the phone from the ear to view detailed information about the detected new content item) (2017).

In some embodiments, the electronic device detects an input through the touch-sensitive surface (e.g., entering a request corresponding to opening an application accepting geographic location data (received at a later time after the voice communication ends), or entering a selection corresponding to an optional description of a physical location displayed during or after the voice communication), and in response to detecting the input, the device opens the application accepting the geographic location data and populates the application with information about the physical location (2019).

In some embodiments, detecting the input includes detecting an input on the selectable description (e.g., selecting or tapping the selectable description 1905 shown in fig. 19A) while displaying the user interface that includes the most recent call. For example, in response to detecting a touch on the selectable depiction 1905 shown in fig. 19A, the electronic device opens a mapping application (or an application capable of displaying map objects, such as a windmilling application) and populates the mapping application with information about the physical location (e.g., a pin to identify the physical location, as shown in fig. 19F).

In some embodiments, detecting the input includes detecting the input on the selectable description (e.g., selecting or tapping the selectable description 2101 or 2103 shown in fig. 21A-21B) while displaying the user interface associated with the voice communication. For example, in response to detecting a touch on the selectable depiction 2101 shown in FIG. 21A, the electronic device opens a mapping application (or an application capable of displaying map objects, such as a windmilling application) and populates the mapping application with information about the physical location (e.g., a pin to identify the physical location, as shown in FIG. 19F). As another example, in response to detecting a touch on the optional depiction 2103 (fig. 21B), the device opens a mapping application (or an application capable of providing route guidance to a physical destination) and populates the mapping application with information about the physical location (e.g., a pin for identifying the physical location and a route to the physical location as shown in fig. 19F, where the route is based on voice extractions provided during voice communications).

In some embodiments, since the application is populated in response to detection of the input, the population is performed before any additional user input is received within the application (e.g., when the map application is open, the pins are populated into the map application shown in fig. 19F without any user input in the map application). In this way, the user is presented with information about the physical location based only on information extracted from the speech during the voice communication, and the user does not provide any additional input to populate the application with that information (in other words, the application is pre-populated with that information).

In some other embodiments, the detected geographic location is stored for display in an appropriate application that is displayed whenever the user later opens the appropriate application (e.g., an application that is capable of accepting geographic location information) and thus does not provide any indication to the user during the voice communication.

It should be understood that the particular order of operations that have been described in fig. 20 is merely exemplary and is not intended to suggest that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with reference to other methods described herein (e.g., method 1800) also apply in a similar manner to method 2000 described above with reference to fig. 20. For example, the operations described above with reference to method 2000 are optionally implemented or supplemented by the operations described herein with reference to other methods described herein (e.g., method 1800). In addition, in section 4: the following details provided in the "structured suggestion" may also be used in conjunction with method 2000 (e.g., the details discussed in section 4 regarding detecting contact and event related information in a message may also be used to extract the same information from a voice communication). In some embodiments, any of the relevant details of sections 1 through 11 may be used for any suitable purpose in conjunction with method 2000. For the sake of brevity, these details are not repeated here.

Fig. 22A-22B are a flowchart representation of a method of proactively suggesting physical locations for use in an instant messaging application, according to some embodiments. Fig. 23A-23O are used to illustrate the method and/or process of fig. 22A-22B. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 2200 is performed by an electronic device (e.g., portable multifunction device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 2200 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 2200 performed by the apparatus 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 2200 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 2200 are optionally combined, and/or the order of some operations is optionally changed.

Method 2200 provides an intuitive way to proactively suggest a physical location for an instant messaging application on an electronic device with a touch-sensitive display, as described below. The method reduces the number of user inputs required to add information about the physical location in the instant messaging application, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, there is an active suggestion for a physical location for an instant messaging application that both conserves power and increases the time interval between battery charges (e.g., by avoiding a user from searching for this information unintentionally before entering the instant messaging application, thereby saving time and eliminating energy-consuming operations).

As shown in fig. 22A, the electronic device presents the text entry field and the copy of the session in an instant messaging application on the display (e.g., an email or iMessage application on a desktop computer, laptop computer, smart phone, or smart watch) (2201). In some embodiments, the conversation transcript includes messages (such as email messages, text messages, audio messages, video messages, picture messages, etc.) exchanged between one or more users. In some embodiments, the conversation transcript includes a text entry field (e.g., as shown in FIG. 23A, the conversation transcript 2301 includes text that the user typed in when drafting a new email response). In some embodiments, the conversation transcript and the text entry field are separate (e.g., as shown in fig. 23C, the conversation transcript 2303 is generally positioned above a separate text entry field 2305 in which the user can draft a new message). In some embodiments (e.g., where the electronic device is in communication with a display and the device remain physically separate such as a desktop computer or smart television device), presenting includes causing the display to present relevant content (e.g., the device provides information to the display so that the display can present text entry fields and a copy of the session (and other user interface elements discussed below)).

When the instant messaging application is presented on the display, the electronic device determines that the next possible input from the user of the electronic device is information about the physical location (2203) (e.g., the address or the device-determined current location of the user). In some embodiments, determining that the next possible input from the user of the electronic device is information about the physical location comprises: processing content associated with the text entry field and the transcript of the conversation to detect that the transcript of the conversation includes a question 2205 about the user's current location (e.g., the second user sends a message asking the user "where you are" as shown in fig. 23A-23B and 23C-23D). In some embodiments, processing the content includes applying a natural language processing algorithm to detect one or more predefined keywords that constitute the question. In some embodiments, the one or more keywords may be searched by the electronic device directly in content associated with the text entry field and the transcript of the conversation, while in other embodiments, the one or more keywords are detected by performing semantic analysis to find phrases similar to the one or more keywords (e.g., words that are a distance away from the meaning of the phrase), and in some embodiments, both techniques are used simultaneously. In some embodiments, the question is included in a message received from a second user different from the user (2209) (as shown in fig. 23A-23B and 23C-23D).

In some embodiments, the electronic device analyzes content associated with the text entry field and the conversation transcript, and determines a suggested physical location based at least in part on a portion of the analyzed content (e.g., content from a most recently received message) (2211). In some embodiments, the suggested physical location corresponds to a location that the user recently viewed in an application other than the instant messaging application (2213). (e.g., the user begins typing "we should eat dinner at [ automatically insert the most recently viewed address ]), for example, the user previously searched for restaurants using a review search application (such as that shown in FIG. 25A), and the device then uses information based on searching for restaurants in the review search application to identify suggested physical locations.

In some embodiments, the electronic device presents a selectable user interface element in the instant messaging application on the display for identifying the suggested physical location (2215). For example, the instant messaging application includes a virtual keyboard, and the selectable user interface elements are displayed in a suggestion portion adjacent to and above the virtual keyboard (2217). As shown in FIG. 23A, suggestion portion 2307 includes an optional user interface element that, when selected, will cause the device to include the user's current location in the text entry field (as shown in FIG. 23B). In some embodiments, selecting 2307 the selectable UI element shown will cause the device to send the user's current location to the remote user immediately in a new message.

Turning now to fig. 22B, in some embodiments, the electronic device receives a selection of a selectable user interface element (2219). In response to receiving the selection, the electronic device presents a representation of the suggested physical location in the text entry field (2221). In some embodiments, the representation of the proposed physical location includes information (2223) identifying the current geographic location of the electronic device (e.g., GPS information retrieved from a location sensor of the electronic device identifying the current geographic location, and this information is subsequently presented in the representation (as shown in fig. 23B and 23D)). As shown in fig. 23B and 23D, in some embodiments, the representation of the suggested physical location is a map object that includes an identifier for the suggested physical location (2227).

In some embodiments, the representation of the proposed physical location is an address (2225). For example, referring to FIG. 23E, in response to detecting selection of a selectable user interface element shown in suggestion portion 2307, the device updates the text input field to include the address displayed in suggestion portion 2307. In some embodiments, the address may correspond to the user's own address (home, work, etc.), the address of a contact stored in the device (as shown in fig. 23G-23H), the address that the user has recently viewed on the electronic device (e.g., the restaurant location viewed in some other application, as shown in fig. 23F), the address sent to the user in this or other session copy, or an address shared to the user by other users (e.g., via email, social networking application, etc.).

In some embodiments, upon determining that the user is typing (i.e., that the user is continuously entering text into the instant messaging application, such as through the virtual keyboard shown in fig. 23E) and has not selected a selectable user interface element (e.g., after a predefined period of time such as 2 seconds, 3 seconds, 4 seconds, where it is reasonably determined that the user will not select a selectable user interface element), the device ceases to present selectable user interface elements (2229). In some embodiments, once the user begins typing, the device ceases to present the selectable user interface elements.

In some embodiments, determining that the next possible input from the user of the electronic device is information about the physical location comprises: a text input portion of the instant messaging application is monitored for a typed input received from a user. In such embodiments, the method further comprises: while monitoring typing inputs, determining whether there is any typing input matching one or more trigger phrases, each trigger phrase associated with a respective content item; in accordance with a determination that the order of typing the input matches the first trigger phrase, displaying a suggested content item associated with the first trigger phrase on the display; and detecting a selection of the suggested content item, in response to detecting the selection, displaying information about the suggested content item in a text entry portion of the instant message application. In some embodiments, in accordance with a determination that the user has provided additional input indicating that the user will not select a selectable user interface element (e.g., consecutive keystrokes no longer match the trigger phrase), the electronic device stops presenting the selectable user interface element (2231).

In some embodiments, in accordance with a determination that a predetermined period of time (e.g., 10 seconds) has elapsed since the first display of the selectable user interface object, the device ceases to present the selectable user interface object.

In some embodiments, the techniques associated with method 2200 may also be available via other types of applications (in addition to instant messaging applications, such as document writing applications) and for other object types (in addition to physical location, such as contacts and events). For example, as shown in fig. 23I and 23J, some embodiments also enable the electronic device to proactively suggest an availability window for scheduling an event (discussed in more detail below with reference to fig. 22C and method 2280). Additionally, as shown in fig. 23K-23J, some embodiments also enable the electronic device to proactively suggest contact information (such as a phone number of a user or a contact stored on the device), or proactively suggest an appropriate response based on a previous conversation (e.g., as shown in fig. 23M) or proactively suggest an appropriate reference (e.g., as shown in fig. 23O). Method 2280 below also provides some more details regarding other types of applications and other object types. In some embodiments, various aspects of methods 2200 and 2280 are combined, exchanged, and/or interchanged.

It should be understood that the particular order in which the operations have been described in fig. 22A-22B is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., method 2280 and method 2900) also apply in a similar manner to method 2200 described above in connection with fig. 22A-22B. For example, the operations described above with reference to method 2200 optionally include one or more operations or features of other methods described herein (e.g., methods 2280 and 2900). In some embodiments, any relevant details of sections 1 through 11 may be used in conjunction with method 2200 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 22C is a flowchart representation of a method of proactively suggesting information related to a location, event, or contact, according to some embodiments. Fig. 23A-23O are used to illustrate the method and/or process in fig. 22C. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 2280 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 2280 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 2280 performed by the device 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 2280 are performed at least in part by or using: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 2280 are optionally combined, and/or the order of some operations is optionally changed.

As described below, method 2280 provides an intuitive way to proactively suggest information related to a location, event, or contact on an electronic device with a touch-sensitive display. The method reduces the number of user inputs that are intended to locate information about contacts, locations or events and input that information for use in an application, thereby creating a more efficient human-machine interface. For battery-powered electronic devices, proactively suggesting information about a location, event, or contact may improve user satisfaction with the electronic device (by automatically invoking the information and presenting it to the user at the relevant time for immediate use), conserve power, and increase the time interval between battery charges.

As shown in fig. 22C, the electronic device presents textual content associated with the application on the display (2281). In some embodiments, the application is a document writing application (e.g., notepad application, word processing application, etc.) or an instant messaging application (such as an email or text messaging application) or any other application in which a virtual keyboard is displayed to enter text into an input-receiving field.

In some embodiments, the device determining a portion of the textual content involves (2283) (or reference to the portion of the textual content) the following: (i) location (e.g., current location information that may be obtained through a location sensor of the electronic device), (ii) contact (e.g., information that may be obtained through a contacts application on the electronic device), or (iii) event (e.g., information that may be obtained through a calendar application on the electronic device). In some embodiments, the portion of the textual content is a statement/question that is best accomplished by information about a location, contact, or event (e.g., such as the examples shown in fig. 23A-23O). In some embodiments, the portion of the text content is for the most recently presented text content in the application (2285) (such as text content typed by the user or text content in a message received from a remote user). For example, the portion is the current text that the user types in the notepad or instant messaging application (e.g., "I am at present" in FIG. 23A, "I am's address is" in FIG. 23E, "John's address is" in FIG. 23H, "I am available" in FIG. 23I, "My phone number is" in FIG. 23K, "make I phone call" in FIG. 23L, and "what type of tumor" in FIG. 23M). In other words, the portion of the text content is an input (i.e., a series of typed inputs) provided by the electronic device user in an input-receiving field within an application (e.g., field 2305 of the instant messaging application shown in fig. 23C, or field 2301 of the email application shown in fig. 23A) (e.g., the user provides a series of typed inputs on a virtual keyboard or adds text to an input-receiving field using dictation).

In some embodiments, the portion of the textual content is a message in the transcript of the conversation that was most recently received from some other user. For example, the application is an instant messaging application, and the portion of text content is a question received in the instant messaging application from a remote user of a remote device (e.g., "are you there.

In some embodiments, upon determining that the portion of the textual content is associated with (i) location (2289), (ii) contact (2291), or (iii) event (2293), the electronic device in turn identifies an appropriate content item available on the electronic device (in some embodiments, without having to retrieve any information from the server) and presents the content item to the user for use in the application (e.g., to respond to a question or effectively complete the user's own typing input). In this way, a user can quickly and easily include information about contacts, events, and locations in an application without having to leave the current application, search for appropriate content, copy or remember the content, return to the current application, and then include the content in the current application (thereby reducing the number of inputs required by the user to include information about contacts, events, and locations in the application).

More specifically, as for (i) when it is determined that the portion of the text content is associated with a location, the electronic device acquires location information from a location sensor on the electronic device and prepares the acquired location information to be displayed as a predicted content item (2289). For example, in a message received from a remote user, the phrase "where are you included? "which the device determines is location dependent, then the device obtains information from a GPS sensor on the device and prepares the information for presentation as a predicted content item (see fig. 23D, where a map object including the user's current location is sent to the remote user). As another example, as the user types in a new email, based on the portion of the text that includes the phrase "i am present" (as shown in fig. 23A), the device determines that the portion is relevant to location, then the device obtains information from a GPS sensor on the device, and prepares the information for presentation as a predicted content item (see fig. 23B, where a map object that includes the user's current location is included in the new email the user is preparing). Further examples are shown in fig. 23E (e.g., the device determines that the portion of the text content includes information related to location based on the user typing "my address is") and fig. 23F (e.g., the device determines that the portion of the text content includes information related to location based on the user receiving a message including the text "where restaurant" for example). As shown in fig. 23F, in some embodiments, the device obtains location information based on the user's previous interactions with a different application (e.g., the user searches for a restaurant application in a different application, such as an application that provides crowd-sourced reviews, and therefore does not use a location sensor to obtain the information). Further details regarding sharing information between two different applications are discussed below with reference to methods 2400, 2500, and 2800, and for the sake of brevity, are not repeated here.

As to (ii) upon determining that the portion of the textual content is related to a contact, the electronic device searches on the electronic device for contact information related to the portion of the textual content and prepares information associated with at least one contact retrieved by the search for display as a predicted content item (2291). For example, the portion of the text content is "what is the address of john? "(FIG. 23G)," John's address is "(FIG. 23H), or" My phone number is "(FIG. 23K), or" make me phone call "(FIG. 23L), the device analyzes the contact information stored in the contact application to retrieve contacts predicted to be responsive to the portion, and provides the retrieved contact information as a predicted content item.

As to (iii), upon determining that the portion of textual content is relevant to the event, the electronic device conducts a new search on the electronic device, searches for event information relevant to the portion of textual content, and prepares information based at least in part on the at least one event (retrieved by the new search) for display as a predicted content item (2293). In some embodiments, the information based at least in part on the at least one event may be event details (such as event time, duration, location) or information derived from event details (such as availability of a user to schedule a new event, as shown in fig. 23I and 23J). For example, the portion of the text content is "in which meeting room a meeting is held? "or" when a meeting starts? ", the device analyzes information associated with an event stored in the calendar application to retrieve information responsive to the question, and provides the retrieved information as a predicted content item.

As described above, the electronic device displays an affordance (2294) including a predicted content item within an application (e.g., displays an affordance "Add Current position" within suggestion portion 2307 of FIG. 23A, displays an affordance "Send My Current position" within suggestion portion 2309 of FIG. 23C, etc., displays other exemplary affordances within suggestion portions 2307 or 2309 of FIGS. 23E, 23F, 23G, 23H, 23I, 23J, 23K, 23L, 23M, 23N, and 23O.). The electronic device also detects selection of the affordance through the touch-sensitive surface (2295); and in response to detecting the selection, the device displays information associated with the predicted content item on the display adjacent to the textual content (2297) (e.g., in response to selecting the affordance of "add current location" (figure 23B), displaying a map object having the user's current location).

In some embodiments, the portion of the textual content is identified (2287) in response to a user input selecting a user interface object that includes the portion of the textual content. For example, the application is an instant messaging application and the user interface object is a message bubble in a conversation displayed within the instant messaging application. In this way, a user can retrieve a predicted content item for a particular portion displayed in an application program, such that if they forget to respond to a particular portion, they can select a user interface object associated with the portion in order to easily view the predicted content item for the particular portion. As a specific example, referring to fig. 23M, the portion of the text content is initially the most recently displayed text content (e.g., "what type of lesion. The device then detects a selection of a second user interface object (e.g., a second message bubble received before the most recently displayed text content, the message bubble including the text content "by the way, where are you. In response to detecting the selection, the device stops displaying the affordance with the predicted content item and determines that the textual content associated with the second user interface object is relevant to the location, contact, or event (in this example, the device determines that "do you are.

As described in the preceding paragraph, in some embodiments, the device is also capable of determining whether the portion of the text content is related to other types of information available on the electronic device (in addition to contacts, locations, and events). For example, the device can detect issues (e.g., what type of tumor) related to information discussed by the user in exchanged emails, documents written by the user or received from other users, or information from other knowledge sources. Additionally, in some embodiments, the electronic device determines a document that is responsive to a particular portion of the textual content in the application (e.g., as shown in fig. 23O, suggest two different documents as a response to "what do you know about the tumor. In some implementations, in response to selecting either of the two different documents, the device may open the respective document and allow the user to view the document and then return to the application.

In some embodiments, the affordance displayed within the suggestion portion and including the predicted content item is displayed proximate (e.g., above) a virtual keyboard within the application. For example, as shown in FIG. 23A, an affordance for "Add Current position" is displayed in a suggestion portion 2307 above the virtual keyboard.

In some embodiments, information associated with the predicted content item and displayed adjacent to the text content is displayed in an input-receiving field, and the input-receiving field is used to display typing input received at a virtual keyboard (e.g., such as a document displayed in a notepad application, or an input-receiving field displayed above a virtual keyboard in an instant messaging application, such as shown in input-receiving field 2305 of fig. 23D, where field 2305 is located above the virtual keyboard).

In some embodiments, the determining 2283 includes parsing the textual content as the application receives the textual content (e.g., as the user types in or the application receives a message) to detect storage patterns known to be associated with contacts, events, and/or locations. In some embodiments, the neural network is trained to perform detection on stored patterns and/or a finite state grammar is used for the detection, after which the electronic device passes information to a system-level service (e.g., using one or more predictive models discussed in section 9 below) to retrieve the appropriate predictive content item.

It should be understood that the particular order of operations that have been described in fig. 22C is merely exemplary and is not intended to suggest that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., method 2200 and method 2900) also apply in a similar manner to method 2280 described above in connection with fig. 22C. For example, the operations described above with reference to method 2280 optionally have one or more features described herein with reference to other methods described herein (e.g., methods 2200 and 2900), or use one or more operations described herein with reference to other methods described herein. In some embodiments, any relevant details of sections 1 through 11 can be used for any suitable purpose in conjunction with method 2280. For the sake of brevity, these details are not repeated here.

Fig. 24A-24B are a flowchart representation of a method of actively populating an application with information that a user previously viewed in another application, according to some embodiments. Fig. 25A-25J are used to illustrate the method and/or process in fig. 24A-24B. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 2400 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of the electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 2400 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 2400 performed by the device 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 2400 are performed at least in part by or using the following: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 2400 are optionally combined, and/or the order of some operations is optionally changed.

As described below, method 2400 provides an intuitive way to populate an application on an electronic device with a touch-sensitive display with information that the user previously viewed in another application. This approach reduces the amount of user input required to use information from a first application in a different second application, thereby creating a more efficient human-machine interface. For battery-driven electronic devices, actively using information that a user previously viewed in another application to populate an application both conserves power and increases the time interval between battery charges.

As shown in fig. 24A, while displaying a first application, an electronic device obtains information identifying a first physical location viewed by a user in the first application (2401). For example, the first application is a foreground application currently displayed on the touch-sensitive display (e.g., the first application is an application that provides crowd-sourced comments such as those shown in fig. 25A). In some embodiments, obtaining comprises sending information about the location data to an operating system component of the electronic device, or obtaining comprises using an auxiliary function to obtain the information. Details regarding the use of accessibility functionality to obtain information are provided above (see, e.g., the description provided above with respect to method 1800, and in particular, the description provided above with respect to operations 1807 and 1809).

In some embodiments, the electronic device exits the first application (2403) (e.g., the user taps the Home hardware button to request to exit the first application and view the Home screen, or the user double-clicks the Home hardware button to request to exit the first application and view the application toggle user interface). After exiting the first application, the electronic device receives a request from a user to open a second application different from the first application (2405). In some embodiments, receiving the request to open the second application comprises: after exiting the first application, an input on the affordance of the second application is detected (2407) (in other words, the request does not correspond to clicking on a link within the first application). For example, the user selects the second application from the home screen (e.g., the user taps an icon (affordance) of the windmilling application displayed on the home screen shown in fig. 25B). In some embodiments, the home screen is a system level component of the operating system, including icons for calling applications available on the electronic device.

As another example, the user selects the second application from the application-switching user interface (e.g., the user taps a representation of a wind-up application included in the application-switching user interface shown in fig. 25C). More specifically, in yet another example, detecting the input includes: a double-click is detected on a physical Home button (e.g., Home 204), an application-switching user interface is displayed in response to detecting the double-click, and an affordance selection is detected from within the application-switching user interface (2411).

As an additional example with respect to operation 2405, the user selects a user interface object that when selected will cause the device to open a second application (e.g., the affordance 2503 illustrated in FIGS. 25B and 25C). In some embodiments, the request is received without receiving any input in the first application (e.g., the request does not include clicking on a link or button within the first application).

In response to receiving the request, the electronic device determines whether the second application is capable of accepting the geographic location information (2413). In some embodiments, the determining operation 2413 includes one or more of the following (2415): (i) determining that the second application includes an input-receiving field capable of accepting and processing the geographic location data; (ii) determining that the second application is capable of displaying the geographic location information on the map; (iii) determining that the second application is capable of using the geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using the geographic location information. In some embodiments, determining that the second application is capable of accepting the geographic location information comprises: it is determined that the second application includes an input-receiving field capable of accepting and processing the geographic location data, and the input-receiving field is a search box that allows searching within a map displayed in the second application. For example, the second application is a windmilling application that includes such input-receiving fields (as shown in fig. 25E, the exemplary windmilling application includes input-receiving fields 2507 that allow searching within a displayed map), or the second application is a map application that also includes such input-receiving fields (as shown in fig. 25F).

Turning now to fig. 24B, in some embodiments, in response to receiving the request, the electronic device determines whether the second application is associated with the first application based on the user's application usage history (e.g., the second application has been opened a threshold number of times after the first application was opened) and also determines that the second application is capable of accepting and processing the location data (as described above). In other words, in some embodiments, the electronic device simultaneously determines that the second application has a field that accepts location data, and that the first application and the second application are associated (by virtue of the user often opening the second application after opening the first application). In some embodiments, prior to presenting the second application, the electronic device provides the second application with access to the information identifying the first physical location, wherein the second application cannot access the information identifying the first physical location prior to providing the access (2417). For example, the second application has previously no access to information about the user's view in the first application and is only now provided with access rights, and the user can only populate the input receipt field in the second application with information identifying the first geographic location. In this way, since the device knows that the user often uses the first application and the second application together, the device is able to actively fill in the text input field without any user input (other than those used to establish the connection between the first application and the second application).

In some embodiments, in response to receiving the request and in accordance with a determination (discussed above with reference to operations 2413 and 2415) that the second application is capable of accepting geographic location information, the electronic device presents the second application, and presenting the second application includes populating the second application with information based at least in part on the first physical location identifying information. In some embodiments, populating the second application includes displaying a user interface object that includes information that identifies information based at least in part on the first physical location (2421). For example, as shown in FIG. 25D, user interface object 2505 includes information based at least in part on first physical location identifying information (e.g., a restaurant address viewed by the user in the first application, as shown in FIG. 25A). In some embodiments, the user interface object 2505 may include the name of the restaurant (e.g., "Gary Danko" instead of or in addition to an address, or the UI object 2505 may include other relevant information about the restaurant location). In some embodiments, the user interface object includes a textual description that informs the user that the first physical location was recently viewed in the first application (2423) (e.g., an icon associated with the first application is included in the user interface object 2505, as shown in fig. 25D).

In some embodiments, the user interface object is a map displayed in the second application (e.g., the map shown in fig. 25D), and populating the second application includes populating the map to include the identifier of the first physical location (2425). In some embodiments, the electronic device uses the name of the first physical location, the phone number of the first physical location, the address of the first physical location, or other information identifying the first physical location (and allowing searching) to find a particular geographic location and populate the particular geographic location into the second application. In some embodiments, the second application presents a virtual keyboard and displays user interface objects (2427) over the virtual keyboard (e.g., user interface object 2505 is displayed over the virtual keyboard as shown in fig. 25D).

In some embodiments, obtaining the information includes (2429) obtaining information about the second physical location, displaying the user interface object includes displaying the user interface object with information about the second physical location (e.g., the map includes identifiers of the first physical location and the second physical location), and/or the affordance includes information about the first physical location and the second physical location. In some embodiments, receiving the request (e.g., operation 2405) includes receiving a request to open a second application having information related to one of the first physical location or the second physical location (e.g., displaying a user interface object 2503 such as shown in fig. 25G and 25H, and the user being able to select either of the physical locations previously viewed in the first application).

In some embodiments, a user's search within the mapping application may also be used to obtain information about physical locations (e.g., the first and second physical locations discussed above). As shown in fig. 25F, a user may search for a location and receive a plurality of search results, including results 2511A, 2511B, 2511C, and 2511D. In some embodiments, the user can select one of the results, such as 2511A shown in fig. 25F, and then highlight the location on the map. After conducting the search, the user may be presented with the option to utilize the physical location as part of the search results (e.g., as shown in FIG. 25G, user interface object 2505 presents the option to ride up the wind to any location using information based on at least two physical locations). In some embodiments, the user interface object 2505 shown in fig. 25G is also accessible via an application switch user interface (as shown in fig. 25H). In some embodiments, in response to receiving a selection of one of the physical locations shown in user interface object 2505 (e.g., a toggle of applications or home screen of fig. 25G or 25H), the user is taken to the appropriate application (e.g., the wind-up application shown in fig. 25I) and the application is populated with information based on the selected physical location (e.g., user interface object 2505 is shown in fig. 25I and includes an address).

The shared location data is used as a primary example to illustrate the above-described method 2400, but the same methods and techniques described above are also applicable to sharing other types of data between two different applications. For example, the techniques described above with reference to method 2400 can also facilitate sharing search queries between a social networking application (e.g., Facebook) and a social sharing application (e.g., Twitter). For example, after a user searches for a name in Facebook, the user is provided with a suggestion to search for the same name in Twitter. As another example, a list of meeting attendees of the upcoming meeting may be shared between the calendar application and the email application such that if the user views the upcoming meeting in the calendar application and then switches to use the email application and clicks on the "compose" button, the recipient list of the new email will be populated to include the list of meeting attendees of the upcoming meeting.

It should be understood that the particular order in which the operations have been described in fig. 24A-24B is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., method 2600 and method 2700) also apply in a similar manner to method 2400 described above in connection with fig. 24A-24B. For example, the operations described above with reference to method 2400 optionally have one or more features of, or incorporate the operations described herein with reference to, other methods described herein (e.g., methods 2600 and 2700). In some embodiments, any relevant details of sections 1-11 may be used in conjunction with method 2400 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 26A-26B are flow diagram representations of proactively suggesting that suggestion information previously viewed by a user in a first application be used in a second application, according to some embodiments. Fig. 25A-25J are used to illustrate the method and/or process of fig. 26A-26B. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 2600 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any one of computing devices a-D of fig. 1E) and/or one or more components of an electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 2600 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, the method 2600 performed by the device 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 2600 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 2600 are optionally combined, and/or the order of some operations is optionally changed.

As described below, method 2600 provides an intuitive way to proactively suggest, on an electronic device with a touch-sensitive display, information that a user previously viewed in a first application for a second application. The method creates a more efficient human-machine interface by invoking useful information for the user and retrieving that information without requiring the user to perform multiple inputs. For battery-driven electronic devices, the user is actively advised of information previously viewed in a first application for a second application to both save power and increase the time interval between battery charges.

As shown in fig. 26A, the electronic device obtains information identifying a first physical location viewed by a user in the first application (2601). The details described above with reference to operation 2401 also apply to operation 2601. The electronic device detects a first input (2603). In some embodiments, the first input corresponds to a request to open an application toggle user interface (2605) (e.g., the first input is a double-click on a physical Home button of the electronic device). In some embodiments, the first input corresponds to a request to open a home screen of the electronic device (2607). (e.g., the first input is a single tap on the Home button of the electronic device). In some embodiments, the first input is an input that causes the device to at least partially exit or switch the application.

In response to detecting the first input, the electronic device identifies a second application that is capable of accepting geographic location information (2609). In some embodiments, identifying that the second application is capable of accepting the geographic location information includes one or more of the following (2611): (i) determining that the second application includes an input-receiving field capable of accepting and processing the geographic location data; (ii) determining that the second application is capable of displaying the geographic location information on the map; (iii) determining that the second application is capable of using the geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using the geographic location information. In some embodiments, identifying that the second application is capable of accepting the geographic location information comprises: it is determined that the second application includes an input-receiving field capable of accepting and processing the geographic location data, and that the input-receiving field is a search box that allows searching within a map displayed in the second application.

In response to detecting the first input, the electronic device presents an affordance different from the first application on at least a portion of the display (in addition to identifying the second application) and suggests opening the second application with information about the first physical location (2613). For example, if the first input corresponds to a request to open a home screen, the electronic device is presented on a portion of the home screen (2617) (e.g., an affordance 2505 is displayed on top of the home screen as shown in fig. 25B and 25G). As another example, if the first input corresponds to a request to open an application-switching user interface, the electronic device presents an affordance (2615) within the application-switching user interface (e.g., an area on the display that is below an application representation executing on the electronic device, such as affordance 2505 shown in fig. 25H). In some embodiments, the suggestion includes a textual description (2619) that is specific to a type associated with the second application (e.g., a description of an action to be performed in the second application using information identifying the first physical location, or a description of a search to be conducted in the second application, e.g., do you want to travel to location X.

Turning now to fig. 25B, the electronic device detects a second input at the affordance (2621). In response to detecting the second input at the affordance, the electronic device opens a second application and populates the second application to include information that is based at least in part on the first physical location identifying information (2623). Operations 2627, 2629, and 2631 correspond to operations 2423, 2425, and 2427, respectively, discussed above with reference to method 2400, and the discussion also applies to method 2600 (these details are not repeated here for brevity).

In contrast to method 2400, method 2600 will not receive a specific request from the user to open the second application before providing a suggestion to the user to open the second application with the first physical location related information. As such, by associating operations with both methods 2400 and 2600 (and using a combination of some of the processing steps from each of these methods), the electronic device can provide an efficient user experience, allowing a user to predictively use location data before or after opening an application that can accept geographic location information. Further, it is determined that the second application is able to accept geographic location information (in method 2600) even has been executed before the second application is opened, and in this way, it will only be suggested to open the application with previously viewed location information (if it is known that the application can accept location data). For simplicity, some details regarding method 2400 are not repeated herein with respect to method 2600, but such details still apply to method 2600 (such as the first application and the second application may share location data directly).

It should be understood that the particular order in which the operations have been described in fig. 26A-26B is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., methods 2400 and 2700) also apply in a similar manner to method 2600 described above in connection with fig. 26A-26B. For example, the operations described above with reference to method 2600 optionally have one or more features of the operations described herein with reference to other methods described herein (e.g., methods 2400 and 2700), or use the operations described herein with reference to other methods described herein. In some embodiments, any of the relevant details of sections 1-11 can be used for any suitable purpose in conjunction with method 2600. For the sake of brevity, these details are not repeated here.

Fig. 27 is a flowchart representation of proactively suggesting a physical location for use as a destination for route guidance in a vehicle, according to some embodiments. FIG. 28 is used to illustrate the method and/or process of FIG. 27. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 2700 is performed by an electronic device (e.g., portable multifunction device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of the electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 2700 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, method 2700 performed by apparatus 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 2700 are performed or used, at least in part, by: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 2700 are optionally combined, and/or the order of some operations is optionally changed.

As described below, the method 2700 provides an intuitive way to actively suggest physical locations on an electronic device with a touch-sensitive display for use as destinations for route guidance in a vehicle. This approach creates a more efficient human-machine interface by requiring less touch input (or no input) to use the physical location for route guidance. For battery-driven electronic devices, actively suggesting physical locations for use as a destination for route guidance in a vehicle both conserves power and increases the time interval between battery charges.

As shown in fig. 27, the electronic device obtains information identifying a first physical location, wherein the first physical location is a location viewed by a user in a first application executing on the electronic device (2701). The electronic device determines that the user has entered the vehicle (2703). In some embodiments, determining that the user has entered the vehicle includes detecting that the electronic device has established a communication link with the vehicle (2705). In other embodiments, determining that the user has entered the vehicle may include detecting whether the user is within a predetermined distance of a storage location of the vehicle, thereby prompting the user to use the first geographic location as a destination for route guidance prior to entering the automobile. In some embodiments, any of the other determinations discussed above with respect to method 1400 may also be used to prove that the user has entered the vehicle.

In response to determining that the user has entered the vehicle, the electronic device provides a prompt (e.g., in a user interface object on the device, such as user interface object 2801 shown in fig. 28, or by Siri, or both) to the user to use the first physical location as a route guidance destination (2707). In response to providing the prompt, the electronic device receives an instruction from the user instructing the user to use the first physical location as a destination for the route guidance (2709).

The electronic device then facilitates route guidance to the first physical location (2711). In some embodiments, facilitating route guidance includes providing route guidance through a display of the electronic device (2713). In some embodiments, facilitating route guidance includes sending information identifying the first physical location to the vehicle (2715). In some embodiments, facilitating route guidance includes providing route guidance (2717) through an audio system in communication with the electronic device (e.g., vehicle speakers or internal speakers of the device itself). In some embodiments, two or more of operations 2713, 2715, and 2717 are performed to ensure that the user accurately follows the route guidance.

In some embodiments, the electronic device detects that the electronic device has received a message (voice mailbox, text, email, or other social media message), including detecting that the message includes information identifying the second physical location (2719) (in some embodiments, the detecting is performed using one or more of the techniques discussed above with reference to methods 1800 and 2000). In some embodiments, detecting that the message contains information identifying the second physical location comprises: the detection is performed when a virtual assistant available on the electronic device is reading a message to the user through an audio system in communication with the electronic device (e.g., Siri reads the message through a device speaker or the vehicle's audio system).

In some embodiments, in response to the detection, the electronic device provides a prompt to the user to use the second physical location as a new destination for route guidance (2723) (e.g., the second physical location may correspond to a new meeting point, such as a restaurant location that changes while the user is driving, while in other embodiments the second physical location is not identified until after the user has reached the first physical location). In some embodiments, in response to receiving an instruction from a user to use the second physical location as a new destination, the electronic device facilitates route guidance to the second physical location (2725) (e.g., using one or more of the facilitating techniques discussed above with reference to operations 2711, 2713, 2715, and 2717).

It should be understood that the particular order of operations that have been described in FIG. 27 is merely exemplary and is not intended to suggest that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., method 2400 and method 2600) also apply in a similar manner to method 2700 described above in connection with fig. 27. For example, the operations described above with reference to method 2700 optionally have one or more features of the operations described herein with reference to other methods described herein (e.g., methods 2400 and 2600), or use the operations described herein with reference to other methods described herein. In some embodiments, any of the relevant details of sections 1-11 may be used in conjunction with method 2700 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Fig. 29 is a flowchart representation of a method of proactively suggesting a paste action, according to some embodiments. Fig. 30A-30D are used to illustrate the method and/or process of fig. 29. Although some of the examples that follow will be given with reference to input on a touch-sensitive display (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface 195 that is separate from the display 194, as shown in fig. 1D.

In some embodiments, method 2900 is performed by an electronic device (e.g., portable multi-function device 100 of fig. 1A, configured according to any of computing devices a-D of fig. 1E) and/or one or more components of the electronic device (e.g., I/O subsystem 106, operating system 126, etc.). In some embodiments, method 2900 is managed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a device, such as one or more processors 122 of device 100 (fig. 1A). For ease of explanation, method 2900 performed by device 100 is described below. In some embodiments, referring to fig. 1A, the operations of method 2900 are performed at least in part by or using: an active module (e.g., active module 163) and its components, a contact/motion module (e.g., contact/motion module 130), a graphics module (e.g., graphics module 132), and a touch-sensitive display (e.g., touch-sensitive display system 112). Some operations in method 2900 are optionally combined, and/or the order of some operations is optionally changed.

As described below, method 2900 provides an intuitive way to proactively suggest a paste action on an electronic device with a touch-sensitive display. The method may reduce user input required to perform the paste action, thereby creating a more efficient human-machine interface. For battery-driven electronic devices, the active proposal for the paste action both helps to save power and increases the time interval between two battery charges.

As shown in FIG. 29, the electronic device presents content in a first application (2901) (e.g., as shown in FIG. 30A, the device presents content corresponding to an instant messaging application including a message from a remote user "go to big time band, they are really very baseball!"). In some embodiments, the electronic device receives a request to copy at least a portion of the content (2903) (e.g., the user copied the text "big time band"). In some embodiments, no request to copy the portion of content is received (in other words, the user only views the content in the first application, and does not request to copy any content).

The electronic device receives a request from a user to open a second application different from the first application, the second application including an input-receiving field (2905) (e.g., input-receiving field 3011 shown in fig. 30C). For example, as shown in fig. 30B, the user provides an input (e.g., touch 3003) on an icon of a second application (e.g., the browser application in the example shown in fig. 30B), the input corresponding to a request to open the second application. As shown in fig. 30C, in response to receiving the request, the electronic device presents a second application having an input-receiving field (e.g., input-receiving field 3011 shown in fig. 30C) (2907).

In some embodiments, the electronic device identifies the input-receiving field as a field that is capable of accepting the portion of content (2909). In some embodiments, in response to detecting selection of an input-receiving field (e.g., a user tapping within the input-receiving field 3011 shown in fig. 30C), recognition is performed (2911). In other words, the user places focus within the first input-receiving field, which the electronic device then determines whether the portion of content that is actively copied is acceptable.

In some embodiments, prior to receiving any user input at the input-receiving field, the electronic device provides a selectable user interface object (or more than one selectable user interface object, such as the user interface object shown within the suggestion portion 3007 shown in fig. 30C) to allow the user to paste at least a portion of the content into the input-receiving field (2913). For example, the suggestion portion 3007 is populated with two suggested content items (e.g., "big time band" and "big time band video") based on the portion of content, where the suggestion portion is displayed substantially above a virtual keyboard within the second application. In response to detecting selection of the selectable user interface object (e.g., input 3009 of fig. 30C), the electronic device pastes the portion of content into the input-receiving field (2915) (e.g., "big time band video" is pasted into the input-receiving field 3011, as shown in fig. 30D). By providing such active paste functionality, the user does not need to leave the second application, reopen the first application, copy the portion from the first application, reopen the second application, and then perform a paste action in the second application. Conversely, the user need only select the selectable user interface object associated with the portion of the content that the user wants to paste, thereby greatly reducing the number of additional inputs required to perform the same paste function and enabling a more efficient and energy efficient user interface for the electronic device.

In some embodiments, the portion of content corresponds to an image, textual content, or both textual content and an image (2915). In this way, the electronic device can proactively suggest a paste action for various content types, depending on the data that the second application is able to accept.

In some embodiments, the selectable user interface object is displayed with an indication of: the portion of the content was recently viewed in the first application (e.g., suggestion portion 3007 shown in FIG. 30C includes a textual description such as "you recently viewed a piece of information about the' big time band"). In this way, the user is provided with a clear indication as to why the paste suggestion was made.

In some embodiments, the user interface object may also be presented on a home screen or a portion of an application switching user interface that provides the user with an option to enable the user to perform an action based on the content viewed in the first application. In some embodiments, the user interface object is presented before requesting the second application to be opened (operation 2905) and may be presented on the first application, home screen, or application toggle user interface. An example of a user interface object 3005 is shown in fig. 30B. Exemplary user interface object 3005 allows a user to perform a search using text presented in a first application (e.g., perform a system-wide search (e.g., Spotlight search) on "big time band" or open a particular application (such as Safari) and perform the search).

While an instant messaging application and a browser application are used as the primary examples described above, many other types of applications also benefit from the techniques associated with method 2900. For example, the first application may be a photo browsing application and the second application may be an instant messaging application (e.g., such that the active paste suggestions presented in the instant messaging application correspond to photos viewed by the user in the photo browsing application).

It should be understood that the particular order of operations that have been described in fig. 29 is merely exemplary and is not intended to suggest that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein in connection with other methods described herein (e.g., method 2200 and method 2900) also apply in a similar manner to method 2900 described above in connection with fig. 29. For example, the operations described above with reference to method 2900 optionally have one or more features of the operations described herein with reference to other methods described herein (e.g., methods 2200 and 2900), or use the operations described herein with reference to other methods described herein. In some embodiments, any relevant details of sections 1-11 may be used in conjunction with method 2900 for any suitable purpose. For the sake of brevity, these details are not repeated here.

Further details of suggested information regarding physical locations are provided below, and these details can be used to supplement methods 2200, 2280, 2900, 2400, 2600, and 2700. In some embodiments, methods 2200, 2280, 2900, 2400, 2600, and 2700 (or any other method described herein) also obtain information about: physical location (or other type of content) from a web browsing application (e.g., Safari from application INC of cupertino, california) where the user views, an address that the user has copied (e.g., to a sticky pad), a location associated with an upcoming calendar event (e.g., if an event is scheduled to occur within a predetermined time period (such as 1 hour, 30 minutes, etc.), the location associated with the event may also be used in a wind turbine or other application and simply suggested to the user), and the location in question when the user interacts with a virtual assistant (e.g., Siri of the application INC) on the electronic device (such as when the user asks Siri for nearby restaurants, then information about those restaurants may be used by or as suggestions to the user in other applications).

In some embodiments, the location may be available to other applications or as a recommendation for use by the user without any prior user interaction related to the location. For example, if a particular location is associated with an upcoming calendar event, the particular location may be proactively suggested for the windward application even if the user has not recently viewed the upcoming calendar event or the particular location.

In some embodiments, location suggestions (e.g., suggestions of locations available using any of the techniques discussed herein) are provided throughout various applications and components of an electronic device (e.g., device 100). For example, in some embodiments, location suggestions may be obtained from within:

a suggestion portion (also referred to as a QuickType bar) above the virtual keyboard, such as discussed with reference to user interface object 2505 in FIG. 25D;

an application-switching user interface, such as discussed with reference to user interface object 2503 in FIG. 25C;

a map application on the home screen, without any user action;

a map desktop applet (e.g., such as one that may be displayed within the main interface in response to a user sliding in a generally left-to-right direction over the first page of the home screen), in some embodiments, a user performing a gesture with increasing intensity over the map desktop applet causes a suggested location to be displayed within the map desktop applet;

The CarPlay map application on the home screen, without any user action (e.g., as discussed for method 2700);

a search interface (e.g., to show search query suggestions corresponding to locations within the search interface, such as the locations in FIG. 11B); and

the virtual assistant component of device 100 (e.g., in response to a text or spoken question from the user, such as "navigate me to there" or "call this place," the virtual assistant can disambiguate references to "that" and "this" based on a suggested location determined according to any of the techniques discussed herein).

In some embodiments, with respect to making a location available to the virtual assistant application, the device 100 can respond to questions such as "navigate me to" or "call this place" based on the data the user is currently viewing in the foreground application. In some embodiments, any requests submitted to the server in response to questions posed to the virtual assistant are performed in a privacy-preserving manner. For example, when parsing and responding "navigate me to there", a request is sent to a server associated with the virtual assistant, and only an indication that a location is available in the current application is provided to the server, without any other user identifying information, nor explicitly published location data. In some embodiments, the server interprets and responds to the command/question, and instructs the device 100 to begin navigating to the appropriate location (e.g., the location the user viewed in the foreground location or some other appropriate location, such as the location of an upcoming calendar event).

In some embodiments, if the user copies textual content, device 100 automatically (i.e., without any explicit instruction from the user to perform) determines whether the copied textual content includes location information (e.g., an address or some other information that may be used to retrieve the address, such as a restaurant name, etc.). Upon determining that the copied textual content does include location information, device 100 publishes the address for use by other system components (e.g., examples provided above, such as the QuickType field and application toggle user interface, etc.) that are capable of displaying and using the location information. For example, a user receives a text message with an address, then the user copies the address, provides an input (e.g., double-clicking on the home button to present an application toggle user interface), and in response to the input, device 100 displays a user interface object, such as a banner that reads "get a route in a map for < address >" or some other suitable and indicative phrase whose location is available in the application (e.g., user interface 2503 discussed above).

In some embodiments, the location information suggested for use by the user (e.g., in the QuickType column, in the application switch user interface, etc.) differs depending on the type of application that is to use the location information. For example, if a user views a location in a crowd-sourced commentary application (e.g., Yelp), and the user subsequently navigates to a windward application (e.g., Uber), the user may see a full address corresponding to the location they previously viewed. However, if the user navigates to the weather application, the user may only be presented within the city and state of the location they previously viewed, rather than the full address, because the weather application only requires information of the city and state, and not the full address. In some embodiments, the application can specify a level of granularity at which location information should be provided, and then provide suggested location information accordingly (e.g., with a windmilling application at a first level of granularity, and a weather application at a second level of granularity).

In some embodiments, the location information inserted in response to the user's selection of the suggested location depends on the trigger phrase. For example, if a user views a location in a crowd-sourced commentary application, then switches to a messaging application, and begins typing "meet up at a place," the device may display the location that the user previously viewed in the crowd-sourced commentary application (e.g., within user interface object 2309 in fig. 23F). In some embodiments, if the user selects the suggested location (e.g., tapping on user interface object 2309), the device may insert the restaurant name and restaurant address (and may also insert other relevant information, such as menu links, phone numbers, etc.). In some embodiments, if the user has typed "address yes," only addresses may be inserted (not names or other details, as the trigger phrase "address yes" means that only addresses are needed) in response to the user selecting a suggestion. In some embodiments, the apparatus 100 maintains more than one representation of a particular location available for proposal to selectively provide this information at different levels of granularity. For example, if a user copies an address from a crowd-sourced reviews application, device 100 may retain the copied address and may additionally store other information (e.g., including phone numbers, restaurant names, menu links, etc.) that may be obtained from the crowd-sourced reviews application.

In some embodiments, the device 100 (or a component, such as the active module of fig. 1A) actively monitors calendar events and suggests locations associated with upcoming events (e.g., events with a start time within a predetermined amount of time, such as 30 minutes, 1 hour, or 1.5 hours), even if no user interaction is received with a particular event or its associated location. In some embodiments, traffic conditions are taken into account in order to adjust the predetermined amount of time.

In some embodiments, when an application is suggested to use location information (e.g., in an application-switching user interface, such as the windmill application is suggested to use the user interface in fig. 25C like the location of Gary Danko in 2503), the application is selected based on various contextual information/heuristics that help identify the application (e.g., based on application usage patterns, time of day, day of week, most recent of the application installation, etc.), with more details provided below with reference to section 8). In some embodiments, the recent usage of the respective application is an additional factor utilized to identify the application (e.g., if the user recently went to dinner and arrived there using the windward application, the device 100 may determine that the user attempted to go home in approximately one hour, and will suggest the windward application because it was the most recently used).

As described above, any of methods 2200,2280,2900,2400,2600 and 2700 (or any other method described herein) may utilize the above details in connection with identifying, storing, and providing information about physical locations.

Additional detailed description of the invention

The additional description provided below in sections 1 through 11 provides additional details that supplement that provided above. In certain instances or embodiments, any of the above-described methods (e.g., methods 600,800, 1000,1200,1400,1600,1800,2000,2200,2280,2400,2600,2700, and 2900) can use some of the details provided below with reference to sections 1-11 to improve or improve the operation of any of the methods as appropriate. Those of ordinary skill in the art will appreciate the numerous ways in which the following descriptions in sections 1-11 supplement the disclosure provided herein (e.g., with reference to fig. 1A-30D).

Part 1: dynamic adjustment of mobile devices

The content "dynamic adjustment of a mobile device" in this section relates to dynamically adjusting a mobile device based on user activity, peer event data, system data, voter feedback, adaptive prediction of system events, and/or thermal conditions according to some embodiments, and provides information that complements the disclosure provided herein. For example, as described in more detail below, this section describes predicting when an application will be used/invoked during the day, and also describes examining usage statistics to determine whether the application is likely to be invoked by a user in the near future, which is in addition to the disclosure provided herein regarding, for example, operations 604 and 608 of method 600 and operation 808 of method 800. As another example, section 1 describes when the temporal prediction used indicates that an event associated with an attribute is likely to occur during the day (e.g., the likely time within 24 hours that a user will launch a particular type of application such as a mail application), which complements the disclosure provided herein (e.g., that related to the collection/storage of usage data (fig. 3A-3B) and the creation/storage of trigger conditions (fig. 4A-4B) and operation 808 of method 800). As another example, section 1 discusses using additional data (position data, motion data, etc.) to improve temporal predictions and generate a panoramic prediction that assigns a percentage value about a particular application that will be launched within a particular time period, which supplements the disclosure provided herein (e.g., the disclosure related to the creation/storage of trigger conditions (fig. 4A-4B)). As another example, section 1 describes the use of a voting system to manage the execution of predicted events, which is complementary to the disclosure provided herein (e.g., the disclosure related to the collection/storage of usage data (fig. 3A-3B) and the creation/storage of trigger conditions (fig. 4A-4B) and operation 808 of method 800). As another example, section 1 describes predicting the likelihood that an event associated with an attribute will occur over a period of time (based on various types of predictions), which complements the disclosure provided herein (e.g., the disclosure related to the collection/storage of usage data (fig. 3A-3B) and the creation/storage of trigger conditions (fig. 4A-4B)). As another example, section 1 describes management of thermal conditions, which complements the disclosure provided herein regarding conserving power (e.g., to ensure that methods 600 and 800, or any other methods described above, operate in an energy efficient manner).

Dynamically adjusted abstract for mobile device

In some implementations, a mobile device (e.g., device 100 of fig. 1A) can be configured to monitor environmental, system, and user events. The mobile device may be configured to detect the occurrence of one or more events that may trigger an adjustment to a system setting.

In some implementations, the mobile device can be configured with predefined and/or dynamically defined attributes. The system may use the attribute to track system events. The attribute events may be stored and later used to predict future occurrences of the attribute events. The system may use the stored attribute events to make decisions about the processing performed by the mobile device. Attributes may be associated with budgets that allow the budget resource to support future events or activities on the system.

In some implementations, various applications, functions, and processes running on the mobile device can submit attribute events. Applications, functions and processes may later request predictions based on the submitted events. The applications, functions, and processes may execute budgets based on budgets associated with attributes and costs associated with reported events. For example, the applications, functions, and processes may be associated with an operating system of the mobile device or a third party application.

In some implementations, the mobile device can be configured to keep frequently invoked applications up-to-date. The mobile device may record when the user invokes the application. Based on the invocation information, the mobile device may predict when the application is invoked during the day. The mobile device can then pre-launch the application and download the update so that the user can invoke the application and view the current updated content without having to wait for the application to download the updated content.

In some implementations, the mobile device can receive a push notification associated with the application indicating that new content is available for download by the application. The mobile device can launch the application associated with the push notification in the background and download the new content. After downloading the content, the mobile device may present a graphical interface indicating to the user that a push notification was received. The user may then invoke the application and view the updated content.

In some implementations, the mobile device can be configured to perform download and/or upload processes of content for applications on the mobile device. For example, a dedicated process may be configured on a mobile device to download and/or upload content for an application on the mobile device.

The application may be paused or terminated while the upload/download is performed. After the upload/download is completed, the application may be invoked.

In some implementations, prior to running an application or accessing a network interface, the mobile device can be configured to check battery power and cellular data usage budgets to ensure that sufficient power and data are available for the user-invoked operation. Before launching the application in the background, the mobile device may check the usage statistics to determine whether the application is likely to be invoked by the user in the near future.

In some implementations, the attribute event data can be shared between mobile devices owned by the same user. The mobile device may receive event data from the peer device and make a decision regarding an interaction or operation involving the peer device based on the received event data. Event data may be shared as predicted, statistical, and/or raw (e.g., unprocessed) event data. For example, the mobile device may determine whether to communicate with the peer device based on the received event data.

Particular implementations provide at least the following advantages: by dynamically adjusting components of the mobile device in response to detected events, battery power may be conserved. The user experience may be improved by anticipating when the user will invoke the application and download the content such that the user will view the updated content when the application is invoked.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, aspects, and potential advantages will become apparent from the description and drawings, and from the claims.

Detailed description of mobile device dynamic adjustment

SUMMARY

This section describes a system architecture for implementing mobile device adaptation based on various system events to facilitate a trade-off between battery life, power requirements, thermal management, and performance. The system provides a basic event collection architecture and a set of heuristics learned from system events to maximize battery life without significantly degrading user experience. The system monitors the system-defined and client-defined attributes and may use the system-defined and client-defined attributes to predict or forecast the occurrence of future events. The system may anticipate future behavior of the system and user expectations for device performance based on dynamically collected statistics and/or explicitly specified user intents. The system may determine hardware and software control parameters to be set and values for the set parameters to improve the user's experience with the expected system behavior. The system utilizes system monitoring and hardware control to achieve an overall improvement in user experience while extending the system and network resources available to the mobile device. Thus, the system may minimize the impact on the user experience while maximizing system and network resources.

Data collection-user centric statistics

Fig. 31_1 shows an exemplary mobile device 31_100 configured to perform dynamic adjustment of the mobile device 31_ 100. In some implementations, the mobile device 31_100 can include a sampling daemon 31_102 that collects events related to device conditions, network conditions, system services (e.g., daemon processes), and user behavior. For example, the sampling daemon 31_102 may collect statistical information related to applications, sensors, and user inputs received by the mobile device 31_100 and store the statistical information in the event data store 31_ 104. This statistical information may be reported to the sampling daemon 31_102 by various clients (e.g., applications, utilities, functions, third party applications, etc.) running on the mobile device 31_100 using predefined or client defined attributes that are reported as events.

Data collection-events and attributes

In some implementations, the mobile device 31_100 can be configured with a framework for collecting system and/or application events. For example, the mobile device 31_100 may be configured with an Application Programming Interface (API) that allows various applications, utilities and other components of the mobile device 31_100 to submit events to the sampling daemon 31_102 for later statistical analysis.

In some implementations, each event recorded by the sampling daemon 31_102 in the event data store 31_104 can include an attribute name (e.g., "package name (bundleId)"), an attribute value (e.g., "contact"), anonymous beacon information, anonymous location information, date information (e.g., GMT date of the event), time information (e.g., localized 24 hour time of the event), network quality information, processor utilization metrics, disk input/output metrics, identification of the current user, and/or event type (e.g., start, stop, occurrence). For example, the attribute name may identify the type of attribute associated with the event. The attribute name may be used to identify a particular metric, for example, tracked by the sampling daemon 31_ 102. An attribute value may be a value associated with an attribute (e.g., string, integer, floating point). The anonymous beacon information can indicate which wireless beacon (e.g., bluetooth low energy, Wi-Fi, etc.) is within range of the mobile device without binding or associating the beacon information with the user or device. Similarly, the anonymous location information may identify the location of the mobile device without binding or associating the location information with the user or device. For example, the location information may be derived from satellite data (e.g., global positioning satellite system), cellular data, Wi-Fi data, Bluetooth data using various transceivers configured on the mobile device 31_ 100. The network quality information may indicate a mobile device network (e.g., Wi-Fi, cellular, satellite, etc.) connection quality detected by the mobile device 31_100 when the event occurs.

In some implementations, the event data for each event can indicate the event occurred, started, or stopped. For example, pairs of events of the same attribute that indicate a start event and a stop event of the attribute may be timed (e.g., duration timing). For example, sampling daemon 31_102 may receive a start event with the attribute "package name" of the value "contact". Thereafter, sampling daemon 102 can receive a stop event with the attribute "package name" of the value "contact". Sampling daemon 31_102 can compare the time of the start event to the time of the stop event to determine the time (e.g., duration) that the "contacts" application was active. In some implementations, an untimed event can be recorded as a point event (e.g., a single occurrence). For example, an event associated with a "battery charge" system attribute (which specifies the instantaneous battery charge at the time the event occurred) may simply be recorded as the occurrence of the event.

Table 1 below provides an example of attribute event entries recorded in the event data store 31_104 by the sampling daemon 31_ 102. The first entry records a "package name" event indicating that the user "Fred" has invoked the "contacts" application. This "package name" event is a start event indicating that Fred has started using the contacts application. The second entry is a "battery level" event entry indicating that the battery level of the mobile device 31_100 is 46%; the event is an occurrence type event (e.g., a single point event). The third entry is a "person name" event associated with the value "George". The "person name" event is used to record the fact that the user Fred has accessed contact information for contact "George" in the contacts application; this is an occurrence type event. The fourth entry records a "package name" event indicating that the user "Fred" has closed or hidden the "contacts" application. This "package name" event is a stop event indicating that Fred has stopped using the contact application. By recording the start and stop events for the "package name" attribute, sampling daemon 31_102 can determine that user Fred has used the contact application for 8 minutes 5, 12 months 2014 based on the timestamps corresponding to the start and stop events. This property event information may be used, for example, to forecast user activity related to applications on the mobile device 31_100, in particular user activity with respect to contact applications.

TABLE 1

Attribute name

Value of

Beacon

Position of

Date

Time

Network quality

User ID

Status of state

Name of bag

Contact person "

B1,B2…

Position 1

20140512

1421

8

Fred

Start of

Electric quantity of battery

46

B1,B2…

Position 2

20140512

1424

8

Fred

Take place of

Name of a person

“George”

B1,B2…

Position 2

20140512

1426

8

Fred

Take place of

Name of bag

Contact person "

B1,B2…

Position 1

20140512

1429

8

Fred

Stop

Predefined attributes

In some implementations, event data can be submitted to the sampling daemon 31_102 using well known or predefined attributes. The well-known or predefined attributes may be general system attributes that may be used by various applications, utilities, functions, or other components of the mobile device 31_100 to submit event data to the sampling daemon 31_ 102. While the attribute definition (e.g., attribute name, data type of associated value, etc.) is predefined, the value assigned to the predefined attribute may vary from event to event. For example, the mobile device 31_100 may be configured to have a predefined attribute "package name" for identifying an application, and a predefined attribute "person name" for identifying a person of interest. The value assigned to the "package name" may vary based on which application on the mobile device 31_100 is active. The value assigned to "person name" may differ based on user input. For example, if the user selects an email from "George", the "person name" attribute value may be set to "George". If the user selects the contact entry associated with "Bob," the "person name" attribute value may be set to "Bob". When an application, utility, function, or other component of the mobile device 31_100 submits an event to the sampling daemon 31_102 using predefined attributes, the application, utility, function, or other component may specify a value associated with the predefined attributes of the event. Examples of predefined or well-known system events are described in the following paragraphs.

In some implementations, the mobile device 31_100 can be configured with predefined attributes (e.g., "system package name") that indicate the name or identifier of an application (e.g., application package) installed on the mobile device 31_ 100. When an application starts, the application manager 31_106 (e.g., responsible for starting the application) may submit an identifier or name of the application (e.g., "contact" for the contact application) as a value of the "system. The sampling daemon 31_102 may record the occurrence of a "contacts" application launch as an event in the event data store 31_104, for example, along with other event data as described above. Alternatively, the application may use the API of sampling daemon 31_102 to indicate start and stop events corresponding to when the application "contacts" is called and when the application is hidden or closed, respectively. For example, the "package name" attribute may be used to record an application start event on the mobile device 31_ 100. The "package name" attribute may be used to record an application termination event on the mobile device 31_ 100. By specifying start and stop events associated with the "package name" attribute, rather than merely specifying the occurrence of an event, the sampling daemon 31_102 can determine how long the user of the mobile device 31_100 has used the "contacts" application.

In some implementations, the mobile device 31_100 can be configured to have a name or identifier that indicates the user of the mobile device 31_100 or a predefined attribute (e.g., "system. For example, upon logging in, waking up, or otherwise accessing the mobile device 31_100, an event associated with the "person name" attribute may be generated and submitted to the sampling daemon 31_102 that identifies the current user of the mobile device 31_ 100. When a user accesses data associated with another person, a "person name" attribute event may be generated and submitted to sampling daemon 31_102, which identifies the other person as a person of interest to the user.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system. anonymous location") that indicates the location of the mobile device 31_ 100. For example, the mobile device 31_100 may generate and submit an event associated with an "anonymous location" attribute to the sampling daemon 31_102 that specifies the location of the mobile device 31_100 at the time of event generation. The location data may be anonymous such that the location cannot be later bound or associated with a particular user or device. For example, an "anonymous location" attribute event may be generated and stored each time a user uses a location-based service of the mobile device 31_ 100.

In some implementations, the mobile device 31_100 can be configured to have predefined attributes (e.g., "system. flight mode") that indicate that the flight mode of the mobile device 31_100 is on or off. For example, when a user turns on or off flight mode, the mobile device 31_100 may generate and submit an event to the sampling daemon 31_102, which indicates the flight mode status at the time of the event. For example, the value of the "airplane mode" attribute may be set to true (e.g., one) when the airplane mode is open, and may be set to false (e.g., zero) when the airplane mode is closed. The sampling daemon 31_102 may then store the "flight mode" events, including the "flight mode" attribute values, in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system. cable plug") that indicates that a power cable of the mobile device 31_100 is plugged in or unplugged. For example, when the mobile device 31_100 detects that a power cable has been unplugged, the mobile device 31_100 may generate an event indicating that the "cable plug in" attribute value is false (e.g., zero). When the mobile device 31_100 detects that a power cable has been plugged into the mobile device 31_100, the mobile device 31_100 may generate an event indicating that the "cable plugged in" attribute is true (e.g., one). The mobile device 31_100 may submit a "cable plug-in" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predefined attribute (e.g., "system. screen lock") that indicates whether the display screen of the mobile device 31_100 is locked or unlocked. For example, the mobile device 31_100 may detect when a display screen of the mobile device 31_100 has been locked (e.g., by the system or by the user) or unlocked (e.g., by the user). Upon detecting the locking or unlocking of the display screen, the mobile device 31_100 may generate an event including a "screen lock" attribute and set the "screen lock" attribute value of the event to true (e.g., locked, integer one) or false (e.g., unlocked, integer zero) to indicate whether the display screen of the mobile device 31_100 is locked or unlocked. The mobile device 31_100 may submit a "screen lock" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predefined attribute (e.g., "system. sleep awake") that indicates whether the mobile device 31_100 is in a sleep mode. For example, the mobile device 31_100 may detect when the mobile device 31_100 enters the sleep mode. The mobile device 31_100 may detect when the mobile device 31_100 exits the sleep mode (e.g., wakes up). Upon detecting entry into or exit from sleep mode, the mobile device may generate an event including a "sleep wake" attribute and set the attribute value to true or false (e.g., an integer one or zero, respectively) to indicate the sleep mode state of the mobile device 31_100 at the time the "sleep wake" event occurred. The mobile device 31_100 may submit a "sleep wake" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system backlight") that indicates whether a display of the mobile device 31_100 is illuminated. The "backlight" attribute may be assigned a value indicative of the backlight intensity or level. For example, the user of the mobile device 31_100 may adjust the intensity of the illumination (backlight) of the display of the mobile device 31_ 100. When the ambient light is bright, the user may increase the intensity of the backlight. When the ambient light is dark, the user may decrease the intensity of the backlight. Upon detecting a change in backlight settings, the mobile device 31_100 may generate an event including a "backlight" attribute and set the attribute value to an adjusted backlight setting (e.g., intensity level). The mobile device 31_100 may submit the "backlight" change event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system ALS") that indicates an ambient light intensity value detected by an ambient light sensor of the mobile device 31_ 100. The "ALS" attribute may be assigned a value indicating the intensity or level of ambient light around the mobile device 31_ 100. For example, an ambient light sensor of the mobile device 31_100 may detect a change in the intensity of ambient light. The mobile device 31_100 may determine that the intensity change exceeds a certain threshold. When detecting that the change in ambient light exceeds the threshold, the mobile device 31_100 may generate an event including an "ALS" attribute and set the attribute value to the detected ambient light intensity value. The mobile device 31_100 may submit the "ALS" change event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., system proximity) that indicates when a proximity sensor of the mobile device 31_100 detects that a display of the mobile device 31_100 is near an object (e.g., a user's face, on a table, etc.). A value may be specified for the "proximity" attribute that indicates whether the display of the mobile device is proximate to an object (e.g., true, false, 0, 1). For example, a proximity sensor of the mobile device 31_100 may detect a change in proximity. When a change in proximity is detected, the mobile device 31_100 may generate an event that includes a "proximity" attribute and set the attribute value to true (e.g., one) when the mobile device 31_100 is in proximity to an object and to false (e.g., zero) when the mobile device 31_100 is not in proximity to an object. The mobile device 31_100 may submit the "proximity" change event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system. motion status") that indicates the type of motion detected by the mobile device 31_ 100. The "motion state" attribute may be specified with a value that indicates whether the mobile device is at rest, moving, running, driving, walking, and so forth. For example, a motion sensor (e.g., an accelerometer) of the mobile device 31_100 may detect movement of the mobile device 31_ 100. The mobile device 31_100 may classify the detected movement based on the motion pattern detected in the detected movement. The movement patterns may be classified into a variety of user activities, such as user stationary, moving, running, driving, walking, and the like. Upon detecting a change in movement, the mobile device 31_100 may generate an event including an attribute of "motion state" and set the value of the attribute to the type of movement detected (e.g., stationary, running, walking, driving, etc.). The mobile device 31_100 may submit the "motion state" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system. network quality") that indicates the quality of the network connection detected by the mobile device 31_ 100. The "network quality" attribute may be assigned a value that indicates a network throughput value over a period of n seconds (e.g., 1 millisecond, 2 seconds, etc.). For example, the mobile device 31_100 may be connected to a data network (e.g., cellular data, satellite data, Wi-Fi, Internet, etc.). The mobile device 31_100 may monitor the data throughput of the network connection for a period of time (e.g., 5 seconds). The mobile device may calculate the amount of data transmitted per second (e.g., bits/second, bytes/second, etc.). Upon detecting a change in throughput or upon creating a new network connection, the mobile device 31_100 may generate an event that includes a "network quality" attribute and set the value of the attribute to the calculated throughput value. The mobile device 31_100 may submit a "network quality" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system. battery charge") that indicates an instantaneous charge level of an internal battery of the mobile device 31_ 100. The "battery level" attribute may be assigned a value indicative of a charge level (e.g., percentage) of the battery. For example, the mobile device 31_100 may periodically (e.g., every 5 seconds, every minute, every 15 minutes, etc.) determine the charge level of the battery and generate a "battery charge" event to record the charge level of the battery. The mobile device 31_100 may monitor the battery charge level and determine when a threshold amount of change in charge level has occurred and generate a "battery charge" event to record the charge level of the battery. The mobile device 31_100 can submit a "battery level" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system. thermal level") that indicates a thermal level of the mobile device 31_ 100. For example, the hot level of the mobile device 31_100 may be the current operating temperature of the mobile device (e.g., degrees celsius). The hot level of the mobile device 31_100 may be a level representing a range of temperature values (e.g., high, medium, low, normal, abnormal, etc.). For example, the mobile device 31_100 may be configured with a utility or function for monitoring the thermal status of the mobile device 31_ 100. When a temperature change or a thermal level change is detected, the thermal utility of the mobile device 31_100 may generate an event including a "thermal level" attribute and set the attribute value to an operating temperature or a current thermal level. The mobile device 31_100 may submit the "hot level" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have a predetermined attribute (e.g., "system. energy") that indicates an amount of energy usage of the mobile device 31_100 over a period of n seconds (e.g., 2 milliseconds, 3 seconds, etc.). For example, when a user invokes a function of the mobile device 31_100 (e.g., an application call, display lighting, data transfer, etc.), the mobile device 31_100 may monitor the amount of energy usage during this period of time to perform the function to estimate the amount of energy used per activity or function. The mobile device 31_100 may then generate an event comprising the "energy" attribute and set the attribute value to the calculated average energy usage. The mobile device 31_100 may submit the "energy" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

In some implementations, the mobile device 31_100 can be configured to have predetermined attributes (e.g., "system. network bytes") that indicate network data for the mobile device 31_100 for a period of n seconds (e.g., 2 milliseconds, 3 seconds, etc.). For example, when a user invokes or initiates a function that requires data to be transferred over the network connection of the mobile device 31_100, the mobile device 31_100 may monitor the amount of network data usage over a certain period of time to estimate the amount of network data used or transferred by each activity or function. The mobile device 31_100 may then generate an event comprising a "network byte" attribute and set the attribute value to the calculated average network data usage. The mobile device 31_100 may submit the "network byte" event to the sampling daemon 31_102 for storage in the event data store 31_ 104.

Other predefined attributes may include: a "system charge status" attribute having a true/false (e.g., one/zero) attribute value indicating whether the mobile device 31_100 is charging its battery; a "system.battery capacity" attribute having an attribute value indicating a current battery charge condition (e.g., in mAh, proportional to battery charge), and a "system.device presence" attribute having a device identifier (e.g., string) attribute value tracking the appearance of the peer device. For example, when scheduling peer-to-peer data sharing, the "device presence" attribute may be used to predict the appearance of the peer-to-peer device.

Custom attributes

In some implementations, the client-specific attributes may be dynamically defined by the client of sampling daemon 31_ 102. For example, a client (e.g., a third party application) may dynamically define its own event attributes, rather than using attributes predefined (e.g., in sampling daemon 31_102 or operating system) and configured on mobile device 31_ 100. For example, an email application may create a "mailbox" attribute dynamically (e.g., at runtime). An email application program ("mail application") may use the API of sampling daemon 31_102 to define an attribute name (e.g., "mail application. mailbox") and an attribute value type (e.g., string, integer, floating point number). Once a client has created (registered) a new custom attribute, the client can use that attribute to generate an event to be stored in the event data store 31_ 104. For example, a mail application may use a "mailbox" attribute to report which mailbox in an email application program is being accessed by a user. If the user is accessing the "work" mailbox, the mail application may use the "mail application mailbox" attribute to create an event and set the value of the attribute to "work" to record the user's access to the "work" mailbox. The sampling daemon 31_102 and client may then use the stored mailbox event information to predict, for example, when the user is likely to access a "work" mailbox in the future.

In some implementations, when a client application is removed (e.g., deleted, uninstalled) from the mobile device 31_100, the properties created by the client application can be deleted from the mobile device 31_ 100. Further, when a client application is removed, event data associated with the client application may be deleted. For example, if a mail application is deleted from the mobile device 31_100, the attributes "mail application, mailbox" and all event data associated with the mail application may be deleted from the mobile device 31_ 100.

Exemplary event generating client

In some implementations, the sampling daemon 31_102 may receive an application event (e.g., a "system package name" event) from the application manager process 31_ 106. For example, the application manager 31_106 may be a process that starts, stops, and monitors applications (e.g., the application 31_108) on the mobile device 31_ 100. In some implementations, the application manager 31_106 can report start and stop times (e.g., "package name" start and stop events) of applications running on the mobile device 31_100 to the sampling daemon 31_ 102. For example, when a user invokes or launches an application, the application manager 31_106 can notify the sampling daemon 31_102 of the invocation of the application by submitting a "package name" start event for the invoked application (which specifies the name or identifier of the application). In some implementations, the application manager 31_106 can indicate to the sampling daemon 31_ 102: application launch is initiated in response to a push notification, a user invocation, or a predicted or forecasted user application invocation. When an application terminates, the application manager 31_106 can notify the sampling daemon 31_102 that the application is no longer running by submitting a "package name" stop event for the application (which specifies the name or identifier of the application).

In some implementations, the sampling daemon 31_102 can use application start and end events (e.g., "package name" property events) to generate a usage time history for each application. For example, the usage time history for each application may include, for each execution of the application, the amount of time that has elapsed since the last execution of the application and the execution duration. The sampling daemon 31_102 may maintain a separate history of user invoked application launches and/or system launched (e.g., automatically launched) applications. Thus, the sampling daemon 31_102 can maintain usage statistics for all applications executing on the mobile device 31_ 100.

In some implementations, the sampling daemon 31_102 may receive power events from the power monitoring process 31_ 108. For example, the power monitor 31_109 may monitor battery capacity, discharge, usage, and charging characteristics of the mobile device 31_ 100. The power monitor 31_109 can determine when the mobile device 31_100 is plugged into an external power source and when the mobile device 31_100 relies on battery power. When the mobile device 31_100 is plugged into an external power source, the power monitor 31_109 may notify the sampling daemon 31_ 102. For example, when the power monitor detects that the mobile device 31_100 is plugged into an external power source, the power monitor 31_109 may send a "cable plug" event to the sampling daemon 31_102 with a "cable plug" attribute value of one (e.g., true). The event may include a battery charging condition when an external power source is connected. The power monitor 31_109 may send an "energy" attribute event to the sampling daemon 31_102 to report battery usage.

In some implementations, the power monitor 31_109 can notify the sampling daemon 31_102 when the mobile device 31_100 is disconnected from an external power source. For example, when the power monitor detects that the mobile device 31_100 is disconnected from an external power source, the power monitor 31_109 may send a "cable plug" event to the sampling daemon 31_102 with a "cable plug" attribute value of zero (e.g., false). The message may include the battery charge condition when the external power source is disconnected. Thus, the sampling daemon 31_102 may maintain statistical information describing the charge distribution (e.g., charging over time) of the battery of the mobile device 31_ 100. The charge distribution statistics may include the amount of time since the last charge (e.g., the time since the external power source was plugged in), and the change in the charge condition of the battery due to the charge (e.g., the start level of the charge, the end level of the charge).

In some implementations, the power monitor 31_109 can notify the sampling daemon 31_102 of a change in battery charge throughout the day. For example, the power monitor 31_109 may be notified when an application starts and stops, and in response to the notification, the power monitor may determine the amount of battery power released and the amount of charge remaining in the battery during that period and send this information to the sampling daemon 31_ 102. For example, power monitor 31_109 may send a "system energy" event to sampling daemon 31_102 to indicate the amount of energy consumed during the period of time that the application is active.

In some implementations, sampling daemon 31_102 can receive device temperature statistics from thermal daemon 31_ 110. For example, the thermal daemon 31_110 may use one or more temperature sensors to monitor the operating temperature conditions of the mobile device 31_ 100. The thermal daemon 31_110 can be configured to periodically report temperature changes to the sampling daemon 31_ 102. For example, the thermal daemon 31_110 may determine the operating temperature of the mobile device 31_100 once every five seconds and report the temperature or thermal level of the mobile device 31_100 to the sampling daemon 31_ 102. For example, the thermal daemon 31_110 may send a "system. thermal level" event to the sampling daemon 31_102 to report the current operating temperature or thermal level of the mobile device 31_ 100. The sampling daemon 31_102 may store the reported temperature in the event data store 31_ 104.

In some implementations, the sampling daemon 31_102 may receive device setup statistics from the device setup process 31_ 112. For example, the device setup process 31_112 may be a function or process of an operating system of the mobile device 31_ 100. The device setup process 31_112 may, for example, receive user input to adjust various device settings, such as turn on/off flight mode, turn on/off Wi-Fi, turn on/off roaming, and so forth. The device setup process 31_112 may report changes in device settings to the sampling daemon 31_ 102. Each device setting may have a corresponding predefined event attribute. For example, when the user turns on or off the flight mode on the mobile device 31_100, the device setup process 31_112 may send a "system. flight mode" event to the sampling daemon 31_ 102. The sampling daemon 31_102 may generate and store statistical information for device settings based on received events and attribute values. For example, whenever a certain setting is enabled (or disabled), sampling daemon 31_102 may store data indicating: the amount of time that has passed since the setting was previously enabled, and the amount of time (e.g., duration) that the setting was enabled.

Similarly, in some implementations, the sampling daemon 31_102 can receive notifications from other mobile device 31_100 components (e.g., device sensors 31_114) when other events occur. For example, sampling daemon 31_102 may receive notification when: when the screen of the mobile device is turned on or off (e.g., a "system backlight" event), when the mobile device 31_100 is held next to the user's face (e.g., a "system proximity" event), when a cell tower switch is detected, when the baseband processor is in a search mode (e.g., a "system. baseband search" event), when the mobile device 31_100 detects that the user is walking, running, and/or driving (e.g., a "system. motion state" event). In each case, sampling daemon 31_102 can receive notifications at the beginning and end of an event. In each case, sampling daemon 31_102 can generate and store statistical information indicating the amount of time that has elapsed since the last detection of the event and the duration of the event. The sampling daemon 31_102 may receive other event notifications and generate other statistical information, as described further below with respect to particular use cases and scenarios.

Application events

In some implementations, the sampling daemon 31_102 can receive event information from an application on the mobile device 31_ 100. For example, an application on the mobile device 31_100 may generate events to the sampling daemon 31_102 that include predefined or dynamically defined attributes to track various application specific events. For example, sampling daemon 31_102 may receive calendar events (e.g., including "calendar. appointment", "calendar. meeting", or "calendar. reminder" attributes, etc.) from calendar application 31_ 116. Calendar events may include "calendar.appointment", "calendar.meeting", or "calendar.reminder" attributes having values that specify a location, time, or other data associated with various calendar events or functions. For example, sampling daemon 31_102 may store, for example, the name of the attribute, the duration of the attribute, and/or the time at which the attribute is scheduled to occur. In some implementations, sampling daemon 31_102 can receive clock events (e.g., including a "clock" attribute) from clock application 31_ 118. For example, sampling daemon 31_102 may store an attribute name (e.g., "clock. alarm") and a value indicating the time at which the alarm is scheduled to occur. The sampling daemon 31_102 may receive event information from other applications (e.g., media applications, card coupon applications, etc.), as described further below.

Application statistics

In some implementations, sampling daemon 31_102 can collect application statistics across application launch events. For example, sampling daemon 31_102 may collect statistics (e.g., event, "package name" attribute values) for each application across many invocations of the application. For example, each application may be identified with a hash of its executable file system path and executable content hash, such that different versions of the same application may be treated as different applications. For example, the application hash value may be submitted to the sampling daemon 31_102 as the value of the "package name" attribute in a "package name" event.

In some implementations, the sampling daemon 31_102 may maintain a counter that tracks background task completion assertion events for each application. For example, whenever an application runs as a background task (e.g., not visible in the foreground and/or not currently used by the user), the application or application manager 31_106 may notify the sampling daemon 31_102 when the application terminates or pauses, and the sampling daemon 31_102 may increment a counter. The sampling daemon 31_102 may maintain a counter that tracks the cumulative number of seconds an application starts in the background across applications. For example, the sampling daemon 31_102 may analyze "package name" start and stop events to determine when an application starts and stops, and use the timestamps of the start and stop events to determine how long the application has been running. In some implementations, the sampling daemon 31_102 can maintain the following independent counters: it counts the number of data connections; tracking network data traffic (e.g., in bytes); tracking the duration and size of file system operations; and/or track the number of threads associated with each application. Sampling daemon 31_102 can maintain a count of the cumulative amount of time that an application remains active, e.g., across application launches. These are just a few examples of application statistics that may be generated by sampling daemon 31_102 based on event and attribute data received by sampling daemon 31_102 and stored in event data store 31_ 104. Other statistics may be generated or collected as described below.

Heuristic method

In some implementations, the mobile device 31_100 can be configured with a heuristic process that can adjust the settings of the device components based on events detected by the sampling daemon 31_ 102. For example, the heuristic process 31_120 may include one or more of: these processes are configured (e.g., programmed) to adjust various system settings (e.g., CPU power, baseband processor power, display lighting, etc.) in response to one or more triggering events and/or based on statistical information collected or generated by sampling daemon 31_ 102.

In some implementations, the heuristic process 31_120 can register with the sampling daemon 31_102 to be invoked or activated when a set of predefined criteria are met (e.g., certain triggering events occur). The triggering event may include an invocation of a media player application (e.g., a "package name" event) or detection that the user has begun walking, running, driving, etc. (e.g., an "exercise state" event). The triggering event may be summarized as invoking the heuristic 31_120 when certain properties, data, statistics, events, attributes, attribute values, etc. are detected in the event data 31_104 or by the sampling daemon 31_ 102. For example, the heuristic 31_120 may be invoked when the sampling daemon 31_102 receives an application start notification (e.g., a "package name" start event specifying a particular application) or when the temperature is above a particular threshold (e.g., a "hot level" event). The heuristic process 31_120 may be invoked when the sampling daemon 31_102 receives an event associated with a specified attribute or attribute value. The heuristic 31_120 may register to be invoked when a single event occurs or statistical information is observed. The heuristic 31_120 may register to be invoked when a combination of events, data, attributes, attribute values, and/or statistics is observed or detected. The heuristic 31_120 may be triggered or invoked in response to a particular user input (e.g., an "airplane mode" event, a "sleep wake" event, etc.). The sampling process 31_102 may invoke the heuristic 31_120 when the sampling process 31_102 detects an event registered by the heuristic 31_ 120.

In some implementations, when the heuristic 31_120 is invoked, the heuristic 31_120 can communicate with the sampling daemon 31_102 to retrieve event data from the event data store 31_ 104. The heuristic 31_120 may process event data and/or other data collected by the heuristic 31_120 itself to determine how to adjust system settings to improve performance of the mobile device 31_100, improve a user's experience when using the mobile device 31_100, and/or avoid future problems with the mobile device 31_ 100.

In some implementations, the heuristic 31_120 may make setting suggestions that may cause settings of various device components 31_122 of the mobile device 31_100 to change. For example, device components may include a CPU, GPU, baseband processor, display, GPS, Bluetooth, Wi-Fi, vibration motor, and other components.

In some implementations, the heuristic 31_120 can make setup suggestions to the control mux 31_ 124. For example, the control multiplexer 31_124 may be a process that arbitrates between the component settings provided by the heuristic process 31_120 and other processes and/or functions of the mobile device 31_100 that affect or change the mobile device 31_100 component settings. For example, the thermal daemon 31_110 may be a heuristic process: which is configured to adjust CPU power, display brightness, baseband processor power, and other component settings (e.g., above a threshold temperature) based on detecting that the mobile device 31_100 is in the middle of a thermal event. However, the heuristic 31_120 may also be configured to adjust CPU power, display brightness, baseband processor power, and other component settings. Thus, in some implementations, the heuristic 31_120 and thermal daemon 31_110 may make setting adjustment suggestions to the control multiplexer 31_124, and the control multiplexer 31_124 may determine which setting adjustments to make. For example, the control multiplexer 31_124 may prioritize the processes and perform adjustments based on the priority of the recommended processes. Thus, if the thermal daemon 31_110 is a higher priority process than the heuristic process 31_120, the control multiplexer 31_124 may adjust the settings of the CPU, display, baseband processor, etc. according to the suggestions of the thermal daemon 31_110 instead of the heuristic process 31_ 120.

In some implementations, the mobile device 31_100 can be configured with multiple heuristics 31_ 120. The heuristic 31_120 may be configured or reconfigured wirelessly. For example, the parameters (e.g., trigger conditions, thresholds, criteria, and outputs) of each heuristic 31_120 may be set or adjusted via a network (e.g., cellular data connection, Wi-Fi connection, etc.). In some implementations, a new heuristic 31_120 can be added to the mobile device 31_ 100. For example, new correlations between trigger events, statistics, and device settings over time may be determined by a system developer. As these new correlations are identified, one can develop new heuristics 31_120 to adjust the system settings taking into account the newly determined relationships. In some implementations, a new heuristic 31_120 can be added to the mobile device 31_100 over the network. For example, the new heuristic 31_120 may be downloaded or installed on the mobile device 31_100 over the air (e.g., cellular data connection, Wi-Fi connection, etc.).

Exemplary heuristic procedure

In some implementations, the heuristic 31_120 can be configured to adjust system settings of the mobile device 31_100 to prevent the mobile device 31_100 from becoming overheated in the user's pocket. For example, the in-pocket heating heuristic may be configured to register with the sampling daemon 31_102 to be invoked when the display of the mobile device is off (e.g., a "system. backlight" event has a zero/false attribute value), and when the mobile device 31_100 is not playing any entertainment media (e.g., music, movies, videos, etc.). When the in-pocket heating heuristic is invoked, the process may propose, for example, a recommendation to reduce the CPU power and GPU power to reduce the operating temperature of the mobile device 31_ 100.

In some implementations, the heuristic 31_120 can be configured to adjust the position accuracy when the display of the mobile device is not in use (e.g., a "system. backlight" event has a zero/false attribute value). For example, if the display of the mobile device is not used (e.g., the display is off, as indicated by the "backlight" attribute event above), the mobile device 31_100 cannot display map information or directions to the user. Thus, the user may not be able to use the location service of the mobile device 31_100, and the location service (e.g., GPS location, Wi-Fi location, cellular location, etc.) may be adjusted to use less power. The location accuracy heuristic may register with the sampling daemon 31_102 to be invoked when the display of the mobile device is turned off. When invoked, the heuristic procedure may adjust the power level of a GPS processor, Wi-Fi transmitter, cellular transmitter, baseband processor, or termination procedure used to determine the location of the mobile device 31_100 in order to conserve power on the mobile device 31_ 100.

In some implementations, the heuristic 31_120 can be configured to adjust settings of an ambient light sensor of the mobile device in response to the user's behavior. For example, the user adaptive Ambient Light Sensor (ALS) heuristic may be invoked by sampling daemon 31_102 when sampling daemon 31_102 receives data (e.g., an "ALS" attribute event) indicating that: the ambient light sensor has detected that the ambient light around the mobile device 31_100 has changed, the ambient light sensor system has adjusted the brightness of the display and/or the user has provided input to adjust the brightness of the display.

When the user-adaptive ALS heuristic is invoked, the process may request additional information about the ALS display adjustment and the user-initiated display adjustment from sampling daemon 31_102 to determine whether there is a user input pattern that instructs the ALS to adjust the display brightness and the user to adjust the display brightness in the opposite direction (e.g., a "system. For example, the user may be on a bus or train to work. During driving, the lights of the bus may be turned on and off. The ambient light sensor can detect changes in ambient light and increase the display brightness when the vehicle lights are on. Since the lights are only temporarily lit, the user may reduce the display brightness when the lights are turned off again. This user input pattern may be tracked (e.g., by "backlight" attribute events) and associated with a time of day, calendar or alarm event entry, or travel pattern through a heuristic to determine in what circumstances or contexts the user will adjust the display brightness in response to ALS display adjustments. Once the user-adaptive ALS heuristic determines the pattern and context of the input, the heuristic may adjust the settings of the ALS to be more or less positive. For example, the ALS may be adjusted to check the level of ambient light more or less frequently during certain times of day, calendar or alarm entries, or travel modes, and adjust the display brightness accordingly.

The heuristics described above are a few examples of heuristics and their implementation in the systems described in this section. Over time, other heuristics developed may be implemented and added to the system. For example, additional heuristic processes may be configured or programmed to adjust the CPU, GPU, baseband processor, or other components of the mobile device in response to detecting events or patterns of events related to temperature measurements, user inputs, clock events (e.g., alarm clocks), calendar events, and/or other events occurring and detected on the mobile device.

Exemplary heuristic registration and invocation Process

Fig. 31_2 shows an exemplary procedure 31_200 for invoking a heuristic procedure. At step 31_202, the sampling daemon 31_102 may be initialized. For example, the sampling daemon 31_102 may be initialized during the start up of the mobile device 31_ 100.

At step 31_204, sampling daemon 31_102 may invoke a heuristic process deployed on mobile device 31_100 during initialization of sampling daemon 31_ 102. For example, the sampling daemon 31_102 may cause each heuristic 31_120 to execute on the mobile device 31_100 and run its initialization subroutines.

At step 31_206, the sampling daemon 31_102 may receive event registration messages from each heuristic 31_ 120. For example, during an initialization subroutine of the heuristic 31_120, the heuristic 31_120 can send information to the sampling daemon 31_102 indicating which attribute events should trigger the invocation of the heuristic 31_ 120. The sampling daemon 31_102 may store the registration information in a database, such as the event data store 31_ 104. The registration information may include an identification of the heuristic (e.g., executable name, file system path, etc.) and event criteria (attribute identification, attribute values, thresholds, ranges, etc.), such that upon detecting a specified event, sampling daemon 31_102 may invoke heuristic process 31_ 120.

At step 31_208, the sampling daemon 31_102 may receive the attribute event data. For example, as described above, the sampling daemon 31_102 may receive attribute event data from various system components (including the application manager 31_106, the sensor 31_114, the calendar 31_116, and the clock 31_ 118).

At step 31_210, sampling daemon 31_102 can compare the received attribute event data to heuristic registration data. For example, when attribute event data is reported to sampling daemon 31_102, sampling daemon 31_102 can compare the event data (e.g., attribute values) or statistics generated from the event data to registration information received from heuristic process 31_ 120.

At step 31_212, sampling daemon 31_102 can invoke a heuristic process based on the comparison performed at step 31_ 210. For example, the sampling daemon 31_102 may invoke the heuristic process 31_120 if event data (e.g., attribute data) and/or statistical information satisfies criteria specified in the heuristic registration data of the heuristic process 31_ 120. For example, if event data and/or statistics cross some threshold specified by the heuristic process for some event during registration, the heuristic process may be invoked by sampling daemon 31_ 102. Alternatively, the mere occurrence of a certain attribute event may cause invocation of the heuristic process 31_ 120.

Fig. 31_3 shows a process 31_300 for adjusting mobile device 31_100 settings using a heuristic process 31_ 120. At step 31_302, the heuristic 31_120 is initialized. For example, the heuristic process 31_120 may be called by the sampling daemon 31_102 so that the heuristic process 31_120 may run its initialization subroutines. For example, the call may be parameterized to indicate that the heuristic 31_120 should run its initialization subroutine during the call.

At step 31_304, the heuristic process 31_120 can register with the sampling daemon 31_102 for system events. For example, during initialization, the heuristic 31_120 can send a message to the sampling daemon 31_102 that includes an identification of an event, threshold, attribute value, or other criteria for invoking the heuristic 31_ 120. The sampling daemon 31_102 can invoke the heuristic process 31_120 when an event occurs and/or meets criteria.

At step 31_306, the heuristic process 31_120 may shut down or terminate. For example, the heuristic 31_120 may be required by the system after registration criteria for the heuristic 31_120 are satisfied. Thus, to conserve device resources (e.g., battery power, processing power, etc.), the launch-type process 31_120 is terminated, shut down, or paused until needed to be invoked (e.g., triggered by the sampling daemon 31_ 102).

At step 31_308, the heuristic 31_120 may be restarted. For example, when sampling daemon 31_102 determines that the criteria specified by heuristic 31_120 in the registration message have been met, sampling daemon 31_102 may invoke heuristic 31_ 120.

At step 31_310, the heuristic process 31_120 can obtain event data from the sampling daemon 31_ 102. For example, once the heuristic 31_120 restarts, the sampling daemon 31_102 may be queried to obtain additional attribute event data. The heuristic processes 31_120 may be configured to interact with other system resources, processes, sensors, etc. as needed to collect data.

At step 31_312, the heuristic 31_120 may process the event data to determine component settings. For example, the heuristic 31_120 may use event data and/or statistical information from the sampling daemon 31_102, and/or data collected from other components of the system, to determine the manner in which to adjust the various component settings of the mobile device 31_ 100. For example, if the heuristic 31_120 determines that the mobile device 31_100 is overheating, the heuristic 31_120 may determine which power settings of the mobile device 31_100 will lower the operating temperature of the mobile device 31_ 100.

At step 31_314, the heuristic 31_120 may send the determined component settings to the control multiplexer 31_ 124. For example, the control multiplexer 31_124 may arbitrate device setting recommendations received from the heuristic 31_120 and other system components (e.g., the thermal daemon 31_ 110). The control multiplexer 31_124 may then adjust various components of the mobile device 31_100 (e.g., CPU, GPU, baseband processor, display, etc.) according to the received setup recommendations.

Forecasting events

In some implementations, the attribute event data (e.g., historical data) stored in the event data store 31_104 can be used by the sampling daemon 31_102 to predict the occurrence of future events. For example, a "package name" property event may be analyzed to predict when a user invokes an application (e.g., any application or a particular application). A "mail application" specified with a particular email folder (e.g., a "mailbox" attribute value set to a "work" folder) may be analyzed.

Event history window specification

In some implementations, the event prediction can be generated based on an event history window specification. For example, a window specification may be generated by the client to specify a time period of interest or periodic time periods of interest on which the client wishes to base event predictions. The window specification may include four components: start time, end time, recurrence width, and recurrence frequency. The start time can indicate the date and/or time in the calendar history when the window should start. The end time may indicate the date and/or time in the history when the window should end. The recurrence width may indicate a period of time of interest to the client (e.g., four hours from the start time). The recurrence frequency may indicate a frequency at which the period of time should be repeated since the start time (e.g., every 8 hours, every two days, every week, every two weeks, etc.).

In some implementations, only events that occur within a specified time period (e.g., a time period of interest) are analyzed in generating the event forecast. For example, if the current date is 2014 5-13, the window specification may specify a start date of 2014 5-11 noon 12:00, an end date of 5-12 noon 12, a recurrence width of 1 hour, and a recurrence frequency of 4 hours. This window specification will cause the sampling daemon 31_102 to analyze event data (e.g., time periods of interest) for each 1 hour period (e.g., time period of interest) that occurs every 4 hours from 12:00 am on 5/11 th 2014 to 12:00 pm on 5/12 th 2014 (e.g., period 1: 2014 12 pm on 5/11 th noon-1:00 pm, period 2: 2014 5/11 pm 4:00 pm-5 pm 5:00 pm, period 3: 2014 5/11 th pm 8:00 pm-9 pm, etc.). In some implementations, when no recurrence width is specified, the entire time period from the start time to the end time will be analyzed to predict the event.

In some implementations, the sampling daemon 31_102 may automatically generate an event history window specification. For example, the sampling daemon 31_102 may identify patterns in the event history data stored in the event data store 31_ 104. If the client requests a forecast of a "package name" event but does not provide a window specification, sampling daemon 31_102 may, for example, identify a pattern of "package name" attributes/events that indicates that the application is typically invoked by the user 8:00 am-9:00 am, 11:30 am-1:30 pm, and 7:00 pm-11:00 pm. The sampling daemon 31_102 may automatically generate a window specification that includes these time periods and excludes other time periods of the day so that the requested forecast will focus on the time period associated with the requested attribute. Similarly, the sampling daemon 31_102 may automatically generate an event history window specification for a particular (e.g., specified) attribute value. For example, if a client requests a forecast for a "package name" event with an attribute value of "mail application," sampling daemon 31_102 may analyze the event history data to identify patterns of occurrence associated with the "mail application" value. If "mail application" and "package name" attribute values are recorded in the event history data at 10:00 am, 12:00 pm, and 5:00 pm each day, the sampling daemon 31_102 may generate window specifications that specify time periods of interest around these times in the day.

Temporal prediction

In some implementations, a temporal forecast may be generated for a certain attribute or attribute value. The temporal forecast may indicate, for example, at what time of day an event associated with the attribute or attribute value may occur. For example, the client of the sampling daemon 31_102 may request a time forecast for the last week (e.g., the last 7 days) for a "package name" attribute (e.g., application launch). To generate the forecast, 24 hours a day may be divided into 96 15-minute time slots. For a particular time slot for each of the last seven days (e.g., 1:00 PM-1:15 PM), the sampling daemon 31_102 may determine whether a "package name" event has occurred and generate a score for that time slot. If a "package name" event occurred within the particular time slot in 2 of the 7 days, the probability (e.g., score) that a "package name" event occurred during the particular time slot (e.g., 1:00 PM-1:15 PM) is 0.29 (e.g., 2 divided by 7). If a "package name" event occurs during a different one of the 7 days (e.g., 12:15 PM-12:30 PM), the probability (e.g., score) that a "package name" event occurred during that time slot is 0.57 (e.g., 4 divided by 7).

Similarly, the client may request a temporal forecast for a particular attribute value. For example, the client may request a time forecast for a "package name" event with a "package name" attribute value of "mail application" instead of a "package name" attribute (e.g., "package name" event). Thus, the client may receive an indication of the time of day (e.g., a 15 minute time slot) at which the user will likely invoke the "mail application" application.

In some implementations, the temporal forecast may be generated based on an event history window specification. For example, if the client provides a window specification specifying a 4 hour period of interest, the time forecast will only generate a probability score for the 15 minute time slot within the 4 hour period of interest. For example, if the time period of interest corresponds to 12:00 am-4:00 pm of each of the last 3 days, 16 time slots would be produced during the 4 hour time period of interest, and a score would be generated for these 16 15 minute time slots. No score is generated for time slots outside the specified 4 hour time period of interest.

Peer to peer forecasting

In some implementations, sampling daemon 31_102 can generate a peer-to-peer forecast for attributes. For example, peer forecasts may indicate the relative probability that a value of a certain attribute will occur during a time period of interest relative to all values (e.g., incidence) of the same attribute. For example, the client of the sampling daemon 31_102 may request a peer-to-peer declaration for the "package name" attribute for a time period of interest (e.g., 11:00 am 1:00 pm) specified by a window specification submitted with the request. If a "package name" event with attribute values "mail application", "contact", "calendar", "web browser", "mail application" or "mail application" occurs during the time period of interest, the relative probability (i.e., score) of the occurrence of "mail application" is 0.43 (e.g., 3/7), the relative probability of the occurrence of "web browser" is 0.29 (e.g., 2/7), and the relative probability of the occurrence of "contact" or "calendar" is 0.14 (e.g., 1/7).

In some implementations, a client of sampling daemon 31_102 may request a peer forecast for a certain attribute. For example, if a client requests a peer forecast for an attribute without specifying the value of the attribute, the sampling daemon 31_102 will generate a peer forecast and return various probability scores for all values of the attribute within the time period of interest. Using the above exemplary peer-to-peer pre-report, sampling daemon 31_102 will return a list of attribute values and scores to the requesting client, e.g.: "mail application": 0.43; "Web browser": 0.29; "contact person": 0.14; the 'calendar': 0.14.

in some implementations, a client of sampling daemon 31_102 may request a peer forecast for a certain attribute value. For example, a client may request a peer forecast for a "package name" attribute with a value of "mail application". As described above, the sampling daemon 31_102 can generate a peer forecast for the "package name" attribute according to the window specification provided by the client. For example, the sampling daemon 31_102 may calculate that the relative probability (i.e., score) of "mail application" occurring is 0.43 (e.g., 3/7), the relative probability of "web browser" occurring is 0.29 (e.g., 2/7), and the relative probability of "touch" or "calendar" occurring is 0.14 (e.g., 1/7). Sampling daemon 31_102 may return a score (e.g., 0.43) for the requested "mail application" value to the client. If the requested value is not represented within the time period of interest specified by the window specification, a zero value will be returned to the client.

Panoramic forecast

In some implementations, a panoramic forecast may be generated to predict the occurrence of the property event. For example, the temporal and peer-to-peer forecasts described above use the relative frequency of event occurrences for a single attribute or attribute value to predict future occurrences of that attribute. This "frequency" forecast type (e.g., frequency of occurrence) uses only data associated with the attributes or attribute values specified in the forecast request. Instead, the "full-scene" forecast may use other data (e.g., location data, beacon data, network quality, etc.) in the event data received for the attributes or attribute values specified in the forecast request. In some implementations, the panorama forecast may use data for events associated with other attributes or attribute values. For example, when a client requests a time-forecast peer forecast of a specified attribute or attribute value, and specifies a forecast type (i.e., forecast flavor) as panoramic, sampling daemon 31_102 will analyze event data for that specified attribute or attribute value, as well as for other attributes and attribute values and event data, to identify correlations between the specified event and other events received by sampling daemon 31_ 102. For example, a frequency forecast with the attribute "package name" having the value "mail application" might assign 0.4 points to a 15 minute time slot of 9:00 am. However, panoramic forecasts may determine that there is a strong correlation between the "mail application" attribute values and the user's work site. For example, a panorama forecast may determine that if a user is at a location associated with work, the mail application will be invoked 90% of the time within the 15 minute time slot of 9:00 am. Thus, the sampling daemon 31_102 may assign a higher score (e.g., 0.9) to the "mail application" forecast score for the 15 minute time slot of 9:00 am.

Similarly, sampling daemon 31_102 may find a strong correlation between the "mail application," "package name" attribute values, and the occurrence of events associated with the "motion state" attribute value "stationary. For example, the sampling daemon 31_102 may determine that the correlation between the use of the mail application and the mobile device 31_100 being stationary is 95%. Sampling daemon 31_102 may determine that there is a 5% correlation between the use of the mail application and the mobile device 31_100 being in motion. Thus, the sampling daemon 31_102 may adjust the forecast score (e.g., 0.95 or 0.05) for a "mail application" attribute value for a particular time slot based on whether the mobile device is moving or stationary.

Scoreboard-frequency and panorama

In some implementations, the sampling daemon 31_102 can track which forecast type is the better predictor of events. For example, when sampling daemon 31_102 receives an attribute event, sampling daemon 31_102 can generate frequency forecasts and panorama forecasts for the attributes or attribute values associated with the received event, and determine which forecast type will be a better predictor of the received attribute event. In other words, even if the prediction is generated immediately prior to receiving the property event, sampling daemon 31_102 can determine whether the frequency forecast type or panorama forecast type is a better predictor of the received property event.

In some implementations, the sampling daemon 31_102 may maintain a scoreboard for each prediction type (e.g., default, panorama). For example, each time sampling daemon 31_102 determines that a frequency forecast type will be a better predictor for a received event, sampling daemon 31_102 may increment a score (e.g., counter) for the frequency forecast type. Whenever sampling daemon 31_102 determines that the panorama forecast type will be a better predictor for the received event, sampling daemon 31_102 may increment a score (e.g., a counter) for the panorama forecast type.

In some implementations, the sampling daemon 31_102 may determine a default prediction type based on scores generated for each forecast type (e.g., frequency, panorama). For example, if the scoreboard process generates a higher score for a panorama forecast type, the panorama will be assigned a default forecast type. If the scoreboard process generates a higher score for the frequency forecast type, the frequency will be assigned to the default forecast type. When a client requests a peer-to-peer or temporal forecast, the client may specify the forecast type (e.g., panorama, frequency, default). If the client does not specify a forecast type, then a default forecast type will be used to generate the peer-to-peer and/or temporal forecasts.

Attribute statistics

In some implementations, the client may request sampling daemon 31_102 to generate statistical information for the attributes or attribute values. For example, similar to prediction generation, the client may specify a history window through which statistics for attributes or attribute values should be generated. When statistics are generated for a specified attribute or attribute value, sampling daemon 31_102 will analyze attribute events that occur within a specified historical window. The client request may specify which statistics the sampling daemon 31_102 should generate.

In some implementations, sampling daemon 31_102 may generate "count" statistics for attributes or attribute values. For example, the "count" statistics may count the number of events associated with a specified attribute or attribute value that occurred in a specified history window.

In some implementations, sampling daemon 31_102 can generate statistical information based on the attribute values. For example, the client may request the first and/or last value of a certain attribute in a specified history window, and sampling daemon 31_102 may return that value. The client may request the minimum, maximum, average, mode, and standard deviation of all values associated with a specified attribute within a specified history window, and the sampling daemon 31_102 may return the values. The sampling daemon 31_102 can generate or determine which values are associated with the requested percentile (e.g., 10 th, 25 th, 50 th, 75 th, 90 th, etc.)

In some implementations, the sampling daemon 31_102 may generate duration statistics. For example, sampling daemon 31_102 may determine the duration associated with an attribute value by comparing the start event of the attribute to the stop event of the attribute. The time difference between the occurrence of the start event and the occurrence of the stop event will be the duration of the event. The client may request the minimum, maximum, mean, mode, and standard deviation of all durations associated with a specified attribute or attribute value within a specified history window, and the sampling daemon 31_102 may return the value. The sampling daemon 31_102 can generate or determine which duration values are associated with the requested percentile (e.g., 10 th, 25 th, 50 th, 75 th, 90 th, etc.)

In some implementations, the sampling daemon 31_102 may generate event interval statistics. For example, sampling daemon 31_102 may determine the time interval associated with the arrival or reporting of an event associated with a certain attribute value by comparing the first occurrence of the attribute event with subsequent occurrences of the attribute event. The time difference between the occurrence of the first event and the occurrence of the subsequent event will be the time interval between the occurrence of the two events. The client may request the minimum, maximum, average, mode, and standard deviation of all time interval values associated with a specified attribute or attribute value within a specified history window, and the sampling daemon 31_102 may return the values. The sampling daemon 31_102 can generate or determine which interval values are associated with the requested percentile (e.g., 10 th, 25 th, 50 th, 75 th, 90 th, etc.)

Keeping application up-to-date-get updates

Fig. 31_4 illustrates an exemplary system 31_400 for performing background fetch application updates. In some implementations, the mobile device 31_100 can be configured to predictively launch an application as a background process for the mobile device 31_100 so that the application can download content and update its interface in anticipation of the user invoking the application. For example, user application launch history data (e.g., a "system package name" start event) maintained by the sampling daemon 31_102 may be used to predict (predict) when a user will invoke an application of the mobile device 31_ 100. These predicted applications may be launched by the application manager 31_106 prior to the user invocation so that the user does not need to wait for the user invoked application to download the current content and update the graphical interface of the application.

Determining when to launch an application-time forecast

In some implementations, the application manager 31_106 can request an application call forecast from the sampling daemon 31_ 102. For example, the sampling daemon 31_102 may provide such an interface: which allows the application manager 31_106 to request a time forecast of application launches on the mobile device 31_100 (e.g., a "package name" start event). The sampling daemon 31_102 may receive an event (e.g., a "package name" start event) that indicates when a user invokes an application on the mobile device 31_100, as described above. When the application manager 31_106 requests a time forecast for the "package name" attribute, the sampling daemon 31_102 can analyze the "package name" events stored in the event data store 31_104 to determine when in the day (e.g., in which 15 minute time slot) the user will typically invoke the application. For example, the sampling daemon 31_102 may calculate the probability that a particular time of day or period of time will include an application call by the user using the above-described time prediction mechanism.

In some implementations, the application manager 31_106 can request a temporal forecast of the "package name" attribute from the sampling daemon 31_102 during initialization of the application manager 31_ 106. For example, the application manager 31_106 may be invoked or launched during the startup of the mobile device 31_ 100. When the application manager 31_106 is initializing, the application manager 31_106 may request a time forecast of application calls (e.g., "package name" start event) for the next 24 hours. Once the initial 24-hour period has elapsed, the application manager 31_106 may request a further 24-hour forecast. This 24-hour forecast period may continue until, for example, the mobile device 31_100 is turned off.

In some implementations, the sampling daemon 31_102 may generate an application call (e.g., a "package name" start event) time forecast for a 24 hour period. For example, the sampling daemon 31_102 may divide the 24 hour period into 96 15 minute time slots. The sampling daemon 31_102 may determine which applications have been invoked based on application launch history data (e.g., "package name" start event data) collected by the sampling daemon 31_102 and stored in the event data store 31_104, and when the application was invoked more than a certain number of times (e.g., 1 to 7) over the previous few days of operation.

In some implementations, when the sampling daemon 31_102 generates a temporal forecast for the "package name" attribute, each 15-minute slot may be ranked according to the probability that the (e.g., any) application will be invoked in the 15-minute slot, as described above in the temporal forecast section.

Once the application invocation probability for each of the 96 slots is calculated, the sampling daemon 31_102 may select a certain number (e.g., up to 64) of slots with the largest non-zero probability and return information identifying these slots to the application manager 31_ 106. For example, the sampling daemon 31_102 may send a list of times to the application manager 31_106 corresponding to the start of 15 minute time slots (e.g., 12:00 pm, 1:45 pm, etc.) corresponding to possible user invoked application launches (e.g., time slots with a score greater than zero).

In some implementations, the application manager 31_106 can set the timer based on the time slot provided by the sampling daemon 31_ 102. For example, the application manager 31_106 can create or set one or more timers (e.g., alarm clocks) corresponding to the time slots identified by the sampling daemon 31_ 102. When each timer is off (e.g., at 12:00 PM), the application manager 31_106 may wake up (e.g., if asleep, paused, etc.) and determine which applications should be started for the current 15 minute slot. Thus, the timer may trigger the acquisition of background updates for applications that may be invoked by the user within the corresponding time slot.

In some implementations, other events may trigger the acquisition of background updates for the application. For example, the application manager 31_106 may register interest in various events with the sampling daemon 31_ 102. For example, the application manager 31_106 may register an interest in events (e.g., attributes) related to opening a cellular radio, a baseband processor, or establishing a network connection (e.g., cellular or Wi-Fi), such that the application manager 31_106 may be notified when these events occur, and may trigger a background application launch such that application updates may utilize the active network connection. Unlocking the mobile device 31_100, opening the display, and/or other interactions may trigger the background application to launch and obtain updates, as described further below. In some implementations, if any background updates were performed within some number of minutes (e.g., seven minutes) before, the application manager 31_106 will not trigger the background application to launch nor obtain the updates.

Determining what application to launch-peer forecasting

In some implementations, the application manager 31_106 can request the sampling daemon 31_102 to provide a list of applications to be launched for the current time. For example, when a timer for a 15 minute time slot is down (e.g., time out) or a triggering event is detected, the application manager may request a peer forecast from sampling daemon 31_102 for the "package name" attribute so that sampling daemon 31_102 can determine which applications to start for the current time slot. The sampling daemon 31_102 may then generate a peer forecast that includes a list of application identifiers and corresponding scores that indicate the probability that each application will be invoked by the user at about the current time.

Fig. 31_5 shows a peer forecast for determining the user invocation probability of an application on the mobile device 31_ 100. For example, fig. 31_500 shows peer forecasts for recent history window specifications (e.g., the previous 2 hours). Fig. 31_530 shows peer-to-peer forecasts for a daily historical window specification (e.g., 4 hour period in each of the past 7 days). Fig. 31_560 shows peer-to-peer forecasts for a weekly history window specification (e.g., 4 hour period, once every 7 days). In some specific implementations, the sampling daemon 31_102 may perform time series modeling using peer-to-peer predictions for different overlapping window specifications to determine user invocation probabilities for applications on the mobile device 31_ 100. If an application is not present in the peer forecast, a zero probability value may be assigned to the application.

In some implementations, time series modeling can be performed by generating peer-to-peer forecasts for different time windows. For example, recent, daily, and weekly peer forecasts can be generated based on recent, daily, and weekly event history window specifications. Recent, daily, and weekly peer predictions may then be combined to determine the application to be launched at the current time, as described further below.

In some implementations, a user call probability may be generated based on recent application calls. For example, a user invocation probability may be generated by performing a peer forecast for the "package name" attribute, where the peer forecast has a window specification that specifies the previous two hours as a time period of interest (e.g., user-initiated application launch within the last two hours).

As shown in fig. 31_500, application startup history data (e.g., "package name" event data) may indicate the number of applications (e.g., four) that were started within the previous two hours. For example, dots and circles may represent applications, open circles may represent a single specific application (e.g., email, social networking application, etc.), and dashed open circles represent calls to other applications. The peer prediction probability score associated with a particular application using the most recent history (e.g., the first 2 hours) may be calculated by dividing the number of calls of the particular application within the first two hours (e.g., 2) by the total number of application calls within the first two hours (e.g., 4). In the illustrated case, the probability associated with a particular application using recent application launch history data is 2/4 or 50%.

The user invocation probability may be generated based on a daily history of application launches (e.g., which applications will be launched at the current time + -2 hours on each of the last seven days). For example, a user invocation probability may be generated by performing a peer forecast for the "package name" attribute, where the peer forecast has a window specification specifying the current time of day + -2 hours (e.g., a 4 hour recurrence width) as the time period of interest (e.g., user-initiated application launch within the last two hours), and the recurrence frequency is 24 hours (e.g., recurrence repeats every 24 hours).

Fig. 31_530 illustrates a daily calendar history of application startup (e.g., a "package name" start event), which can be used to determine the user invocation probability of an application. For example, each box of fig. 31_530 represents a time window (e.g., + -2 hours of current time) in each day (e.g., day 7) of a previous day (e.g., as specified in the window specification of peer forecast), which may be analyzed to determine a user invocation probability (e.g., a peer forecast score) for a particular application (e.g., an open circle). The probability associated with a particular application using daily historical data may be calculated by dividing the number of calls (e.g., 6) for that particular application in all windows by the total number of application calls (e.g., 22) in all windows. In the illustrated case, the probability associated with a particular application using daily startup history data is 6/22 or 27%.

The user invocation probability may be generated based on a weekly history of application launches (e.g., which applications will be launched at the current time of + -2 hours seven days ago). For example, a user invocation probability may be generated by performing a peer forecast for the "package name" attribute, where the peer forecast has a window specification that specifies the current time of day + -2 hours (e.g., a 4 hour recurrence width) as the time period of interest (e.g., user-initiated application launch within the last two hours), and the recurrence frequency is 7 days (e.g., recurrence repeats every 7 days).

Fig. 31_560 illustrates a weekly history of application launch (e.g., a "package name" start event), which can be used to determine the user invocation probability of an application. For example, if the current date and time is wednesday afternoon 1, the user invocation probability (e.g., peer forecast score) of the application may be based on the application launch conditions during the time window of last wednesday afternoon 1 or around afternoon 1 (e.g., + -2 hours). In the illustrated case, the probability (e.g., open circles) associated with a particular application using the weekly application launch history data is 1/4 or 25%.

In some implementations, a score may be generated for each application from a combination of recent, daily, and weekly user invocation probabilities. For example, the recent, daily, and weekly probabilities may be combined by calculating a weighted average of the recent (r), daily (d), and weekly (w) probabilities. Each probability may have an associated weight, and each weight may correspond to a predetermined importance of each probability as determined empirically. The sum of all weights may be equal to one. For example, the probability based on recent activations may be weighted 0.6, the daily probability may be weighted 0.3, and the weekly probability may be weighted 0.1. Thus, the combined probability score may be the sum of 0.6(r), 0.3(d), and 0.1(w) (e.g., a score of 0.6r +0.3d +0.1 w).

Referring back to fig. 31_4, once a probability score is determined for each application based on recent, daily, and weekly probabilities, the sampling daemon 31_102 may recommend a configurable number (e.g., three) of applications with the highest non-zero probability scores to the application manager 31_106 for launching to perform background fetch downloads/updates.

In some implementations, the sampling daemon 31_102 may exclude the following applications from the "what to start" analysis described above: applications that do not support background updates (e.g., get) application updates, applications that the user has closed background updates, applications that have opted out of background updates, and/or any applications that the user is currently using or in the foreground of the mobile device 31_100 display (since foreground applications are likely to already be up-to-date).

In some implementations, once the application manager 31_106 receives the recommended applications from the sampling daemon 31_102, the application manager 31_106 can query the sampling daemon 31_102 whether each recommended application can be launched. The sampling daemon 31_102 can use its local admission control mechanism (described below) to determine whether an application manager can launch a particular application. For example, the application manager 31_106 may send a "package name" attribute with an attribute value identifying one of the recommended applications to the sampling daemon 31_102 and request the sampling daemon 31_102 to perform admission control on the attribute value.

Local admission control

In some implementations, the sampling daemon 31_102 can perform admission control on attribute events on the mobile device 31_ 100. For example, admission control may be performed on an attribute or attribute value to determine whether a client application may perform an activity, action, function, event, etc. associated with the attribute. For example, a client of sampling daemon 31_102 may request admission of an attribute "package name" with a value "mail application". In response to receiving the admission request, the sampling daemon can determine whether the client can perform the activity associated with the "mail application" attribute value (e.g., execute the "mail application" application).

In some implementations, admission control can be performed based on budgets and feedback from the voter. For example, when sampling daemon 31_102 receives an admission control request, the request may include a cost associated with allowing the attribute event (e.g., a start application, "package name" start event). The sampling daemon 31_102 can examine the system level data budget, the system level energy budget, and/or the specific attribute budget to determine if the budget associated with the attribute has sufficient remaining credits to cover the attribute event. If there is no budget associated with the attribute (e.g., the attribute is not a budget attribute), the attribute event may be allowed to proceed (e.g., sampling daemon 31_102 will return a "OK" value in response to the admission control request). If there is a budget associated with the attribute and there are not enough remaining credits in the associated budget to pay for the cost of the event, the attribute event is not allowed to proceed (e.g., sampling daemon 31_102 will return a "no" value in response to the admission control request).

If there is a budget associated with the attribute and there are enough remaining credits in the budget to pay for the cost of the event, the voter will be required to make a decision on whether to allow the attribute to proceed. If all voters vote "yes," the attribute event will be allowed to proceed (e.g., sampling daemon 31_102 will return a "OK" value in response to the admission control request). If any voter votes "no," the attribute events will not be allowed to proceed (e.g., sampling daemon 31_102 will return a "no" value in response to the admission control request). Details about budgets and voters are described in the following paragraphs.

In some implementations, sampling daemon 31_102 may return a "never" value in response to an admission control request if an attribute or attribute value has not been reported to sampling daemon 31_102 in an event some time (e.g., 7 days, a month, etc.) before the admission control request. For example, sampling daemon 31_102 may generate temporal or peer forecasts to determine when events associated with attributes or attribute values are allowed or permitted. For example, events that are not expected to occur need not be preemptively retrieved (e.g., data need not be pre-read for applications that are not invoked by the user).

Admission control-budget

In some implementations, the sampling daemon 31_102 can perform admission control based on a budget associated with an attribute or attribute value. For example, sampling daemon 31_102 can determine whether to allow (e.g., grant) an activity (e.g., an event) associated with an attribute or attribute value based on a budget associated with the attribute or attribute value. In some implementations, the sampling daemon 31_102 can determine whether a certain attribute or attribute value can be granted based on a system level energy budget and/or a system level data budget configured for the mobile device 31_ 100. The sampling daemon 31_102 can store a budget in the count data store 31_402, including counters for tracking the remaining data and energy budget for the current time period (e.g., the current hour). When a client requests admission control to be performed for an attribute or attribute value, the client may specify a number representing the cost of allowing or granting the occurrence of an event associated with the attribute or attribute value. If there is sufficient integration in the budget associated with the attribute, the attribute event will be voted by the voter described below. If there is not enough credit in the budget associated with the attribute, the attribute event is not allowed to proceed.

System level energy budget

In some implementations, sampling daemon 31_102 can determine whether a certain attribute or attribute value can be granted based on the energy budget. For example, the energy budget may be a percentage (e.g., 5%) of the mobile device battery capacity (in milliamp hours).

In some implementations, the energy budget can be allocated every hour for 24 hours. For example, the sampling daemon 31_102 may utilize battery utilization statistics (e.g., "system. energy" events) collected and stored in the event data store 31_104 to determine allocations that reflect typical historical battery usage per hour over 24 hours. For example, as described above, a certain percentage of the energy budget may be specified for each hour based on historically or statistically determined energy usage allocation or application usage forecasts. There is at least a minimum energy budget amount (e.g., 0.1%, 1%, etc.) greater than zero per hour. For example, 10% of the energy budget may be allocated within hours of unused data, while the remaining 90% of the energy budget may be allocated within hours of active usage based on historical energy or application usage. With every hour elapsed, the current energy budget will be replenished with a new/current hour energy budget. Any energy budget remaining from the last hour will be added to the budget for the current hour.

In some implementations, the count data store 31_402 may include a counter for determining how much remaining energy budget is available. For example, count data store 31_402 may include one or more counters that are initialized with the energy budget for the current hour. When the energy budget is used by the attribute event, the energy budget may be decremented by a corresponding amount. For example, when an application is started or terminated using a "package name" start or stop event, the application manager 31_106 may notify the sampling daemon 31_ 102. The sampling daemon 31_102 can then notify the power monitor 31_109 when the application starts and when the application terminates. From the start and stop times, the power monitor 31_109 can determine how much energy is used by the application. The power monitor 31_109 can send the amount of power used by the application (e.g., by submitting a "system. energy" attribute event) to the sampling daemon 31_102, and the sampling daemon 31_102 can decrement the appropriate counter by the amount of power used.

In some implementations, the sampling daemon 31_102 may deny admission requests for this attribute when no energy budget remains for the current hour. For example, when the energy budget counter in the count data store 31_402 decrements to zero, no energy budget remains and no activities, events, etc. associated with the property bound to the energy budget can be admitted. If the energy budget remaining for the current hour is sufficient to cover the cost of the attribute event, sampling daemon 31_102 may return a "yes" value in response to the admission control request and allow the attribute event to proceed.

In some implementations, the sampling daemon 31_102 will not make admission control decisions based on the energy budget when the mobile device 31_100 is plugged into an external power source. For example, when the mobile device 31_100 is plugged into an external power source, a remaining energy budget of zero will not prevent the performance of the attributed event.

System level data budget

In some implementations, sampling daemon 31_102 can determine whether a certain attribute can be granted based on a data budget. For example, the sampling daemon 31_102 may determine the average amount of network data consumed by the mobile device 31_100 based on statistical data (e.g., "system. network bytes" attribute events) collected by the sampling daemon 31_102 and stored in the event data store 31_ 104. The network data budget may be calculated as the average daily percentage of network data consumed by the user/mobile device 31_ 100. Alternatively, the network data budget may be a predefined or configurable value.

In some implementations, the network data budget may be allocated every hour for 24 hours. For example, a certain minimum budget (e.g., 0.2MB) may be allocated for each hour. The remaining amount of network data budget may be allocated within 24 hours based on historical network data usage. For example, the sampling daemon 31_102 may determine how much network data is consumed per hour per day based on historical statistics (e.g., "system. network bytes" attribute events) and assign a percentage based on the amount of data consumed per hour. With every hour elapsed, the pre-data budget will be complemented with the new/current hour data budget. Any data budget remaining from the last hour may be added to the data budget for the current hour.

In some implementations, count data store 31_402 may maintain a data counter for network data pre-computation. As network data is consumed, the data counter may be decremented according to the amount of network data consumed. For example, the amount of network data consumed may be determined based on application start and stop events (e.g., "packet name" start or stop events) provided by the application manager 31_106 to the sampling daemon 31_ 102. Alternatively, the amount of network data consumed can be provided by the process that manages the network interface (e.g., network daemon 31_406, background transfer daemon 31_ 1302). For example, the network interface management process may report "system. network byte" events to the sampling daemon 31_102, which may be associated with application start and stop events (e.g., "package name" events), to determine how much data the application consumes.

In some implementations, the sampling daemon 31_102 may keep track of which network interface type (e.g., cellular or Wi-Fi) is used to consume network data and determine the amount of network data consumed based on that network interface type. The amount of network data consumed may be adjusted according to the weights or coefficients assigned to each interface type. For example, network data consumed on the cellular data interface may be assigned a coefficient of one (1). One in a few tenths (0.1) of a network data allocation may be allocated for consumption on a Wi-Fi interface. The total network data consumed may be calculated by adding the consumed cellular data plus the consumed Wi-Fi data and dividing by ten (e.g., total data +0.1 Wi-Fi). Thus, the data consumed over Wi-Fi will have a much smaller impact on the data budget than the data consumed over a cellular data connection.

In some implementations, the sampling daemon 31_102 may reply a "no" to the in-control request when no data budget remains for the current hour. For example, when the data budget counter in count data store 31_402 decrements to zero, no data budget remains and activities associated with the attributes bound to the data budget will all be allowed. If the remaining data budget in the current hour is sufficient to cover the data cost of the attribute event, the sampling daemon 31_102 may make a "yes" reply to the admission control request.

Attribute budget

In some implementations, attributes can be associated with budgets. For example, predefined attributes or self-defined (dynamically defined) attributes may be associated with a budget through the API of the sampling daemon 31_ 102. A client (e.g., an application, a function, a third party application, etc.) of the sampling daemon 31_102 can issue a request to the sampling daemon 31_102 to associate attributes with a client-defined budget. The budget may be, for example, some credit.

Once the budget is allocated, the reported event associated with the budget attribute can indicate a cost associated with the event, and the budget can be reduced according to the specified cost. For example, the predefined system attribute "system scan" may be configured on the mobile device 31_100 to indicate when the mobile device 31_100 scans for signals from other bluetooth low energy devices. The bluetooth LE scan may be run, for example, as a background task. Bluetooth LE scanning requires the bluetooth radio to be turned on, which in turn consumes energy from the mobile device 31_100 battery. To prevent the bluetooth LE scan from consuming too much energy, a certain budget (e.g., 24 points) may be allocated for the bluetooth LE scan attribute. Each time a bluetooth LE scan event is generated and reported to sampling daemon 31_102, the event may be reported as having a certain cost (e.g., 1) report. This cost may be deducted from the budget so that the budget of 24 is decremented by 1 each time the bluetooth LE scan attribute is reported in an event.

In some implementations, the attribute budget can be allocated over a period of time. For example, the "bluetooth LE scan" attribute budget may be evenly distributed over 24 hours, so that the "bluetooth LE scan" attribute can only cost 1 credit per hour. In some implementations, the attribute budget can be replenished at the end of a certain time period. For example, if the period of the "bluetooth LE scan" attribute budget is 24 hours, the "bluetooth LE scan" attribute budget may be replenished every 24 hours.

In some implementations, the budget associated with an attribute can be a subset (e.g., a sub-budget) of another budget. For example, the budget for an attribute may be specified as part of another budget (such as the system level data or system level energy budget described above). For example, a "mail application. mailbox" attribute may be associated with such a budget: this budget is 5% of the data budget allocated for the system. The "bluetooth LE scan" attribute may be associated with such a budget: this budget is 3% of the energy budget allocated for the system. A sub-budget (e.g., a "mailbox" budget) may be tied to a super-budget (e.g., a system data budget) such that reducing the sub-budget also reduces the super-budget. In some implementations, if the over-budget is reduced to zero, the sub-budget is also reduced to zero. For example, if the system data budget is zero, the "mailbox" attribute budget will also be zero, even if no event is reported for the "mailbox" attribute that would reduce the "mailbox" attribute budget.

In some implementations, the sampling daemon 31_102 client can request that the sampling daemon 31_102 return the remaining budget amount for the attribute. For example, the client may make a request to the sampling daemon 31_102 to get the remaining budget for the "bluetooth LE scan" attribute. If three of the 24 budgeted credits have been used, sampling daemon 31_102 may return a value of 21 to the requesting client.

In some implementations, the client may report an event that a specified amount of budget credit is spent when no credit remains in the budget for the associated attribute. In the case where there are no credits remaining in the budget, when sampling daemon 31_102 receives an event that requires 1 credit (e.g., a "bluetooth LE scan" event), sampling daemon 31_102 may decrement the budget (e.g., -1) and report back to the client the error of the event. The error may indicate, for example, that the attribute has no budget remaining.

Attribute budget modeling

In some implementations, the attribute budget can be allocated based on historical usage information. For example, when an event is reported for a budget attribute, requests to use the attribute budget (e.g., events associated with costs) can be tracked over time. If a budget of 24 is allocated for the bluetooth LE scan attribute, for example, the budget may initially be allocated evenly over 24 hours, as described above. As events are reported over time for the attributes associated with the budget, the sampling daemon 31_102 may analyze the reported events to determine the time during the 24 hours when the event is most likely to occur. For example, the sampling daemon 31_102 may determine that the "bluetooth LE scan" event occurs often around 8 am, 12 pm, and 6 pm, but rarely around 2 am. The sampling daemon 31_102 can use this event frequency information to form an allocation of the "bluetooth LE scan" attribute budget over 24 hours. For example, the sampling daemon may allocate two budget points for each time slot corresponding to 8 am, 12 pm, and 6 pm, and zero budget points for the time slot associated with 2 am.

Admission control-voting machine

In some implementations, the sampling daemon 31_102 can perform admission control based on feedback from other software (e.g., plug-ins, utilities, applications, heuristics) running on the mobile device 31_ 100. For example, other software may be configured to act as a voter for admission control with the sampling daemon 31_ 102. For example, sampling daemon 31_102 may be registered with voters (e.g., applications, utilities, daemons, heuristics, etc.) to vote on in-control decisions. For example, sampling daemon 31_102 may be configured to interact with the following voter: a voter that monitors the thermal condition of the mobile device 31_100, a voter that monitors the CPU usage of the mobile device 31_100, and/or a voter that monitors the battery power level of the mobile device 31_ 100. When the sampling daemon 31_102 receives an admission control request, each voter (e.g., hot, CPU, and battery) may be required to vote whether the activity associated with the specified attribute should be allowed. When all voters vote "yes," the attribute will be allowed (e.g., the activity associated with the attribute will be allowed to occur). When a single voter votes "no," the attribute will not be allowed (e.g., the activity associated with the attribute will not be allowed). In some implementations, the voter may be configured as plug-in software that may be dynamically (e.g., at runtime) added to the sampling daemon 31_102 to provide additional functionality to the admission control system. In some implementations, the voter may use the temporal forecasting mechanism and the peer forecasting mechanism described above when determining whether to permit or allow an event associated with a certain attribute or attribute value.

Network daemon

In some implementations, the network daemon 31_406 may be configured as an admission control block. Network daemon 31_406 can be configured to use the voting API of sampling daemon 31_102, which allows network daemon 31_406 to receive voting requests from sampling daemon 31_102 and provide voting responses to sampling daemon 31_102 (e.g., yes, no). For example, web daemon 31_406 may receive a table request from sampling daemon 31_102 that includes attributes and/or attribute values. When the mobile device 31_100 is connected to a voice call, for example, and not connected to a Wi-Fi network connection, the network daemon 31_406 may indicate that the sampling daemon 31_102 should not permit or allow events associated with attributes or attribute values. For example, to prevent background update processes (e.g., acquisition processes) from interfering with or degrading the quality of voice calls, the network daemon 31_406 will not allow events associated with starting background update processes (e.g., "package name" start events) when the user is connected to a voice call and not connected to a Wi-Fi connection. Thus, when the mobile device 31_100 is connected to a call and not connected to Wi-Fi, the network daemon 31_406 may return a "no" value in response to the voting request.

In some implementations, when the mobile device 31_100 has a poor quality cellular network connection, the network daemon 31_406 may indicate that the sampling daemon 31_102 should not allow or grant the attribute event. A poor cellular connection quality may be determined when the transmission rate and/or throughput is below a predetermined threshold. For example, if the mobile device 31_100 has a poor quality cellular network connection and is not connected to Wi-Fi, the network daemon 31_406 may prevent the admission or execution of an attribute event that would waste battery energy and cellular data by using the poor quality network connection (e.g., launching an application that would attempt to download or upload data over a poor cellular connection) by returning a "no" value when the sampling daemon 31_102 makes a decision request.

In some implementations, the network daemon 31_406 may vote "yes" for admission of the requested attribute when the network daemon 31_406 does not have information indicating a bad network condition or some other condition that would affect network data usage or system performance.

Hot daemon process

In some implementations, the hot daemon 31_110 application may be configured as an admission control voter. Thermal daemon 31_110 can be configured to use a voting API of sampling daemon 31_102 that allows thermal daemon 31_110 to receive voting requests from sampling daemon 31_102 and provide voting responses to sampling daemon 31_102 (e.g., yes, no). For example, the hot daemon may receive a voting request from sampling daemon 31_102 that includes the attribute and/or the attribute value. When thermal daemon 31_110 has detected a thermal event, thermal daemon 31_110 may indicate that sampling daemon 31_102 should not permit or allow the event associated with the attribute or attribute value. For example, the thermal daemon 31_110 can monitor the temperature of the mobile device 31_100 and report the temperature value to the sampling daemon 31_102 by generating an event that includes a "hot level" attribute and a corresponding temperature value.

In some implementations, when thermal daemon 31_110 determines that the temperature of mobile device 31_100 is above a threshold temperature value, thermal daemon 31_110 may further prevent thermal daemon 31_102 from allowing attribute events that may increase the operating temperature of mobile device 31_100 by returning a "no" value when sampling daemon 31_102 sends a request to thermal daemon 31_110 to vote on the attribute (e.g., "package name") event.

In some implementations, sampling daemon 31_102 will only require a vote from thermal daemon 31_110 when there is currently an abnormal thermal condition. For example, the sampling daemon 31_102 may maintain a thermal condition value (e.g., true, false) that indicates whether the mobile device 31_100 is operating under normal thermal conditions. For example, if the current thermal condition of the mobile device 31_100 is normal, the thermal condition value may be true. The thermal condition value may be false if the current thermal condition of the mobile device 31_100 is abnormal (e.g., overheated, above a threshold temperature). Initially, the thermal condition value may be set to true (e.g., normal operating temperature). Upon detecting that the operating temperature rises above the threshold temperature, the thermal daemon 31_110 may send an updated value (e.g., false) of the thermal condition value to the sampling daemon 31_102 indicating an abnormal operating temperature. Once the mobile device 31_100 cools to a temperature below the threshold temperature, the thermal daemon 31_110 may update the thermal condition value (e.g., true) to indicate that the operating temperature is normal.

When sampling daemon 31_102 receives an admission control request for a certain attribute, sampling daemon 31_102 can check the thermal condition value to determine if thermal daemon 31_110 is required to vote on the admission (permission) of that attribute event. If the thermal condition value indicates that the operating temperature is normal (e.g., the value is true), then sampling daemon 31_102 will interpret the thermal condition value as a "yes" vote from thermal daemon 31_ 110.

If the thermal condition value indicates that the operating temperature is abnormal (e.g., the value is false), sampling daemon 31_102 will send the attribute and/or attribute value to thermal daemon 31_110 to allow thermal daemon 31_110 to vote on a particular attribute or attribute value.

In some implementations, the thermal daemon 31_110 can determine how to vote on the attributes and/or attribute values (e.g., yes, no) based on the current thermal conditions of the mobile device 31_100 and peer forecasts of the attributes. For example, the thermal daemon 31_110 can request a peer forecast for that attribute from the sampling daemon 31_ 102. The thermal daemon 31_110 may request peer forecasting for the current time by generating a window specification that includes the current time in the time period of interest (e.g., + -1 hour, 2 hours, etc.). The hot daemon 31_110 will receive a peer forecast from the sampling daemon 31_102 indicating a likelihood score for each value of the attribute that occurs in the time period of interest. For example, if thermal daemon 31_110 requests an equal forecast for the "package name" attribute, then thermal daemon 31_110 may receive a list of "package name" values (e.g., application identifiers) and associated forecast (e.g., probability, likelihood) scores. For example, if a "package name" event with attribute values "mail application", "contact", "calendar", "web browser", "mail application" and "mail application" occurs during a time period of interest, the relative probability (i.e., score) of the "mail application" occurring is 0.43 (e.g., 3/7), the relative probability of the "web browser" occurring is 0.29 (e.g., 2/7), and the relative probability of the "contact" or "calendar" occurring is 0.14 (e.g., 1/7). In some implementations, the hot daemon 31_110 may order the list of attribute values according to score (e.g., top score in front and lowest score in back). For example, the above-mentioned ranked list of "package name" attribute values has, from top to bottom: "mail application", "web browser", "contacts", and "calendar".

In some implementations, the hot daemon 31_110 may determine when to vote "yes" on the attribute value based on the position of the attribute value in the ordered list. For example, if the attribute value considered by thermal daemon 31_110 is not in the peer forecast list received from sampling daemon 31_102, the attribute value will receive a "no" vote from thermal daemon 31_ 110. If the attribute value is in the peer prediction list and below a threshold level (e.g., index) in the list (e.g., at the lowest 25% of the attribute based on the score), then the thermal daemon 31_110 will vote "no" for the attribute. If the attribute value is in the peer forecast list and above a threshold level in the list (e.g., at the highest 75% of the attributes based on the score), then the thermal daemon 31_110 will vote "yes" for the attribute. Once the vote is determined, hot daemon 31_110 will return a "yes" (e.g., true) or "no" (e.g., false) vote to sampling daemon 31_ 102.

In some implementations, the thermal daemon 31_110 may be configured to have a maximum threshold level to avoid voting "no" on all attribute values (e.g., so that certain attribute events will occur). The maximum threshold level may be 50% of the attribute values in the ranked peer forecast list (e.g., the first 50% get a "yes" vote and the last 50% get a "no" vote). Thus, the hot daemon 31_110 may adjust the threshold level that separates the attribute values that will receive a "yes" vote from the attribute values that will receive a "no" vote from 0% to 50% of the attribute values with the lowest score.

In some implementations, the threshold level for determining a "yes" or "no" vote can be proportional to the thermal level (e.g., temperature) of the mobile device 31_ 100. For example, the thermal daemon 31_110 may be configured to have a maximum operational thermal level (Lh) and a normal operational level (Ln). The thermal daemon 31_110 may determine the current operating heat level (Lc) and determine the percentile of the thermal range (e.g., Lh-Ln) that the mobile device 31_100 is currently operating in (e.g., Lc-Ln/Lh-Ln ═ percent). The hot daemon 31_110 may use the calculated percentile to determine which part of the 0-50% attribute values should receive a "no" vote. For example, if the current operating heat level is calculated to be 65% of the hot range, the attribute value of the last 32.5% resulting from the peer forecast score will get a "no" vote from the thermal daemon 31_ 110. Thus, the least important attribute values will get a "no" vote, while the most important attribute values will get a "yes" vote. Referring back to the "package name" example above, if the ordered list of "package name" attribute values is, from top to bottom: "mail application", "web browser", "contact", and "calendar", then "calendar" would receive a "no" vote, while "mail application", "web browser", and "contact" would receive a "yes" vote (e.g., "mail application", "web browser", and "contact" are the most common application programs). For example, if the application manager 31_106 has made an admission control request for the "package name" attribute to determine the application to launch, the "mail application", "web browser", and "contacts" applications will be launched, while the "calendar" application will not be launched.

As another example, the hot daemon 31_110 may be required to perform a table resolution on the "mail application mailbox" attribute. A peer forecast may be generated for the "mail application mailbox" attribute value that produces an ordered list of mail folders that indicates the most frequently accessed folder to the least frequently accessed folder (e.g., "inbox," "personal," "work," "home," "spam," and "trash"). If the latter 32.5% of the attribute values are to receive a "no" vote, then "spam" and "trash" will receive a "no" vote. For example, if a "mailbox" application is directed to a "mail application, the" mail application "application makes admission control requests to determine which folders are to get email, then the" mail application "application will get email for the" inbox "," personal "," work ", and" home "folders, but not for the" spam "and" trash "folders. In some implementations, the attribute or attribute value that has received a "no" vote from the thermal daemon 31_110 may be notified when the thermal condition value maintained by the sampling daemon 31_102 is reset to indicate a normal operating temperature (e.g., true). For example, the sampling daemon 31_102 may store data identifying clients, attributes, and attribute values that have received a "no" vote. Upon receiving an updated thermal condition value (e.g., true) from thermal daemon 31_110, sampling daemon 31_102 can send a notification to the client that received the "no" vote to prompt the client to attempt to make another admission control request for the previously rejected attribute or attribute value. In some implementations, the client may resend the admission control request without getting a prompt from the sampling daemon 31_ 102. For example, a client may have an internal timer that causes the client to retry an admission control request after a period of time has elapsed.

Activity monitor

In some implementations, the activity monitoring application 408 may be configured as an admission control block. The activity monitor 31_408 can be configured to use the voting API of the sampling daemon 31_102, which allows the activity monitor 31_408 to receive voting requests from the sampling daemon 31_102 and provide voting responses to the sampling daemon 31_102 (e.g., yes, no). For example, activity monitor 31_408 can receive a voting request from sampling daemon 31_102 that includes attributes and/or attribute values. When the mobile device 31_100 uses more than a threshold amount (e.g., 90%) of memory resources or CPU resources, the activity monitor 31_408 may indicate that the sampling daemon 31_102 should not grant or allow events associated with attributes or attribute values. For example, if the mobile device 31_100 is already running many applications or processes that are using most of the memory resources or CPU resources of the mobile device 31_100, launching additional applications in the background would likely reduce the performance of the mobile device 31_100 by running out of the remaining memory resources. Thus, when the activity monitor 31_408 determines that the memory or CPU usage exceeds a threshold (e.g., 75%), the activity monitor 31_408 can prevent the application manager 31_106 from launching additional applications by returning a "no" value when the sampling daemon 31_102 sends a request to vote on a "package name" property event. If the activity monitor 31_408 determines that the memory and/or CPU resources of the mobile device 31_100 are below a threshold usage amount, the activity monitor 31_408 may return a "YES" value in response to the voting request from the sampling daemon 31_ 102.

Launching a background fetch application

In some implementations, when the application manager 31_106 makes an admission control request to the sampling daemon 31_102 and receives a "yes" reply, the application manager 31_106 may invoke or launch the identified application (e.g., the application 31_108 as identified by the "package name" attribute value) in the operating environment background of the mobile device 31_ 100. For example, the application 31_108 may be launched in the background so that the launching of the application 31_108 is not noticeable to the user. The application 31_108 may then communicate with the content server 31_404 via a network (e.g., the internet) to download the updated content for display to the user. Thus, when the user subsequently selects the application 31_108 (e.g., brings the application to the foreground), the device will present the current and up-to-date content to the user without having to wait for the application 31_108 to download the content from the server 31_404 and refresh the application's user interface.

In some implementations, the application manager 31_106 can be configured to launch applications that support background acquisition when the mobile device 31_100 is charging and connecting to Wi-Fi. For example, the sampling daemon 31_102 may determine when the mobile device 31_100 is connected to an external power source (e.g., based on a "cable plug in" attribute event) and to a network (e.g., the internet) through Wi-Fi (e.g., based on the received event) and send a signal to the application manager 31_106 to cause the application manager 31_106 to launch an acquisition-enabled application used within a certain amount of time (e.g., seven days) in the past.

Exemplary background acquisition Process

Fig. 31_6 is a flow diagram of an exemplary process 31_600 for predictively launching an application to perform background updates. For example, the process 31_600 may be executed by the application manager 31_106 and the sampling daemon 31_102 to determine when to launch a background application configured to obtain data updates from a network resource (such as the content server 31_404 of fig. 31_ 4). Additional description related to the steps of the process 31_600 may be found with reference to fig. 31_4 and 31_5 above.

At step 31_602, the application manager 31_106 can receive an application call forecast from the sampling daemon 31_ 102. For example, the application manager 31_106 may be started during the start-up of the mobile device 31_ 100. During its initialization, the application manager 31_106 may request a forecast for applications that may be invoked by the user of the mobile device 31_100 in the next 24-hour period. For example, the application manager 31_106 may request a time forecast for the attribute "package name". This forecast may indicate when the application is launched. For example, a 24 hour period may be divided into multiple 15 minute periods, and each 15 minute period may be associated with a probability that the user invoked the application within the 15 minute period. The forecast returned to the application manager 31_106 may identify up to 64 15 minute periods in which the user is likely to invoke the application.

At step 31_604, the application manager 31_106 may set a timer based on the application start prediction. For example, the application manager 31_106 can set a timer or alarm for each 15-minute period identified in the application launch forecast returned to the application manager 31_106 by the sampling daemon 31_ 102.

At step 31_606, application manager 31_106 can request sampling daemon 31_102 to identify the application to launch. For example, when a timer expires or an alarm is stopped, the application manager (if sleeping or paused) may wake up and request a list of applications to be launched from the sampling daemon 31_102 for the current 15 minute period. The sampling daemon 31_102 may return a list of applications that should be started in the mobile device 31_100 background. For example, the application manager 31_106 may request a peer forecast for the attribute "package name". The peer forecast may indicate which values of the "package name" attribute are most likely to be reported in the current 15 minute time slot (e.g., which applications the user is most likely to invoke).

At step 31_607, the application manager 31_106 may send a request to the sampling daemon 31_102 asking if the application can be started. For example, for each application identified by sampling daemon 31_102 in response to a "package name" peer forecast request, application manager 31_106 can query sampling daemon 31_102 whether to launch that application. For example, the application manager 31_106 may request the sampling daemon 31_102 to perform admission control for a particular value of the "packet name" attribute, where the particular value corresponds to the application that the application manager 31_106 is attempting to launch. The daemon 31_102 may return yes if the application can be started, no if the application cannot be started, and never if the application cannot be started.

At step 31_610, the application manager 31_106 may launch the application. For example, if the sampling daemon 31_102 returns a "determine" (e.g., determine, yes, true, etc.) answer to the admission control request, the application manager 31_106 will launch the application as a background process for the mobile device 31_ 100. If the sampling daemon 31_102 returns a "no" or "never" response to the admission control request, the application manager 31_106 will not start the application.

At step 31_612, the application manager 31_106 can send an application start notification to the sampling daemon 31_ 102. For example, the application manager 31_106 may send a "package name" start event to the sampling daemon 31_102 to record the execution of the started application.

At step 31_614, the application manager 31_106 may detect that the launched application has terminated. For example, the application manager 31_106 may determine when a launched application is no longer running on the mobile device 31_ 100.

At step 31_616, the application manager 31_106 can send an application termination notification to the sampling daemon 31_ 102. For example, the application manager 31_106 may send a "package name" end event to the sampling daemon 31_102 to record the termination of the application.

Fig. 31_7 is a flow diagram of an exemplary process 31_700 for determining when to launch an application on a mobile device 31_ 100. For example, the process 31_700 may be used to determine when an application is launched, what applications should be launched, and whether an application may be launched based on application usage statistics (e.g., "package name" attribute event data), data and energy budgets, and mobile device operating conditions and environmental conditions, as described in detail above with reference to fig. 31_ 4.

At step 31_702, the sampling daemon 31_102 may receive an application launch forecast request from the application manager 31_ 106. For example, the application manager 31_106 may request a time forecast for the next 24 hours for the "package name" attribute from the sampling daemon 31_ 102. Once the 24 hour period has elapsed, the application manager 31_106 may request a time forecast for the subsequent 24 hour period for the "package name" attribute. For example, the application manager 31_106 may request a time forecast for the "package name" attribute every 24 hours.

At step 31_704, sampling daemon 31_102 can determine that the application starts a pre-report. For example, an application launch forecast (e.g., a time forecast for the "package name" attribute) may be used to predict when a user-initiated application launch may occur during a 24 hour period. The 24 hour period may be divided into multiple 15 minute periods. For each 15-minute period (e.g., 96 15-minute periods within 24 hours), the sampling daemon 31_102 may use historical user invocation statistics (e.g., "package name" start events) to determine the probability that an application launch initiated by the user will occur in that 15-minute period, as described above with reference to FIG. 31_ 4.

At step 31_706, the sampling daemon 31_102 can transmit an application launch forecast to the application manager 31_ 106. For example, the sampling daemon 31_102 may select a maximum of 64 15 minute periods with the highest non-zero probability of application launch initiated by the user. Each of the selected 15 minute periods may be identified by a start time of the 15 minute period (e.g., 12:45 pm). The sampling daemon 31_102 can send the list of 15 minute period identifiers for the application manager 31_106 as an application launch forecast (e.g., a time forecast for the "package name" attribute).

At step 31_708, sampling daemon 31_102 can receive a request for what application to start at the current time. For example, the application manager 31_106 can send a request to the sampling daemon 31_102 for the sampling daemon 31_102 to determine which applications should be started at or around the current time. For example, the request may be a request for peer forecasting of the current 15 minute time slot for the "package name" attribute.

At step 31_710, sampling daemon 31_102 can score the application at the current time based on historical event data. The sampling daemon 31_102 can determine applications that the user is likely to start in the near future based on the historical start of application data (e.g., "package name" property start event data) initiated by the user collected by the sampling daemon 31_ 102. The sampling daemon 31_102 can score applications based on historical probabilities that a user will invoke an application at or around the current time using the most recent application startup data, daily application startup data, and/or weekly application startup data, as described above with reference to fig. 31_4 and 31_ 5.

At step 31_712, sampling daemon 31_102 can communicate the application and application score to application manager 31_ 106. For example, the sampling daemon 31_102 may select a certain number (e.g., three) of applications (e.g., the "package name" attribute values) with the highest scores (e.g., the highest probability of being invoked by the user) to communicate to the application manager 31_ 106. The sampling daemon 31_102 may exclude applications that have been launched within a previous period of time (e.g., the previous 5 minutes). The sampling daemon 31_102 may send information to the application manager 31_106 identifying the highest scoring application and its respective score, as described above with reference to fig. 31_ 4.

At step 31_714, sampling daemon 31_102 may receive a request from application manager 31_106 to determine if an application may be launched. For example, the sampling daemon 31_102 may receive an admission control request that identifies an application (e.g., a "packet name" value).

At step 31_716, the sampling daemon 31_102 can determine that the current mobile device conditions and budget allow the application to start. For example, in response to an admission control request, the sampling daemon 31_102 may examine the system level data and energy budgets, the attribute budgets, and the voter feedback to determine that the application should be launched as a background task on the mobile device 31_100, as described in detail above with reference to fig. 31_ 4.

At step 31_718, sampling daemon 31_102 may transmit a reply to application manager 31_106 indicating that the identified application may be launched. For example, if the condition applies to a background application launch, the sampling daemon 31_102 may return a "yes" value (e.g., ok, yes, true, etc.) to the application manager 31_106 in response to the admission control request so that the application manager 31_106 may launch the identified application.

Short term trends

In some implementations, sampling daemon 31_102 may be configured to detect when attributes tend to appear. For example, a client application may register interest in a particular attribute with sampling daemon 31_ 102. When sampling daemon 31_102 detects that a particular attribute is trending to occur, sampling daemon 31_102 can notify the client that the particular attribute is trending to occur.

For example, the application manager 31_106 may register an interest for the "package name" attribute (or a specific value of the "package name" attribute). When the sampling daemon 31_102 determines that the "package name" attribute (or its value) tends to occur, the sampling daemon 31_102 may notify the application manager 31_106 of the trend so that the application manager 31_106 may predictively launch the application that tends to occur in the background of the mobile device 31_ 100. For example, if an application is being repeatedly invoked by the user of the mobile device 31_100, the application is tending to appear. In some cases, the emerging applications are new applications that may not be included in the "package name" property peer forecast described above, or applications that are rarely used until they are emerging. Thus, the emerging application may not remain current using the application launch forecasting method described above.

The purpose of attribute trend detection is to detect attributes (e.g., attribute events) that are repeatedly reported to sampling daemon 31_102 and determine the approximate cadence (e.g., periodicity) at which the attributes are initiated, making errors when the reported cadence is small. The attribute being repeatedly reported to sampling daemon 31_102 is said to be "prone to occur". The sampling daemon 31_102 client can then use the determined rhythm to perform a function or operation in anticipation of the next event associated with the attribute that tends to occur.

For example, the application manager 31_106 may use the determined cadence to set a timer that will trigger the application manager 31_106 to start a trending application in the background so that when the user invokes the application, the application will be updated, as described above. For example, if the rhythm of the application is 5 minutes, the application manager 31_106 may set a timer that will expire every 4 minutes and cause the application manager 31_106 to launch the application so that the application may receive updated content and update the interface of the application before being invoked again by the user.

In some implementations, the trend detection mechanisms described in this section can be used to detect other system event trends, including application startup, such as repeated software or network notifications, application crashes, and the like. For example, a client may register an interest in any attribute or attribute value and may receive a notification when the attribute of interest tends to appear.

In some implementations, the sampling daemon 31_102 may maintain a trend table that may be used to track multiple attribute behaviors. The trend table may include an attribute value identification field (ATTID), a status field (STATE), a most recent launch timestamp (LLT), an inter-launch rhythm (ILC) indicating an amount of time between launches, and a confidence field (C).

Fig. 31_8 is a flow chart 31_800 showing state transition of a certain entry (e.g., application) in the trend table. Initially at step 31_802, the trend table may include an empty entry (e.g., a record) with the ATTID, LLT, ILC, and C fields being empty (e.g., N/a) and STATE being set to "invalid" (I). When an attribute event is reported at time t, the trend table is scanned for available entries (e.g., entries in state I). Among the possible invalid entries, several methods may be used to select the entry to use. For example, a random invalid entry may be selected. Alternatively, an invalid entry may be selected such that all empty entries in the trend table maintain a continuous order. If no invalid entries exist, the oldest entry in the transient (T) state (or random entry) may be selected to track the newly launched application. If there are no I or T state entries, the oldest new (N) state entry may be selected to track the newly reported attribute events.

At step 31_804, once the trend table entry is selected, the STATE field of the selected entry for tracking new report attribute events may be set to new (N), the ATTID may be set to the attribute value of the new report attribute, the LLT field may be set to the current time t (e.g., wall clock time), and the ILC and C fields are set to predefined minimum values ILC _ MIN (e.g., 1 minute) and C _ MIN (e.g., zero).

At step 31_806, on the next report of the same attribute event at time t', the entry is found if the entry for that attribute in the table is still present and not evicted (e.g., selected to track another attribute). The entry's STATE is set to instantaneous (T), the ILC is set to the difference between LLT and the current system time (e.g., T ' -T or T ' -LLT), and the C field is incremented (e.g., by a predefined value, C DELTA). Alternatively, the ILC field may be set to other functions of its old and new values, such as a running average.

At step 31_808, on the next report of the same attribute event at time t', the entry is found if the entry for that attribute in the table is still present and not evicted (e.g., selected to track another attribute). The STATE of the entry may remain set to transient (T), the ILC is set to the difference between the LLT and the current clock (e.g., wall clock) time (e.g., T '-T or T' -LLT), and the C field is incremented again (e.g., by a predefined value, C DELTA).

At step 31_810, if the C value of the trend table entry reaches (e.g., equals) a threshold (e.g., C _ high threshold) after several reports of the attribute event, the STATE of the attribute entry may be changed to STATE ═ a at step 31_ 811. If at step 31_810 the C value of the trend table entry does not reach a threshold (e.g., C _ HIGHTHRESHOLD), the value of the entry may be updated according to step 31_ 808.

In state "a," whenever an attribute event is reported, if the time between the last report and the current report time is within a certain amount of time (e.g., ILC _ EPSILON 5 minutes), the confidence (C) field of the attribute entry is incremented until a predefined maximum value (e.g., C _ MAX) is reached. When an attribute entry in the trend table is in the active (a) state, the ILC value of the entry may be used as an estimate of the firing rate (e.g., cadence) and the ATTID of the entry may be used to identify the attribute value that tends to occur.

In some implementations, sampling daemon 31_102 may send an attribute value (ATTID) and a cadence value (ILC) to the client so that the client may perform certain actions or functions in anticipation of the next event associated with the attribute value. For example, the attribute value and the tempo value may be sent to the application manager 31_106 so that the application manager 31_106 may launch the identified application in the background (e.g., ATTID, "package name" attribute value) in anticipation of the application user invoking the application so that the application may receive updated content before the user launches the application, as described above. For example, the application manager 31_106 may start a timer based on a tempo value that will wake up the application manager 31_106 to start the application in anticipation of a user invoking the application.

In some implementations, sampling daemon 31_102 may notify the client of the expected next occurrence of a property event based on the detected property trend. For example, the sampling daemon 31_102 may send a signal or notification to the application manager 31_106 indicating that the application manager 31_106 should launch a trending application. The application manager 31_106 may register interest in the application by sending an application identifier (e.g., a "package name" attribute value) to the sampling daemon 31_ 102. The sampling daemon 31_102 may monitor the user invocation of the application (e.g., based on the reported "package name" start event) to determine if the application is trending to appear, as described above. If the application is trending, the sampling daemon 31_102 can determine the cadence of the invocation and send a notification or signal to the application manager 31_106 at the determined time based on the cadence, as described above. For example, if the cadence is four minutes, the sampling daemon 31_102 can send a signal to the application manager 31_106 every 3 minutes (e.g., some time period before the next occurrence of an event) to cause the application manager 31_106 to launch an application. If the cadence change is 6 minutes, the sampling daemon 31_102 can detect the cadence change and adjust it when the application manager 31_106 receives the signal. For example, the sampling daemon 31_102 may signal the application manager 31_106 to start the application every 5 minutes instead of every 3 minutes, adjusting to a slower pace (e.g., the time period between calls increases).

Each time the attribute trend table is checked for any reason (e.g., adding a new entry, updating an existing entry, etc.), the C values of all entries in either STATE T or STATE a (ILC _ EPSILON greater than its ILC since its last boot) will be decremented. Any entry at this point with a C value below the lowest threshold (e.g., C _ LOWTHHRESHOLD) is demoted. For example, an entry may demote from State A to State T, or from State T to State I.

In some implementations, the above-described trend detection mechanism may be used to detect trend events other than application calls or launches. For example, any reproducibility event (e.g., any attribute event) on the mobile device 31_100 may be detected and tracked using the above-described trend detection method and trend table. Trending events may include screen touches, network connections, application failures, network intrusions, and/or any other event that may be reported or signaled to sampling daemon 31_ 102.

Push notification

Fig. 31_9 is a block diagram 31_900 showing a system for providing push notifications to a mobile device 31_ 100. In some implementations, the mobile device 31_100 can be configured to receive push notifications. For example, the push notification may be a message initiated by the push provider 31_902 and sent through the push notification server 31_906 to the push service daemon 31_904 running on the mobile device 31_ 100.

In some implementations, the push provider 31_902 can receive authorization to send push notifications to the mobile device 31_100 through a user authorization request presented by the application 31_908 to the user of the mobile device 31_ 100. For example, the push provider 31_902 may be a server owned, operated, and/or maintained by the same vendor that created (e.g., programmed, developed) the application 31_ 908. When the application 31_908 presents a user interface on the mobile device 31_100 requesting authorization of the push provider 31_902 to send a push notification to the mobile device 31_100, and the user indicates that the push notification is authorized, the push provider 31_902 may receive authorization from the user to send the push notification to the mobile device 31_100 (e.g., the push service daemon 31_ 904). For example, the user may select a button presented by the application 31_908 on the user interface to indicate that a push notification is authorized for the push provider 31_902 and/or the application 31_ 908. The push provider 31_902 may then receive a device token that identifies the mobile device 31_100 and may be used to route push notifications to the mobile device 31_ 100. For example, the push notification server 31_906 may use the push notification receiving device token and use the device token to determine which mobile device 31_100 should receive the push notification.

In some implementations, the mobile device 31_100 can send information identifying the authorized push application to the push notification server 31_ 906. For example, the mobile device 31_100 may send a message 31_926 containing the push notification filter 31_914 and a device token for the mobile device 31_100 to the push notification server 31_ 906. The push notification server 31_906 may store a mapping of device tokens (e.g., identifiers of the mobile devices 31_ 100) to the push filter 31_914 for each mobile device served by the push notification server 31_ 906. The push filter 31_914 may include, for example, information identifying such applications: the application has received authorization to receive push notifications on the mobile device 31_ 100.

In some implementations, the push filter 31_914 may be used by the push notification server 31_906 to filter out (e.g., block sending) push notifications sent to applications that have not been authorized by the user of the mobile device 31_ 100. Each push notification sent by the push provider 31_902 to the push notification server 31_906 may include information (e.g., an identifier) identifying an application 31_908 (e.g., a device token) associated with the push provider 31_902 and the mobile device 31_ 100.

When notification server 31_906 receives a push notification, notification server 31_906 may use mobile device identification information (e.g., a device token) to determine which push filter 31_914 to apply to the received push notification. The notification server 31_906 may compare the application identification information in the push notification with the push filter 31_914 of the identified mobile device to determine whether the application associated with the push provider 31_902 and identified in the push notification is identified in the push filter 31_ 914. If an application associated with the push notification is identified in the push filter 31_914, the notification server 31_906 may send the push notification received from the push provider 31_902 to the mobile device 31_ 100. If the application identified in the push notification is not identified in the push filter 31_914, the notification server will not send the push notification received from the push provider 31_902 to the mobile device 31_100 and the push notification may be deleted.

Non-wakeup push notification

In some implementations, notification server 31_906 may be configured to handle high priority push notifications and low priority push notifications. For example, the push provider 31_902 may send a high priority push notification 31_910 and/or a low priority push notification 31_912 to the push notification server 31_ 906. The push provider 31_902 may identify the push notification as high priority or low priority, for example, by specifying the priority of the push notification in the data contained in the push notification sent to the push notification server 31_906 and the mobile device 31_ 100.

In some implementations, the push notification server 31_906 may handle the low priority push notification 31_912 differently than the high priority push notification 31_ 910. For example, the push notification server 31_906 may be configured to compare the application identification information contained in the high priority push 31_910 with the authorized application identification information in the push filter 31_914 to determine whether the high priority push notification 31_910 may be sent to the mobile device 31_ 100. The push notification server 31_906 may send a high priority push notification to the mobile device 31_100 if the application identification information in the high priority push notification 31_910 matches the authorized application identifier in the push filter 31_ 914. If the application identification information in the high priority push notification 31_910 does not match the authorized application identifier in the push filter 31_914, the push notification server 31_906 will not send the high priority push notification to the mobile device 31_ 100.

In some implementations, the push notification server 31_906 may be configured to delay delivery of low priority push notifications. For example, when the mobile device 31_100 receives a push notification from the push notification server 31_906, the receipt of the push notification causes the mobile device 31_100 to wake up (e.g., if in a sleep or low power state). When the mobile device 31_100 wakes up, the mobile device 31_100 will turn on a depletable battery, use cellular data, cause the mobile device 31_100 to warm up, or otherwise affect the various subsystems and processors of the mobile device 31_ 100. By preventing or delaying delivery of the low priority push notification to the mobile device 31_100, the mobile device 31_100 may conserve network (e.g., cellular data) and system (e.g., battery) resources.

In some implementations, the push notification filter 31_914 may include a wake up list 31_916 and a do not wake up list 31_ 918. The wake-up list 31_916 may identify for which application a low priority push notification should be delivered to the mobile device 31_ 100. In some implementations, when an application is authorized to receive push notifications at the mobile device 31_100, the application identification information is added to the wake list 31_914 by default. The do-not-wake list 31_918 may identify authorized applications that should delay low priority push notifications. The specific mechanisms for populating the do-not-wake list 31_918 and/or manipulating the wake list 31_916 and the do-not-wake list 31_918 are described in detail below in describing the push notification initiated background update. In some implementations, the high priority push notification will not be delayed at the push notification server 31_906 and will be pushed to the mobile device 31_100 as long as the application identified in the high priority push notification is identified in the push filter 31_914 (e.g., the wake list 31_914 and/or the do not wake list 31_ 918).

In some implementations, when the push notification server 31_906 receives the low priority push notification 31_912, the push notification server 31_906 may compare the application identifier in the low priority push notification 31_912 to the wake up list 31_916 and/or the do not wake up list 31_ 918. For example, if the application identification information in the low priority push notification 31_912 matches the authorized application identifier in the wake list 31_916, the low priority push notification 31_912 will be delivered to the mobile device 31_100 in the notification message 31_ 920.

In some implementations, the delivery of low priority push notifications associated with the applications identified in the do-not-wake list 31_918 may be delayed. For example, if an application identified in the low priority push notification 31_912 is also identified in the do-not-wake list 31_918, the low priority push notification 31_912 may be stored in the push notification data store 31_922, rather than delivered immediately to the mobile device 31_ 100. In some implementations, if the mobile device 31_100 identified by the push notification (high or low priority) is not currently connected to the push notification server 31_906, the push notification for the unconnected mobile device 31_100 may be stored in the push notification data store 31_922 for later delivery to the mobile device 31_ 100.

In some implementations, the push notifications stored in the push data store 31_922 will remain in the push data store 31_922 until the application identifier associated with the stored push notifications is moved from the do-not-wake list 31_918 to the wake list 31_916 or until a network connection is established between the push notification server 31_906 and the mobile device 31_ 100.

For example, a network connection may be established between the push notification server 31_906 and the mobile device 31_100 when another (high priority or low priority) push notification is delivered to the mobile device 31_100, or when the mobile device 31_100 sends other transmissions 31_924 (e.g., status messages, heartbeat messages, keep-alive messages, etc.) to the push notification server 31_ 906. For example, the mobile device 31_100 may send a message 31_924 to the push notification server 31_905 indicating that the mobile device 31_100 will be active for a period of time (e.g., 5 minutes), and the push notification server 31_906 may send all received push notifications to the mobile device 31_100 during the specified activity period. In some implementations, all push notifications stored in the push notification memory 31_922 will be delivered to the mobile device 31_100 when a network connection is established between the mobile device 31_100 and the push notification server 31_ 906. For example, the push notifications stored in the push notification data store 31_922 may be transmitted over a connection created by other transmissions between the mobile device 31_100 and the push notification server 31_ 906.

In some implementations, the mobile device 31_100 can establish two different communication channels with the push notification server 31_ 906. For example, two communication channels may be established simultaneously or at different times. The mobile device 31_100 may have a cellular data connection and/or a Wi-Fi connection with the push notification server 31_906, for example. In some implementations, the mobile device 31_100 can generate and send a different push filter 31_914 to the push notification server 31_906 for each communication channel. For example, the cellular data connection may be associated with a first set of push filters 31_914 for determining when to send high-priority and low-priority push notifications on the cellular data connection. The Wi-Fi data connection may be associated with a second set of push filters 31_914, which may be the same or different from the cellular data push filters, for determining when to send high-priority and low-priority push notifications on the Wi-Fi data connection. When the push notification server 31_906 receives the push notification, the push notification server may compare the application identified in the push notification with a push notification filter of the communication channel (e.g., Wi-Fi, cellular) that the push notification server 31_906 will use to send the push notification to the mobile device 31_ 100.

Push initiated background updates

In some implementations, receipt of the push notification by the mobile device 31_100 can trigger a background update of an application on the mobile device 31_ 100. For example, when the mobile device 31_100 (e.g., the push service daemon 31_904) receives the push notification message 31_920 from the push notification server 31_906, the push service daemon 31_904 can compare the application identifier in the push notification message 31_920 to the filter 31_928 stored on the mobile device 31_100 to determine whether the push notification message 31_920 was delivered correctly or should have been filtered (e.g., not delivered) by the push notification server 31_ 906. For example, the push filter 31_928, the wake up list 31_930, and the do not wake up list 31_932 may correspond to the push filter 31_914, the wake up list 31_916, and the do not wake up list 31_918, respectively. In some implementations, if the push service daemon 31_904 determines that the push notification message 31_920 should not have been delivered to the mobile device 31_100, the push notification message 31_902 will be deleted.

Low priority push notifications

In some implementations, the push notification message 31_920 received by the mobile device 31_100 may include a low priority push notification. For example, a low priority push notification may indicate that a content update is available for an application associated with the push notification. Thus, when a low priority push notification causes the application 31_908 to start, the application 31_908 may download updated content from one or more network resources (e.g., the push provider 31_ 902).

In some implementations, when the push service daemon 31_904 receives a low priority push notification associated with an application (e.g., application 31_908) on the mobile device 31_100, the push service daemon 31_904 can query whether the sampling daemon 31_102 can launch the application associated with the received low priority push notification. For example, the push service daemon 31_904 may request the sampling daemon 31_102 to perform admission control by sending an identifier (e.g., a "packet name" attribute value) of the application associated with the received low priority push notification to the sampling daemon 31_ 102. The sampling daemon 31_102 may perform admission control by examining the data budget, energy budget, attribute budget, and voter feedback, as described above with reference to fig. 31_ 4. The sampling daemon 31_102 may return such values to the push service daemon 31_904 based on the results of the admission control process: the value indicates whether the application identified by the low priority push notification can be launched.

In some implementations, if the value returned from the admission control request indicates "yes" -it is feasible to start the application, the push service daemon 31_904 will send a low priority push notification to the application manager 31_106, and the application manager 31_106 can invoke the application (e.g., application 31_ 908). The application 31_908 may then communicate with the push provider 31_902 over a network (e.g., the internet) to receive updated content from the push provider 31_ 902.

In some implementations, if the value returned from the admission control request indicates "no" -it is not feasible to start the application, the push service daemon 31_904 will store the low priority push notification in the push notification data store 31_ 934. For example, when storing a low priority push notification, the push service daemon 31_904 will only store the latest push notifications received for the application identified in the push notification. In some implementations, when sampling daemon 31_102 indicates that push service daemon 31_904 should not start an application now (e.g., admission control replies no), push service daemon 31_904 can move the application identifier for the application from wake list 31_930 to do-not-wake list 31_ 932. For example, if sampling daemon 31_102 determines that the budget and/or conditions of the mobile device do not allow the application to be launched, then allowing push notification server 31_906 to wake mobile device 31_100 for additional low priority push notifications associated with the presence of the application may simply consume further data and energy budget of mobile device 31_100, or degrade environmental conditions (e.g., cause the device to warm up). Thus, by moving the application identifier into the do-not-wake list 31_932 and sending a message 31_926 to the push notification server 31_906 that includes the update filter 31_928 (e.g., the wake up list 31_930 and the do-not-wake list 31_932), the notification server 31_906 can update its own push filter 31_914, wake up list 31_916, and do-not-wake list 31_918 to reflect the change to the push filter 31_928 and prevent additional low priority push notifications for the application from being delivered to the mobile device 31_ 100.

In some implementations, if the value returned from the admission control request indicates that it is "never" feasible to launch the application, the push service daemon 31_904 will delete the low priority push notification and delete the application identifier 31_928 associated with the push notification from the push filter. The updated push filter may be sent to the push notification server 31_906 and the push filter 31_914 on the push notification server 31_906 may be updated to prevent the push notification server 31_906 from sending any more push notifications associated with the application identifier.

In some implementations, the sampling daemon 31_102 can send a "stop" signal to the push service daemon 31_904 to temporarily prevent future low priority push notifications from being sent from the push notification server 31_906 to the mobile device 31_ 100. For example, the sampling daemon 31_102 may send a stop signal to the push service daemon 31_904 in the following case: the sampling daemon 31_102 determines that the data budget is exhausted at the current hour, the energy budget is exhausted at the current hour, the system is experiencing a thermal event (e.g., the mobile device 31_100 is overheating), the mobile device 31_100 has a poor cellular connection, and the mobile device 31_100 is not connected to Wi-Fi and/or the mobile device 31_100 is connected to a voice call and not connected to Wi-Fi. When the push service daemon 31_904 receives the stop signal, the push service daemon 31_904 may move the application identifiers in the wake up list 31_930 to the do-not-wake list 31_932 and send the updated push screener 31_928 to the push notification server 31_906 to update the push screener 31_ 914. Thus, the push notification server 31_906 will temporarily prevent future low priority push notifications from waking up the mobile device 31_100 and affecting the budget, limits, and operating conditions of the mobile device 31_ 100.

In some implementations, the sampling daemon 31_102 can send a retry signal to the push service daemon 31_ 904. For example, the sampling daemon 31_102 can monitor the status of budgets, network connections, restrictions, and device conditions, and will send a retry message to the push service daemon 31_904 if: when the energy budget is not exhausted, when the push data budget is not exhausted, when the mobile device 31_100 has not experienced a thermal event, when the mobile device 31_100 has a good quality cellular connection or is connected to Wi-Fi, when the mobile device 31_100 is not connected to a voice call, and when the boot rate limit has been reset. Once the push service daemon 31_904 receives the retry signal, the push service daemon 31_904 will send an admission control request to the sampling daemon 31_102 for each push notification in the push notification data store 31_934 to determine if each application (e.g., a "packet name" attribute value) associated with the stored push notification can be launched.

If the sampling daemon 31_102 returns a "yes" from the admission control request, the push service daemon 31_904 may send a push notification to the application manager 31_106, and the application manager 31_106 may launch the application associated with the push notification as a background process on the mobile device 31_100, as described above. Once the application is launched, the application may download content or data updates and update the application user interface based on the downloaded data. The application manager 31_106 does not ask the sampling daemon 31_102 whether the application associated with the low priority push notification can be started.

High priority push notifications

In some implementations, the push notification message 31_920 received by the mobile device 31_100 may include a high priority push notification. For example, a high priority push notification may indicate that a content update is available for an application associated with the push notification. Thus, when a high priority push notification results in invocation of an application, the application may download updated content from one or more network resources. In some implementations, when the push service daemon 31_904 receives a high priority push notification, the push service daemon 31_904 will send the high priority push notification to the application manager 31_106 without making an admission control request to the sampling daemon 31_ 102.

In some implementations, when the application manager 31_106 receives a push notification associated with an application, the application manager 31_106 will make an admission control request to the sampling daemon 31_ 102. In response to the admission control request, sampling daemon 31_102 may reply with a "yes", "no" or "never" answer as described above. When the application manager 31_106 receives a "yes" reply to the inbound control request, the application manager 31_106 may launch the application associated with the received high priority push notification as a background process on the mobile device 31_ 100.

In some implementations, when the application manager 31_106 receives a "no" reply to the inbound control request, the application manager 31_106 may store the high priority push notification in the high priority push notification store 31_ 936. When the application manager 31_106 receives the "never" response, the application manager 31_106 may delete the high priority push notification and delete any push notifications stored in the push notification data store 31_936 for the application associated with the push notification.

In some implementations, the sampling daemon 31_102 can send a "can retry" signal to the application manager 31_ 106. For example, when the application manager 31_106 receives a "retriable" message from the sample daemon 31_102, the application manager 31_106 may make an admission control request for the application associated with each high priority push notification in the high priority push notification data store 31_936 and start the respective application as a background process when a "yes" reply is received in response to the admission control request.

Delayed display push notification

In some implementations, the high priority push notification may cause a graphical user interface to be displayed on the mobile device 31_ 100. For example, receipt of a high priority push notification may cause a banner, balloon-like object, or other graphical object to be displayed on the graphical user interface of the mobile device 31_ 100. The graphical object may include, for example, information indicating the subject or content of the received push notification.

In some implementations, when the application manager 31_106 receives a high priority push notification, the application manager 31_106 can cause the notification to be displayed on a graphical user interface of the mobile device 31_ 100. However, when the high priority push notification indicates that there is a data update to be downloaded to the application associated with the high priority push notification, the application may be launched in the background of the mobile device 31_100 before the push notification is displayed. For example, the application manager 31_106 may be configured to have a certain amount of delay time (e.g., 30 seconds) between launching the application associated with the high priority push notification and displaying the graphical object (e.g., banner) that announces the push notification to the user. This delay may, for example, allow the application sufficient time to download the content update and update the user interface of the application before the user invokes it. Thus, when a user provides input to a graphical object or otherwise invokes an application associated with a high priority push notification, the user interface of the application will be up to date and the user will not be forced to wait for updates to the application. In some implementations, if the application manager 31_106 is unable to launch an application associated with a high priority push notification, the mobile device 31_100 will display a graphical object (e.g., a banner) to notify the user that a high priority push notification was received.

Exemplary push Notification Process

Fig. 31_10 is a flow diagram of an exemplary process 1000 for performing a non-wakeup push at the push notification server 31_ 906. At step 31_1002, the push notification server 31_906 may receive the push notification. For example, the push notification server 31_906 may receive push notifications from the push notification provider 31_902 (e.g., a server operated by an application provider).

At step 31_1004, the push notification server 31_906 may determine that the push notification is a low priority push notification. For example, the push notification provider may include data in the push notification that specifies a priority of the push notification. The push notification server 31_906 may analyze the content of the push notification to determine the priority of the push notification.

At step 31_1006, the push notification server 31_906 may compare the push notification with the push notification filter. For example, the push notification may identify an application installed or configured on the mobile device 31_100 to which the low priority push notification is directed. The push notification may include, for example, an application identifier (e.g., a "package name" attribute value). The push notification server 31_906 may compare the application identifier in the push notification with the application identifier in the do-not-wake list 31_918 of the push notification filter.

At step 31_1008, the push notification server 31_906 may determine that a low priority push notification should be stored. For example, if the application identifier from the low priority push notification is in the do-not-wake list 31_918 of the push notification filter, the push notification server 31_906 may determine that the low priority push should be stored in the push notification data store 31_ 922.

At step 31_1010, based on the determination at step 31_1008, the low priority push notification will be stored in the database or data store 31_922 of the push notification server 31_906 and will not be sent immediately to the mobile device 31_ 100.

At step 31_1012, the push notification server 31_906 may determine that a network connection has been established with the mobile device 31_ 100. For example, the push notification server 31_906 may create a network connection with the mobile device 31_100 to deliver another high-priority or low-priority push. The mobile device 31_100 may establish a network connection with the push notification server 31_906 to send notification filter changes, periodic status updates, keep-alive messages for the push notification server 31_906, or other messages.

At step 31_1014, the push notification server 31_906 may send the stored push notification in response to determining that a network connection has been established with the mobile device 31_ 100. For example, the push notification server 31_906 may send a low priority push notification stored at the push notification server 31_906 to the mobile device 31_ 100.

Fig. 31_11 is a flow diagram of an exemplary process 1100 for performing an application background update in response to a low priority push notification. At step 31_1102, the mobile device 31_100 may receive a low priority push notification from the push notification server 31_ 906.

At step 31_1104, the mobile device 31_100 may determine whether it is feasible to launch an application associated with the low priority push notification. For example, the application may be launched as a background process on the mobile device 31_ 100. The mobile device 31_100 may use the admission control procedure described above to determine whether it is feasible to start an application. For example, the mobile device 31_100 (e.g., sampling daemon 31_102) may determine whether launching an application is feasible based on data, energy, and/or attribute predictions determined for the mobile device 31_ 100. The mobile device 31_100 may determine whether it is feasible to launch an application based on the status of the mobile device and/or the status of the network connection of the mobile device based on responses from various tables. The details for determining whether launching an application (e.g., admission control) is feasible are described in more detail above with reference to fig. 31_ 4.

At step 31_1106, the mobile device 31_100 may store a low priority push notification when device conditions, budgets, restrictions, and other data indicate that it is not feasible to launch an application. For example, the mobile device 31_100 may store the low priority push notification in a database or other data store on the mobile device 31_ 100.

At step 31_1108, the mobile device 31_100 may update its push notification filter in response to determining that launching the background application is not feasible. For example, the mobile device 31_100 may move an application associated with a low priority push notification to the do-not-wake list of the push notification filter on the mobile device 31_ 100.

At step 31_1110, the mobile device 31_100 may send the updated notification filter to the push notification server 31_ 906. The push notification server 31_906 may update its own push notification filter based on the filter received from the mobile device 31_100 to determine when to send low priority push notifications to the mobile device 31_100 and when not to send low priority push notifications to the mobile device.

At step 31_1112, the mobile device 31_100 may determine that it is feasible to retry launching the application associated with the low priority push notification. For example, the mobile device 31_100 may determine that budgets, limitations, and device conditions as described above allow for the launching of additional background applications on the mobile device 31_ 100.

At step 31_1114, the mobile device 31_100 may determine whether it is feasible to launch a particular application associated with the stored low priority push notification. For example, the sampling daemon 31_102 of the mobile device 31_100 may perform admission control to determine that the budget configured on the mobile device 100 has been reset or supplemented for the current time and that the environmental conditions of the mobile device 31_100 and the network connection are good enough to start a specific background application.

At step 31_1116, the mobile device 31_100 may launch the particular application when the mobile device 31_100 determines that the launching of the application is feasible. For example, the particular application may be launched as a background process to download new content and update the application's user interface before the user invokes the application. This process would allow the user to invoke the application without having to wait for the content update to be downloaded and refresh the user interface of the application.

Fig. 31_12 is a flow diagram of an exemplary process 1200 for performing an application background update in response to a high priority push notification. At step 31_1202, the mobile device 31_100 may receive a high priority push notification.

At step 31_1204, the mobile device 31_100 may determine whether it is feasible to launch an application associated with the high priority push notification. For example, the sampling daemon 31_102 of the mobile device 31_100 may perform admission control to determine whether it is feasible to launch an application based on the budget of the mobile device 31_100 and environmental conditions (e.g., device conditions, network conditions, etc.).

At step 31_1206, when it is not feasible to launch an application associated with the high priority push notification (e.g., admission control returns no), the mobile device 31_100 may store the high priority push notification. For example, the mobile device 31_100 may store the high priority push notification in a database, queue, or other suitable data structure.

At step 31_1208, the mobile device 31_100 may determine that it is feasible to retry launching the application associated with the stored high priority push notification. For example, the mobile device 31_100 may determine that it is feasible to retry launching the application when data, energy and/or attribute pre-calculations have been supplemented, device conditions have improved, network conditions have improved, or other conditions of the mobile device 31_100 have changed, as discussed above in the admission control description.

At step 31_1210, the mobile device 31_100 may determine whether it is feasible to launch an application associated with the stored high priority push notification. For example, the mobile device 31_100 may determine whether launching an application is feasible based on the criteria described above.

At step 31_1212, the mobile device 31_100 may start the application in the background of the mobile device 31_ 100. For example, the application may be launched as a background process on the mobile device 31_100 so that the application may download updated content from a network resource (e.g., a content server) on a network (e.g., the internet).

At step 31_1214, the mobile device 31_100 may wait a period of time before presenting the push notification to the user. For example, the mobile device may be configured to allow the application to download the content for a period of time before notifying the user of the received high priority push notification.

At step 31_1216, the mobile device 31_100 may present the push notification on the user interface of the mobile device 31_ 100. For example, the mobile device 31_100 may present a graphical object (e.g., a banner) that includes information describing the high priority push notification. For example, the user may select the graphical object to invoke an application. Since the application has time to download the content before the notification is presented to the user, the application will be able to display the updated content to the user when the user invokes the application, without forcing the user to wait for the updated content to be downloaded from the network.

Background upload/download

Fig. 31_13 is a block diagram of an exemplary system 31_1300 for performing background downloads and/or data uploads on a mobile device 31_ 100. Background downloading and/or uploading may be network data transfers initiated by the application without explicit user input. For example, while the user is playing an electronic game application, a background download may be performed to retrieve the next gate of the electronic game. In contrast, a foreground download or upload may be a network data transfer performed in response to an explicit indication from the user that a download or upload should occur. For example, a web page link may be selected by a user to download a picture, movie, or document, thereby initiating a foreground download. Similarly, background uploads can be distinguished from foreground uploads based on whether an explicit user request to upload data to a network resource (e.g., a server) is received from a user.

In some implementations, the foreground download/upload (e.g., the download/upload explicitly requested by the user) is performed immediately for the user. For example, the download/upload requested by the user is performed immediately and is not constrained by budget constraints or other considerations. The foreground download/upload may be over a cellular data connection. In contrast, background downloads and/or uploads may be performed opportunistically, and within budget limits and taking into account environmental conditions (such as the temperature of the mobile device 31_ 100). For example, when an attribute is approved by the admission control mechanism described above, a background download or upload may be performed for that attribute or attribute value. In some implementations, background downloads and/or uploads may be limited to Wi-Fi network connections.

In some implementations, system 31_1300 may include a background transfer daemon 31_ 1302. In some implementations, the background transfer daemon 31_1302 can be configured to perform background downloading and uploading of data or content on behalf of applications or processes running on the mobile device 31_ 100. For example, the background transfer daemon 31_1302 may perform background downloads and/or uploads between the application 31_1304 and the server 31_1306 on behalf of the application 31_ 1304. Thus, background downloads/uploads may be performed apart from (e.g., not in/through) the process of requesting downloads/uploads from the application 31_ 1304.

In some implementations, the application 31_1304 can initiate background downloading/uploading by sending a request to the background transfer daemon 31_1304 to download or upload data. For example, a request to download data (e.g., content) may identify a network location where the data may be downloaded. The request to upload data may identify the network location where the data may be uploaded, and the location where the data is currently stored on the mobile device 31_ 100. The request may also identify the application 31_ 1304. Once the request has been made, the application 31_1304 can be shut down or paused so that it does not continue to consume computing and/or network resources on the mobile device 31_100 while the background download/upload is performed by the background transfer daemon 31_ 1304.

In some implementations, upon receiving a request to perform a background upload or download of data, the background transfer daemon 31_1302 may send a request to the sampling daemon 31_102 to determine whether the background transfer daemon 31_1302 can perform data transfer over a network. For example, the background transfer daemon 31_1302 may request that the sampling daemon 31_102 perform admission control on the data transfer. In the admission control request, the background transfer daemon 31_1302 may provide an identifier (e.g., a "packet name" attribute value) for the background transfer daemon 31_1302 or an identifier for the application requesting the background transfer, so that admission control may be performed on the background transfer daemon or the application. The admission control request may include an amount of data transmitted as a request cost to be deducted from the system level data budget.

In response to receiving an admission control request from the background transfer daemon 31_1302, the sampling daemon 31_102 can determine whether the system level data and/or energy budget for the current hour has been exhausted. In some implementations, if the sampling daemon 31_102 determines that the mobile device 31_100 is connected to an external power source, the sampling daemon 31_102 will not prevent background downloads/uploads based on the energy budget. The sampling daemon 31_102 may determine whether the mobile device 31_100 is connected to Wi-Fi. The sampling daemon 31_102 may also determine whether the mobile device 31_100 is in a thermal event (e.g., operating temperature is above a predetermined threshold). In some implementations, if the sampling daemon 31_102 determines that the data budget has been exhausted and the mobile device 31_100 is not connected to Wi-Fi, the energy budget has been exhausted and the mobile device 31_100 is not connected to an external power source, or the mobile device 31_100 is in a thermal event, the sampling daemon 31_102 will return a "no" reply to the admission control request through the background transfer daemon 31_ 1302.

In some implementations, when the background transfer daemon 31_1302 receives a "no" reply to the inbound control request from the sampling daemon 31_102, the process 31_1302 may store the background download/upload request from the application 31_1304 in the request repository 31_ 1308.

In some implementations, the sampling daemon 31_102 can send a retry signal to the background transfer daemon 31_ 1302. For example, sampling daemon 31_102 can send a retry signal to the background transfer daemon 31_1302 when the data and energy budgets are replenished and the system is no longer experiencing a thermal event. The sampling daemon 31_102 may send a retry signal to the background transfer daemon 31_1302 when the mobile device 31_100 is connected to Wi-Fi, connected to an external power source, and when the system is not experiencing a thermal event. For example, when connecting to Wi-Fi, there may be no need to control data usage. Similarly, there may be no need to conserve battery power when connected to an external power source. Thus, when performing admission control, the data and energy budgets may be ignored by the sampling daemon 31_ 102.

In some implementations, when the background transfer daemon 31_1302 receives the retry signal, the background transfer daemon 31_1302 can send an admission control request to the sampling daemon 31_ 102.

If the sampling daemon 31_102 returns an "ok" reply in response to the admission control request, the background transfer daemon 31_1302 may perform a background download or upload for the application 31_ 1304. Once the background download is complete, the background transfer daemon 31_1302 may wake up or invoke the application 31_1304 and provide the downloaded data to the application 31_ 1304.

In some implementations, the background transfer daemon 31_1302 can notify the sampling daemon 31_102 when the background download/upload begins and ends so that the sampling daemon 31_102 can adjust the budget and maintain statistics of the background download/upload performed on the mobile device 31_ 100. For example, the background transfer daemon 31_1302 may send a "background transfer" attribute start or stop event to the sampling daemon 31_ 102. In some implementations, the background transfer daemon 31_1302 can transfer the number of bytes transferred over cellular data, over Wi-Fi, and/or over both together (e.g., a "system. network bytes" attribute event) so that the sampling daemon 31_102 can adjust the budget and maintain statistics of background downloads/uploads performed on the mobile device 31_ 100.

In some implementations, the sampling daemon 31_102 can return a timeout value to the background transmission daemon 31_1302 in response to the admission control request. For example, the timeout value may indicate a period of time (e.g., 5 minutes) that the background transfer daemon must perform a background download or upload. After the timeout period has elapsed, background transfer daemon 31_1302 will suspend background downloading or uploading.

In some implementations, the timeout value may be based on the remaining energy budget for the current hour. For example, the sampling daemon 31_102 can determine how much energy is consumed per second when a download or upload is performed over Wi-Fi based on historical event data collected by the sampling daemon 31_ 102. The sampling daemon 31_102 may determine the timeout period by dividing the remaining energy budget by the energy consumption rate when performing the background download or upload (e.g., energy budget/energy consumption/time — timeout period).

In some implementations, the background downloads and/or uploads are recoverable. For example, if the mobile device 31_100 moves out of Wi-Fi range, background downloads/uploads can be paused (e.g., paused). When the mobile device 31_100 re-enters Wi-Fi range, the suspended download/upload may be resumed. Similarly, if the background download/upload energy budget is exhausted (e.g., a timeout period has elapsed), the background download/upload may be suspended. The suspended downloading/uploading may be resumed when the extra budget is allocated (e.g., within the next hour).

In some implementations, the background download/upload may be suspended based on the quality of the network connection. For example, even though the mobile device 31_100 may have a good cellular data connection between the mobile device 31_100 and a serving cellular tower, and may have a good data connection between the cellular tower and a server to/from which the mobile device 31_100 is transmitting data, the mobile device 31_100 may not have a good connection with the server. For example, the transmission rate between the mobile device 31_100 and the server may be slow, or the throughput of the cellular interface may be low. If the transfer rate of the background download/upload is below the threshold transfer rate value and/or the throughput of the background download/upload is below the threshold throughput value, the background download/upload (e.g., data transfer) may be suspended or quiesced based on the detected poor quality network connection until there is a better network connection available. For example, if a Wi-Fi connection is available, suspended background downloads/uploads may be resumed over the Wi-Fi connection.

In some implementations, the background transfer daemon 31_1302 may be configured to have a limit on the number of background downloads and/or uploads that can be performed at the same time. For example, the background transfer daemon 31_1302 may limit the number of concurrent background downloads and/or uploads to three.

Exemplary background download/upload Process

Fig. 31_14 is a flow diagram of an exemplary process 31_1400 for performing background downloads and uploads. For example, background downloads and/or uploads may be performed by the background transfer daemon 31_1302 on behalf of applications on the mobile device 31_ 100.

At step 31_1402, a background transfer request may be received. For example, the background transfer daemon 31_1302 may receive a background download/upload request from an application running on the mobile device 31_ 100. Once the application issues the request, the application may be terminated or paused, for example. The request may identify the application and identify a source of the data and/or a target location for the data. For example, in downloading data, the source location may be a network address of the server and the destination location may be a directory in the file system of the mobile device 31_ 100. In uploading data, the source location may be a file system location and the target may be a network location.

At step 31_1404, the mobile device 31_100 can determine that the budget and device conditions do not allow data transfer. For example, the background transfer daemon 31_1302 may query whether the sampling daemon 31_102 can perform the requested background transfer by making an admission control request to the sampling daemon 31_1302 that identifies the background transfer daemon 31_1302, the application that is performing the background transfer, and/or the amount of data to be transferred. The sampling daemon 31_102 can determine if the energy and data budgets are exhausted and if the mobile device 31_100 is in a thermal event. If the budget has been exhausted, or if the mobile device 31_100 is in the middle of a thermal event, the sampling daemon 31_102 may send a message to the background transfer daemon 31_1302 indicating that background data transfer may not be performed (e.g., admission control returns "no").

At step 31_1406, the mobile device 31_100 may store the background transfer request. For example, when the sampling daemon 31_102 returns a "no" value in response to an admission control request, the background transfer daemon 31_1302 may store the transfer request in a transfer request repository.

At step 31_1408, the mobile device 31_100 may determine that the background transfer may be retried. For example, the sampling daemon 31_102 may determine that the data and energy budget has been replenished and that the mobile device 31_100 is not in the middle of a thermal event. The sampling daemon 31_102 may send a retry message to the background transfer daemon 31_ 1302. The background transfer daemon 31_1302 may then attempt to perform the requested transfer stored in the transfer request repository by making another admission control request for each transfer request stored.

At step 31_1410, the mobile device 31_100 may determine that the budget and status of the mobile device 31_100 allows background data transfer. For example, the background transfer daemon 31_1302 may ask the sampling daemon 31_102 whether the requested background transfer can be performed. The sampling daemon 31_102 may perform admission control to determine that the energy and data budgets are replenished and that the mobile device 31_100 is not in the middle of a thermal event. If the budget is not exhausted, and if the mobile device 31_100 is not in the middle of a thermal event, the sampling daemon 31_102 can send a message to the background transfer daemon 31_1302 indicating that a background data transfer can be performed.

At step 31_1412, the mobile device 31_100 may perform a background transfer. For example, the background transfer daemon 31_1302 may perform the requested background download or background upload for the requesting application. The background transfer daemon 31_1302 may notify the sampling daemon 31_102 when the background transfer starts and ends (e.g., start and stop events using the "background transfer" property). The backend transfer daemon 31_1302 may send such a message: this message informs the sampling daemon of the number of bytes transferred during a background download or upload (e.g., using a "network bytes" attribute event). Once the background transfer is complete, the background transfer daemon 31_1302 may invoke (e.g., launch or wake up) the application making the background transfer request and send completion status information (e.g., success, error, downloaded data, etc.) to the requesting application.

Enabling/disabling background updates

Fig. 31_15 illustrates an exemplary Graphical User Interface (GUI)31_1500 for enabling and/or disabling application background updates on a mobile device. For example, the GUI 31_1500 may be an interface presented on a display of the mobile device 31_100 for receiving user input to adjust background update settings of an application on the mobile device 31_ 100.

In some implementations, as described above, GUI 31_1500 may enable or disable background updates performed on the application based on a user invocation forecast. For example, sampling daemon 31_102 and/or application manager 31_106 can determine whether to enable or disable background updates for an application and prevent the application from being launched by application manager 31_106 or from being included in the application call forecast generated by sampling daemon 31_ 102. For example, if background updates are disabled for an application, sampling daemon 31_102 will not include the application in the user-invoked application declaration requested by application manager 31_ 106. Thus, when background updates are disabled, the application manager 31_106 will not launch the application. Conversely, if background updates are enabled for an application, that application may be included in the application call prediction generated by the sampling daemon 31_102 based on the user call probabilities, as described above.

In some implementations, as described above, GUI 31_1500 may enable or disable background updates performed on the application when a push notification is received. For example, the sampling daemon 31_102, application manager 31_106, and/or push service daemon 31_904 may determine whether to enable or disable background updates for an application and prevent the application from being launched by the application manager 31_106 in response to receiving a push notification. For example, if a background update is disabled for an application and a push notification for the application is received, the application manager 31_106 will not initiate the application download update in response to the push notification.

In some implementations, the GUI 31_1500 may display applications 31_1502- & 1514 that have been configured to perform background updates. For example, the application 31_1502- & 1514 may be configured or programmed to run as a background process on the mobile device 31_100 when launched by the application manager 31_ 106. When the applications 31_1502- & 1514 run as background processes, they can communicate with various network resources to download current or updated content. The applications 31_1502- & 1514 may then update their respective user interfaces to present the updated content when invoked by the user of the mobile device 31_ 100. In some implementations, applications that are not configured or programmed to perform background updates will not be displayed on GUI 31_ 1500.

In some implementations, the user can provide input to GUI 31_1500 to enable and/or disable background updates of the application. For example, the user may provide an input (e.g., a touch input) to the mobile device 31_100 using the toggle key 31_1516 to open or close a background update of the application 31_ 1502. The user may provide an input (e.g., a touch input) to the mobile device 31_100 using the toggle key 31_1518 to open or close a background update of the application 31_ 1508.

In some implementations, additional options can be specified for the background update application through the GUI 31_ 1500. For example, the user may select the graphical object 31_1510 associated with application 31_1514 to invoke a graphical user interface (not shown) for specifying additional background update options. Background update options may include, for example, a start time and an end time for opening and/or closing the application 31_1514 background updates.

Sharing data between peers

Fig. 31_16 illustrates an exemplary system for sharing data between peer devices. In some implementations, the mobile device 31_100 can share event data, system data, and/or event forecasts with the mobile device 31_ 1600. For example, the mobile device 31_100 and the mobile device 31_1600 may be devices owned by the same user. Thus, it may be advantageous to share information about user activities on each of the mobile device 31_100 and the mobile device 31_ 1600.

In some implementations, the mobile device 31_1600 can be configured similarly to the mobile device 31_100 described above. For example, the mobile device 31_1600 may be configured with a sampling daemon 31_1602 that provides the functionality (e.g., attributes, attribute events, forecasts, admission control, etc.) described in the preceding paragraphs.

In some implementations, the mobile device 31_100 and the mobile device 31_1600 can be configured with an identity service daemon 31_1620 and an identity service daemon 31_1610, respectively. For example, the identity service daemons 31_1620 and 31_1610 may be configured to communicate information between the mobile device 31_100 and the mobile device 31_ 1600. The identity service daemon may be used to share data between devices owned by the same user through various peer-to-peer and network connections. For example, identity service daemon 31_1620 and identity service daemon 31_1610 may exchange information over bluetooth, bluetooth low energy, Wi-Fi, LAN, WAN, and/or internet connections.

In some implementations, sampling daemon 31_1602 (and sampling daemon 31_102) can be configured to share event forecasts and system state information with other sampling daemons running on other devices owned by the same user. For example, if the mobile device 31_100 and the mobile device 31_1600 are owned by the same user, the sampling daemon 31_102 and the sampling daemon 31_1602 may exchange event forecast information and/or system state information (e.g., battery state). For example, sampling daemon 31_1602 can use identity service daemon 31_1610 to send event forecast information and/or system state information.

The identity service daemon 31_1610 can establish a connection with the identity service daemon 31_1620 and transmit event forecast information and/or mobile device 31_1600 system state information to the sampling daemon 31_102 through the identity service daemon 31_ 1620.

In some implementations, the application 31_1608 (e.g., a client of the sampling daemon 31_ 1602) can request that the sampling daemon 31_1602 send an event forecast for a specified attribute or attribute value to the sampling daemon 31_ 102. For example, the application 31_1608 may be an application synchronized with the application 31_108 of the mobile device 31_ 100. For example, the applications 31_108 and 31_1608 may be media applications (e.g., music libraries, video libraries, email applications, messaging applications, etc.) configured to synchronize data (e.g., media files, messages, status information, etc.) between the mobile device 31_100 and the mobile device 31_ 1600.

In some implementations, to allow a peer device (e.g., mobile device 31_100) to determine when to synchronize data between devices, application 31_1608 may request sampling daemon 31_1602 to generate a temporal and/or peer to peer forecast for a "package name" attribute or a particular "package name" attribute value (e.g., an application identifier of application 31_ 1608) based on attribute event data generated by mobile device 31_1600, and transmit the forecast to sampling daemon 31_ 102. For example, a peer device may be a remote device owned by the same user (e.g., not the current local device). The mobile device 31_100 may be a peer device such as the mobile device 31_ 1600.

In some implementations, the requesting client (e.g., application 31_1608) can specify a schedule for delivery and a duration for forecasting data. For example, the application 31_1608 may request a peer-to-peer and/or time forecast for the "package name" attribute value "mail application". The application 31_1608 may request that the forecasts be generated and exchanged, for example, weekly, and that each forecast cover a duration or period of a week.

In some implementations, data exchange between peer devices may be statically arranged. The sampling daemon 31_1602 can send the attribute data necessary for the mobile device 31_100 to make the remote status of the mobile device 31_1600 consistent under a strict schedule (e.g., application forecasting and battery statistics every 24 hours). In some implementations, the client may request attribute forecasts or statistics from the peer devices on demand. These exchanges are non-reproducible. The requesting client may be notified when the requested data is received.

In some implementations, the sampling daemon 31_1602 can transmit system state data for the mobile device 31_1600 to the sampling daemon 31_ 102. For example, sampling daemon 31_1602 may receive battery charge level events (e.g., "battery level" attribute events), battery charge events (e.g., "cable plug in" events), energy usage events (e.g., "energy" attribute events), and/or other events that may be used to generate battery usage and charging statistics, and transmit battery related event data to sampling daemon 31_ 102. For example, battery status information may be exchanged every 24 hours. Battery status information may be exchanged opportunistically. For example, when a communication channel is established (e.g., peer-to-peer, networked, etc.) between the mobile device 31_100 and the mobile device 31_1600, the mobile device can opportunistically use the already open communication channel to exchange battery status or other system state information (e.g., identification of the current foreground application).

As another example, sampling daemon 31_1602 may receive a thermal level event (e.g., a "thermal level" attribute event), a network event (e.g., a "network quality" attribute event, a "network byte" attribute event), and transmit the thermal and/or network event to sampling daemon 31_ 102. Sampling daemon 31_1602 can receive an event (e.g., a "system. foreground application" property event) from application manager 31_106 indicating an application (e.g., application identifier) that is currently in the foreground of mobile device 31_1600 and send foreground application information to sampling daemon 102. In some implementations, thermal events and foreground application change information may be exchanged with peers once the event occurs (e.g., once a connection is established between the peers). In some implementations, the network status information may be exchanged periodically (e.g., once per day, twice per day, once per hour, etc.).

Upon receiving the forecast and/or system event data from sampling daemon 31_1602, sampling daemon 31_102 may store the forecast and/or event data in peer data store 31_ 1622. Similarly, any forecast and/or event data received by sampling daemon 31_1602 from sampling daemon 31_102 may be stored in peer data store 31_ 1612. In some implementations, the forecast and/or event data received from another device may be associated with a device description. For example, the device description may include a device name, a device identifier, and a model identifier that identifies a model of the device. The device description may be used to look up forecast data and/or event data for the device in peer data store 31_ 1622. Once the mobile device 31_100 and the mobile device 31_1600 have exchanged forecast and/or event data, the two mobile devices can use the exchanged information to determine when to communicate with each other using the remote admission control mechanism below. By allowing devices to share information only when the information is needed and when the battery state of the devices can support sharing the information, power management of the communication can be improved.

Remote admission control

In some implementations, the mobile device 31_100 (or the mobile device 31_1600) can perform admission control based on data received from another device. For example, sampling daemon 31_102 can perform admission control based on forecast and system event data received from sampling daemon 31_1602 and stored in peer data store 31_ 1622. For example, to synchronize data with the application 31_1608, the application 31_108 can send a synchronization message to the identity service daemon 31_ 1620. For example, the synchronization message may include an identifier of the mobile device 31_100, an identifier of the mobile device 31_1600, a priority identifier (e.g., high, low), and a message payload (e.g., data to synchronize).

Low priority messages

In some implementations, low priority messages may be transmitted after admission control has passed. For example, the low priority message may be a message associated with ad hoc processing (e.g., background application, system utility, expected activity, non-user initiated activity). For example, the identity service daemon 31_1620 may send an admission control request to the sampling daemon 31_102 for a "packet name" attribute value as an application 31_1608 packet name identifier (e.g., "packet name" ═ 1608 "). In addition to the "package name" attribute name and value (e.g., "1608"), the identity service daemon 31_1620 may provide a device name (e.g., "device 31_ 1600") in the admission control request to indicate that the application 31_108 is requesting admission control for communication with another device.

In some implementations, in response to receiving an admission control request, the sampling daemon 31_102 may perform local admission control and remote admission control. For example, the sampling daemon 31_102 may perform local admission control as described above to determine whether the mobile device 31_100 is in a condition that allows an event associated with a specified attribute value (e.g., "packet name" ═ 1608 ") to occur. The sampling daemon 31_102 may, for example, examine the local energy, data, and attribute budgets and query the voter feedback to determine if the mobile device 31_100 is in a condition that allows events associated with a specified attribute value (e.g., "package name" 1608 ").

In addition to performing local admission control, the sampling daemon 31_102 may perform remote admission control based on "package name" attribute predictions, event data, and system data received from the mobile device 31_1600 and stored in the peer data store 31_ 1622. For example, the sampling daemon 31_102 can use a device identifier (e.g., "device 31_1600," device name, unique identifier, UUID, etc.) to locate data in the peer data store 31_1622 associated with the mobile device 31_ 1600. The sampling daemon 31_102 may analyze the attribute (e.g., "package name") forecast data received from the sampling daemon 31_1602 to determine whether the application 31_1608 may be invoked by the user on the mobile device 31_1600 in the current 15 minute time slot. If the application 31_1608 is unlikely to be invoked by the user in the current 15 minute time slot, the sampling daemon 31_102 may return a "no" value in response to the admission control request. For example, by allowing the application 31_108 to synchronize with the application 31_1608 only when the application 31_1608 is likely to be used on the mobile device 31_1600, the sampling daemon 31_102 can delay the synchronization process and save system resources (e.g., battery, CPU cycles, network data) until the user is likely to use the application 31_1608 on the mobile device 31_ 1600.

In some implementations, if the application 31_1608 is likely to be invoked by the user of the mobile device 31_1600 in the current 15 minute time slot, the sampling daemon 31_102 may examine the system data associated with the mobile device 31_1600 and stored in the peer data store 31_ 1622. For example, the sampling daemon 31_102 may examine system data associated with the mobile device 31_1600 to determine if the mobile device 31_1600 has sufficient battery power remaining to perform synchronization between the application 31_108 and the application 31_ 1608. For example, the sampling daemon 31_102 may check whether there is currently enough battery power to complete the synchronization between the application 31_108 and the application 31_ 1608. The sampling daemon 31_102 may check if there is enough battery charge to perform synchronization and continue running until the next predicted battery charge (e.g., a "cable plug in" attribute event). For example, sampling daemon 31_102 may generate a temporal forecast for the "cable insertion" attribute that identifies when the next "cable insertion" attribute event is likely to occur. The sampling daemon 31_102 can analyze the energy usage statistics (events) to predict the energy usage until the next "cable plug in" event and determine if there is enough energy left to serve the synchronous transfer between application 31_108 and application 31_ 1608. If the sampling daemon 31_102 determines that the mobile device 31_1600 does not have enough energy (e.g., battery power) to service synchronization, the sampling daemon 31_102 may return a "no" value in response to the remote admission control request.

In some implementations, the sampling daemon 31_102 can examine system data associated with the mobile device 31_1600 to determine if the mobile device 31_1600 is in a normal thermal state (e.g., not overheating), and can process synchronization requests. For example, if the "hot level" attribute event data received from the mobile device 31_1600 indicates that the mobile device 31_1600 is currently operating at a temperature above the threshold, the sampling daemon 31_102 may prevent synchronous communications by returning a "no" value in response to a remote admission control request.

In some implementations, the sampling daemon 31_102 may return a "yes" value to the identity service daemon 31_1620 in response to the admission control request when the forecast data indicates that the user is likely to invoke the application 31_1608 on the mobile device 31_1600 and the energy, thermal, and other system state information indicates that the mobile device 31_1600 is in a condition to process communications from the mobile device 31_ 100. In response to receiving a "yes" value in response to the admission control request, the identity service daemon 31_1620 may transmit a synchronization message for the application 31_108 to the identity service daemon 31_1610 on the mobile device 31_ 1600. Application 31_108 and application 31_1608 can then synchronize data by exchanging messages through identity service daemon 31_1620 and identity service daemon 31_ 1610.

In some implementations, high priority messages may be transmitted after undergoing remote admission control. For example, the high priority message may be a message associated with a user-initiated task, such as a message associated with a foreground application or a message generated in response to a user providing input. In some implementations, admission control for high priority messages may be handled similarly to low priority messages. However, when performing remote admission control on high priority messages, the high priority messages may be admitted (allowed) without taking into account attribute forecast data (e.g., "package name" forecast data), since high priority messages are typically triggered by some user action, rather than being initiated by some arbitrary background task.

In some implementations, when performing admission control for high priority messages, the battery status of a remote device (e.g., mobile device 31_1600) can be checked to ensure that the remote device (e.g., a peer device) has sufficient available battery power to handle the high priority information. If there is sufficient battery power on the remote device, the remote admission control will approve the high priority message. For example, when there is sufficient battery power left to process a high priority message, the sampling daemon 31_102 may transmit a "yes" value to the identity service daemon 31_1620 in response to the remote admission control request. If there is not enough battery power on the remote device, the remote admission control will reject high priority messages. For example, when there is not enough battery charge remaining to handle high priority messages, the sampling daemon 31_102 may transmit a "no" value to the identity service daemon 31_1620 in response to the remote admission control request. Thus, when the peer device has sufficient battery power remaining to process the message in question, the identity service daemon 31_1620 will initiate communication with the peer device (e.g., mobile device 31_ 1600).

In some implementations, when sampling daemon 31_102 is notified of a high priority message, sampling daemon 31_102 can send current battery state information (e.g., current charge level) to identity service daemon 31_ 1620. The identity service daemon 31_1620 may add battery status information to the high priority message. Accordingly, system state information may be efficiently shared between devices by piggybacking battery state information (or other information, e.g., hot level, foreground applications, etc.) on other messages transmitted between the mobile device 31_100 and the mobile device 31_ 1600.

In some implementations, sampling daemon 31_102 can send a retry message to identity service daemon 31_ 1620. For example, when the condition on the mobile device 31_100 or mobile device 31_1600 changes (e.g., battery condition improves), the sampling daemon 31_102 can send a retry message to the identity service daemon 31_ 1620. In some implementations, a retry message can be generated when the remote focus application changes. For example, if a user on a remote peer device is using a "mail application" application, the "mail application" application will become the focused on application. When the user begins to use the "web browser" application, the in focus application will change to the "web browser" application. When peer-to-peer data is exchanged between mobile device 31_100 and mobile device 31_1600, the change to the in focus application may be reported as an event to sampling daemon 31_1602 and transmitted to sampling daemon 31_ 102. Upon receiving event information indicating that a change has occurred to the in focus application at peer device 31_1602, sampling daemon 31_102 may send a retry message to identity service daemon 31_ 1620. The identity service daemon 31_1620 may then retry admission control for each message rejected by the sampling daemon 31_ 102. For example, the identity service daemon 31_1620 may store the rejected message (e.g., the transfer task) and upon receiving the retry message from the sampling daemon 31_102, send the rejected message with admission control. In some implementations, the rejected message may be transmitted after a period of time has elapsed. For example, after a configurable period of time has elapsed, a message that has not passed admission control may be sent to the peer device.

In some implementations, the identity service daemon 31_1620 may interrupt data streaming when the sampling daemon 31_102 indicates that the condition on the mobile device 31_100 or the mobile device 31_1600 has changed. For example, if the sampling daemon 31_102 determines that the battery condition on the mobile device 31_100 or mobile device 31_1600 has changed such that one of the mobile devices may run out of battery power, the sampling daemon 31_102 may inform the identity service daemon 31_1620 to stop transmitting and retry admission control for attribute events associated with data flows.

Process for sharing data between peer devices

Fig. 31_17 illustrates an exemplary process 31_1700 for sharing data between peer devices. Additional details regarding the process 31_1700 can be found in FIG. 31_16, above. At step 31_1702, the mobile device may receive event data from a peer device. For example, event data may be shared as "summary" (e.g., forecast, statistical, etc.) or raw (e.g., unprocessed) event data. For example, when the second device and the mobile device are owned by the same user, the second device (e.g., mobile device 31_1600) is a peer device of mobile device 31_ 100. The mobile device 31_100 may receive event data related to a system state (e.g., battery state, network state, foreground application identifier, etc.) of the mobile device 31_ 1600. The mobile device may receive attribute event forecasts, statistics, or raw event data from the mobile device 31_1600 based on events occurring on the mobile device 31_ 1600. For example, the application 31_1608 on the peer device 31_1600 can instruct the sampling daemon 31_1602 on the peer device 31_1600 to generate and send a forecast for a particular attribute or attribute value to the mobile device 31_ 100.

At step 31_1704, the identity service daemon 31_1620 on the mobile device 31_100 may receive a message to send to the peer device 31_ 1600. For example, an application 31_108 running on a mobile device may need to share, exchange, or synchronize data with a corresponding application 31_1608 on the peer device 31_ 1600. The application 31_108 can send a message to the identity service daemon 31_1620 containing the data to be shared.

At step 31_1706, the sampling daemon 31_102 on the mobile device 100 may determine whether to transmit a message based on the data received from the peer device 31_ 1600. For example, the sampling daemon 31_102 may perform local and remote admission control checks to determine if a message should be sent to the peer device 31_1600 at the current time. The sampling daemon 31_102 may approve transmission of the message if the property event forecast received from the peer device 31_1600 indicates that the user of the peer device 31_1600 is likely to invoke the application 31_1608 at the current time, and if the event data indicates that conditions (e.g., battery status, hot water level, etc.) in the peer device 31_1600 that initiated communication with the peer device 31_1600 do not drain the battery or degrade the thermal status.

At step 31_1708, once sampling daemon 31_102 performs admission control and approves initiation of communication with peer device 31_1600, identity service daemon 31_1620 may transmit the message to peer device 31_ 1600. For example, the identity service daemon 31_1620 may transmit the message to the identity service daemon 31_1610 of the peer device 31_ 1600. The identity service daemon 31_1610 may then transmit a message to the application 31_1608 so that the application 31_108 and the application 31_1608 can synchronize data.

The memory (e.g., memory of device 100 in fig. 1A) may also store other software instructions to facilitate the processes and functions described in section 1, such as the dynamic adjustment processes and functions described with reference to fig. 31-1-31 _ 17.

Exemplary methods, systems, and computer-readable media for dynamically adjusting a mobile device

The memory (e.g., memory of device 100 in fig. 1A) may also store other software instructions to facilitate the processes and functions described in section 1, such as the dynamic adjustment processes and functions described with reference to fig. 31-1-31 _ 17.

In one aspect, a mobile device may be configured to monitor environmental, system, and user events associated with the mobile device and/or a peer device. The occurrence of one or more events may trigger an adjustment to a system setting. The mobile device may be configured to keep frequently invoked applications up to date based on forecasts made for predicted user invocations. In some implementations, the mobile device can receive push notifications associated with such applications: which indicates that there is new content available for the application to download. The mobile device may launch an application associated with the push notification in the background and download the new content. In some implementations, prior to running an application or communicating with a peer device, the mobile device can be configured to check the energy and data budget and environmental conditions of the mobile device and/or the peer device to ensure a high quality user experience.

In some implementations, a method is provided. The method comprises the following steps: receiving, at a mobile device, attribute event data from a peer device, wherein the attribute event data describes an event occurring at the peer device; storing peer event data at a mobile device; receiving a request from an application on a mobile device to communicate with a peer device, wherein the request includes an attribute having a value corresponding to a corresponding application identifier on the peer device; determining, by the mobile device, to initiate communication with the peer device based on the peer event data.

In some implementations, the peer device and the mobile device are owned by a single user. In some implementations, determining, by the mobile device, to initiate communication with the peer device based on the peer event data includes generating one or more forecasts for the attributes based on the peer event data. In some implementations, determining, by the mobile device, to initiate communication with the peer device based on the peer event data includes determining a battery status of the peer device based on the peer event data. In some implementations, determining, by the mobile device, to initiate communication with the peer device based on the peer event data includes determining a thermal status of the peer device based on the peer event data. In some implementations, determining, by the mobile device, to initiate communication with the peer device based on the peer event data includes determining that the user is likely to invoke a corresponding application on the peer device at about the current time.

In some implementations, a non-transitory computer-readable storage medium is provided that includes one or more sequences of instructions which, when executed by one or more processors, cause: receiving, at a mobile device, attributed event data from a peer device, wherein the attributed event data describes an event occurring at the peer device; storing peer event data at a mobile device; receiving a request from an application on a mobile device to communicate with a peer device, wherein the request includes an attribute having a value corresponding to a corresponding application identifier on the peer device; determining, by the mobile device, to initiate communication with the peer device based on the peer event data.

In some implementations, the peer device and the mobile device are owned by a single user. In some implementations, the instructions that cause determining, by the mobile device, to initiate communication with the peer device based on the peer event data include instructions that cause the device to generate one or more forecasts for the attributes based on the peer event data. In some implementations, the instructions that cause the mobile device to determine to initiate communication with the peer device based on the peer event data include instructions that cause the device to determine a battery status of the peer device based on the peer event data. In some implementations, the instructions that cause determining, by the mobile device, to initiate communication with the peer device based on the peer event data include instructions that cause the device to determine a thermal status of the peer device based on the peer event data. In some implementations, the instructions that cause the mobile device to determine to initiate communication with the peer device based on the peer event data include instructions that cause the device to determine that the user is likely to invoke a corresponding application on the peer device at about the current time.

In some implementations, a system is provided that includes one or more processors; and a non-transitory computer-readable medium comprising one or more sequences of instructions which, when executed by the one or more processors, cause: receiving, at a mobile device, attributed event data from a peer device, wherein the attributed event data describes an event occurring at the peer device; storing peer event data at a mobile device; receiving a request from an application on a mobile device to communicate with a peer device, wherein the request includes an attribute having a value corresponding to a corresponding application identifier on the peer device; determining, by the mobile device, to initiate communication with the peer device based on the peer event data.

In some implementations, the peer device and the mobile device are owned by a single user. In some implementations, the instructions that cause determining, by the mobile device, to initiate communication with the peer device based on the peer event data include instructions that cause the device to generate one or more forecasts for the attributes based on the peer event data. In some implementations, the instructions that cause the mobile device to determine to initiate communication with the peer device based on the peer event data include instructions that cause the device to determine a battery status of the peer device based on the peer event data. In some implementations, the instructions that cause determining, by the mobile device, to initiate communication with the peer device based on the peer event data include instructions that cause the device to determine a thermal status of the peer device based on the peer event data. In some implementations, the instructions that cause the mobile device to determine to initiate communication with the peer device based on the peer event data include instructions that cause the device to determine that the user is likely to invoke a corresponding application on the peer device at about the current time.

In another aspect, a mobile device may be configured to monitor environmental, system, and user events. The occurrence of one or more events may trigger an adjustment to a system setting. In some implementations, the mobile device can be configured to keep frequently invoked applications up to date based on forecasts made for predicted user invocations. In some implementations, the mobile device can receive push notifications associated with such applications: which indicates that there is new content available for the application to download. The mobile device can launch the application associated with the push notification in the background and download the new content. In some implementations, prior to running an application or accessing a network interface, the mobile device can be configured to check the energy and data budget and environmental conditions of the mobile device to maintain a high quality user experience.

In some implementations, there is provided a method comprising: receiving event data at a first process running on a mobile device; receiving event registration data from a second process running on the mobile device, the event registration data identifying one or more events for triggering invocation of the second process, wherein the second process pauses or terminates after receiving the event registration data; determining, by a first process, that the one or more events have occurred based on event data; and invokes a second process on the mobile device.

In some implementations, invoking the second process causes the second process to adjust one or more components of the mobile device. In some implementations, the one or more components include a central processing unit, a graphics processing unit, a baseband processor, or a display of the mobile device. In some implementations, the one or more events include a change in an operating temperature of the mobile device, a change in a system setting, a user input, turning on or off a display, setting a clock alarm, or setting a calendar event. In some implementations, the method further comprises: receiving, at a first process, a request from a second process for storing event data by the second process; the requested event data is sent from the first process to a second process, wherein the second process is configured to adjust one or more components of the mobile device based on the event data. In some implementations, the one or more events include a pattern of events, and wherein the first process is configured to, when the pattern of events is detected, identify a pattern in the received event data and invoke the second process.

In some implementations, a non-transitory computer-readable medium is provided that includes one or more sequences of instructions which, when executed by one or more processors, cause: receiving event data at a first process running on a mobile device; receiving event registration data from a second process running on the mobile device, the event registration data identifying one or more events for triggering invocation of the second process, wherein the second process is paused or terminated after the event registration data is received; determining, by a first process, that the one or more events have occurred based on event data; and invokes a second process on the mobile device.

In some implementations, invoking the second process causes the second process to adjust one or more components of the mobile device. In some implementations, the one or more components include a central processing unit, a graphics processing unit, a baseband processor, or a display of the mobile device. In some implementations, the one or more events include a change in an operating temperature of the mobile device, a change in a system setting, a user input, turning on or off a display, setting a clock alarm, or setting a calendar event. In some implementations, the instructions cause: receiving, at a first process, a request from a second process for storing event data by the second process; the requested event data is sent from the first process to a second process, wherein the second process is configured to adjust one or more components of the mobile device based on the event data. In some implementations, the one or more events include a pattern of events, and wherein the first process is configured to, when the pattern of events is detected, identify a pattern in the received event data and invoke the second process.

In some implementations, a system is provided that includes one or more processors; and a non-transitory computer-readable medium comprising one or more sequences of instructions which, when executed by one or more processors, cause: receiving event data at a first process running on a mobile device; receiving event registration data from a second process running on the mobile device, the event registration data identifying one or more events for triggering invocation of the second process, wherein the second process pauses or terminates after receiving the event registration data; determining, by a first process, that the one or more events have occurred based on event data; and invokes a second process on the mobile device.

In some implementations, invoking the second process causes the second process to adjust one or more components of the mobile device. In some implementations, the one or more components include a central processing unit, a graphics processing unit, a baseband processor, or a display of the mobile device. In some implementations, the one or more events include a change in an operating temperature of the mobile device, a change in a system setting, a user input, turning on or off a display, setting a clock alarm, or setting a calendar event. In some implementations, the instructions cause: receiving, at a first process, a request from a second process for storing event data by the second process; the requested event data is sent from the first process to a second process, wherein the second process is configured to adjust one or more components of the mobile device based on the event data. In some implementations, the one or more events include a pattern of events, and wherein the first process is configured to, when the pattern of events is detected, identify a pattern in the received event data and invoke the second process.

In another aspect, a mobile device may be configured to monitor environmental, system, and user events associated with the mobile device and/or a peer device. The occurrence of one or more events may trigger an adjustment to a system setting. The mobile device may be configured to keep frequently invoked applications up to date based on forecasts made for predicted user invocations. In some implementations, the mobile device can receive a push notification associated with an application that indicates new content.

In some implementations, there is provided a method comprising: receiving, by a first process executing on a mobile device, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes, wherein each attribute is associated with a budget and each event has a corresponding cost; reducing a budget for a particular attribute based on an event cost associated with the particular attribute received by the mobile device; storing the event data in an event data store on the mobile device; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; comparing the cost of the event to the remaining budget for the particular attribute; and allowing, by the first process, the event associated with the particular attribute based on the comparison.

In some implementations, at least one of the plurality of attributes is dynamically defined by the client at runtime. In some implementations, determining the allowed events includes generating a prediction for a particular attribute that indicates when an event associated with the attribute is likely to occur. In some implementations, determining the allowable events includes determining that sufficient budget remains to pay for the event cost. In some implementations, the budget for a particular attribute is dynamically defined by the client. In some implementations, the budget corresponds to a portion of a system-level data budget. In some implementations, the budget corresponds to a portion of the system-level energy budget.

In some implementations, a non-transitory computer-readable medium is provided that includes one or more sequences of instructions which, when executed by one or more processors, cause: receiving, by a first process executing on a mobile device, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes, wherein each attribute is associated with a budget and each event has a corresponding cost; reducing a budget for a particular attribute based on an event cost associated with the particular attribute received by the mobile device; storing the event data in an event data store on the mobile device; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; comparing the cost of the event to the remaining budget for the particular attribute; and allowing, by the first process, the event associated with the particular attribute based on the comparison.

In some implementations, at least one of the plurality of attributes is dynamically defined by the client at runtime. In some implementations, the instructions that cause the device to determine that the event is allowed include instructions that cause the device to generate a forecast for a particular attribute that indicates when an event associated with the attribute is likely to occur. In some implementations, the instructions that cause the device to determine that the event is allowed include instructions that cause the device to determine that sufficient budget remains to pay for the event cost. In some implementations, the budget for a particular attribute is dynamically defined by the client. In some implementations, the budget corresponds to a portion of a system-level data budget. In some implementations, the budget corresponds to a portion of the system-level energy budget.

In some implementations, a system is provided that includes one or more processors; and a computer-readable medium comprising one or more sequences of instructions which, when executed by one or more processors, cause: receiving, by a first process executing on a mobile device, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes, wherein each attribute is associated with a budget and each event has a corresponding cost; reducing a budget for a particular attribute based on an event cost associated with the particular attribute received by the mobile device; storing the event data in an event data store on the mobile device; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; comparing the cost of the event to the remaining budget for the particular attribute; and allowing, by the first process, the event associated with the particular attribute based on the comparison.

In some implementations, at least one of the plurality of attributes is dynamically defined by the client at runtime. In some implementations, the instructions that cause the device to determine that the event is allowed include instructions that cause the device to generate a forecast for a particular attribute that indicates when an event associated with the attribute is likely to occur. In some implementations, the instructions that cause the device to determine that the event is allowed include instructions that cause the device to determine that sufficient budget remains to pay for the event cost. In some implementations, the budget for a particular attribute is dynamically defined by the client. In some implementations, the budget corresponds to a portion of a system-level data budget. In some implementations, the budget corresponds to a portion of the system-level energy budget.

In another aspect, a mobile device may be configured to monitor environmental, system, and user events associated with the mobile device and/or a peer device. The occurrence of one or more events may trigger an adjustment to a system setting. The mobile device may be configured to keep frequently invoked applications up to date based on forecasts made for predicted user invocations. In some implementations, the mobile device can receive push notifications associated with such applications: which indicates that there is new content available for the application to download. The mobile device may launch an application associated with the push notification in the background and download the new content. In some implementations, prior to running an application or communicating with a peer device, the mobile device can be configured to check the energy and data budget and environmental conditions of the mobile device and/or the peer device to ensure a high quality user experience.

In some implementations, there is provided a method comprising: receiving, by a first process from one or more plug-in processes executing on a computing device, a request to register the plug-in process as one or more voting processes; receiving, by a first process, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes; storing the event data in an event data store on the mobile device; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; sending information identifying the particular attribute to each registered voting process; receiving a vote from at least one of the registered voting processes in response to sending information identifying the particular attribute to each registered voting process; and determining, by the first process, to allow the event associated with the particular attribute based on the voting.

In some implementations, the one or more voting processes are dynamically inserted into the first process at runtime. In some implementations, determining, by the first process based on feedback from the one or more table processes, the event associated with the particular attribute includes: sending information identifying the specific attribute to each voting process; and receiving a "yes" vote from each voting process when each voting process determines that an event associated with a particular attribute should be allowed to occur. In some implementations, the method includes: when the first process receives a "no" vote from at least one of the one or more voting processes, it is determined by the first process to block a second event associated with the second attribute. In some implementations, the method includes: receiving a request from at least one of the voting processes to make a forecast associated with a particular attribute; generating a requested forecast; and returning the requested forecast to the at least one voting process. In some implementations, the method includes: determining, by the first process, a third event that is allowed to be correlated with the particular attribute value based on feedback from the one or more voting processes. In some implementations, determining, by the first process based on feedback from the one or more table processes, a third event associated with the particular attribute value includes: sending information identifying the specific attribute value to each voting process; and receiving a "yes" vote from each voting process when each voting process determines that an event associated with a particular attribute value should be allowed to occur.

In some implementations, a non-transitory computer-readable medium is provided that includes one or more sequences of instructions which, when executed by one or more processors, cause: receiving, by a first process from one or more plug-in processes executing on a computing device, a request to register the plug-in process as one or more voting processes; receiving, by a first process, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes; storing the event data in an event data store on the mobile device; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; sending information identifying the particular attribute to each registered voting process; receiving a vote from at least one of the registered voting processes in response to sending information identifying the particular attribute to each registered voting process; and determining, by the first process, to allow the event associated with the particular attribute based on the voting.

In some implementations, the one or more voting processes are dynamically inserted into the first process at runtime. In some implementations, the instructions that cause the device to determine, by the first process based on the feedback from the one or more voting processes, the event associated with the particular attribute include instructions that cause the device to: sending information identifying the specific attribute to each voting process; and receiving a "yes" vote from each voting process when each voting process determines that an event associated with a particular attribute should be allowed to occur. In some implementations, the instructions cause the apparatus to: when the first process receives a "no" vote from at least one of the one or more voting processes, it is determined by the first process to block a second event associated with the second attribute. In some implementations, the instructions cause the apparatus to: receiving a request from at least one of the voting processes to make a forecast associated with the particular attribute; generating a requested forecast; and returning the requested forecast to the at least one voting process. In some implementations, the instructions cause the apparatus to: determining, by the first process, to allow a third event associated with the particular attribute value based on feedback from the one or more voting processes. In some implementations, the instructions that cause the device to determine, by the first process based on the feedback from the one or more voting processes, a third event associated with the particular attribute value include instructions that cause the device to: sending information identifying the specific attribute value to each voting process; and a "yes" vote is received from each voting process when it determines that the event associated with the particular attribute value should be allowed to occur.

In some implementations, a system is provided that includes one or more processors; and a computer-readable medium comprising one or more sequences of instructions which, when executed by one or more processors, cause: receiving, by a first process from one or more plug-in processes executing on a computing device, a request to register the plug-in process as one or more voting processes; receiving, by a first process, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes; storing the event data in an event data store on the mobile device; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; sending information identifying the particular attribute to each registered voting process; receiving a vote from at least one of the registered voting processes in response to sending information identifying the particular attribute to each registered voting process; and determining, by the first process, to allow the event associated with the particular attribute based on the voting.

In some implementations, the one or more voting processes are dynamically inserted into the first process at runtime. In some implementations, the instructions that cause the device to determine, by the first process based on the feedback from the one or more voting processes, the event associated with the particular attribute include instructions that cause the device to: sending information identifying the specific attribute to each voting process; and receiving a "yes" vote from each voting process when each voting process determines that an event associated with a particular attribute should be allowed to occur. In some implementations, the instructions cause the apparatus to: when the first process receives a "no" vote from at least one of the one or more voting processes, it is determined by the first process to block a second event associated with the second attribute. In some implementations, the instructions cause the apparatus to: receiving a request from at least one of the voting processes to make a forecast associated with the particular attribute; generating a requested forecast; and returning the requested forecast to the at least one voting process. In some implementations, the instructions cause the apparatus to: determining, by the first process, to allow a third event associated with the particular attribute value based on feedback from the one or more voting processes. In some implementations, the instructions that cause the device to determine, by the first process based on the feedback from the one or more voting processes, a third event associated with the particular attribute value include instructions that cause the device to: sending information identifying the specific attribute value to each voting process; and a "yes" vote is received from each voting process when it determines that the event associated with the particular attribute value should be allowed to occur.

In another aspect, a mobile device may be configured to monitor environmental, system, and user events associated with the mobile device and/or a peer device. The occurrence of one or more events may trigger an adjustment to a system setting. The mobile device may be configured to keep frequently invoked applications up to date based on forecasts made for predicted user invocations. In some implementations, the mobile device can receive push notifications associated with such applications: which indicates that there is new content available for the application to download. The mobile device may launch an application associated with the push notification in the background and download the new content. In some implementations, prior to running an application or communicating with a peer device, the mobile device can be configured to check the energy and data budget and environmental conditions of the mobile device and/or the peer device to ensure a high quality user experience.

In some implementations, there is provided a method comprising: receiving, by a first process executing on a mobile device, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes; storing the event data in an event data store on the mobile device; generating one or more event forecasts for each attribute in the stored event data; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; events associated with a particular attribute are allowed to be determined by a first process based on a forecast generated for the particular attribute.

In some implementations, the one or more forecasts predict a likelihood that an event associated with a certain attribute will occur within a period of time. In some implementations, the one or more forecasts include peer forecasts. In some implementations, the one or more forecasts include a temporal forecast. In some implementations, the one or more forecasts include a frequency forecast based on a frequency of occurrence of a particular attribute in the event data store. In some implementations, the one or more forecasts include a panoramic forecast based on events associated with attributes different from the particular attribute. In some implementations, the method includes: a default forecast type is determined based on a prediction of an occurrence of the received event for each of a plurality of forecast types. In some implementations, the plurality of forecast types includes a frequency forecast type and a panorama forecast type.

In some implementations, a non-transitory computer-readable medium is provided that includes one or more sequences of instructions which, when executed by one or more processors, cause: receiving, by a first process executing on a mobile device, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes; storing the event data in an event data store on the mobile device; generating one or more event forecasts for each attribute in the stored event data; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; events associated with a particular attribute are allowed to be determined by a first process based on a forecast generated for the particular attribute.

In some implementations, the one or more forecasts predict a likelihood that an event associated with a certain attribute will occur within a period of time. In some implementations, the one or more forecasts include peer forecasts. In some implementations, the one or more forecasts include a temporal forecast. In some implementations, the one or more forecasts include a frequency forecast based on a frequency of occurrence of a particular attribute in the event data store. In some implementations, the one or more forecasts include a panoramic forecast based on events associated with attributes different from the particular attribute. In some implementations, the instructions cause: a default forecast type is determined based on each of a plurality of forecast types for a forecast instance of an occurrence of a received event. In some implementations, the plurality of forecast types includes a frequency forecast type and a panorama forecast type.

In some implementations, a system is provided that includes one or more processors; and a non-transitory computer-readable medium comprising one or more sequences of instructions which, when executed by the one or more processors, cause: receiving, by a first process executing on a mobile device, events generated by one or more client processes, each event comprising data associated with one of a plurality of attributes; storing the event data in an event data store on the mobile device; generating one or more event forecasts for each attribute in the stored event data; receiving, by a first process, a request from a client process to initiate an event associated with a particular attribute; determining, by a first process, to allow an event associated with a particular attribute based on a forecast generated for the particular attribute.

In some implementations, the one or more forecasts predict a likelihood that an event associated with a certain attribute will occur within a period of time. In some implementations, the one or more forecasts include peer forecasts. In some implementations, the one or more forecasts include a temporal forecast. In some implementations, the one or more forecasts include a frequency forecast based on a frequency of occurrence of a particular attribute in the event data store. In some implementations, the one or more forecasts include a panoramic forecast based on events associated with attributes different from the particular attribute. In some implementations, the instructions cause: a default forecast type is determined based on each of a plurality of forecast types for a forecast instance of an occurrence of a received event. In some implementations, the plurality of forecast types includes a frequency forecast type and a panorama forecast type.

In yet another aspect, a mobile device may be configured to monitor environmental, system, and user events associated with the mobile device and/or a peer device. The occurrence of one or more events may trigger an adjustment to a system setting. The mobile device may be configured to keep frequently invoked applications up to date based on forecasts made for predicted user invocations. In some implementations, the mobile device can receive push notifications associated with such applications: which indicates that there is new content available for the application to download. The mobile device may launch an application associated with the push notification in the background and download the new content. In some implementations, prior to running an application or communicating with a peer device, the mobile device can be configured to check the energy and data budget and environmental conditions of the mobile device and/or the peer device to ensure a high quality user experience.

In some implementations, there is provided a method comprising: receiving, at a thermal management daemon executing on a mobile device, a request to vote on allowing an event associated with a specified value for a property to occur; requesting peer forecasting for the attribute from the sampling daemon; receiving a score for each of a plurality of values associated with the attribute and predicted to occur near a current time; voting is performed based on the score of the specified attribute value to allow the event.

In some implementations, the method includes: determining a number of highest scoring attribute values of the plurality of values; when the specified attribute value is included in the number of highest scoring attribute values, a vote is made to allow the event. In some implementations, the method includes: when the specified attribute value is not included in the plurality of values, a vote is made to block the event. In some implementations, the method includes: determining a number of lowest scoring attribute values of the plurality of values; when the specified attribute value is included in the number of lowest scoring attribute values, a vote is made to block the event. In some implementations, the method includes: the number of lowest scoring attribute values is determined based on the current operating temperature of the mobile device. In some implementations, the method includes: the number of lowest scoring attribute values is determined based on the location of the current operating temperature within the operating temperature range.

In some implementations, a non-transitory computer-readable medium is provided that includes one or more sequences of instructions which, when executed by one or more processors, cause: receiving, at a thermal management daemon executing on a mobile device, a request to vote on allowing an event associated with a specified value for a property to occur; requesting peer-to-peer forecasting for the attribute from the sampling daemon; receiving a score for each of a plurality of values associated with the attribute and predicted to occur near a current time; voting is performed based on the score of the specified attribute value to allow the event.

In some implementations, the instructions cause: determining a number of highest scoring attribute values of the plurality of values; when the specified attribute value is included in the number of highest scoring attribute values, a vote is made to allow the event. In some implementations, the instructions cause: when the specified attribute value is not included in the plurality of values, a vote is made to block the event. In some implementations, the instructions cause: determining a number of lowest scoring attribute values of the plurality of values; when the specified attribute value is included in the number of lowest scoring attribute values, a vote is made to block the event. In some implementations, the instructions cause: the number of lowest scoring attribute values is determined based on the current operating temperature of the mobile device. In some implementations, the instructions cause: the number of lowest scoring attribute values is determined based on a location of the current operating temperature within the operating temperature range.

In some implementations, a system is provided that includes one or more processors; and a computer-readable medium comprising one or more sequences of instructions which, when executed by the one or more processors, cause: receiving, at a thermal management daemon executing on a mobile device, a request to vote on allowing an event associated with a specified value for a property to occur; requesting peer forecasting for the attribute from the sampling daemon; receiving a score for each of a plurality of values associated with the attribute and predicted to occur near a current time; voting is performed based on the score of the specified attribute value to allow the event.

In some implementations, the instructions cause: determining a number of highest scoring attribute values of the plurality of values; when the specified attribute value is included in the number of highest scoring attribute values, a vote is made to allow the event. In some implementations, the instructions cause: when the specified attribute value is not included in the plurality of values, a vote is made to block the event. In some implementations, the instructions cause: determining a number of lowest scoring attribute values of the plurality of values; when the specified attribute value is included in the number of lowest scoring attribute values, a vote is made to block the event. In some implementations, the instructions cause: the number of lowest scoring attribute values is determined based on the current operating temperature of the mobile device. In some implementations, the instructions cause: the number of lowest scoring attribute values is determined based on a location of the current operating temperature within the operating temperature range.

Section 2: searchingCable technology

The content in this section, "search techniques," describes performing federated searches, multi-domain query completion terms, and the use of user feedback in referencing search indexes, according to some embodiments, and provides information that complements the disclosure provided herein. For example, some portions of this section describe the generation of multiple ranked query results from a query through multiple separate search domains (e.g., search maps, people, and places), which supplement the disclosure provided herein, e.g., related to method 800, and related to populating prediction portion 930 of fig. 9B-9C, as described below. As another example, some portions of this section describe searching and determining search completion terms, which supplement disclosures provided herein, such as those related to automatically displaying related content (e.g., method 800) without receiving any user input, and related to generating predicted content (e.g., as discussed below with reference to fig. 3A-3B) using previous search histories and based on the user's previous search histories. As another example, some portions of this section describe monitoring user interactions with search results in order to improve presentation of search results, which complements the disclosure herein, e.g., those related to using previous search histories to generate predicted content (e.g., as discussed below with reference to fig. 3A-3B).

Search technical introduction

Methods and apparatus to perform multi-domain query searches are described herein. In an exemplary embodiment, the device receives a query prefix from a client of a user. The device also determines a plurality of search completion terms across a plurality of independent search domains. In addition, the device ranks a plurality of search completion terms based on the scores calculated for each of the plurality of search completion terms determined by the corresponding search domain, wherein at least one of the plurality of search completion terms is used to generate a plurality of search results in response to receiving the query prefix without an indication from the user.

In another embodiment, the device uses feedback from the user search session to generate a result cache. In this embodiment, the device receives a feedback package from the client, where the feedback package characterizes user interactions with a plurality of query results in a search session presented to the user in response to a query prefix entered by the user. The device also generates a plurality of results for the plurality of queries by running the plurality of queries using the search feedback index to obtain a plurality of results. Additionally, the device creates a results cache from the plurality of results, wherein the results cache maps the plurality of results to the plurality of queries, and the results cache is to provide the query results to the client.

In another embodiment, the appliance generates a plurality of ranked query results from a query over a plurality of independent search domains. In this embodiment, the device receives a query and uses the query to determine a plurality of results across a plurality of independent search domains. The device also characterizes the query. Further, the device ranks the plurality of results based on the scores computed for each of the plurality of results determined by the respective search domain and query characterization, where the query characterization indicates a query type.

Other methods and apparatus are also described.

Detailed description of search techniques

Methods and apparatus to perform multi-domain query searches are described herein. In the following description, numerous specific details are set forth to provide a thorough explanation of embodiments of the present invention. It will be apparent, however, to one skilled in the art that embodiments of the invention may be practiced without these specific details. In other instances, well-known components, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

In the following description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. "coupled" is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, cooperate or interact with each other. "connect" is used to indicate the establishment of communication between two or more elements coupled to each other.

The processes illustrated in the figures are performed by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. Although the process is described below in terms of certain sequential operations, it should be understood that certain operations described may be performed in a different order.

Further, some operations may be performed in parallel rather than sequentially.

The terms "server," "client," and "device" are intended to refer to data processing systems in general, and not to specific form factors of the server, client, and/or device in particular.

Methods and apparatus to perform multi-domain query searches are described herein. In one embodiment, the device receives from the client an incremented query prefix entered by the user, and uses the incremented query prefix to generate a set of query completion terms for each query prefix. For example, in one embodiment, if the user enters the string "applet," the device receives the incremental query prefixes of "a," ap, "" app, "" appl, "and" applet. For each query prefix, the device generates a set of query completion terms. For example, in one embodiment, the finish item for "a" may be "applet. Similarly, the device may generate a different set of query completion terms for other incremental query prefixes. In one embodiment, the device determines the set of query completion terms from a plurality of search domains. For example, in one embodiment, the device searches for query completion terms across multiple search domains (such as a map search domain, a media search domain, a wiki search domain, a site search domain, and other search domains). In one embodiment, each search field includes one or more query completion tree for determining possible completions for the input query prefix. In one embodiment, each search domain returns a set of scores that the device uses to rank the query completion terms. For example, in one embodiment, each search domain returns a set of raw, local, and global scores that may be used by the device to rank different completion terms for different domains.

As described above, conventional systems will return possible query completion terms to the user, and the user will select one of the possible query completion terms for the query search. In contrast, in one embodiment, the device does not return the set of query completion terms to the user. Instead, the device ranks the set of query completion items and uses the subset of query completion items to determine relevant results for the subset of query completion items without displaying the set of query completion items to the user or obtaining an indication of which of the set of query completion items to use to determine relevant results. In one embodiment, a device performs a search for relevant results across multiple search domains (e.g., a map, media, wiki, site, other search domain, or another search domain). The device receives a set of results from multiple search domains and ranks the results based on scores generated from each search domain and cross-domain information. In one embodiment, the device further ranks the relevant results based on the type of query completion terms used to determine the results. For example, in one embodiment, if a query completion term is characterized as a search for a place, results from the map search domain and wiki entries for that place may get a higher ranking. As another example, if the query completion term is indicated as pertaining to an artist, the media search domain results may get a higher ranking. The device will return the relevant results found for the query completion terms to the client.

In one embodiment, a user viewing the results may engage or dismiss the results. In one embodiment, a participation event may occur if a user interacts with one of the presentation results presented to the user during the user's search session. For example, in one embodiment, a user may click on a link presented for one of the surfaced results. As another example, the user may click on the link and spend longer than a predetermined time interacting with an object (e.g., a website) referenced by the link (e.g., interacting with the referenced object for more than 60 seconds). In this example, a user may receive results of a query search for the current President of the United states and click on a link that references a web page describing the President's latest lecture. If the user interacts with the website for more than a predetermined time (e.g., 60-90 seconds), the device will determine that the user is engaged in the result represented by the link. In another embodiment, the user may ignore or override the results presented to the user. For example, in one embodiment, if the user clicks on a link presented for one of the surfaced results, but does not browse the website within a predetermined time (e.g., less than 60-90 seconds), the device determines that this is a decommissioning event for the results.

In one embodiment, the feedback may be incorporated into a search index, where the feedback may affect the ranking and filtering of relevant results. In this embodiment, the client that presents and renders the relevant results additionally collects participation and disuse events for the user's search session. The client collects the events into a feedback packet and sends the packet to the server for processing. In one embodiment, a server

Receiving a feedback packet and converting the feedback packet into a feedback index entry. In one embodiment, the feedback index entries have a format of < query, result, presentation count, participation count, disuse count >, where the query is an input query and a contextual information query, such as device type, application, region settings, and geographic location; the result is a visualization result; the impression count is the number of times the result is to be rendered for the query; the engagement count is the number of times the results of the query are engaged; the discard count is the number of times the result is discarded. The entry is incorporated into the feedback search index. In one embodiment, the feedback search index is a search index that incorporates user feedback into the scoring results. For example, in one embodiment, each engagement event for a query result pair will facilitate the results of the respective query. In this example, if a user participates in a result of a particular query, future users may also participate with respect to that result of the same query. Thus, in one embodiment, the results of the query will be returned and ranked higher for future users having the same query. Conversely, if a user discards the results of a particular query, future users may also discard the same results for the same query. Thus, in one embodiment, the results of the query may be returned and ranked lower for future users having the same query.

In one embodiment, the server also uses the feedback search index to generate a results cache that maps queries to results. In one embodiment, the results cache is a cache that maps queries to results, which can be used to quickly return results for user queries. In one embodiment, the results cache is stored in an edge server proximate to the user device, which may serve one or more results prior to performing the query search. In one embodiment, the server generates a result cache by running a set of queries from the results that are set to generate an updated result set that incorporates the collected feedback into the results of the updated result set. The updated set of results is sent to the edge server.

FIG. 32_1 is a block diagram of one embodiment of a system 32_100 for returning search results based on an input query prefix. In fig. 32_1, the system 32_100 includes a search network 32_108 coupled to a device 32_102, a smartphone 32_114, and a tablet 32_ 116. In one embodiment, the search network is a network of one or more servers that receive query prefixes for different devices and return query results to those devices. For example, in one embodiment, the search network receives the query prefix 32_ 110A-D from the device 32_102, the smartphone 32_114, and/or the tablet 32_116 and returns the query result 32_112A-D to the respective device (e.g., the device 32_102, the smartphone 32_114, and/or the tablet 32_ 116). In one embodiment, the device 32_102 may be a personal computer, laptop, server, mobile device (e.g., smartphone, laptop, personal digital assistant, music playing device, gaming device, etc.), and/or capable of requesting and/or displaying queries. In one embodiment, the device may be a physical or virtual device. In one embodiment, the smart phone 32_114 may be a cellular phone capable of performing many of the functions of the device 32_ 102. In one embodiment, the tablet computer 32_116 may be a mobile device that accepts input on a display.

In one embodiment, each device includes a browser for entering a query prefix by a user. For example, in one embodiment, the device 32_102 includes a web browser 32_104 and a file browser 32_ 106. Each of these browsers includes a search input field for inputting a query prefix by a user. In one embodiment, web browser 32_104 is a program that allows a user to search for and retrieve web pages for various types of web page documents. In one embodiment, the web browser 32_104 includes a search input field 32_ 128. The search input field 32_128 is used by the user to enter a query prefix string. In one embodiment, the query prefix string is a string of text or other symbol of the query prefix to be used for being sent to the search network 32_ 108. The query prefix string may be an incomplete or complete search string entered by the user. In one embodiment, when the user types a query input string in the search input field 32_120A, the web browser 32_104 captures the query prefix string and sends the query prefix string in the query prefix 32_110A to the search network. For each symbol or text string entered in the search input field 32_120A, the web browser 32_104 will create and send a query prefix 32_110A to the search network 32_ 108. In response to receiving the query prefix 32_110A, the search network creates one or more query completion terms over multiple search domains and selects one or more of these query completion terms to create a set of relevant results 32_112 that are returned to the web browser 32_ 104. For example, in one embodiment, when the user enters the text "app," web browser 32_104 uses the query prefix strings "a," "ap," "app," and "app" to create query prefix 32_ 110A. For each of these query prefixes 32_110A, the search network 32_108 creates a set of query completion terms from multiple search domains, uses the query completion terms to determine relevant results, and returns a different set of results for a different query prefix 32_ 110A. The process of capturing the query prefix as the user enters subsequent characters may also be done in the file browser 32_ 106. In one embodiment, file browser 32_106 includes a search input field 32_120B that a user can use to enter a query prefix string. In this embodiment, as the user enters the query prefix string, the file browser 32_106 creates a different query prefix 32_110B and sends it to the search network 32_ 108. The search network 32_108 receives the different query prefixes 32_110B and determines one or more query completion terms, which in turn returns relevant results, as described above. Further, the query prefix may be used to perform queries using a metadata database of data stored locally on the device 32_ 106.

In one embodiment, the same process of capturing a query input string as it is entered, determining one or more query completion terms, and determining relevant results using the query completion terms may also be performed on the smartphone 32_114 and tablet 32_ 116. In this embodiment, the smartphone 32_114 includes a browser 32_ 116. The browser 32_116 includes a search input field 32_ 120C. Similar to the above, the search input field 32_120C is used by the user to enter a query prefix string. The query prefix string is incrementally captured by the browser 32_116, which in turn creates a different set of query prefixes 32_110C that are sent to the search network 32_ 108. In response to receiving each of these different query prefixes 32_110C, the search network 32_108 determines one or more query completion terms, and uses these query completion terms to determine the relevant results 32_112C that are returned to the browser 32_ 116. In addition, the tablet computer 32_116 includes a browser 32_ 118. The browser 32_118 includes a search input field 32_ 120D. Similar to the above, the search input field 32_120D is used by the user to enter a query prefix string. The query prefix string is incrementally captured by the browser 32_118, which in turn creates a set of different query prefixes 32_110D that are sent to the search network 32_ 108. In response to receiving each of these different query prefixes 32_110D, the search network 32_108 determines one or more query completion terms, and uses these query completion terms to determine the relevant results 32_112D that are returned to the browser 32_ 118. In one embodiment, the search network 32_108 includes a search module 32_118 that processes query completion terms and returns relevant results. Processing the query completion terms and returning relevant results is further described below in FIGS. 32_ 2-32 _ 7.

As described above, the browser on the device sends the query prefix 32_110A-D to the search network 32_ 108. In one embodiment, the query prefix 32_110A-D includes a query prefix string, a location (e.g., latitude/longitude combination), a device type identifier (e.g., computer, smartphone, tablet, etc.), and an application type identifier (e.g., Web browser (and what type of Web browser), file browser), and a region setting. In this embodiment, the context entered into the query prefix string by the user is provided to the search network 32 — 108 by providing the location, device type identifier, application type identifier, and region settings. In one embodiment, the search network 32_108 uses the context and query prefix string to determine query completion terms and relevant results. For example, in one embodiment, the search network 32_108 may use the location information to determine query completion terms and results that are relevant to the location of the device providing the query prefix. For example, the device location may be used to find search results for places near the current device location. As another example, in another embodiment, the search network 32_108 may use the device type identifier to determine completion terms and results for that device type. In this example, if the device type identifier indicates that the query prefix is from a smartphone, the search network 32_108 may give greater weight to the results for the application store for the smartphone than for the personal computer. In another example and further embodiment, application type identifiers and zone settings may also be used to weight completion items and results.

In one embodiment, the search network 32_108 completes the query prefix using a multi-domain query completion term. In this embodiment, the search network 32_108 sends each received query prefix to each search domain used by the search network 32_ 108. For example, in one embodiment, the search network 32_108 sends the received query prefix to a map search domain, a media search domain, a wiki search domain, a site search domain, and other search domains. Each of these search domains will determine one or more query completion terms for the query prefix based on data contained in the search domain. In addition, each search domain will return a set of scores for one or more query completion terms. For example, in one embodiment, the search domain will return a raw, local, and/or global score for each query completion term. Executing the multi-domain query completion term is further described in fig. 3-6.

Rather than returning the query completion terms determined by the search network 32_108 to the device providing the query prefix, the search network 32_108 uses one or more of the query completion terms to determine a set of relevant query results across multiple search domains. In one embodiment, the use of query completion to determine a set of relevant query results is performed without an indication from the user which of the query completion terms to use to determine relevant results. In this embodiment, when the user enters a string into the search input field, the search network 32_108 processes the string and returns the relevant results to the user. In one embodiment, the search network 32_108 uses one or more of the determined query completion terms to find and rank the query results for those query completion terms. In one embodiment, the search network 32_108 searches over a plurality of search domains that are available to search the network 32_ 108. In this embodiment, the search network 32_108 receives a set of results for the query completion term from each search domain. For each of these results, the search network 32_108 additionally receives a set of scores characterizing the result. In one embodiment, the score may include a score determined by the search domain providing the result, another metric, and/or a signal characterizing the query completion term used to provide the result, as described below in fig. 32_ 7. In one embodiment, the signal is based on lexical characterization of the query completion terms using a knowledge base. In one embodiment, the lexical characteristic description determines what type of query completion terms are being used for a multi-domain query search. Performing a multi-domain query search to determine a set of relevant results is further described below in fig. 32_7 and 32_ 13-32 _ 15.

Figure 32_2 is a flow diagram 32_200 of one embodiment of a process for determining query completion terms and related results based on an input query prefix. In fig. 32_2, the process 32_200 begins by receiving a query prefix. In one embodiment, the query prefix includes a query prefix string, location, device type identifier, application type identifier, and region settings as described above in FIG. 32_ 1. In this embodiment, the location, device type identifier, application type identifier, and/or region settings give context to the query prefix for which the user entered the query prefix string. At block 32_204, the process 32_200 determines query completion terms across multiple search domains, while ranking and selecting query completion terms. In one embodiment, the process 32_200 uses the query prefix to determine a set of query completion terms from each different such domain. For example, in one embodiment, if the query prefix string is "ap," the process 32_200 will use the query prefix string to determine the set of query completion terms from different search domains (e.g., maps, media, wiki, site, and/or other search domains). In this example, the map search domain may return the query completion term to the city Apache Junction, the media search domain may return the query completion term to the musical composition, apacharia Spring, the wiki search domain may return the query completion term to Apple company, and the site search domain may return the query completion term to the website Apple. In one embodiment, if the query prefix string has a minimum number of characters (e.g., four characters), the process 32_200 will create the set of query completion terms.

In addition, the process 32_200 ranks and selects possible query completion terms received from different such domains. In one embodiment, the process 32_200 ranks the possible query completion terms based on scores determined by the corresponding search domains and weights based on the context of the query prefix. In this embodiment, the process 32_200 selects the set of query completion terms based on the rankings. In one embodiment, rather than returning the set of query completion terms to the user entering the query prefix string (for large completion terms), the set of relevant results is determined using the set of query completion terms and then returned to the user. Determining a set of query completion terms is further described in FIGS. 32_ 3-32 _6 below.

The process 32_200 determines the set of correlation results at block 32_ 206. In one embodiment, the process 32_200 determines the relevant results based on the query completion terms determined at block 32_ 204. In this embodiment, the process 32_200 searches over multiple search domains available to the process 32_ 200. In this embodiment, the process 32_200 receives a set of results for the query completion term(s) from each search domain. For each of these results, the process 32_200 additionally receives a set of scores characterizing the result. In one embodiment, the score may include a score determined by the search domain providing the result, another metric, and/or a signal characterizing the query completion term used to provide the result, as described below in fig. 32_ 7. In one embodiment, the signal is based on lexical characterization of query completion terms using a knowledge base. In one embodiment, the vocabulary specific description determines what type of query completion terms are being used for a multi-domain query search. Determining the set of correlation results is further described in fig. 32_7 and 32_ 13-32 _15 below. At block 32_208, the process 32_200 returns the set of correlation results to the user. In another embodiment, the feedback index may be used as a signal field to weight the results. This embodiment is further described in fig. 32_14 below.

As described above, the process 32_200 determines query completion terms and related results across multiple search domains. In one embodiment, query completion terms and related results are aggregated using an aggregator. FIG. 32_3 is a block diagram of one embodiment of a system 32_300 including an aggregator 32_302 and a plurality of search domains 32_ 304A-F. In one embodiment, the aggregator 32_302 receives a request for a query completion term based on an input query prefix. In response to receiving the input query prefix, the aggregator 32_302 sends the input query prefix to each search domain 32_ 304A-F. Each search domain 32_304A-F uses the input query prefix to determine possible query completion terms in that domain. For example, in one embodiment, the map search domain 32_304A receives an input query prefix and searches for possible query completion terms in that domain. In one embodiment, aggregator 32_302 receives query completion terms from each search domain and ranks the received query completion terms based on a score for each completion term determined by the respective search domain and a weight based on the query prefix context.

In one embodiment, the map search domain 32_304A is a search domain that includes geographic map related information. In this embodiment, the map information may include information about a map, address, location, merchant, attraction, or other type of information related to a map. In another embodiment, the map information may also include information related to sights, such as open times, ratings and scores, contact information, routes, and/or photographs related to the location. In one embodiment, the media search field 32_304B is a media related search field. In one embodiment, the media search field 32_304B includes information related to music, books, video, lessons, spoken language, podcasts, radio stations, and/or other types of media. In further embodiments, the media search domain 32_304B may include information related to applications that may be run on devices, such as the device 32_102, smartphone 32_114, and tablet 32_116 described above in fig. 32_ 1. In one embodiment, the media search domain is a media store that includes different types of media that may be purchased (e.g., music, books, videos, lessons, spoken language, podcasts, stations, applications, and/or other types of media). In one embodiment, wiki search field 32_304C is an online encyclopedia search field. For example, in one embodiment, wiki search field 32_304C may be WIKIPEDIA (WIKIPEDIA). In one embodiment, site search field 32_304D is a search field for a website. For example, in one embodiment, the site search fields 32_304D include commercial, government, public and/or private websites such as "apple. com", "white house. gov", "yahoo. com", and the like. In one embodiment, the other search domains 32_304E are a set of other search domains (e.g., news search domains) that may be accessed by the aggregator 32_ 302. In one embodiment, the feedback completion items field 32_304F is a search index based on query feedback collected by browsers running on various devices. In one embodiment, the feedback completion items field 32_304F includes a feedback index that maps queries to results based on the collected query feedback. The feedback index is further described in fig. 32_ 8-32 _12 below.

As described above, each search field 32_304A-F includes information that allows each search field to present a set of query completion terms based on an input query prefix. In one embodiment, each search domain includes a query completion term tree for determining query completion terms and determining a score for each query completion term. FIG. 32_4 is a diagram of one embodiment of a query completion term search field 32_ 402. In FIG. 32_4, the query completion term search field 32_402 includes a query completion term tree 32_400 having nodes 32_ 404A-J. In one embodiment, each of the nodes 32_404A-J represents a character in the corresponding language. In this embodiment, different query completion terms may be represented by following nodes 32_404A-J on the tree. For example, in one embodiment, starting at node 32_404A and following to node 32_404C, a completion item starting with the letter "ap" may be represented. Each node also has a frequency, which is the number of times this completion has been matched to the input query prefix. In one embodiment, node 32_404C has a frequency N. In this embodiment, the frequency is represented as the raw score returned to aggregator 32_302 above. In one embodiment, the frequency may be calculated based on a log (e.g., a map or media search field), a page visited (e.g., a wiki search field), or another information source. Under node 32_404C, there are a number of possible other query completion terms. For example, in one embodiment, nodes 32_404D-F represent query completion terms that begin with the letters "apa," apt, "and" app. The total number of possible query-completed terms under the node gives an indication of the closeness of the query-completed term represented by the node. If there are a large number of possible other nodes under a node, the query completion term represented by that node may not be a good completion term. On the other hand, if there are relatively few nodes below a node, then that node may be a good completion item. In one embodiment, the local score for the node is represented by the frequency of the node divided by the number of completion items represented by the sub-tree below the node. In one embodiment, the formula for the local score is represented by formula (1):

Local score (node) ═ frequency (node)/number of completions below the node.

In one embodiment, each query completion term tree includes a total number of completion terms. This value is used to calculate a global score for the completed item (or node). In one embodiment, the formula for the global score is represented by formula (2): global score (node) ═ frequency (node)/number of completed items in query completed item tree

In one embodiment, the raw, local, and global scores for each query completion term are returned to the aggregator by the search domain.

Fig. 32_5 is a diagram of one embodiment of a map search field 32_ 500. In FIG. 32_5, the map search domain 32_500 includes query completion term trees 32_504A-D for different zoom levels of the domain. In one embodiment, the map search domain 32_500 includes a query completion term tree for the city level 32_504A, the county level 32_504B, the state level 32_504C, and the country level 32_504D, which are aggregated by the map aggregator 32_ 502. In this embodiment, the map aggregator 32_502 receives a determination of a query completion term for an input query prefix, which in turn determines the query completion term for that input query prefix at different zoom levels 32_504A-D of the map search domain 32_ 500. Map aggregator 32_502 retrieves possible query completion terms from each of the different zoom levels 32_504A-D, aggregates the query completion terms, and returns the query completion terms to the aggregator (e.g., aggregator 32_ 302). Thus, the map search domain 32_500 determines query completion terms across different zoom levels. In one embodiment, the map search field 32_500 includes information about addresses, locations, businesses, attractions, and/or any other information relevant to a map. In one embodiment, the map search field 32_500 may include directory information, such as white page or yellow page directories. In one embodiment, the media search domains are organized according to storefronts, which are based on a combination of device identifiers and zone settings. In this embodiment, there is a tree of query completion terms for each storefront. FIG. 32_6 is a flow diagram 32_600 of one embodiment of a process for determining query completion terms from multiple search domains. In one embodiment, the aggregator 32_302 performs the process 32_600 to determine query completion terms from multiple search domains. In fig. 32_6, the process 32_600 begins with receiving a query prefix at block 32_ 602. In one embodiment, the query prefix includes a query prefix string in context, as described above in fig. 32_ 2. At block 32_602, the process 32_600 sends the query prefix to a different search domain to determine possible completion terms. In one embodiment, the process 32_600 sends the query prefix to map, media, wiki, site, and/or other search domains, where each search domain determines possible query-completion terms for entering the query prefix based on a query-completion term tree available for each search domain, as described above in FIG. 32_ 4. The process 32_600 receives possible query completion terms from each search domain at block 32_ 606. In addition to receiving possible query completion terms, the process 32_600 also receives a set of scores, e.g., raw, local, and/or global scores, for each possible completion term, as described above in FIG. 32_ 4. At block 32_608, the process 32_600 ranks and filters the possible query completion terms based on the returned scores and the context of the input query prefix. In one embodiment, the process 32_600 ranks the possible query completion terms based on the raw, local, and global scores received from the different search domains and the context contained in the query prefix. The process 32_600 may also filter possible query completion terms based on a set of rules. For example, in one embodiment, the filter rule may be that the process 32_600 filters out possible completion items that have an original score of one or less than some predetermined value. The process 32_600 sends the ranked and filtered completion terms to the search query module at block 32_610, where the search query module uses the set of ranked and filtered completion terms to determine a set of relevant results to be returned to the user.

As described above, the query completion terms determined by the process 32_600 are used to determine relevant results without returning these completion terms to the user. FIG. 32_7 is a flow diagram of one embodiment of a process 32_700 for determining relevant results in multiple search domains from a determined query completion term. In one embodiment, the combiner 32_824 performs the process 32_ 700. In FIG. 32_7, the process 32_700 receives a query completion entry from the completer at block 32_ 702. In one embodiment, the received query completion terms are completion terms determined by the process 32_600 in response to receiving the query prefix. At block 32_704, the process 32_700 sends the query completion term to a different search domain to determine possible correlation results. In one embodiment, each search domain uses the received query completion terms to determine relevant results for that search domain. At block 32_706, the process 32_700 receives query results from different search domains. In one embodiment, the process 32_700 receives a score associated with each result calculated by the relevant search domain.

The process 32_700 ranks and filters the search results at block 32_ 708. In one embodiment, the process 32_700 ranks the search results according to the scores returned by each search domain for the search results and other factors. In this embodiment, scores from different domains may be scored based on domain dependency scores, query independence scores, and query dependency scores. In one embodiment, each different search domain may provide specific data for ranking returned results. For example, in one embodiment, if the result has a Uniform Resource Locator (URL) associated with the result (e.g., if the result is a merchant location, if the merchant has a URL reference website or other social media presence information) and/or counts the number of clicks, the map search domain may provide various query independence information to rank the results: number of online ratings, average rating score, distance from the user (e.g., based on query prefix location information). As another example and another embodiment, the media search domain may provide other types of information for scoring: media rating count, media age, popularity, declining popularity, and/or purchase data result. In another example and embodiment, a wiki search field may provide information about the number of page views, edit histories, and languages available for ranking. Other search areas may provide scoring metrics such as number of citations and age.

In one embodiment, the process 32_700 receives a set of scores from each search domain and uses these scores to determine an initial score for each result. The process 32_700 applies the signal field to each result. In one embodiment, the signal field is a query completion term profile. In this embodiment, the process 32_700 characterizes each query completion item and uses the query completion item characterization to rank the results. For example, in one embodiment, the process 32_700 performs lexical characterization using a knowledge base to determine the type of query completion terms. In this example, the query completion term type indicates whether the query completion term identifies a person, place, thing, and/or another category. For example, in one embodiment, the process 32_700 may determine that the system is using the query completion term to determine the location. In this example, query results from the map search domain will be weighted (and ranked) higher in the ranking of the search results because the query completion terms are used to determine the locality. Query completion term characterization is further described below in FIGS. 32_ 13-32 _ 15.

In another embodiment, the process 32_700 applies an enhancement effect to each result score. In this embodiment, the process 32_700 applies the freshness due to the query to each result. In one embodiment, the freshness that the query should have means that if there is a spike or peak in the number of counts for that result recently, the result is a "fresh" result, which can be promoted. The result of its count fluctuating around the baseline over time is not a "fresh" result and is not boosted. In one embodiment, the count is based on an analysis of social media feeds (e.g., Twitter, etc.).

For example, in one embodiment, if the query completion term is "first love," and four results are returned: (1) song "first love" from the media search domain; (2) a merchant named "person in first love" from a map search domain; (3) a news article that speaks an early love advertisement; and (4) wiki entry "first love". In this embodiment, based on the search domain dependency metric, each result has an initial score as follows: from the media search domain { year, rating, and raw score }; from the map search domain { distance from user, with URL, number of comments, average comment } from; from the news domain { year, news score, reference count }, and the wiki search domain { web page rank, raw score }. Each search domain provides its own score to the process 32_ 700. In this example, the score for each result may initially be ranked as, wiki result > media result > news result > map result. The process 32_700 also applies the signal field to each result. In this example, the query "first love" is characterized as a song, and may also be a place. The characterization can be applied to improve the media storage result and improve the mapping result to a smaller extent. After applying the characterization boost, the result score may be ranked as: wiki results > media results (but scores closer) > map results > news results. In addition, the process 32_700 applies the boost that the query should have to the results. For example, the count of commercials is increased because the "first love" commercial has been played for the first two days. Thus, the "first love" result will get the freshness boost that the query should have. In this example, the news result "first love" will get a large boost, so the result will be ranked as: news results > wiki results > media results > map results.

In one embodiment, the process 32_700 additionally filters the search results. In this embodiment, the process 32_700 removes the results based on certain rules. For example, in one embodiment, the process 32_700 may remove results below a certain overall score. Alternatively, the process 32_700 may filter the results based on another criterion (e.g., text not well matched to the query, low click-through rate, low popularity, results with straightforward content and/or profanity, and/or combinations thereof). At block 32_710, the process 32_700 returns the ranked, filtered results to the user.

FIG. 32_8 is a block diagram of a system 32_800 that incorporates user feedback into a search index. In fig. 32_8, the system 32_800 includes a device 32_802 that sends a query prefix 32_828 to an edge server 32_804, which in turn returns a query result 32_830 to the device. In addition, edge server 32_804 is coupled to core server 32_ 816. In one embodiment, the device 32_802 sends the query prefix 32_828 to the edge server as the user enters the query prefix. For example, in one embodiment, if the user types in the query prefix "applet," a query prefix is generated for "a," "ap," "app," "appl," and "applet" as each character is entered by the user and sent to the edge server 32_ 804. Further, for each query prefix 32_828 sent to the edge server 32_804, the edge server 32_804 returns a correlation result 32_830 to the client. For example, in one embodiment, as the user enters each character, the edge server will return the relevant results for query prefixes 32_828 "a", "ap", "app", "appl", and "applet". In one embodiment, the edge server may also execute the query completion term. In one embodiment, the device 32_802 also collects feedback about the user's search session, collects the feedback into a feedback packet 32_832, and sends the feedback packet to the edge server. The collect and send feedback is further described in fig. 32_10 below. In one embodiment, the device 32_802 includes a collect feedback module 32_838 that collects and sends feedback.

In one embodiment, the edge server 32_804 includes a feedback module 32_806, which further includes a feedback search module 32_808 and a feedback collection module 32_ 810. In one embodiment, the feedback search module 32_808 performs a search for each query prefix 32_828 based on the feedback index 32_814 stored on the edge cache 32_812 of the edge server 32_ 804. In this embodiment, as the user enters the query prefix 32_828, the system returns a new set of relevant results 32_830 to the device 32_802 using the feedback search module 32_808 and the feedback search index 32_ 814. In one embodiment, the feedback search index is an index that incorporates user feedback into the search index. In this embodiment, the feedback search index is a results cache for quickly providing results 32_830 to the device. In one embodiment, the feedback search index is a reference search index, which is further described below with reference to fig. 32_ 11. In one embodiment, the feedback set 32_810 collects feedback packets sent from the device 32_802 and forwards the feedback packets to the core server 32_ 816.

In one embodiment, the core server 32_816 includes a feedback feed pipeline 32_818, a feedback decision pipeline 32_822, a feedback index 32_820, and a consolidator 32_ 824. In one embodiment, the feedback feed pipeline 32_818 receives raw feedback packets 32_834 from the edge server 32_804 and converts each of these raw feedback packets 32_834 into an entry for the feedback index 32_ 820. In one embodiment, feedback feed pipeline 32_816 converts each of the raw feedback packets into a set of index entries having a format of < query, result, presentation count, participation count, disuse count >, where the query is an input query and a contextual information query, such as device type, application, region setting, and geographic location; the result is a visualization result; the impression count is the number of times the result is rendered for the query; the participation count is the number of times the results of the query are participated in; the discard count is the number of times the result is discarded. In the present embodiment, these index entries are added to the feedback index 32_ 820. Updating the feedback index using the original feedback packet is further described below in FIG. 32_ 11. In one embodiment, feedback index 32_820 is a search index that incorporates user feedback. The feedback feed pipeline 32_818 also includes a process feedback module 32_840 that updates the feedback index with the raw feedback packet.

In one embodiment, the feedback decision pipeline 32_822 uses the feedback index 32_820 to update the result set. In one embodiment, a result set is a mapping between a set of queries and results. In this embodiment, the feedback decision pipeline 32_822 runs a set of queries against the feedback index 32_820 to determine an updated result set. In this embodiment, the updated result set is sent to the coalition 32_ 824. The feedback decision pipeline 32_822 additionally sends the updated result set 32_826 to the edge server 32_ 804. The updated result set 32_826 includes results for the set of queries, as determined using the updated feedback index 32_ 820. In one embodiment, the feedback decision pipeline 32_822 includes an update results module 32_842 that updates the results set. The update result set is further described in fig. 32_12 below. In one embodiment, the feedback decision pipeline 32_822 additionally sends the updated result set to a feedback archive 32_836 that stores the updated result set 32_ 826. In one embodiment, the federator 32_824 performs a multi-domain search using the completed query, as described in FIGS. 32_ 13-32 _15 below.

As described above, the search network captures user feedback with respect to the user's search session and uses the feedback to build a search feedback index. Figure 32_9 is a flow diagram according to one embodiment of a process 32_900 of incorporating user feedback into a reference search index. In FIG. 32_9, the process 32_900 begins by collecting user feedback for a user search session. In one embodiment, the process 32_900 begins collecting feedback at the device that received the query results in response to the query prefix sent to the search network. In this embodiment, the process 32_900 collects feedback by detecting an initial appearance event or another event (e.g., beginning to enter a query prefix) and determining the user's interactions in the search session. In one embodiment, the user interaction may be to keep focus on the website referenced by the result, click on a link or other reference on the website, or another type of interaction. In one embodiment, a search session is a set of events initiated by a user starting to enter a query prefix that tracks the user's actions over a coarse period of time (e.g., 15 minutes). In one embodiment, if the user participates in any of these surfaced results ("participation events"), and if the user discards the surfaced results ("discard events"), the process 32_900 records the issued query prefix, the relevant results surfaced for the user. In one embodiment, the process 32_900 records whether the user is engaged in replacing the search option.

In one embodiment, a participation event may occur if a user interacts with one of the presentation results presented to the user. For example, in one embodiment, the user may click on a link presented for one of the revealed results. As another example, the user may click on the link and take longer than a predetermined time to interact with the object (e.g., website) referenced by the link (e.g., interact with the referenced object for more than 60 seconds). In this example, the user may receive results of a query search for the current President of the United states and click on a link that references a web page describing the President's latest lecture. If the user interacts with the website for more than a predetermined time (e.g., 60-90 seconds), the process 32_900 will determine that the user has engaged in the result represented by the link. Thus, this will be a participation event in this outcome. In one embodiment, hovering over a link may be recorded as engaging. In another embodiment, the user may also view the displayed results for a period of time. In this embodiment, actions that would otherwise be recorded as obsolete may be recorded as participating, and vice versa, depending on the type of outcome and the actions after the time period. For example, in one embodiment, if a user queries "Chinese population" and gets the display results, and the user pauses for 10 seconds before deleting the query, the event may be recorded as a participation rather than a disuse event.

In another embodiment, the user may ignore or override the results presented to the user. For example, in one embodiment, if the user clicks on a link presented for one of the surfaced results, but does not browse the website within a predetermined time (e.g., less than 60-90 seconds), the process 32_900 determines that this is a decommissioning event for the result. In one embodiment, there are other types of decommissioning events: continue typing more characters (extended query prefix); clicking to another window or application; deleting the query; backspace one or more characters or otherwise edit the query; any engagement other than what is presented as a result may be recorded as overriding the result. In one embodiment, the user's actions are recorded along with the time interval the user spends, which may change the interpretation of the abstinence to the interpretation of the participation, or vice versa.

In one embodiment, the user's search session may end after a predetermined time, whether during the user session duration, at a time of inactivity, or under some other metric. In response to the search session ending, the process 32_900 assembles the events collected for the search session into a feedback packet that is sent to the search network. Gathering feedback is further described in fig. 32_10 below.

At block 32_904, the process 32_900 processes the received feedback included in the feedback packet. In one embodiment, the process 32_900 converts the received feedback packet into an entry for feeding back a search index. In one embodiment, the feedback search index is a search index that incorporates user feedback into the scoring results. For example, in one embodiment, each participation event for a (query, result) pair will facilitate the results of the respective query. In this example, if a user participates in a result of a particular query, future users may also participate with respect to that result of the same query. Thus, in one embodiment, the results of the query will be returned and ranked higher for future users having the same query. Conversely, if a user discards the results of a particular query, future users may also discard the same results for the same query. Thus, in one embodiment, the results of the query may be returned and ranked lower for future users having the same query.

In one embodiment, the process 32_900 converts the received feedback package into a feedback search index entry having a format of < query, result, presentation count, participation count, disuse count >, wherein the query is an input query and a contextual information query, such as device type, application, region setting, and geographic location; the result is a visualization result; the impression count is the number of times the result is rendered for the query; the engagement count is the number of times the results of the query are engaged; discard counts are the number of times the result is discarded. In one embodiment, the process 32_900 updates the feedback index entry in the feedback search index. In another embodiment, each feedback packet further comprises a unique source identifier, which may comprise a user identifier, a device identifier or a session identifier, with or without a obfuscated identity to protect the privacy method, wherein an update feedback index entry is appended to an index in the reference index table, the unique source identifier being the source of the feedback reference. The feedback index may then be queried to provide results and weights that are personalized or customized for individual users or groups of users. Processing the received feedback is further described in fig. 32_11 below.

The process 32_900 updates the result cache at block 32_ 906. In one embodiment, the results cache is a cache that maps queries to results, which can be used to quickly return results for user queries. In one embodiment, the results cache is stored in an edge server proximate to the user device (e.g., an edge server geographically closer to the client than other edge servers) that may serve one or more results before performing the query search. In one embodiment, the process 32_900 updates the results by running a set of queries using the updated feedback search index to determine a set of results for the queries. The updated results are sent to each result cache stored on the edge server. The update result cache is further described in fig. 32_12 below.

Figure 32_10 is a flow diagram 32_1000 of one embodiment of a process for collecting user feedback during a user search session. In one embodiment, the process 32_100 is performed by a collect feedback module (such as the collect feedback module 32_838 described above in fig. 32_ 8) to collect user feedback during a user search session. In fig. 32_10, the process 32_1000 begins by detecting an event that triggers a feedback set. In one embodiment, the initial event may be the beginning of the entry of a query prefix string for another type of event. In one embodiment, if the user engages in a previous search session for a period of time (e.g., 15 minutes), the start of the entry of the query prefix string marks the start of a new user search session and begins recording user feedback. As described above, a search session is a set of events initiated by a user beginning to enter a query prefix that tracks the user's actions over a coarse period of time (e.g., 15 minutes).

At block 32_1004, the process 32_1000 records an event associated with the user search session. In one embodiment, the process 32_1000 records the surface, participation, and disuse events. In one embodiment, a surface event is a related result that is surfaced to a user in response to the user entering a query prefix or a complete query. In one embodiment, the process 32_1000 records the surface events by recording the results presented for each query prefix or complete query. Further, the process 32_1000 records the participation event at block 32_ 1004. In one embodiment, an engagement event is an event that would occur if the user interacted with one of the presentation results presented to the user. For example, in one embodiment, a user may click on a link presented for one of the surfaced results. As another example, a user may click on the link and take longer than a predetermined time to interact with an object (e.g., a website) referenced by the link (e.g., interact with the referenced object for more than 60 seconds). In this example, a user may receive results of a query search for the current President of the United states and click on a link that references a web page describing the President's latest lecture. If the user interacts with the website for more than a predetermined time (e.g., 60-90 seconds), the process 32_1000 will determine that the user has engaged in the results represented by the link. Thus, this will be the participation event for this result.

In another embodiment, the process 32_1000 may record an abandonment event, where an abandonment event is an event that a user may ignore or abandon a result that is apparent to the user. For example, in one embodiment, if the user clicks on a link presented for one of the surfaced results, but does not browse the website within a predetermined time (e.g., less than 60-90 seconds), the process 32_900 determines that this is a dismissal event for the result. In one embodiment, the user navigates away by closing a tab or window that presents the website, clicking on another application, or some other action that indicates that the user has not interacted with the presented website.

At block 32_1006, the process 32_1000 creates a feedback package from the recorded events of the user search session. In one embodiment, the user search session ends based on the following predetermined times: a predetermined time from the initial search session event (e.g., 15 minutes) or may be a predetermined time that the user is inactive with respect to the user search session. For example, in one embodiment, if the user has not been active or interacted with a result or other type of object referenced by one of the results for a predetermined amount of time (e.g., 10 minutes), the user search session will end. In one embodiment, in response to the end of a user search session, the process 32_1000 will collect the recorded events and create a feedback package from the user search session. In one embodiment, the feedback package includes a set of results presented to the user, a query associated with the results, an engagement event for the user to engage in the query results, and an override event for the user to override the results presented to the user, wherein each override event is associated with the query. The process 32_1000 sends the feedback packet to the search network at block 32_ 1008. In one embodiment, the client sends the feedback packet to the edge server, which forwards the feedback packet to the core server for processing.

Figure 32_11 is a flow diagram of one embodiment of a process 32_1100 to incorporate user feedback into a feedback index. In one embodiment, a process feedback module (such as the process feedback module 32_840 described in fig. 32_8 above) performs process feedback. In fig. 32_11, the process 32_1100 begins by receiving a feedback packet at block 32_ 1102. In one embodiment, the feedback packet is a feedback packet of a user search session as described above in fig. 32_ 10. At block 32_1104, the process 32_1100 converts the feedback packet into one or more feedback index entries. In one embodiment, the feedback index entry is the number of events recorded for a particular query result pair. For example, in one embodiment, the feedback index entry includes < query, result, presentation count, participation count, disuse count >, where the query is an input query and a contextual information query, such as device type, application, region settings, and geographic location; the result is a visualization result; the impression count is the number of times the result is rendered for the query; the participation count is the number of times the results of the query are subjected to participation; the discard count is the number of times the result is discarded.

At block 32_1106, the process 32_1100 inserts a feedback index entry into the feedback index. In one embodiment, the feedback index is a search index that incorporates user feedback into the search index. In one embodiment, the feedback index is a reference index, where a participation event is a positive reference to a result and a decommissioning event is a negative reference to the result. In one embodiment, the references to search indexes are described in U.S. patent application No.12/628,791 entitled "Ranking and Selecting Entities Based on calibrated report or inventory records" filed 12/1 2009 and incorporated in this section. In one embodiment, if there is an entry in the feedback index with the same query result pair, process 32_1100 will update the entry with the number of event counts.

As described above, user feedback incorporating a feedback index may be used to update the results cache. FIG. 32_12 is a flow diagram of one embodiment of a process 32_1200 for updating a results cache using user feedback. In one embodiment, an update results module (such as update results module 32_842 described above in FIG. 32_ 8) performs process 32_1200 to update the results cache. In FIG. 32_12, the process 32_1200 begins by receiving a result set RS that includes a plurality of queries. In one embodiment, the result set is a mapping between a set of queries and results. This result set can be used in a result cache to quickly return relevant results for the query prefix, as described above in fig. 32_ 8. In one embodiment, the result set is generated by a search index that does not include user feedback. In another embodiment, the result set is generated from a previous feedback index incorporating previous user feedback.

At block 32_1204, the process 32_1200 runs each query from the result set RS against the current feedback index. The process 32_1200 uses the results from the running query at block 32_1204 to create an updated result set RS' at block 32_ 1206. In one embodiment, the result set RS' is a feedback weighted result set, where results of queries having primarily participated in events get higher weights in the feedback index, and results of queries having primarily disused events get lower weights in the feedback index. For example, in one embodiment, if query Q in the result set RS has results ranked as R1, R2, and R3, and has those results for Q in the updated feedback index, where R1 has 20 participation events and 50 retirement events, R2 has 32_100 participation events and 2 retirement events, and R3 has 50 participation events and 10 retirement events, running query Q against the updated feedback index may return results ranked R2, R3, and R1. Thus, in one embodiment, the results in the updated result set RS' will be ranked using the feedback index. In another embodiment, the relevant results filter may have the rule: for a result to be presented, the result may require at least x participation events or no more than y disuse events. Thus, in this embodiment, the use of the feedback index can change which results are presented versus which results are not presented. The process 32_1200 sends the updated result set RS' to each edge server at block 32_ 1208. In one embodiment, the process 32_1200 sends the updated result set RS' from the core server 32_816 to the edge server 32_804, as described in fig. 32_8 above.

FIG. 32_13 is a block diagram of one embodiment of a federator 32_824 that performs multi-domain searches using token query completion terms. In one embodiment, the federator includes a done items module 32_1304, a mixer/ranker 32_1306, a plurality of search domains 32_1308A-F, and a vocabulary service 32_ 1314. In one embodiment, the completion entry module 32_1304 determines a query completion entry for each query prefix, as described in fig. 32_ 6. The determined query completion terms are forwarded to the mixer/ranker 32_1306 which uses the query completion terms to perform a multi-domain search for relevant results using the search domains 32_1308A-F, as described above in FIG. 32_ 7. In one embodiment, the search fields 32_1308A-F are the search fields described above in FIG. 32_ 3. For example, in one embodiment, the map search field 32_1308A is a search field that includes information about a geographic map, as described above in fig. 32_ 3. The map search field 32_1308A queries information from a map data source 32_ 1310A. Media search field 32_1308B is a media-related search field, as described above in fig. 32_ 3. Media search field 32_1308B queries information from media data source 32_ 1310B. Wiki search field 32_1308C is an online encyclopedia search field, as described above in fig. 32_ 3. Wiki search field 32_1308C queries for information from wiki data source 32_ 1310C. The site search field 32_1308D is a search field of a website, as described above in FIG. 32_ 3. The site search field 32_1308D queries information from the site data source 32_ 1310D. The other search domains are a set of other search domains that may be accessed by the mixer/ranker 32_1306 as described above in FIG. 32_ 3. Other search fields 32_1308E query information from other data sources 32_ 1310E. In one embodiment, the feedback search field 32_1308F is a search index based on query feedback collected by browsers running on various devices, as described above in fig. 32_ 3. The feedback search field 32_1308 queries information from a feedback data source 32_1310F (e.g., a feedback search index).

In addition, the mixer/ranker 32_1306 receives and ranks results from multiple search domains 32_ 1308A-F. In one embodiment, the mixer/ranker 32_1306 characterizes each query completion using a vocabulary service 32_1302 that determines what type of search is being performed. For example, in one embodiment, the vocabulary service 32_1302 may determine whether the search is for a person, place, thing, etc. In one embodiment, the vocabulary service 32_1302 uses a knowledge base 32_1312 that maps words or phrases to a class. In this embodiment, the characterization of query completion is used to weight the results returned by the search domains 32_ 1308A-F. For example, in one embodiment, if a query completion term is characterized as a search for a place, results from the map search domain and wiki entries for that place may get a higher ranking. As another example, if the query completion term is indicated as pertaining to an artist, the media search field results may get a higher ranking. Weighting the results is further described below in FIG. 32_ 14.

FIG. 32_14 is a flow diagram of one embodiment of a process 32_1400 for determining relevant results using vocabulary services for query completion terms. In one embodiment, the mixer/ranker 32_1306 performs the process 32_1400 to determine relevant results using the vocabulary service for query completion terms, as described above in FIG. 32_ 13. In FIG. 32_14, the process 32_1400 begins with receiving a query completion entry at block 32_ 1402. In one embodiment, the received query completion terms are completion terms determined by the process 32_600 in response to receiving the query prefix. In one embodiment, the process 32_1400 performs blocks 32_1404 and 32_1408 in one parallel flow and blocks 32_1406 and 32_1410 in another parallel flow. At block 32_1404, the process 32_1400 sends the query completion item to a different search domain to determine possible relevant results. In one embodiment, each search domain uses the received query completion terms to determine relevant results for that search domain. In one embodiment, multiple search domains process each query completion term in parallel. The process 32_1400 sends the query completion terms to the vocabulary service to characterize each completion term. In one embodiment, the vocabulary service characterizes each query completion term by determining whether the query completion term is a query about a person, place, thing, or another type of information. The characterization of the query completion terms is further described in FIG. 32_15 below. The process 32_1400 receives search results from multiple search domains at block 32_ 1408. In one embodiment, each search result includes a set of scores that characterize the results from the respective search domain.

At block 32_1410, the process 32_1400 receives a vocabulary search result characterizing the query completion term. In one embodiment, the characterization of the query completion terms indicates the type of information each query completion term is searching for. For example, in one embodiment, a query completion term is a query about a person, place, thing, or other type of information. In one embodiment, the two parallel streams converge at block 32_ 1412. The process 32_1400 uses the query completion term characterization to rank and filter relevant results for the query completion term at block 32_ 1412. In one embodiment, if the query completion term is indicated as a search for a person, results from the wiki domain about the person obtained by the search may get a higher ranking. For example, in one embodiment, if the query completion term is characterized as searching for movies, the results from the movie review or local show time may get a higher ranking. As another example, if a query completion term is indicated as a place, results from the map search domain and wiki entries for the place may get a higher ranking. As another example, if the query completion term is indicated as pertaining to an artist, the media search domain results may get a higher ranking. Ranking using query completion terms is also described above in FIG. 32_ 7. In another embodiment, the feedback index may be a signal field used to rank and/or filter relevant results. In this embodiment, the process 32_1400 ranks the results higher using the number of participation events and ranks the results lower using the number of disuse events. In one embodiment, the process 32_1400 additionally ranks and screens the results as described above at block 32_708 of fig. 32_ 7. The process 32_1400 returns the ranked, filtered results at block 32_ 1414.

As described above, the process 32_1400 uses lexical services to characterize query completion terms. Figure 32_15 is a flow diagram of one embodiment of a process 32_1500 for characterizing query completion terms. In FIG. 32_15, the process 32_1500 receives the query completion entry at block 32_ 1502. At block 32_1504, the process 32_1500 tokenizes each query completion term. In one embodiment, tokenizing the completion terms is the separation of the query completion terms into individual tokens (e.g., words, phrases, plural/single variants). For a tokenized query completion term, the process 32_1500 determines a match for the tokenized completion term in the knowledge base. In one embodiment, the knowledge base is a database of words or phrases that map to a certain category. For example, in one embodiment, the knowledge base may include entries such as { eiffel tower → site }, { Michael Jackson (Michael Jackson) → artist }, { bark Obama (barac Obama) → president }, { black widow → spider }, and the like. In one embodiment, the knowledge base is constructed using ontologies. In one embodiment, the process 32_1500 determines matches for query completion terms in the knowledge base using a term frequency matching algorithm. For example, in one embodiment, if the query completion term is "who is Michael Jackson? ", the process 32_1500 may match the terms" Michael, "" Jackson, "or" Michael Jackson. In this example, the process 32_1500 will attempt to find the longest match in the knowledge database. If the knowledge base has matches for "Michael", "Jackson" and "Michael Jackson", then the match for "Michael Jackson" will be used. If there are matches for one or more query completion terms, the process 32_1500 may return the match term(s) at block 32_ 1508. For example, in one embodiment, the "who is Michael Jackson? ", the process 32_150 may return" people "," artist ", or another type of characterization. If there is no match, the return of the process 32_1500 is without any characterization.

FIG. 32_16 is a block diagram of one embodiment of a completion term module 32_1600 for determining query completion terms from multiple search domains. In one embodiment, the completion items module 32_1600 includes a receive query prefix module 32_1602, a send prefix module 32_1604, a receive completion items module 32_1606, a rank and filter completion items module 32_1608, and a send completion items module 32_ 1610. In one embodiment, the receive query prefix module 32_1602 receives the query prefix as described above at block 32_602 in fig. 32_ 6. The transmit prefix module 32_1604 transmits the query prefix to a different search domain as described above at block 32_604 in fig. 32_ 6. The receive completion items module 32_1606 receives the query completion items, as described above at block 32_606 in FIG. 32_ 6. The rank and filter completion items module 32_1608 ranks and filters the received query completion items as described above at block 32_608 in FIG. 32_ 6. The send completion entry module 32_1610 sends the query completion entry to the relevant results module, as described above at block 32_610 in FIG. 32_ 6.

FIG. 32_17 is a block diagram of one embodiment of a results module 32_1700 that determines relevant results in multiple search domains from determined query completion terms. In one embodiment, the results module 32_1700 includes a receive query completion items module 32_1702, a send completion items module 32_1704, a receive query results module 32_1706, a ranking and filter module 32_1708, and a return results module 32_ 1710. In one embodiment, receive query completion terms module 32_1702 receives query completion terms as described above at block 32_702 in FIG. 32_ 7. The send completion item module 32_1704 sends completion items to multiple search domains as described above at block 32_704 in FIG. 32_ 7. The receive query results module 32_1706 receives query results from multiple search domains as described above at block 32_706 in fig. 32_ 7. The ranking and filter module 32_1708 ranks and filters the query results as described above at block 32_708 in fig. 32_ 7. The return results module 32_1710 returns the query results as described above at block 32_710 in fig. 32_ 7.

Fig. 32_18 is a block diagram of one embodiment of a collect feedback module 32_838 used to collect user feedback during a user search session. In one embodiment, the collect feedback module 32_838 includes a detect present events module 32_1802, a record events module 32_1804, a create feedback packet module 32_1806, and a send feedback module 32_ 1808. In one embodiment, the detect initial event module 32_1802 detects an initial event to begin a user search session, as described above at block 32_1002 in fig. 32_ 10. The record events module 32_1804 records events during the user search session, as described above at block 32_1004 in fig. 32_ 10. The create feedback packet module 32_1806 creates a feedback packet as described above at block 32_1006 in fig. 32_ 10. The send feedback module 32_1808 sends the feedback packet as described above at block 32_1008 in fig. 32_ 10.

Fig. 32_19 is a block diagram of one embodiment of a process feedback module 32_840 for incorporating user feedback into a feedback index. In one embodiment, the process feedback module 32_840 includes a receive feedback packet module 32_1902, a convert feedback packet module 32_1904, and an insert feedback entry module 32_ 1906. In one embodiment, the receive feedback packet module 32_1902 receives the feedback module as described at block 32_1102 in fig. 32_ 11. The convert feedback packet module 32_1904 converts the feedback packet as described at block 32_1104 in fig. 32_ 11. The insert feedback entry module 32_1906 inserts the feedback index entry as described at block 32_1106 in fig. 32_ 11.

FIG. 32_20 is a block diagram of one embodiment of an update query results module 32_842 for updating a results cache using user feedback. In one embodiment, updating the results cache 32_842 includes a receive results set module 32_2002, a run query module 32_2004, an update results set module 32_2006, and a send update results module 32_ 2008. In one embodiment, the receive result set module 32_2002 receives the result set as described at block 32_1202 in FIG. 32_ 12. The run query module 32_2004 uses the feedback index to run the query as described at block 32_1204 in fig. 32_ 12. The update result set module 32_2006 updates the result set as described at block 32_1206 of FIG. 32_ 12. The send update results module 32_2008 sends the updated result set as described at block 32_1202 in fig. 32_ 12.

FIG. 32_21 is a block diagram of one embodiment of a correlation results module 32_2100 that determines correlation results using a vocabulary service for querying completion terms. In one embodiment, the related results module 32_2100 includes a receive completed items module 32_2102, a send completed items module 32_2104, a vocabulary completed items module 32_2106, a receive results module 32_2108, a receive vocabulary results module 32_2110, a rank results module 32_2112, and a return results module 32_ 2114. In one embodiment, the receive completion entry module 32_2102 receives the query completion entry, as described at block 32_1402 in FIG. 32_ 14. The send completion entry module 32_2104 sends the query completion entry to the plurality of search domains that receive the query completion entry, as described at block 32_1404 in FIG. 32_ 14. The vocabulary completion item module 32_2106 sends the query completion item to the vocabulary service as described at block 32_1406 in FIG. 32_ 14. The receive results module 32_2108 receives query results from multiple search domains as described at block 32_1408 in fig. 32_ 14. The receive vocabulary results module 32_2110 receives the vocabulary service representations as described at block 32_1410 in fig. 32_ 14. The rank results module 32_2112 ranks the search domain results as described at block 32_1412 in fig. 32_ 14. The return results module 32_2114 returns the ranking results as described at block 32_1414 in fig. 32_ 14.

FIG. 32_22 is a block diagram of one embodiment of a characterize query module 32_2200 to characterize query completion terms. In one embodiment, the characterize query results module 32_2200 includes a receive completed items module 32_2202, a tokenize completed items module 32_2204, a find matches module 32_2206, and a return characterization module 32_ 2208. In one embodiment, the receive completion entry module 32_2202 receives a completion entry as described above at block 32_1502 in fig. 32_ 15. The tokenize completed items module 32_2204 tokenizes completed items as described above at block 32_1504 in FIG. 32_ 15. The find matching module 32_2206 finds a matching entry in the knowledge base for the tokenized finish entry, as described above at block 32_1506 in fig. 32_ 15. The return characterization module 32_2208 returns characterization as described above at block 32_1508 in fig. 32_ 15.

In some embodiments, apparatus 100 (described above with reference to fig. 1A) is used to implement the techniques described in this section.

Exemplary apparatus, methods, and computer-readable media for search techniques

In one aspect, methods and apparatus to perform multi-domain query searches are described herein. In an exemplary embodiment, a device receives a query prefix from a client of a user. The device also determines a plurality of search completion terms across a plurality of independent search domains. In addition, the device ranks a plurality of search completion terms based on the scores calculated for each of the plurality of search completion terms determined by the corresponding search domain, wherein at least one of the plurality of search completion terms is used to generate a plurality of search results in response to receiving the query prefix without an indication from the user.

In some embodiments, a non-transitory machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method of generating a plurality of ranked completion items using query prefixes over a plurality of independent search domains, the method comprising: receiving a query prefix from a client of a user; determining a plurality of search completion terms across a plurality of independent search domains; and ranking the plurality of search completion terms based on the score calculated for each of the plurality of search completion terms determined by the corresponding search domain, wherein at least one of the plurality of search completion terms is used to generate a plurality of search results in response to receiving the query prefix without an indication from the user.

In some embodiments, the method comprises: and screening the plurality of search completion items. In some embodiments, each of the plurality of independent search domains is selected from a map search domain, a media store search domain, an online encyclopedia search domain, and a site search domain. In some embodiments, the score for one of the plurality of search completion terms is the original score for that search completion term, which is the frequency of times that the search completion term has been received. In some embodiments, the score for one of the plurality of search completion terms is a local score for the search completion term that is based on the search completion term original score and a number of other possible search completion terms that use the search completion term as a prefix. In some embodiments, the score for one of the plurality of search completion terms is a global score for the search completion term that is based on the search completion term original score and a number of other possible search completion terms in the search domain. In some embodiments, the query prefix includes an input string and context, and the input string is input by the user. In some embodiments, the context includes location, device type, application identifier, and zone settings.

In some embodiments, a method of generating a plurality of ranked completion terms using query prefixes over a plurality of independent search domains is provided, the method comprising: receiving a query prefix from a client of a user; determining a plurality of search completion terms across a plurality of independent search domains; and ranking the plurality of search completion terms based on the score calculated for each of the plurality of search completion terms determined by the corresponding search domain, wherein at least one of the plurality of search completion terms is used to generate a plurality of search results in response to receiving the query prefix without an indication from the user. In some embodiments, the method comprises: and screening the plurality of search completion items. In some embodiments, each of the plurality of independent search domains is selected from a map search domain, a media store search domain, an online encyclopedia search domain, and a site search domain. In some embodiments, the score for one of the plurality of search completion terms is the original score for that search completion term, which is the frequency of times that the search completion term has been received. In some embodiments, the score for one of the plurality of search completion terms is a local score for the search completion term that is based on the search completion term original score and a number of other possible search completion terms that use the search completion term as a prefix. In some embodiments, the score for one of the plurality of search completion terms is a global score for the search completion term that is based on the search completion term original score and a number of other possible search completion terms in the search domain. In some embodiments, the query prefix includes an input string and a context, and the input string is input by the user. In some embodiments, the context includes location, device type, application identifier, and zone settings.

In some embodiments, there is provided an apparatus for generating a plurality of ranked completion items using query prefixes over a plurality of independent search domains, the apparatus comprising: a processor; a memory coupled to the processor by a bus; and a process executed by the processor from the memory, the process causing the processor to receive a query prefix from a client of a user, determine a plurality of search completion terms across a plurality of independent search domains, and rank the plurality of search completion terms based on a score calculated for each of the plurality of search completion terms determined by the corresponding search domain, wherein at least one of the plurality of search completion terms is used to generate a plurality of search results in response to receiving the query prefix without an indication from the user. In some embodiments, the process further causes the processor to filter the plurality of search completion terms. In some embodiments, each of the plurality of independent search domains is selected from a map search domain, a media store search domain, an online encyclopedia search domain, and a site search domain. In some embodiments, the score for one of the plurality of search completion terms is an original score for that search completion term that is the frequency of times that the search completion term has been received.

In another aspect, a method and apparatus are provided for generating a results cache using feedback from a user search session. In this embodiment, the device receives a feedback package from the client, where the feedback package characterizes user interactions with a plurality of query results in a search session presented to the user in response to a query prefix entered by the user. The device also generates a plurality of results for the plurality of queries by running the plurality of queries using a search feedback index to obtain a plurality of results. Additionally, the device creates a results cache from the plurality of results, wherein the results cache maps the plurality of results to the plurality of queries and is operable to provide the query results to the client.

In some embodiments, a non-transitory machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method of generating a result cache using feedback from a search session, the method comprising: receiving a feedback package from the client, wherein the feedback package characterizes user interactions with a plurality of query results in a search session presented to the user in response to a query prefix entered by the user; adding an entry in the search feedback index using the feedback packet; generating a plurality of results for the plurality of queries by running the plurality of queries using a search feedback index to obtain a plurality of results; and creating a result cache from the plurality of results, wherein the result cache maps the plurality of results to the plurality of queries and is operable to provide the query results to the client. In some embodiments, the feedback package includes a query prefix, the plurality of query results, and a plurality of events recorded during the user interaction. In some embodiments, the plurality of events includes a surface event, which is an event that displays results from the query prefix to the user. In some embodiments, the plurality of events includes an engagement event for one of the query results, which is an event indicating that the user has engaged in the query result. In some embodiments, the participation event for the query result is a click on a query result link. In some embodiments, the plurality of events includes a discard event for one of the query results, which is an event that instructs the user to discard the query result. In some embodiments, the results cache is a cache used by the client to return query results for query requests. In some embodiments, the feedback index entry includes a query prefix, a result for the query prefix, and a set of events for the result.

In some embodiments, a method of generating a results cache using feedback from a search session is provided, the method comprising: receiving a feedback package from the client, wherein the feedback package characterizes user interactions with a plurality of query results in a search session presented to the user in response to a query prefix entered by the user; adding an entry in the search feedback index using the feedback packet; generating a plurality of results for the plurality of queries by running the plurality of queries using a search feedback index to obtain a plurality of results; and creating a result cache from the plurality of results, wherein the result cache maps the plurality of results to the plurality of queries and is operable to provide the query results to the client. In some embodiments, the feedback package includes a query prefix, the plurality of query results, and a plurality of events recorded during the user interaction. In some embodiments, the plurality of events includes a render event, which is an event that displays results from the query prefix to the user. In some embodiments, the plurality of events includes an engagement event for one of the query results, which is an event indicating that the user has engaged in the query result. In some embodiments, the engagement event for the query result is a click on a query result link. In some embodiments, the plurality of events includes a discard event for one of the query results, which is an event that instructs the user to discard the query result. In some embodiments, the results cache is a cache used by the client to return query results for query requests. In some embodiments, the feedback index entry includes a query prefix, a result for the query prefix, and a set of events for the result.

In some embodiments, there is provided an apparatus for generating a results cache using feedback from a search session, the apparatus comprising: the apparatus comprises: a processor; a memory coupled to the processor by a bus; and a process executed by the processor from the memory, the process causing the processor to add an entry in a search feedback index using the feedback package, generate a plurality of results for the plurality of queries by running the plurality of queries using the search feedback index to obtain a plurality of results, and create a result cache from the plurality of results, wherein the result cache maps the plurality of results to the plurality of queries and is used to provide the query results to the client. In some embodiments, the feedback package includes a query prefix, the plurality of query results, and a plurality of events recorded during the user interaction. In some embodiments, the plurality of events includes a render event, which is an event that displays results from the query prefix to the user. In some embodiments, the plurality of events includes an engagement event for one of the query results, which is an event indicating that the user has engaged in the query result.

In another aspect, a method and apparatus are provided for generating a plurality of ranked query results from a query over a plurality of independent search domains. In this embodiment, the device receives a query and uses the query to determine a plurality of results across a plurality of independent search domains. The device also characterizes the query. Further, the device ranks the plurality of results based on a score calculated for each of the plurality of results determined by the respective search domain and query characterization, where the query characterization indicates a query type.

In some embodiments, a non-transitory machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method of generating a plurality of ranked query results from a query over a plurality of independent search domains, the method comprising: receiving a query; determining a plurality of results across a plurality of independent search domains using the query; characterizing the query; ranking the plurality of query results based on the score calculated for each of the plurality of results determined by the respective search domain and query characterization, wherein the query characterization indicates a query type. In some embodiments, the query type is selected from a person, a place, and a thing. In some embodiments, the method comprises: and screening the plurality of search results. In some embodiments, each of the plurality of independent search domains is selected from a map search domain, a media store search domain, an online encyclopedia search domain, and a site search domain. In some embodiments, characterizing the query comprises: symbolizing the query; and find the matching term of the symbolized query in the knowledge base. In some embodiments, finding a match comprises: the longest match is found between the symbols in the query. In some embodiments, tokenizing the query comprises: the query is split into symbols. According to some embodiments, the symbol is selected for a group consisting of words and phrases. In some embodiments, the query is a query completion term that completes from the query prefix without an indication from the user as to which query completion term to use.

In some embodiments, a method of generating a plurality of ranked query results from a query over a plurality of independent search domains is provided, the method comprising: receiving a query; determining a plurality of results across a plurality of independent search domains using the query; characterizing the query; ranking the plurality of query results based on the score calculated for each of the plurality of results determined by the respective search domain and query characterization, wherein the query characterization indicates a query type. In some embodiments, the query type is selected from a person, a place, and a thing. In some embodiments, the method comprises: and screening the plurality of search results. In some embodiments, each of the plurality of independent search domains is selected from a map search domain, a media store search domain, an online encyclopedia search domain, and a site search domain. In some embodiments, characterizing the query comprises: symbolizing the query; and find a match for the tokenized query in the knowledge base. In some embodiments, finding a match comprises: the longest match is found between the symbols in the query. In some embodiments, tokenizing the query comprises: the query is split into symbols. In some embodiments, the query is a query completion term that completes from the query prefix without an indication from the user as to which query completion term to use.

In some embodiments, there is provided an apparatus for generating a plurality of ranked query results from a query over a plurality of independent search domains, the apparatus comprising: a processor; a memory coupled to the processor by a bus; and a process, executed by a processor from memory, that causes the processor to receive a query, determine a plurality of results across a plurality of independent search domains using the query, characterize the query, rank the plurality of query results based on a score calculated for each of the plurality of results determined by a respective search domain and query characterization, wherein the query characterization indicates a query type. In some embodiments, the query type is selected from a person, a place, and a thing. In some embodiments, the process further causes the processor to filter the plurality of search results.

Section 3: multi-domain search techniques

The content "multi-domain search techniques" in this section describes, according to some embodiments, operations for conducting multi-domain searches on computing devices, and provides information that complements the disclosure provided herein. For example, some portions of this section describe utilizing local learning on a computer device to improve search results obtained from one or more domains, which complements the disclosures provided herein, such as those related to FIGS. 4A-4B, FIG. 5, and those related to identifying and using patterns of user behavior. In some embodiments, the specific content in this section is used to help improve the search results presented in the search interface (e.g., as discussed above with reference to methods 600,800,1000 and 1200).

Introduction to Multi-Domain search technology

Embodiments of the following are described: search results returned to a user from a local database of private information, and results returned from one or more search domains, are improved with query and result features learned locally on the user's computing device. In one embodiment, one or more search domains may inform a computing device of one or more features related to a search query to which the computing device may apply local learning.

In one embodiment, the computing device may use information obtained from the computing device to learn one or more characteristics relevant to the search query. The information obtained from the computing device and the information obtained by the computing device may be used locally on the computing device to train a machine learning algorithm to learn features related to the search query, or features related to results returned from the search query. This functionality may be sent to a remote search engine to return more relevant personalized results for a query without violating the privacy of the device user. In one embodiment, this feature is used to expand queries. In one embodiment, this feature is used to bias queries. This function may also be used to filter the results returned from a search query. The results returned from the query may be local results, remote search engine results, or both.

In one example, a user of a computing device may subscribe to news or RSS feeds that push daily information about sports scores to the computing device. The only information about the subscribing user that is known by the news or RSS feed is that the user is interested in the sports scores. The user may query the information for "football scores" received by the computing device from the RSS feeds using a local query interface on the computing device.

For us users, soccer refers to american football, for example, football kicked by dallas cowboy. For european or south american users, soccer is often referred to as soccer in america. Thus, the distinction of "soccer" from "american football" in the context of the query term "football" may be a feature that is relevant to a search query to be trained by a computing device. If a user of a computing device interacts with local results of a soccer score, a local predictor of a news or RSS feed may know that when the user of the device queries for a soccer score, the user means a soccer score.

In one embodiment, the remote search engine may learn the features "soccer and soccer". However, while the remote search engine may learn that there is a clear distinction between american football and soccer, the remote search engine does not know whether a particular user's query for soccer scores is interested in american football results or soccer results. Once the remote search engine learns of this difference, the next time the remote search service receives a query for football scores, the remote search engine may return American football scores and soccer scores and also send a feature to the query computing device to train on that feature so that the computing device may learn whether a particular user of the computing device is interested in American football scores or in soccer scores.

In one embodiment, after the local client learns of the feature using the computing device private information, the computing device may send a bias to the feature to the remote search service along with the query when the user of the computing device next queries the remote search service for football scores. For example, the bias may indicate whether this particular user is interested in football or soccer.

In one embodiment, the computing device may learn the feature using one of the following statistical analysis methods: linear regression, bayesian classification, or naive bayesian classification.

Some embodiments include one or more application programming interfaces (AP1) in the environment, where the calling program code interacts with other program code that is called through the one or more interfaces. Various function calls, messages, or other types of calls may also include various parameters, which may be transmitted via an APl between the caller and the called code. In addition, the API may provide the caller code with the ability to use the data types or classes defined in the API and implemented in the callee code.

At least some embodiments include such environments: wherein the calling software component interacts with the called software component through the APL. A method for operating through an API in this environment includes passing one or more function calls, messages, other types of calls, or parameters via the APL

Other features and advantages will be apparent from the accompanying drawings and from the detailed description.

Detailed description of multi-domain search techniques

In the following detailed description of the embodiments, reference is made to the accompanying drawings in which like references indicate similar elements, and in which is shown by way of illustration specific embodiments that may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, mechanical, electrical, functional and other modifications may be made without departing from the scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

Embodiments of the following are described: query and result features are learned using locally available information on a computing device that improves local and remote search results for a computing device user without disclosing private information about the user to a remote search engine.

FIG. 33_1 shows a block diagram of a local search subsystem 33_130 and a remote search subsystem 33_135 on a computing device 33_100 as is known in the art. The local search subsystem 33_130 may include a local search interface 33_110 and a local database of searchable information 33_111 in communication with each other.

The local database 33_111 indexes local information on the computing device 33_100 for searching using the local search interface 33_ 110. The local information is private to computing device 33_100 and is not shared with remote search subsystem 33_ 135. The local information may include data, metadata, and other information about the applications 33_112 and data 33_113 on the computing device 33_ 100.

Local database 33_111, applications 33_112, and data 33_113 are not accessible to remote search subsystem 33_ 135. Queries entered into the local search interface 33_110, local results returned from the local queries, and user interactions with local results returned from the local queries are not shared with, nor accessible by, the remote search subsystem 33_ 135.

The local search interface 33_110 may communicate with the local database 33_111 via the communication interface 33_ 1. The local database may communicate with applications 33_112 and data 33_113 via communication interface 33_ 3.

The remote search subsystem 33_135 may include a remote search interface 33_120 and a remote query service 33_ 121. The remote query service 33_121 may send queries to and return results from the remote search engine 33_150 via the web service 33_122 and the network 33_ 140. The remote results are not available to the local search subsystem 33_ 130.

The remote search interface 33_120 may communicate with the remote query service 33_121 via interface 2. The remote query service 33_121 may communicate with the web service 33_122 via the interface 4.

FIG. 33_2 illustrates, in block diagram form, a local search subsystem 33_130 having a local learning system 33_116 that can be used to improve search results returned from local searches and searches of a remote search engine 33_150 without exposing private information. In one embodiment, the local learning system 33_116 may be reset so that learning is refreshed.

The local search subsystem 33_130 may include a local search interface 33_110 and a local database 33_111 for data and metadata about applications 33_112 and data 33_113 on the computing device 33_ 100. Local databases 33_111 may include local information about data sources, such as a contacts database stored on a client, a document title or word in a document stored on a computing device, an application title and data associated with an application on a computing device and metadata, such as email, instant messages, spreadsheets, presentations, databases, music files, pictures, movies and other data local to a computing device. In one embodiment, local database 33_111 may include information about data sources stored in the user's cloud storage. The applications 33_112 may include calculator programs, dictionaries, instant messaging programs, email applications, calendars, telephones, cameras, word processors, spreadsheet applications, presentation applications, contact management applications, mapping applications, music, video or media players, local and remote search applications, and other software applications.

A query may be generated using the local search interface 33_110 and query results may be returned from the local database 33_111 via the communication interface 33_1 and displayed in the local search interface 33_ 110. The local search subsystem 33_130 may additionally have a local query service 33_114, a local search and feedback history 33_115, and a local learning system 33_ 116. The local query service 33_114 may receive a query from the local search interface 33_ 110. In one embodiment, the local search interface 33_110 may also communicate the query to the remote query server 33_121 via the communication interface 33_7, such that the local search interface 33_110 receives search results from the local database 33_111 and the remote search engine 33_ 150. The local query service 33_114 may remove redundant spaces, remove high frequency low relevance query terms such as "the" and "a," and package the query into a form usable by the local database 33_ 111. The remote query service 33_121 may perform similar functions for the remote search engine 33_ 150. In one embodiment, the local search interface 33_110 may pass the query to the remote query service 33_121 via the communication interface 33_7 to obtain the query results from the remote search engine 33_ 150. In one embodiment, the remote query service 33_121 can receive query features learned by the local learning system 33_116 via the communication interface 33_ 8. This feature may be used to expand the query and/or bias the query features toward the remote search engine 33_ 150. In one embodiment, the remote query service 33_121 may pass query features returned from the remote search engine 33_150 to the local learning system 33_116 via the communication interface 33_8 to train the features.

The local search and feedback history 33_115 may store a history of all search queries issued using the local query interface 33_110, including queries sent to the remote query service 33_121 via the communication interface 33_ 7. The local search and feedback history 33_115 may also store user feedback associated with local and remote results returned from queries. The feedback may include indications of: whether the user is participating in the result, for example, by clicking on the result; the time taken for the user to view the results; whether the result is the first result the user interacted with or other ordinal value; whether the result is the only result with which the user interacted; and whether the user has not interacted with the result, discarding the result. User feedback may be encoded and stored with the query that generated the results for which feedback was obtained. In one embodiment, the local search and feedback history 33_115 may store references to one or more results returned by the query. The information stored in the local search and feedback history 33_115 is considered private user information and cannot be used or accessed by the remote search subsystem 33_ 135. In one embodiment, the local search and feedback history 33_115 may be refreshed. In one embodiment, the local search and feedback history 33_115 may be eliminated. The elimination time can be analyzed so that stable long-term trends are maintained for a longer time than search and feedback histories that do not show stable trends.

The local learning system 33_116 may analyze the local search and feedback history 33_115 to identify features that the local learning system 33_116 may train. Once the features are identified, the local learning system 33_116 may generate a local predictor to train the features. In one embodiment, a predictor is an instance of a software component that operates on one or more data. In one embodiment, the local predictor may be trained using statistical classification methods, such as regression, bayes, or na iotave bayes. In one embodiment, the predictor may be specific to a particular category of results. The categories are discussed in more detail below with respect to operation 33_420 of FIG. 33_ 4: the results are mixed, ranked, and presented on the local device for those.

The computing device 33_100 may also includeAnd a remote search subsystem 33_135 including a remote search interface 33_120 and a remote query service 33_ 121. The remote search interface 33_120 may include a Web browser, such as

Or

The query service 33_121 may perform intermediate processing on the query before passing the query to the web service 33_122 via the network 33_140 and then to the remote search engine 33_ 150. The web service 33_122 pass may receive results from the remote search engine 33_150 for display on the remote query interface 33_120 or on the local search interface 33_ 110. The remote query service 33_121 may be communicatively coupled to the web service 33_122 via the communication interface 33_ 4.

The network 33_140 may include the internet, an 802.11 wired or wireless network, a cellular network, a local area network, or any combination of these networks.

The interfaces 33_1-33_8 may be implemented using interprocess communication, shared memory, sockets, or Application Programming Interfaces (APIs). The API is described in more detail below with reference to fig. 33_ 7.

FIG. 33_3 illustrates, in block diagram form, a method 33_300 for learning query and result features locally using local search queries, local search results, and local feedback and search history based on local search results 33_ 115.

In operation 33_305, the user may issue a query using the local query interface 33_ 110.

In operation 33_310, the local query may be stored in the local search history and feedback history 33_ 115.

In operation 33_315, the local results may be returned from the local database 33_111 to the local search interface 33_110 for display to the user. The local database 33_111 indexes data and metadata 33_112 generated or processed by one or more applications 33_112, such as documents, images, music, audio, video, calculator results, contacts, queries, filenames, file metadata, and other data generated by the applications 33_112 or associated with the data 33_ 113. In one embodiment, the local database may not return any local results to the query for one or more applications 33_ 112. For example, if a query of "ham" is entered into the local search interface 33_110 in operation 33_305, the local database 33_111 may return results from the dictionary application 33_112, from the document 33_113 containing the word "ham", from a contact having the word "ham" (such as "Cunningham"), but not the calculator application 33_112 because the calculator application has no data or metadata 33_113 related to "ham". However, if a query for "Pi" is entered in the local search interface 33_110 in operation 33_305, the local database 33_111 may return results related to the calculator application 33_112, such as "3.141592654," Greek notation "π," or a formula that utilizes the value of Pi, such as a circle length or area formula for a circle, or a volume formula for a sphere or cylinder. Similarly, if a query for "Lake Tahoe pictures" is entered in the local search interface 33_110 in operation 33_305, the local database 33_111 may return the results of a picture of taho Lake, which may be generated by the camera application 33_112, downloaded from the email application 33_112 and/or the document 33_113 containing the taho Lake picture generated by the word processing application 33_ 112. In one embodiment, the local results may be sorted for display according to the application 33_112 that obtained or generated the local results. For example, the taihao lake pictures downloaded from the email application 33_112 may be sorted together for display, the taihao lake pictures generated by the camera application 33_112 may be sorted together for display, and the taihao lake pictures incorporated into one or more documents generated by the word processing application 33_112 may be sorted together for display.

In operation 33_320, the user may interact with one or more of the displayed local results. The cases of interaction with results, or of non-interaction with results, may be stored as feedback to local results in the local search and feedback history 33_ 115.

In operation 33_325, the local learning system 33_116 may analyze the local search and local feedback history 33_115 to determine one or more features relevant to the query.

In operation 33_330, if the local learning system 33_116 has identified a new feature, in operation 33_335, a new local predictor may be generated for the feature, and the local learning system 33_116 may train on the identified feature.

In operation 33_340, the next time a query is issued (to which a feature is relevant), the feature may be used to perform one or more of the following: expanding the query, biasing the terms of the query, or screening the results returned from the query.

Fig. 33_4 illustrates, in block diagram form, a method 33_400 of locally learning query features using search results returned from local search queries and remote search queries and local feedback of local search queries and remote search query results.

In operation 33_405, the user issues a query using the local search interface 33_ 110. As described above, the local search interface 33_110 may communicate queries to one or both of the local database 33_111 and the remote search engine 33_150 via the local query service 33_114 or the remote query service 33_121, respectively.

In operation 33_410, the query may be stored in the local search history and feedback history 33_ 115.

Local results from the local database 33_111 and remote results from the remote search engine 33_150, respectively, may be returned simultaneously or asynchronously, as shown in operations 33_315 and 33_ 415. In one embodiment, timer 33_417 may be set to determine when to display results that have been received until the timer expires. In one embodiment, additional results may be received after the timer expires. The time value may be configured locally on the computing device 33_100 and/or on the remote search engine 33_150 such that the local and remote search results are displayed at different times.

In operation 33_420, the local search results and remote results may be blended and ranked and then presented to the user on the local search interface 33_ 110. In one embodiment, if the local learning system 33_116 determines that the calculator results are highly correlated, it is ranked to the top. If a user issues a query from a calculator application and the query "looks" like an operation or unit conversion, the calculator results may be highly relevant. In one embodiment, the local results 33_315 that match the query may be ranked higher than the remote search engine results 33_ 415. In one embodiment, the results may be ranked and/or filtered using previously learned features. In one embodiment, the local results 33_315 may be presented in a variety of categories, such as email, contacts, iTunes, movies, tweets, text messages, documents, images, spreadsheets, and the like, and ordered within each category. For example, local results may be presented in multiple categories and ordered by first displaying the most recently created, modified, accessed, or viewed local results 33_315 in each category. In another embodiment, the categories may be ordered by context. For example, if the user issues a local query from within his music player application 33_112, the results returned by the local database 33_111 associated with the music player application 33_112 may be sorted and displayed before other local results. In another embodiment, the categories may be ordered by how often the user interacts with the results of a certain category. For example, if a user rarely interacts with email results, the email results may be sorted and displayed lower than other local results. In one embodiment, the display order of the local categories is fixed. This may facilitate user recognition since the local result categories change infrequently. In another embodiment, the categories may be displayed according to a relevance ranking order, and the results within each category may be displayed by the relevance ranking order.

In one embodiment, the results 33_415 returned from the remote search engine may include a score based on at least one of: whether the query term is equal to the result title, whether the query term is encompassed within the result body, or a score based on a term frequency versus document frequency relationship for one or more query terms. In addition, the remote search engine search results 33_415 may have a query dependency engagement score that indicates whether other users that have issued the query have engaged in the results, indicating that the user found the results to be relevant to the query. The results may also have a query independence engagement score that indicates whether other users engaged in the results, meaning that other users find the results relevant regardless of which query was used to retrieve the results. The results may also have a "top-level" score, indicating that there are many users who find the results to be relevant, which should be ranked at the top of the result set. In one embodiment, the local learning system 33_116 may generate, for each outcome, a probability that the computing device 33_110 user is likely to also find that the outcome is relevant.

In operation 33_425, the local search interface may receive feedback from the user indicating whether the user has participated in the result (if so, how long the user participated in the result), or whether the user has abandoned the result. Regardless of whether the results are local database results or remote search engine results, user feedback may be collected and stored in the local search and feedback history 33_ 115. The query may also be stored in the local search and feedback history 33_ 115. In one embodiment, the query and feedback history may be associated with a particular user of the computing device 33_ 100. In one embodiment, the queries, feedback history 33_115, and associations with a particular user may be used by the local learning 33_116 to generate a social graph for the particular user.

For example, assume that a particular user Bob issues one or more queries for "Bill" and "Steven" to the local device and the remote search engine in operation 33_ 405. Local results 33_315 may be received from, for example, the contacts application 33_112, and may be for people named Bill and Steven, for example

The material returns remote results 33_415 and other remote results 33_ 415. After the results are mixed, ranked, and presented to the user Bob in operation 420, the search query and feedback history 33_115 of interactions that Bob has taken place with the local results 33_315, the remote results 33_415, or both, may be stored in operation 33_ 425. From this stored search history and feedback 33_115, a social graph may be generated by the local learning system 33_116 from Bob's interactions with the local results 33_315, the remote results 33_415, or both.

In one embodiment, local learning of remote results may also be used to filter out results that have been repeatedly presented to the user but with which the user has not interacted. For example, in operation 33_405, the user may issue a query for the current political topic to the local device and the remote search engine 33_ 150. Remote results 33_415 returned in response to the query may include results from "

(Herfenton post) "and"

(Fox News) ". In operation 33_425, the learning system 33_116 may learn from the locally stored feedback that the user rarely or never matched

Results any/all of the results of the interaction occur. The learning system 33_116 may determine a new function to train on, i.e., "news source," and learn to exclude future remote results when mixing, ranking, and displaying the results on the local device in operations 33_420

And (6) obtaining a result.

In operation 33_430, only the feedback history of the remote search engine results may be returned to the remote search engine 33_ 150. The feedback history may be anonymized so that no particular user and/or machine is identified in the information sent to the remote search engine 33_ 150. In one embodiment, queries associated with anonymous feedback are not sent to a remote search engine, thereby protecting user privacy.

In operation 33_435, the local learning system 33_116 can analyze the local search and feedback history 33_115 to determine if features can be identified from the results and feedback to the results. The local learning system 33_116 may utilize feedback of all results for local and remote queries to determine whether features can be identified.

If a feature is identified in operation 33_435, the local learning system 33_116 may generate a local predictor for the feature and train on the feature in operation 33_ 440.

In operation 33_445, the local learning system 33_116 may optionally send a feature vector to the remote search engine based on the features identified by the local learning system 33_ 116. Again using the example of a news source, in operation 33_405, the user may query the local device and the remote search engine 33_150 for the current political topic. The remote results 33_415 returned in response to the query may include results from

And

the result of (1). The remote search engine 33_150 may be configured to perform the search based on interactions made by a number of users of the remote search engine 33_150

The result of (2) is returned as the highest ranked result. However, the local feedback record for the particular user may indicate that the particular user is not in contact with

The results interact, which is derived by the remote search engine 33_150

The opposite is true for the highest ranking result. The local learning system 33_116 may identify as a feature in operation 33_435 even though the remote search engine will be

The result is ranked at the highest level, and the user is also not affiliated with

The results interact and the local learning system may perform local learning on the feature in operation 33_440 and optionally send the feature back to the remote search engine 33_150 in operation 33_ 445.

FIG. 33_5 illustrates, in block diagram form, a method 33_500 for locally learning the characteristics of a query passed by the remote search engine 33_150 to the computing device 33_100 in response to the query sent by the computing device 33_100 to the remote search engine 33_ 150. Many of the operations of method 33_500 have been described above.

In operation 33_405, the user may issue a query using the local search interface 33_ 110. As described above, the local search interface 33_110 may pass the query to one or both of the local database 33_111 and the remote search engine 33_ 150.

In operation 33_310, the local query may be stored in the local search history and feedback history 33_ 115.

In operation 33_315, the computing device 33_100 may receive local results returned from the local database 33_111 in response to the query. The local results may be received asynchronously independent of the search results returned by the remote search engine 33_ 150.

In operation 33_515, the computing device 33_100 may receive results returned from the remote search engine 33_150 in response to the query. In operation 33_515, the remote search engine may also return features related to the query and results for training by the local learning system 33_ 116.

In one embodiment, timer 33_417 may be set to determine when to display results that have been received until the timer expires. In one embodiment, additional results may be received after the timer expires. The time value of the timer may be configured locally on the computing device 33_100 and/or on the remote search engine 33_150 such that the local and remote search results are displayed at different times.

In operation 33_420, the local results and remote results may be mixed and ranked as described above in operation 33_420 of fig. 33_ 4.

In operation 33_425, the local search interface may receive feedback from the user indicating whether the user has participated in the result (if so, how long the user participated in the result), or whether the user has abandoned the result. Regardless of whether the results are local database results or remote search engine results, user feedback may be collected and stored in the local search and feedback history 33_ 115. The query may also be stored in the local search and feedback history 33_ 115. In one embodiment, the query and feedback history may be associated with a particular user of the computing device 33_ 100.

In operation 33_430, only the feedback history of the remote search engine results may be returned to the remote search engine 33_ 150. The feedback history may be anonymized so that no particular user and/or machine is identified in the information sent to the remote search engine 33_ 150. In one embodiment, queries associated with anonymous feedback are not sent to a remote search engine, thereby protecting user privacy.

In operation 33_520, the local learning system 33_116 may generate a local predictor for the features received from the remote search engine 33_150 in operation 33_515 and train on the features. The local learning system 33_116 may utilize the local feedback and search history 33_115 to determine how a particular user interacts with the local and remote search results for features received from the remote search engine 33_ 150. The local learning system 33_116 can track such situations: whether the features are determined by the local learning system 33_116 or whether the features are received from the remote search engine 33_150 for learning by the local learning system 33_ 116. In embodiments where the characteristic information is sent to the remote search engine 33_150 (such as operation 33_630 of fig. 33_6, below), the characteristic information may be anonymized before being sent to the remote search engine 33_150 to protect the privacy of a particular user.

FIG. 33_6 illustrates, in block diagram form, a method 33_600 of receiving or determining a new feature, training a feature locally, and utilizing a feature.

In operation 33_605, the remote search engine 33_150 may return to the computing device 33_100 new features that the computing device will train locally. The remote search engine 33_150 may return the feature to the computing device 33_100 along with the results returned by the computing device 33_100 from the query. In one embodiment, this feature may be returned to the computing device regardless of whether the query was generated from the local search interface 33_110 or the remote search interface 33_ 120. In one embodiment, the remote query server 33_121 may intercept the feature and communicate the feature to the local learning system 33_116 via the communication interface 33_ 8.

In operation 33_610, the method 33_600 may alternatively begin with the local learning system 33_116 determining the features by analyzing the local search history and the feedback history 33_ 115. The local search history and feedback history 33_115 may be analyzed in various ways to learn a certain function. Several examples are given below:

the user may issue a query for "football scores". The remote search engine 33_150 may return the results of american football scores and soccer scores. The remote search engine 33_150 may have determined that the computing device 33_100 sending the query is located at an IP address in the united states. Thus, the remote search engine prioritizes american football (such as dallas cowboy) scores as the most relevant result. In many europe and south american countries, football means soccer. Suppose that the user issuing the query is interested in, and interacts with, soccer results. The local learning system 33_116 may analyze the local search history and feedback history 33_115 to determine that the user has not interacted with the higher ranked american football score. The local learning system 33_116 may then analyze the results and determine the following characteristics: football has at least two meanings and the user of this computing device 33_100 has a greater preference for soccer than american football.

Again using the football scoring example, after receiving the results of the football scoring, the user may want to know why he would obtain an american football score. In the local result returned from the local database 33_111, a dictionary entry for the word "football" may appear. The user clicks on the "football" dictionary entry. In response, the local learning system 33_116 may determine such new features: football has an alternative definition and the user's preference for soccer is greater than american football.

As another example, assume that a user enters the query "Montana", and receives the local result "Mary Montana" from his address book, the local result from his dictionary, the remote result of Joe Montana (American football legend), and the remote result of Montana, USA. Mary Montana is clicked from his local address book almost always each time the user queries Montana. The local learning system 33_116 can determine the characteristics for Montana and that the user prefers the contact record "Mary Montana".

As another example, a user issues a query for "MG". The user has a number of photographs of the english MG car on his local computer and these photographs are indexed in the local database 33_ 111. The remote search engine 33_150 may return the result of the element "magnesium" (symbol Mg). The user may also have a number of songs by the band "book t.and the MGs" on his computer and receive local results accordingly. The local learning system 33_116 may determine the differences in these results and may determine the characteristics of the "MG".

Once a feature is received in operation 33_605 or determined in operation 33_610, the local learning system 33_116 may generate a local predictor for the feature in operation 33_ 620.

In operation 33_625, the local learning system 33_116 may use the local predictor to train the feature "MG" with the local search history and feedback history 33_ 115. The local learning system 33_116 may also use the context of the computing device 33_100 to train on a certain feature.

Using the example of MG described above, if a user issues a query for MG from within a computer program, the local learning system 33_116 may use the context to learn that the user is most likely interested in the molecular weight of magnesium or other attributes of magnesium, and train for MG accordingly. If the user issues a query from within the picture viewing application while viewing a picture of an MG vehicle, the local learning system 33_116 can use the context to know that the user is most likely interested in MG vehicles in the uk.

In operation 33_630, the features learned by the local learning system 33_116, or received from the remote search engine 33_150, may be utilized in several different ways. When a new query is issued for an MG, the query may be extended, for example, with learned MG preferences (e.g., magnesium). In one embodiment, when a new query is issued for an MG, the query may be biased toward the results of magnesium, for example. The local learning system 33_116 may calculate a bias probability (learned preference) associated with each query feature and provide the bias as a feature vector to the remote search engine 33_ 150. In one embodiment, the feature vector may be sent to the remote search engine the next time the user queries the remote search engine using the query terms associated with the features. In one embodiment, this feature may be used to filter the results returned from one or both of the local database 33_111 or the remote search engine 33_150, limiting the results returned to the query MG to, for example, magnesium results.

In fig. 33_7 ("software stack"), the exemplary embodiment, application, may use several service apis to invoke service a or B and several OS apis to invoke an Operating System (OS), a and B may use several OS apis to invoke the OS.

Note that service 2 has two APs, one of which (service 2 API 1) receives calls from and returns values to application 1, the other of which (service 2 API 2) receives calls from and returns values to application 2, service 1 (which may be, for example, a software library) makes calls to and receives return values from OS API 1, service 2 (which may be, for example, a software library) makes calls to and receives return values from API 1 and OS API 2, and application 2 calls API 2 and receives return values from it.

Exemplary systems, methods, and computer-readable media for Multi-Domain search techniques

In some embodiments, there is provided a computer-implemented method comprising: learning, on the computing device, features related to the search query, wherein the features are learned at least in part using information generated on the computing device that is not sent to a remote search engine; sending the search query and an indication of the feature to a remote search engine; and receiving, by the computing device, search results in response to the search query and the indication of the characteristic. In some embodiments, the indication of the characteristic comprises at least one of: a bias towards a feature or a feature vector. In some embodiments, the information obtained from the computing device includes at least one of: a search query of information on a computing device executing on the computing device, or feedback interacted with by a user of the computing device, the feedback having results returned from a search query of information stored on the computing device executing on the computing device. In some embodiments, learning includes performing a statistical analysis on information obtained from the computing device, wherein the statistical analysis includes one of: linear regression, bayesian (Bayes) classification, or Naive Bayes (Naive Bayes) classification. In some embodiments, the method further includes receiving, from a remote search engine, features related to a search query for the computing device to learn. In some embodiments, the method further comprises learning, on the computing device, the features received from the remote search engine, wherein the features received from the remote search engine are learned at least in part using information generated on the computing device that is not transmitted to the remote search engine. In some embodiments, learning the features includes eliminating query terms relevant to the search query based on information obtained from the computing device.

In some embodiments, a non-transitory machine readable medium is provided that when executed by a processing system performs a method comprising: learning, on a computing device, features related to a search query, wherein the features are learned at least in part using information generated on the computing device that is not sent to a remote search engine; sending a search query and an indication of the feature to a remote search engine; and receiving, by the computing device, search results in response to the search query and the indication of the feature. In some embodiments, the indication of the characteristic comprises at least one of: a bias towards a feature or a feature vector. In some embodiments, the information obtained on the computing device includes at least one of: a search query of information on a computing device executing on the computing device, or feedback interacted with by a user of the computing device, the feedback having results returned from a search query of information stored on the computing device executing on the computing device. In some embodiments, learning includes performing a statistical analysis on information obtained from the computing device, wherein the statistical analysis includes one of: linear regression, bayesian classification, or naive bayesian classification. In some embodiments, the method further includes receiving, from a remote search engine, characteristics related to a search query for the computing device to learn. In some embodiments, the method further comprises learning, on the computing device, the features received from the remote search engine, wherein the features received from the remote search engine are learned at least in part using information generated on the computing device that is not transmitted to the remote search engine. In some embodiments, learning the features includes eliminating query terms relevant to the search query based on information obtained from the computing device.

In some embodiments, there is provided a system comprising: a processing system programmed with executable instructions that, when executed by the processing system, perform a method. The method comprises the following steps: learning, on the system, features relevant to the search query, wherein the features are learned at least in part using information generated on the system that is not sent to a remote search engine; sending the search query and an indication of the feature to a remote search engine; and receiving, by the system, search results in response to the search query and the indication of the characteristic. In some embodiments, the indication of the characteristic comprises at least one of: a bias towards a feature or a feature vector. In some embodiments, the information obtained on the system includes at least one of: a search query of information on the system executing on the system, or feedback interacted with by a user of the system with results returned from a search query of information stored on the system executing on the system. In some embodiments, learning comprises performing a statistical analysis on information obtained from the system, wherein the statistical analysis comprises one of: linear regression, bayesian classification, or naive bayesian classification. In some embodiments, the method further comprises receiving, from a remote search engine, characteristics related to a search query for the system to learn. In some embodiments, the method further comprises learning, on the system, features received from the remote search engine, wherein the features received from the remote search engine are learned at least in part using information generated on the system that is not transmitted to the remote search engine. In some embodiments, learning the features includes eliminating query terms relevant to the search query based on information obtained from the system.

Section 4: structured suggestions

Content in this section "structured suggestions" describes, according to some embodiments, structuring suggestions and suggesting contacts and calendar events for a user using context-aware computing based on analysis of content (e.g., text messages) associated with the user, and providing information that complements the disclosure provided herein. For example, a portion of this section describes ways to identify and suggest new contacts, which complements the disclosure provided herein, such as those related to method 600 and method 800 discussed below, and particularly to populating suggestions in prediction section 930 of fig. 9B-9C. Further, techniques for analyzing content may also be applied to those discussed above with reference to methods 1800 and 2000, and techniques for suggesting contacts and calendar events may be used to perform these suggestions based on the analysis of the voice communication content.

Structured advice profiles

In some embodiments, a method of suggesting contacts includes: at an electronic device: receiving a message; identifying an entity and contact information associated with the entity in the received message; determining that a contact associated with the identified entity does not exist among a plurality of contacts in a database; and in response to the determination, generating a contact associated with the entity, the generated contact including contact information and an indication that the generated contact is a suggested contact.

In some embodiments, a method of suggesting contacts includes: at an electronic device: receiving a message; identifying an entity and a contact information item associated with the entity in the received message; determining that a contact associated with the identified entity exists among a plurality of contacts in a database and that the contact does not include the identified item of contact information; and in response to the determination, updating the contact to include the item of contact information and an indication that the item of contact information is a suggested item of contact information.

In some embodiments, a method of suggesting contacts includes: at an electronic device having a display: receiving a message; identifying in the received message an entity and contact information associated with the entity; generating an indication that the identified contact information is suggested contact information; and displaying a first user interface corresponding to a contact associated with the entity, the first user interface including a first user interface object that indicates, based on the generated indication, that the identified contact information is suggested contact information.

In some embodiments, a method of suggesting contacts includes: at an electronic device having a display: receiving a message; identifying in the received message an entity and contact information associated with the entity; and displaying a first user interface corresponding to the received message, the first user interface comprising: a first portion comprising content of a message received by the electronic device; and a second portion comprising: a first user interface object corresponding to the identified entity; a second user interface object corresponding to the identified contact information; and a third user interface object associated with the identified contact information, the electronic device adding the identified contact information to the database if the third user interface object is selected by the user.

In some embodiments, a method of suggesting a calendar event comprises: at an electronic device: receiving a message; identifying event information in the received message; and generating a calendar event associated with the identified event information, the generated calendar event including the event information and an indication that the generated calendar event is a suggested calendar event.

In some embodiments, a method of suggesting a calendar event comprises: at an electronic device having a display: receiving a message; identifying event information in the received message; and displaying a first user interface corresponding to the received message, the first user interface including: a first portion comprising content of a message received by the electronic device; and a second portion comprising: a first user interface object corresponding to the identified event information; and a second user interface object associated with the identified event information, the electronic device adding the identified event information to a database including a plurality of calendar events if the second user interface object is selected by a user.

In some embodiments, a method of suggesting a plurality of contacts and/or calendar events comprises: at an electronic device having a display: receiving a message; identifying a plurality of instances of contact information or event information in the received message; and displaying a first user interface corresponding to the received message, the first user interface comprising: a first portion comprising content of a message received by the electronic device; and a second portion that, if selected by the user, the electronic device displays a second user interface that includes a list of multiple instances of the identified contact or event information.

Detailed description of structured suggestions

In the following description of the present disclosure and embodiments, reference is made to the accompanying drawings, in which are shown by way of illustration specific embodiments that may be practiced. It is to be understood that other embodiments and examples may be practiced and that changes may be made without departing from the scope of the present disclosure.

As described above, managing contacts and calendar events on an electronic device can be very cumbersome for a user because adding or updating contacts and calendar events requires multiple steps to be performed manually, which in total will take a significant amount of time. Thus, many users are only reluctant to update their contacts and calendars at any time, but they still spend time searching their devices manually to find specific contact or event information at a later time. This can frustrate the user and affect efficiency.

The present disclosure addresses this problem by providing an electronic device that automatically suggests contacts and calendar events for a user based on a message. Such devices may analyze a user's messages for contact and event information and then automatically generate or update suggested contact and calendar events for the user based on the information. The suggested contacts and calendar events may be searchable as if manually entered by the user, and the user may choose to add or ignore the suggested contacts and calendar events. Therefore, the user can maintain own contacts and calendar events with little effort, the time of the user can be saved, the efficiency is improved, and a more effective human-computer interface can be generated.

1. Structured suggestions

In embodiments of the present disclosure, the electronic device may suggest contacts and calendar events for the user organization from the user's messages. The suggested contacts and calendar events may be searchable as if manually entered by the user, and the user may choose to add or ignore (e.g., reject) the suggested contacts and calendar events. Therefore, the user can maintain own contacts and calendar events with little effort, the time of the user can be saved, the efficiency is improved, and a more effective human-computer interface can be generated.

2.1 suggesting contact information

Fig. 34_5A illustrates an exemplary data architecture 34_502A for suggesting contacts, according to some embodiments. As shown in fig. 34_5A, the electronic device 34_500 can associate (e.g., store) the contact information 34_520A from the message 34_510 with the corresponding contact 34_ 530A. The message 34_510 may comprise any type of message capable of being sent or received by a user of the device 34_500, such as an email, an instant message, a message that is posted via an application on the device 34_500, etc., and may comprise any attachment to the message 34_ 510.

Contact information 34_520A may include information typically associated with a contact entry in a directory database, such as the name, phone number, address, business or social network process, etc. of an entity. Contact entries are typically organized or indexed by entity, which may include individuals, groups, organizations, companies, and the like. Contact information 34_520A may be stored in any suitable format so long as an application, such as contact module 137, can recognize when processing contact information 34_ 520A. The contact information 34_520 may also be formatted according to a standard protocol, such as the CardDAV protocol, to allow updates or synchronization with other clients via a network.

In some embodiments, the identified contact information 34_520A may be associated with the contact 34_530A in any of three mutually exclusive states-a suggested state 34_540, an added state 34_550, and a rejected state 34_ 560. The suggested state 34_540 may reflect a state in which the user has not confirmed or approved the addition of the contact information 34_520A to the contact. The added state 34_550 may reflect a state in which the user has confirmed or approved the addition of contact information 34_520A to the contact. The rejected status 34_560 can reflect a status where the user has rejected the addition of contact information 34_520A to the contact. Contact 34_530A may also be associated with any of these three states when all of the associated contact information belongs to the same state.

In some implementations, the added state 34_550 may be considered a default state by the device 34_500, meaning that no additional data need be added in association with such contacts to indicate that it is in the added state 34_ 550. For example, a contact that the user added on the device 34_500 may be considered by default to be in the added state 34_ 550.

In embodiments where the added state 34_550 is considered the default state, the device 34_500 may associate data with the contact information 34_520A to indicate that the contact information 34_520A belongs to the suggested state 34_540 or the rejected state 34_ 560. This data may take any suitable form, such as metadata may be used, and an application processing the contact information 34_520A will use the corresponding form to recognize whether the contact information 34_520A is in the suggested state 34_540 or the declined state 34_ 560.

The device 34_500 may also associate data with the contact 34_530A to indicate whether the contact 34_530A and all associated contact information belong to the suggested state 34_540 or the rejected state 34_ 560.

By storing the contact information 34_520A in the suggested state 34_540, the device 34_500 (e.g., via an application running on the device 34_ 500) may include suggested contact information in a contact search. To avoid user confusion, the device 34_500 may also indicate to the user that the contact information 34_520A is in the suggested state 34_520B by providing a visual indication (e.g., via adding a mark or highlighting) and/or preventing the user from directly manipulating the contact information 34_520A (e.g., by first asking the user to provide additional input and then allowing the user to manipulate the contact information 34_ 520A). Input may refer to any suitable input means, such as touch, mouse, voice, etc.

Storing the contact information 34_520A in the rejected state 34_560, the device 34_500 may remember that the user has rejected previously suggested contact information to avoid suggesting the information to the user again. Contact information 34_520A in the rejected state 34_560 may be ignored by applications that process contact information in the added state 34_550 and the suggested state 34_ 540.

The device 34_500 may store the contact information 34_520A locally on the device 34_500 and refrain from synchronizing the contact information 34_520A to the remote database until the contact information 34_520A is changed from the suggested state 34_540 to the added state 34_ 550. In other embodiments, the contact information 34_520A may be updated to a remote database while in the suggested state 34_ 540.

The device 34_500 may identify the contact information 34_520A from the structured content or unstructured content in the message 34_ 510. Structured content refers to content having a formal organization or structure arranged according to a predefined format, such as automated e-mail provided by an online travel agency that lists flight, hotel, and/or rental car reservation information in the same predefined manner (e.g., using the same HTML structure). In some embodiments, to identify contact information 34_520A from structured content, device 34_500 may use a template configured to recognize contact information in a particular format provided by such messages. In some embodiments, the device 34_500 may add and/or update these templates via the network.

Unstructured content refers to content that has no formal organization or structure, such as natural language content (e.g., someone says in a message that he/she has a new number) and email signatures. To identify contact information 34_520A from unstructured content, device 34_500 may use a data detector configured to identify a predefined contact information index, such as "i am enabled for a new number, which is a particular phrase such as < number >". The device 34_500 may also add and/or update these data detectors via the network. The device 34_500 may improve the predefined indexing relied upon by the data detector by correlating contact information on the device 34_500 (e.g., in an address book database) with the language on the device 34_500 associated with the contact information (e.g., in a message). The relevant language can then be used to refine the predefined index for subsequent use. The message content analyzed by the device 34_500 may include any information that the device 34_500 is capable of recognizing, including message metadata.

2.2 suggested event information

Fig. 34_5B illustrates an exemplary data architecture 34_502B for suggesting calendar events, according to some embodiments. As shown in fig. 34_5B, the electronic device 34_500 can associate (e.g., store) the event information 34_520B from the message 34_510 with the corresponding calendar event 34_ 530B. The message 34_510 may comprise any type of message capable of being sent or received by a user of the device 34_500, such as an email, an instant message, a message that is posted via an application on the device, etc., and may comprise any attachment to the message.

The event information 34_520B may include information typically associated with calendar entries in a calendar database, such as time, date, location, and the like. The event information 34_520B may be stored in any suitable format as long as an application, such as the calendar module 148, can recognize the format to process the event information 34_ 520B. Event information 34_520B may also be formatted according to standard protocols, such as the CalDAV protocol, to allow updates or synchronization with other clients via the network.

In some embodiments, the identified event information 34_520B may be associated with the calendar event 34_530B in any one of three mutually exclusive states, the suggested state 34_540, the added state 34_550, and the rejected state 34_ 560. The suggested state 34_540 may reflect a state in which the user has not confirmed or approved the addition of the event information 34_520B to the calendar event. The added state 34_550 may reflect a state in which the user has confirmed or approved the addition of the event information 34_520B to the calendar event. The rejected state 34_560 may reflect a state in which the user has rejected the addition of event information 34_520B to a calendar event. The calendar event 34_530B may also be associated with any of these three states when all of the associated calendar event information belongs to the same state.

In some embodiments, the added state 34_550 may be considered a default state by the device 34_500, meaning that no additional data need be added in association with such calendar events to indicate that it is in the added state 34_ 550. For example, a calendar event added by the user on the device 34_500 may be considered as the added state 34_550 by default.

In embodiments where the added state 34_550 is considered the default state, the device 34_500 may associate data with the event information 34_520B to indicate that the event information 34_520B belongs to the suggested state 34_540 or the rejected state 34_ 560. This data may take any suitable form, such as metadata may be employed, and the application processing the event information 34_520B will use the corresponding form to recognize whether the event information 34_520B is in the advised state 34_540 or the declined state 34_ 560.

The device 34_500 may also associate data with the calendar event 34_530B to indicate whether the calendar event 34_530B and all associated event information belongs to the suggested state 34_540 or the rejected state 34_ 560.

By storing the event information 34_520B in the suggested state 34_540, the device 34_500 (e.g., via an application running on the device 34_ 500) may include suggested event information in a calendar event search. To avoid user confusion, the device 34_500 may also indicate to the user that the event information 34_540 is in the suggested state 34_520B by providing a visual indication (e.g., via adding a mark or highlighting) and/or preventing the user from directly manipulating the event information 34_520B (e.g., by first requiring the user to provide additional input and then allowing the user to manipulate the event information 34_ 520B). Input may refer to any suitable input means, such as touch, mouse, voice, etc.

Storing the event information 34_520B in the rejected state 34_560, the device 34_500 may remember that the user has rejected previously suggested event information in order to avoid suggesting that information to the user again. Event information 34_520B in the rejected state 34_560 may be ignored by applications that handle event information in the added state 34_550 and the suggested state 34_ 540.

The device 34_500 may store the event information 34_520B locally on the device 34_500 and refrain from synchronizing the event information 34_520B to the remote database until the event information 34_520B is changed from the suggested state 34_540 to the added state 34_ 550. In other embodiments, the event information 34_520B may be updated to a remote database while in the advice state 34_ 540.

The device 34_500 may identify the event information 34_520B from the structured content or unstructured content in the message 34_ 510. Structured content refers to content having a formal organization or structure arranged according to a predefined format, such as automated e-mail provided by an online travel agency that lists flight, hotel, and/or rental car reservation information in the same predefined manner (e.g., using the same HTML structure). In some embodiments, to identify event information 34_520B from structured content, device 34_500 may use a template configured to recognize event information in a particular format provided by such messages. In some embodiments, the device 34_500 may add and/or update these templates via the network.

Unstructured content refers to content that has no formal organization or structure, such as natural language content (e.g., someone saying in a message that he/she will see you somewhere at a particular time) and email signatures. To identify event information 34_520B from unstructured content, device 34_500 may use a data detector configured to identify predefined event information references a particular phrase such as "see me at < time > at < address >. The device 34_500 may also add and/or update these data detectors via the network. The device 34_500 may improve the predefined indexing relied upon by the data detector by correlating event information on the device 34_500 (e.g., in a calendar database) with the language on the device 34_500 (e.g., in a message) associated with the event information. The relevant language can then be used to refine the predefined index for subsequent use. The message content analyzed by the device 34_500 may include any information that the device 34_500 is capable of recognizing, including message metadata.

It should be appreciated that the exemplary data architectures 34_520A and 34_520B may be the same or different. For example, the same data schema may be used for suggested contacts and suggested calendar events. Alternatively, one data architecture is used for suggesting contacts and another, different data architecture is used for suggesting calendar events.

It should also be appreciated that the message 34_510 may be processed for suggesting only contacts, suggesting only calendar events, or both suggested contacts and suggested calendar events. When processing is performed for both suggested contacts and suggested calendar events, messages 34_510 may be processed for suggested contacts and suggested calendar events in series or in parallel. For example, the message 34_510 may be processed first for suggested contacts, followed by a message for suggested calendar events. Alternatively, copies of message 34_510 and message 34_510 may be processed for suggested contacts and suggested calendar events in parallel.

3. User interface and associated process

34_ 6A-34 _13 depict embodiments of a user interface ("UI") and associated processes that may be implemented on the device 34_ 500. In some embodiments, device 34_500 corresponds to device 100 (fig. 1A).

34_6A and 34_6G illustrate exemplary user interfaces for providing suggested contacts and calendar events according to some embodiments.

In particular, fig. 34_6A illustrates the display of a user interface, such as by the contacts module 137, corresponding to contacts having suggested contact information (i.e., contact information in the suggested state 34_ 540), such as after processing a message as described above. In this embodiment, the contact is associated with a person named John applied and includes a company name ("a company"), an office telephone number ("405-. The company name and office telephone number are confirmed contact information items belonging to the added status 34_ 550. The phone number is a suggested contact information item and belongs to the suggested state 34_ 540.

The device 34_500 may provide a user interface object 34_600 (e.g., the phrase "suggest") in the user interface to indicate to the user that the cell phone number is a suggested item of contact information and has not been confirmed by the user. Any suitable user interface object may be used for this purpose, including a label, icon, or other visual indication that can indicate that the cell phone number is a suggested item of contact information. When the same contact includes a contact information item in the suggested state 34_540 and a contact information item in the added state 34_550, as in the case of fig. 34_6A, the device 34_500 may display the items in the suggested state 34_540 below all the items in the added state 34_550 or at a lower priority than all the items in the added state.

The device 34_500 may also prevent the user from calling John applesed from this initial user interface at the suggested number by directly invoking an application (e.g., the phone module 138). For example, the device 34_500 may provide text and/or fields associated with the suggested number with a visual appearance different from the visual appearance of the confirmed contact information item, such as a gray display appearance (not shown), to indicate that the user will not directly call the suggested number when selecting the number. After the user selects the proposed number, the device 34_500 may replace the current user interface with a second user interface through which the user may review and call the proposed number.

As shown in fig. 34_6B, the second user interface (labeled "check suggestions") includes a suggestion portion 34_606 in the form of a banner that includes a user interface object 34_602 (labeled "add to contact") associated with a suggestion number. The user selecting the user interface object 34_602 may cause the device 34_500 to add the suggested number to the contact in the added state 34_550 (e.g., change the state of the suggested number from the suggested state 34_540 to the added state 34_ 550). After the user selects the cell phone number or similar indication in this subsequent user interface, such as a phone icon displayed next to the cell phone number, the device 34_500 invokes an application (e.g., the phone module 138) to call John Appleseed at the proposed number. In some embodiments, if the user does not select the user interface object 34_602, but rather selects a cell phone number or similar indication, the device 34_500 will maintain the cell phone number in the suggested state 34_540 (e.g., the user's operation of calling the suggested number, would not be considered an implicit approval of the suggested number for the contact). In other embodiments, the device 34_500 may change the state of the cell phone number to the added state 34_550 after the user selects the cell phone number, even if the user has not selected the user interface object 34_602 (e.g., the user's operation of calling the proposed number may be considered an implicit approval of the proposed number for the contact).

The second user interface in fig. 34_6B also includes a user interface object 34_604 (labeled "ignore") associated with the suggested number. The user selecting the user interface object 34_604 may cause the device 34_500 to stop displaying the user interface object 34_602, which removes the option of adding a number to a contact. Upon selection of the user interface object 34_604, the device 34_500 may change the state of the proposed number from the proposed state 34_540 to the rejected state 34_ 560. While in the declined state 34_560, the device 34_500 may be configured to no longer display, nor suggest, the suggested number associated with the contact.

In addition, the second user interface in fig. 34_6B includes a message portion 34_608 (labeled "relevant email") that includes a portion of the message from which the device 34_500 identified the proposed number. Thus, in providing an interface for checking suggested contact information, the user interface of fig. 34_6B may provide the user with a message context associated with the suggested contact information. As shown in fig. 34_6B, the device 34_500 can display a limited portion of the e-mail that is associated with the portion containing the cell phone number. When the user selects the portion of the message displayed, the device 34_500 may cause a messaging application (e.g., the email client module 140) to open the entire email for the user. In some embodiments, the entire email may be displayed with the suggested contact information in a user interface corresponding to the user interface shown in FIG. 34_ 6D.

Fig. 34_6C shows a user interface displayed in response to the user selecting the "edit" user interface object in fig. 34_ 6A. In this editing user interface, the user may also call directly to the suggested number represented by user interface object 34_610, which is highlighted (i.e., in bold) to indicate that the number is in the suggested state 34_ 540. Any suitable visual indication may be used to indicate that the user interface object 34_610 is in the suggested state 34_ 540.

FIG. 34_6D illustrates a screen viewable by a user after opening a message (e.g., an email displayed by the email client module 140) on the device 34_500, where the device 34_500 has identified suggested contact information in the message. The user interface of fig. 34_6D includes a suggestion portion 34_612 and a message portion 34_ 614. The message part 34_614 includes the content of the message received by the device 34_ 500. The suggestion portion 34_612 includes a user interface object corresponding to the identified entity ("John applied"); a user interface object corresponding to the identified contact information ("405- & 123- & 6633"); and a user interface object 34_618 (labeled "add to contact") associated with the identified contact information, which if selected by the user, the device will add the suggested number to the contact in the added state 34_ 550. The suggested portion 34_612 includes a user interface object 34_620 (labeled "ignore") associated with the identified contact information, which if selected by the user, the device 34_500 changes the state of the identified contact information from the suggested state 34_540 to the rejected state 34_ 560. While in the rejected state 34_560, the device 34_500 may be configured to no longer display, and no longer suggest, this suggested contact information associated with the contact. Upon selection of the identified contact information 34_612 of the suggestion portion 34_616 above the "ignore" and "add to contact" tiles, a user interface corresponding to the contact associated with the identified entity appears on the device. For example, in this embodiment, the device 34_500 may present the contact information for "John Appleseed" in a user interface corresponding to the user interface shown in fig. 34_ 6A.

FIG. 34_6E shows a screen viewable by a user after opening a message (e.g., an email displayed by the email client module 140) on the device 34_500, where the device 34_500 has identified suggested event information in the message. The user interface of fig. 34_6E includes a suggestion portion 34_620 and a message portion 34_ 622. The message portion 34_622 includes the content of the message received by the device 34_ 500. The suggested component 34_620 includes a user interface object corresponding to the identified event information ("dinner," "certain sushi store," "day 7 3 month, friday," or "9: 50 pm"), and a user interface object 34_626 (labeled "add to calendar") associated with the identified event information, which if selected by the user, the device 34_500 will add the suggested event information to the calendar event in the added state 34_ 550. The recommendation portion 34_620 includes a user interface object 34_628 (labeled "ignore") that, if selected by the user, the device 34_500 changes the status of the identified event information from the recommendation state 34_540 to the rejected state 34_ 560. While in the rejected state 34_560, the device 34_500 may be configured to no longer display the piece of suggested event information associated with the calendar event and no longer suggest the piece of suggested event information associated with the calendar event. Upon selection of the identified event information 34_624 of the suggestion portion 34_620 located above the "ignore" and "add to calendar" tiles, a user interface (not shown) corresponding to the calendar event associated with the identified event information may appear on the device (e.g., as displayed by, for example, the contacts module 137) through which the user may select a user interface object to add the suggested event information to the calendar event in the added state 34_ 550.

FIG. 34_6F shows a screen viewable by a user after opening a message (e.g., an email displayed by the email client module 140) on the device 34_500, where the device 34_500 has identified a plurality of suggested contacts and/or calendar events in the message. The user interface of fig. 34_6F includes a presentation part 34_630 and a message part 34_ 632. The message portion 34_632 includes the content of the message received by the device 34_ 500. The suggestion portion 34_630 also includes a user selectable region, and if the user selects the user selectable region, the device 34_500 displays a subsequent user interface containing a list of multiple instances of the identified contact or event information, as shown in fig. 34_ 6G. Limiting the suggestions portion 34_630 of fig. 34_6F to a single banner, rather than incorporating all of the suggestions of fig. 34_6G into the user interface of fig. 34_6F, prevents the suggestions portion of fig. 34_6F from interfering with the user, resulting in the user not being able to easily view and read the message in the message portion.

FIG. 34_6G shows a subsequent user interface associated with the user interface of FIG. 34_6F containing a list of suggested contact information and event information identified in the message. As shown in FIG. 34_6G, the suggested information is organized by type (e.g., suggested calendar events are grouped in one group and suggested contacts are grouped in another group), and each piece of suggested information includes the above-described "ignore" and "add to contact" and "add to calendar" functions. The user interface of fig. 34_6G also includes a user interface object 34_634 ("add all"), which, if selected by the user, the device 34_500 will add each of the identified groupings of multiple instances of contact or event information (e.g., the two suggested calendar events shown in fig. 34_ 6G) to the corresponding contact or calendar event in an added state 34_ 550.

34_7A and 34_7B illustrate a flow diagram of an exemplary process for generating suggested contacts, according to some embodiments. The process may be performed on an electronic device (e.g., device 34_ 500).

The electronic device may receive (34_702) the message (e.g., email in fig. 34_6D, message portion 34_ 614) and then identify (34_704) the entity (e.g., fig. 34_6D, "John Appleseed") and the contact information associated with the entity (e.g., fig. 34_6D, "405-" 123- "6633") in the received message. The device may determine (34_722) that a contact (e.g., fig. 34_5A, contact 34_530A) associated with the identified entity is not present among a plurality of contacts in a database (e.g., storage on device 34_500, such as a contacts database), and in response to this determination (34_724), the device may generate a contact associated with the entity, the generated contact including contact information and an indication (e.g., metadata) that the generated contact is a suggested contact (e.g., in suggested state 34_ 540). Note that when the device generates the contact "John applesed" as a suggested contact, each item of contact information in the contact may be indicated as a suggested item of contact information and stored in the suggested state 34_540, or the entire contact as a whole may be indicated as a suggested contact and stored in the suggested state 34_ 540. It is also noted that any message resident on the device, such as incoming and outgoing messages, can be analyzed using the disclosed processes.

In some embodiments, the identified entity is a (34_706) name and the identified contact information is a phone number, address, business, or social network transaction.

In some implementations, the apparatus may recognize a signature block in the message to identify unstructured content in the message. For example, to identify entities and associated contact information in a message, a device may identify (34_708) a signature block of the message and analyze the identified signature block for entity and contact information. The e-mail may be included (34_710) in the message and the signature block may be the e-mail signature. The email may include (34_712) one or more previous emails in the email thread, and identifying the email signature may include analyzing the one or more previous emails in the email thread. The reference layer of the e-mail is expanded, so that the device is prevented from wrongly associating the contact information positions in different e-mails in the e-mail thread.

In some embodiments, the device may identify unstructured content in the message by searching for a defined phrase using a data detector. For example, to identify an entity and associated contact information in a message, the device may identify (34_714) one or more phrases in the message based on a set of predefined phrases, and analyze the one or more identified phrases for the entity and contact information. The device may update (34_716) the set of predefined phrases via the network so that the device can continue to use the accurate phrases. The device may also downgrade (34_718) one or more of the predefined phrases if the user requests to reject the suggested contact. In other words, if the user continues to reject suggested information identified by using particular phrases, such behavior may be deemed by the device to be an inaccurate indication of those phrases. The device may also generate (34_720) one or more predefined phrases by correlating the contact information in the database with a language on the electronic device associated with the contact information, such as a message, a calendar event, etc. In this way, the device can determine what exact language in the message containing the contact information, for example, has caused the user to create or update a contact with the contact information.

In some embodiments, the data architecture of fig. 34_5A may be used to make suggested contacts easier to search. For example, the device may receive (34_726) a request for a contact (e.g., a user request to search for a contact via an application on the device) and in response to the request for a contact, search for a suggested contact.

In some embodiments, the device may (34_728) refrain from storing the suggested contact in the remote database via the network in response to the generation of the contact. For example, if the suggested contact is in the suggested state 34_540, the device may refrain from pushing the contact to an update or synchronization service (e.g., an application on the device) for updating the contact on multiple clients via the network.

In some embodiments, the device may (34_730) receive a request to add the suggested contact to the database (e.g., fig. 34_6D, "add to contact" 34_618) and, in response to the request, store the generated contact in the database without storing an indication that the generated contact is the suggested contact (e.g., change the state of the contact from the suggested state 34_540 to the added state 34_ 550). In response to a request to add the suggested contact to the database, the device may store (34_732) the generated contact in a remote database via the network without storing an indication that the generated contact is a suggested contact, for example, by pushing the contact to an update service or a synchronization service.

In some embodiments, the device may (34_734) receive a request to reject the suggested contact (e.g., fig. 34_6D, "ignore" 34_620) and reject the suggested contact in response to the request, thereby preventing future generation of the suggested contact due to the entity and the contact information being identified in future messages. This can be done by storing rejected contacts in rejected state 34_560, allowing the device to know which have been rejected.

34_8A and 34_8B illustrate a flow diagram of an exemplary process for updating an existing contact using suggested contact information items, according to some embodiments. The process may be performed on an electronic device (e.g., device 34_ 500).

The electronic device may receive (34_802) the message (e.g., email in fig. 34_6D, message portion 34_ 614) and then identify (34_804) the entity (e.g., fig. 34_6D, "John Appleseed") and the contact information items associated with the entity (e.g., fig. 34_6D, "405-" 123- "6633") in the received message. The device may determine (34_822) that a contact (e.g., fig. 34_5A, contact 34_530A) associated with the identified entity exists among the plurality of contacts in the database and that the contact does not include the identified item of contact information. In response to this determination (34_824), the device may update the contact to include the item of contact information and an indication (e.g., metadata) that the item of contact information is a suggested item of contact information (e.g., in a suggested state 34_ 540). It is also noted that any message resident on the device, such as incoming and outgoing messages, may be analyzed using the disclosed process.

In some embodiments, the identified entity is a (34_806) name and the identified contact information item is a phone number, an address, a business, or a social networking process.

In some implementations, the device may recognize the signature in the message to identify unstructured content in the message. For example, to identify an entity and associated contact information item in a message, the device may identify (34_808) a signature block of the message and analyze the identified signature block for the entity and contact information item. The e-mail may be included (34_810) in the message and the signature block may be an e-mail signature. The email may include (34_812) one or more previous email pieces in the email thread, and identifying the email signature may include analyzing the one or more previous emails in the email thread. The reference layer of the e-mail is expanded, which helps the device avoid mistakenly associating the contact information positions in different e-mails in the e-mail thread.

In some embodiments, the device may identify unstructured content in the message by searching for a defined phrase using a data detector. For example, to identify an entity and associated contact information items in a message, the device may identify (34_814) one or more phrases in the message based on a set of predefined phrases, and analyze the one or more identified phrases for the entity and the contact information items. The device may update (34_816) the set of predefined phrases via the network so that the device can continue to use the accurate phrases. The device may also downgrade (34_818) one or more predefined phrases if the user requests to decline to suggest items of contact information. In other words, if the user continues to reject suggested information identified by using particular phrases, such behavior may be considered by the device as an indication that those phrases are inaccurate. The device may also generate (34_820) one or more predefined phrases by correlating the contact information in the database with a language on the electronic device associated with the contact information, such as a message, a calendar event, etc. In this way, the device can determine what exact language in the message containing the contact information, for example, has caused the user to create or update a contact with the contact information.

In some embodiments, the data architecture of fig. 34_5A may be used to make suggested contacts easier to search. For example, the device may receive (34_826) a request for a contact (e.g., a user request to search for a contact via an application on the device) and, in response to the request for a contact, search for a suggested item of contact information.

In some embodiments, the device may (34_828) refrain from storing the suggested item of contact information in the remote database via the network in response to the update of the contact. If the suggested contact information item is in the suggested state 34_540, the device may refrain from pushing the contact information item to an update service or synchronization service (e.g., an application on the device) for updating contacts on multiple clients via the network.

In some embodiments, the device may (34_830) receive a request to add a suggested item of contact information to the database (e.g., fig. 34_6B, "add to contact" 34_602) and, in response to the request, store the updated contact in the database without storing an indication that the item of contact information is a suggested item of contact information (e.g., changing the state of the contact information from the suggested state 34_540 to the added state 34_ 550). In response to a request to add a suggested item of contact information to the database, the device may store (34_832) the updated contact in the remote database via the network without storing an indication that the item of contact information is a suggested item of contact information, for example, by pushing the contact information to an update/synchronization service.

In some embodiments, the device may (34_834) receive a request to reject a suggested contact information item (e.g., 34_6B, "ignore" 34_604) and reject the suggested contact information item in response to the request, thereby preventing future updates to the contact with the suggested contact information item because the entity and the contact information item are identified in future messages. This may be accomplished by storing rejected contact information in the rejected state 34_560, allowing the device to know which have been rejected.

34_9A and 34_9B illustrate a flow diagram of an exemplary process for displaying contacts with suggested contact information, according to some embodiments. The process may be performed on an electronic device (e.g., device 34_500) having a display.

The electronic device may receive (34_902) the message (e.g., email in fig. 34_6D, message portion 34_ 614) and then identify (34_904) the entity (e.g., fig. 34_6D, "John Appleseed") and the contact information associated with the entity (e.g., fig. 34_6D, "405-" 123- "6633") in the received message. The device may generate (34_906) an indication (e.g., metadata) that the identified contact information is suggested contact information and display (34_908) a first user interface (e.g., fig. 34_6A) corresponding to the contact associated with the entity. The first user interface may include a first user interface object (e.g., "suggest") based on the generated indication, indicating that the identified contact information is suggested contact information.

In some embodiments, the device may prevent (34_910) input corresponding to selection of suggested contact information from invoking an application to contact the entity (e.g., fig. 34_6A, the device does not call the suggested number when the number is selected).

In some embodiments, the device may (34_912) detect an input in the first user interface corresponding to selection of suggested contact information and, in response to the detection, display a second user interface (e.g., fig. 34_6B) including a second user interface object associated with the identified contact information (e.g., fig. 34_6B, "add to contact" 34_602) that, if selected by the user, the electronic device will add the identified contact information to the database. The second user interface may (34_914) include a third user interface object (e.g., "ignore" 34_604, fig. 34_6B) associated with the identified contact information, and if the user selects the third user interface object, the electronic device may cease displaying the second user interface object. The display of the first user interface is stopped (34_916) while the second user interface is displayed. The device may cease (34_918) displaying the first user interface object in response to adding the identified contact information to the database.

In some embodiments, the second user interface may display (34_920) at least a portion of the message (e.g., fig. 34_6B, "related emails"). The device may (34_922) detect an input corresponding to selection of a displayed message and, in response to the detection, invoke an application (e.g., email client module 140) to open the message (e.g., fig. 34_ 6D). The message may (34_924) be an email and the application may be an email application.

In some embodiments, the device may detect (34_926) an input in the second user interface corresponding to selection of suggested contact information and, in response to the detection, invoke an application (e.g., the telephony module 138) to contact the entity using the identified contact information. In response to detection of an input corresponding to selection of suggested contact information in the second user interface, the device may (34_928) add the identified contact information to the database (e.g., change the state of the contact information from the suggested state 34_540 to the added state 34_ 550). The device may cease (34_918) displaying the first user interface object in response to adding the identified contact information to the database.

Fig. 34_10 illustrates a flow diagram of an exemplary process for displaying suggested contact information with a message, according to some embodiments. The process may be performed on an electronic device (e.g., device 34_500) having a display.

The electronic device may receive (34_1002) the message (e.g., the email in FIG. 34_6D, message portion 34_614) and then identify (34_1004) the entity (e.g., FIG. 34_6D, "John applied") and the contact information associated with the entity (e.g., FIG. 34_6D, "405-" 123- "6633") in the received message. The message may (34_1006) be an email. The identified entity may (34_1008) be a name and the identified contact information may be a phone number, address, business, or social network process.

The device may display (34_1010) a first user interface (e.g., fig. 34_6D) corresponding to the received message. The first user interface may include: a first portion (e.g., fig. 34_6D, message portion 34_614) that includes the content of the message received by the electronic device; and a second portion (e.g., fig. 34_6D, suggestion portion 34_612) that includes a first user interface object (e.g., fig. 34_6D, "John applied"), corresponding to the identified entity, a second user interface object (e.g., fig. 34_6D, "405-" 123- "6633"), corresponding to the identified contact information, and a third user interface object (e.g., fig. 34_6D, "add to contact," 34_618) associated with the identified contact information, the electronic device adding the identified contact information to the database (e.g., storing the contact information as a contact) if the third user interface object is selected by the user. The second portion (34_1012) can include a fourth user interface object associated with the identified contact information (e.g., 34_6D, "ignore" 34_620), which if selected by the user, the electronic device can cease displaying the third user interface object.

34_11A and 34_11B illustrate a flow diagram of an exemplary process for generating suggested calendar events, according to some embodiments. The process may be performed on an electronic device (e.g., device 34_ 500).

The electronic device may receive (34_1102) the message (e.g., E-mail in fig. 34_6E, message part 34_ 622) and identify (34_1104) event information (e.g., fig. 34_6E, "dinner," "certain sushi store," "day 7/3, friday," or "9: 50 pm"). The device may generate (34_1122) a calendar event (e.g., fig. 34_5B, calendar event 34_530B) associated with the identified event information, where the generated calendar event includes the event information and an indication (e.g., metadata) that the generated calendar event is a suggested calendar event (e.g., in suggested state 34_ 540).

In some embodiments, the identified event information is a (34_1106) date and time. In some embodiments, the device may identify structured content in the message by using a template configured to recognize event information in a particular format provided by such a message. For example, to identify event information in a message, the device may (34_1108) identify a format of content in the message, identify a template from a set of predefined templates configured to recognize event information in the message in the format of the content, and analyze the content using the identified template to obtain the event information. The message may include (34_1110) an email and the content may include the subscription (e.g., FIG. 34_ 6E). The device may update (34_1112) the predefined template set via the network so that the device can continue to use the accurate template.

In some embodiments, the device may identify unstructured content in the message by searching for event information indexes using a data detector. For example, to identify event information in a message, a device may identify (34_1114) one or more date and time references in the message based on a predefined set of references to date and time, and analyze the one or more identified date and time references for event information.

The device may update (34_1116) the predefined set of indices for the date and time via the network so that the device can continue to use the exact indices. If the user requests a rejection of the suggested calendar event, the device may downgrade (34_1118) the predefined index for one or more dates and times. In other words, if the user continues to reject suggested information identified by using a particular index for the date and time, such behavior may be deemed by the device to be an indication that those indices are inaccurate. The device may generate (34_1120) a predefined index of one or more dates and times by correlating event information in a database comprising a plurality of calendar events with a language on the electronic device associated with the event information. In this way, the device may better determine what language in the message containing the event information, for example, to cause the user to create or update a calendar event with the event information.

In some embodiments, the data architecture of fig. 34_ 5A-34 _5B may be used to make suggested calendar events easier to search. For example, the device may receive (34_1124) a request for a calendar event (e.g., a user request to search for calendar events via an application on the device) and search for suggested calendar events in response to the request for calendar events.

In some embodiments, the device may, in response to generation of the calendar event, refrain (34_1126) from storing the suggested calendar event in a remote database via the network. For example, if the suggested calendar event is in the suggested state 34_540, the device may refrain from pushing the calendar event to an update or synchronization service (e.g., an application on the device) for updating the calendar event on multiple clients via the network.

In some embodiments, the device may (34_1128) receive a request to add a suggested calendar event to a database including a plurality of calendar events (e.g., fig. 34_6E, "add to calendar" 34_626) and, in response, store the generated calendar event in the database without storing an indication that the generated calendar event is a suggested calendar event (e.g., change the state of the calendar event from suggested state 34_540 to added state 34_ 550). In response to a request to add a suggested calendar event to the database, the device may store (34_1130) the generated calendar event in a remote database via the network, without storing an indication that the generated calendar event is a suggested calendar event, for example, by pushing the calendar event to an update or synchronization service.

In some embodiments, the device may (34_1132) receive a request to reject the suggested calendar event (e.g., fig. 34_6E, "ignore" 34_628), and in response to rejecting the request, prevent future generation of the suggested calendar event because the event information was identified in a future message. This can be done by storing rejected events in rejected state 34_560, allowing the device to know which have been rejected.

Fig. 34_12 illustrates a flow diagram of an exemplary process for displaying suggested event information with a message, according to some embodiments. The process may be performed on an electronic device (e.g., device 34_500) having a display.

The electronic device may receive (34_1202) the message (e.g., E-mail in fig. 34_6E, message portion 34_622) and identify (34_1204) event information (e.g., fig. 34_6E, "dinner," "certain sushi store," "day 7/3, friday," or "9: 50 pm"). The message may be (34_1206) an email. The identified event information may (34_1208) be a date and time.

The device may display (34_1210) a first user interface (e.g., fig. 34_6E) corresponding to the received message. The first user interface may include: a first portion (e.g., fig. 34_6E, message portion 34_622) that includes the content of the message received by the electronic device; and a second portion (e.g., fig. 34_6E, suggestion portion 34_620) including a first user interface object (e.g., fig. 34_6E, "dinner," certain sushi store, "" 3 month 7 day, friday, or "9: 50 pm") corresponding to the identified event information and a second user interface object (e.g., fig. 34_6E, "add to calendar" 34_626) associated with the identified event information, the electronic device adding the identified event information to a database including a plurality of calendar events (e.g., storing the event information as a calendar event) if the second user interface object is selected by the user. The second portion (34_1212) may include a third user interface object (e.g., fig. 34_6E, "ignore" 34_628) associated with the identified event information, which if selected by the user, the electronic device may cease displaying the second user interface object.

Fig. 34_13 illustrates a flow diagram of an exemplary process for displaying multiple suggested contact information or event information with a message, according to some embodiments.

The process may be performed on an electronic device (e.g., device 34_500) having a display.

The electronic device may receive (34_1302) a message (e.g., email in fig. 34_6F, message portion 34_632) and identify (34_1304) multiple instances of contact information or event information (e.g., "2 events, 1 contact" in the accompanying itinerary, fig. 34_6F) in the received message.

The device may display (34_1306) a first user interface (e.g., fig. 34_6F) corresponding to the received message. The first user interface may include: a first portion (e.g., fig. 34_6F, message portion 34_632) that includes the content of the message received by the electronic device; and a second portion (e.g., fig. 34_6F, suggestion portion 34_ 630); if the user selects the second portion, the electronic device displays a second user interface (FIG. 34_6G) that includes a list of multiple instances of the identified contact or event information.

In some embodiments, the device may (34_1308) detect an input corresponding to selecting a second portion of the first user interface and, in response to the detection, display a second user interface that includes a list of the identified multiple instances of contact information or event information and a first user interface object for each of the identified multiple instances of contact information or event information (e.g., fig. 34_6G, "add to calendar" or "add to contact"), and if the first user interface object is selected by the user, the electronic device may add the identified information to the database (e.g., store the event information as a calendar event, or store the contact information as a contact). The second user interface may (34_1310) include a second user interface object (e.g., fig. 34_6G, "ignore") for each of the identified multiple instances of contact information or event information, and if the user selects the second user interface object, the electronic device may cease displaying the first user interface object. The second user interface (34_1312) may include a third user interface object (e.g., fig. 34_6G, "add all" 34_634) that, if selected, the electronic device may add each of the identified contact information or grouping of multiple instances of event information (e.g., multiple calendar events or contacts) to the database. The display of the first user interface is stopped (34_1314) while the second user interface is displayed.

It should be understood that the specific order of the operations in fig. 34_7A through 34_13 that has been described is exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize a variety of ways to reorder the operations described in this section. For the sake of brevity, these details are not repeated here. Further, it is noted that aspects of the processes 34_700 through 34_1300 (fig. 34_7 through 34_13) may be combined with one another.

The operations in the information processing procedure described above may be implemented by executing one or more functional blocks in an information processing apparatus (such as a general-purpose processor or a dedicated chip). Such modules, combinations of such modules, and/or combinations thereof with general hardware (e.g., as described above with respect to fig. 1A, 1B, and 3) are included within the scope of the invention.

Fig. 34_14 shows exemplary functional blocks of the electronic device 34_1400, which in some embodiments performs the functions described above. As shown in fig. 34_14, the electronic device 34_1400 includes a display unit 34_1402 configured to display a graphic object; a touch-sensitive surface unit 34_1404 configured to receive user gestures; one or more RF units 34_1406 configured to detect and communicate with external electronic devices; and a processing unit 34_1408 coupled to the display unit 34_1402, the touch-sensitive surface unit 34_1404, and the RF unit 34_ 1406.

In some embodiments, the processing unit 34_1408 is configured to support an operating system 34_1410 running one or more applications 34_ 1412. In some embodiments, the processing unit 34_1408 is configured to receive data from the RF unit 34_1406 representing an external device within range of wireless communication; displaying a graphical user interface affordance on the touch-sensitive surface unit 34_ 1404; and in response to detecting contact on the displayed affordance, launch an application on device 34_1400 that corresponds to the application being executed on the external device.

The functional blocks of the device 34_1400 are optionally implemented by hardware, software, or a combination of hardware and software that perform the principles of the various described examples. Those skilled in the art will appreciate that the functional blocks described in fig. 34_14 are optionally combined or separated into sub-blocks to implement the principles of the various described examples. Thus, the description in this section optionally supports any possible combination or separation or further definition of the functional blocks described in this section.

Fig. 34_15 shows exemplary functional blocks of another electronic device 34_1500, which in some embodiments performs the functions described above. As shown in fig. 34_14, the electronic device 34_1500 includes a display unit 34_1402 configured to display a graphic object; a touch-sensitive surface unit 34_1504 configured to receive user gestures; one or more RF units 34_1506 configured to detect and communicate with external electronic devices; and a processing unit 34_1508 coupled to the display unit 34_1502, the touch-sensitive surface unit 34_1504, and the RF unit 34_ 1506.

In some embodiments, processing unit 34_1508 is configured to support one or more of units 34_1510 through 1520 to perform the various functions described above. For example, the receiving unit 34_1510 is configured to perform one or more of the receiving functions described above (e.g., receiving a message). The identifying unit 34_1512 is configured to perform one or more of the identifying functions described above (e.g., identifying an entity and contact information associated with the entity in a received message; identifying event information in a received message; or identifying contact information or multiple instances of event information in a received message). The determination unit 34_1514 is configured to perform one or more of the above-described determination functions (e.g., determine that a contact associated with the identified entity is not present among the plurality of contacts in the database; determine that a contact associated with the identified entity is present among the plurality of contacts in the database, and that the contact does not include the identified item of contact information). The generation unit 34_1516 is configured to perform one or more of the generation steps described above (e.g., in response to determining, generating a contact associated with the entity; generating an indication that the identified contact information is suggested contact information; generating a calendar event associated with the identified event information). The update unit 34_1518 is configured to perform one or more of the update steps described above (e.g., in response to determining, to update the contact, to join the contact information item, and to indicate that the contact information item is a suggested contact information item). The display unit 34_1520 is configured to perform one or more of the display steps described above (e.g., display a first user interface corresponding to a contact associated with the entity or the received message on the display unit 34_ 1502).

The functional blocks of the device 34_1500 are optionally implemented by hardware, software, or a combination of hardware and software that perform the principles of the various described examples. Those skilled in the art will appreciate that the functional blocks described in fig. 34_15 are optionally combined or separated into sub-blocks to implement the principles of the various described examples. Thus, the description in this section optionally supports any possible combination or separation or further definition of the functional blocks described in this section.

Exemplary method, apparatus System, and computer-readable Medium for structuring suggestions

In one aspect, the present disclosure provides an electronic device that suggests contacts and calendar events for a user based on the user's messages. Such devices may analyze a user's messages for contact and event information and then automatically generate or update suggested contact and calendar events for the user based on the information. The suggested contacts and calendar events may be searchable as if manually entered by the user, and the user may choose to add or ignore the suggested contacts and calendar events.

In some implementations, a method performed in an electronic device (e.g., device 100 in fig. 1A, which is implemented according to any of the configurations shown in fig. 1E) is provided. The method comprises the following steps: receiving a message; identifying an entity and contact information associated with the entity in the received message; determining that a contact associated with the identified entity does not exist among a plurality of contacts in a database; and in response to the determination, generating a contact associated with the entity, the generated contact including contact information and an indication that the generated contact is a suggested contact.

In some implementations, the identified entity includes a name and the identified contact information includes a phone number, an address, a business, or a social networking transaction. In some implementations, the identifying includes identifying a signature block of the message, and analyzing the identified signature block for entity and contact information. In some implementations, the email includes one or more previous emails in the email thread, and the identification of the email signature includes analyzing the one or more previous emails in the email thread. In some implementations, the identifying includes identifying one or more phrases in the message based on a set of predefined phrases; and analyzing the one or more identified phrases for entity and contact information. In some implementations, the method includes: the set of predefined phrases is updated via a network. In some implementations, the method includes: if the user requests to decline to suggest a contact, one or more of the predefined phrases are demoted. In some implementations, the method includes: the contact information in the database and a language on the electronic device associated with the contact information are correlated to generate one or more of the predefined phrases. In some implementations, the method includes: receiving a request for a contact; and searching for suggested contacts in response to the request for the contact. In some implementations, the method includes: in response to the generation of the contact, refraining from storing the suggested contact in a remote database via a network. In some implementations, the method includes: receiving a request to add a suggested contact to a database; and in response to a request to add the suggested contact to the database, storing the generated contact in the database without storing an indication that the generated contact is a suggested contact. In some implementations, the method includes: in response to a request to add the suggested contact to the database, the generated contact is stored in a remote database via the network without storing an indication that the generated contact is the suggested contact. In some implementations, the method includes: receiving a request for rejecting suggested contacts; and in response to rejecting the request to suggest a contact, preventing future generation of the suggested contact because the entity and the contact information were identified in future messages.

In some implementations, a system is provided, the system comprising: means for receiving a message; means for identifying, in the received message, an entity and contact information associated with the entity; means for determining that a contact associated with the identified entity does not exist among a plurality of contacts in a database; and means for generating, in response to the determination, a contact associated with the entity, wherein the generated contact includes contact information and an indication that the generated contact is a suggested contact.

In another aspect, a method performed in an electronic device (e.g., device 100 in FIG. 1A, implemented according to any of the configurations shown in FIG. 1E) is provided. The method comprises the following steps: receiving a message; identifying an entity and an item of contact information associated with the entity in the received message; determining that a contact associated with the identified entity exists among a plurality of contacts in a database and that the contact does not include the identified item of contact information; and in response to the determination, updating the contact to include the item of contact information and an indication that the item of contact information is a suggested item of contact information.

In some implementations, the identified entity includes a name and the identified contact information item includes a phone number, an address, a business, or a social networking transaction. In some implementations, the identifying includes identifying a signature block for the message and analyzing the identified signature block for entities and contact information items. In some implementations, the message includes an email and the signature block includes an email signature. In some implementations, the email includes one or more previous emails in the email thread, and the identification of the email signature includes analyzing the one or more previous emails in the email thread. In some implementations, the identifying includes identifying one or more phrases in the message based on a predefined set of phrases; and analyzing the one or more identified phrases for entity and contact information items. In some implementations, the method includes: the set of predefined phrases is updated via a network. In some implementations, the method includes: if the user requests a rejection of suggested contact information items, one or more of the predefined phrases are ranked down. In some implementations, the method includes: the contact information in the database and a language on the electronic device associated with the contact information are correlated to generate one or more of the predefined phrases. In some implementations, the method includes: receiving a request for a contact; and in response to a request for the contact, search for suggested contact information items. In some implementations, the method includes: in response to the update of the contact, refraining from storing the suggested item of contact information in a remote database via a network. In some implementations, the method includes: receiving a request to add a suggested item of contact information to a database; and in response to a request to add the suggested item of contact information to the database, storing the updated contact in the database without storing an indication that the item of contact information is a suggested item of contact information. In some implementations, the method includes: in response to a request to add a suggested item of contact information to the database, the updated contact is stored in the remote database via the network without storing an indication that the item of contact information is a suggested item of contact information. In some implementations, the method includes: receiving a request to reject suggested contact information items; and in response to the request to decline the suggested contact information item, preventing future updates to the contact with the suggested contact information item because the entity and the contact information item were identified in future messages.

In some implementations, a system is provided, the system comprising: means for receiving a message; means for identifying, in the received message, an entity and a contact information item associated with the entity; means for determining that a contact associated with the identified entity exists among a plurality of contacts in a database and that the contact does not include the identified item of contact information; and means for updating the contact to include the item of contact information and an indication that the item of contact information is a suggested item of contact information in response to the determination.

In yet another aspect, a method performed in an electronic device (e.g., device 100 in FIG. 1A, implemented according to any of the configurations shown in FIG. 1E) is provided. The method comprises the following steps: receiving a message; identifying an entity and contact information associated with the entity in the received message; generating an indication that the identified contact information is suggested contact information; and displaying a first user interface corresponding to a contact associated with the entity, the first user interface including a first user interface object that indicates, based on the generated indication, that the identified contact information is suggested contact information. In some implementations, the method includes: input corresponding to selection of suggested contact information is prevented from invoking the application to contact the entity. In some implementations, the method includes: detecting an input in the first user interface corresponding to selecting suggested contact information; and in response to detecting an input in the first user interface corresponding to selection of suggested contact information, displaying a second user interface that includes a second user interface object associated with the identified contact information, the electronic device adding the identified contact information to the database if the second user interface object is selected by the user. In some implementations, the second user interface includes a third user interface object associated with the identified contact information, and the electronic device stops displaying the second user interface object if the third user interface object is selected. In some implementations, the first user interface is stopped from being displayed while the second user interface is displayed. In some implementations, the second user interface displays at least a portion of the message. In some implementations, the method includes: detecting an input corresponding to a selection of the displayed message; and in response to detecting an input corresponding to a selection of the displayed message, invoking an application to open the message. In some implementations, the message includes an email and the application includes an email application. In some implementations, the method includes: detecting an input in the second user interface corresponding to selecting the suggested contact information; and in response to detecting an input in the second user interface corresponding to selection of the suggested contact information, invoking an application to contact the entity using the identified contact information. In some implementations, the method includes: in response to detecting an input in the second user interface corresponding to selection of suggested contact information, the identified contact information is added to the database. In some implementations, the method includes: in response to adding the identified contact information to the database, ceasing to display the first user interface object.

In some implementations, a system is provided, the system comprising: means for receiving a message; means for identifying, in the received message, an entity and contact information associated with the entity; means for generating an indication that the identified contact information is suggested contact information; and means for displaying a first user interface corresponding to a contact associated with the entity, the first user interface including a first user interface object that indicates, based on the generated indication, that the identified contact information is suggested contact information.

In yet another aspect, a method performed in an electronic device (e.g., device 100 in FIG. 1A, implemented according to any of the configurations shown in FIG. 1E) is provided. The method comprises the following steps: receiving a message; identifying an entity and contact information associated with the entity in the received message; and displaying a first user interface corresponding to the received message, the first user interface including: a first portion comprising content of a message received by the electronic device; and a second portion comprising: a first user interface object corresponding to the identified entity; a second user interface object corresponding to the identified contact information; and a third user interface object associated with the identified contact information, the electronic device adding the identified contact information to the database if the third user interface object is selected by the user. In some implementations, the second portion includes a fourth user interface object associated with the identified contact information, and the electronic device stops displaying the third user interface object if the fourth user interface object is selected. In some implementations, the message includes an electronic mail. In some implementations, the identified entity includes a name and the identified contact information includes a phone number, an address, a business, or a social networking transaction.

In some implementations, a system is provided, the information comprising: means for receiving a message; means for identifying, in the received message, an entity and contact information associated with the entity; and means for displaying a first user interface corresponding to the received message, the first user interface comprising: a first portion comprising content of a message received by the electronic device; and a second portion comprising: a first user interface object corresponding to the identified entity; a second user interface object corresponding to the identified contact information; and a third user interface object associated with the identified contact information, the electronic device adding the identified contact information to the database if the third user interface object is selected by the user.

In yet another aspect, a method performed in an electronic device (e.g., device 100 in FIG. 1A, implemented according to any of the configurations shown in FIG. 1E) is provided. The method comprises the following steps: receiving a message; identifying event information in the received message; generating a calendar event associated with the identified event information, the generated calendar event including the event information and an indication that the generated calendar event is a suggested calendar event. In some implementations, the identified event information includes a date and a time. In some implementations, the identifying includes: identifying a format of content in the message; identifying, from a set of predefined templates, a template configured to recognize event information in a format of content in a message; and analyzing the content using the identified template for event information. In some implementations, the message includes an email and the content includes a subscription. In some implementations, the method includes: the set of predefined templates is updated via a network. In some implementations, the identifying includes identifying an index for one or more dates and times in the message based on a predefined set of indices for the dates and times; and analyzing the one or more identified date and time indices for event information. In some implementations, the method includes: a predefined set of indices for date and time is updated via the network. In some implementations, the method includes: if the user requests a rejection of the suggested calendar event, the predefined index of one or more dates and times is demoted. In some implementations, the method includes: event information in a database including a plurality of calendar events is correlated with a language associated with the event information on the electronic device to generate a predefined index of one or more dates and times. In some implementations, the method includes: receiving a request for a calendar event; and in response to a request for a calendar event, search for suggested calendar events. In some implementations, the method includes: in response to generation of the calendar event, refraining from storing the suggested calendar event in a remote database via a network. In some implementations, the method includes: receiving a request to add a suggested calendar event to a database comprising a plurality of calendar events; and in response to a request to add a suggested calendar event to the database, storing the generated calendar event in the database without storing an indication that the generated calendar event is a suggested calendar event. In some implementations, the method includes: in response to a request to add a suggested calendar event to the database, the generated calendar event is stored to a remote database via the network without storing an indication that the generated calendar event is a suggested calendar event. In some implementations, the method includes: receiving a request to decline to suggest a calendar event; and in response to rejecting the request to suggest a calendar event, preventing future generation of the suggested calendar event because the event information was identified in a future message.

In some implementations, a system is provided, the system comprising: means for receiving a message; means for identifying event information in the received message; means for generating a calendar event associated with the identified event information, wherein the generated calendar event includes the event information and an indication that the generated calendar event is a suggested calendar event.

In a further aspect, a method performed in an electronic device (e.g., device 100 in fig. 1A, which is implemented according to any of the configurations shown in fig. 1E) is provided. The method comprises the following steps: receiving a message; identifying event information in the received message; and displaying a first user interface corresponding to the received message, the first user interface including: a first portion comprising content of a message received by the electronic device; and a second portion comprising: a first user interface object corresponding to the identified event information; and a second user interface object associated with the identified event information, the electronic device adding the identified event information to a database including a plurality of calendar events if the second user interface object is selected by a user. In some implementations, the second portion includes a third user interface object associated with the identified event information, and the electronic device stops displaying the second user interface object if the third user interface object is selected. In some implementations, the message includes an email. In some implementations, the identified event information includes a date and time. In some implementations, a system is provided, the system comprising: means for receiving a message; means for identifying event information in the received message; and means for displaying a first user interface corresponding to the received message, the first user interface comprising: a first portion comprising content of a message received by the electronic device; and a second portion comprising: a first user interface object corresponding to the identified event information; and a second user interface object associated with the identified event information, the electronic device adding the identified event information to a database including a plurality of calendar events if the second user interface object is selected by the user.

In yet another aspect, a method performed in an electronic device (e.g., device 100 in FIG. 1A, implemented according to any of the configurations shown in FIG. 1E) is provided. The method comprises the following steps: receiving a message; identifying a plurality of instances of contact information or event information in the received message; and displaying a first user interface corresponding to the received message, the first user interface comprising: a first portion comprising content of a message received by the electronic device; and a second portion that, if selected, the electronic device displays a second user interface that includes a list of multiple instances of the identified contact or event information. In some implementations, the method includes: detecting an input corresponding to selection of a second portion of the first user interface; and in response to detecting an input corresponding to selection of a second portion of the first user interface, displaying a second user interface, the second user interface comprising: a list of multiple instances of the identified contact or event information; and a first user interface object for each of the identified multiple instances of contact or event information, the electronic device adding the identified information to the database if the first user interface object is selected by the user. In some implementations, the second user interface includes a second user interface object for each of the identified multiple instances of contact or event information, and the electronic device stops displaying the first user interface object if the second user interface object is selected. In some implementations, the second user interface includes a third user interface object that, if selected, the electronic device adds each of the identified groupings of the multiple instances of contact information or event information to the database. In some implementations, the display of the first user interface is stopped while the second user interface is displayed.

In some implementations, a system is provided, the information comprising: means for receiving a message; means for identifying a plurality of instances of contact information or event information in the received message; and means for displaying a first user interface corresponding to the received message, the first user interface comprising: a first portion comprising content of a message received by the electronic device; and a second portion that, if selected, the electronic device displays a second user interface that includes a list of multiple instances of the identified contact or event information.

In some implementations, an electronic device is provided, the electronic device including: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing any of the methods described above in this section. In some implementations, a computer-readable storage medium is provided that stores one or more programs, the one or more programs including instructions, which when executed by an electronic device, cause the device to perform any of the methods described in this section. In some implementations, a system is provided that includes means for performing any of the methods described in this section.

Section 5: learning decision tree segmentation of generative models of complex user patterns in the context of data sparsity

The content "learning decision tree segmentation of a generation model of a complex user pattern in a data sparsity context" in this section describes decision tree segmentation of a generation module for learning a complex user pattern in a data sparsity context according to some embodiments, and provides information that complements the disclosure provided in this section. For example, some portions of this section describe ways to suggest applications in response to events on the device, which supplement the disclosures provided in this section, such as those related to populating the affordances corresponding to applications and deep links in the prediction section 930 of FIGS. 9B-9C. In some embodiments, the predictive models described in this section are used to help identify applications that are suitable for prediction and display to a user (i.e., these predictive models are used in conjunction with methods 600,800,1000 and 1200).

Invention of learning decision tree segmentation of generative models of complex user patterns in the context of data sparsity

Embodiments may provide systems, methods, and apparatuses for suggesting one or more applications to a user of a computing device based on an event. Examples of computing devices are phones, tablets, laptops or desktop computers. Exemplary events include connecting to an accessory device and changing power states (e.g., waking from a power off or sleep state).

The predictive model may correspond to a particular event. The suggested application may be determined using one or more attributes of the computing device. For example, a particular sub-model may be generated from a subset of historical data collected about user interactions after an event occurred and when the device has one or more attributes (e.g., this user interaction of which application was selected after the event of connecting to a car of someone, which has the property of a particular time of day). The sub-model tree may be determined corresponding to different contexts of computing device attributes. Also, various criteria may be used to determine when to generate a sub-model, such as a confidence level in the sub-model that provides a correct prediction in the historical data subset, and an information gain (entropy reduction) in the historical data distribution relative to the parent model.

Other embodiments are directed to systems, portable consumer devices, and computer-readable media associated with the methods described in this section.

The nature and advantages of the embodiments of this section may be better understood by reference to the following detailed description and the accompanying drawings.

Learning detailed descriptions of decision tree segments of generative models of complex user patterns in the context of data sparsity

Embodiments may provide for suggesting a customized and personalized experience of an application to a user of a device, thereby making use of the device easier. A user may have a sufficient set of interactions with the user device after a particular event (e.g., which applications are launched in association with the event or are running in association with the event). Examples of computing devices are phones, tablets, laptops, or desktop computers. Exemplary events include connecting to an accessory device and changing power states (e.g., waking from a power off or sleep state).

Each data point in the historical data may correspond to a particular context (e.g., to one or more attributes of the device), and over time, more and more data for the particular context may be obtained. Historical data for a particular event may be used to suggest applications to a user. Embodiments may provide a personalized experience since different users will have different historical data.

To provide an accurate personalized experience, various embodiments may begin with a broad model that is simply trained to provide no application suggestions, or to suggest the same set of applications for various contexts. With enough historical data, the broad model can be segmented into submodels, e.g., decision trees as submodels, each corresponding to a different subset of the historical data. Then, when an event does occur, a particular sub-model may be selected to provide a suggested application corresponding to the current context of the device. Various criteria may be used to determine when to generate a sub-model, such as a confidence level in the sub-model that provides a correct prediction in the historical data subset, and an information gain (entropy reduction) in the historical data distribution relative to the parent model.

In some embodiments, the "confidence level" corresponds to the probability that the model can make a correct prediction (i.e., that at least one of the predicted applications is selected after an event) based on historical data. One example of a confidence level is the percentage of events that are correctly predicted. Another example uses a Cumulative Distribution Function (CDF) of probability distributions (e.g., beta distributions) generated from the number of correct predictions and the number of incorrect predictions. The CDF can be calculated by integrating the probability distribution. In various implementations, the confidence level may be an amount of increase in the CDF beyond an input value (e.g., between 0 and 1, where 1 corresponds to a correct prediction), or an input value that provides a specified CDF beyond an input value. The probability that an application is selected may need to be a threshold probability, which is an inference of models with confidence levels above a confidence threshold. The confidence level may be inversely proportional to the entropy measure, so an increase in confidence level from the parent model to the child model may correspond to a decrease in entropy.

Thus, some embodiments may decide when and how to segment the user's historical data in the context of the user's recommendations. For example, after collecting a period of user activity, embodiments may accumulate a list of possible segmentation candidates (e.g., location, day of week, etc.). Embodiments may also train the model across the data set and compute a confidence metric for the joint distribution of the data set and the model. A set of models may be trained, one for each segmented data set (i.e., subset), and then the confidence of each data model distribution measured. If the confidence of all data model distributions is acceptable, then the implementation may perform segmentation (segmentation) and then recursively examine the segmented space for additional segmentation.

In this way, some embodiments may use inference to explore the trade-off between segmentation and generalization, create more complex models for users with clearer and complex patterns, and create simple general models for users with noisier and simple patterns. Also, some embodiments may generate a probabilistic model tree based on determining a divergence distribution among the potential candidate models.

I. Event-based suggestion applications

Embodiments may suggest applications based on events that may be limited to certain predetermined events (also referred to as trigger events). For example, when a headset is plugged into the headset interface, a music application may be suggested. In some embodiments, the context information may be used in conjunction with the event to identify the application to suggest to the user. For example, when a set of headphones is plugged into a headphone jack, contextual information relating to location may be used. If the device is located in, for example, a gym, application a may be suggested when the headset is plugged into the headset jack. Alternatively, if the device is located in the home, application B may be suggested when the headset is plugged into the headset jack. Thus, applications that may be used in some context may be suggested at appropriate times, thereby enhancing the user experience.

In some embodiments, "context information" is collectively referred to as any data that may be used to define the context of a device. The context information for a given context may include one or more context data, each corresponding to a different attribute of the device. The potential attributes may belong to different categories, such as a temporal category or a location category. Context data is used as a feature of the model (or submodel), while data used to train the model may include different attributes of the same class. A particular context may correspond to a particular combination of device attributes or to only one attribute.

FIG. 35_1 is a flow diagram of a method 35_100 for suggesting applications based on detected events, according to an embodiment of the present invention. The method 35_100 may be performed by a mobile device (e.g., phone, tablet) or a non-mobile device and utilize one or more user interfaces of the device.

In some embodiments, a "user interface" corresponds to any interface through which a user interacts with a device. The user interface of an application allows a user to interact with the application. The user interface may be an interface of the application when the application is running. As another example, the user interface may be a system interface that provides a simplified set of applications for selection by the user, thereby making it easier for the user to use the applications.

At block 35_110, an event is detected. In some embodiments, it may be determined whether the event is a triggering event for a suggested application. In some implementations, the suggested applications are determined only for certain predetermined events (e.g., trigger events). In other implementations, suggested applications may be determined for a dynamic event list that may be updated based on historical user interactions that occurred with applications on the device.

In some embodiments, the trigger event may be identified as being sufficiently likely to be associated with a unique operation of the device. An event list of trigger events may be stored on the device. Such events may be default lists and maintained as part of the operating system and may or may not be configured by the user.

The trigger event may be an event caused by a user and/or an external device. For example, the triggering event can be when the accessory device is connected to the mobile device. Examples include plugging a headset into a headset jack, making a bluetooth connection, turning on the device, waking the device from sleep, reaching a particular location (e.g., a location identified as frequently visited), and so forth. In this example, each of these events may be classified as a different trigger event, or the trigger events may collectively be any accessory device connected to the mobile device. As other examples, the triggering event may be a particular interaction of the user with the device. For example, a user may move a mobile device in a manner consistent with operation, in which case the operational state of the device is a triggering event. Such an operating state (or other state) may be determined based on sensors of the device.

At block 35_120, the application associated with the event is identified. For example, when a headset is plugged into the headset interface, a music application may be identified. In some embodiments, more than one application may be identified. The predictive model may identify the associated application, where the predictive model may be selected for a particular event. The predictive model may use the context information to identify the application because, for example, different applications may be more likely to be used in different contexts. Some embodiments may only identify an application when there is a sufficient probability of being selected by the user (e.g., as determined by the user's historical interactions with the device).

The predictive model may be composed of sub-models, each for a different combination of context data. Different combinations may have different amounts of context data. These sub-models may be generated in a hierarchical tree, with the more specific combined sub-models being located lower down in the hierarchical tree. In some embodiments, a sub-model is generated only if it can predict the application more accurately than models higher up in the tree. In this way, a more accurate prediction of which application the user will select can be made. In some embodiments, the prediction model and sub-model may identify the top N applications (e.g., a fixed percentage number) selected by the user after the event when there is a particular combination of context data.

The context information may specify one or more attributes of the device for a particular context. The context may be the surrounding environment of the device (type of context) when the trigger event is received. For example, the context information may be the time of day at which the event was detected. In another example, the context information may be a particular location of the device when the event is detected. In yet another example, the context information may be a particular date when the triggering event was detected. Such contextual information may provide more meaningful information about the context of the device so that the prediction engine may accurately suggest applications that the user may use in that context. Thus, a prediction engine using context information may suggest applications to a user more accurately than if the context information was not used.

At block 35_130, an action is performed in association with the application. In one embodiment, the action may be to display a user interface for the user to select to run the application. The user interface may be provided in various ways, such as by being displayed on a screen of the device, projected onto a surface, or provided as an audio interface.

In other embodiments, an application may be run and a user interface specific to the application may be provided to the user. Either user interface may be provided in response to identifying an application, for example, on a lock screen. In other implementations, a user interface to interact with the application may be provided after the user is authenticated (e.g., by password or biometric identification), but such a user interface would be more specific than a home screen only case, such as a smaller list of suggested applications to run.

In some embodiments, a "lock screen" is a screen shown when the user is not authenticated and thus the device is locked in most use cases. Some functions may be exposed, such as a camera. In some embodiments, some functionality associated with the suggested application may be obtained if a user interface corresponding to the suggested application is exposed on the lock screen. For example, the application may be run. If the application is running from the lock screen, the functionality may be limited and the limited functionality may be extended when the user is authenticated.

In some embodiments, a "home screen" is a device screen that appears when the device is first powered on. For mobile devices, the home screen typically displays an array of icons corresponding to various applications that may be run on the device. Other screens may be accessed to browse for other applications that do not appear on the home screen.

Segmentation II

Each time a particular event occurs (e.g., plugging in a headset or powering up the device), the device can track which application is used in association with the event. In response to each occurrence of a particular event, the device may save a data point corresponding to the selected application, the action performed using the application, and the event. In various embodiments, data points may be held individually or aggregated with a count determined for the number of times a particular application was selected, where the count may include a count for a particular action. Thus, different counts are determined for different actions of the same selected application. Historical data of previous user interactions with the device may be used as input to determine the predictive model, and as input to determine whether and how many sub-models to create.

Once a particular event is detected, a predictive model corresponding to the particular event may be selected. The predictive model will be determined using historical data corresponding to a particular event as input to the training process. However, historical data may occur in many different contexts (i.e., different combinations of context information), where different applications are selected in different contexts. Thus, in general, historical data may not provide an application that will be explicitly selected when a particular event occurs.

Models such as neural networks or regression may be trained to identify a particular application for a particular context, but this may be more difficult when all of the corresponding historical data is used. Using all historical data may result in overfitting of the predictive model and reduced accuracy. Embodiments of the present invention may segment historical data into different input sets of historical data, each input set corresponding to a different context. Different submodels may be trained for different sets of historical data inputs.

Segmentation may improve the performance of the machine learning system. In one step of segmentation, the input space may be divided into two subspaces, each of which may be solved independently with a separate submodel. Such a segmentation process may increase the number of idle parameters available to the system and may improve training accuracy, but at the cost of reducing the amount of data in each model, which may reduce the accuracy of the system when it is being displayed with new data, for example, if the amount of data for a sub-model is small. An embodiment can segment the input space only if the joint distribution of data and model parameters created from the resulting subspace is more confident.

A. Different models based on different context data

When a particular event occurs, the device may be in various contexts, for example, at different locations, at different times, in different states of motion of the device (such as running, walking, driving, or standing still), or in different states of power usage (such as being turned on or off from a sleep mode). The context information may be retrieved in conjunction with the detected event, e.g., after the event is detected. Contextual information may be used to help predict which application may be used in relation to a detected event. Motion sensors such as accelerometers, gyroscopes or GPS sensors may be used to determine the different motion states.

Embodiments may use the context information in various ways. In one example, a piece of context data (e.g., corresponding to one attribute of the device) may be used as a feature of a particular sub-model to predict which recipient is most likely to be selected. For example, a specific location of the device may be provided as an input to the submodel. These features are part of the sub-model composition.

As another example, some or all of the context data of the context information may be used in the segmentation process. The input historical data may be segmented using a piece of context data such that only historical data corresponding to corresponding attributes of the piece of context data is used to determine a particular sub-model. For example, a specific location of the device is not used as an input for a submodel, but for selecting the submodel to be used, and which input data to use accordingly to generate a specific submodel.

Thus, in some embodiments, particular context data may be used to identify which sub-model to use, and other context data may be used as input to the sub-models for predicting which application a user is likely to interact with. A particular attribute (e.g., a particular location) does not correspond to a particular submodel, which may be used as a submodel for future (input) use. If the particular attribute corresponds to a particular sub-model, the usage of the attribute may become richer because the entire model is dedicated to the particular attribute.

One drawback to dedicating a submodel to a particular attribute (or combination of attributes) is that there may not be a large amount of historical data corresponding to that particular attribute. For example, a user may perform a particular event (e.g., insert a headset) only a few times at a particular location. This limited amount of data is also referred to as data sparseness. When a combination of attributes is used (e.g., a particular location at a particular time), the data may become more sparse. To overcome this drawback, embodiments may selectively determine when to generate new sub-models as part of the segmentation process.

B. Segmentation when more data is acquired

When a user first begins to use the device, there will not be any historical data predicting actions that the user may take with the application after a particular event. In the initial mode, historical data may be obtained without providing any predictions. As more historical data is obtained, it may be determined whether to segment the predictive model into sub-models. With more history data, a sub-model may be segmented into more sub-models. When limited historical data is available for user interaction with the device, no action can be taken, or a more general model can be used, for example.

Fig. 35_2 shows a segmentation process 35_200 according to an embodiment of the invention. The segmentation process 35_200 may be performed by a user device (e.g., a mobile device, such as a phone) that may maintain data privacy. In other embodiments, the segmentation process 35_200 may be performed by a server in communication with the user equipment. The segmentation process 35_200 may be performed partially over a period of time (e.g., days, months, or years), or the entire segmentation process 35_200 may be performed together and may be periodically resumed. The segmentation process 35_200 may be executed as a routine of a prediction engine.

FIG. 35_2 shows a timeline 35_230 corresponding to the collection of more data. As more data is collected, the predictive model may be segmented into several sub-models. At different points in the collected data, fragmentation may occur (e.g., fragmentation 35_ 201). As more data is acquired, another segmentation may occur. Although FIG. 35_2 shows new submodels for certain segments occurring at different points along timeline 35_230, each segment may involve a fully redo segment, which may or may not result in the same submodel being created as in the previous segment.

In this example, the event model 35_205 may correspond to a particular event (e.g., connected to a particular device, such as an automobile). The event model 35_205 may correspond to the top level of the prediction engine for that particular event. At the outset, there can only be one model for this particular event, since there is minimal historical data available. At this point, the event model 35_205 may simply track historical data for training purposes. The event model 35_205 may make predictions and compare those predictions to actual results (e.g., whether the user interacted with the predicted application within a specified time after the event was detected). If no application has a probability greater than the threshold, then no application will be executed when the particular event occurs.

In some embodiments, the event models 35_205 only use data collected for a particular device. In other embodiments, the event models 35_205 may be seeded with historical data aggregated from other users. Such historical data may allow the event model 35_205 to be able to provide some suggestions, which may then allow additional data points to be obtained. For example, it may be tracked whether the user interacts with the suggested application via the user interface, which may provide more data points than whether the user actually selected the application to provide.

As more data is collected, it may be periodically determined whether segmentation should be performed. Such a determination may be based on whether a higher accuracy may be obtained by segmentation. This accuracy may be measured as a level of probability that a prediction may be made, as described in more detail below. For example, if an application can be predicted to have a higher level of sub-model probability than event model 35_205, segmentation can be performed. One or more other criteria may also be used to determine whether a sub-model should be created as part of the segmentation process. For example, one criterion may be that a submodel must have a statistically significant amount of input history data before the submodel is applied. Since models trained on small amounts of data may not be accurate, the requirement for the amount of data may provide greater stability for the sub-models, and ultimately more accurate data.

At segment 35_201, it is determined to segment the event model 35_205 into a fitness house model 35_210 and another sub-model 35_ 240. Such segmentation may occur when a user has explicit behavior with respect to a particular context. In this embodiment, there is a definite behavior when the context is that the device is located at a gym (which may be a particular gym or any gym, which may be determined by cross-referencing the merchant location). Such cross-referencing may use an external database stored on the server. The determined behavior may be measured when the gym sub-model 35_210 can predict the correct application selected by the user with a greater probability than the event model 35_ 205.

As part of the segmentation 35_201, the input historical data is used to generate the gym sub-model 35_210 and to generate the sub-model 35_240 corresponding to all other contexts except the gym. Other sub-models 35_240 may be used to predict applications that the user may interact with when the context is not a gym.

At segment 35_202 after collecting more data, it is determined that further segments from event model 35_205 can be made to generate supermarket model 35_ 220. This determination may be made after the supermarket has obtained a sufficient number of data points so that the supermarket model 35_220 can predict with sufficient confidence. A sufficient degree of confidence may be measured with respect to the confidence obtained from the other submodels 35_ 240. Segmentation may be performed once the supermarket submodel 35_220 can predict an application with greater confidence than the other submodel 35_ 240. After the segmentation 35_202, the sub-model 35_240 will correspond to any other context than gyms and supermarkets.

At segment 35_203 after more data has been collected, it is determined that segmentation of the gym house model 35_210 can be performed. In this case, it is determined that the application can be predicted with a higher degree of confidence in the history data of the gym segmented into specific times, particularly afternoon times (e.g., 12 o 'clock-4 o' clock). Thus, when the user is at the gym in the afternoon, the afternoon gym model 35_211 may be used to predict which applications the user is likely to interact with. If the user is at the gym at other times, the gym sub-model 35_210 can be used, which is equivalent to having some other sub-model at a certain position in the tree, i.e. in the same way as the other sub-models 35_240 described exist.

At segment 35_204 after more data has been collected, it is determined that the gym house model 35_210 can be further segmented to generate the morning gym house model 35_ 212. In this case, enough historical data has been collected for the morning hours during which the application can be predicted with greater accuracy than using the more general fitness house model 35_210 (which uses only data that does not correspond to the afternoon fitness house model 35_ 211).

1. Default model

The default model may be used when the device is first obtained (e.g., brought up) by the user. The default model may be applied to a set of events (e.g., all events specified as trigger events). As described above, the default model that can be seeded is aggregated data from other users. In some embodiments, the default model may simply choose the most popular application regardless of context, for example, because there is not enough data available for any one context. After data is collected again, the default model may be discarded.

In some embodiments, the default model may have hard-coded logic that specifies the intended application to suggest and the action to perform. In this way, it can be detected how the user responds (e.g., a negative response is that the user does not select a suggested application), which can provide additional data that simply tracks a positive response from the user. Similar to such default models, predictive models can be run to compare their predictions to actual results. The predictive model may then be refined based on the actual results. When the predictive model has sufficient confidence, a switch may be made from the default model to the predictive model. Similarly, the performance of the submodels may be tracked. When a sub-model has sufficient confidence, the sub-model can be used for a given context.

2. Initial training

The predictive model (e.g., event model 35_205) may be initially trained using historical data collected so far, where the model provides no suggestions to the user. This training may be referred to as initial training. The predictive model may be updated periodically (e.g., daily) as part of a background process, which may occur when the device is charged and not in use. Training may involve optimizing the coefficients of the model to optimize the number of correct predictions and comparing them to actual results in historical data. In another example, training may include identifying the top N (e.g., a predetermined number or percentage) of applications actually selected. After training, the accuracy of the model may be measured to determine whether the model should be used to provide the suggested application (and potentially corresponding action) to the user.

Once the model obtains sufficient accuracy (e.g., the top selected application is selected with sufficiently high accuracy), the model may be implemented. For the top level model (e.g., event model 35_205), this may not happen, but may happen when a sub-model is tested for a particular context. Thus, such initial training may be similarly performed on the sub-models.

As historical information accumulates through the use of the mobile device, the predictive model may be periodically trained (i.e., updated) in view of new historical information. After training, the predictive model may suggest applications and actions more accurately based on the latest interaction patterns between the user and the mobile device. Training a predictive model may be most efficient when a large amount of historical information is recorded. Thus, training may occur at sufficiently long time intervals to allow the mobile device to detect a large number of interactions with the user. However, waiting too long between training sessions may hamper the fitness of the prediction engine. Thus, a suitable time period between training sessions may be between 15 and 20 hours, such as 18 hours.

Training the predictive model may take time and may interfere with the use of the mobile device. Thus, training may occur when the user is least likely to use the device. One way to predict that the user will not use the device is to wait for a certain period of time when the device is not being used, for example when no buttons are pressed and when the device is not moving. This may indicate that the user is in a state where the user will not interact with the handset for a short period of time, such as when the user is asleep. Any suitable duration may be used as the period to wait, such as one to three hours. In a particular embodiment, the period of waiting is two hours.

At the end of these two hours, the predictive model may be updated. However, if the user interacts with the mobile device (e.g., presses a button or the mobile device) before the end of the two hours, the two hour period countdown may be restarted. If the time period is continuously restarted before two hours of inactivity are reached, the mobile device may force training of the predictive model after some absolute time period. In one embodiment, the absolute time period may be determined as a threshold time period for which the user-friendliness of the mobile device begins to decline due to the predictive model being too old. The absolute time period may be between 10 and 15 hours, or in particular embodiments 12 hours. Thus, the maximum time between training may be between 28 hours (18+10 hours) and 33 hours (18+15 hours). In a particular embodiment, the maximum time is 30 hours (18+12 hours).

Selecting a model based on context information

The prediction model and any sub-models may be organized as a decision tree, for example, as shown in FIG. 35_ 2. The submodels of the decision tree may also be referred to as nodes. Each node of the decision tree may correspond to a different context, e.g., a different combination of context data. Context data using context information may be traversed through the decision tree to determine which sub-model to use.

A. Traversing decision trees

FIG. 35_3 illustrates a decision tree 35_300 that may be generated according to an embodiment of the present invention. The event model 35_305 corresponds to the top level model of the decision tree 35_ 300. The event model 35_305 may correspond to a particular event, e.g., as described in this section. The event model 35_305 may be selected in response to detection of a corresponding event. Once the event model 35_305 is selected, it can be determined which sub-model to use. Each sub-model may use different historical data, such as mutually exclusive data sets. For different detection events, there will be different decision trees with different submodels.

The first level of the decision tree 35_300 corresponds to a location category. Node 35_310 corresponds to location 1, which may be defined as a bounding region (e.g., within a specified radius) of location 1. Node 35_320 corresponds to position 2. Node 35_330 corresponds to position 3. The node 35_340 corresponds to any other position.

When the context information corresponds to a particular location, rather than a location to which the more general node 35_340 may correspond, each of the nodes 35_310, 35_320, and 35_330 may be generated if the sub-model may predict the application with higher confidence. Nodes 35_310 and 35_320 have more children, while node 35_330 has no children.

Embodiments may traverse the decision tree 35_300 by searching whether any of the nodes 35_310, 35_320, and 35_330 match the context information for a particular occurrence. If the context information of the user equipment for a particular occurrence of an event indicates a context comprising location 3, a match is found for node 35_ 330. Since node 35_330 does not have any other child nodes, the child model for node 35_330 may be used.

Node 35_310 has two child nodes: node 35_311 and node 35_ 312. Node 35_311 corresponds to a particular time (time 1), and node 35_312 corresponds to all other times that do not match time 1. If the context information for the current occurrence of the event includes location 1 (and thus a match with node 35_ 310), a search may be performed to determine if the context information includes time 1 (i.e., a match with node 35_ 311). If the context information includes time 1 (i.e., combined with location 1), then the sub-model for node 35_311 may be used for prediction. If the context information does not include time 1, then the sub-model for node 35_312 may be used for prediction.

Node 35_320 has two child nodes: node 35_321 and node 35_ 322. The node 35_321 corresponds to whether the user equipment is connected to a specific device (device 1), and the node 35_322 corresponds to when the user equipment is not connected to the device 1. If the context information for the current occurrence of the event includes location 2 (and thus a match with node 35_ 310), a search may be performed to determine if the context information includes a connection with a device (i.e., a match with node 35_ 321). If the context information includes a connection with device 1 (i.e. in combination with location 2), the sub-model for node 35_321 may be used for prediction. If the context information does not include a connection to device 1, a sub-model for node 35_322 may be used for prediction.

Thus, once the bottom of the tree is detected, the sub-model of the final node can be used for prediction. All branches of the tree 35_300 may be deterministic, always selecting the final node for the same context information. Having all nodes of the same level of the decision tree 35_300 correspond to the same category may avoid conflicts when selecting an applicable node. For example, if a child node of event model 35_305 corresponds to time 1, then there may be a conflict because it may conflict with node 35_ 311. In such an embodiment, nodes at the same level but under different parent nodes may correspond to different categories, as is the case with the set of nodes 35_311 and 35_312 and the set of nodes 35_321 and 35_ 322.

Once a sub-model is selected based on the detected event and context information, more applications and any corresponding actions may be predicted using the selected sub-model. In some embodiments, the action taken with respect to a predicted application may depend on the confidence level of predicting the application.

B. Method of producing a composite material

FIG. 35_4 is a flow diagram of a method 35_400 of suggesting applications to a user of a computing device based on events, according to an embodiment of the present invention. The method 35_400 may be performed by a computing device (e.g., by a user device that is tracking user interactions with the user device). The method 35_400 may suggest applications using a set of historical interactions, including interactions having different sets of one or more attributes of the computing device.

At block 35_410, the device detects an event at the input device. Examples of input devices are a headset interface, a network connection device, a touch screen, buttons, etc. An event may be any action of the mobile device interacting with an external entity, such as an external device or a user. The event may be of a type that is repeated for the device. Thus, historical statistics may be obtained for different occurrences of an event. Such historical data may be used to train models and sub-models.

At block 35_420, a predictive model corresponding to the event is selected. The selected predictive model may depend on the event. For example, when an event involves establishing a bluetooth connection with an external device, a predictive model designed for the bluetooth connection may be selected. As another example, when an event involves inserting a set of headphones into a headphone interface, a predictive model designed for headphone connection may be selected.

At block 35_430, one or more attributes of the computing device are received. The one or more attributes may be received by an application suggestion engine executing on the device. As described in this section, attributes may correspond to time, location, motion state, current or previous power state (e.g., on, off, or asleep), charge state, current music selection, calendar events, and so forth. Such one or more attributes may correspond to context data defining a particular context of the device. The one or more attributes may be measured around the time the event was detected (e.g., over a period of time). The time period may include times before and after the event is detected, a time period just before the event is detected, or a time just after the event is detected.

At block 35_440, a particular sub-model of the predictive model is selected using the one or more attributes. For example, the decision tree may be traversed to determine the particular sub-model. A particular submodel may correspond to one or more attributes, for example, because the one or more attributes may uniquely identify the particular submodel. This may occur when a decision tree is defined that does not have different categories of attributes under the same parent node.

A particular sub-model may be generated using a particular subset of the historical interactions of the user with the device. This particular subset may be generated by a segmentation process that improves accuracy by creating sub-models. A particular subset of historical interactions may be obtained by tracking user interactions with the device after an event occurs. The computing device has the one or more attributes when the particular subset is obtained. Thus, the current context of the device corresponds to the context of the particular subset of the device in which the historical interactions were obtained.

At block 35_450, the particular sub-model identifies one or more applications to suggest to the user. The one or more applications may have at least one threshold probability of at least one of the one or more applications being accessed by the user in association with the event. One of the one or more applications in the prediction history may be identified as a correct prediction. The threshold probability may be measured in various ways, and may use a probability distribution determined from historical data, as described in more detail below. For example, it may be desirable for the mean (average) probability, median probability, or peak value of the probability distribution to be above a threshold probability (e.g., above 0.5, equivalent to 50%). Thus, the confidence level may be the mean, median or peak of the probability distribution. Another example is that the regions on the probability distribution above a certain value are greater than a threshold probability.

At block 35_460, a user is provided with a user interface for interacting with one or more applications. For example, the device may display the identified application to the user via an interface that the user may interact with to indicate whether the user wishes to access the identified application. For example, the user interface may include a touch-sensitive display that displays one or more of the identified applications to the user and allows the user to access the one or more applications identified by the device by interacting with the touch-sensitive display. The user interface may allow for interaction on a display screen with fewer applications than are provided on the home screen of the computing device.

For example, one or more suggested applications may be provided on the lock screen. The user may choose to open the application from the lock screen, thereby making it easier for the user to interact with the application. A user interface may be provided on the other screen that may appear after the button is activated to begin using the device. For example, the application-specific user interface may appear after the user is authenticated (e.g., via a password or biometric).

C. Exemplary model

In some embodiments, the model may select the top N applications for a given data set (or subset). Since these N applications were selected the most in the past, it can be predicted that future behavior will reflect past behavior. N may be a predetermined number (e.g., 1,2, or 3) or percentage of applications, which may be the percentage of applications actually used in association with the event (i.e., not all applications on the device). Such a model may select the top N applications to provide to the user. Further analysis may be performed, for example, to determine a probability (confidence) level for each of the N applications to determine whether to provide them to the user, and how to provide them to the user (e.g., actions), which may depend on the confidence level.

In the example where N is equal to 3, when an event occurs where the context information corresponds to a particular sub-model, the model will return to launching the first three applications at the most.

In other embodiments, rather than using only context information to select a submodel, a submodel may use a composite signal, with some context information being used to determine the predicted application. For example, a neural network or logistic regression model may predict an application using location (or other features) and constructing a type of linear weighted combination of these features. Such more complex models may be more suitable when the data volume of the sub-model is significantly large. Some embodiments may switch the type of sub-model used at a particular node (i.e., a particular combination of context data) after obtaining the data for that node again.

Generation of models and decision trees

In some embodiments, the decision tree may be regenerated periodically (e.g., daily) based on historical data at the time of regeneration. Thus, the decision tree may have different forms on different days. The generation of a child node (another sub-model) may be controlled according to the confidence increase (also referred to as information gain) for the prediction application. The generation of a child node may also be controlled based on whether the data for the child node is statistically significant. In some embodiments, all child nodes at a given level (e.g., gym child model 35_210 and other child models 35_240) may need to be statistically significant and provide information gain relative to the parent model.

In determining the nodes of the decision tree, segmentation may be performed in various ways to produce different decision trees. For example, both a particular location and a particular time may be used. In some embodiments, higher attributes may be generated in the decision that provide the highest increase in information gain (confidence) for the predicted application. Such a segmentation process may ensure the highest probability of predicting the correct application with which the user will interact.

A. Accuracy distribution of model

The accuracy of the model can be tested against historical data. For a given event, the historical data may identify which applications were used in association with the event (e.g., just before the event or just after the event, such as within one minute). For each event, the context data may be used to determine a particular model. In addition, context data may be used as input features for the model.

In an example where the model (or sub-model) selects the top application, the plurality of historical data points when the top application was actually selected (launched) may be determined to be a correct count, and the plurality of historical data points when the top application was not selected may be determined to be an incorrect count. In embodiments where N is greater than one for the model that selects the first N applications, the correct count may correspond to any historical data points at which one of the first N applications was launched.

Both correct and incorrect counts can be used to determine a distribution that specifies how accurate the model is. A binomial distribution can be used as the accuracy distribution. The binomial distribution with parameters m and p is a discrete probability distribution of the number of successes in the sequence of m independent yes/no experiments. Here the yes/no test is on whether one of the predicted N applications is correct. For example, if the model predicts that a music application will be launched and the music application has been launched, the data point will be added to the "yes" ("true") number of experiments. If the music application is not launched (e.g., another application is launched or no application is launched), the data point will be added to the "no" ("false") number of experiments.

According to bayesian theorem, P (a-B) ((P (B | a) (P (a))/(P (B)). B is the event that gets a specified determined correct count T and incorrect count f.a is the predicted application correct event.p (a) is the a priori (expected) probability of randomly selecting the correct application, which can be assumed to be 1, since no particular application will be more expected than any other application, at least no historical data.p (a) is the probability of the model being correct (which corresponds to the correct count divided by the total historical events.) P (B | a) is the likelihood function that gets the correct count T and incorrect count F for a given probability r (i.e. event a, which can be taken to be 0.5 to represent the same probability of being correct or incorrect.) P (a | B) is the a posterior probability to be determined, i.e. the probability of selecting one of the predictive applications given the historical data B.

If the a priori distribution is uniform, then P (a) does not exist, leaving P (a | B)/P (B), which is equal to Beta [ correct, incorrect ], i.e., Beta distribution with parameters α ═ correct and β ═ incorrect. Since the β function is ambiguous for α -0 or β -0, embodiments may assume an initial value of 1 for both correct and incorrect numbers. Beta [1+ exact, 1+ incorrect ] is a binomial distribution.

For bayesian statistics, given evidence X (e.g., correct and incorrect counts of historical data), the posterior probability p (θ | X) is the probability of the parameter θ (e.g., the actual selected application is one of the predicted applications). It is in contrast to a likelihood function p (X | θ), which is the probability of evidence X (e.g., correct and incorrect counts of historical data) for a given parameter (e.g., the selection of the predictive application for an event). The relationship between the two is as follows: let us have a prior belief that the probability distribution function is P (θ) (e.g., the expected probability of correct selected application) and the observed value X has a likelihood of P (X | θ), then the posterior probability is defined as P (θ | X) ═ P (X | θ) (P (θ))/(P (X)). the posterior probability can be considered to be proportional to the product of the likelihood and the prior probability.

Other accuracy profiles may be used. For example, a Dirichlet distribution can be used, which is a multivariate generalization of beta distribution. Dirichlet distributions are the conjugate priors of both the class distributions and the polynomial distributions, similar to beta distributions which are the conjugate priors of the binomial distributions. Dirichlet returns the belief that the probability density function returns that the probability of K oppositional events is x _ i, given that each event has been observed α _ i-1 times. The Dirichlet distribution can be used to generate an entire histogram of application launches (i.e., predicted application launch times for a particular event) as a multi-term distribution.

Instead, embodiments may separate them into two categories (correct and incorrect), so two distributions are used and the entire histogram need not be provided. Other embodiments may use Dirichlet distributions (conjugate priors of multiple distributions) to try to solve the problem of more difficult description of the entire histogram, but this requires more data to be made trustworthy as more data needs to be interpreted.

B. Exemplary binomial distribution

Fig. 35_ 5A-35 _5D show graphs of exemplary binomial distributions for various correct and incorrect numbers according to embodiments of the present invention. These plots were generated from Beta [1+ correct number, 1+ incorrect number ]. On the horizontal axis in the graph, 1 corresponds to correct prediction and 0 corresponds to incorrect prediction. The vertical axis provides the probability of the frequency at which the model will be correct. These distributions are also referred to as Probability Density Functions (PDFs). The distribution may be normalized for comparison.

Fig. 35_5A shows binomial distributions for two correct predictions and two incorrect predictions. Such a model will have equal correct and incorrect numbers, so the highest probability occurs at 0.5. The maximum value at 0.5 indicates that most likely only half of the temporal predictions of the model are accurate. The distribution is rather broad in view of the small number of data points. Therefore, the confidence regarding the accuracy of the model is low. The perceived probability is that the model is less accurate 50% of the time, or more accurate 50% of the time. However, since the number of data points is small, the confidence of the determination accuracy is low.

Fig. 35_5B shows the binomial distribution for 2 correct predictions and 1 incorrect prediction. This pattern was correct 66% of the time. Thus, the peak of the distribution occurs at about 0.66. However, the confidence is very low in view of the small number of data points. The perceived probability is that the model is only possible to be accurate 10% or 20% of the time.

Fig. 35_5C shows binomial distributions for four correct predictions and two incorrect predictions. This model was also correct 66% of the time. However, still in view of the small number of data points, there is still an appreciable probability that the model will be only 30% accurate if the data is available again.

Fig. 35_5D shows the binomial distribution for 40 correct predictions and 20 incorrect predictions. Such a pattern is also correct 66% of the time. However, given the large number of data points, the probability of the accuracy of the model being only 30% is very low. Thus, the distribution shows a higher confidence in being able to determine the model with an accuracy of 66%. Furthermore, more of the area under the distribution is to the right of 0.5, so it can be determined more confidently that the model is accurate at least 50% of the time than in the case of fig. 35_ 5B.

C. Statistical significance

A model can be considered statistically significant if it can accurately distinguish when it is correct and when it is incorrect with sufficient confidence. A posterior probability distribution determined based on the number of incorrect predictions and correct predictions may be used to determine with sufficient confidence whether the model is sufficiently accurate.

The confidence level required for statistical significance may be provided in various ways and may have various criteria. The average accuracy (correct number/total number) of the distribution, the peak value of the distribution, or the median of the distribution may be required to have a certain value. For example, the model may be required to be correct at least 50% of the time, e.g., as measured from the mean of the distribution (i.e., greater than 0.5). The correct number/total is also called maximum likelihood estimation.

Another criterion (confidence level) may be used for confidence in accuracy. Confidence may be measured by an integration of the distribution above a lower limit (e.g., an area of the distribution above 0.25 or other value). The area under the distribution curve is also referred to as the cumulative distribution function. In one embodiment, the criterion may be that 95% of the area of the PDF is greater than 0.25. The point where the interval [ x,1.0] covers 95% of the area below the PDF is called the "lower confidence limit". Thus, if you are right twice and wrong once, it means you are right 66% of the time, but this is not statistically significant because the distribution is very wide, as in fig. 35_ 5B.

Some embodiments begin using the model (e.g., top-level model or sub-model) only if the model is accurate enough and with enough confidence to know the accuracy. For example, the initial model may be trained for a period of time before use. Only if the accuracy and confidence are above the respective thresholds, embodiments may begin using the model to provide suggestions to the user. In some embodiments, the requirement for a PDF a certain amount of area may provide a single criterion for determining whether to use the model, since the accuracy may be considered high enough if the area is moved sufficiently to the right.

In some embodiments, the initial model may use data from others to provide more statistics, at least initially. Then, once sufficient statistics are obtained, only the data of a particular person can be used. In addition, user-specific data may have a higher weight in order to phase out data from others.

D. Information gain (entropy)

A comparison may be made between the first probability distribution of the model and the second probability distribution of the submodel to determine whether to segment the model. In some embodiments, the comparison may determine whether there is an information gain (e.g., Kullback-Leibler divergence), or equivalently, a reduction in entropy. High entropy will cause many applications to have similar probabilities of being selected, and maximum entropy causes all applications to have the same probability. With maximum entropy, the likelihood of selecting the correct application is minimal, since all applications have equal probability and all applications are not more likely to be selected than another application.

Such a measure of variance may be used to determine whether a sub-model may be used to make more accurate predictions (including confidence) for a given context for which the sub-model applies. If the difference measure value is greater than the difference threshold, segmentation may be performed. The difference measure may be signed with a positive sign to ensure that the information achieves a gain. The Kullback-Leibler divergence can be used as a measure of difference. Other exemplary metrics include keney purity and variance reduction.

For example, if one model applies to all cases, the model will only choose the top application (e.g., music application) for all contexts. The music application will be a prediction of all contexts (e.g., gym, driving work, etc.). Since the sub-models are generated for more specific contexts, the predictions may become more specific, e.g., when the user goes to a gym, a single application dominates, or a particular playlist dominates. Thus, the number of times an application is selected may have a peak, and then everything else is zero. Thus, the goal of the decision tree is to maximize the information gain (minimize the entropy).

Other submodels may be identified when more specific contexts may provide more information gain. For example, a morning gym may be a more specific context, where a particular playlist is dominant. As another example, a morning connection to a car may provide a more accurate prediction of a news application because the historical data is organized more (entropy is reduced) to primarily select a news application (or a group of news applications).

Fig. 35_6A and 35_6B illustrate parent models and child models generated from segmentation according to an embodiment of the present invention. Fig. 35_6A shows a binomial distribution of parent models that provide 80 correct predictions and 60 incorrect predictions. A child model may be created from a portion of the historical data for the parent model. Fig. 35_6B shows the binomial distribution of submodels providing 14 correct predictions and 2 incorrect predictions. Even if the submodel has fewer data points, the prediction is more accurate because evidence of a move indicates higher accuracy. Therefore, the entropy has decreased and there is an information gain.

E. When to segment

As described above, various embodiments may use one or more criteria to determine whether to segment a model to generate sub-models. One criterion may be that the confidence level (selecting one of a set of one or more prediction applications) for making a correct prediction is greater than a confidence threshold. For example, the average probability of correct prediction is greater than an accuracy threshold (an example of a confidence threshold). As another example, the CDFs distributed above a particular value may be required to be above a confidence level.

Another criterion may be to use a sub-model that provides information gain (entropy reduction) instead of a model. For example, the value for the Kullback-Leibler divergence may be compared to a difference threshold. One or more criteria for segmentation may ensure that a sub-model will outperform a base model. One or more criteria may be required for all sub-models of a certain parent model, such as the gym sub-model 35_210 and the other sub-models 35_ 240.

In some cases, the confidence limits for the two sub-models may be reduced relative to the parent model, but still have an information gain and a confidence limit above a threshold. The confidence lower limit may also increase. As long as all submodels have a high enough confidence limit and the information gain is large enough, embodiments may choose to segment (partition) the more general model.

In some embodiments, any accuracy and information gain criteria may be met by ensuring that confidence levels are increased as a result of segmentation. For example, a first attribute of a device may be selected to test a first sub-model of a first context, which may include other attributes relative to a parent model. A first subset of historical interactions that occurred when the computing device had a first attribute may be identified. The first subset is selected from a set of historical interactions for the parent model and is smaller than the set of historical interactions.

Based on the first subset of historical interactions, the first submodel may predict at least one application of the first set of one or more applications that the user will access in association with the event having the first confidence level. The first submodel may be created based at least on the first confidence level being greater than the initial confidence level by at least a threshold amount (which may be 0 or greater). The threshold amount may correspond to a difference threshold. In some implementations, the first submodel may not necessarily be created when the criterion is met, as further criteria may be used. If the confidence level is not greater than the initial confidence level, another attribute may be selected for testing. This comparison of confidence levels may correspond to a test of information gain. The same process may be repeated to determine a second confidence level for predicting a second submodel (for the second attribute) for the first submodel for the first set of one or more applications. A second subset of the historical interactions may be used for a second submodel. A third attribute or more may be tested in a similar manner.

F. Regeneration of decision trees

Embodiments may generate the decision tree for the model periodically (e.g., daily). This generation may use historical data available at the time. Thus, the decision tree can be changed from one generation to another. In some implementations, the decision tree is constructed without knowledge of the previous decision tree. In other embodiments, a new decision tree may be constructed from such prior knowledge, e.g., knowing which submodels are possible, or starting from a previous decision tree.

In some embodiments, all contexts (or a predetermined list of contexts) are tried to determine which submodels provide the greatest information gain. For example, if a location provides the maximum information gain for segmentation into submodels, a submodel for at least one particular location may be created. At each segmentation level, the contexts may be tested in such a greedy manner to determine which contexts provide the highest increase in information gain.

In other embodiments, a subset of contexts is selected (e.g., randomly selected, including pseudo-randomly) to test whether segmentation is appropriate. This option is advantageous when there are many contexts that can be tested. Context selection may be performed using a Monte Carlo based method that may use the probability that a context will likely result in a segmentation. A random number (an example of a random process) may be generated and then used to determine (for a particular attribute) which context to test.

The probability may be used as a weight so that contexts with higher weights are more likely to be selected in this "random" selection process. The probability may be determined based on which sub-models were generated in the past. For example, if the gym (and possibly a particular time of day) was highly successful before, the generation process would select the context with a 90%, 95%, or 99% probability, depending on how often it was selected in the past, and possibly also depending on how high the information gain was in the past. A certain number of divisions will be tried for each level or for the whole tree generation process.

Determining actions based on probability levels

The predictive model may test not only the selected application, but also certain actions, and possibly media content (e.g., a particular playlist). In some embodiments, once the probability of selecting an application is sufficiently accurate, a more aggressive action may be provided than merely providing a launch option. For example, the content may be automatically played when the application is started. Alternatively, the application may be automatically launched.

When an application is selected with sufficient probability (e.g., confidence level above a high threshold) prediction, the prediction may begin a test action. Thus, the test is not just for predicting the application, but whether a particular action can be predicted with sufficient accuracy. Different possible actions (including media items) may be obtained from the historical data. Multiple actions may be selected for execution with one application. Each of the plurality of actions may correspond to one of a plurality of different submodels of the first submodel. The confidence level of each of the plurality of different submodels may be tested to determine whether to generate a second submodel for at least one of the plurality of actions.

Thus, when there is greater confidence, the embodiment may more aggressively perform the action to be performed. The predictive model may provide a particular user interface if a particular action is performed with a high probability. Thus, in some embodiments, the higher the likelihood of use, the more aggressive actions may be taken, such as automatically opening an application with a corresponding user interface (e.g., visual or voice commands), rather than merely providing a mechanism by which the application is more easily opened.

For example, the base model may have some degree of statistical significance (accuracy and confidence) and the action may be to suggest an application on the lock screen. As another example, a higher level of statistical significance may cause the screen to light up (thereby drawing attention to the application, only one application may be selected, or a User Interface (UI) for the application may be provided (i.e., not the UI of the system for selecting the application) some embodiments may consider actions taken in determining whether to segment, and not segment when the actions fail, which would typically correspond to having an information gain.

The action may depend on whether the model predicts only one application or a group of applications. For example, if there is an opportunity to make three suggestions instead of one, this will also change the probability distribution, since selecting any of the three will provide the correct prediction. A model in which a recommendation for one application is not trusted may have sufficient confidence in recommendations for three applications. Embodiments may perform the addition of another application to a group of applications predicted by the model (e.g., a second most common application not already in the group), thereby giving the model a higher confidence. If the model is based on predictions of more than one application, the provided user interface will provide for interaction with more than one application, which may affect the form of the UI. For example, all applications may be provided on the lock screen and one application will not automatically launch.

There may also be multiple operations and one proposal for different operations. For example, there may be two playlists in a gym as part of a sub-model (e.g., one application is identified when two actions have similar selected possibilities, but two actions are identified in the model). Both actions together may be statistically significant, while they alone are not.

For example, when the model is first trained for some event (e.g., insertion of a headphone), the confidence of the model may not be sufficient to perform any action. At an initial confidence level, an icon or other object may be displayed on the lock screen. At the next higher confidence level, the screen may light up. With a further level of confidence, a user interface specific to a particular function of the application may be displayed (e.g., a control for playing music or a scroll window for accessing headline news of a new application). The next higher level may correspond to some function of the application being automatically initiated. This action may even replace the current operation of the application (e.g., playing one song) to play another song or playlist. These different levels may represent various values for defining confidence levels.

Other example actions may include altering a song being played, providing a notification (which may be a prominent location on the screen). The action may occur after unlocking the device, e.g., a UI specific to the application may be displayed after unlocking. Operations may be defined using deep links to start specific functions of an application.

Some embodiments may display a notification to a user on a display screen. The notification may be sent by, for example, a push notification. The notification may be a visual notification that includes a picture and/or text that informs the user of the suggested application. The notification may suggest an application to the user for the user to select and run at leisure. After being selected, the application may be run. In some embodiments, the notification may also include suggested actions within the suggested application for more aggressive prediction. That is, the notification may notify the user of the suggested application and the suggested action within the suggested application. Thus, the user may be provided with the option to run a suggested application or to perform a suggested action within the suggested application. For example, the notification may inform the user that the suggested application is a music application and that the suggested action is playing a certain song in the music application. The user may indicate that he or she wants to play the suggested song by clicking on the icon showing the song. Alternatively, the user may indicate that he or she prefers to run the application to play another song by swiping the notification across the screen.

In addition to outputting the suggested applications and suggested actions to the user interface in one notification, the prediction engine may output two suggested actions to the user interface in one notification. For example, the prediction engine may output one suggested action to play a first song and a second suggested action to play a second song. The user can select which song to play by clicking on the corresponding icon in the notification. In an embodiment, the suggested actions may be determined based on different criteria. For example, one suggested action may be for playing a recently played song regardless of the contextual information, while another suggested action may be for playing a song that was last played under the same or similar contextual information. For example, for the case where the user enters his or her car and a triggering event causes the prediction engine to suggest two actions related to playing a certain song, Song A may be the song that was last played at home, while Song B may be the song that was last played by the user in the car. When the user selects a song to play, the song may continue from the beginning, or from where it last stopped (e.g., in the middle of the song).

To enable the prediction engine to suggest actions, the prediction engine 35_302 may access a memory device that stores information about the active state of the device. The active state of the device may represent an action performed after the suggested application is selected. For example, the active state for a music application may be playing a certain song. The active state may record the time the song last stopped. In an embodiment, the historical database may record historical data relating to the activity state of the device. Thus, the prediction engine may suggest actions to be run by the suggested application.

Architecture VI

Fig. 35_7 illustrates an exemplary architecture 35_700 for providing a user interface for a user to interact with one or more applications. The architecture 35_700 shows elements for detecting events and providing suggestions to applications. The architecture 35_700 may also provide other suggestions, for example, for suggesting contacts. The architecture 35_700 may exist within a user device (e.g., device 100 in fig. 1A).

The top is the UI element. As shown, there is a lock screen 35_710, a search screen 35_720, and a voice interface 35_ 725. The user interface may be provided to the user in these ways. Other UI elements may also be used.

The bottom is the data source. The event manager 35_742 may detect the event and provide information about the event to the application suggestion engine 35_ 740. In some embodiments, the event manager may determine whether an event triggers an application suggestion. A list of predetermined events for triggering application suggestions may be specified. The location unit 35_744 may provide the location of the user equipment. For example, the location unit 35_744 may include a GPS sensor and a motion sensor. The location unit 35_744 may also include other applications that may store the last location of the user, which may be sent to the application suggestion engine 35_ 740. Other context data may be provided from other context units 35_ 746.

The application suggestion engine 35_740 may identify one or more applications and corresponding actions. At the same level as the application suggestion engine 35_740, the contact suggestion engine 35_750 may provide suggested contacts for presentation to the user.

The suggested applications may be provided to the display center 35_730, which may determine what to provide to the user. For example, the display center 35_730 may determine whether to provide a suggested application or to provide a contact. In other examples, both an application and a contact may be provided. The display center may determine the best way to provide to the user. Different suggestions to the user may use different UI elements. In this way, the display center 35_730 can control the suggestions to the user so that different engines do not interrupt the suggestions provided by other engines. In various embodiments, the engine may push the suggestions (recommendations) to the display center 35_730 or receive requests for suggestions from the display center 35_ 730. The display center 35_730 may store the suggestion for a certain time and then determine to delete the suggestion if the suggestion is not provided to the user or the user does not interact with the user interface.

The display center 35_730 may also identify other actions that the user device takes in order for the device to send suggestions when. For example, if the user is using an application, no suggestion may be provided. The display hub 35_730 may determine when to send the suggestion based on various factors, such as the motion state of the device, whether the lock screen is on, or authorized access is provided, whether the user is using the device, and the like.

In some embodiments, the software component included on device 100 (FIG. 1A) includes an application suggestion module. The application suggestion module may include various sub-modules or systems, such as described above in fig. 35_ 7. The application suggestion module may perform all or part of the method 37_ 400.

Example of decision tree segmentation for learning generative models of complex user patterns in the context of data sparsity Sexual method, apparatus system and computer readable medium

This section provides systems, methods, and apparatus for suggesting one or more applications to a user based on an event. The predictive model may correspond to a particular event. The suggested application may be determined using one or more attributes of the computing device. For example, a particular sub-model may be generated from a subset of historical data collected about user interactions after an event occurs and when a device has one or more attributes. The sub-model trees may be determined corresponding to different contexts of computing device attributes. Also, various criteria may be used to determine when to generate a sub-model, such as a confidence level in the sub-model that provides a correct prediction in the historical data subset, and an increase in information in the historical data distribution (entropy reduction) relative to the parent model.

In some embodiments, a method for suggesting one or more applications to a user of a computing device based on an event is provided, the method comprising: at a computing device: detecting an event at an input device of a computing device, the event being of a type that recurs for the computing device; selecting a prediction model corresponding to the event; receiving one or more attributes of a computing device; using the one or more attributes to select a particular sub-model of the predictive model that corresponds to the one or more attributes, wherein the particular sub-model is generated using a particular subset of historical interactions of the user with the computing device that occurred after the event was detected and when the computing device had the one or more attributes; identifying, by the particular sub-model, one or more applications to suggest to the user, the one or more applications having at least one threshold probability of at least one of the one or more applications being accessed by the user in association with the event; and providing a user interface to the user for interacting with the one or more applications.

In some embodiments, user interaction is provided on a display screen having fewer applications than are provided on the home screen of the computing device. In some embodiments, a particular sub-model predicts one or more applications with a confidence level greater than a confidence threshold. In some embodiments, the method comprises: determining how to provide the user interface to the user based on the confidence level. In some embodiments, the method comprises: determining a confidence level by: determining a first probability distribution; and for points greater than the lower limit, calculating a cumulative distribution of the first probability distribution to obtain a confidence level. In some embodiments, the method comprises: determining a confidence level by: determining a first probability distribution; and calculating a mean, median, or peak value of the first probability distribution to obtain a confidence level. In some embodiments, the particular submodel provides a first probability distribution for a particular subset of correctly predicted historical interactions using an information gain relative to a second probability distribution for correct prediction by the predictive model. In some embodiments, the information gain is greater than a difference threshold, and wherein the information gain is determined using a Kullback-Leibler divergence. In some embodiments, the method comprises: receiving a set of historical interactions of the user with the computing device after detecting the event, wherein the set of historical interactions includes and is greater than the particular subset of historical interactions, the set of historical interactions including interactions having a different set of one or more attributes of the computing device; using an initial model of the predictive model to calculate an initial confidence level for predicting one or more applications that the user will access after the event based on the set of historical interactions; and generating a sub-model tree for the predictive model by: selecting a first attribute of a computing device; identifying a first subset of historical interactions that occurred when the computing device had a first attribute, the first subset being selected from the set of historical interactions and being smaller than the set of historical interactions; calculating a first confidence level using the first submodel, the first confidence level for predicting at least one application of the first set of one or more applications that the user will access in association with the event based on the first subset of historical interactions; creating a first sub-model based on the first confidence level being greater than the initial confidence level by at least a threshold amount; and selecting another attribute for testing when the first confidence level is not greater than the initial confidence level. In some embodiments, the method comprises: when the first confidence level is not greater than the initial confidence level: another application is added to the first set of one or more applications and the first sub-model is tested again. In some embodiments, the method comprises: generating a sub-model tree for the predictive model further by: selecting a second attribute of the computing device; identifying a second subset of historical interactions that occurred when the computing device had the first attribute and the second attribute, the second subset being selected from the first subset of historical interactions and being smaller than the first subset of historical interactions; calculating a second confidence level using the second submodel, the second confidence level for predicting, based on a second subset of the historical interactions, one of a second set of one or more applications that the user will access in association with the event; creating a second sub-model based on the second confidence level being greater than the first confidence level by at least a threshold amount; and selecting a third attribute for testing when the second confidence level is not greater than the first confidence level. In some embodiments, a sub-model tree for the predictive model is generated periodically. In some embodiments, the first attribute is selected using a random process. In some embodiments, the first set of one or more applications is one application, and the method comprises: selecting a plurality of actions to be performed by the one application, each of the plurality of actions corresponding to one of a plurality of different sub-models of the first sub-model; the confidence level of each of the plurality of different submodels is tested to determine whether to generate a second submodel for at least one of the plurality of actions.

In some embodiments, a computer product is provided that includes a non-transitory computer-readable medium storing a plurality of instructions for suggesting one or more applications to a user of a computing device based on an event, the plurality of instructions, when executed on one or more processors of a computer system, performing: at a computing device: detecting an event at an input device of a computing device, the event being of a type that recurs for the computing device; selecting a prediction model corresponding to the event; receiving one or more attributes of a computing device; using the one or more attributes to select a particular sub-model of the predictive model that corresponds to the one or more attributes, wherein the particular sub-model is generated using a particular subset of historical interactions of the user with the computing device that occurred after the event was detected and when the computing device had the one or more attributes; identifying, by the particular sub-model, one or more applications to suggest to the user, the one or more applications having at least one threshold probability of at least one of the one or more applications being accessed by the user in association with the event; and performing an action on the one or more applications. In some embodiments, the particular sub-model predicts the one or more applications having a confidence level greater than a confidence threshold, and wherein the particular sub-model provides a first probability distribution for a particular subset of correct prediction historical interactions using an information gain relative to a second probability distribution for correct prediction of the prediction model. In some embodiments, the action is providing a user interface to the user for interacting with the one or more applications.

In some embodiments, there is provided a computing device for suggesting one or more applications to a user of the computing device based on an event, the computing device comprising: an input device; one or more processors configured to: detecting an event at an input device of a computing device, the event being of a type that recurs for the computing device; selecting a prediction model corresponding to the event; receiving one or more attributes of a computing device; using the one or more attributes to select a particular sub-model of the predictive model that corresponds to the one or more attributes, wherein the particular sub-model is generated using a particular subset of historical interactions of the user with the computing device that occurred after detecting the event and when the computing device had the one or more attributes; identifying, by the particular sub-model, one or more applications to suggest to the user, the one or more applications having at least one threshold probability of at least one of the one or more applications being accessed by the user in association with the event; and providing a user interface to the user for interacting with the one or more applications. In some embodiments, the particular sub-model predicts the one or more applications having a confidence level greater than a confidence threshold, and wherein the particular sub-model provides a first probability distribution for a particular subset of correct prediction historical interactions using an information gain relative to a second probability distribution for correct prediction of the prediction model. In some embodiments, the one or more processors are further configured to: receiving a set of historical interactions of the user with the computing device after detecting the event, wherein the set of historical interactions includes and is greater than the particular subset of historical interactions, the set of historical interactions including interactions having a different set of one or more attributes of the computing device; using an initial model of the predictive model to calculate an initial confidence level for predicting one or more applications that the user will access after the event based on the set of historical interactions; and generating a sub-model tree for the prediction model by: selecting a first attribute of a computing device; identifying a first subset of historical interactions that occurred when the computing device had a first attribute, the first subset being selected from the set of historical interactions and being smaller than the set of historical interactions; calculating a first confidence level using the first sub-model, the first confidence level for predicting at least one application of the first set of one or more applications that the user will access in association with the event based on the first subset of historical interactions; creating a first sub-model based on the first confidence level being greater than the initial confidence level by at least a threshold amount; and selecting another attribute for testing when the first confidence level is not greater than the initial confidence level.

Section 6: application suggestion based on detected trigger events

The material in this section, "application suggestions based on detected trigger events," describes application suggestions based on detected trigger events according to some embodiments, and provides information that supplements the disclosure provided herein. For example, some portions of this section describe recommending applications for use based on a triggering event (inserting headphones into the device and suggesting different applications depending on the user's current location), which complements the disclosure provided herein, such as those related to populating the predicted content within prediction section 930 of fig. 9B-9C and related to creating and detecting triggering conditions (fig. 4A-4B). In some embodiments, the predictive models described in this section are used to help identify applications that are suitable for prediction and display to a user (i.e., these predictive models are used in conjunction with methods 600,800,1000 and 1200).

Concise summarization of application suggestions based on detected trigger events

Embodiments provide improved apparatus and methods for recommending applications based on triggering events. For example, certain events may be detected by the device and identified as triggering events. Different trigger events may have different predictive models, which may enable more accurate recommendations. The selected predictive model may use contextual information (e.g., collected before or after the event is detected) to identify applications for presentation to the user for easier access (e.g., to allow access on a locked screen).

In some embodiments, one or more input devices are monitored for the occurrence of a triggering event. When a trigger event is detected, context information may be collected from one or more sources (e.g., another application of the device that has obtained the context information). The context information may relate to the context of the device at or near the time of the triggering event, such as location or time of day. Once the context information is received, historical information may be collected from a historical event database. The database may maintain a record of historical interactions between the user and the device. Based on the trigger event, context information and historical information may be utilized to identify a set of one or more applications for the user. The identified application may then be suggested to the user by providing a user interface in a manner that is different from how, when, or where the identified application is normally accessed (e.g., on the home screen), providing the user with an option to run the application when needed.

Other embodiments relate to systems, portable consumer devices, and computer-readable media associated with the methods described in this section.

The nature and advantages of embodiments of the present invention may be better understood by reference to the following detailed description and accompanying drawings.

Detailed description of application suggestions based on detected trigger events

Current mobile devices can store many applications on their solid state drives. In some cases, a mobile device may store hundreds of applications on its solid state drive. When a user wants to run a certain application on his mobile device, he or she must unlock the device, search all applications in the device to identify the desired application, and then initiate execution of the application. The process of finding the desired application can be very time consuming and redundant, especially for applications that are used more often than other applications.

The user may pre-program the device to automatically perform a specified action of a predetermined application when a particular condition is met (e.g., a triggering event occurs). For example, the device may be programmed to suggest a predetermined application when a triggering event occurs. However, such operations are static and require configuration by the user.

Rather than automatically suggesting a predetermined application, embodiments of the present invention may utilize a predictive model to suggest applications that are likely to be run by a user in a given context upon the occurrence of a triggering event. Different applications for different contexts may be identified for the same trigger event. For example, one application may be suggested in a first context, while another application may be suggested in a second context.

Identifying applications that a user may use has several benefits. The user interface may be provided to the user in a suitable manner or in a suitable screen, which may save time and simplify device operation. The user does not have to search through many applications to identify the application to use. The user interface for the application may be provided in various ways, which may depend on how high a probability that the user will use the application is. Further, the predictive model may provide a particular user interface if a particular action is performed with a high probability. Thus, in some embodiments, the higher the likelihood of use, the more positive actions may be taken, such as automatically opening an application with a corresponding user interface (e.g., visual or voice commands), rather than merely providing a mechanism by which the application is more easily opened.

Application prediction

Embodiments may suggest applications based on a triggering event. For example, when a headset is plugged into the headset interface, a music application may be suggested. In some embodiments, the context information may be used in conjunction with a triggering event to identify an application to suggest to the user. For example, when a set of headphones is plugged into a headphone jack, contextual information relating to location may be used. If the device is located in, for example, a gym, application a may be suggested when the headset is plugged into the headset jack. Alternatively, if the device is located in the home, application B may be suggested when the headset is plugged into the headset jack. Thus, applications that may be used in certain contexts may be suggested at appropriate times, thereby enhancing the user experience.

FIG. 36_1 is a flow diagram of a method 36_100 for suggesting applications based on a triggering event, according to an embodiment of the invention. Method 100 may be performed by a mobile device (e.g., phone, tablet computer) or a non-mobile device, and utilize one or more user interfaces of the device.

At block 36_102, a triggering event is detected. Not all events occurring on the device are trigger events. The trigger event may be identified as being sufficiently likely to be associated with a unique operation of the device. An event list of trigger events may be stored on the device. Such events may be a default list and maintained as part of the operating system and may or may not be configured by the user.

The trigger event may be an event caused by a user and/or an external device. For example, the triggering event can be when the accessory device is connected to the mobile device. Examples include plugging a headset into a headset interface, making a bluetooth connection, etc. In this example, each of them may be classified as a different triggering event, or the triggering events may collectively be any accessory device connected to the mobile device. As other examples, the triggering event may be a particular interaction of the user with the device. For example, a user may move a mobile device in a manner consistent with operation, in which case the operational state of the device is a triggering event. Such an operational state (or other state) may be determined based on sensors of the device.

At block 36_104, the application associated with the trigger event is identified. For example, when a headset is plugged into the headset interface, a music application may be identified. In some implementations, more than one application may be identified. The predictive model may identify the associated application, where the predictive model may be selected for a particular triggering event. The predictive model may use context information to identify applications because, for example, different applications may be more likely to be used in different contexts. Some embodiments may only identify an application when there is a sufficient probability of being selected by the user (e.g., as determined by the user's historical interactions with the device). Various types of predictive models may be used. Examples of predictive models include neural networks, decision trees, multi-label logistic regression, and combinations thereof.

At block 36_106, an action is performed in association with the application. In one embodiment, the action may be providing a user interface for the user to select to run the application. The user interface may be provided in various ways, such as by being displayed on a screen of the device, projected onto a surface, or provided as an audio interface.

In other embodiments, an application may be run and a user interface specific to the application may be provided to the user. Either user interface may be provided in response to identifying an application, for example, on a lock screen. In other implementations, a user interface for interacting with the application may be provided after the user is authenticated (e.g., by password or biometric). When the user interface is displayed, such a user interface will be more specific than a home screen alone, i.e., less than the applications on the home screen, of the list of suggested applications to run. The user interface may be displayed on a display of the device immediately after the trigger event is detected. In other embodiments, the user interface may be displayed after the user provides some input (e.g., one or more tap gestures) that may still be less user input (e.g., number of taps) than if no application was suggested.

Initiating a predicted event

The trigger event may be a set of predetermined events that trigger the identification of one or more applications to be provided to the user. The event may be detected using a signal generated by a device component. This section will discuss more details of how a trigger event is detected in further detail.

Fig. 36_2 shows a simplified block diagram of a detection system 36_200 for determining a trigger event according to an embodiment of the present invention. Detection system 36_200 may reside within a device for which a trigger event is being determined. As shown, detection system 36_200 can detect a number of different events. One or more of the detected events may be determined by detection system 36_200 to be a triggering event. Other processing modules may then perform processing using the triggering event.

A. Detecting events

In an embodiment, detection system 36_200 includes hardware and software components for detecting events. For example, detection system 36_200 may include a plurality of input devices, such as input device 36_ 202. The input device 36_202 may be any suitable device capable of generating a signal in response to an event. For example, the input devices 36_202 may include a device connection input device 36_204, a user interaction input device 36_206, and a location input device 36_208 that may detect a device connection event, a user interaction event, and a location event, respectively. When an event is detected at the input device, the input device may send a signal indicating the particular event for further analysis.

In some embodiments, a collection of components may participate in a single event. For example, a person may be detected as running based on a motion sensor and a GPS location device.

1. Device connection event

A device connection event may be an event that occurs when another device is connected to the device. For example, the device connection input device 36_204 may detect an event that the device is communicatively coupled to the device. Any suitable device components forming a wired or wireless connection to an external device may be used as the device connection input device 36_ 204. Embodiments of the device connection input device 36_204 include a headset interface 36_210 and a data connection 36_212, such as a wireless connection circuit (e.g., bluetooth, Wi-Fi, etc.) or a wired connection circuit (e.g., ethernet, etc.).

The headset interface 36_210 allows a set of headsets to be coupled to the device. The signal may be generated when the headset is coupled, for example by making an electrical connection when plugged into the headset interface 36_ 210. In a more complex embodiment, the headset interface 36_210 may include circuitry that provides an identification signal that identifies the type of headset interface of the device. Events may be detected in various ways and signals may be generated and/or transmitted in various ways.

The data connection 36_212 may be communicatively coupled with an external device, for example, by a wireless connection. For example, the bluetooth connection may be coupled to a computer of a vehicle or a computer of a wireless headset. Thus, when an external device is coupled to the mobile device via the data connection 36_212, it may be determined that the external device is connected and a corresponding device connection event signal may be generated.

2. User interaction events

The user interaction input device 36_206 may be used to detect user interaction events. A user interaction event may occur when a user interacts with a device. In some embodiments, the user may directly activate the displayed user interface via one of the user interaction input devices 36_ 206. In other embodiments, the user interface may not be displayed, but the user may still access it, for example via the user shaking the device or providing some other type of gesture. Further, the interaction may not include a user interface, for example, when the state engine uses values from sensors of the device.

Any suitable device component of the user interface may be used as the user interaction input device 36_ 206. Examples of suitable user interaction input devices are a button 36_214 (e.g., Home button or power button), a touch screen 36_216, and an accelerometer 36_ 218. For example, a button 36_214 of the mobile device, such as a Home button, a power button, a volume button, etc., may be the user interaction input device 36_ 204. Further, a switch, such as a mute mode switch, may be the user interaction input device 36_ 204. When a user interacts with the device, it may be determined that the user has provided user input, and a corresponding user interaction event may be generated. Such events may depend on the current state of the device, e.g., when the device is first turned on or activated in the morning (or other long inactivity). Such information may also be used when determining whether a certain time is a triggering event.

The touch screen 36_216 may allow a user to provide user input via the display screen. For example, the user may swipe his or her finger across the display to generate the user input signal. When the user performs the action, a corresponding user interaction event may be detected.

The accelerometer 36_218 or other motion sensor may be a passive component that detects movement (e.g., shaking and tilting) of the mobile device (e.g., using a gyroscope or compass). Such movement of the mobile device may be detected by the event manager 36_230, which may determine that the movement is of a particular type. The event manager 36_230 may generate an event signal 36_232 corresponding to a particular type of user interaction event in a given state of the device. The state of a device may be determined by a state engine, further details of which may be found in U.S. patent publication 2012/0310587 entitled "Activity Detection" and U.S. patent publication 2015/0050923 entitled "detecting Exit From A Vehicle," both of which are incorporated herein by reference in their entirety.

One example is that while the user is running, an accelerometer may sense shaking and generate a signal to be provided to event manager 36_ 230. Event manager 36_230 may analyze the accelerometer signal to determine the type of event. Having determined the type of event, the event manager 36_230 may generate an event signal 36_232 corresponding to the event type. The mobile device may move to indicate that the user is running. Thus, the particular user interaction may be identified as a running event. The event manager 36_230 may then generate and transmit an event signal 36_232 indicating that a running event has been detected.

3. Location events

The location input device 36_208 may be used to generate location events. Any suitable positioning system may be used to generate the location events. For example, a Global Positioning System (GPS) may be used to generate the location event. The location event may be an event corresponding to a particular geographic location. For example, if the mobile device arrives at a particular location, the GPS component may generate an input signal corresponding to the location event. Typically, mobile devices may move to tens or even hundreds of locations per day, many of which may not be important enough to be considered a location event. Thus, not every detected location will be a location event. In an embodiment, a location event may be a location that occurs more frequently than other events. For example, an event may be a location event if it occurs frequently at least a threshold number of times over a period of time, e.g., five times over a span of six months to one year. Thus, important locations may be distinguished from unimportant locations and determined as location events.

B. Determining a trigger event

As further shown in fig. 36_2, the input device 36_202 may output the detected event 36_222, for example, as a result of any corresponding event. The detection event may include information about which input device is sending a signal for detecting event 36_222, a subtype for a particular event (e.g., which headset type or data connection type). Such information may be used to determine whether the detected event 36_222 is a triggering event, and such information may be passed to a later module for determining which predictive model to use or which action to perform for the proposed application.

The detection event 36_222 may be received by the event manager 36_ 230. The event manager 36_230 may receive signals from the input device 36_202 and determine what type of event was detected. Depending on the type of event, the event manager 36_230 may output a signal (e.g., event signal 36_232) to a different engine. Different engines may have a subscription to the event manager 36_230 to receive specific event signals 36_232 that are important to their function. For example, the trigger event engine 36_224 may be subscribed to receive the event signal 36_232 generated in response to the detection event 36_222 from the input device 36_ 202. The event signal 36_232 may correspond to the type of event determined from the detection event 36_ 222.

The trigger event engine 36_224 may be configured to determine whether the detection event 36_222 is a trigger event. To make this determination, the trigger event engine 36_224 may reference a specified trigger event database 36_226, which may be coupled to the trigger event engine 36_ 224. The designated trigger event database 36_226 may include a list of predetermined events designated as trigger events.

The trigger event engine 36_224 may compare the received detection event 36_222 to a list of predetermined events and output a trigger event 36_228 if the detection event 36_222 matches a predetermined event listed in a designated trigger event database 36_ 226. For example, the list of predetermined events may include one or more of the following: (1) inserting a headset into a headset interface, (2) connecting an external device via a bluetooth connection, (3) pressing a button after a period of time has elapsed (e.g., when waking up in the morning), (4) sensing certain types of motion of the device, and (5) reaching a certain location. For (5), the specified trigger event database 226 may include specifications for a particular location.

As described in this section, one aspect of the present technology is to collect and use data from a variety of sources to suggest applications to a user. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or may be used to contact or locate a particular person. Such personal information data may include location-based data, home address, or any other identifying information.

The present disclosure recognizes that the use of such personal information data in the present technology may be useful to benefit the user. For example, the personal information data may be used to suggest applications that are of greater interest to the user. Thus, using such personal information data enables planned control of delivered content. In addition, the present disclosure also contemplates other uses for which personal information data is beneficial to a user.

The present disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with established privacy policies and/or privacy practices. In particular, such entities should enforce and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining privacy and security of personal information data. For example, personal information from a user should be collected for legitimate and legitimate uses by an entity and not shared or sold outside of these legitimate uses. In addition, such collection should only be done after the user has informed consent. In addition, such entities should take any required steps to secure and protect access to such personal information data, and to ensure that others who have access to the personal information data comply with their privacy policies and procedures. In addition, such entities may subject themselves to third party evaluations to prove compliance with widely accepted privacy policies and practices.

Regardless of the foregoing, the present disclosure also contemplates embodiments in which a user selectively prevents use or access to personal information data. That is, the present disclosure contemplates that hardware elements and/or software elements may be provided to prevent or block access to such personal information data. For example, the user may choose not to provide location information for the targeted content delivery service. As another example, the user may choose not to provide accurate location information, but to permit transmission of location area information.

IX. determination of suggested applications

After detecting the trigger event, the application may be identified based on the trigger event. In some embodiments, the identification of the application is not a preprogrammed action. Instead, the identification of the application may be a dynamic action that may change based on the additional information. For example, the identification of suggested applications may be determined based on contextual information and/or historical information, as well as based on other information.

A. System for determining applications based on trigger events

Fig. 36_3 shows a simplified block diagram of a prediction system 36_300 for identifying an application and a corresponding action command based on a triggering event and context information, according to an embodiment of the present invention. Prediction system 36_300 resides within a device that identifies an application. Prediction system 36_300 may include hardware and software components.

Prediction system 36_300 includes a prediction engine 36_302 for identifying suggested applications. The prediction engine 36_302 may receive a trigger event, such as the trigger event 36_228 discussed in fig. 36_ 2. The prediction engine 36_302 may use information collected from the triggering events 36_228 to identify suggested applications 36_ 304. As shown, the prediction engine 36_302 may receive context data 36_306 in addition to the triggering event 36_ 228. The prediction engine 36_302 may use information collected from both the triggering event 36_228 and the context information 36_306 to identify the suggested application 36_ 304. Prediction engine 36_302 may also determine actions to perform, for example, how and when a user interface may be provided for a user interacting with the suggested application.

In some embodiments, the suggested application 36_304 may be any application that exists on a solid state drive associated with the mobile device. Thus, the prediction engine 36_302 may have the ability to suggest any application when a triggering event is detected. Alternatively, in an embodiment, the prediction engine 36_302 may have the ability to suggest less than all applications when a triggering event is detected. For example, the user may select certain applications to be inaccessible to prediction engine 36_ 302. Thus, prediction engine 36_302 may not be able to suggest these applications when a triggering event is detected.

1. Context information

Context information may be collected from the context data 36_ 306. In an embodiment, the context information may be received at any time. For example, the context information may be received before and/or after detecting the trigger event 36_ 228. Additionally, context information may be received during the detection of the trigger event 36_ 228. The context information may specify one or more attributes of the device for a particular context. The context may be the ambient environment of the device (type of context) when the trigger event 36_228 is detected. For example, the context information may be the time at which the trigger event 36_228 was detected. In another example, the context information may be a particular location of the device when the trigger event 36_228 is detected. In yet another example, the context information may be a particular date when the trigger event 36_228 is detected. Further, the contextual information may be data collected from a calendar. For example, the amount of time between the current time and the event time (e.g., days or hours). Such context information may provide more meaningful information about the context of the device so that prediction engine 36_302 may accurately suggest applications that the user may use in that context. Thus, the prediction engine 36_302 using the context information may suggest applications to the user more accurately than if the context information was not used.

The context data 36_306 may be generated by a context source 36_ 308. The context source 36_308 may be a component of the mobile device that provides data related to the current situation of the mobile device. For example, the context source 36_308 may be a hardware device and/or software code that functions as an internal digital clock 36_310, a GPS device 36_312, and a calendar 36_314, respectively, for providing information related to the time of day, device location, and date of year. Other context sources may be used.

The context data 36_306 for the prediction engine 36_302 may be collected in a power efficient manner. For example, continuously polling the GPS 36_312 to determine the location of the device may be overly power hungry, which may shorten battery life. To avoid shortening battery life, the prediction engine 36_302 may determine the location of the device by requesting the location of the device from a source other than the GPS 36_ 312. Another source of location information may be an application that has recently polled the GPS 36_312 for device location. For example, if application a is the application that most recently polled GPS 36_312 for device location, prediction engine 36_302 may request and receive location data from application a, rather than polling GPS 36_312 alone.

2. Historical information

In certain embodiments, the historical event database 36_316 may also be used by the prediction engine 36_302 in addition to the context source 36_ 308. The historical event database 36_316 may include historical information of previous interactions between the user and the mobile device after the trigger event is detected.

The historical event database 36_316 may record the number of times an application is opened after a certain trigger event. For example, database 36_316 may record: after the headset is plugged into the headset interface, the user has opened application a eight of the ten times. Thus, the prediction engine 36_302 may receive this information as historical data 36_318 to determine whether application a should be identified for the user when a set of headphones is plugged into the headphone interface.

The historical event database 36_316 may also record the number of times an application is opened in different contexts when a triggering event is detected. For example, database 36_316 may record: when the user is at home, application a is opened nine out of ten times by the user after the headset is plugged into the headset interface, and when the user is at a gym, application a is opened one out of ten times by the user. Thus, prediction engine 36_302 may receive this information as historical data 36_318 and determine that application a should be identified when the headset is inserted into the device at home rather than at a gym. It should be appreciated that although the embodiments discussed in this section refer to a location as a "home" or "gym," the context data 36_306 representing a "home" or "gym" may be in the form of digital coordinates. Those skilled in the art will appreciate that information related to the time of day and the date of the year may be utilized in a similar manner in place of the location to identify other applications.

The historical event database 36_316 may also record the frequency with which a user decides not to run an identified application and under what circumstances such a decision is made. For example, database 36_316 may record: when the user inserts the headset into the device at home, the user has not selected application B suggested to the user twice out of ten times. Thus, the prediction engine 36_302 may receive this information as historical data 36_318 to adjust the probability that application B is suggested when the user inserts headphones into the device at home.

In some embodiments, the context information 36_306 and/or historical information (discussed further in this section) may not be available or limited when a trigger event is detected. In this case, the default application may be suggested upon detection of the triggering event. The default application may be a type of application that is typically associated with a type of triggering event. For example, if a set of headphones is plugged into the headphone interface, a music application may be suggested. Alternatively, the map application may be suggested when a bluetooth connection is made with the car. After obtaining more historical information, the suggested application may be provided instead of the default application.

B. Multiple prediction models

Since different trigger events may result in different suggested applications, embodiments may use different predictive models for different trigger events. In this way, the predictive model can be improved to provide more accurate recommendations for specific triggering events.

Fig. 36_4 shows the prediction engine 36_302 in more detail according to an embodiment of the invention. The prediction engine 36_302 may be program code stored on a memory device. At an embodiment, prediction engine 36_302 includes one or more prediction models. For example, prediction engine 36_302 may include prediction models 1 through N. Each predictive model may be a portion of code and/or data specifically designed to identify an application for a particular trigger event 36_ 228. For example, predictive model 1 may be specifically designed to identify an application for a triggering event in which a set of headphones are plugged into a headphone interface. The predictive model 2 may be designed to identify an application for a triggering event when a bluetooth device is connected.

The predictive model 3 may be designed to identify applications for trigger events where a user interacts with the user interface of the device after a long period of time (e.g., when the user first interacts with the mobile device after waking up in the morning). Other predictive models may be designed to identify applications for triggering events associated with particular detected motion patterns (e.g., when a user is running with a mobile device), reaching a particular location, and selecting a particular application (e.g., selecting an application that communicates with a computer of an automobile). Any number of prediction models may be included in the prediction engine 36_302 depending on the number of triggering events 36_ 228.

As shown, each predictive model 1 through N may be coupled to a context source and a historical event database to receive context data 36_306 and historical data 36_ 318. Thus, according to embodiments discussed in this section, each predictive model may use context data 36_306 and historical data 36_318 to identify suggested applications 36_ 304.

Referring again to fig. 36_3, the prediction engine 36_302 may send the suggested applications 36_304 to the expert center module 36_ 320. In an embodiment, the event manager 36_320 may be part of code that manages how content is displayed on the device (e.g., on a lock screen, when a search screen is opened, or other screen). For example, the expert center 36_320 may coordinate which information is displayed to the user, such as suggested applications, suggested contacts, and/or other information. The expert center 36_320 may also determine when to provide such information to the user.

X. user interface

If the event manager 36_320 determines that this is an opportunity to output the suggested application to the user, the event manager 36_320 may output the application 36_322 to the user interface 36_ 324. In an embodiment, the output application 36_322 may correspond to the suggested application 36_ 304. The user interface 36_324 may communicate the output application 36_322 to the user and request a response from the user regarding the output application 36_ 322.

In an embodiment, the user interface 36_324 may be a combination of device components with which a user may interact. For example, the user interface 36_324 may be a combination of device components having the ability to output information to a user and/or allow a user to input signals to the device.

A. Display device

The user interface 36_324 may be displayed on a display of the device. The display may be sensitive to touch such that input signals may be generated through physical interaction with the display. In such an embodiment, the display may include a touch sensitive layer superimposed on the image display layer to detect user touches of the display. Thus, the display may be part of the user interface 36_324 capable of outputting information to and inputting information from a user. For example, the display may display an icon for a suggested application and input a signal to run the application when the user taps a corresponding position of the display panel.

Modern devices have security measures to prevent unauthorized use of the device. Such devices may require that the user unlock the device first in order for the user to have access to all applications stored on the device. The device may limit the accessibility of all applications depending on the state of the device's security. For example, a device may require a user to unlock the device first to allow the device to allow access to all of its applications. The unlocked device may have a display that displays a home screen. The home screen may display and/or provide access to all applications of the device. However, the locked device may have a display that displays the lock screen. Certain areas of the display may be occupied by prompts for unlocking the device. Thus, the lock screen may allow interaction with fewer applications than the home screen due to the increased state of device security and limited display space. For example, the lock screen may only allow access to less than all applications of the device, such as one to three applications. In some embodiments, the suggested applications 36_304 discussed in this section with respect to fig. 36_3 may be displayed on the lock screen.

B. Other input and output device parts

While the display may be part of the user interface 36_324 that is capable of outputting information to and inputting information from a user, other parts of the user interface 36_324 are not so limited. For example, other device components that can input information from a user are also contemplated in embodiments of this section. For example, buttons and switches may be part of the user interface 36_ 324. The button may be a device component that generates an input when a user applies pressure thereto. The switch may be a component of the device that generates an input when the user turns the lever to another position. Thus, according to the embodiments discussed in this section, the user may activate buttons and/or switches to run the suggested application 36_ 304.

Device components that can output information from a user are also contemplated in embodiments of this section. For example, a speaker or haptic device may be part of a user interface that outputs information to a user. The speaker may output an audio notification to indicate that the identified application has been suggested. The haptic device may output a haptic notification to indicate that the identified application has been suggested. It should be understood that such an apparatus is merely an embodiment, and other embodiments are not limited to such an apparatus.

C. Level of interaction

The user interface 36_324 may require different levels of interaction for the user to run the output application 36_ 322. The respective levels may correspond to a degree of probability that the user will run the suggested application 36_ 304. For example, if the prediction engine 36_302 determines that the proposed application 36_304 has a probability of being run by the user that is greater than a threshold probability, the user interface 36_324 may output a prompt that allows the user to run the application faster by skipping intermediate steps.

For example, if the prediction engine 36_302 determines that the user's probability of running the suggested music application is greater than a high threshold probability, the suggested music application may be automatically run, and the user interface 36_324 may thus display controls for the music application, such as play, pause, and fast forward/rewind. Thus, the user may not have to perform a click to run an intermediate step of the application.

Alternatively, if the prediction engine 36_302 determines that the user has a probability of running the music application that is less than a high threshold probability but still higher than a lower threshold probability, the music application may be displayed as an icon. The lower threshold probability may be higher than the baseline threshold probability. The baseline threshold probability may establish a minimum probability that the corresponding application will be suggested. Thus, the user may have to perform the additional step of clicking on the icon to run the suggested music application. However, the number of clicks may still be less than that required when the application is not being suggested to the user. In an embodiment, the threshold probability may vary depending on the type of application. In various embodiments, the high threshold probability may be in a range between 75% and 100%, the lower threshold probability may be in a range between 50% and 75%, and the baseline threshold may be in a range between 25% and 50%. In a particular embodiment, the high threshold probability is 75%, the lower threshold probability is 50%, and the baseline probability is 25%.

In embodiments, a higher probability may result in more aggressive application recommendations. For example, if an application has a high probability of about 90%, prediction engine 36_302 may provide an icon on the lock screen of the device to allow the user to access the application by clicking the icon once. If the application has a higher probability of about 95%, prediction engine 36_302 may even automatically run the suggested application for the user without the user clicking on anything. In such cases, prediction engine 36_302 may output not only the suggested application, but also commands specific to that application, such as a command to play selected music in a music application or a command to initiate guidance for a particular route in a map application.

According to an embodiment of the invention, the prediction engine 36_302 may determine what level of interaction is required and then output this information to the event manager 36_ 320. The event manager 36_320 may then send this information to the user interface 36_324 for output to the user.

In an embodiment, the user interface 36_324 may display a notification to the user on a display screen. The notification may be sent by, for example, a push notification. The notification may be a visual notification that includes a picture and/or text that informs the user of the suggested application. The notification may suggest an application to the user for the user to select and run at their leisure. After being selected, the application may be run. In some embodiments, the notification may also include suggested actions within the suggested application for more aggressive prediction. That is, the notification may notify the user of the suggested application and the suggested action within the suggested application. Accordingly, the user may be provided with the option to run the suggested application or to perform the suggested action within the suggested application. For example, the notification may inform the user that the suggested application is a music application and that the suggested action is playing a certain song in the music application. The user may indicate that he or she wants to play the song by clicking on the icon showing the suggested song. Alternatively, the user may indicate that he or she prefers to run the application to play another song by swiping the notification across the screen.

In addition to outputting suggested applications and suggested actions to the user interface 36_324 in one notification, the prediction engine 36_302 may output two suggested actions to the user interface 36_324 in one notification. For example, prediction engine 36_302 may output one suggested action to play a first song and a second suggested action to play a second song. The user can select which song to play by clicking on the corresponding icon in the notification. In an embodiment, the suggested actions may be scaled based on different criteria. For example, one suggested action may be for playing a recently played song regardless of the contextual information, while another suggested action may be for playing a song that was last played under the same or similar contextual information. For example, for the case where the user enters his or her car and a triggering event causes prediction engine 36_302 to suggest two actions related to playing a certain song, Song A may be the last song played at home, and Song B may be the last song played by the user in the car. When the user selects a song to play, the song may continue from the beginning, or from where it last stopped (e.g., in the middle of the song).

To enable prediction engine 36_302 to suggest actions, prediction engine 36_302 may access a memory device that stores information about the active state of the device. The active state of the device may represent an action performed after the suggested application is selected. For example, an active state for a music application may be playing a certain song. The active state may record the time the song last stopped. In an embodiment, the history database 36_316 of fig. 36_3 may record historical data related to the activity state of the device. Thus, prediction engine 36_302 may suggest actions to be run by the suggested application.

XI. method of determining suggested applications

Fig. 36_5 is a flow diagram of a method 36_500 of identifying an application based on a triggering event, according to an embodiment of the invention. The method 36_500 may be performed in whole or in part by a device. As various examples, the device may be a cell phone, tablet, laptop, or other mobile device as already discussed in this section.

At block 36_502, a device (e.g., mobile device) detects an event. For example, a set of headphones may be plugged into the headphone interface of the device. As another example, a wireless headset may be coupled to a device via a bluetooth connection. The input device 36_202 in fig. 36_2 may be used to detect events. An event may be any action by the mobile device that interacts with an external entity, such as an external device or a user.

At block 36_504, the device determines whether the detection event is a triggering event. To determine whether a detected event is a triggering event, the detected event may be compared to a predetermined event list, such as the event list in the designated triggering event database 36_226 in FIG. 36_ 2. If the detection event matches one of the predetermined event list, the detection event may be determined to be a trigger event.

At block 36_506, the device selects a predictive model, such as one of predictive models 1 through N in fig. 36_ 4. The selected predictive model may depend on the triggering event. For example, when the triggering event involves establishing a bluetooth connection with an external device, a predictive model designed for the bluetooth connection may be selected. As another example, when the triggering event involves inserting a set of headphones into the headphone interface, a predictive model designed for headphone connection may be selected.

At block 36_508, the device receives context information. The context information may be received from various sources, such as the context source 36_308 in FIG. 36_ 3. In an embodiment, the context information may relate to the surrounding situation of the device. For example, the contextual information may relate to the time of day, day of the year, or location of the device. Further, the device may also receive history information. The historical information may be related to a history of interactions between the device and the user (e.g., historical events database 36_316) stored in the database.

At block 36_510, the device may identify one or more applications that are accessed by the user with a probability of at least a threshold probability. As already mentioned in this section, there may be multiple thresholds. In some embodiments, the threshold probability may be a baseline threshold probability, a lower threshold probability, or a high threshold probability. For example, one or more applications may each have a probability greater than a threshold probability. In another example, one or more applications may have a combined probability greater than a threshold probability. The one or more applications may be the application with the highest probability and may be selected with various criteria (e.g., all having a probability greater than a threshold, as many applications above a threshold but limited to a maximum number, as desired, etc.). In some embodiments, applications with probabilities less than the baseline threshold probability may be ignored.

The probability of being visited by the user may be determined by a predictive model. The predictive model may determine this probability by utilizing context information as well as historical information. In an embodiment, the identified application is the application discussed in this section with respect to fig. 36_3 and 36_ 4.

In some embodiments, if the applications have equal probabilities, they may be ignored, i.e., not identified. In these cases, the device may need to generate additional historical information to correctly identify the one or more applications. The more historical information that is collected, the more accurate the device can be in identifying the correct application, e.g., an application that is desired to be accessed by the user in a given context. In other embodiments, two applications may be provided, for example, if their combined probability is sufficiently high, such as would occur if the two applications had the highest two probabilities.

At block 36_512, the device may provide a user interface to the user. For example, the device may display the identified application to the user via an interface that the user may interact with to indicate whether the user wishes to access the identified application. For example, the user interface may include a touch-sensitive display that displays one or more of the identified applications to the user and allows the user to access the one or more applications identified by the device by interacting with the touch-sensitive display.

In some embodiments, the user interface may be provided in a lock screen or home screen. The home screen may be a screen displayed after pressing the home button in the unlocked state. The lock screen may be a screen that is displayed after pressing the home button to wake up the device after a long period of inactivity. In an embodiment, the lock screen has less available display space for displaying the application than the home screen because a portion of the lock screen is reserved for unlocking the device. In some embodiments, the user interface may be associated with an application that is already running. For example, the user interface may be a music player interface having audio controls associated with a running music application, as shown in fig. 36_ 6.

Fig. 36_6 shows an exemplary user interface 36_600 for the device 36_602 associated with an already running application. The user interface 36_600 may be a user interface for a music application, although other user interfaces for different applications are also contemplated in this section. The user interface 36_600 may be provided by a touch screen display 36_ 604. The touch screen display 36_604 may display audio controls 36_608, volume controls 36_610, song titles 36_612, and/or album art 36_ 614. Audio controls 36_608 may provide a user interface for fast forwarding, fast rewinding, playing, and pausing songs. The volume control 36_610 allows the user to adjust the volume of the output sound. The song title 36_612 and album art 36_614 may display information about the currently playing song. In an embodiment, the backlight of the device 36_602 may be lit up when the user interface 36_600 is displayed by the touch screen display 36_ 604. The illumination of the backlight allows the user to see the running application and to note that the device 36_602 has run the proposed application. By automatically running a music application and providing the user with a user interface 36_600, the device 36_602 can enhance the user experience by allowing the user to access his or her desired application without having to click on one or more icons.

Some portions of the user interface 36_600 may be hidden in some cases. For example, if an expert center of the device 36_602, such as the expert center 36_320 in FIG. 36_3, decides that another application has precedence over the suggested application, the album art 36_614 may be hidden and may instead be displayed. Other applications may be displayed on the display 36_604 as accessible icons for running the other application. In other embodiments, the other application may be displayed as a notification, and when the user clicks on the notification, access to the icon of the other application is allowed. In such a case, the notification would be displayed in place of the album art 36_ 614. In an embodiment, if the user interface is displayed on the lock screen, the notification may also be displayed on the lock screen. Thus, the user may be made aware of and given the opportunity to run applications that are deemed to have a higher priority.

XII time limit for running applications

In an embodiment, if the identified application is not accessed within a certain period of time, the device may remove the user interface as if the user interface was not initially provided. If the user does not access the application for a certain period of time, it is assumed that the user is not interested in accessing the application. Thus, the user interface is removed so that the user cannot access the identified application and the user is not distracted.

Fig. 36_7A and 36_7B are flowcharts illustrating a method for removing a user interface according to an embodiment. In particular, fig. 36_7A is a flow chart illustrating a method 36_700 of removing a user interface after a period of time has elapsed. FIG. 36_7B is a flow diagram illustrating a method 36_703 for removing a user interface after a triggering event has been removed within a threshold period of time. The methods 36_700 and 36_703 may be performed in whole or in part by the device.

Referring to fig. 36_7A, the method 36_700 begins with providing a user interface to a user at block 36_ 701. Block 36_701 may be performed as described at block 36_512 discussed in this section with respect to fig. 36_ 5.

At block 36_702, the device determines whether a threshold period of time has elapsed since the user interface was first provided to the user. The user interface may be provided to the user in a lock screen or home screen. In an embodiment, the threshold time period represents a predetermined time period that begins immediately after the user interface is provided to the user without the user interacting with the device.

The threshold time period may vary depending on the type of triggering event. For example, if the triggering event is of an event type involving direct user interaction (e.g., a cognitive action performed by a user expected to cause the event), the threshold time period may be relatively short, such as 15 to 30 seconds. One example of such a triggering event includes inserting a set of headphones into the headphone interface. Another example includes pressing a button to wake up the device after a relatively long period of time. The threshold period of time may be relatively short because it may be assumed that the user is interacting directly with the phone and may immediately know the identified application that is output. Since the user immediately knows the identified application, a short period of time without the identified application being accessed will indicate that the user does not intend to access the identified application.

Alternatively, if the trigger event is of an event type that does not involve direct user interaction, the threshold time period may be longer than the threshold time period for trigger events involving direct user interaction. In one embodiment, the threshold time period for a triggering event that does not involve direct user interaction may be relatively long, such as 15 to 30 minutes. One such example includes reaching a location. When the device arrives at a particular location, it is assumed that the user is traveling and is not paying attention to the device. The user may not immediately know the identified application that is output. Thus, a longer time may elapse before the user checks the device and knows the identified application.

At block 36_704, if the threshold period of time has elapsed, the user interface may be removed so that the user may not realize that the application is suggested at all. If, however, the threshold time period has not elapsed, at block 36_706, the device determines whether the user wishes to access the application. The user may indicate by any form of user input via the user interface that he or she wants to access the application, such as by interacting with a touch screen, pressing a button, toggling a switch, or using a biometric device.

If it is determined that the user has not indicated that he or she would like to access the application, the device may continue to provide a user interface to the user at block 36_ 701. However, if the device receives an indication that the user wants to access the application, the device may run the application at block 36_ 708. Thus, the device may save the user time by providing shortcuts to the desired applications, thereby enhancing the user experience.

In some embodiments, the user interface may be removed before the duration of the threshold time period. As shown in fig. 36_7B, at block 36_710, the device determines whether the trigger event has been removed, e.g., an action opposite to the trigger event has been detected. For example, if the triggering event is the insertion of a set of headphones into the headphone interface, the removal of the triggering event is the unplugging of the set of headphones from the headphone interface. In another example, if the triggering event is the establishment of a bluetooth connection, the removal of the triggering event is the disconnection of the bluetooth connection. The removal of the trigger event may be interpreted by the device as indicating that the user does not intend to access the suggested device. Thus, if the trigger event is removed, the user interface may be removed at block 36_704, e.g., the application may be cleared and any user interface for the application may be hidden.

Xiii. training routine

As historical information accumulates through use of the mobile device, predictive models (e.g., predictive models 1 through N discussed in fig. 36_ 4) may be trained (i.e., updated) periodically in view of new historical information. After training, the predictive models 1 through N may suggest applications and actions more accurately based on the latest interaction pattern between the user and the mobile device. Training predictive models 1 through N may be most efficient when a large amount of historical information is recorded. Thus, training may occur at sufficiently long time intervals to allow the mobile device to detect a large number of interactions with the user. However, waiting too long between training sessions may hamper the fitness of the prediction engine. Thus, a suitable time period between training sessions may be between 15 and 20 hours, such as 18 hours.

Training predictive models 1 through N may take time and may interfere with the use of the mobile device. Thus, training may occur when the user is least likely to use the device. One way to predict that the user will not use the device is to wait for a certain period of time when the device is not being used, for example when no buttons are pressed and when the device is not moving. This may indicate that the user is in a state where the user will not interact with the handset for a short period of time, such as when the user is asleep. Any suitable duration may be used as the period to wait, such as one to three hours. In a particular embodiment, the period of waiting is two hours.

At the end of these two hours, the predictive models 1 through N may be updated. However, if the user interacts with the mobile device (e.g., presses a button or the mobile device) before the end of the two hours, the two hour period countdown may be restarted. If the time period is continuously restarted before two hours of inactivity are reached, the mobile device may force training predictive models 1 through N after some absolute time period. In one embodiment, the absolute time period may be determined as a threshold time period for which the user-friendliness of the mobile device begins to decline due to the expiration of the predictive model. This absolute time period may be between 10 and 15 hours, or in particular embodiments 12 hours. Thus, the maximum time between training may be between 28 hours (18+10 hours) and 33 hours (18+15 hours). In a particular embodiment, the maximum time is 30 hours (18+12 hours).

In some embodiments, the software components of device 100 (fig. 1A) include a trigger event module and a prediction module. The trigger event module may include various sub-modules or systems, for example, as described in this section with respect to fig. 36_ 2. Further, the prediction module may include various sub-modules or systems, for example, as described in this section with respect to fig. 36_ 3.

Exemplary methods, apparatus, and computer-readable mediums for application suggestion based on detected trigger events Quality of food

In some embodiments, the event may be detected by an input device. An event may be determined to be a triggering event by comparing the event to a set of triggering events. A first predictive model corresponding to the event is then selected. Context information is then received for the device specifying one or more attributes of the computing device in the first context, and a set of one or more applications is identified. The probability that the set of one or more applications are accessed by the user when the event occurs in the first context may be at least a threshold probability. Thereafter, a user interface is provided for interacting with the set of one or more applications.

In some embodiments, there is provided a computer-implemented method for providing a user interface to a user for interacting with a suggestion application executing on a computing device, the method comprising: at a computing device: detecting an event at an input device of a computing device; determining that the event corresponds to one of a set of trigger events specified for identifying one or more suggested applications; selecting a first predictive model corresponding to the event; receiving context information about the computing device, the context information specifying one or more attributes of the computing device for a first context; identifying, by a first predictive model, a set of one or more applications that are accessed by a user with a probability of at least a threshold probability when the event occurs in association with a first context, the first predictive model using historical interactions of the user with the computing device after detecting the event; a user interface is provided to the user for interacting with the set of one or more applications. In some embodiments, detecting an event at an input device of a computing device comprises: a connection of a computing device with an accessory device is detected. In some embodiments, the accessory device includes a headset or a computer of the vehicle. In some embodiments, the context information specifies a location of the computing device. In some embodiments, the user interface allows interaction on a screen with fewer applications than are provided on the home screen of the computing device. In some embodiments, detecting an event at an input device of a computing device comprises: detecting motion of the computing device using one or more motion sensors; and determining a motion state of the computing device based on the movement, wherein the designation of the set of trigger events is to identify one of the one or more suggested applications that includes the motion state of the computing device. In some embodiments, the second predictive model is selected when another trigger event is detected, the other trigger event being different from the event, wherein the second predictive model is different from the first predictive model. In some embodiments, the set of one or more applications comprises a plurality of applications, and wherein the set of one or more applications as a whole has a probability greater than a threshold probability. In some embodiments, a user interface is provided on a lock screen of the computing device that allows selection of one of the set of applications from the lock screen. In some embodiments, the method comprises: the set of one or more applications is run, and the user interface is specific to the running one or more applications. In some embodiments, the one or more attributes comprise at least one of: a location of the computing device, a time of day determined by the computing device, and a date of year determined by the computing device. In some embodiments, the method comprises: determining whether a threshold period of time has elapsed; removing the user interface when it is determined that the threshold period of time has elapsed; determining whether the user is attempting to access the set of one or more applications when it is determined that the threshold time period has not elapsed; and running the set of one or more applications when it is determined that the user is attempting to access the set of one or more applications. In some embodiments, the threshold time period for a trigger event involving direct user interaction is greater than the threshold time period for a trigger event not involving direct user interaction.

In some embodiments, a computer product comprising a non-transitory computer readable medium stores a plurality of instructions that, when executed, control a device comprising one or more processors, the instructions comprising: detecting an event at an input device of a device; determining that the event corresponds to one of a set of trigger events designated for identifying one or more suggested applications; selecting a first predictive model corresponding to the event; receiving context information about a device, the context information specifying one or more attributes of the device for a first context; identifying, by a first predictive model, a set of one or more applications having a probability of being accessed by a user when the event occurs in a first context of at least a threshold probability, the first predictive model using historical interactions of the user with the device at the time the event was detected; a user interface is provided to the user for interacting with the set of one or more applications. In some embodiments, detecting an event at an input device of the device comprises: a connection of a computing device with an accessory device is detected.

In some embodiments, there is provided an apparatus comprising: a trigger event store for storing trigger events; a history storage area for storing history data; one or more input devices; one or more context sources; and one or more processors configured to: detecting events at one or more input devices; determining that the events correspond to one of a set of trigger events designated for identifying one or more suggested applications; selecting a first predictive model corresponding to the event; receiving context information about the device from one or more context sources, the context information specifying one or more attributes of the computing device for a first context; identifying, by a first predictive model, a set of one or more applications that are accessed by a user with a probability of at least a threshold probability when the event occurs in a first context, the first predictive model using historical interactions of the user with the computing device at the time the event was detected; a user interface is provided to the user for interacting with the set of one or more applications. In some embodiments, the one or more input devices include at least one of a headset interface, a bluetooth device, a button, a touch screen, an accelerometer, and GPS. In some embodiments, the triggering event is a predetermined event. In some embodiments, the user interface allows for interaction with fewer applications than are provided on the home screen of the computing device. In some embodiments, the set of one or more applications includes a plurality of applications, and wherein each of the plurality of applications has a probability greater than a threshold probability.

Section 7: person-centric prediction/techniques for suggesting recipients based on device context

The material in this section, "person-centric prediction" describes person-centric predictions and techniques that suggest recipients based on device context according to some embodiments, and provides information that complements the disclosure provided herein. For example, a portion of this section describes ways to identify and predict contacts and recommend those contacts for use by a user, which complements the disclosure provided herein, such as those related to methods 600 and 800 discussed below, and particularly to filling the prediction portion 930 of fig. 9B-9C with suggestions. In some embodiments, the predictive models and historical interaction databases described in this section for assisting in predicting and suggesting contacts are used to assist in identifying appropriate contacts for prediction, suggestion, and/or inclusion in a user interface, such as a search interface or lock screen, for immediate use by a user (i.e., these predictive models are used in conjunction with methods 600,800,1000 and 1200 to suggest/predict contacts).

Brief summary of human-centric predictions

Embodiments suggest recipients of communications and interactions that are most likely to be relevant to a user of a computing device based on the current context of the device. Examples of computing devices are phones, tablets, laptops or desktop computers. An exemplary system collects knowledge of previous interactions and suggests predicted recipients based on this knowledge. This knowledge may be stored in a historical interaction database along with information indicating when, where, and how the user interacted with other users. The system may recommend recipients (e.g., people) and mechanisms to interact with them given a particular context. Context may be described in terms of state variables indicating time, location, and account identifier (e.g., an email account). Context may also be based on keywords (e.g., keywords from an email topic or calendar event topic) as well as other factors such as, for example, a set of recipients with which the user has previously interacted. Additional restrictions may be imposed to help narrow suggestions for particular users, accounts, applications (e.g., communication applications), or interaction mechanisms.

Embodiments may provide systems, methods, and apparatuses for suggesting one or more recipients to contact a computing device based on events and context. Exemplary events include receiving an input to initiate a search, receiving an input to access an email application, composing an email, receiving an input to access a text messaging application, composing a text message, receiving an input to access a calendar application, creating a calendar entry, editing a calendar entry, initiating a telephone call, initiating a video call, and initiating a video conference. Exemplary contexts include location and time. Implementations may predict the recipient of a communication based on the context of the device that the user is using to initiate or compose the communication (e.g., at home, commute, at business, etc.). For example, the recipient for the communication is predicted based on information about the communication (e.g., whether the communication is an email, instant message, text message, video conference, or calendar invitation). The recipient for the communication is also predicted based on previous communications. For example, a user or contact with which the user has previously interacted via email, message, or call may be suggested as a recipient for the communication.

Embodiments may provide a method for suggesting recipients to contact by using contextual information to predict people with whom a user may want to interact at a certain time and place. Some embodiments determine a current context that represents a current state when a user of a device (e.g., a mobile device) composes or initiates a communication in an application program. In embodiments, the current context may include context information such as time, location, next calendar entry, title or subject of the communication (e.g., email subject or calendar entry title), last recipient of similar communication, and account information (e.g., personal email account or work email account). Some implementations use the current context to predict the most likely recipient that the user will add as a recipient of the communication.

Other embodiments relate to systems, portable consumer devices, and computer-readable media associated with the methods described herein.

The nature and advantages of embodiments of the present invention may be better understood by reference to the following detailed description and accompanying drawings.

Detailed description of human-centric prediction

Embodiments may provide a customized and personalized experience for suggesting recipients to a user of a computing device, thereby utilizing the device to more easily interact and communicate with other users. Embodiments may provide a method for suggesting recipients to contact using human-centric prediction. Human-centric prediction uses contextual information to predict people with whom a user may want to interact at a certain time and place. A user of a computing device may interact and communicate with a set of other users (e.g., contacts). Examples of computing devices are phones, tablets, laptops or desktop computers. Interactions and communications with other users may occur after a particular event. Exemplary events include initiating a search, accessing a communication application, and composing or initiating a communication. Exemplary communication applications include email applications, calendar applications, video call applications, instant messaging applications, text messaging applications, video conferencing applications, web conferencing applications, and voice call applications. Exemplary communications include voice and data communications, such as, for example, email messages, calendar invitations, text messages, instant messages, video calls, voice calls, and video conferences. When using a communication application on a device, recipients of the communication may be suggested based on comparing the current context of the device with the historical information.

In an embodiment, data from past historical interactions is stored in a table of a database and used to suggest recipients of the communication. The database may include contextual information of past interactions, such as, for example, timestamps, applications used for the interactions, account information (e.g., account identifiers for email accounts), and locations. Past interactions may be compared to the context of the device to suggest recipients of the communication initiated on the device. For example, the current context of the device may be compared to historical interaction data to match the current context to similar past interactions of previous recipients.

Each data point (e.g., record) in the historical data may correspond to a particular context (e.g., to one or more attributes of the device), and over time, more and more data for the particular context is obtained. Historical data for a particular event may be used to suggest recipients to the user. Embodiments may provide a personalized experience since different users will have different historical data.

In some embodiments, recipients for previous similar communications are used to suggest recipients for the communication being composed or initiated. For example, if a user selects a first recipient for a current communication, additional recipients for the current communication may be predicted using other recipients added to past communications with the selected first recipient. In one embodiment, the recipient may be suggested based on periodic context data indicative of the interaction (e.g., a communication repeatedly sent on a similar event of the day or the same day of the week). Recipients may also be suggested based on location information indicating that the user's current location is similar to the location where the user was when past communications were sent to certain contacts.

In an embodiment, the recipient may be predicted using information provided by the user. The information provided by the user may include the subject of the email, the content of the email, the calendar entry title, the time of the event, and/or the recipient selected by the user. Such user-provided information may be compared to historical context information to predict recipients. For example, recipients of past communications having characteristics similar to the information provided by the user may be presented to the user as suggested recipients of the current communication. Some embodiments may use information that the user has entered a communication (e.g., if the user has included a theme or an add-on) to determine that such information is relevant to the identification of the potential recipient. For example, embodiments may parse the subject matter of an email message or calendar entry to identify one or more keywords that may be relevant to suggesting potential recipients when such information is available.

To provide an accurate personalized experience, various embodiments may begin with a broad model that is simply trained to provide no recipient suggestions, or to suggest the same set of recipients for various contexts. With enough historical data, the broad model can be segmented into sub-models, e.g., as a set of people or interactions, each sub-model corresponding to a different subset of the historical interaction data. Then, when an event does occur, a particular sub-model may be selected to provide one or more suggested recipients corresponding to the current context of the device. Various criteria may be used to determine when to generate a sub-model, such as the confidence level in the sub-model that provides the correct prediction in the historical data subset, and the information gain (entropy reduction) in the historical data distribution relative to the parent model.

Thus, some embodiments may decide when and how to segment the historical data in the context of recipient recommendations. For example, after collecting a period of user interaction activity, embodiments may accumulate a list of possible segmentation candidates (e.g., location, day of week, time of day, etc.). Embodiments may also train the model over the entire data set and compute a confidence measure for the joint distribution of the data set and the model. A set of models may be trained, one for each segmented data set (i.e., subset), and then the confidence of each data model distribution measured. If the confidence of all data model distributions is acceptable, then the embodiment may perform segmentation (segmentation) and then recursively examine the segmented space for additional segmentation.

In this way, some embodiments may use inference to explore the trade-off between segmentation and generalization, create more complex models for users with clearer and complex patterns, and create simple general models for users with noisier and simple patterns. Also, some embodiments may generate a probabilistic model tree based on determining a divergence distribution among the potential candidate models.

I. Suggesting recipients based on events

Embodiments may suggest one or more recipients based on events that may be limited to some predetermined event (also referred to as a triggering event). Exemplary triggering events may include initiating a search, composing an email, creating a calendar entry, and so forth. For example, when a user begins composing an email using an email account, the user may be advised of the contacts to whom the user previously sent the email using the email account. In some embodiments, the context information may be used in conjunction with the event to identify the recipient to suggest to the user. For example, contextual information related to a location may be used when a calendar entry (e.g., calendar event, meeting, or appointment) is being created or modified. If the device is at an office location, for example, recipient A, who has an office at that location, may be suggested as an invitee to the calendar event. Alternatively, if the device is at home, recipient B (i.e., a family member or a roommate) associated with the home location may be suggested as an invitee to the calendar entry. Thus, recipients predicted to be relevant in certain contexts may be suggested at appropriate times, thereby enhancing the user experience. As another example, time-related context information may be used when opening a calendar entry for creation or modification. Recipient a, who is a co-worker, may be suggested as an invitee to the calendar event if the scheduled start time of the calendar entry corresponds to the user's typical work hours. Alternatively, if the calendar entry has a start time corresponding to evening or weekend, recipient B, who is a friend or family member, may be suggested as an invitee to the calendar event.

Fig. 37_1 is a flow diagram of a method 37_100 for suggesting recipients based on detecting an event, according to an embodiment of the invention. Method 37_100 may be performed by a mobile device (e.g., phone, tablet computer) or a non-mobile device, and using one or more user interfaces of the device.

At block 37_102, a user input at the user device is detected. In some embodiments, it may be determined whether the input corresponds to a triggering event for suggesting a recipient. In some implementations, one or more suggested recipients are determined only for certain predetermined events (e.g., triggering events). In other implementations, one or more suggested recipients may be determined for a dynamic event list, which may be updated based on historical user interactions with the user device.

In some embodiments, the trigger event may be identified as being sufficiently likely to be associated with the operation of the communication application of the device. An event list of trigger events may be stored on the device. Such events may be a default list and maintained as part of the operating system, and may or may not be configured by the user.

The trigger event may be an event caused by a user and/or an external device. For example, the triggering event may be when an input is received at the mobile device. Examples include receiving input to initiate a search, receiving input to access a communication application, and so forth. In this example, each of these events may be classified as a different triggering event. As other examples, the triggering event may be a particular interaction of the user with the device. For example, a user may initiate a search on the device, access a communication application on the device, or begin composing a communication message on the device. Further, for example, a user may move the mobile device to a work site, in which case the location state of the device is a triggering event. Such a location state (or other state) may be determined based on sensors of the device.

At block 37_104, context information representing the current state of the device is determined. In one example, the context information may indicate an application executing on the device. For example, the context information may indicate a state of a communication application for initiating the communication. The context information may also indicate a state of a search application used to initiate the search. For example, block 37_104 may include determining a time, account information (e.g., an email account identifier), and/or location corresponding to a communication application being used on the device. Block 37_104 may also include determining a sub-state of the device, the sub-state being an application state of the executing application. For example, the application state may indicate a state of an email application for composing email messages, a state of a calendar application for creating calendar events, a state of an instant messaging client for initiating instant messages, a state of an application for composing text messages, or a state of an application for initiating phone calls, video calls, or video conferences.

The context information may specify one or more attributes of the device for a particular context. The context may be the surrounding environment of the device (type of context) when the trigger event is received. For example, the context information may be the time of day at which the event was detected. In another example, the context information may be a particular location of the device when the event is detected. In yet another example, the context information may be a particular date when the triggering event was detected. Such contextual information may provide more meaningful information about the context of the device so that the suggestion engine may accurately suggest recipients that the user may select in that context. Thus, a prediction engine using contextual information may suggest recipients to a user more accurately than if the contextual information was not used.

At block 37_106, historical data representing past interactions between the user and other users is retrieved. The retrieval is based on the context information. For example, block 37_106 may include retrieving data corresponding to past emails, messages, phone calls, calendar entries, video calls, and video conferences. The historical data may be retrieved from tables corresponding to previous communications made using the user device, where each table corresponds to a different device sub-state of the user device and includes a plurality of contact metrics for previous communications of different recipients. For example, block 37_106 may include using one or more state variables to identify a first set of tables corresponding to the one or more state variables and then obtaining contact metrics for one or more potential recipients from the first set of tables.

At block 37_108, the context information is compared to historical data. Block 37_108 may include querying the first set of tables identified at block 37_106 to determine a correlation between the historical data and the contextual information in the set of tables.

At block 37_110, one or more recipients for the communication are predicted. As shown in fig. 37_1, the recipient is predicted based on the comparison performed at block 37_ 108. For example, when composing an email using a work email account during work while the user device is in a work location, a previously used contact with a work email address may be identified as the predicted recipient. In some embodiments, more than one recipient may be identified.

Block 37_110 may use a prediction engine or prediction model to identify the predicted recipient. For example, a predictive model may be selected for a particular triggering event. The predictive model may use context information to identify the recipient, e.g., interactions or communications with different recipients may be more likely to occur in different contexts. Some embodiments may suggest a recipient only if there is a sufficient probability that the user selects the suggested recipient (e.g., as determined by the user's historical interactions with the recipient while using the device). Examples of historical interactions may include at least partial communications exchanged by a user with a recipient using an email application, text messaging (e.g., SMS-based messaging), an instant messaging application, and a video conferencing application.

In some embodiments, social elements based on past communications and interactions may be used to predict recipients. For example, the historical data obtained at block 37_106 may be used to weight the recipients of previously sent electronic mail pieces. The social elements reflect historical interaction data between a user of the user device and a group of past recipients (e.g., contacts and contact groups) of the user's communication. Co-occurrence (i.e., communications sent to the same group of recipients) can be used to predict email recipients. For example, the social element may weight each recipient to whom the user has sent an email, with a higher weight assigned to recipients that have been repeatedly included in a group of recipients (e.g., a carbon copy list or a defined group of contacts). The recipients may be uniquely identified in the history data by their respective email addresses. The social element weight may be higher for a sent email than for a received email. The social elements may also be weighted based on the email account (e.g., personal account or work account) that the user has used to send emails. When the contextual information indicates that an email is being composed, social elements may be used to identify co-occurrences of recipients of past emails. These co-occurrences may in turn be used to predict the recipients of the email being composed, particularly when the user selects one of a group of recipients that have been included in past email messages.

At block 37_112, an indication of one or more predicted recipients is provided to the user. Block 37_112 may include presenting the list of one or more predicted recipients in a user interface of the user device or in a communication application executing on the user device. In some embodiments, the action may be performed at block 37_112 in association with the executing application. In one embodiment, the action may be displaying a user interface for the user to select one or more predicted recipients. The user interface may be provided in various ways, such as by being displayed on a screen of the device, projected onto a surface, or provided as an audio interface.

In other embodiments, an application may be run and a user interface specific to the application may be provided to the user. Any of the user interfaces may be provided in response to identifying a recipient, such as a potential recipient of the communication. In other implementations, a user interface to interact with the application may be provided after the user is authenticated (e.g., by password or biometric identification), but such a user interface would be more specific than a home screen only, such as an interface with a list of suggested recipients.

Initiating a receiver predicted event

The trigger event may be a set of predetermined events that trigger the identification of one or more recipients to be provided to the user. The event may be detected using a signal generated by a device component. Details of how the trigger event is detected will be discussed in more detail below with reference to fig. 37_ 2.

Fig. 37_2 shows a simplified block diagram of a detection system 37_200 for determining a trigger event according to an embodiment of the present invention. Detection system 37_200 may reside within the device for which the trigger event is being determined. As shown, detection system 37_200 may detect a number of different events. One or more of the detected events may be determined by detection system 37_200 as a triggering event. Other processing modules may then perform processing using the triggering event.

A. Detecting events

In an embodiment, detection system 37_200 includes hardware and software components for detecting a triggering event. For example, the detection system 37_200 may include a plurality of input devices, such as input device 37_ 202. The input device 37_202 may be any suitable device capable of generating a signal in response to an event. For example, the input device 37_202 may include a user interaction input device 37_204 and a position input device 37_206 that may detect a device connection event, a user interaction event, and a position event, respectively. When an event is detected at the input device, the input device may send a signal indicating the particular event for further analysis.

In some embodiments, a collection of components may participate in a single event. For example, a person may be detected as being on-duty or off-duty based on a motion sensor, a GPS location device, and a timestamp.

1. User interaction events

The user interaction input device 37_204 may be used to detect user interaction events. A user interaction event may occur when a user interacts with a device. In some embodiments, the user may provide input to the displayed user interface of the application via one of the user interaction input devices 37_ 204. In other embodiments, the user interface may not be displayed, but the user may still access it, such as via the user shaking the device or providing some other type of gesture. Further, the interaction may not include a user interface, for example, when the state engine uses values from sensors of the device.

Any suitable device component of the user interface may be used as the user interaction input device 37_ 204. Examples of suitable user interaction input devices are button 37_208 (e.g. home button or power button), touch screen 37_210, camera 37_212, accelerometer 37_214, microphone 37_216 and mouse 37_ 218. For example, a button 37_208 of the mobile device, such as a home button, a power button, a volume button, etc., may be the user interaction input device 37_ 204. Further, a switch such as a silent mode switch may be the user interaction input device 37_ 204. Also, for example, a microphone 37_216 of the mobile device, such as an integrated microphone configured to detect voice commands, may be the user interaction input device 37_ 204. As another example, a mouse 37_218 or pointing device, such as a stylus, may be the user interactive input device 37_204 for providing user input to the communication application.

When a user interacts with the device, it may be determined that the user has provided user input to the application, and a corresponding trigger event may be generated. Such events may depend on the current state of the device, such as where the device is located or when the event occurred. That is, the trigger event may be generated based in part on input from the user interaction input device 37_204 in conjunction with the location state of the device (e.g., at work location) and the temporal context (e.g., weekday morning). Such information may also be used when determining whether a certain time is a triggering event.

The touch screen 37_210 may allow a user to provide user input via the display screen. For example, the user may swipe his or her finger across the display to generate the user input signal. When the user performs this action, a corresponding trigger event 37_228 may be detected.

The accelerometer 37_218 or other motion sensor may be a passive component that detects movement (e.g., shaking and tilting) of the mobile device (e.g., using a gyroscope or compass). Such movement of the mobile device may be detected by the event manager 37_230, which may determine that the movement is of a particular type. The event manager 37_230 may generate an event signal 37_232 corresponding to a particular type of user interaction event in a given state of the device. The state of a device may be determined by a state engine, further details of which may be found in U.S. patent publication 2012/0310587 entitled "Activity Detection" and U.S. patent publication 2015/0050923 entitled "detecting Exit From A Vehicle," both of which are incorporated herein by reference in their entirety.

One example is that when the user is running, an accelerometer may sense shaking and generate a signal to be provided to the event manager 37_ 230. The event manager 37_230 may analyze the accelerometer signal to determine the type of event. Having determined the type of event, the event manager 37_230 may generate an event signal 37_232 corresponding to the event type. The mobile device may move to indicate that the user is running. Thus, the particular user interaction may be identified as a running event. The event manager 37_230 may then generate and transmit an event signal 37_232 indicating that a running event has been detected.

2. Location events

The location input device 37_206 may be used to generate location events. The location event may be used in conjunction with a user interaction event to trigger recipient suggestions. Any suitable positioning system may be used to generate the location events. For example, a Global Positioning System (GPS) may be used to generate the location event. The location event may be an event corresponding to a particular geographic location. For example, if the mobile device arrives at a particular location, the GPS component may generate an input signal corresponding to the location event.

B. Determining a trigger event

As further shown in fig. 37_2, the input device 37_202 may output the detected event 37_222, for example, as a result of any corresponding event. The detection event may include information about which input device is sending a signal for detecting event 37_222, a subtype for a particular event (e.g., which headset type or data connection type). Such information may be used to determine whether a detected event 37_222 is a triggering event, and such information may be passed to a later module for determining which predictive model to use or which action to perform for the suggested recipient (e.g., compose an email, create a calendar invitation, initiate a voice call or video call).

The detection event 37_222 may be received by the event manager 37_ 230. The event manager 37_230 may receive a signal from the input device 37_202 and determine what type of event was detected. Depending on the type of event, the event manager 37_230 may output a signal (e.g., event signal 37_232) to a different engine. Different engines may have a subscription to the event manager 37_230 to receive specific event signals 37_232 that are important to their function. For example, the trigger event engine 37_224 may be subscribed to receive the event signal 37_232 generated in response to the detection event 37_222 from the input device 37_ 202. The event signal 37_232 may correspond to the type of event determined from the detection event 37_ 222.

The trigger event engine 37_224 may be configured to determine whether the detection event 37_222 is a trigger event. To make this determination, the trigger event engine 37_224 may reference a specified trigger event database 37_226, which may be coupled to the trigger event engine 37_ 224. The specified trigger event database 37_226 may include a list of predetermined events specified as trigger events.

The trigger event engine 37_224 may compare the received detected event 37_222 with a list of predetermined events and output a trigger event 37_228 if the detected event 37_222 matches a predetermined event listed in a designated trigger event database 37_ 226. For example, the list of predetermined events may include one or more of the following: (1) accessing a communication application, (2) initiating a search, (3) composing a communication, (4) sensing a particular type of movement of a device, and (5) arriving at a particular location. For (5), the specified trigger event database 37_226 may include specifications for a particular location. For each of the predetermined events (1) through (5), the time or time range at which the event occurred may be included in the specified trigger event database 37_ 226. For example, the designated trigger event database 37_226 may store designated trigger events corresponding to sensed arrival at a work location between 8 and 10 am.

Determination of suggested recipients

Upon detecting a trigger event, one or more potential recipients may be identified based on the trigger event. In some embodiments, the identification of the recipient is not a preprogrammed action. Instead, the recipient's identification may be a dynamic action that may change based on additional information. For example, the identity of the suggested recipient may be determined based on contextual information and/or human-centric historical interaction information, as well as based on other information.

Each time a particular triggering event occurs (e.g., accessing an email client on the device, a calendar application, an instant messaging application, or a video conferencing application), the device may track which recipient(s) are selected as recipients for the communication associated with the event. In response to each occurrence of a particular event, the device may save data points corresponding to the selected recipient, the interactions with the recipient performed using the application, and the event. In various embodiments, data points may be saved separately or aggregated with a count determined for the number of times a particular recipient was selected, where the count may include a count for a particular action. For example, a count indicating the number of emails sent to the recipient may be kept along with information indicating which email account was used to send these emails, the time at which the emails were sent, and the location of the device at the time the emails were sent. In this embodiment, the data point may also indicate the number of times the recipient is the first recipient of the email, the number of times the recipient is included as part of the email group or correspondence group list, and the number of times the recipient is copied (e.g., carbon copy/CC or sealed carbon copy/BCC). Thus, different counts are determined for different actions of the same selected recipient.

Historical data indicating previous interactions and communications of the user with the recipient may be used as input to a predictive model that predicts whether a given recipient should be suggested as a recipient for future communications. For example, historical data used to predict/suggest recipients may include a record of past interactions (i.e., historical interactions) with other users. Examples of such historical interactions include voice calls, emails, calendar entries/events, instant messages, text messages (e.g., SMS-based messages), video conferences and video calls. For example, the historical interactions may include a call history indicating the time, duration, and recipient (identified by a phone number) corresponding to a past voice call. Further, for example, the history interaction can include an email history indicating a time, periodicity (e.g., daily, weekly), and recipient (identified by an email address) corresponding to past email messages.

Once a particular event is detected, a predictive model corresponding to the particular event may be selected. The predictive model will be determined using historical interaction data corresponding to a particular event as input to a training process. However, historical data may occur in many different contexts (i.e., different combinations of context information), where different recipients are selected in different contexts. Thus, in general, historical interaction data may not suggest recipients that will be explicitly selected by the user at the time of a particular event.

The predictive model may correspond to a particular event. One or more attributes of the computing device may be used to determine a suggested recipient to contact. For example, a particular sub-model may be generated from a subset of historical data corresponding to user interactions with other users after an event occurs. This subset of historical interaction data may be collected when the device has one or more attributes (e.g., user interaction with a selected recipient after an event accessing the email application, attributes having a particular location and/or time of day). The predictive model may be composed of sub-models, each for a different combination of context data. Different combinations may have different amounts of context data. These sub-models may be generated in a hierarchical tree, with the more specific combined sub-models being located lower down in the hierarchical tree. In some embodiments, a sub-model is generated only if it can predict the recipient more accurately than models higher up in the tree. In this way, a more accurate prediction of which application the user will select can be made. In some embodiments, the predictive model and submodel may identify the top N recipients (e.g., a fixed percentage) selected by the user after the event when there is a particular combination of context data.

Models such as neural networks or regression may be trained to identify a particular application for a particular context, but this may be more difficult when all of the corresponding historical data is used. Using all historical interaction data may result in overfitting of the predictive model and reduced accuracy. Embodiments of the present invention may segment historical data into different input sets of historical data, each input set corresponding to a different context. Different submodels may be trained for different sets of historical data inputs.

A. Different models based on different context data

When a particular event occurs, the device may be in various contexts, for example, at different locations (such as at work, at home, or at school), at different times, on different days of the week (such as weekdays or weekends), in different states of motion of the device (such as running, walking, driving, or standing still), or in different states of use of the communication application (such as for composing an email or creating a calendar entry). The context information may be retrieved in conjunction with the detected event, e.g., after the event is detected. Contextual information may be used to help predict which predicted recipients are likely to be selected as recipients of a communication associated with a detected event. A GPS sensor may be used to determine different locations and the time may be determined based on when previous communications were sent. Motion sensors such as accelerometers, gyroscopes or GPS sensors may be used to determine the different motion states.

Embodiments may use the context information in various ways. In one example, a piece of context data (e.g., corresponding to one attribute of the device) may be used to predict which recipient is most likely to be selected. For example, a particular location of the device may be provided as an input to the predictive model.

As another example, some or all of the context data of the context information may be used in the segmentation process. The input historical data may be segmented using a piece of context data such that only historical data corresponding to corresponding attributes of the piece of context data is used to determine a particular sub-model. For example, a specific location of the device is not used as an input for a submodel, but for selecting the submodel to be used, and which input data to use accordingly to generate a specific submodel.

Thus, in some embodiments, particular context data may be used to identify which sub-model to use, and other context data may be used as input to the sub-model for predicting with which recipient a user is likely to interact. A particular attribute (e.g., a particular location) does not correspond to a particular submodel, which may be used as a submodel for future (input) use. If the particular attribute corresponds to a particular sub-model, the usage of the attribute may become richer because the entire model is dedicated to the particular attribute.

One drawback to dedicating a submodel to a particular attribute (or combination of attributes) is that there may not be a large amount of historical data corresponding to that particular attribute. For example, a user may perform a particular event (e.g., compose an email) only a few times at a particular location. This limited amount of data is also referred to as data sparseness. When a combination of attributes is used (e.g., a particular location at a particular time), the data may become more sparse. To overcome this drawback, embodiments may selectively determine when to generate a new sub-model as part of the segmentation process.

1. Default model

The default model may be used when the device is first obtained (e.g., brought up) by the user. The default model may be applied to a set of events (e.g., all events specified as trigger events). The default model may be seeded with aggregated data from other devices associated with the user. In some embodiments, the default model may simply choose the most popular recipients regardless of context, for example, because there is insufficient data available for any one context. After data is collected again, the default model may be discarded.

In some embodiments, the default model may have hard-coded logic that specifies the intended recipient to suggest and the action to perform. In this way, it can be detected how the user responds (e.g., a negative response is that the user does not select a suggested recipient), which can provide additional data that simply tracks the positive response used. Similar to such a default model, a predictive model may be run to compare its prediction to actual results. The predictive model may then be refined based on the actual results. When the predictive model has sufficient confidence, a switch may be made from the default model to the predictive model. Similarly, the performance of the submodels may be tracked. When a sub-model has sufficient confidence, the sub-model can be used for a given context. In some embodiments, there are different submodels for different events. For example, an email sub-model may be used in the email context to predict recipients of the email, and a separate calendar sub-model may be used to predict invitees to a calendar event. These different submodels may use data from corresponding tables in the historical interaction database to identify recipients of previous email and calendar invitations. In this embodiment, the email table may have records of past email messages that indicate recipients that the user previously added to the message. Similarly, the calendar in the historical interaction database may have records of past calendar events that indicate users who were invited to the calendar event.

2. Initial training

A predictive model (e.g., an event model) may be initially trained using historical data collected so far, where the model does not provide recipient suggestions to the user. This training may be referred to as initial training. The predictive model may be updated periodically (e.g., daily) as part of a background process, which may occur when the device is charged and not in use. Training may involve optimizing the coefficients of the model to optimize the number of correct predictions and comparing them to actual results in historical interaction data. In another example, training may include identifying the top N (e.g., a predetermined number or percentage) of applications actually selected. After training, the accuracy of the model may be measured to determine whether the model should be used to provide suggested recipients (and potentially corresponding interaction types) to the user.

Once the model obtains sufficient accuracy (e.g., the top selected application is selected with sufficiently high accuracy), the model may be implemented. For a top-level model (e.g., the first event model), this may not happen, but may happen when a sub-model is tested for a particular context. Thus, such initial training may be similarly performed on the sub-models.

B. Segmentation when more data is acquired

When a user first begins using the device, there will be no historical interaction data predicting recipients with which the user may choose to interact after a particular event (e.g., after accessing an email application, a calendar application, a video conferencing application, or a calendar application). In the initial mode, historical interaction data may be obtained without suggesting a predicted recipient. As more historical data is obtained, it may be determined whether to segment the predictive model into sub-models. With more historical interaction data, the sub-model may be segmented into more sub-models. When limited historical data is available for user interaction with the recipient, the recipient is not advised, or a more general model may be used.

The segmentation process may be performed by a user device (e.g., a mobile device, such as a smartphone), which may maintain data privacy. In other embodiments, the segmentation process may be performed by a server in communication with the user device. The segmentation process may be performed partially over a period of time (e.g., days or months), or the entire segmentation process may be performed together and may be periodically resumed. The segmentation process may be performed as a routine of a recipient prediction engine.

As more data is collected, the predictive model may be segmented into several sub-models. At different points in the collected data, segmentation may occur. As more data is acquired, another segmentation may occur. Each segment may involve a full redo segment, which may or may not result in the same submodel being created as in the previous segment.

In this embodiment, the first event model may correspond to a particular event (e.g., sending an email to a particular contact, such as a colleague). The event model may correspond to a top level of a prediction engine for the particular event. Initially, there can only be one model for this particular event, since there is a minimum of historical interaction data available. At this point, the event model may simply track historical data for training purposes. The event model may make recipient predictions and compare those predictions to actual results (e.g., whether the user selected a suggested recipient to interact with within a specified time after the event was detected). If no recipient has a probability greater than the threshold, then no recipient will be advised when the particular event occurs.

In some embodiments, the event model uses only data collected for a particular device. In other embodiments, the event model may be seeded with historical interaction data aggregated from other devices associated with the user. Such historical interaction data may allow the event model to provide some recipient suggestions, which may then allow additional data points to be obtained. For example, it may be tracked whether a user interacts with a suggested recipient via a particular application (e.g., email, audio call, video conference, instant message, or text message), which may provide more data points than whether the user actually selected the recipient to provide.

As more data is collected, it may be periodically determined whether segmentation should be performed. Such a determination may be based on whether a higher accuracy may be obtained by segmentation. This accuracy may be measured as a level of probability that a prediction may be made, as described in more detail below. For example, segmentation may be performed if the recipient can be predicted to have a higher level of sub-model probability than the event model. One or more other criteria may also be used to determine whether a sub-model should be created as part of the segmentation process. For example, one criterion may be that a submodel must have a statistically significant amount of input history data before the submodel is applied. Since models trained on a small amount of data may not be accurate, the requirement for the amount of data may provide greater stability for the sub-models, and ultimately more accurate data.

C. System for suggesting recipients based on triggering events

Fig. 3 shows a simplified block diagram of a prediction system 37_300 for identifying a recipient and a corresponding action command based on a triggering event and context information, according to an embodiment of the present invention. Prediction system 37_300 resides within the device that identifies the application. Prediction system 37_300 may include hardware and software components.

The prediction system 37_300 includes a prediction engine 37_302 for identifying suggested recipients. The prediction engine 37_302 may receive a trigger event. Prediction engine 37_302 may use information collected from triggering event 37_328 to identify suggested recipients 37_ 304. As shown, in addition to the triggering event 37_328, the prediction engine 37_302 may also receive context data 37_ 306. Prediction engine 37_302 may use information collected from both triggering event 37_328 and context data 37_306 to identify suggested recipients 37_ 304. In an embodiment, based on the received context data 37_306, the prediction engine 37_302 uses different models to identify the proposed recipients for different types of communications. For example, prediction engine 37_302 may use an email sub-model when context data 37_306 indicates that an email application is being accessed or an email is being composed. The email model sub-model may use such context data 37_306 in conjunction with historical email data from the historical event database 37_316 to predict email recipients. An email sub-model may be used to predict recipients of an email, and a separate calendar sub-model may be used to predict invitees to a calendar event. The prediction engine 37_302 may also determine the actions to be performed, e.g., how and when a user interface may be provided for a user interacting with the proposed recipient.

1. Context information

Context information may be collected from the context data 37_ 306. In an embodiment, the context information may be received at any time. For example, the context information may be received before and/or after the trigger event 37_328 is detected. Additionally, context information may be received during the detection of the trigger event 37_ 328. The context information may specify one or more attributes of the device for a particular context. The context may be the ambient environment of the device (type of context) when the trigger event 37_328 is detected. For example, the context information may be the time at which the trigger event 37_328 was detected. In another example, the context information may be a particular location of the device when the triggering event 37_328 is detected. In yet another example, the context information may be a particular date when the triggering event 37_328 is detected. Such contextual information may provide more meaningful information about the context of the device so that prediction engine 37_302 may accurately suggest that the user is likely to select a recipient in that context as the recipient. Accordingly, the prediction engine 37_302 using the context information can suggest the recipient to the user more accurately than without using the context information.

Context data 37_306 may be generated by context source 37_ 308. The context source 37_308 may be a component of the mobile device that provides data related to the current situation of the mobile device. For example, context source 37_308 may be a hardware device and/or software code that functions as an internal digital clock 37_310, a GPS device 37_312, and a calendar 37_314, respectively, for providing information related to the time of day, device location, and date of year. Other context sources may be used.

The context data 37_306 for the prediction engine 37_302 may be collected in a power efficient manner. For example, continuously polling the GPS 37_312 to determine the location of the device may be overly power hungry, which may shorten battery life. To avoid shortening battery life, the prediction engine 37_302 may determine the location of the device by requesting the location of the device from a source other than the GPS 37_ 312. Another source of location information may be an application that has recently polled GPS 37_312 for device location. For example, if application a is the application that most recently polled GPS 37_312 for device location, prediction engine 37_302 may request and receive location data from application a, rather than polling GPS 37_312 alone.

2. Historical information

In certain embodiments, the historical event database 37_316 may also be used by the prediction engine 37_302 in addition to the context source 37_ 308. The historical event database 37_316 may include historical information of previous interactions between the user and the mobile device after the trigger event is detected.

The historical event database 37_316 may record the number of user interactions with the recipient after a certain triggering event. For example, the database 37_316 may record: when recipient a is included, eight out of ten users include recipient B in the email or calendar invitation. Thus, the prediction engine 37_302 may receive this information as historical data 37_318 to determine whether recipient a should be identified for the user when recipient a is selected for email or calendar communication.

The historical event database 37_316 may also record the number of interactions with the recipient in different contexts when a triggering event is detected. For example, the database 37_316 may record: when the user is at home, nine out of ten times the user interacts with recipient a after the user accesses the personal email account, and when the user is at a work location, one out of ten times the user interacts with recipient a and uses the work email account. Thus, the prediction engine 37_302 may receive this information as historical data 37_318 and determine that recipient a should be suggested when the user accesses a personal email account at home, but not when accessing a work email account at work. It should be appreciated that although the examples discussed in this section refer to a location as "home" or "work," the context data 37_306 representing "home" or "work" may be in the form of digital coordinates, such as geographic coordinates. Those skilled in the art will appreciate that information related to the time of day, day of week, and date of year may be used in a similar manner in place of location to identify the recipient.

The historical event database 37_316 may also record the frequency with which the user decides not to select the identified recipient as the recipient for communication and under what circumstances such a decision is made. For example, the database 37_316 may record: when the user inserts the headset into the device at home, the user has not selected recipient B as the recipient of the telephone call twice out of ten. Thus, the prediction engine 37_302 may receive this information as historical data 37_318 to adjust the probability of suggesting recipient B when the user inserts headphones into the device at home.

As described above, one aspect of the present technology is to collect and use data from a variety of sources to improve predictions of users with whom the users may be interested in interacting. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or may be used to contact a particular person. Such personal information data may include location-based data, telephone numbers, email addresses, work addresses, home addresses, past interaction records, or any other identifying information.

The present disclosure recognizes that the use of such personal information data in the present technology may be useful to benefit the user. For example, personal information data may be used to predict users who may want to communicate at a certain time and place. Therefore, it is possible to predict, with human centers, persons with whom a user may want to interact at a specific time and place using such personal information data included in the context information.

The present disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with established privacy policies and/or privacy practices. In particular, such entities should enforce and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining privacy and security of personal information data. For example, personal information from a user should be collected for legitimate and legitimate uses by an entity and not shared or sold outside of these legitimate uses. In addition, such collection should only be done after the user has informed consent. Furthermore, such entities should take any required steps to secure and protect access to such personal information data, and to ensure that others who have access to the personal information data comply with their privacy policies and procedures. In addition, such entities may subject themselves to third party evaluations to prove compliance with widely accepted privacy policies and practices.

Regardless of the foregoing, the present disclosure also contemplates embodiments in which a user selectively prevents use or access to personal information data. That is, the present disclosure contemplates that hardware elements and/or software elements may be provided to prevent or block access to such personal information data. For example, in the case of a human-centric predictive service, the techniques of the present invention may be configured to allow a user to opt-in to "join" or "opt-out of" participating in the collection of personal information data during registration with the service. As another example, the user may choose not to suggest to the recipient that the service provide location information. As another example, the user may choose not to provide accurate location information, but to permit transmission of location area information.

D. User interface

Fig. 37_4 and 37_5 illustrate exemplary user interfaces for presenting a list of suggested recipients. In particular, fig. 37_4 shows an exemplary user interface 37_400 for device 37_402 associated with an already running e-mail application. The user interface 37_400 may be a user interface for a client-side email application, although other user interfaces for different applications are also contemplated in this section. For example, the user interface 37_400 may be the user interface of any application that may be used to interact with a recipient, such as an instant messaging application, a video conferencing application, and a calendar application. The user interface 37_400 may be provided by a touch screen display 37_ 404. The touch screen display 37_404 may display an email interface 37_406 that includes a subject line 37_ 408. The subject line 37_408 may allow the user to enter the subject of the email message. Suggested recipients list 37_410 allows the user to select one or more suggested recipients. As shown, embodiments may present suggested recipients in the search results window 37_410 based on the context of the device 37_402 without any subject matter or title being entered in the subject line 37_ 408. Such a zero keyword search may be performed by the device 37_402 based on the current context of the device 37_ 402. For example, based on the current location of the device 37_402, the current time, as an email account identifier for use on the device 37_402, and other contextual information, suggested email recipients may be determined and displayed in the search results window 37_410 without relying on providing a complete or partial subject of the email message in the subject line 37_ 408. In some embodiments, when the user enters a topic (or portion thereof) on the topic row 37_408, the suggested recipient list 37_410 may be updated based on the keywords in the topic row.

Some portions of the user interface 37_400 may be hidden in some cases. For example, if a suggestion center in device 37_402, such as suggestion center 37_320 in fig. 37_3, determines that another recipient (e.g., recipient B shown in fig. 37_ 4) has precedence over a first suggested recipient (e.g., recipient a shown in fig. 37_ 4), the first recipient may be hidden and the other recipient may be turned to be displayed. The other recipient may then be first displayed in the suggested recipient list 37_410 on display 37_ 404. Thus, the user may be made aware of and given the opportunity to interact with recipients that are deemed to have a higher priority. In the exemplary embodiment of fig. 37_4, the user does not provide any input regarding the recipient in the interface 37_ 400. As shown, embodiments may present suggested recipients in suggested recipient list 37_410 based on the context of device 37_402 without entering any recipients (or partial recipient names) in interface 37_ 400. That is, suggested recipients may be identified and displayed in suggested recipient list 37_410 without using any auto-completion techniques to predict contacts based on partially entered contact names or email addresses. Such a zero keyword search may be performed by the device 37_402 based on the current context of the device 37_ 402. For example, based on the current location of the device 37_402, the current time, and other contextual information, such as an email account (e.g., work or personal email account) used on the device 37_402, suggested email recipients may be determined and displayed in the search results window 37_ 410.

Fig. 37_5 shows a user interface 37_500 for the device 37_502 associated with a search application that has already been run. The user interface 37_500 may be provided by a touch screen display 37_ 504. The touch screen display 37_504 may display a search interface 37_506 including a search window 37_ 508. Search window 37_508 may allow a user to enter one or more search terms. Search results window 37_510 may present suggested recipients. In the embodiment of fig. 37_5, the user does not provide any keywords in the search window 37_ 508. As shown, embodiments may present suggested recipients in the search results window 37_510 based on the context of the device 37_502 without entering any search terms or keywords in the search window 37_ 508. Such a zero keyword search may be performed by the device 37_502 based on the current context of the device 37_ 502. For example, based on the current location of the device, the current time, and other contextual information, such as a user account identifier used on the device 37_502, the suggested recipients may be determined and displayed in the search results window 37_ 510. The user may then interact with search results window 37_510 to select one or more suggested recipients. In an embodiment, when a user enters a search term (or portion thereof) in search window 37_508, the suggested recipients in search results window 37_510 may be updated based on the search term.

In some embodiments, the search results window 37_510 may be more than the exemplary contact list shown in fig. 37_ 5. For example, search results window 37_510 may include information indicating how the user interacted with the suggested recipient. The search results window 37_510 may also indicate why an interaction should occur. For example, search results window 37_510 may suggest that the user initiate a video call to recipient a based on the recipient's personal account using the user's personal account, as the user often does so during this time of day. The suggestion may even suggest using a particular communication application to contact recipient a.

E. Method of producing a composite material

FIG. 37_6 is a flow diagram of a method 37_600 of suggesting one or more recipients to a user of a computing device based on an event, according to an embodiment of the invention. The method 37_600 may be performed by a computing device (e.g., by a user device that is tracking user interactions with the user device). The method 37_600 may suggest recipients using a set of historical interactions, including interactions having different sets of one or more attributes of the computing device.

At block 37_602, the device detects an event at the input device. As shown, block 37_602 may include detecting a user input at a user device associated with the user. Examples of input devices are a touch screen, a microphone for providing voice commands, a camera, buttons, a mouse, a stylus, a keyboard, etc. An event may be any action of the mobile device interacting with an external entity, such as an external device or a user. The event may be of a type that is repeated for the device. Thus, historical statistics may be obtained for different occurrences of an event. Such historical data may be used to train models and sub-models.

Block 37_602 may include receiving one or more attributes of the user device. The one or more attributes may be received by a recipient suggestion engine executing on the device. As described in this section, attributes may correspond to time, location, motion state, calendar events, and the like. Such one or more attributes may correspond to context data defining a particular context of the device. The one or more attributes may be measured around the time the event was detected (e.g., over a period of time). The time period may include times before and after the event is detected, a time period just before the event is detected, or a time just after the event is detected.

At block 37_604, it is determined that the user input corresponds to a trigger for a recipient to provide a suggestion via the suggestion engine. For example, if user input is received for composing an electronic mail in an electronic mail application, block 37_604 may determine that a suggested recipient for the electronic mail should be provided. Further, for example, if a user input is received to initiate a search in a search application, block 37_604 may include determining that a predicted contact is to be included in the search results.

At block 37_606, one or more tables corresponding to previous communications using the user equipment are populated. In the embodiment of fig. 37_6, each of the one or more tables corresponds to a different sub-state of the user device and includes a plurality of contact metrics for previous communications with different recipients. For example, the previous communication may include previous interactions with other users, such as, for example, previous emails, voice calls, text messages, instant messages, video calls, and calendar invitations.

At block 37_608, a first set of one or more tables corresponding to the one or more state variables is identified using the one or more state variables. For example, if the location state variable indicates that the user device is at the user's home, block 37_608 may include identifying a table corresponding to previous communications associated with the user's home. That is, the table corresponding to previous communications made using the user device may be filtered to include only the table corresponding to previous communications initiated or performed by the user while at home. In this embodiment, when the location state variable indicates that the user device is at the user's home, a set of tables can be identified that compose, read, or edit past emails while the user is at home. Also, for example, if the account status variable indicates that the user is using a work email account, block 37_608 may include identifying a set of tables corresponding to past communications made using the work email account. Embodiments may use multiple state variables (e.g., location status and account status) to enable a user to select a location for a particular account

At block 37_610, a first set of tables is queried to obtain contact metrics for one or more potential recipients. The contact metrics may include, for example, contact metrics for recipients of a calendar invitation to a previous calendar event made using the user device, a time at which a previous email message was created (i.e., composed or sent), an email account identification associated with the previous email message, other recipients who copied on the email message, a number of emails sent to each recipient. In one embodiment, the querying of the first set of tables may be completed to calculate a total number of prior communications sent to each of the one or more potential recipients. For example, querying the first set of tables may include querying an email table to determine an accumulated number of emails sent to or received from each potential recipient. Querying the first set of tables may include querying a calendar event table to determine a cumulative number of calendar invitations sent to or received from each potential recipient.

Block 37_610 may look up the table based on individual interactions with potential recipients and group interactions with groups of recipients. For example, block 37_610 may predict a next e-mail recipient, where context data from the e-mail table indicates previous e-mail interactions (e.g., sent or received e-mail messages) between the user and the recipient. Box 37_610 may include ranking history interactions corresponding to the current context. For example, the weight of the interaction may include a social element indicating a co-occurrence of multiple recipients. In this embodiment, the ranking of the user's historical interactions with a group of recipients may be increased based on whether the user previously interacted with the group in other past interactions. That is, the rankings of the members of a group of recipients may be boosted based on the group of recipients having been included in common past interactions (e.g., the recipients are repeatedly duplicated on emails sent by the user). Thus, if the user previously selected two recipients for past email, and both were copied to email sent to the third recipient, the third recipient would be ranked up based on having been previously included in the two recipients. However, if the user has another interaction that includes only one of the three recipients, then such an interaction will not result in the same rank boost.

At block 37_612, an overall contact metric for previous communications and interactions using the obtained contact metrics is calculated for each of the one or more potential recipients. In one embodiment, the total contact metric for the previous communications is a cumulative total of the previous communications sent to each of the one or more potential recipients. In this embodiment, the total number of emails, messages, calls, and calendar invitations sent to each potential recipient may be calculated by querying one or more tables.

At block 37_614, the prediction engine is to identify one or more predicted recipients to suggest to the user based on the total contact metrics for the one or more potential recipients and using one or more criteria. In some embodiments, the criteria may include a minimum number of predicted recipients to suggest (e.g., top N recipients), a percentage of predicted recipients to suggest (e.g., top 25%), and/or a threshold confidence level for suggesting predicted recipients. Block 37_614 may include using a hard interrupt as a criterion. For example, the recipient may only be considered to have a minimum number of past interactions with the user. In some embodiments, social criteria are used to suggest recipients. For example, predicted recipients may be suggested when they co-occur with another suggested recipient with whom the user previously interacted. In some embodiments, recipients having similar characteristics to other predicted recipients may be suggested. For example, recipients having the same email address field and associated with the same location as the predicted recipient may be suggested as additional recipients for the communication.

Block 37_614 may include using a particular sub-model to identify one or more recipients to suggest to the user. The one or more recipients may have at least one threshold probability of at least one of the one or more recipients being interacted with by the user in association with the trigger event. One of the one or more recipients in the prediction history data may be identified as a correct prediction. The threshold probability may be measured in various ways, and may use a probability distribution determined from historical data, as described in more detail below. For example, it may be desirable for the mean (average) probability, median probability, or peak of the probability distribution to be above a threshold probability (e.g., above 0.5, equivalent to 37 — 60%). Thus, the confidence level may be the mean, median or peak of the probability distribution. Another example is that the regions above a certain value on the probability distribution are larger than a threshold probability.

At block 37_616, the one or more predicted recipients are provided to the user. Block 37_614 may include providing a user interface to the user for communicating with the one or more recipients. For example, the device may display the identified recipients to the user via a list interface that the user may interact with to indicate whether the user wishes to access the identified recipients. For example, the user interface may include a touch-sensitive display that displays one or more of the identified recipients to the user and allows the user to communicate with the one or more recipients identified by the device by interacting with the touch-sensitive display. The user interface may allow interaction on a display screen with fewer recipients than are provided in a list containing all recipients of the user.

For example, one or more suggested recipients may be provided in a list of recipients on a search screen. The user may select a recipient and then select from the search screen how to communicate with the selected recipient, thereby making it easier for the user to interact with the selected recipient. For example, a user interface specific to a communication application (e.g., an email application) may appear after authenticating the user (e.g., via a password or biometric).

In the email context, block 37_614 may provide the suggested recipients as potential recipients of the email message. In this case, the exemplary email application interface of FIG. 37_4 may be used to provide suggested email recipients to the user. In the search context, block 37_614 may include providing the suggested recipients as search results in a search interface. For example, the search interface of fig. 37_5 may be used to present suggested recipients in a search result list.

F. Exemplary model

In some embodiments, the model may select the top N recipients for a given data set (or subset). Since the N recipients were selected most often in the past, it is predicted that future behavior will reflect past behavior. N may be a predetermined number of recipients (e.g., 1, 2, or 3) or a percentage, which may be the number of recipients of the actual past recipients associated with the event. Such a model may select the top N recipients to provide to the user. Further analysis may be performed, for example, to determine a probability (confidence) level for each of the N recipients, to determine whether to provide them to the user, and how to provide them to the user (e.g., action), which may depend on the confidence level.

In the example where N is equal to 3, when an event occurs where the context information corresponds to a particular sub-model, the model will return the first three recipients that are most often selected.

In other embodiments, rather than using only context information to select a submodel, a submodel may use a composite signal, with some context information being used to determine the predicted recipient. For example, a neural network or logistic regression model may predict a recipient using location (or other features) and constructing a type of linear weighted combination of these features. Such more complex models may be more suitable when the data volume of the sub-model is significantly large. Some embodiments may switch the type of submodel used at a particular node (i.e., a particular combination of context data) after obtaining the data for that node again.

The accuracy of the model may be tested against historical interaction data. For a given event, the history interaction data may identify which recipients the user interacted with in association with the event (e.g., just before the event or just after the event, such as within one minute). For each event, the context data may be used to determine a particular model. Further, the context data may be used as input features for the model.

In the example where the model (or sub-model) selects the top recipient, the number of historical data points at the time the top recipient was actually selected (i.e., sent the communication) may be determined to be a correct count, and the number of historical data points at the time the top recipient was not selected may be determined to be an incorrect count. In embodiments where N is greater than one for the model that selects the first N recipients, the correct count may correspond to any historical data points when one of the first N recipients was selected as the recipient of the communication.

Based on the first subset of historical interactions, the first submodel may predict at least one recipient of the first set of one or more recipients with which the user will interact in association with the event having the first confidence level. The first submodel may be created based at least on the first confidence level being greater than the initial confidence level by at least a threshold amount (which may be 0 or greater). The threshold amount may correspond to a difference threshold. In some implementations, the first submodel may not necessarily be created when the criterion is met, as further criteria may be used. If the confidence level is not greater than the initial confidence level, another attribute may be selected for testing. This comparison of confidence levels may correspond to a test of information gain. The same process may be repeated to determine a second confidence level for a second submodel (for the second attribute) of the first submodel for predicting the first set of one or more recipients. A second subset of the historical interactions may be used for a second submodel. A third attribute or more may be tested in a similar manner.

G. Regeneration of decision trees

Embodiments may generate the decision tree for the model periodically (e.g., daily). This generation may use historical interaction data available at the time. Thus, the decision tree can be changed from one generation to another. In some embodiments, the decision tree is constructed without knowledge of the previous decision tree. In other embodiments, a new decision tree may be constructed from such prior knowledge, e.g., knowing which submodels are possible, or starting from a previous decision tree.

In some embodiments, all contexts (or a predetermined list of contexts) are tried to determine which submodels provide the greatest information gain. For example, if a location provides the maximum information gain for segmentation into submodels, a submodel for at least one particular location may be created. At each segmentation level, the contexts may be tested in such a greedy manner to determine which contexts provide the highest increase in information gain.

Determining actions based on probability levels

The predictive model may test not only the selected recipients, but also certain operations (e.g., copying the recipients onto emails based on previously added recipients). In some embodiments, once the probability of selecting a recipient is sufficiently accurate, more aggressive actions may be provided than merely providing the proposed recipient. For example, when a recipient is provided, the email application may automatically launch and the recipient is included as the recipient in the new email message.

When a recipient is selected with a sufficient probability (e.g., a confidence level above a high threshold) prediction, the prediction may begin a test action. Thus, the test is not merely for predicting the recipient, but whether a particular action can be predicted with sufficient accuracy. Different possible actions may be obtained from historical interaction data (including launching an email, text messaging, calendar, or video conferencing application).

Thus, when there is greater confidence, the embodiment may more aggressively perform the action to be performed. The predictive model may provide a particular user interface for a communication application if a particular communication means (e.g., email, text message, voice call, video call, and video conference) has a high probability of being used to communicate with the recipient. For example, if the user has a high probability that an email will be sent to the suggested recipients, the predictive model may provide an interface for an email application. Thus, in some embodiments, the higher the probability of use, more aggressive actions may be taken, such as automatically providing an interface for interacting with recipients using a corresponding communication application (e.g., email, calendar, instant message, text message, voice call, or video conference), rather than merely providing suggested recipients.

For example, the base model may have some degree of statistical significance (accuracy and confidence) and the action may be to suggest a recipient on the search screen. As another example, a higher level of statistical significance may cause the screen to light up (thereby drawing attention to the recipients, only one recipient may be selected, or a User Interface (UI) for a particular application (i.e., a UI for an email application) may be provided.

The action may depend on whether the model predicts only one recipient or a group of recipients. For example, if there is an opportunity to propose three recipient suggestions instead of one, this will also change the probability distribution, since selecting any one of the three recipients will provide the correct prediction. A model where the recommendation for one recipient is not trusted may have sufficient confidence in the recommendations for three recipients. Embodiments may perform the addition of another recipient to a group of recipients predicted by the model (e.g., the second potential contact not already in the group), thereby giving the model a higher confidence. If the model is based on predictions of more than one contact, the provided user interface will provide interaction with more than one contact, which may affect the form of the UI. For example, all contacts may be provided in a list and one contact may not be automatically selected. In one embodiment, the prediction may include the top contact, and if that contact is selected, other contacts may be duplicated on the message (i.e., due to co-occurrence in the historical interaction data). In the embodiment of fig. 37_4, these other recipients may be listed in the CC/BCC section of the email application interface 37_ 406.

There may also be multiple operations and one proposal for different operations. For example, there may be two playlists in a gym as part of a sub-model (e.g., one application is identified when two actions have similar selected possibilities, but two actions are identified in the model). Both actions together may be statistically significant, while they alone are not.

For example, when the model is first trained for an event (e.g., composing an email), the confidence of the model may not be sufficient to perform any action. At an initial confidence level, a recipient name, icon, or other recipient identifier may be displayed. At the next higher confidence level, the recipient's contact address (e.g., email address or phone number) may be displayed. With a further level of confidence, a user interface specific to a particular communication application may be displayed (e.g., controls for adding a predicted recipient as a recipient of a new email, instant message, phone call, or video call). These different levels may represent various values for defining confidence levels.

Other example actions may include altering a song being played, providing a notification (which may be a prominent location on the screen). The action may occur after unlocking the device, e.g., a UI specific to the application may be displayed after unlocking. Operations may be defined using deep links to start specific functions of an application.

V. data flow and module

Fig. 37_7 is an exemplary data flow diagram 37_700 for suggesting recipients to contact. The data flow diagram 37_700 provides recipient suggestions 37_702 to various communication applications and interaction mechanisms 37_ 701. In the embodiment of fig. 37_7, the applications and mechanisms 37_701 include a calendar 37_704, mail 37_706, message 37_708, phone 37_710, and video call 37_ 712. As shown in FIG. 37_7, an exemplary mail 37 uThe 706 application is an email application and the exemplary messaging 37_708 application is an instant messaging application. As shown, the phone application 37_710 may be used to initiate voice calls and compose text messages. An example of a video call application is

An application program.

The data flow diagram 37_700 shows that the recipient suggestions 37_702 may be based on data from various data sources 37_ 714. Data source 37_714 may include information for past communications. The data sources may include events 37_716, searches 37_718, found contacts 37_720, recent activities 37_722, collection daemons 37_724, communication histories 37_726, and contacts 37_ 728. The data source 37_714 may be populated with data from the communication application and interaction mechanism 37_ 701. For example, calendar 37_704 may provide calendar event data to event 37_ 716. Similarly, the telephone 37_710 and the video call 37_712 can provide call histories of the voice call and the video call to the communication history 37_726, respectively. In the fig. 37_7 embodiment, found contacts 37_720 may include contacts found in email messages and other types of messages (e.g., instant messages and text messages).

Fig. 37_8 is a block diagram of an exemplary interaction module 37_ 810. As shown, the interaction module 37_810 can be implemented as a daemon including a recording engine 37_814 and a suggestion engine 37_ 816. The interaction module 37_810 maintains a store of interactions in the interaction database 37_818 and executes the recipient suggestion algorithm using the suggestion engine 37_ 816. The interaction module 37_810 comprises an interaction store service 37_817 for communicating with an interaction database 37_ 818. The interaction module 37_810 may query the interaction database 37_818 to retrieve information of past interactions. The interaction module 37_810 may also send interaction data to the interaction database 37_818 to populate the table in the interaction database 37_ 818. For example, database tables in the interaction database 37_818 corresponding to previous communications using the application 37_800 may be populated. In the embodiment of fig. 37_8, each table in the interaction database 37_818 corresponds to a different sub-state of the user device on which the application 37_800 is executing and includes previously recorded contact metrics for interactions with other users performed using the device. For example, the recorded interactions may include previous interactions with other users, such as, for example, previous email, voice calls, text messages, instant messages, video calls, and calendar invitations.

The interaction module 37_810 also includes an XPC service 37_813 for communicating with the application 37_ 800. The application 37_800 may be one of the communication applications or interaction mechanisms shown in fig. 37_ 7. The application program 37_800 comprises a framework 37_820, which in turn comprises an interaction recorder 37_824 for recording interactions and communications performed using the application program 37_ 800. The framework 37_820 also includes an interactive advisor 37_826 that can be used to provide advised recipients to the application 37_ 800. The framework 37_820 may use an interaction recorder 37_824 to provide an interface for recording interactions. The interfaces of the interaction recorder 37_824 and interaction recommender 37_826 transfer data to the interaction module 37_810 via the XPC service 37_ 822.

Architecture VI

Fig. 37_9 shows an exemplary architecture 37_900 for providing a user interface for interaction with one or more recipients. The architecture 37_900 shows elements for detecting events and providing recipient suggestions. The architecture 37_900 may also provide other suggestions, such as for communication applications. The recipient suggestions may be provided with suggested applications. For example, the architecture 37_900 may provide suggested recipients and also suggest contacting the suggested recipients via a particular communication application. The architecture 37_900 may exist within the user equipment.

The top is the UI element. As shown, there is a lock screen 37_910, a search screen 37_920, and a speech interface 37_ 925. The user interface may be provided to the user in these ways. Other UI elements may also be used.

At the bottom, are the data sources for the application suggestion engine 37_940 and the recipient suggestion engine 37_ 950. The event manager 37_942 may detect events and provide information about the events to the application suggestion engine 37_ 940. In some embodiments, the event manager 37_942 may determine whether an event triggers an application suggestion. A list of predetermined events for triggering application suggestions can be specified. The location unit 37_944 may provide the location of the user equipment. For example, location unit 37_944 may include a GPS sensor and a motion sensor. The location unit 37_944 may also include other applications that may store the last location of the user, which may be sent to the application suggestion engine 37_ 940. Other context data may be provided from other context units 37_ 946.

The application suggestion engine 37_940 may identify one or more applications and corresponding actions. At the same level as the application suggestion engine 37_940, the recipient suggestion engine 37_950 may provide suggested recipients for presentation to the user. The event manager 37_952 may detect an event related to a receiver and provide information about the event to the receiver suggestion engine 37_ 950. In some embodiments, the event manager 37_952 may determine whether an event triggers a recipient suggestion. A list of predetermined events for triggering the recipient's suggestion may be specified. The interaction history 37_954 may provide data for previous interactions and communications with other users. For example, the interaction history 37_954 may be a data source for information recorded from previous emails exchanged between the user of the device and other users. The location unit 37_956 may provide the location of the user equipment. For example, the location unit 37_956 may include a GPS and a motion sensor. The location unit 37_956 may also include other applications that may store the last location of the user device, which may be sent to the recipient suggestion engine 37_ 950. Other context data may be provided from other context units 37_ 958.

The suggested recipients may be provided to the suggestion center 37_930, which may determine what to provide to the user. For example, the suggestion center 37_930 may determine whether to provide the suggested application or the recipient. In other examples, both an application and a recipient may be provided. The suggestion center may determine the best way to provide to the user. Different suggestions to the user may use different UI elements. In this way, the suggestion center 37_930 may control suggestions to the user so that different engines do not interrupt suggestions provided by other engines. In various embodiments, the engine may push suggestions (recommendations) to the suggestion center 37_930 or receive requests for suggestions from the suggestion center 37_ 930. The suggestion center 37_930 may store the suggestion for a certain time and then determine to delete the suggestion if the suggestion is not provided to the user or the user does not interact with the user interface.

The suggestion center 37_930 may also identify other actions that occur with the user device to inform the device when to send the suggestion. For example, if the user is using an application, the recipient of the suggestion may be provided, but the application suggestion may not be provided. The suggestion center 37_930 may determine when to send the suggestion based on various factors, such as the motion state of the device, whether the lock screen is on, or whether authorized access is provided, whether the user is working at a work site, using the device at home, and the like.

In some embodiments, the software component of device 37_100 (fig. 1A) includes a recipient suggestion/prediction module (or set of instructions). In some embodiments, the recipient suggestion module may include various sub-modules or systems, for example, as described above with reference to fig. 37_7 through 37_ 9. The recipient suggestion module may perform all or part of the method 37_100 or 37_ 600.

Exemplary methods, apparatus, and computer-readable media for human-centric prediction

Some embodiments provide systems and methods for advising a recipient. Upon detecting that a user input on the device corresponds to a trigger for providing a suggested recipient, contextual information of the device is determined that is representative of a current state of the device, wherein the current state is defined by state variables. Populating tables corresponding to previous communications made using the device, each table corresponding to a different sub-state of the device and including contact metrics for previous communications with different recipients. The state variables may be used to identify a set of tables corresponding to the state variables. Contact metrics for potential recipients are obtained from the set of tables. An overall contact metric for the previous communication is calculated for each potential recipient. A predicted recipient to suggest is identified based on the overall contact metrics of the potential recipients and using criteria, and the predicted recipient is provided to the user.

In some embodiments, there is provided a computer-implemented method of providing suggested recipients to be contacted with a user device of a user, the method comprising: at the user equipment: detecting a user input at a user device; determining that the user input corresponds to a trigger of a recipient providing a suggestion via a suggestion engine; determining context information of the user equipment, the context information representing a current device state of the user equipment, wherein the current device state is defined by one or more state variables; populating one or more tables corresponding to previous communications with the user device, each of the one or more tables corresponding to a different device substate of the user device and including a plurality of recipient metrics for the previous communications with the different recipients; identifying a first set of one or more tables corresponding to the one or more state variables using the one or more state variables; obtaining contact metrics for one or more potential recipients from the first set of tables; for each of the one or more potential recipients: calculating an overall contact metric for the previous communication using the obtained contact metrics; identifying, using a suggestion engine, one or more predicted recipients to suggest to the user based on the total contact metrics of the one or more potential recipients and using one or more criteria; and providing the one or more predicted recipients to the user. In some embodiments, the context information includes one or more recipients of an open communication in a communication application executing on the user device, a current device state including a current application state, and a current application state including the one or more recipients. In some embodiments, the context information includes an account identifier corresponding to a communication application executing on the user device, a current device state including a current application state, and a current application state including the account identifier. In some embodiments, the method comprises: a current location of the user device is determined, wherein the context information includes the current location, a current device state including a current location state, and a current location state including the current location. In some embodiments, the context information includes a current time and a current date, and wherein the one or more criteria include a minimum number of predicted recipients to suggest. In some embodiments, the one or more criteria include a threshold confidence level, the method further comprising: at a user equipment: determining how to provide the one or more predicted recipients to the user based on the respective confidence levels for each of the one or more predicted recipients. In some embodiments, the context information includes a topic of open communication in a communication application executing on the user device, a current device state including a current application state, and a current application state including the topic. In some embodiments, the subject of the open communication is one or more of an email message subject, a calendar event subject, and a video conference subject. In some embodiments, the context information includes a scheduled time for an open calendar event in a calendar application executing on the user device, a current device state including a current application state, and a current application state including a scheduled time. In some embodiments, the context information includes a location of an open calendar event in a calendar application executing on the user device, a current device state including a current application state, and a current application state including a location. In some embodiments, one of the one or more tables is a calendar table corresponding to a calendar sub-state of the user device, the calendar table including contact metrics for recipients of calendar invitations to previous calendar events conducted using the user device. In some embodiments, one of the one or more tables is an email table corresponding to an email sub-status of the user device, the email table including contact metrics for recipients of previous email messages created using the user device, the contact metrics including a time at which the previous email message was created, an email account identifier associated with the previous email message, other recipients that were duplicated on the previous email message, and a number of email messages sent to each recipient. In some embodiments, calculating the total contact metric for the previous communications includes querying the one or more tables to calculate a total number of previous communications sent to each of the one or more potential recipients.

In some embodiments, a computer product comprising a non-transitory computer-readable medium storing instructions for providing suggested recipients to contact a user device of a user, the instructions, when executed by one or more processors of the user device, perform the following: detecting a user input at a user device; determining that the user input corresponds to a trigger of a recipient providing a suggestion via a suggestion engine; determining context information of the user equipment, the context information representing a current device state of the user equipment, wherein the current device state is defined by one or more state variables; populating one or more tables corresponding to previous communications with the user device, each of the one or more tables corresponding to a different device sub-state of the user device and including a plurality of contact metrics for the previous communications with different recipients; identifying a first set of one or more tables corresponding to the one or more state variables using the one or more state variables; obtaining contact metrics for one or more potential recipients from the first set of tables; for each of the one or more potential recipients: calculating an overall contact metric for the previous communication using the obtained contact metrics; identifying, using a suggestion engine, one or more predicted recipients to suggest to the user based on the total contact metrics of the one or more potential recipients and using one or more criteria; and providing the one or more predicted recipients to the user. In some embodiments, the context information includes one or more recipients of an open communication in a communication application executing on the user device, a current device state including a current application state, and a current application state including the one or more recipients. In some embodiments, the context information includes a current time and account identifier corresponding to a communication application executing on the user device, a current device state including the current application state, and a current application state including the current time and account identifier.

In some embodiments, there is provided a user device for providing suggested recipients to be contacted with the user device, the user device comprising: an input device; one or more processors configured to: detecting a user input at an input device; determining that the user input corresponds to a trigger to provide a recipient of a suggestion via a suggestion engine; determining context information of the user equipment, the context information representing a current device state of the user equipment, wherein the current device state is defined by one or more state variables; populating one or more tables corresponding to previous communications with the user device, each of the one or more tables corresponding to a different device substate of the user device and including a plurality of contact metrics for the previous communications with different recipients; identifying a first set of one or more tables corresponding to the one or more state variables using the one or more state variables; obtaining contact metrics for one or more potential recipients from the first set of tables; for each of the one or more potential recipients: calculating an overall contact metric for the previous communication using the obtained contact metrics; identifying, using a suggestion engine, one or more predicted recipients to suggest to the user based on the total contact metrics of the one or more potential recipients and using one or more criteria; and providing the one or more predicted recipients to the user. In some embodiments, the context information includes one or more recipients of an open communication in a communication application executing on the user device, a current device state including a current application state, a current application state including the one or more recipients. In some embodiments, one of the one or more tables is an email table corresponding to an email sub-state of the user device, the email table including contact metrics for recipients of previous email messages created using the user device, the contact metrics including a time at which the previous email message was created, an email account identifier associated with the previous email message, other recipients duplicated on the previous email message, and a number of email messages sent to each recipient. In some embodiments, the one or more criteria include a threshold confidence level, the one or more processors are further configured to, at the user equipment: determining how to provide the one or more predicted recipients to the user based on the respective confidence levels for each of the one or more predicted recipients.

Section 8: application model for active assistant

The material in this section, "application model for active assistant," describes an application model for active assistant and details related to actively providing a suggestion to a user of a computing device, according to some embodiments, and provides information that complements the disclosure provided herein. For example, portions of this section describe applications that are predicted that a user may be interested in accessing, which complements the disclosure provided herein, such as those related to populating the predicted content within prediction section 930 of fig. 9B-9C and to creating and detecting trigger conditions (fig. 4A-4B). In some embodiments, details related to the application prediction engine and details related to predicting an application for inclusion in a search interface are also applicable to other methods described herein (e.g., methods 600,800, 1000, and 1200).

Abstraction of application model for active assistant

The embodiments described in this section present techniques for identifying when a user activates a search application on his or her mobile computing device. In particular, the techniques relate to presenting a prediction of one or more applications that a user may be interested in accessing before receiving search parameter input from the user, which may reduce the likelihood or necessity that the user must manually provide search parameters to the search application. According to some embodiments, the search application may be configured to interface with a prediction engine (referred to in this section as an "application prediction engine") whenever the search application is activated (e.g., displayed within a user interface of the mobile computing device). More specifically, when a search application interfaces with an application prediction engine, the search application may issue prediction requests for one or more applications that a user may be interested in accessing. In turn, the application prediction engine may analyze information associated with the applications installed on the mobile computing device to generate predictions. The search application may then display the predicted one or more applications within a user interface of the search application for selection by the user.

One embodiment presents a method for providing predictions to a user of a mobile computing device. In particular, the method is implemented by an application prediction engine executing on a mobile computing device and includes the steps of: (1) receiving a request from a search application executing on a mobile computing device, the request being for providing a prediction of one or more applications installed on the mobile computing device and likely to be of interest to a user to activate, (2) identifying a list of applications installed on the mobile computing device, (3) for each application included in the list of applications: (i) generating a score for an application by performing one or more functions on one or more data signals corresponding to the application, and (ii) associating the score with the application, (4) filtering a list of applications according to the generated score to produce a filtered list of applications, (5) populating a prediction with the filtered list of applications, and (6) providing the prediction to a search application.

Another embodiment presents a method for presenting predictions to a user of a mobile computing device. In particular, the method is implemented by a search application executing on a mobile computing device and includes the steps of: (1) detecting activation of a search application, (2) issuing a request to an application prediction engine regarding predicting one or more applications that are installed on the mobile computing device and that may be of interest to the user for activation, (3) receiving a prediction from the application prediction engine, wherein the prediction comprises a list of the one or more applications and each application is associated with a respective score, and (4) displaying, within a user interface of the search application, a user interface entry for at least one of the one or more applications based on the scores.

Another embodiment addresses a mobile computing device configured to present predictions to a user of the mobile computing device. In particular, the mobile computing device includes a processor configured to execute a search application configured to perform steps comprising: prior to (1) detecting activation of a search application, and (2) receiving input from a user within a user interface of the search application: (i) sending a request to an application prediction engine executing on a mobile computing device, the request pertaining to a list of one or more applications installed on the mobile computing device and that a user may be interested in activating, (ii) receiving the list from the application prediction engine, and (iii) displaying, within a user interface of a search application, a user interface entry for at least one of the one or more applications included in the list. As described above, the processor is further configured to execute the application prediction engine, wherein the application prediction engine is configured to perform the steps comprising (1) receiving a request from the search application for a list of one or more applications that the user may be interested in activating, (2) generating the list, and (3) providing the list to the search application.

Other embodiments include a non-transitory computer-readable medium configured to store instructions that, when executed by a processor, cause the processor to implement any of the foregoing techniques as set forth in this section.

This summary is provided merely for purposes of summarizing some example embodiments so as to provide a basic understanding of some aspects of the subject matter described in this section. Thus, it should be recognized that the above-described features are only examples, and should not be considered as narrowing the scope or spirit of the subject matter described in this section in any way. Other features, aspects, and advantages of the subject matter described in this section will become apparent from the following detailed description, the accompanying drawings, and the claims.

Other aspects and advantages of the embodiments described in this section will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the embodiments.

Detailed description of application models for active assistant

Representative applications of the apparatus and methods according to the described embodiments of the invention are provided in this section. These examples are provided solely to add context and aid in the understanding of the described embodiments. It will thus be apparent to one skilled in the art that the embodiments of the present invention may be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order to not unnecessarily obscure the embodiments of the invention. Other applications are possible, such that the following examples should not be considered limiting.

The embodiments described in this section propose techniques for identifying when a user activates a search application on his or her mobile computing device, and presenting predictions of one or more applications that the user may be interested in accessing before receiving input of search parameters from the user. According to some embodiments, the search application may be configured to interface with the application prediction engine whenever the search application is activated (e.g., displayed within a user interface of the mobile computing device) and query the application prediction engine to predict one or more applications that the user may be interested in accessing. Conversely, the application prediction engine may analyze information associated with an application installed on the mobile computing device to generate a prediction. This information may include, for example, an application installation timestamp, an application activation total, an application usage metric, a location of an application icon within a main user interface (e.g., on a home screen, within a folder, etc.), search parameters recently provided by the user, feedback collected indicating whether a previous prediction was accurate, etc., which enables the application prediction engine to provide meaningful and relevant predictions for the search application. In turn, the search application may display the predicted one or more applications within a user interface of the search application for selection by the user. Notably, such techniques can substantially reduce the cumbersome process of a user entering search parameters each time he or she attempts to access a particular application, which can provide a substantial improvement in the user's overall satisfaction with his or her mobile computing device.

Although the embodiments set forth in this section relate primarily to an application prediction engine configured to predict applications that a user may wish to access, it should be noted that other prediction engines for providing different kinds of predictions (e.g., people that a user may want to contact) may be implemented within a mobile computing device. More specifically, and according to some embodiments, each prediction engine may be configured to assign itself as an "expert" for a particular prediction category within the mobile computing device. For example, the application prediction engine may designate itself as an expert on the "application" prediction category to instruct the application prediction engine to specifically predict applications that a user of the mobile computing device may be interested in accessing. According to some embodiments, the application prediction engine may employ a learning model that enables the application prediction engine to analyze data (e.g., the information described above) and provide predictions based on the data. Although this disclosure primarily discusses an application prediction engine configured to implement a learning model, it should be noted that any technique for analyzing behavioral data and providing predictions may be employed by the application prediction engine described in this section. Further, it should be noted that the functionality that the application prediction engine may vary between different types of user devices (e.g., smart phones, tablets, watches, laptops, etc.) in order to provide specialized predictions for these different types of user devices. For example, a first type of application prediction engine may be assigned to a smartphone, a second type of application prediction engine may be assigned to a tablet, and so on.

As described above, each prediction engine implemented on a mobile computing device may specify itself as an expert on one or more prediction categories within the mobile computing device. Thus, in some cases, two or more application prediction engines may designate themselves as experts on the "application" prediction category. In this case, when a search application described in this section issues a prediction request, each of the two or more application prediction engines will perform its own analysis (e.g., according to a learning model employed by the application prediction engine) and generate a prediction from the request. In this case, at least two or more predictions are generated in response to the prediction request, which may establish redundancy and competing predictions that the search application may not be able to interpret.

Thus, embodiments also propose a "prediction center" configured to act as a mediator between the application prediction engine and the search application. To provide such functionality, the prediction center may be configured to act as a registrar for the prediction engine (e.g., application prediction engine) when the prediction engine initializes and attempts to assign itself as an expert for one or more prediction categories (e.g., an "application" prediction category). Similarly, and according to some embodiments, the prediction center may also be configured to manage different types of prediction categories within the mobile computing device, such that a consumer application (e.g., a search application described in this section) may query the prediction center to identify prediction categories that may be provided. In this way, when a consumer application issues a prediction request for a particular prediction category and two or more prediction engines respond with their respective predictions, the prediction center may be configured to receive and process the predictions prior to responding to the request issued by the consumer application. Processing the predictions may involve, for example, removing duplicate information present in the predictions, applying weights to the predictions based on historical performance (i.e., accuracy) metrics associated with the prediction engine, ranking the predictions based on scores advertised by the prediction engine when generating its predictions, and so forth. In this way, the prediction center can extract multiple predictions into an optimized prediction and provide the optimized prediction to the consumer application. Thus, the design advantageously simplifies the operating requirements of the consumer application (as they need not be able to process multiple predictions), integrates heavy processing into the prediction center, and enables the consumer application to obtain input representing various prediction engines that have assigned themselves as experts on the prediction category of interest.

Thus, the different techniques described above enable a search application to interact with a prediction center to receive predictions that may be used to enhance the overall user experience. In some cases, it may be valuable for the search application to provide feedback to the prediction center/application prediction engine to indicate whether the prediction is accurate. Such feedback may be beneficial, for example, when the learning algorithm is implemented by an application prediction engine, as the feedback may be used to "train" the learning algorithm and improve the overall accuracy of its prediction. For example, when the application prediction engine generates a prediction about a particular application that is most likely to be activated by the user (e.g., when displayed in a search application prior to receiving search input from the user), the search application may provide feedback indicating that the prediction is valid (e.g., the particular application was selected and activated by the user). In turn, the application prediction engine may increase the score advertised when the prediction engine produces a similar subsequent prediction.

Additionally, it should be noted that the architecture of the prediction center may be configured to enable different entities described in this section (such as the application prediction engine) to be modular components within the mobile computing device. In one architectural approach, each application prediction engine can be configured as a bundle, with the prediction center understanding its format (e.g., tree structure), and it enables the prediction center to serve as a platform for implementing the application prediction engine functionality. According to the method, the prediction center may be configured to, for example, parse different file system paths (e.g., upon initialization) to identify different bundles residing within the mobile computing device. In this way, bundles can be conveniently added to, updated in, and removed from a file system of a mobile computing device, thereby facilitating a modular configuration that can evolve efficiently over time without requiring extensive updates to the mobile computing device (e.g., operating system upgrades). For example, the application prediction engine may be configured to enable updating all or a portion of the logic implemented by the application prediction engine (e.g., via over-the-air (OTA) updates). Note that the foregoing architecture is exemplary, and any architecture that enables the various entities described in this section to communicate with each other and provide their different functionality may be used.

In addition, the prediction center/application prediction engine may also be configured to implement one or more caches, which may be used to reduce the amount of processing that occurs when generating predictions. According to some embodiments, after generation, a prediction may be accompanied by a "validity parameter" that indicates when the prediction should be removed from the cache in which it is stored. The validity parameter (also referred to in this section as "expiration information") may define, for example, a time-based expiration, an event-based expiration, and the like. In this way, when the application prediction engine frequently receives prediction requests from a search application, the application prediction engine may generate and cache predictions in order to substantially reduce the amount of future processing that would otherwise occur when processing duplicate prediction requests. Note that the prediction center/application prediction engine may be configured to cache predictions using various methods. For example, when available cache memory is limited, the prediction center/application prediction engine may be configured to generate a threshold number of predictions (e.g., within a time window), and when the threshold is met, transition to cache predictions and reference the cache for predicted subsequent requests (as long as the expiration information indicates the predictions are valid).

Accordingly, the embodiments described in this section propose techniques for identifying when a user activates a search application on his or her mobile computing device, and presenting a prediction of one or more applications that the user may be interested in accessing before receiving input of search parameters from the user. A more detailed discussion of these techniques is shown below and described in conjunction with fig. 38_1 through 38_4, which show detailed illustrations of system, method core user interfaces that can be used to implement these techniques.

Fig. 38_1 illustrates a block diagram of different components of a mobile computing device 38_100 configured to implement various techniques described in this section, in accordance with some embodiments. More specifically, fig. 38_1 illustrates a high-level overview of the mobile computing device 38_100, which, as shown, is configured to implement a prediction center 38_102, an application prediction engine 38_104, and a search application 38_ 116. According to some embodiments, the prediction center 38_102, the application prediction engine 38_104, and the search application 38_116 may be implemented in an Operating System (OS) (not shown in fig. 38_ 1) configured to execute on the mobile computing device 38_ 100. As shown in fig. 38_1, the prediction center 38_102 may be configured to act as a mediator between the application prediction engine 38_104 and the search application 38_ 116. Although not shown in fig. 38_1, the prediction center 38_102 may be configured to implement an aggregator configured to integrate multiple predictions, such as when two or more application prediction engines 38_104 are implemented and generate two or more predictions in response to requests issued by the search application 38_ 116. It should be noted, however, that both the application prediction engine 38_104 and the search application 38_116 may be configured to communicate directly with each other to reduce or even eliminate the need to implement the prediction center 38_102 within the mobile computing device 38_ 100. It should further be noted that the application prediction engine 38_104 and the search application 38_116 need not be logically separate from each other, and the different functions implemented by these entities may be combined to create different architectural approaches that provide the same results.

As shown in fig. 38_1, the prediction 38_112 may be communicated between the application prediction engine 38_104 and the search application 38_116, e.g., the prediction center 38_102 may receive the prediction 38_112 generated by the application prediction engine 38_104 and forward the prediction 38_112 to the search application 38_ 116. The feedback 38_114 may also be communicated between the application prediction engine 38_104 and the search application 38_116, for example, the prediction center 38_102 may receive the feedback 38_114 from the search application 38_116 and provide the feedback 38_114 to the application prediction engine 38_104 so that the application prediction engine 38_104 may increase the accuracy of the predictions 38_112 over time.

Additionally, the prediction center 38_102 may be configured to implement a cache that enables the prediction center 38_ 102/application prediction engine 38_104 to cache the predictions 38_112 in an attempt to increase processing and energy consumption efficiency at the mobile computing device 38_ 100. For example, the cache may include a plurality of entries, where each entry includes a prediction 38_112 and expiration information indicating how long the prediction 38_112 is considered valid. The expiration information may include, for example, a time-based expiration, an event-based expiration, and the like. As such, when the application prediction engine 38_104 frequently receives requests for predictions 38_112, the application prediction engine 38_104 may generate and cache predictions 38_112 to substantially reduce the amount of processing that would otherwise occur at the mobile computing device 38_100, thereby improving performance.

As described in this section, the application prediction engine 38_104 may be implemented using various architectural approaches, for example, the application prediction engine 38_104 may be a stand-alone executable program external to the prediction center 38_102 and in communication with the prediction center 38_102 via Application Programming Interface (API) commands that are supported by the prediction center 38_102 and used by the application prediction engine 38_104, the application prediction engine 38_104 may be a bundle stored within the file system of the mobile computing device 38_100 and interpreted and implemented by the prediction center 38_102, and so on. As shown in fig. 38_1, the application prediction engine 38_104 may include a configuration parameter 38_106 that indicates the manner in which the application prediction engine 38_104 generates the prediction for the search application 38_ 116. In particular, the configuration parameters 38_106 may define the manner in which the data signal 38_110 (which corresponds to the installed application information 38_108 available to the application prediction engine 38_104 within the mobile computing device 38_ 100) is received by the application prediction engine 38_104 and processed by the application prediction engine 38_ 104. According to some embodiments, the data signals 38_110 may represent application installation timestamps (e.g., when each application was installed), application activation timestamps (e.g., the time each application was last activated), application activation totals (e.g., the total number of times an application has been activated), application usage metrics (e.g., the frequency with which an application has been activated), and the like. The data signals 38_110 may also include the location of an application icon within a main user interface (e.g., on a home screen, within a folder, etc.) of the mobile computing device 38_100, application search parameters recently provided by the user, feedback collected indicating whether previous predictions provided by the application prediction engine 38_104 were accurate, and so forth.

Although not shown in fig. 38_1, the application prediction engine 38_104 can be configured to implement a learning model that enables the application prediction engine 38_104 to provide predictions 38_112 that evolve over time and remain relevant to a user of the mobile computing device 38_ 100. According to some embodiments, the learning model may represent an algorithm configured to analyze information (e.g., data signals 38_110) and generate predictions 38_112, which may enhance the overall experience of the user as the user operates mobile computing device 38_ 100. According to some embodiments, the information processed by the application prediction engine 38_104 may be collected from various sources within the mobile computing device 38_100, such as a file system implemented on the mobile computing device 38_100, feedback information provided by the search application 38_116, information collected by sensors of the mobile computing device 38_100 (e.g., a Global Positioning System (GPS) sensor, a microphone sensor, a temperature sensor, an accelerometer sensor, etc.), information provided by external sources (e.g., other applications executing on the mobile computing device 38_100, an OS kernel, etc.), and so forth.

Additionally, as shown in fig. 38_1, the mobile computing device 38_100 may be configured to interface with one or more servers 38_120 (e.g., via an internet connection) to receive Over The Air (OTA) updates 38_122 that may be used to partially or fully update one or more of the application prediction engine 38_104, prediction center 38_102, and search application 38_ 116. Thus, FIG. 38_1 provides a high-level overview of various components that may be used to implement the techniques described in this section.

Fig. 38_2 illustrates a method 38_200 implemented by the application prediction engine 38_104, according to some embodiments. Although the method 38_200 is described as the application prediction engine 38_104 and the search application 38_116 communicating directly with each other, it should be noted that the prediction center 38_102 may act as a mediator between the application prediction engine 38_104 and the search application 38_116, depending on the various functions provided by the prediction center 38_102 described in this section. As shown, the method 38_200 begins at step 38_202, where the application prediction engine 38_104 receives a request from the search application 38_116 to provide a prediction 38_112 of one or more applications installed on the mobile computing device 38_100 that may be of interest to a user of the mobile computing device 38_100 for access to the one or more applications. The request may be issued by the search application 38_116 in response to the search application being activated on the mobile computing device 38_100, such as when a user of the mobile computing device 38_100 inputs a gesture that causes the search application to be activated.

In step 38_204, the application prediction engine 38_104 identifies a list of applications installed on the mobile computing device 38_ 100. This information may be obtained, for example, by the installed application information 38_108 and the data signal 38_ 110. In step 38_206, the application prediction engine 38_104 sets the current application as the first application in the application list. In step 38_208, the application prediction engine 38_104 generates a score for the current application by performing one or more functions on one or more data signals 38_110 corresponding to the current application. According to some embodiments, performing a function on the data signal 38_110 may include calculating a score for the data signal 38_110 and adjusting the score according to a fixed weight associated with the data signal 38_ 110. For example, when data signal 38_110 corresponds to an installation date of an application, the score may be based on an amount of time that has elapsed since the application was installed, e.g., a newer installation date corresponds to a higher score. In some cases, the score may be adjusted according to a decay value (e.g., half-life), which may be particularly applicable to data signals 38_110, etc. representing time information associated with the application (e.g., application installation timestamp, application activation timestamp, etc.). Conversely, a fixed weight associated with the data signal 38_110 may be applied to the score to produce an updated version of the score. In this way, and upon completion of one or more functions on one or more data signals 38_110 corresponding to the current application, the prediction engine may produce a final form of the score for the current application (e.g., a sum of the individual scores).

In step 38_210, the application prediction engine 38_104 determines whether additional applications are included in the application list. In step 38_210, if the application prediction engine 38_104 determines that additional applications are included in the application list, the method 38_200 proceeds to step 38_ 212. Otherwise, the method 38_200 proceeds to step 38_214, which is described in more detail below. In step 38_212, the application prediction engine 38_104 sets the current application as the next application in the application list. In step 38_214, the application prediction engine 38_104 screens the list of applications based on (1) the generated score, and (2) the request received in step 38_ 202. For example, the request may indicate that only three application suggestions are displayed in the user interface of the mobile computing device 38_100 (e.g., based on screen size or resolution settings), which may cause the application prediction engine 38_104 to eliminate from the list of applications any application that scores other than the top three of the list. In step 38_216, the application prediction engine 38_104 populates the prediction 38_112 with the screened list of applications and provides the prediction 38_112 to the search application 38_ 116.

Fig. 3 illustrates a method 38_300 implemented by the search application 38_116, according to some embodiments. As shown, the method 38_300 begins at step 38_302, where the search application 38_116 is activated. In step 38_304, the search application 38_116 issues a request for a prediction 38_112 of one or more applications that the user may be interested in accessing. In step 38_306, the search application 38_116 receives the prediction 38_112 in response to the request, wherein the prediction 38_112 comprises a list of the one or more applications, and each application is associated with a respective score. In step 38_308, the search application 38_116 displays, within the user interface of the search application 38_116, a user interface entry for at least one of the one or more applications according to the score (e.g., as shown in fig. 38_4 and described below). In step 38_310, the search application 38_116 receives user input through the user interface.

In step 38_312, the search application 38_116 determines whether the user input corresponds to a user interface entry. In step 38_312, if the search application 38_116 determines that the user input corresponds to a user interface entry, the method 38_300 proceeds to step 38_ 314. Otherwise, the method 38_300 proceeds to step 38_318, which will be described in more detail below. In step 38_314, the search application 38_116 activates the application corresponding to the user interface entry. In step 38_316, the search application 38_116 provides feedback indicating that the application is activated. Finally, in step 38_318, the search application 38_116 deactivates itself.

Fig. 38_4 illustrates a conceptual diagram 38_400 of an exemplary user interface 38_402 of the search application 38_116 described in this section, according to some embodiments. As shown in fig. 38_4, the user interface 38_402 may include a search field 38_404 that enables a user of the mobile computing device 38_100 to enter search parameters (e.g., using a virtual keyboard 38_408 included in the user interface 38_ 402). Further, the user interface 38_402 may include a list of a plurality of user interface entries 38_406 for applications that the user may be interested in activating, which may be obtained using the predictions 38_112 generated by the application prediction engine 38_104 described in this section. Conversely, when the user provides feedback (which may include, for example, canceling a search, ignoring a suggested application, and entering search parameters, or selecting one of the user interface entries 38_ 406), the feedback may be forwarded to the application prediction engine 38_104 for processing.

FIG. 1A, described above, illustrates a detailed view of a computing device 100 that may be used to implement the various components described in this section, according to some embodiments. In particular, the detailed view illustrates various components that may be included in the mobile computing device 37_100 shown in fig. 37_ 1.

Exemplary methods and apparatus for application model for active Assistant

The embodiments described in this section propose techniques for identifying when a user activates a search application on his or her mobile computing device, and presenting predictions of one or more applications that the user may be interested in accessing before receiving input of search parameters from the user. According to some embodiments, the search application may be configured to interface with the application prediction engine whenever the search application is active, and query the application prediction engine to predict one or more applications that the user may be interested in accessing. In turn, the application prediction engine may analyze information associated with applications installed on the mobile computing device to generate predictions. Using the prediction, the search application may display the predicted one or more applications within a user interface of the search application for selection by the user.

In some embodiments, a method for proactively providing predictions to a user of a mobile computing device is provided, the method comprising: at an application prediction engine executing on a mobile computing device: for each application included in the list of applications installed on the mobile computing device: performing at least one function on at least one data signal corresponding to the application to establish a score for the application, wherein the score indicates a likelihood that the application will be activated by a user, and associating the score with the application; and providing the prediction to a search application executing on the mobile computing device, wherein the prediction includes a list of applications and their associated scores. In some embodiments, the method includes, prior to providing the prediction to the search application: a request for the prediction is received from a search application, where the request is issued by the search application in response to activation of the search application and prior to receiving a search input from a user. In some embodiments, the request indicates a particular number of applications that should be included in the list of applications included in the prediction. In some embodiments, the method comprises: for each application included in the list of applications: the score is adjusted according to a weight associated with the at least one data signal. In some embodiments, when the at least one data signal corresponds to a temporal aspect of application access within the mobile computing device, performing the at least one function on the at least one data signal further comprises: prior to adjusting the score according to the weight associated with the at least one data signal: adjusting the score of the at least one data signal according to an attenuation factor applied to the at least one data signal. In some embodiments, the at least one data signal is selected from one or more of: an application installation timestamp, an application activation total, an application usage metric, a location of an application icon in a main user interface of the mobile computing device, search parameters recently provided by the user, and collected feedback indicating whether a previous prediction was accurate. In some embodiments, the location of the application icon within the main user interface of the mobile computing device may indicate: a page number of the main user interface that includes the application icon, and whether the application is included in a folder within the main user interface. In some embodiments, the method comprises: after providing the prediction to the search application: receiving feedback from the search application, wherein the feedback is indicative of a behavior of the user after providing the prediction to the search application; and updates the collected feedback to reflect feedback received from the search application.

In some embodiments, a method for proactively presenting predictions to a user of a mobile computing device is provided, the method comprising: at a search application executing on a mobile computing device: detecting activation of a search application; issuing a request to an application prediction engine regarding predicting one or more applications that are installed on a mobile computing device and that a user may be interested in activating; receiving a prediction from an application prediction engine, wherein the prediction comprises a list of one or more applications and each application is associated with a respective score; and displaying, within a user interface of the search application, a user interface entry for at least one of the one or more applications based on the score. In some embodiments, the request is issued to the application prediction engine prior to receiving a search input via a search field included in a user interface of the search application. In some embodiments, the method comprises: receiving user input through a user interface of a search application; information associated with the user input is provided in the form of feedback. In some embodiments, the feedback indicates whether the user selected a user interface entry for the at least one application or entered search parameters. In some embodiments, the request indicates a particular number of applications that should be included in the forecast, and the particular number of applications enables a number of user interface entries based on the applications that can be displayed to the user within the user interface of the search application.

In some embodiments, a mobile computing device configured to proactively present predictions to a user of the mobile computing device is provided, the mobile computing device comprising a processor configured to: a search application configured to perform steps comprising: detecting activation of a search application, and prior to receiving input from a user in a user interface of the search application: issuing a request to an application prediction engine executing on the mobile computing device for a list of one or more applications installed on the mobile computing device and that the user may be interested in activating, receiving the list from the application prediction engine, and displaying, within a user interface of the search application, a user interface entry for at least one of the one or more applications included in the list; and an application prediction engine, wherein the application prediction engine is configured to perform steps comprising: the method includes receiving a request from a search application for a list of one or more applications that a user may be interested in activating, generating the list, and providing the list to the search application. In some embodiments, generating the list comprises: for each application installed on the mobile computing device: generating a score for the application by performing one or more functions on one or more data signals corresponding to the application and associating the score with the application; and filtering the application according to the generated score; and merging the applications into the list as filtered. In some embodiments, performing a function of the one or more functions on a data signal of the one or more data signals comprises: establishing a score for the data signal based on information associated with the data signal; and adjusting the score according to the weight associated with the data signal. In some embodiments, the data signal corresponds to a temporal aspect of application access within the mobile computing device, performing at least the function on the data signal further comprises: prior to adjusting the score according to the weight associated with the data signal: the score of the data signal is adjusted according to the attenuation factor applied to the data signal. In some embodiments, the one or more data signals include an application installation timestamp, an application activation total, an application usage metric, a location of an application icon in a main user interface of the mobile computing device, a search parameter recently provided by the user, and collected feedback indicating whether a previous prediction was accurate. In some embodiments, the location of the application icon within the main user interface of the mobile computing device may indicate: a page code of the main user interface including the application icon, and whether the application is included in a folder within the main user interface. In some embodiments, the application prediction engine is further configured to: after providing the list to the search application, performing steps comprising: receiving feedback from the search application, wherein the feedback is indicative of a user's behavior after providing the list to the search application; and updates the collected feedback to reflect the feedback received from the search application.

Section 9: expert center (providing predictive content items for components of electronic devices)

The material in this section, "expert center," describes the expert center, and describes components that provide predictive content items to an electronic device (e.g., to any component of device 100 in fig. 1A) according to some embodiments, and provides information that complements the disclosure provided herein. For example, portions of this section describe the creation of prediction engines and prediction categories, which supplement the disclosure provided herein, such as those related to the creation/storage of trigger conditions (fig. 4A-4B), as well as those related to retrieving and presenting predicted content items in a search interface (e.g., methods 600,800, 1000, and 1200), or those related to being suggested items in a messaging application (e.g., methods 2200, 2900, 2280, 2400, 2600, and 2700). In some embodiments, the methods disclosed herein utilize or use a prediction engine described in section 9 below to provide various related content items (e.g., predicted applications, predicted people/contacts, predicted locations, information related to events/contacts/locations for quickly adding content to various types of applications, and other related content items discussed above with reference to any of the methods) to a user at appropriate times.

Expert center abstract

The embodiments described in this section set forth techniques for implementing various "prediction engines" that may be configured to provide different kinds of predictions within a mobile computing device. According to some embodiments, each prediction engine may assign itself as an "expert" on one or more "prediction categories" within the mobile computing device. When a consumer application issues a prediction request for a particular prediction category, and two or more prediction engines respond with predictions, a "prediction center" may be configured to receive and process the predictions prior to responding to the request. Processing the predictions may include removing duplicate information that exists between predictions, ranking the predictions according to a confidence level advertised by the prediction engine, and so forth. In this way, the prediction center can extract multiple predictions into an optimized prediction and provide the optimized prediction to the consumer application.

One embodiment proposes a method for synchronously providing predictions to an application executing on a mobile computing device. In particular, the method is implemented by a prediction center executing on a mobile computing device and includes the steps of: (1) receiving a request from an application to synchronously provide predictions for a prediction category, (2) identifying one or more prediction engines associated with the prediction category, (3) receiving one or more predictions from the one or more prediction engines based on the request, (4) aggregating the one or more predictions to produce a prediction requested by the application, and (5) providing the prediction to the application.

Another embodiment proposes a method for asynchronously providing predictions to an application executing on a mobile computing device. In particular, the method is implemented at a prediction center executing on a mobile computing device and includes the steps of: (1) receiving a request from an application to asynchronously provide a prediction for a prediction category, (2) identifying one or more prediction engines associated with the prediction category, and (3) notifying each of the one or more prediction engines to asynchronously provide one or more predictions in accordance with the request.

Another embodiment provides a mobile computing device configured to generate predictions from user behavior. In particular, the mobile device is configured to implement: (1) a prediction center configured to act as a mediator between one or more prediction engines and one or more applications, wherein the prediction center manages a plurality of prediction categories, (2) one or more prediction engines, wherein each of the one or more prediction engines acts as an expert on at least one of the plurality of prediction categories managed by the prediction center, and (3) one or more applications, wherein each of the one or more applications is configured to perform steps comprising: (i) issuing a request to a prediction center for a prediction for a particular prediction category of the plurality of prediction categories, and (ii) receiving a prediction from the prediction center according to the request, wherein the prediction is an aggregation of at least two predictions produced by a prediction engine that is an expert on the particular prediction category.

Other embodiments include a non-transitory computer-readable medium configured to store instructions that, when executed by a processor, cause the processor to implement any of the foregoing techniques as set forth in this section.

This summary is provided merely for purposes of summarizing some example embodiments so as to provide a basic understanding of some aspects of the subject matter described in this section. Thus, it should be recognized that the above-described features are only examples, and should not be considered as narrowing the scope or spirit of the subject matter described in this section in any way. Other features, aspects, and advantages of the subject matter described in this section will become apparent from the following detailed description, the accompanying drawings, and the claims.

Other aspects and advantages of the embodiments described in this section will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the embodiments.

Detailed description of expert centers

Representative applications of apparatus and methods according to the described embodiments of the invention are provided in this section. These examples are provided solely to add context and aid in the understanding of the described embodiments. It will thus be apparent to one skilled in the art that the embodiments of the present invention may be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order to not unnecessarily obscure the embodiments of the invention. Other applications are possible, such that the following examples should not be considered limiting.

The embodiments described in this section present techniques for collecting and organizing behavioral data to enable mobile computing devices to provide meaningful predictions for their end users. According to some embodiments, the mobile computing device may be configured to implement various "prediction engines," each of which may be configured to provide different kinds of predictions within the mobile computing device. More specifically, and according to some embodiments, each prediction engine may assign itself as an "expert" on one or more "prediction categories" that may be used to enhance the overall operation of the mobile computing device. Examples of prediction categories may include applications (e.g., activation/deactivation), people (e.g., phone calls, chats, etc.), geographic data (e.g., mobile computing device movement/zone settings), notifications (e.g., push notification arrival), physical input (e.g., connecting headset/power to the mobile computing device), and so forth. Note that the foregoing prediction categories are merely exemplary, and the embodiments set forth in this section may employ any prediction category that a mobile computing device is capable of maintaining. According to some embodiments, the prediction engine may employ a learning model that enables the prediction engine to analyze data (e.g., behavioral data associated with a user's operation of the mobile computing device) and provide predictions in accordance with the data. Although this disclosure primarily discusses a prediction engine configured to implement a learning model, it should be noted that any technique for analyzing behavioral data and providing predictions may be employed by the prediction engine described in this section.

As previously described in this section, and according to some embodiments, the prediction engine may assign itself as an expert on one or more prediction categories within the mobile computing device. Thus, in some cases, two or more prediction engines may assign themselves as experts of the same prediction category within the mobile computing device. Thus, when a requesting entity (referred to in this section as a "consumer application") issues a prediction request that two or more prediction engines have allocated themselves as the prediction categories of their experts, each of the two or more prediction engines will perform its own analysis (e.g., according to the learning model that the prediction engine employs) and generate a prediction (or predictions) from the request. In this case, at least two or more predictions are generated in response to the prediction request, which may establish redundancy and competing predictions that the search consumer application may not be able to interpret.

Thus, embodiments also propose a "prediction center" configured to act as a mediator between the prediction engine and the consumer application. According to some embodiments, the prediction center may be configured to act as a registrar for the prediction engine when the prediction engine initializes and attempts to assign itself as an expert in one or more prediction categories. Similarly, and according to some embodiments, the prediction center may also be configured to manage different types of prediction categories within the mobile computing device, such that the consumer application may query the prediction center to identify prediction categories that may be provided. In this way, when a consumer application issues a prediction request for a particular prediction category, and two or more prediction engines respond with their respective predictions, the prediction center may be configured to receive and process the predictions prior to responding to the request issued by the consumer application. Processing the predictions may involve, for example, removing duplicate information present in the predictions, applying weights to the predictions based on historical performance (i.e., accuracy) metrics associated with the prediction engines, ranking the predictions based on a confidence level that the prediction engines advertised when generating their predictions, and so forth. In this way, the prediction center can extract multiple predictions into an optimized prediction and provide the optimized prediction to the consumer application. Thus, the design advantageously simplifies the operating requirements of the consumer application (since they need not be able to process multiple predictions), integrates heavy processing into the prediction center, and enables the consumer application to obtain input representing various prediction engines that have allocated themselves as experts on the prediction category of interest.

According to some embodiments, the prediction center may enable the consumer application to receive predictions in a "synchronized" manner. More specifically, the consumer application may be configured to issue a request to the prediction center that causes the prediction center to interact with one or more prediction engines and provide a degree of instantaneous (i.e., synchronized) response/prediction to the consumer application. This synchronization configuration may be used, for example, when a consumer application (such as a chat application) launches and attempts to pre-identify contacts (e.g., based on a current time) with which a user of the mobile computing device is most likely to send and receive messages. According to other embodiments, the prediction center may enable the consumer application to receive predictions in an "asynchronous" manner. More specifically, the consumer application may be configured to issue a request to the prediction center that causes the prediction center to notify/configure one or more prediction engines to provide predictions on an as-needed (i.e., asynchronous/triggered) basis. This asynchronous configuration may be used, for example, when a consumer application, such as an OS kernel configured to activate (i.e., launch) and deactivate (i.e., close) applications on a mobile computing device, attempts to reactively load the application in response to physical input occurring at the mobile computing device. For example, the prediction engine may determine that a particular music application was manually launched by the user most of the time the headset was plugged into his or her mobile computing device. Conversely, when the headset is connected to the mobile computing device, the prediction engine may indicate the particular music application to the OS kernel via prediction. In turn, the OS kernel may pre-load (based on predictions) the appropriate music application, which may help improve the user experience and enhance the performance of the mobile computing device.

Thus, the different techniques described above enable a consumer application to interact with a prediction center to receive predictions that may be used to enhance the overall user experience. In some cases, the consumer application provides feedback to the prediction center to indicate whether it may be valuable to have the predictions produced by the prediction engine accurate. Such feedback may be beneficial, for example, when the learning algorithm is implemented by a prediction engine, as the feedback may be used to "train" the learning algorithm and improve the overall accuracy of its prediction. For example, when the prediction engine generates a prediction that a user will take a particular action, and the consumer application provides feedback indicating that the prediction is valid (i.e., the user has taken the particular action), the prediction engine may increase the confidence level advertised by the prediction engine when producing similar subsequent predictions. As the confidence level increases, predictions produced by a prediction engine may be prioritized over competing predictions produced by other prediction engines (if any). Alternatively, when the prediction engine predicts that the particular action will be taken by the user, and the consumer application provides feedback indicating that the prediction is not valid (i.e., the user did not take the particular action), the prediction engine may decrease the confidence level advertised when similar subsequent predictions are produced by the prediction engine. As the confidence level decreases, predictions produced by the prediction engine may be exceeded by competing predictions produced by other prediction engines (if any).

Additionally, and according to some embodiments, the prediction center/prediction engine may be configured to implement a recorder that maintains a record of the predictions generated and their corresponding feedback. These records may be beneficial in various aspects, for example, a developer of the prediction engine may receive records from a large number of mobile computing devices indicating that the prediction engine is continuously generating inaccurate predictions. In turn, the developer of the prediction engine may revisit the configuration of the prediction engine to improve its accuracy. The prediction centers between different mobile computing devices may also be configured to exchange information with one another in order to identify advanced trends that are observed and that may be used to enhance the overall user experience. For example, the prediction centers may identify each other that when most mobile computing devices enter a particular geographic area (e.g., the perimeter of a movie theater), the user of the mobile computing device manually places their mobile computing device in a silent mode. This identification, in turn, can be used to provide suggestions to the user to place their mobile computing device in a silent mode when entering a particular geographic area. The identification can also be used to suggest that automatic rules are set such that when the mobile computing device enters a particular geographic area, the mobile computing device will automatically enter a silent mode, thereby eliminating the need for the user to access his or her mobile computing device and manually place the mobile computing device in a silent mode.

In addition to the techniques described above, the prediction center may also be configured to implement one or more "filters," which may be used to further enhance the manner in which predictions are generated within the mobile computing device. According to some embodiments, the filter may be used to provide an additional layer of processing that helps to reduce or eliminate the occurrence of predictions that, while correct and reliable (within the scope of the prediction engine), are not practical and functional in real-world scenarios. For example, consider a scenario in which a lock screen application on a mobile computing device represents a consumer application, where the lock screen application displays static icons for a camera application, as well as dynamic icons for applications that are most likely to be accessed by a user (e.g., based on the current time). In this embodiment, the lock screen application may issue a request to the prediction center for a prediction associated with the "applications" prediction category in an attempt to identify applications that should be associated with the dynamic icons displayed within the lock screen application. Consider further that in this embodiment, a single prediction engine is associated with the "application" prediction category, where the single prediction engine determines that the camera application is most likely to be accessed by the user (as this typically occurs when the screen application display is locked). It is worth noting that in this embodiment, this prediction is somewhat meaningless, as it would be wasteful to display two different icons for the same camera application within the lock screen application. Thus, a filter may be used to help prevent these situations from occurring, for example, the filter may be configured to remove a camera application from predictions associated with an "application" prediction category when a lock screen application is active on a mobile computing device.

In addition, the prediction center/prediction engine may also be configured to implement one or more caches, which may be used to reduce the amount of processing that occurs when generating predictions. According to some embodiments, a prediction may be accompanied by a "validity parameter" that indicates when the prediction should be removed from the cache storing the prediction, after generation. The validity parameter (also referred to in this section as expiration information) may define, for example, a time-based expiration, an event-based expiration, and the like. In this way, when the prediction engine frequently receives requests for predictions for a particular prediction category, the prediction engine may generate and cache predictions in order to substantially reduce future processing that would otherwise occur when processing duplicate prediction requests. Note that the prediction center/prediction engine may be configured to cache predictions using various methods. For example, when the available cache is limited, the prediction center/prediction engine may be configured to generate a threshold number of predictions (e.g., within a time window), and when the threshold is met, transition to cache predictions and reference the cache for subsequent requests for predictions (as long as the expiration information indicates that the predictions are valid).

Additionally, it should be noted that the architecture of the prediction center may be configured such that the different entities described in this section (including the prediction engine, prediction classes, filters, recorders, etc.) are capable of being modular components within the mobile computing device. In one architectural approach, each entity may be configured to implement a set of Application Programming Interface (API) function calls that enable the entity to communicate with the prediction center and provide the different functions described in this section. For example, according to the architectural approach, an entity may be configured as a self-contained executable that may operate outside of a prediction center and that is capable of providing the various functions described in this section. In another architectural approach, each entity can be configured as a bundle, with the prediction center understanding its format and enabling it to serve as a platform for implementing the functionality of the entity. According to the method, the prediction center may be configured to, for example, parse different file system paths (e.g., upon initialization) to identify different bundles residing within the mobile computing device. In this way, bundles can be conveniently added to, updated in, and removed from a file system of a mobile computing device, thereby facilitating a modular configuration that can evolve efficiently over time without requiring extensive updates to the mobile computing device (e.g., operating system upgrades). Note that the foregoing architectures are exemplary, and any architecture that enables the various entities described in this section to communicate with each other and provide their different functionality may be used.

Embodiments therefore present techniques for collecting and organizing behavioral data to enable a mobile computing device to provide meaningful predictions for its end user. A more detailed discussion of these techniques is illustrated below and described in connection with fig. 39_1, 39_2, 39_3A through 39_3C, 39_4A through 39_4B, 39_5A through 39_5C, and the mobile device 100 illustrated in fig. 1A, which show detailed illustrations of systems and methods that can be used to implement these techniques.

Fig. 39_1 illustrates a block diagram of different components of a mobile computing device 39_100 configured to implement various techniques described in this section, in accordance with some embodiments. More specifically, FIG. 39_1 illustrates a high-level overview of a mobile computing device 39_100, which, as shown, is configured to implement a prediction center 39_102 and various consumer applications 39_ 112. According to some embodiments, the prediction center 39_102 various consumer applications 39_112 may be implemented in an Operating System (OS) (not shown in fig. 39_ 1) configured to execute on the mobile computing device 39_ 100. As also shown in fig. 39_1, the prediction center 39_102 may be configured to manage various recorders 39_105, various prediction categories 39_106, various prediction engines 39_108, and various filters 39_ 110. The prediction center 39_102 may also implement a manager 39_104 configured to act as a mediator between the prediction engine 39_108 and the consumer application 39_112, e.g., the manager 39_104 may receive the predictions generated by the prediction engine 39_108 (shown in fig. 39_1 as predictions 39_114) and forward the predictions 39_114 to the consumer application 39_ 112. The prediction center 39_102 may also be configured to receive feedback information 39_116 from the consumer applications 39_112 and provide the feedback information 39_116 to the prediction engine 39_108 so that they may produce more accurate predictions 39_114 over time. Thus, FIG. 39_1 provides a high-level overview of various components that may be used to implement the techniques described in this section.

Fig. 39_2 illustrates a block diagram of a more detailed view 39_200 of certain components of the mobile computing device 39_100 in fig. 39_1, according to some embodiments. As shown in fig. 39_2, each prediction engine 39_108 may be configured to include one or more learning models 39_202, corresponding states 39_204, and a list of prediction categories 39_106 to which the prediction engine 39_108 has assigned itself as an expert. According to some embodiments, the learning model 39_202 may represent an algorithm configured to analyze information (e.g., state 39_204) and generate predictions, which may enhance the overall experience of the user as the user operates the mobile computing device 39_ 100. According to some embodiments, the status 39_204 may be collected from various sources within the mobile computing device 39_100, such as feedback information 39_116 provided by a consumer application, information collected by sensors of the mobile computing device 39_100 (e.g., Global Positioning System (GPS) sensors, microphone sensors, temperature sensors, accelerometer sensors, etc.), information provided by external sources (e.g., applications executing on the mobile computing device 39_100, OS kernels, etc.), and so forth.

As also shown in fig. 39_2, the manager 39_104 may be configured to manage various recorders 39_105, various prediction classes 39_106, various prediction engines 39_108, and various filters 39_ 110. As described above, these entities may be implemented using various architectural methods, for example, the entities may be stand-alone executable files that are external to the prediction center 39_102 and communicate with the hypervisor 39_104 via API commands, the entities may be bundles that are stored in a file system of the mobile computing device 39_100 and that can be interpreted/implemented by the hypervisor 39_104, and so on. As also shown in fig. 39_2, the manager 39_104 may implement an aggregator 39_220 configured to merge multiple predictions 39_114 (e.g., when generated by different prediction engines 39_ 108). Further, as shown in FIG. 39_2, the manager 39_104 may be configured to maintain a record of the consumer applications 39_112 interacting with the prediction center 39_ 102. As described in more detail in this section, these records may be used to associate the prediction engine 39_108 with the consumer application 39_112 registered to receive predictions from the prediction engine 39_108 asynchronously.

Additionally, as shown in fig. 39_2, the prediction center 39_102 may be configured to implement a cache 39_206 that enables the prediction center 39_ 102/prediction engine 39_108 to cache the predictions 39_114 in an attempt to increase processing and energy consumption efficiency at the mobile computing device 39_ 100. As shown in fig. 39_2, the cache 39_206 may include entries 39_208, where each entry 39_208 includes a prediction 39_114 and expiration information 39_210 indicating how long the prediction 39_114 is considered valid. The expiration information 39_210 may include, for example, a time-based expiration, an event-based expiration, and the like. As such, when the prediction engine 39_108 frequently receives requests for predictions 39_114 for a particular prediction category 39_106, the prediction engine 39_108 may generate and cache predictions 39_114 to substantially reduce the amount of processing that would otherwise occur at the mobile computing device 39_100, thereby improving performance.

Fig. 39_3A illustrates a method 39_300 for advanced initialization and operation of the prediction engine 39_108, according to some embodiments. As shown in fig. 39_3A, the method 39_300 begins at step 39_302, where the prediction engine 39_108 loads one or more learning models 39_ 202. At optional step 39_304, the prediction engine 39_108 loads the previously established states 39_204 associated with the one or more learning models 39_ 202. According to some embodiments, the previously established state 39_204 may be retrieved from any storage resource (e.g., local non-volatile memory, cloud storage, etc.) available to the prediction engine 39_ 108. At step 39_306, the prediction engine 39_108 issues a request to the prediction center 39_102 as an expert on at least one prediction category 39_106 (and provides the prediction 39_ 114). At step 39_308, the prediction engine 39_108 receives a request to synchronously provide the predictions 39_114 or asynchronously provide the predictions 39_114 for at least one prediction class 39_ 106. At step 39_310, the prediction engine 39_108 asynchronously and/or synchronously provides predictions according to one or more learning models 39_202, where each prediction 39_114 includes confidence level information. At step 39_312, the prediction engine 39_108 receives feedback information indicating the level of accuracy associated with the provided prediction 39_ 114. Such feedback information 39_116 may be used to "train" the learning model 39_202 and improve the overall accuracy of its predictions 39_ 114. For example, when the prediction engine 39_108 generates a prediction 39_114 that a user of the mobile computing device 39_100 will take a particular action, and the consumer application 39_112 provides feedback indicating that the prediction 39_114 is valid (i.e., the user has taken the particular action), the prediction engine 39_108 may increase the confidence level advertised by the prediction engine 39_108 when producing similar subsequent predictions 39_ 114. At step 39_314, the prediction engine 39_108 updates one or more learning models 39_202 according to the feedback information.

Fig. 39_3B illustrates a method 39_330 for synchronously providing predictions 39_114 at prediction engine 39_108, according to some embodiments. As shown in fig. 39_3B, the method 39_330 begins at step 39_332, where the prediction engine 39_108 receives a request to synchronously provide predictions 39_114 for a particular prediction class 39_ 106. According to some embodiments, the request may be generated by the prediction center 39_102 on behalf of the consumer application 39_112 that is requesting the prediction 39_114 for the particular prediction category 39_ 106. Alternatively, the request may be generated by the consumer application 39_112 and provided directly to the prediction engine 39_ 108. In this way, the overall involvement of the prediction center 39_102 may be reduced or even eliminated relative to the prediction center 39_102 serving as an intermediary between the prediction engine 39_108 and the consumer application 39_ 112. At step 39_334, the prediction engine 39_108 identifies at least one learning model 39_202 associated with a particular prediction class 39_ 106. At step 39_336, the prediction engine 39_108 generates predictions 39_114 for a particular prediction class 39_106 from at least one learning model 39_ 202. At step 39_338, the prediction engine 39_108 associates the prediction 39_114 with the confidence level information. At step 39_340, prediction engine 39_108 provides prediction 39_ 114. More particularly, and depending on the configuration (e.g., as described above in connection with step 39_ 332), the prediction engine 39_108 may provide the predictions 39_114 to the prediction center 39_102 or directly to the consumer application 39_ 112. Conversely, when other prediction engines 39_108 provide similar predictions 39_114, the predictions 39_114 are aggregated with other predictions 39_114 (if any) (e.g., aggregated by aggregator 39_220 when predictions 39_114 are provided to prediction center 39_ 102).

Fig. 39_3C illustrates a method 39_350 for asynchronously providing predictions 39_114 at prediction engine 39_108, according to some embodiments. As shown in fig. 39_3C, the method 39_350 begins with step 39_352, where the prediction engine 39_108 receives a request to asynchronously provide predictions 39_114 for a particular prediction class 39_ 106. At step 39_354, the prediction engine 39_108 identifies at least one learning model 39_202 associated with a particular prediction class 39_ 106. At step 39_356, the prediction engine 39_108 identifies at least one trigger associated with a particular prediction class 39_106 and/or at least one learning model 39_ 202. At step 39_358, the prediction engine 39_108 determines whether a trigger is active/occurring. At step 39_358, if the prediction engine 39_108 determines that the trigger is activated, the method 39_350 proceeds to step 39_ 360. Otherwise, the method 39_350 repeats at step 39_358 until a trigger is activated/occurs. At step 39_360, the prediction engine 39_108 generates a prediction 39_114 of a particular prediction class 39_106 from at least one learning model 39_ 202. At step 39_362, the prediction engine 39_108 associates the prediction 39_114 with confidence level information. At step 39_364, prediction engine 39_108 provides predictions 39_114 (e.g., to prediction center 39_102 for aggregation).

Fig. 39_4A illustrates a method 39_400 of a consumer application 39_112 requesting to synchronously receive predictions 39_114, according to some embodiments. As shown in fig. 39_4A, the method 39_400 begins at step 39_402, where the consumer application 39_112 issues a request for a prediction 39_114 of a particular prediction category 39_ 106. According to some embodiments, the consumer application 39_112 may be configured to issue a request to the prediction center 39_102, where the prediction center 39_102 interfaces with the prediction engine 39_108 registered as an expert on a particular prediction category 39_ 106. Alternatively, the consumer application 39_112 may be configured to issue the request directly to the prediction engine 39_108, for example, when the prediction engine 39_108 is the only expert on a particular prediction category 39_106 within the mobile computing device 39_ 100. At step 39_404, the consumer application 39_112 synchronously receives the predictions 39_114 for the particular prediction category 39_106 in conjunction with the request issued at step 39_ 402. At step 39_406, the consumer application 39_112 observes behavior (e.g., user behavior) at the mobile computing device 39_100 to determine whether the prediction 39_114 is accurate. At step 39_408, the consumer application 39_112 provides feedback information 39_116 indicating the level of accuracy associated with the prediction 39_ 114.

Fig. 39_4B illustrates a method 39_450 by which a consumer application 39_112 registers to asynchronously receive predictions 39_114, according to some embodiments. As shown in fig. 39_4B, the method 39_450 begins with step 39_452, where the consumer application 39_112 issues a request to asynchronously receive predictions 39_114 for a particular prediction category 39_ 106. At step 39_454, the consumer application 39_112 asynchronously receives the predictions 39_114 for the particular prediction category 39_ 106. At step 39_456, the consumer application 39_112 observes behavior (e.g., user behavior) at the mobile computing device 39_100 to determine whether the prediction 39_114 is accurate. At step 39_458, the consumer application 39_112 provides feedback information 39_116 indicating the level of accuracy associated with the prediction 39_ 114.

Fig. 39_5A illustrates a method 39_500 for managing registration of a prediction engine 39_108 at a prediction center 39_102, according to some embodiments. As shown, the method 39_500 begins at step 39_500, where the manager 39_104 of the prediction center 39_102 receives a request from the prediction engine 39_108 to act as the prediction engine 39_108 and provide the prediction 39_114 of at least one prediction category 39_ 106. At step 39_504, the supervisor 39_104 adds the prediction engine 39_108 to the list of prediction engines 39_108 assigned to provide the predictions 39_114 of the at least one prediction class 39_ 106. At optional step 39_506, the manager 39_104 assigns a weight to the prediction engine 39_108 based on the historical performance metrics associated with the prediction engine 39_ 108. At optional step 39_508, the manager 39_104 initializes the filter 39_110 (if any) associated with the prediction engine 39_108 and/or the at least one prediction category 39_ 106. At step 39_510, the manager 39_104 updates the configuration of the prediction center 39_102 to enable the consumer application 39_112 to issue a request to synchronously and/or asynchronously receive predictions 39_114 associated with at least one prediction category 39_ 106.

Fig. 39_5B illustrates a method 39_550 for synchronously providing predictions 39_114 to consumer applications 39_112 at the prediction center 39_102, according to some embodiments. As shown in FIG. 39_5B, the method 39_550 begins at step 39_552, where the manager 39_104 receives a request from the consumer application 39_112 to synchronously provide the predictions 39_114 for a particular prediction category 39_ 106. One exemplary scenario may involve activating a messaging application at the mobile computing device 39_100 and issuing a request for a prediction 39_114 regarding the three contacts with which the user operating the messaging application is most likely to communicate. At step 39_554, the manager 39_104 identifies a list of prediction engines 39_108 assigned to the particular prediction category 39_ 106. Continuing with the foregoing exemplary scenario, consider further that two different prediction engines 39_108 have registered themselves as experts on the "people" prediction category 39_ 106. At step 39_556, the manager 39_104 queries each prediction engine 39_108 included in the list of prediction engines 39_108 for the prediction 39_ 114.

At step 39_558, the manager 39_104 receives from each prediction engine 39_108 included in the list of prediction engines 39_108 a corresponding prediction 39_114 associated with the confidence level information. Continuing with the foregoing exemplary scenario, consider further that two prediction engines 39_108 provide predictions 39_114, each of which includes a separate list of three contacts that the user is most likely to contact. For example, the first list may include the values read as "Greg: 0.7 "," Amy: 0.5 "and" mom: 0.3 "(where the name (e.g.," Greg ") indicates the predicted individual to be contacted and the number following the name (e.g., 0.7) indicates the corresponding confidence level that the predicted individual will be contacted), the second list may include entries read as" mom: 0.7 "," Greg: 0.4 "and" Julie: entry of 0.2 ". At step 39_560, the manager 39_104 updates the confidence level information associated with the predictions 39_114 according to the weights (if any) assigned to the respective prediction engines 39_ 108. For example, if the prediction engine 39_108 that generated the first list had an assigned weight of 0.75 based on the continuously poor performance observed by the manager 39_104 (e.g., via the feedback information 39_116), the confidence level information for each entry in the first list would be reduced by 0.75. At step 39_562, the manager 39_104 aggregates the predictions 39_114 according to their associated confidence level information (e.g., using the aggregator 39_ 220). Continuing with the previous embodiment, and assuming no weights are applied in step 39_560, step 39_562 would involve manager 39_104 building the following update list: "Greg: 1.1 "(i.e., 0.7+0.4 ═ 1.1)," Mom: 1.0 "(i.e., 0.3+0.7 ═ 1.0)," Amy: 0.5 "and" Julie: 0.2 ", wherein for" Julie: the 0.2 "entry is removed because the messaging application only needs to receive predictions about three contacts. At step 39_564, the manager 39_104 provides the prediction 39_114 to the consumer application 39_112 according to the aggregated prediction 39_114, which prediction will include "Greg: 1.1 "," Mom: 1.0 "and" Amy: 0.5".

Fig. 39_5C illustrates a method 39_570 for asynchronously providing predictions 39_114 to consumer applications 39_112 at the prediction center 39_102, according to some embodiments. As shown, the method 39_570 begins at step 39_572, where the manager 39_104 receives a request from the consumer application 39_112 to asynchronously receive predictions 39_114 for a particular prediction category 39_ 106. At step 39_574, the manager 39_104 identifies a list of prediction engines 39_108 assigned to the particular prediction category 39_ 106. At step 39_576, the manager 39_104 notifies each prediction engine 39_108 included in the list of prediction engines 39_108 to asynchronously provide the prediction 39_114 associated with the particular prediction class 39_ 106. At step 39_578, the manager 39_104 receives from each prediction engine 39_108 included in the list of prediction engines 39_108 the corresponding prediction 39_114 associated with the location level information. At step 39_580, the manager 39_104 updates the confidence level information associated with the prediction 39_114 according to the weight (if any) assigned to the corresponding prediction engine 39_ 108. At step 39_582, the manager 39_104 aggregates the predictions 39_114 according to their associated confidence level information. At step 39_584, the manager 39_104 provides the prediction 39_114 to the consumer application 39_112 in accordance with the aggregated prediction 39_ 114.

Exemplary method and apparatus for expert center

The described embodiments set forth techniques for implementing various "prediction engines" that may be configured to provide different kinds of predictions within a mobile computing device. According to some embodiments, each prediction engine may assign itself as an "expert" on one or more "prediction categories" within the mobile computing device. When a consumer application issues a prediction request of a particular category, and two or more prediction engines respond with their respective predictions, the "prediction center" may be configured to receive and process the predictions prior to responding to the request. Processing the predictions may include removing duplicate information that exists between predictions, ranking the predictions according to a confidence level advertised by the prediction engine, and so forth. In this way, the prediction center can extract multiple predictions into an optimized prediction and provide the optimized prediction to the consumer application.

In some embodiments, a method for synchronously providing predictions to an application executing on a mobile computing device is provided, the method comprising: at a prediction center executing on a mobile computing device: receiving a request from the application to synchronously provide predictions for a prediction category; identifying one or more prediction engines associated with the prediction category; receiving one or more predictions generated by the one or more prediction engines based on the request; aggregating the one or more predictions to produce a prediction requested by the application; and providing the prediction to the application. In some embodiments, aggregating the one or more predictions comprises performing one or more operations selected from: removing duplicate predictions from the one or more predictions; filtering the one or more predictions according to one or more filters implemented by the prediction center; for each of the one or more predictions: adjusting a confidence level associated with the prediction according to a weight assigned to a prediction engine that produced the prediction, wherein the confidence level associated with the prediction was generated by the prediction engine when producing the prediction; and ranking each of the one or more predictions according to a confidence level associated with the prediction. In some embodiments, the method comprises: prior to receiving the prediction request: for each prediction engine of the one or more prediction engines: receiving a request from a prediction engine for the prediction engine to be an expert on the prediction category; and associating a prediction engine with the prediction category. In some embodiments, the method comprises: after generating the prediction requested by the application: establishing a validity parameter associated with the prediction; associating a validity parameter with the prediction; the prediction and validity parameters are stored in a cache. In some embodiments, the validity parameter defines one or more of a time-based expiration or a trigger-based expiration. In some embodiments, the method comprises: after storing the prediction and validity parameters in the cache: receiving a second request from a second application to synchronously provide predictions for the prediction category; locating the prediction within a cache; and when the validity parameter associated with the prediction indicates that the prediction is valid: the prediction is provided to the second application. In some embodiments, the prediction category is included in a plurality of prediction categories managed by the prediction center, and each prediction category of the plurality of prediction categories is associated with: activation and deactivation of applications executing on the mobile computing device, contacts known to the mobile computing device, Global Positioning System (GPS) information that may be provided to the mobile computing device, notifications processed by the mobile computing device, or physical inputs made to the mobile computing device. In some embodiments, the method comprises: after providing the prediction to the application: receiving feedback information from the application indicating an accuracy of the prediction; and providing the feedback information to the one or more prediction engines, wherein the feedback information is utilizable by the one or more prediction engines to increase an accuracy of subsequent predictions produced by the one or more prediction engines.

In some embodiments, a method for asynchronously providing predictions to an application executing on a mobile computing device is provided, the method comprising: at a prediction center executing on a mobile computing device: receiving a request from the application to asynchronously provide predictions for a prediction category; identifying one or more prediction engines associated with the prediction category; each of the one or more prediction engines is notified to asynchronously provide one or more predictions in accordance with the request. In some embodiments, the method comprises: after notifying each of the one or more prediction engines: receiving one or more predictions; aggregating the one or more predictions to produce a prediction required by the application; and providing the prediction to the application. In some embodiments, the method comprises: after generating the prediction requested by the application: establishing a validity parameter associated with the prediction; associating a significance parameter with the prediction; the prediction and validity parameters are stored in a cache. In some embodiments, the validity parameters are provided by the one or more prediction engines and define one or more of a time-based expiration or a trigger-based expiration. In some embodiments, aggregating the one or more predictions comprises performing one or more operations selected from: removing duplicate predictions from the one or more predictions; screening the one or more predictions according to one or more filters implemented by the prediction center; for each of the one or more predictions: adjusting a confidence level associated with the prediction according to a weight assigned to a prediction engine that produced the prediction, wherein the confidence level associated with the prediction was generated by the prediction engine when producing the prediction; and ranking each of the one or more predictions according to a confidence level associated with the prediction. In some embodiments, the one or more prediction engines asynchronously provide the one or more predictions to the prediction center in response to a trigger-based event occurring at the mobile computing device. In some embodiments, the method comprises: prior to receiving the prediction request: for each prediction engine of the one or more prediction engines: receiving a request from a prediction engine for the prediction engine to be an expert on the prediction category; and associating a prediction engine with the prediction category. In some embodiments, the prediction category is included in a plurality of prediction categories managed by the prediction center, and each prediction category of the plurality of prediction categories is associated with: activation and deactivation of applications executing on the mobile computing device, contacts known to the mobile computing device, Global Positioning System (GPS) information that may be provided to the mobile computing device, notifications processed by the mobile computing device, or physical inputs made to the mobile computing device.

In some embodiments, there is provided a mobile computing device configured to generate predictions as a function of what a user should be, the mobile computing device comprising: a processor configured to perform: a prediction center configured to act as an intermediary between one or more prediction engines and one or more applications, wherein the prediction center manages a plurality of prediction categories; one or more prediction engines, wherein each prediction engine of the one or more prediction engines functions as an expert on at least one prediction category of a plurality of prediction categories managed by a prediction center; and one or more applications, wherein each of the one or more applications is configured to perform steps comprising: the method includes issuing a request to a prediction center for a prediction for a particular prediction category of the plurality of prediction categories, and receiving a prediction from the prediction center in accordance with the request, wherein the prediction is an aggregation of at least two predictions produced by a prediction engine that is an expert on the particular prediction category. In some embodiments, aggregating the at least two predictions comprises performing one or more operations selected from: removing duplicate predictions from the at least two predictions; filtering the at least two predictions according to one or more filters implemented by a prediction center; for each of the at least two predictions: adjusting a confidence level associated with the prediction according to a weight assigned to a prediction engine that produced the prediction, wherein the confidence level associated with the prediction was generated by the prediction engine when producing the prediction; and ranking each of the at least two predictions according to a confidence level associated with the prediction. In some embodiments, the prediction center is configured to perform steps comprising: after providing the prediction to the application: receiving feedback information from the application indicating an accuracy of the prediction; and providing the feedback information to prediction engines that generate the at least two predictions, wherein the feedback information is utilizable by the prediction engines to increase accuracy of subsequent predictions generated by the prediction engines. In some embodiments, each prediction category of the plurality of prediction categories is associated with: activation and deactivation of applications executing on the mobile computing device, contacts known to the mobile computing device, Global Positioning System (GPS) information that may be provided to the mobile computing device, notifications processed by the mobile computing device, or physical inputs made to the mobile computing device.

Section 10: context monitoring, context notification, context prediction and efficient context monitoring

The material in this section "context monitoring, context notification, context prediction, and efficient context monitoring" describes device context monitoring, context notification, context prediction, and efficient context monitoring in accordance with some embodiments, and provides information that complements the disclosure provided herein. For example, a person-in-one section of this section describes monitoring the operational context of a computing device, which supplements the disclosure provided herein, such as those related to the collection/storage of usage data (fig. 3A-3B), the creation/storage of trigger conditions (fig. 4A-4B), and the provision of relevant content to a user based on the usage data and trigger conditions (e.g., methods 600 and 800). In some embodiments, the context monitoring/prediction details discussed in this section are used to provide context information used to provide data to improve the presentation of search results and other suggested content for any of the methods discussed herein (e.g., to supplement method 600,800,1000,1200,2200,2280,2900 or any other method discussed herein that may benefit from using additional context information).

Concise summary of context monitoring/prediction

Systems, methods, and non-transitory computer-readable storage media for monitoring a current context of a computing device are disclosed. In some implementations, the context daemon can collect context information about the computing device. The context information may include current device hardware state information. The context information may include current software state information. The context may be derived or implied from a combination of hardware state information, software state information, or any other type of state information. For example, the derived context may be a user state (e.g., user activity, sleep, running, etc.) derived from or implied by hardware or software state information.

In some implementations, the context information can be reported by the context monitor to the context daemon. The context monitor may be a specialized means for collecting context information monitored by the context daemon. A context monitor may be an application, utility, tool, etc. that constructs, uses, or generates hardware or software state information for other purposes and reports the state information to a context daemon. After collecting the context information, the context daemon can store the current context of the computing device in a central location so that a context client (e.g., software, applications, utilities, operating system, etc.) can obtain the current context information from a single source. In some implementations, the context daemon can generate and/or collect historical context information. The historical context information may include old or outdated context information. Historical context information may be derived from the context information. Thus, the context daemon can provide a central repository of context information that the context client (e.g., process) can use to determine the current context of the computing device.

Systems, methods, and non-transitory computer-readable storage media for notifying a context client of a change in a current context of a computing device are disclosed. In some implementations, the context client can register to callback when the context daemon detects the specified context. For example, a context client may specify a context of interest to the context client. When the context daemon detects that the current context of the computing device corresponds to a registered context, the context daemon can notify the context client that the current context matches a context of interest to the context client. Thus, the context client need not independently obtain the programming required for context updates and detect context changes relevant or of interest to the context client.

Systems, methods, and non-transitory computer-readable storage media for efficiently monitoring an operating context of a computing device are disclosed. In some implementations, the context daemon and/or the context client can be terminated to conserve system resources. For example, if the context daemon and/or the context client are idle, they may be shut down to conserve battery power or free up other system resources (e.g., memory). When an event occurs that requires the context daemon and/or the context client to run (e.g., a current context change), the context daemon and/or the context client may be restarted to process the event. Thus, system resources may be conserved while still providing relevant context information collection and callback notification functionality.

Systems, methods, and non-transitory computer-readable storage media for predicting a future context of a computing device are disclosed. In some implementations, the context daemon can use historical context information to predict future events and/or context changes. For example, the context daemon may analyze historical context information to predict user sleep patterns, user exercise patterns, and/or other user activities. In some implementations, the context client may register a callback for a predicted future context. For example, the context client may request that the notification be received ten minutes before the event and/or context change is predicted. The context daemon can use the prediction to notify the context client prior to the predicted event.

Detailed description of context monitoring/prediction

Determining a current context

FIG. 40_1 is a block diagram of an exemplary system 40_100 for monitoring, predicting, and notifying changes in the operating context of a contextual client computing device. The computing device may be, for example, a desktop computer, a laptop computer, a smart phone, a tablet computer, or any other type of computing device. For example, system 40_100 may be configured to run on a computing device. In some implementations, the system 40_100 can include a context daemon 40_ 102. For example, context daemon 40_102 may be a background process executing on the computing device. For example, context daemon 40_102 may be a process included in the operating system of the computing device.

In some implementations, context daemon 40_102 may be configured to collect information about the current operating context of the computing device. For example, the context information may include information describing internal and/or external context of the computing device. In some implementations, the internal context information may include hardware state information. For example, hardware state information may identify the hardware being used and how the hardware is used. If the hardware is a wireless transceiver for communicating with another device, the hardware state information may identify the other device, when the connection was created, how much data was transferred, etc. In some implementations, the internal context information may include software state information. For example, the state information for the calendar application may include calendar events, meetings, names of contacts to attend the meetings, start and end times of the meetings, and the like.

In some implementations, the external context information can include user activity. For example, external context information may be derived from hardware state information and/or software state information. For example, context daemon 40_102 may derive user behavior (e.g., sleep mode, operational mode, eating mode, travel mode, etc.) from hardware and/or software state information, as described further below.

In some implementations, the context daemon 40_102 may include a monitor group 40_104 for collecting various types of context information. Each of the monitor banks 40_106 in the monitor bank 40_104 may be configured to collect context information about a corresponding context item. For example, the monitor group 40_106 may be a process external to the context daemon 40_ 102. The monitor group 40_106 may be a dynamically loaded software package that may be executed within the context daemon 40_ 102.

In some implementations, the monitor bank 40_106 can include a context monitor 40_ 108. For example, the context monitor 40_108 may be configured to collect information about the current context of the computing device. In some implementations, the set of monitors 40_106 can include a history monitor 40_ 110. For example, the history monitor 40_110 may be configured to collect or determine a historical context of the computing device, as described further below.

In some implementations, each of the monitor banks 40_106 in the monitor bank 40_104 can be configured to collect a particular type of context information. For example, the context daemon 40_102 may load many different monitor groups 40_ 106. Each monitor group 40_106 may be configured to collect different context information from different sources 40_130 within the computing device. For example, one monitor group 40_106 may collect context information regarding bluetooth context items, while another monitor group 40_106 may collect context information regarding lock state context items.

In some implementations, the monitor bank 40_106 may be configured to collect device location context information from the location API 40_ 132. For example, the context monitor 40_108 may receive current Global Navigation Satellite System (GNSS) location data received by a GNSS receiver from the location API 40_ 132. The monitor bank 40_106 may receive current cellular and/or WiFi-derived location data from the location API 40_ 132.

In some implementations, the monitor bank 40_106 may be configured to collect lock state context information from the lock state API 40_ 134. For example, the context monitor 40_108 may collect lock state context information that describes a current lock state (e.g., locked, unlocked, etc.) of the computing device. For example, a user of a computing device must unlock the computing device in order to use or interact with the computing device. When the device is locked, the device will not accept user input. When the device is unlocked, the device will accept user input. For handheld devices with touch screen displays, when the device is unlocked, the display may light up and may accept touch input from the user. When the touch screen device is locked, the display may be dark and the touch screen display will not accept touch input. Thus, the locked state of the computing device may provide evidence that the user has interacted with the computing device.

In some implementations, the monitor bank 40_106 can be configured to collect application context information from the application manager API 40_ 136. For example, the context monitor 40_108 may receive information from the application manager API 40_136 describing which applications are currently running on the computing device, how long the applications have been running, when the applications are invoked, and/or which applications are currently in focus (e.g., in the foreground, visible on the display).

In some implementations, the monitor group 40_106 may be configured to collect bluetooth context information from the bluetooth API 40_ 138. For example, the context monitor 40_108 may receive information from the bluetooth API 40_138 describing active bluetooth connections, including the identity and type of bluetooth devices connected to the computing device, when the connection was established, and for how long (e.g., duration) the computing device has been connected to the bluetooth device.

In some embodiments, the monitor bank 40_106 may be configured to collect headphone interface information from the headphone API 40_ 138. For example, the context monitor 40_108 may receive information from the headset API 40_138 describing whether a wired headset or headset (or other device) is currently connected to the headset interface of the computing device. In some implementations, the monitor bank 40_106 may receive information from the headset API 40_138 regarding the type of device connected to the headset interface.

In some implementations, the monitor bank 40_106 may be configured to collect other context information from other device state APIs 40_ 142. For example, the context monitor 40_108 may receive information from the other status APIs 40_138 describing WiFi connections, phone connections, application usage, calendar events, photos, media usage information, battery charge status, and/or may be used to describe or infer current internal and/or external context of the computing device.

In some implementations, the monitor group 40_106 can be dynamically loaded into the context daemon 40_102 as needed. For example, when the context daemon 40_102 needs location context information (e.g., the client has requested location information), the context daemon 40_102 may load the location specific monitor group 40_106 into the monitor group 40_ 104. After loading, the context monitors 40_108 in the monitor group 40_106 will begin to collect context specific to the current location. By loading the monitor bank 40_106 as needed, the context daemon 40_102 can conserve system resources of the computing device, such as memory and battery power. In some implementations, the monitor group 40_106 may be an external process, such as the reporting client 40_ 124. The context daemon 40_102 may invoke the external process monitor group 40_106 to collect context information as needed. For example, the context daemon 40_102 may load or invoke the monitor group 40_106 in response to receiving a callback request, as described below.

In some implementations, the context daemon 40_102 may receive context information from the reporting client 40_ 124. For example, the reporting client 40_124 (context client) may be any software running on a computing device that generates or collects context information and reports the context information to the context daemon 40_ 102. For example, a mapping application running on the computing device may use the location API 40_132 to obtain location information to determine how to route a user from a starting location to a destination location. In addition to determining the route, the mapping application may report the location information obtained from the location API 40_132 to the context daemon 40_ 102. Thus, although the reporting client 40_124 is not built to collect and report contextual information, such as the monitor bank 40_106, the reporting client 40_124 may be configured to report contextual information as the reporting client 40_124 obtains contextual information in the course of performing its primary function.

In some implementations, the context daemon 40_102 may include the current context 40_ 112. For example, the current context 40_112 may be an in-memory repository of context information received from the monitor bank 40_104 (e.g., monitor bank 40_106) and/or reporting client 40_ 124. When the monitor group 40_104 and/or reporting client 40_124 reports context information to the context daemon 40_102, the context daemon 40_102 may update the current context 40_112 with the newly received context information. Thus, the current context 40_112 may include context information (e.g., context items) that describe the current context of the computing device.

Fig. 40_2A and 40_2B illustrate exemplary current contexts 40_200 and 40_ 250. Fig. 40_2A illustrates an embodiment of context items that may constitute the current context 40_ 200. For example, the current context 40_200 (e.g., the current context 40_112) may include context information of the computing device at time T. For example, the current context 40_200 may include a context item representing the current lock state (false). The current context 40_200 may include a context item (false) indicating the insertion state of the headphone interface. The current context 40_200 may include a context item (false) representing the state of charge of the battery. The current context 40_200 may include a context item (false) indicating a connection state of the bluetooth transceiver. The current context 40_200 may include a context item that identifies the application (social application) that is currently the focus of the computing device. For example, the context information shown in the current context 40_200 may be received from the monitor bank 40_106 and/or from the reporting client 40_ 124.

FIG. 40_2B illustrates an embodiment of a new context item added to the current context 40_ 250. The current context 40_250 (current context 40_112) may include context information for the computing device at a later time T'. For example, current context 40_250 includes a new context item that identifies the current location of the computing device. When a new location monitor group 40_106 is loaded into context daemon 40_102 and begins reporting location context information to context daemon 40_102, a new location context item may be added. For example, the new set of location monitors 40_106 may be loaded in response to a request for location information from a contextual client.

Callback request

Referring to fig. 40_1, in some implementations, the context daemon 40_102 can expose an API that allows context client software running on the computing device to access (e.g., query, view, etc.) information in the current context 40_ 112. In some implementations, the context daemon 40_102 may receive a request from a requesting client 40_126 (context client) to callback the requesting client 40_126 when the context daemon 40_102 detects a particular context. For example, the requesting client 40_126 may send a callback request to the context daemon 40_ 102. The callback daemon 40_102 may store callback request information in the callback registry 40_ 114. The callback registration table 40_114 may be a memory repository for callback information. For example, the callback request may specify a predicate (e.g., context condition) for notifying the requesting client 40_ 126. The callback request may include a client identifier of the requesting client 40_ 126.

In some implementations, when a callback request is received, the callback registry 40_114 may generate a unique identifier for the callback request and store the callback request identifier, the client identifier, and the callback predicate in the callback predicate database 40_ 116. The context daemon 40_102 may return a callback request identifier to the requesting client 40_126 in response to receiving the callback request. When the context information in the current context 40_112 satisfies the predicate, the context daemon 40_102 will notify the requesting client 40_ 126. For example, the callback notification may include a callback request identifier such that the requesting client 40_126 may determine the callback request corresponding to the notification. For example, the requesting client 40_126 may register a number of callback requests with the context daemon 40_ 102. When the callback daemon 40_102 sends a callback notification to the requesting client 40_126, the requesting client 40_126 may use the callback request identifier to determine which callback request the callback notification relates to.

In some implementations, the context daemon 40_102 may load the monitor group 40_106 in response to receiving a callback request from the requesting client 40_ 126. For example, the context daemon 40_102 may support delayed initialization of the monitor group 40_ 106. In other words, the context daemon 40_102 may load and initialize the monitor group 40_106 when a service callback request is needed. For example, if no client is interested in location information, the context daemon 40_102 may not load the location monitor set 40_106 so that system resources (e.g., battery, memory, etc.) are not wasted monitoring unneeded context items. However, when a callback request for a location context item is received, the content daemon 40_102 may load, initialize, or invoke the monitor set 40_106 associated with the location context item and begin receiving context information regarding the location of the computing device.

In some implementations, the monitor group 40_106 can be a software plug-in to the context daemon 40_ 102. For example, the monitor group 40_106 may be software code (e.g., library, object code, java jar file, etc.) that may be dynamically loaded into the context daemon 40_102 and executed to monitor context information. In some implementations, the monitor group 40_106 can be a separate process external to the context daemon 40_ 102. For example, the monitor group 40_106 may be a separate executable file that the context daemon 40_102 may call to monitor and report context information.

FIG. 40_3 illustrates an exemplary callback predicate database 40_ 300. For example, the predicate database 40_300 may correspond to the predicate database 40_116 of FIG. 1. In some implementations, each entry 40_302 through 40_316 in the predicate database 40_300 may include a callback request identifier. For example, when the context daemon 40_102 receives a callback request from the requesting client 40_126, the context daemon 40_102 may generate a unique request identifier for the callback request. As described above, the context daemon 40_102 may return a callback request identifier to the requesting client 40_126 in response to a callback request. Context daemon 40_102 may associate the generated callback request identifier with the client identifier and callback predicate in the callback database. When the context daemon 40_102 sends a callback notification to the requesting client 40_126, the context daemon 40_102 may include a callback identifier in the notification so that the requesting client 40_126 may determine why the context daemon 40_102 sent the callback notification. For example, the requesting client 40_126 may send a plurality of callback requests to the context daemon 40_ 102. The requesting client 40_126 may determine based on the callback request identifier from which callback request the context daemon 40_102 is sending notifications.

In some implementations, each entry 40_ 302-40 _316 in the predicate database 40_300 can include a client identifier and a callback predicate. The client identifier may correspond to a client that is required to receive a notification (e.g., called back) when the current context of the computing device satisfies the respective predicate specified by the requesting client 40_ 126. In some implementations, the client identifier can be generated by a launch daemon configured to launch (e.g., execute, invoke, etc.) a process on the computing device, as described further below. For example, entry 40_302 corresponds to a requesting Client 40_126 having a Client identifier "Client _ ID 1" that is required to be notified when the current context of the computing device indicates that the device is locked and the focused application is a music application. In other words, the contextual Client (e.g., requesting Client 40_126) corresponding to the Client identifier "Client _ ID 1" specifies a predicate for notifying (e.g., recalling) that the contextual Client is a device locked and that the application currently being used by the user is a music application. For example, a predicate specified by the requesting client 40_126 may identify one or more context conditions (e.g., hardware state values, software state values, derived contexts, etc.) separated by logical (e.g., boolean) operators. When the current state of the computing device corresponds to the specified predicate, the context daemon 40_102 will notify (e.g., callback) the requesting client 40_ 126.

In some implementations, the predicate may include a temporal component. For example, predicates may include "before" and/or "after" operators (expressions) that allow the requesting client 40_126 to indicate an amount of time before or after an event (e.g., a state change, a context change, etc.) when the requesting client 40_126 should receive a notification. For example, the context daemon 40_102 may receive calendar application state information indicating that a meeting at a particular time in the future is scheduled. The requesting client 40_126 can register a predicate (e.g., entry 40_316) that specifies that the context daemon 40_102 should notify the requesting client 40_126 thirty minutes before the meeting. When the current time corresponds to thirty minutes before the meeting, the context daemon 40_102 may send a notification to the requesting client 40_ 126. Similarly, the context daemon 40_102 may predict future events (e.g., user sleep cycle, user arrival at home, user arrival at office, user wake up, etc.) based on historical context information. For example, the requesting client 40_126 may register a predicate (e.g., entry 40_306) that specifies that the context daemon 40_102 should notify the requesting client 40_126 thirty minutes before the predicted period of user sleep. When the current time corresponds to thirty minutes before the predicted sleep period, the context daemon 40_102 may send a notification to the requesting client 40_ 126. Similarly, the requesting client 40_126 may register a predicate (e.g., entry 40_310) that specifies that the context daemon 40_102 should notify the requesting client 40_126 five minutes after predicting that the user wakes up based on the predicted sleep period. For example, the context daemon 40_102 may send a notification to the requesting client 40_126 when the current time corresponds to five minutes after the user wakes up.

Event stream

Referring to FIG. 40_1, in some implementations, the context daemon 40_102 may include a historical knowledge repository 40_ 118. For example, while the current context 40_112 includes context information reflecting the current state of the computing device, as described above, the historical knowledge 40_118 includes historical context information. The historical knowledge 40_118 may be an internal memory store of historical context information. For example, the calendar knowledge 40_118 can include event streams that represent changes in context (e.g., state) over time. For example, each event or context item tracked in the current context 40_112 has a corresponding value. When a context item in the current context 40_112 changes value, the old value may be recorded in the history knowledge 40_ 118. By analyzing the state change, a start time, an end time, and a duration may be calculated for each context item value.

FIG. 40_4 is a diagram 40_400 illustrating exemplary value changes over time associated with a context item. For example, the map 40_400 includes a current context 40_402 indicating current values for lock, headset, charge, bluetooth, focus application, sleep, and location context items. The graph 40_400 includes past (historical) values 40_404 over time for the same context item. By analyzing changes in the context item values, context daemon 40_102 may determine the start time, end time, and duration of each value associated with a context item, as shown in FIG. 5 below.

Fig. 40_5 is a diagram 40_500 illustrating an exemplary event stream associated with a context item. In some implementations, each state change represented in the graph 40_400 of the graph 40_4 can be converted into a data object (e.g., object 40_502) that includes data describing the duration of time a particular state exists within the system and metadata associated with the state. In some implementations, the history monitor 40_110 may be configured to convert current and/or historical context information into a history event stream object. For example, when a change in the value of a context item corresponding to a particular monitor group 40_106 is detected, the history monitor 40_110 may generate a historical event stream object based on the previous value of the context item. For example, some context monitors 40_106 may be configured to periodically report the status of software and/or hardware components of the computing device. For example, the context monitor 40_106 may be configured to periodically report bluetooth status information. The reported bluetooth status information may include a sequence of identical status values followed by a status change. For example, the context monitor 40_106 may report that the status of bluetooth is "off, on". The history monitor 40_110 may combine a series of "off" bluetooth context values, determine the start time and end time of the "off" value, and calculate how long the bluetooth component is in the "off state.

In some implementations, the history monitor 40_110 may collect additional information (e.g., metadata) for the event stream object. For example, continuing with the bluetooth embodiment described above, the history monitor 40_110 may determine that the bluetooth context item has an "on" value and request additional information from the bluetooth API 40_ 138. For example, the history monitor 40_110 may receive information from the bluetooth API 40_138 identifying the type of bluetooth device connected to the computing device, the bluetooth protocol used, the amount of data transferred over the bluetooth connection, and/or any other information related to the bluetooth connection.

In another embodiment, while the context monitor 40_108 may be configured to collect current context information (e.g., call information) from the telephony API (e.g., phone number called, time the call was initiated, time the call was terminated, etc.), the history monitor 40_110 may collect event stream metadata for the call from a contacts API (e.g., name of the person called, etc.) or a call history API (e.g., last name of the person called, duration of the call, etc.) and add this additional information to the event stream object of the telephony context item. Accordingly, the history monitor 40_110 may be configured to generate or collect additional data about historical events to make the historical events (e.g., event streams, event stream objects) more valuable for historical reference and predicting future events. After the history monitor 40_110 collects or generates event stream metadata, the history monitor 40_110 may store the event stream metadata in the history knowledge repository 40_ 118. In some implementations, the context daemon 40_102 and/or history monitor 40_110 may store event stream objects (e.g., including start times, stop times, durations, and/or metadata) in the history database 40_ 120.

Fig. 6 illustrates an exemplary historical event stream database 40_ 600. For example, the historical event stream database may correspond to the historical database 40_ 120. The example historical event stream database 40_600 represents a conceptual description of historical event stream data stored in the historical database 40_600 and may not reflect the actual implementation of the historical database 40_ 600. Those skilled in the art will recognize that the historical event stream data in the database 40_600 may be organized and stored in many different ways.

In some implementations, the historian 40_600 may include event flow tables 40_602 through 40_ 614. For example, each event flow table 40_ 602-614 may correspond to a single event flow (e.g., context entry). Each event stream table (e.g., table 40_602) may include records (e.g., 40_616, 40_618, etc.) corresponding to event stream objects in the event stream. For example, the "locked" event stream table 40_602 may include event stream object records 40_616, 40_618 describing the locked (or unlocked) state of the "locked" event stream. The event stream object record may include a "start time" field with a Timestamp (TS) value indicating when the event started. The event stream object record may include a "duration" field indicating the duration (D) of the event. The event stream object record may include state information describing a state change corresponding to the event (e.g., "locked: false" indicates that the device is not locked).

In some implementations, the event stream object record may include metadata describing other data associated with the event. For example, when generating historical event stream data, the history monitor 40_110 may collect and/or generate metadata describing additional attributes of the system state or the surrounding environment at the time of the event. For example, for the "charge" event stream 40_606, the history monitor 40_110 may collect information related to the battery state of charge at the beginning and/or end of the charging event (e.g., percent charge, charge level, etc.). For the bluetooth event stream 40_608, the history monitor 40_110 may collect information related to the type of bluetooth device connected to the computing device and/or the source of media transmitted to the bluetooth device. For the location event stream 40_612, the history monitor 40_110 may convert raw location data (e.g., grid coordinates, GNSS data, cell tower identification data, Wi-Fi network identifiers, etc.) into location words (e.g., home, company, school, grocery store, restaurant name, etc.) that the human user may understand.

In some implementations, the history monitor 40_110 may generate or obtain more accurate location information than the context monitor 40_ 108. For example, the context monitor 40_108 may provide current (e.g., instantaneous) location information without much, if any, processing. This initial position data may be inaccurate due to various problems with positioning techniques (e.g., signal multipath problems, difficulty connecting to adequate satellites, etc.). Given the additional time and the additional data, the location may be determined more accurately. Since the history monitor 40_110 processes historical data (rather than current or instantaneous data), the history monitor 40_110 may take time to obtain more accurate location information from, for example, the location API 40_ 132. This additional metadata describing the event may be stored in the event stream record of the history database 40_ 600.

In some implementations, the history monitor 40_110 may obtain history information about the context items after initializing the monitor set 40_ 106. For example, if the monitor bank 40_106 is configured to monitor location context, the context daemon 40_102 may load, invoke, and/or initialize the monitor bank 40_106 as needed, as described above. When the monitor set 40_106 is initialized, the context monitor 40_108 will collect context information for the location context item. However, after the set of monitors 40_106 is initialized, there is no historical data for the location context item since the location context item has not been previously monitored. Thus, in some implementations, the history monitor 40_110 may request location history data from the location API 40_132 and generate historical context information (e.g., event streams, event stream objects, etc.) based on the location history data received from the location API 40_ 132.

Event stream privacy

In some implementations, each event stream may have a corresponding privacy policy. In some implementations, the event stream may be configured with a default privacy policy. In some implementations, an administrator user can provide input to the computing device to configure the privacy policy for each event stream (e.g., each context item). For example, the privacy policy corresponding to each event stream may change over time.

In some implementations, the event stream of the context item may have a privacy policy that prevents maintenance of historical information of the context item. For example, an event stream corresponding to a location context item may have a policy that does not allow for maintaining a history of the location of the computing device. When such a "no history" policy is configured for event streams, the history monitor 40_110 will not generate historical context information (e.g., event stream objects) for the event streams.

In some implementations, the event stream of context items may have a privacy policy that specifies an amount of time (e.g., time-to-live) that historical context information should be stored before being deleted. For example, an event stream corresponding to a "focus application" context item may have a time-to-live policy that specifies that event stream data for the "focus application" context item should be deleted for a longer amount of time than specified (e.g., 3 days, 1 month, etc.). The context daemon 40_102 may periodically perform maintenance on the event stream to delete event stream objects longer than the amount of time specified in the time-to-live policy.

In some implementations, the event stream of context items can have a timestamp resolution policy. For example, when the timestamp resolution policy is in effect, the history monitor 40_110 makes the precise timestamp associated with an event (e.g., a state change) in the event stream less precise. For example, a change of location event may have a timestamp accurate to milliseconds. When a parsing policy is applied to the event stream, the history monitor 40_110 may use a less accurate timestamp accurate to seconds or minutes. For example, by using less accurate timestamps for location event streams, the system may protect user privacy by preventing the contextual client from determining the exact time the user moved.

In some implementations, the event stream of context items can have a storage location policy. For example, a computing device may be configured with different storage locations corresponding to the security state of the computing device. For example, a computing device may have a class "a" database that is only accessible when the computing device is unlocked (e.g., a user has entered a password to unlock the device). The computing device may have a "class B" database that may be accessed after a first unlock (e.g., without a subsequent unlock) after the computing device is restarted or started. The computing device may have a "C" class database that is accessible at all times (e.g., regardless of password entry). The storage location privacy policy for the event stream may identify the type of database in which the corresponding event stream data is stored.

Efficient context monitoring

In some implementations, the computing device can be configured to terminate software running on the computing device when the software is not in use. For example, an operating system of a computing device may be configured to identify idle processes. The operating system may shut down (e.g., terminate, disable) idle processes to free up memory or conserve battery resources for use by other components (e.g., software, hardware) of the system. However, if the operating system terminates the idle context daemon 40_102, the context daemon 40_102 will no longer be able to monitor the current context of the system and will not be able to notify the requesting client 40_126 of the context change. Similarly, if the operating system terminates the idle requesting client 40_126, the requesting client 40_126 will not run to receive callback notifications from the context daemon 40_ 102. The following paragraphs describe various mechanisms by which the context daemon 40_102 and/or the requesting client 40_126 may be restarted to handle context monitoring and/or callback operations.

FIG. 40_7 is a block diagram of an exemplary system 40_700 for providing context callback notifications to a requesting client 40_ 126. For example, system 40_700 may correspond to system 40_100 of FIG. 1 above. In some implementations, system 40_700 can include a launch daemon 40_ 702. For example, the launch daemon 40_702 may be configured to launch (e.g., call, launch, execute, initialize, etc.) an application, utility, tool, and/or other process on the computing device. The launch daemon 40_702 may be configured to monitor processes and terminate idle processes on the computing device.

In some implementations, the launch daemon 40_702 may launch the requesting client 40_ 126. For example, a process (e.g., an operating system, a user application, etc.) running on the computing device may invoke the requesting client 40_ 126. The launch daemon 40_702 may receive the message corresponding to the invocation and may launch the requesting client 40_ 126. Upon launching the requesting client 40_126, the launch daemon 40_702 may provide the requesting client 40_126 with a client identifier 40_704 that may be used to identify the requesting client 40_126 within the computing device.

In some implementations, the client identifier 40_704 can be a token (e.g., encrypted data) that the launch daemon 40_702 generates and that is assigned to the requesting client 40_126 by the launch daemon 40_ 702. The launch daemon 40_702 may store a mapping between the token and the software package corresponding to (e.g., defining) the requesting client 40_126 in the client identifier database 40_ 706. The token may be generated such that the token itself does not identify the corresponding requesting client 40_ 126. However, when the launch daemon 40_702 later receives the token, the launch daemon 40_702 may use the token to look up the corresponding requesting client 40_126 in the client identifier database 40_ 706. Thus, launch daemon 40_702 may use the token as an index to identify the corresponding requesting client when the token is opaque to other software within the computing device.

In some implementations, the client identifier 40_704 may be an instance identifier corresponding to a particular instance of the requesting client 40_ 126. In some implementations, the client identifier 40_704 may identify the software package (e.g., application, utility, tool, etc.) in all instances of the requesting client 40_ 126. For example, when the launch daemon 40_702 launches the first instance of the requesting client 40_126, the client identifier 40_704 may identify the first instance of the requesting client 40_ 126. When the requesting client 40_126 is terminated (e.g., because the requesting client 40_126 has become idle), the same client identifier 40_704 may be used to identify subsequent instances of the requesting client 40_126 that were initiated by the launch daemon 40_ 702. The launch daemon 40_702 may use a similar mechanism as the requesting client 40_126 to launch the context daemon 40_ 102.

In some implementations, the requesting client 40_126 may send a callback request 40_708 to the context daemon 40_ 102. For example, as described above, the callback request 40_708 may include the client identifier 40_704 and a callback predicate. As described above, upon receiving the callback request 40_708, the context daemon 40_102 may store the client identifier 40_704 and the predicate in the predicate database 40_ 116.

In some implementations, when the requesting client 40_126 sends the callback request 40_708 to the context daemon 40_102, the requesting client establishes a communication session 40_709 between the requesting client 40_126 and the context daemon 40_ 102. In some implementations, the system 40_700 can be configured such that a communication session between the requesting client 40_126 and the context daemon 40_102 can only be initiated by the requesting client 40_ 126. For example, the context daemon 40_102 may not be able to establish a communication session directly with the requesting client 40_ 126. Thus, in some implementations, the context daemon 40_102 can only communicate with the requesting client 40_126 (e.g., send a callback notification to the requesting client) while the communication session 40_709 established by the requesting client 40_126 is still open.

In some implementations, context daemon 40_102 may collect context information about events that occur on the computing device. For example, as described above, the context daemon 40_102 may collect context information from the monitor bank 40_106 and the reporting client 40_ 124. In some implementations, the context daemon 40_102 may store the current context in the context database 40_ 712. For example, context daemon 40_102 may store the current context in context database 40_712 to facilitate restoring the context information to context daemon 40_ 102. When the context daemon 40_102 is terminated and restarted, the context daemon 40_102 may restore the current context (e.g., the now old context) from the context database 40_712 while the context daemon 40_102 waits for a context update from the monitor bank 40_ 106.

In some implementations, the context daemon 40_102 can determine whether the current context corresponds to a predicate received from the requesting client 40_ 126. For example, when obtaining new context data that updates the current context (e.g., changes the state of a context item), the context daemon 40_102 may compare the callback predicate stored by the context daemon 40_102 in the callback registry 40_114 or predicate database 40_116 to the context item in the current context 40_112 to determine whether the current context matches (corresponds to) the condition specified by the predicate. The context daemon 40_102 may send a notification 40_701 to the requesting client 40_126 when the current context matches a predicate registered by the requesting client 40_ 126. For example, as described above, the notification 40_701 may identify a callback request previously sent by the requesting client 40_126 to the context daemon 40_ 102. Thus, when the context daemon 40_102 detects the current context of interest to the requesting client 40_126, the context daemon 40_102 may notify (e.g., callback) the requesting client 40_ 126.

40_8A and 40_8B are block diagrams illustrating an exemplary system 40_700 for restarting a requesting client that has been terminated. For example, in FIG. 8A, the system 40_700 has determined that the requesting client 40_126 is idle and has terminated the requesting client 40_126 (e.g., the dashed outline of the requesting client 40_126 indicates termination). In FIG. 8A, context daemon 40_102 is still running. However, since the requesting client 40_126 has been terminated, the communication session 40_709 is also terminated.

FIG. 40_8B is a block diagram illustrating an exemplary mechanism for restarting a requesting client 40_126 using the system 40_ 700. Continuing with the embodiment of FIG. 8A, context daemon 40_102 may receive context information that matches callback predicates registered by the requesting client 40_ 126. In response to determining that the context information matches the callback predicate, the context daemon 40_102 may attempt to notify the requesting client 40_ 126. In attempting to notify the requesting client 40_126, the context daemon 40_102 may determine that the communication session 40_709 between the context daemon 40_102 and the requesting client 40_126 has been terminated. In response to determining that the communication session 40_709 is terminated, the context daemon 40_102 may request the launch daemon 40_702 to restart the requesting client 40_ 126. For example, the context daemon 40_102 may send the client identifier 40_704 received from the requesting client 40_126 to the launch daemon 40_702 that is requested to restart the requesting client 40_ 126.

In some implementations, the launch daemon 40_702 may launch the requesting client 40_126 upon receiving the client identifier 40_ 704. For example, launch daemon 40_702 may determine that context daemon 40_102 is authorized to request that the requesting client 40_126 be restarted based on the client identifier provided by context daemon 40_ 102. For example, if the requesting client 40_126 has not previously requested a callback from the context daemon 40_102 and has not provided the client identifier 40_704 to the context daemon 40_102, the context daemon 40_102 will not have the client identifier 40_704 (e.g., token).

In some implementations, upon reboot, the requesting client 40_126 may send a callback request 40_708 to the context daemon 40_ 102. For example, the requesting client 40_126 may establish a new communication session 40_802 between the requesting client 40_126 and the context daemon 40_102 by sending a callback request 40_708 to the context daemon 40_ 102. Once the communication session 40_802 is established, the context daemon 40_102 may send a notification 40_710 to the requesting client 40_126 to notify the requesting client 40_126 that the callback predicate provided by the requesting client 40_126 has been satisfied by the current context.

40_9A and 40_9B are block diagrams illustrating an exemplary system 40_700 that restarts context daemons that have been terminated. For example, in FIG. 9A, system 40_700 has determined that context daemon 40_102 is idle and has terminated context daemon 40_102 (e.g., the dashed outline of context daemon 40_102 indicates termination). In FIG. 9A, the requesting client 40_126 is still running. However, since context daemon 40_102 has been terminated, communication session 40_709 is also terminated.

FIG. 40_9B is a block diagram illustrating an exemplary mechanism for restarting the context daemon 40_102 using the system 40_ 700. Continuing with the embodiment of FIG. 9A, the system 40_700 can restart the context daemon 40_102 in response to receiving a message from the requesting client 40_126 for the context daemon 40_ 102.

In some implementations, the requesting client 40_126 can detect that the communication session 40_709 between the requesting client 40_126 and the context daemon 40_102 has been terminated. In response to detecting that the communication session 40_709 has been terminated, the requesting client 40_126 may reestablish the communication session by sending a message to the terminated context daemon 40_ 102. In some implementations, the requesting client can send a message to the context daemon 40_102 using the message system 40_ 902. Message system 40_902 of system 40_700 may determine that context daemon 40_102 is not running and send a message to launch daemon 40_702 to cause launch daemon 40_702 to restart context daemon 40_ 102. In response to receiving the message, launch daemon 40_702 may restart context daemon 40_ 102. Once the context daemon 40_102 is running, the message system 40_902 may send a message from the requesting client 40_126 to the context daemon 40_102, thereby reestablishing the communication channel between the requesting client 40_126 and the context daemon 40_ 102.

In some implementations, upon reboot, the context daemon 40_102 may restore its callback registry 40_114 and current context 40_ 112. For example, the callback registry 40_114 may be restored from the predicate database 40_ 116. The current context 40_112 may be restored from the context database 40_ 712. Upon reboot, the context daemon 40_102 may load the monitor set 40_106 needed to collect context information to service callback requests recovered from the predicate database 40_ 116. As described above, the context daemon 40_102 can update the current context 40_112 with the context information reported by the loaded monitor bank 40_104, and notify the requesting client 40_126 when a context item in the current context 40_112 matches a predicate registered by the requesting client 40_ 126.

40_10A and 40_10B are block diagrams of an exemplary system 40_700 illustrating restarting a context daemon that has been terminated and a requesting client. For example, in FIG. 40_10A, the system 40_700 has determined that both the context daemon 40_102 and the requesting client 40_126 are idle and have terminated the context daemon 40_102 and the requesting client 40_126 (e.g., the dashed outlines indicate termination). In FIG. 40_10A, the communication session 40_709 is terminated because both the context daemon 40_102 and the requesting client 40_126 are terminated.

FIG. 40_10B is a block diagram illustrating an exemplary mechanism for restarting the context daemon 40_102 and the requesting client 40_126 using the system 40_ 700. Continuing with the embodiment of FIG. 40_10A, the system 40_700 can restart the context daemon 40_102 in response to receiving a message from the intervention client 40_1002 for the terminated context daemon 40_ 102. For example, similar to the requesting client 40_126 in FIG. 9B, the intervention client 40_1002 may send a message to the now-terminated context daemon 40_ 102. The message system 40_902 may receive the message and determine that the context daemon 40_102 is not running. In response to determining that context daemon 40_102 is not running, message system 40_902 may send a message to launch daemon 40_702 to cause launch daemon 40_702 to restart context daemon 40_ 102.

In some implementations, upon reboot, the context daemon 40_102 may restore its callback registry 40_114 from the predicate database 40_ 116. Upon reboot, the context daemon 40_102 may restore its current context 40_112 from the context database 40_712 and may begin collecting updated context information, as described above. When the context daemon 40_102 determines that the registered predicate matches the current context information, the context daemon 40_102 may attempt to notify the requesting client 40_ 126. When the context daemon 40_102 determines that there is no communication session 40_709 between the requesting client 40_126 and the context daemon 40_102, the context daemon 40_102 may request the launch daemon 40_702 to restart the requesting client 40_126 so that the communication session may be reestablished, and the context daemon 40_102 may callback the requesting client 40_126, as described above with reference to FIG. 8B.

FIG. 40_11 is a block diagram of an exemplary system 40_1100 configured to restart a requesting client 40_126 and/or context daemon 40_102 based on device state information received by the launch daemon 40_ 702. For example, system 40_1100 may correspond to system 40_700 and may perform similar functions as system 40_700, as described above.

In some implementations, the launch daemon 40_702 may be configured to receive the device state 40_ 1104. For example, device state 40_1104 may be low-level, specific state data generated by various hardware and/or software components of the computing device. For example, the launch daemon 40_702 may receive a device state 40_1104 that includes location data generated by a location service component (e.g., a GPS receiver, a Wi-Fi or cellular data component, etc.) of the computing device. In some implementations, the device status 40_1104 may indicate a change in location, but may not provide high-level location information (e.g., a human-readable tag).

For example, the requesting client 40_126 may send a callback request 40_708 with a location-based predicate to the context daemon 40_ 102. The predicate may specify that the requesting client 40_126 should be notified that the current location (e.g., current context) of the computing device is the user's home (e.g., location-home). To determine that the device location is the user's home, the context daemon 40_102 and/or the monitor group 40_106 may collect information from the location API 40_132 and a contacts application running on the user device that defines the location of the "home" (e.g., defines the geographic location associated with the "home" tag). By comparing the location information from the location API 40_132 to the definition of "home" in the contacts application, the context daemon 40_102 can determine when the context item "location" equals "home". As shown in this embodiment, determining that the location predicate (e.g., "home") defined by the requesting client 40_126 is satisfied depends on combining the current geographic location data (e.g., grid coordinates) with the user data that associates the tag (e.g., "home") with the geographic location. Thus, the abstract location context "home" may be determined by analyzing concrete state data generated by the location service and contact applications of the computing device.

In some implementations, when the context daemon 40_102 receives the callback request 40_708 from the requesting client 40_126, the context daemon 40_102 may send a device state request 40_1102 to the launch daemon 40_702 to register an interest in a state change of a particular component of the computing device. When the launch daemon 40_702 receives the device state 40_1104, the launch daemon 40_702 may determine that a state change has occurred to the specified component and notify the context daemon 40_102 and/or the requesting client 40_ 126.

In some implementations, the device state request 40_1102 may specify that the launch daemon 40_702 should notify the context daemon 40_102 when a specified state change occurs. For example, when requesting client 40_126 to send a callback request to context daemon 40_102 specifying a location-based callback predicate, context daemon 40_102 may send a device state request 40_1102 to launch daemon 40_702 to request that launch daemon 40_702 notify context daemon 40_102 when launch daemon 40_702 detects a change in location component state.

In some implementations, the device state request 40_1102 may specify that the launch daemon 40_702 should notify the requesting client 40_126 when a specified state change occurs. For example, when requesting client 40_126 sends a callback request to context daemon 40_102 specifying a location-based callback predicate, context daemon 40_102 may send a device state request 40_1102 to launch daemon 40_702 requesting that launch daemon 40_702 notify requesting client 40_126 when launch daemon 40_702 detects a location component state change. In some implementations, the device status request 40_1102 may include a client identifier 40_704 corresponding to the requesting client 40_126, such that the launch daemon 40_702 may determine which requesting client 40_126 to notify.

40_12A and 40_12B are block diagrams illustrating an exemplary system 40_1100 for restarting a context daemon using a startup daemon. For example, in FIG. 40_12A, system 40_1100 has determined that both context daemon 40_102 and requesting client 40_126 are idle and that context daemon 40_102 and requesting client 40_126 have terminated (e.g., the dashed outline indicates termination). In FIG. 40_12A, since both the context daemon 40_102 and the requesting client 40_126 are terminated, the communication session 40_709 is also terminated.

FIG. 40_12B is a block diagram illustrating an exemplary mechanism for restarting the context daemon 40_102 using the startup daemon 40_702 of system 40_ 1100. As described above with reference to FIG. 40_11, the context daemon 40_102 may receive a callback request from the requesting client 40_126 specifying a context predicate that sends the notification 40_710 from the context daemon 40_102 to the requesting client 40_ 126. In response to receiving the predicate, context daemon 40_102 may send a device state request 40_1102 to launch daemon 40_702 to register interest in device state changes associated with the predicate. For example, if requesting client 40_126 specifies a location-based callback predicate, context daemon 40_102 may require launch daemon 40_702 to notify context daemon 40_102 when the location of the computing device changes. When the launch daemon 40_702 receives the device state 40_1104 indicating the change in location, the launch daemon 40_702 may attempt to notify the context daemon 40_ 102. Continuing with the embodiment of fig. 40_12A, because context daemon 40_102 is not running on the computing device, launch daemon 40_702 may determine that context daemon 40_102 is not running and launches (e.g., reboots, initiates, calls, executes, etc.) context daemon 40_ 102. Upon restarting the context daemon 40_102, the context daemon 40_102 may request the startup daemon 40_702 to restart the requesting client 40_126, as described above with reference to FIG. 8B.

40_13A and 40_13B are block diagrams illustrating an exemplary system 40_1100 for restarting a requesting client 40_126 using a startup daemon. For example, in FIG. 40_13A, the system 40_1100 has determined that both the context daemon 40_102 and the requesting client 40_126 are idle and have terminated the context daemon 40_102 and the requesting client 40_126 (e.g., the dashed outlines indicate termination). In FIG. 40_13A, since both the context daemon 40_102 and the requesting client 40_126 are terminated, the communication session 40_709 is also terminated.

FIG. 40_13B is a block diagram illustrating an exemplary mechanism for restarting a requesting client 40_126 using the startup daemon 40_702 of system 40_ 1100. As described above with reference to FIG. 40_11, the context daemon 40_102 may receive a callback request from the requesting client 40_126 specifying a context predicate that sends a callback notification 40_710 from the context daemon 40_102 to the requesting client 40_ 126. In response to receiving the predicate, the context daemon 40_102 may send a device state request 40_1102 to the launch daemon 40_702 to register an interest in a device state change associated with the predicate representing the requesting client 40_ 126. For example, when registering an interest in a device state change associated with a predicate, the context daemon 40_102 may provide the client identifier 40_704 to the launch daemon 40_ 702. For example, if the requesting client 40_126 specifies a location-based callback predicate, the context daemon 40_102 may require the launch daemon 40_702 to notify the requesting client 40_126 when the location of the computing device changes. When the launch daemon 40_702 receives the device state 40_1104 indicating the change in location, the launch daemon 40_702 may attempt to notify the requesting client 40_126 (e.g., identified by the client identifier 40_ 704). Continuing with FIG. 40_13A, because the requesting client 40_126 is not running on the computing device, the launch daemon 40_702 may determine that the requesting client 40_126 is not running and launches (e.g., reboots, initiates, calls, executes, etc.) the requesting client 40_ 126. Upon restarting the requesting client 40_126, the requesting client 40_126 may cause the launch daemon 40_702 to restart the context daemon 40_102 by sending a message to the context daemon 40_102, as described above with reference to FIG. 9B.

Predicting future events

In some implementations, the context daemon 40_102 may predict future events based on the event stream information. For example, the context daemon 40_102 may analyze historical context information (e.g., event streams, event stream objects, etc.) to determine historical user behavior patterns. Context daemon 40_102 may predict future user behavior based on these past behavior patterns. For example, predictable user behavior may include sleep patterns, work patterns, exercise patterns, diet patterns, and other repetitive user behavior. The context daemon 40_102 may determine when these user behaviors occur based on clues in the event stream that reflect how the user interacted with the computing device during these user activities.

For ease of explanation, the following description will describe an exemplary sleep prediction implementation based on historical device lock state event stream data. However, the mechanism for sleep prediction may also be used to predict other user behavior by analyzing other event stream data. For example, the context daemon 40_102 may use the location data to infer the user's mode of operation. The context daemon 40_102 may use the accelerometer data to infer user motion patterns. The context daemon 40_102 may use application data (e.g., restaurant registrations on a social media software application) to infer user eating patterns.

In some implementations, the context daemon 40_102 may use device lock state event flow data to determine and/or predict a user sleep mode. For example, if the computing device (e.g., handheld device, smartphone, etc.) remains locked for a long time (e.g., 5 hours or more), the context daemon 40_102 may infer that the user is sleeping. In some implementations, other event flow information (e.g., accelerometer data, application usage data, etc.) may be used to confirm sleep patterns and/or sleep predictions. In some implementations, context daemon 40_102 may perform the day's sleep prediction at some time after the user wakes up from the previous day's sleep and before the next predicted sleep cycle. For example, context daemon 40_102 may perform calculations to predict the next sleep cycle after detecting that the user has woken up from the current sleep cycle. For example, the context daemon 40_102 may detect that the user has woken up by determining that the current value of the "locked" context item is false (e.g., the user has unlocked the device) and that the current time is after the predicted sleep period has ended.

Slot mode prediction of future events

In some implementations, the context daemon 40_102 may perform slot-wise averaging to predict future events. For example, to predictively determine a sleeping user sleep mode, context daemon 40_102 may analyze the lock state event stream described above. The context daemon 40_102 may analyze the lock state event stream by dividing it into consecutive 24 hour time periods within the first 40_28 days. The context daemon 40_102 may divide each 24 hour period into 96 15 minute time slots. The context daemon 40_102 may determine the lock status for each 15 minute period of each 24 hour period. For example, if the computing device remains locked for the entire 15 minute time slot, the lock status of that time slot may be true (e.g., 1). If the computing device unlocks within a 15 minute slot, the lock state of the slot may be false (e.g., 0). The lock status data for the 15 minute time slot for each 24 hour time period may be combined to generate 28 data vectors representing each of the first 40_28 days. For example, each vector (e.g., of length 96) may include 96 lock state values corresponding to each of the 15 minute time slots within a day. The context daemon 40_102 may then average each 15 minute time slot over a 40_28 day period to determine the user's historical sleep pattern.

Fig. 40_14 is a diagram 40_1400 illustrating an embodiment of slot-wise averaging for predicting future events. For example, fig. 40_1400 illustrates using a device lock state to determine a sleep mode and predict future sleep periods. For example, each horizontal line represents a lock status data vector for a 24 hour period. The 24-hour period may range from t-n to t + n, where "t" is some time corresponding to about the middle (e.g., presumed, typical, calculated, etc.) of the user's sleep cycle, and "n" may be 40_ 12. For example, if a typical person is sleeping from 10 pm to 6 am, "t" may be 2 am. In fig. 40_1400, "C" represents the current day. Thus, C-1 was yesterday, C-2 was two days ago, C-7 was one week ago, and C-28 was four weeks ago. With 96 corresponding 15 minute time slots per day. For example, FIG. 40_1400 depicts 15 minute time slots corresponding to 3:30-3:45 AM, 5:00-5:15 AM, and 6:15-6:30 AM. Although only three 15-minute slots are shown on fig. 40_1400 to reduce clutter on the graph 40_1400, each vector has 96 15-minute slots, and the operations described with reference to the three 15-minute slots on fig. 40_1400 will be performed for each of the 96 slots within each 24-hour period.

Referring to one of the vectors C-1, 3:30 slots and 5:00 slots on FIG. 40_1400, the value of one (e.g., 1, true) indicates that the computing device remains locked throughout the corresponding 15 minute slot. During the 6:15 time slot, a zero value (e.g., 0, false) indicates that the computing device is unlocked at some time during the 15 minute time period. For example, the context daemon 40_102 may infer that the user must have woken up to unlock the computing device in a 6:15 time slot. The context daemon 40_102 may infer that the user is asleep when the device remains locked for a threshold period of time (e.g., 5 hours), as described further below.

To determine the probability that the user will keep the computing device locked (and thus asleep) during each 15-minute slot in the current day, the context daemon 40_102 may average the values of each 15-minute slot over the previous 28 days to predict the value of each 15-minute slot in the current 24-hour period. The context daemon 40_102 may use the average 15 minute slot value calculated for the current 24 hour period to identify periods of time in the current 24 hours that exceed a sleep threshold (e.g., 5 hours) at which the device may remain locked. For example, where the average of the 15-minute time slots is above a certain threshold (e.g., 0.5, 40_ 50%, etc.), the context daemon 40_102 may determine that the computing device will remain locked for the 15-minute time slot. The context daemon 40_102 may determine a series of 15 minute time slots with a continuity (or mostly continuity) greater than a threshold that, when combined, exceeds a sleep threshold time period. Once the series of 15 minute time slots is determined, context daemon 40_102 may identify the time period covered by the series of 15 minute time slots as the predicted sleep period.

In some implementations, context daemon 40_102 may perform a weighted average in the lock state data vector. For example, each vector may be weighted such that older lock state data has less of an impact on the average than newer lock state data. In some implementations, the context daemon 40_102 may perform short-term averaging over a series of recent days (e.g., over each of the last 7 days) and/or long-term averaging over a series of weeks (e.g., 7 days ago, 14 days ago, 21 days ago, 28 days ago). For example, short-term averaging may be better for predicting daily patterns, while long-term averaging may be better for predicting what a user does on a particular day of the week. For example, if today is saturday, the user's last saturday activity may better predict the user's today behavior than the user's yesterday (e.g., friday), especially if the user is working monday through friday.

Short term averaging

In some implementations, the context daemon 40_102 may determine the Probability (PS) that the device will remain locked in a 15 minute time slot using the following short term weighted average algorithm:

Where V1 corresponds to C-1, V2 corresponds to C-2, and so on, V7 corresponds to C-7, and λ is a weight between 0 and 1, determined experimentally. For example, a short term weighting algorithm may be used to calculate a weighted average every 15 minutes over the first seven days.

Long term average

In some implementations, the context daemon 40_102 may determine the Probability (PL) that a device will remain locked in a 15 minute time slot using the following long term weighted average algorithm:

where V7 corresponds to C-7, V14 corresponds to C-14, V21 corresponds to C-21, and V28 corresponds to C-28, and "λ" is a weight between 0 and 1, which is an experimentally determined value. For example, a long term weighting algorithm may be used to calculate a weighted average of 15 minutes for the same day of the week over the past four weeks.

In some implementations, the short-term weighted averaging algorithm and the long-term weighted averaging algorithm can be combined to generate a combined (e.g., composite, global, etc.) probability (P) that the computing device will remain locked for the 15-minute time slot, which is calculated as follows:

where "r" is an experimentally determined number (e.g., 0.5) that can be used to adjust the impact of the long-term weighted average on the probability calculation.

Proportional time slot value

Fig. 15 minutes depicts exemplary fig. 15 minutes 00 and 15 minutes 50 showing the calculation of the scaled slot value. For example, rather than assigning a true (1) and a false (0) value for each 15 minute time slot within the 24 hour period C-n as described above, context daemon 40_102 may determine how many of each 15 minute time slot the computing device is locked or unlocked during and assign a proportional value to the time slot representing the proportional amount of time within the time slot that the device is locked.

Referring to fig. 15 minutes 00, the shaded area of each 15 minute slot may represent the time within the slot in which the device is locked. For example, the device is locked during the entire 3:30-3:45 AM and 5:00-5:15 AM slots. Thus, the 3:30 and 5:00 time slots may be assigned a value of one (1). However, the computing device is locked only during a portion of the 6:15-6:30 am time slots. If the computing device is locked for the first 40-10 minutes of the 6:15 time slot, the 6:15 time slot may be assigned a value of 40-10/15 or 0.67, which represents the proportional amount of 15 minute time slots for which the device is locked, as shown in FIG. 15 minutes 50. If the computing device is repeatedly locked and unlocked (e.g., 5 minutes locked, 2 minutes unlocked, 1 minute locked, 5 minutes unlocked, etc.), the computing device may sum the time periods of locking, sum the time periods of unlocking, and calculate the proportion of 15 minute time slots during which the computing device is locked. In some implementations, a scale value may be determined for each 15 minute time slot within a 24 hour time period (e.g., a data vector). In some implementations, a proportional value for each 15 minute time slot may be used when calculating the short-term and/or long-term probabilities described above.

Generating sleep curves

Fig. 40_16A is a diagram 40_1600 illustrating an exemplary method for predicting a future context. For example, the method illustrated in fig. 40_16 may be used to predict future sleep periods for a user of a computing device. For example, each column in fig. 40_1600 (e.g., column 40_1602, column 40_1606) may represent a 15 minute time slot, as described above. The value of each 15 minute time slot may be represented by the height of the column and may correspond to the combined weighted average probability (P) of the time slot, as described above. For example, the probability (P) may range from zero (e.g., 0, 0%) to one (e.g., 1, 40 — 100%). The probability may represent, for example, a probability that the computing device will remain locked during a 15 minute time slot, as described above. The probability may be calculated based on a binary (e.g., 0, 1)15 minute slot value. The probability may be calculated based on a proportional 15 minute slot value.

In some implementations, the context daemon 40_102 may convert the probability map 40_1600 into a probability curve representing the sleep cycle of a user of the computing device. For example, the context daemon 40_102 may determine the sleep probability threshold 40_1606 to determine which 15 minute time slot corresponds to the user's sleep period. In some implementations, the sleep probability threshold 40_1606 can be determined dynamically. For example, given a minimum sleep period (e.g., 5 hours, 7 hours, etc.), context daemon 40_102 may determine a value (e.g., 0.65, 40_ 50%, etc.) of sleep probability threshold 40_1606 that forms a segment of consecutive 15 minute slots that is at least as long as the minimum sleep period and that includes 15 minute slots having a (e.g., probabilistic, average) value that exceeds sleep probability threshold 40_ 1606. In other words, the context daemon 40_102 may adjust the sleep probability threshold 40_1606 up or down until a series of 15 minute time slots, when combined, meet or exceed the minimum sleep period and have a value above the sleep probability threshold.

In some implementations, once the sleep probability threshold 40_1606 is determined, the context daemon 40_102 may determine the sleep period 40_1608 of the user based on consecutive 15 minute time slots. For example, sleep period 40_1608 may correspond to a period covered by consecutive 15-minute time slots. Referring to fig. 40_16, the sleep cycle may correspond to a period of time starting at 11 pm to ending at 7 am.

Fig. 40_16B is a diagram 40_1650 illustrating an exemplary method for converting slot-wise probabilities to probability curves. For example, to enable consistent prediction of a user's sleep cycle, it may be useful to generate a probability curve (e.g., similar to a bell-shaped curve) that monotonically increases as the user falls asleep (e.g., increasing the probability that the device will remain locked) and monotonically decreases as the user wakes (e.g., decreasing the probability that the device will remain locked).

In some implementations, to generate the probability curve 40_1652, the context daemon 40_102 may convert the probability (e.g., average) computed for each 15 minute slot to a binary (e.g., 1 or 0) value using the sleep probability threshold determined above. For example, 15-minute slots within a sleep period (e.g., above sleep threshold 40_1606) may be assigned a value of one (1), and 15-minute slots outside the sleep period may be assigned a value of zero (0). Once a binary value is assigned to each 15 minute time slot, context daemon 40_102 may fit a curve (e.g., probability curve 40_1652) to the binary value. Once generated, the context daemon 40_102 may use the probability curve 40_1652 to estimate the probability that the user will be in a sleep state at a particular time of day and/or for a particular period of time of day. For example, the context daemon 40_102 may use the probability curve 40_1652 to predict when the user is likely to be asleep in the future. Referring to FIG. 16B, since the calculated sleep time segment represented by FIG. 40_1650 is between 11 PM and 7 AM, the context daemon 40_102 may predict that the user will be in a sleep state between 11 PM and 7 AM in the future. For example, if the sleep prediction is done daily (e.g., after the user wakes up in the morning), the context daemon 40_102 may predict that the user will be in a sleep state between 11 pm and 7 am of the day.

Handling irregularities-outliers and missing data

In some implementations, the context daemon 40_102 may be configured to process outlier data within the historical event stream data. In some implementations, the context daemon 40_102 may be configured to process outlier 15 minute time slots that otherwise correspond to sleep time periods over a period of time. For example, a period of time that may be a sleep period candidate (e.g., a continuous 15 minute slot having a value that exceeds the sleep threshold and that, when combined, exceeds the minimum sleep period) may include 15 minute slots that do not exceed the sleep threshold. For example, if the minimum sleep period is 5 hours, there are at least twenty 15 minute slots within the sleep period. In the case where there are twenty 15-minute slots, there may be ten slots that exceed the sleep threshold, then one slot that does not exceed the sleep threshold (e.g., an outlier), and then nine slots that exceed the sleep threshold. An embodiment of this case can be seen with reference to fig. 40_16, where the outlier slot 40_1608 does not exceed the sleep threshold 40_1606 and is surrounded by slots (e.g., 40_1604) that exceed the sleep threshold 40_ 1606. When there are a few (e.g., one, two) outlier 15-minute slots within a segment of 15-minute slots that exceed the sleep threshold, then the context daemon 40_102 may treat the outlier 15-minute slot as if it exceeded the sleep threshold, such that a sleep time period (e.g., a sleep curve) as described above may be generated. For example, the context daemon 40_102 may ignore outlier 15 minute slots when determining that the segment corresponding to the sleep period continues for 15 minute slots. When converting the 15-minute time slot to a binary value to generate a probability curve (as shown in fig. 40_ 16B), the context daemon 40_102 may assign a value of one to the outlier 15-minute time slot so that a probability curve may be generated.

In some implementations, the context daemon 40_102 may be configured to process outlier dates (e.g., 24 hour time periods, historical data vectors, etc.) within the stream of historical events when predicting sleep time periods. For example, prior to calculating the short term average, the context daemon 40_102 may compare the lock event data (e.g., historical context data) for the first seven days. For example, the context daemon 40_102 may perform similarity analysis on the historical data vectors for each of the first seven 24 hour time periods. If the data for one of the seven days (e.g., the outlier date) is completely different from the other six days, the context daemon 40_102 may remove the outlier date from the short term average calculation. For example, it may be normal to produce a small time-to-date change in historical device lock status data for a 15 minute time slot. However, it is not normal to produce changes in large segments of locked data (e.g., corresponding to periods of sleep of the user). The context daemon 40_102 may detect outlier dates by comparing the inter-date patterns in the history data and detecting that the observed usage patterns for the day (e.g., user behavior) do not correspond to the observed usage patterns for other dates in the week. For example, when a 15 minute slot is typically locked on other dates, the context daemon 40_102 may determine that the same 15 minute slot is unlocked on an outlier date (24 hour period). Once the outlier date is detected, the context daemon 40_102 may omit the outlier date from the short term average calculation described above.

Similarly, prior to calculating the long-term average, the context daemon 40_102 may, for example, compare the lock event data (e.g., historical context data) for the same day for each of the previous four weeks. If the data for one of the days (e.g., outlier date) differs significantly from the other four days, the context daemon 40_102 may remove the outlier date from the long term average calculation. For example, it may be normal to produce a small inter-cycle variation in historical device lock status data for a 15 minute time slot. However, it is not normal to produce changes in large segments of locked data (e.g., corresponding to periods of sleep time for a user). The context daemon 40_102 may detect outlier dates by comparing inter-week patterns in the historical data and detecting that an observed usage pattern (e.g., user behavior) for one day does not correspond to an observed usage pattern for the same day in the previous weeks. Once the outlier date is detected, the context daemon 40_102 may omit the outlier date from the long term average calculation described above.

In some implementations, the context daemon 40_102 may detect outlier dates based on changes in user behavior patterns. For example, if the user is typically sleeping between 11 pm and 7 am, the historical lock event data will indicate that the device is kept (mostly) locked between 11 pm and 7 am. However, on very few days, the user may learn or work overnight, so the sleep period of the day may change to another period (e.g., 6 am to 12 pm). In some implementations, context daemon 40_102 may detect this change in sleep mode based on historical lock state data and remove the day from the average calculation.

In some implementations, the context daemon 40_102 may detect outlier dates based on known or commonly recognized constraints in human behavior. For example, a user may travel on a weekend and accidentally leave a computing device (e.g., a smartphone) at home for the entire weekend. In this case, the entire weekend (e.g., two days) device will remain locked, generating a piece of locked data that can be mistakenly interpreted by context daemon 40_102 as a sleep period. Context daemon 40_102 may detect this by comparing the time period (e.g., sleep time period) corresponding to the segment of locked data to a maximum sleep time period (e.g., 12 hours, 24 hours, etc.). For example, the maximum sleep period may be determined based on common sense (e.g., human sleep typically does not exceed 24 hours) or based on observed data (e.g., the maximum observed user sleep period is 10 hours). If the period of time exceeds the maximum sleep period, the context daemon 40_102 may ignore the date or dates corresponding to the period of time when performing the long and/or short term calculations described above.

In some implementations, the context daemon 40_102 may be configured to handle lost data in the stream of historical events. For example, a user may turn off a computing device for a period of time, or the device may lose battery power after being unplugged from an external power source for an extended period of time. When the device is turned off, the device cannot collect context information and cannot generate a historical event stream. When the computing device is turned on again, the context daemon 40_102 may attempt to predict future events (e.g., future sleep periods) based on the missing data corresponding to the time period that the device was turned off. In this case, context daemon 40_102 may determine that there are no event (e.g., context item) data values for the time period and ignore (e.g., omit) the date or dates (e.g., historical data vectors) corresponding to the time period when performing the short-term and/or long-term average calculations described above.

Scheduling activities based on predicted events

Fig. 40_17 shows an exemplary event stream 40_1700 that includes predicted future events. For example, using the mechanisms described above, context daemon 40_102 may predict future sleep periods 40_ 1702. In some implementations, the predicted future event can be used to schedule an activity (e.g., a context callback) within the computing device. Referring to FIG. 1, the requesting client 40_126 may request callback notifications prior to a predicted event. For example, the requesting client 40_126 can send a callback request including a predicate to the context daemon 40_102 specifying that the context daemon 40_102 should notify the requesting client 40_126 thirty minutes before the user falls asleep. When a callback request is received, the context daemon 40_102 may schedule a notification within 30 minutes before the start of the predicted sleep period. Similarly, the requesting client may send a callback request including a predicate to context daemon 40_102 specifying that context daemon 40_102 should notify requesting client 40_126 one hour after the user falls asleep. When a callback request is received, context daemon 40_102 may schedule a notification within one hour after the start of the predicted sleep period.

In some implementations, context daemon 40_102 may confirm the prediction of a future event based on the current context at the predicted time of the event. For example, if the requesting client 40_126 requests that the context daemon 40_102 notify the requesting client 40_126 thirty minutes after the predicted sleep period begins, the context daemon 40_102 may confirm that the user has actually gone to sleep at that time by analyzing the current context 40_112 (e.g., context item values) to determine if the device is locked. If the device is unlocked within thirty minutes after the predicted sleep period begins (e.g., the user is not asleep), the context daemon 40_102 will not notify the requesting client 40_ 126. In some implementations, other context information may be used to confirm the predicted sleep period. For example, accelerometer status may be used to confirm sleep periods. For example, most smartphone users will place the smartphone on a table or floor while asleep. Tables and floors are usually stationary objects. Thus, the smartphone does not generate a large amount (if any) of accelerometer data. If the smartphone is generating accelerometer data, the smartphone is most likely in the user's pocket while the user is moving. Thus, the accelerometer data may indicate that the user is moving and not falling asleep within the predicted sleep period.

In some implementations, context daemon 40_102 may improve the prediction of future events by identifying precursor events. For example, the context daemon 40_102 may analyze historical event stream data to identify relationships between user activities and predicted events. For example, a user may have a habit of checking a mail application, social networking application, news application, or other application before going to bed. Context daemon 40_102 may detect these patterns (e.g., use an alarm application and then sleep) and identify one or more precursor applications (e.g., a clock application, a news application, etc.). Once the one or more precursor applications are identified, context daemon 40_102 may use the precursor applications to predict that the user is going to sleep. For example, the context daemon 40_102 may determine, based on historical event data, that the user typically falls asleep within 40_10 minutes after using the alarm clock application. If context daemon 40_102 has predicted that the user will sleep at 11 PM and the user will use the alarm application at 10 PM, context daemon 40_102 may adjust the start of the predicted sleep period from 11 PM to 10:10 PM based on the precursor alarm application activity. In some implementations, the context daemon 40_102 may adjust the start of the predicted sleep period by adjusting the start time and stop time of the predicted sleep period without adjusting the duration of the predicted sleep period. In some implementations, the context daemon 40_102 may adjust the start of the predicted sleep period by adjusting the start time without adjusting the stop time of the predicted sleep period, thereby extending the duration of the predicted sleep period. Alternatively, when context daemon 40_102 detects that the user is using a precursor application (e.g., the current context indicates that the focused application is a precursor application), context daemon 40_102 may monitor the user's activity to determine when the user locks the computing device and, once the device is locked, begin the current sleep period.

Other use cases

In some implementations, a contextual client running on a computing device can use the sleep prediction described above to schedule background tasks when a user falls asleep. For example, an operating system process (e.g., an application updater) may need to schedule some system maintenance tasks (e.g., download and/or install application updates) while a user is asleep so that the user does not inconvenience system maintenance due to allocation of system resources. Context daemon 40_102 may analyze the state of various device components (e.g., hardware, software, etc.) to determine whether scheduled activities may interfere with any user activity, as described further below.

In some cases, the operating system process may require a user password (e.g., password) before performing the system maintenance task. Since the user will not be able to provide the password when the user falls asleep, the operating system process may request callback notifications from the context daemon 40_102 at some time (e.g., 40_10 minutes) before the user's predicted sleep period. After receiving the callback request, the context daemon 40_102 may schedule a callback notification within 40_10 minutes before the predicted sleep period begins. When the scheduled time arrives (e.g., the current time equals the scheduled time), context daemon 40_102 may send a callback notification to the operating system process. When the operating system process receives the callback notification, the operating system process may prompt the user to enter a user password so that the operating system process may perform maintenance tasks while the user is asleep. For example, an operating system process may receive a password from a user and store the password for use during performance of system maintenance tasks. Once the system maintenance task is completed and the password is no longer needed, the operating system process may delete the user password from the computing device.

To initiate maintenance tasks while the user is asleep, the operating system process may request callback notifications at some time (e.g., 40_30 minutes) after the start of the predicted sleep period. After receiving the callback request, the context daemon 40_102 may schedule callback notifications within 45 minutes after the predicted sleep period begins. When the scheduled time arrives (e.g., the current time equals the scheduled time), the context daemon 40_102 may verify that the user is not using and/or will not use the computing device before sending the callback notification to the operating system process.

In some implementations, context daemon 40_102 may verify that the user is not using the computing device by determining whether the computing device is servicing the user-initiated activity. For example, even if the computing device is locked (e.g., indicating that the user may be sleeping), the computing device may perform navigation-related activities for the user's navigation requests. Thus, when context daemon 40_102 determines to turn on a navigation component (e.g., global navigation satellite system receiver) of the computing device, context daemon 40_102 may determine that the user is using the computing device and cancel or delay sending callback notifications to operating system processes within the predicted sleep period. Similarly, when context daemon 40_102 determines that the computing device is providing personal hotspot services, synchronizes data with another user device in response to a user request (e.g., as opposed to automatic background synchronization), or provides some other user-initiated service when scheduling callback notifications, context daemon 40_102 may cancel or delay sending callback notifications to operating system processes because the user is still using the computing device even though the device is locked.

In some implementations, context daemon 40_102 may verify that the user is not using the computing device by determining whether the computing device is about to initiate an activity visible to the user. For example, various processes running on the computing device may notify or get the attention of the user. A communication application (e.g., instant message, text message, email, phone, etc.) may alert the user of a received message. For example, the communication application may be configured to remind the user to read or respond to a received message ten minutes after the message is received. The clock application may include an alarm clock function configured to notify (e.g., wake up) the user at some future time. The calendar application program may be configured to remind the user of a scheduled calendar event in the future. If the user is scheduled to attend the meeting, the navigation application may present a departure time reminder to the user based on an amount of travel time of the user from the user's current location to the meeting location. The exercise application may be configured to remind the user to stand up, walk around, go to run, or do other types of exercise. Each of these notifications, reminders, alerts, and the like are directed to the user and will prompt or cause the user to interact with the computing device. Context daemon 40_102 may determine whether one of these user-visible events will occur within a threshold period of time (e.g., one minute, ten minutes, time required to complete a system maintenance task, etc.), and delay or cancel sending callback notifications to operating system processes because the user is about to begin using the computing device.

Procedure

FIG. 40_18 is a flow diagram of an exemplary process 40_1800 for notifying a change in context on a client computing device. For example, a computing device corresponding to the system 40_100 described above may execute the process 40_ 1800.

At step 40_1802, the computing device may receive a context callback request. For example, the context daemon 40_102 may receive a callback request from the requesting client 40_126, as described above. The callback request may include an identifier for the requesting client 40_126 and a predicate that defines the context (e.g., device state) condition under which the context daemon 40_102 should send a callback notification to the requesting client 40_ 126. In some implementations, upon receiving a callback request, the context daemon 40_102 may generate a callback identifier that may be used by the context daemon 40_102 and/or the requesting client 40_126 to identify the callback request. For example, the context daemon 40_102 may return a callback identifier to the requesting client 40_126 in response to receiving a callback request from the requesting client 40_ 126. For example, the context daemon 40_102 may store callback requests in the callback registry 40_114 and/or the predicate database 40_ 116.

At step 40_1804, the computing device may initialize a context monitor to service the callback request. For example, the context daemon 40_102 may load the monitor group(s) 40_106 corresponding to the context items specified in the callback request predicates, as described above.

At step 40_1806, the computing device may receive current context information from the context monitor bank 40_106 (context monitor 40_ 108). For example, each context monitor 40_108 may interface with various system components to obtain the state of the system components. The context monitor 40_108 may then report the status to the context daemon 40_ 102. Alternatively, the context daemon 40_102 may receive status information from the reporting client 40_ 124. The context daemon 40_102 may generate the current context 40_112 based on the received state information, as described above.

At step 40_1808, the computing device may determine that the current context matches the requested context. For example, the context daemon 40_102 may compare the context request predicate with the current context to determine that the current context satisfies the condition specified in the predicate.

At step 40_1810, the computing device may send a callback notification to the requesting client 40_ 126. For example, in response to determining that the current context matches the requested context, the context daemon 40_102 may send a callback notification to the requesting client 40_126 identifying the callback request. The requesting client 40_126 may use the callback request identifier to determine which callback predicate triggers the callback (e.g., to determine the current operating context of the computing device). The requesting client 40_126 may then perform an action appropriate for the current context.

FIG. 40_19 is a flow diagram of an exemplary process 40_1900 for restarting a context daemon to service callback requests. For example, a process running on a computing device may be terminated by a process manager service of an operating system when the process manager determines that the process has been idle for a period of time. When the process manager (or some other process) terminates context daemon 40_102, the computing device may execute process 40_1900 to restart context daemon 40_102 so that context daemon 40_102 may collect context information and send a dispatch notification back to requesting client 40_ 126. For example, process 40_1900 may correspond to the context daemon restart mechanism described with reference to fig. 7-13.

At step 40_1902, the computing device can initiate a communication session between the context daemon 40_102 and the requesting client 40_ 126. In some implementations, the requesting client 40_126 can initiate a communication session with the context daemon 40_102 by sending a callback request to the context daemon 40_102 as described above. The callback request may include a client identifier and a callback predicate, as described above. In some implementations, the context daemon 40_102 can only communicate with the requesting client 40_126 using a communication session initiated by the requesting client 40_126 (e.g., sending callback notifications). In some implementations, the context daemon 40_102 may store the callback request in a callback database (e.g., predicate database 40_ 116).

At step 40_1904, the computing device may determine that context daemon 40_102 is inactive. For example, when the context daemon 40_102 does not receive any callback requests or context information updates within a period of time, the process manager may determine that the context daemon 40_102 is idle or inactive.

At step 40_1906, the computing device may close context daemon 40_ 102. For example, based on determining that context daemon 40_102 is inactive, the process manager may shut down or terminate context daemon 40_102 to conserve system resources (e.g., battery power, memory, etc.). Upon closing the context daemon 40_102, the communication session between the requesting client 40_126 and the context daemon 40_102 will also be terminated.

At step 40_1908, the computing device may detect an event associated with context daemon 40_ 102. For example, the event may be a context client (e.g., requesting client 40_126, reporting client 40_124, etc.) sending a message to context daemon 40_ 102. For example, the message may be a callback request from the requesting client 40_ 126. The message may be a context information update received from the reporting client 40_ 124. In some implementations, the event can be a device state update received by launch daemon 40_702, where context daemon 40_102 has registered an interest.

At step 40_1910, the computing device may restart context daemon 40_ 102. For example, when a context client sends a message to the terminated context daemon 40_102, the computing device may restart the context daemon 40_102 so that the context daemon 40_102 may receive and process the message. When boot daemon 40_702 receives a device state update corresponding to the request received from context daemon 40_102, boot daemon 40_702 may restart context daemon 40_ 102.

At step 40_1912, the computing device may restore the registered callback requests to the callback daemon 40_ 102. For example, upon restart, the callback daemon 40_102 may recover callback requests received before the callback daemon 40_102 was terminated. For example, the callback daemon 40_102 may recover previously received callbacks from the callback database.

At step 40_1914, the computing device may initialize the event monitor needed to service the resumed callback request. For example, the context daemon 40_102 may load the event monitor group 40_106 needed to collect the context information needed to service callback requests.

At step 40_1916, the computing device may reestablish the communication session between the context daemon 40_102 and the requesting client 40_ 126. For example, once the context daemon 40_102 is running again, the requesting client 40_126 may send a message (e.g., a callback request) to the context daemon 40_102 to reestablish the communication session. Context daemon 40_102 may send callback notifications to the client using the re-established communication session and according to predicates specified in the callback request.

FIG. 40_20 is a flow diagram of an exemplary process 40_2000 for restarting a callback client to receive callback notifications. For example, a process running on a computing device may be terminated by a process manager service of an operating system when the process manager determines that the process has been idle for a period of time. When the process manager (or some other process) terminates the requesting client 40_126, the computing device may execute the process 40_1900 to restart the requesting client 40_126 so that the requesting client 40_126 may receive callback notifications from the context daemon 40_ 102. For example, process 40_1900 may correspond to the requesting client restart mechanism described with reference to fig. 7-13.

At step 40_2002, the computing device may initiate a communication session between the context daemon 40_102 and the requesting client 40_ 126. In some implementations, the requesting client 40_126 can initiate a communication session with the context daemon 40_102 by sending a callback request to the context daemon 40_102 as described above. The callback request may include a client identifier and a callback predicate, as described above. In some implementations, the context daemon 40_102 can only communicate with the requesting client 40_126 using a communication session initiated by the requesting client 40_126 (e.g., sending callback notifications). In some implementations, the context daemon 40_102 may store the callback request in a callback database (e.g., predicate database 40_ 116).

At step 40_2004, the computing device may determine that the requesting client 40_126 is inactive. For example, when the requesting client 40_126 is not performing significant processing within the computing device (e.g., the requesting client 40_ 126's CPU usage is below a threshold level), the processing manager may determine that the requesting client 40_126 is idle or inactive.

At step 40_2006, the computing device may shut down the requesting client 40_ 126. For example, based on determining that the requesting client 40_126 is inactive, the process manager may shut down or terminate the requesting client 40_126 to conserve system resources (e.g., battery power, memory, etc.). Upon closing the requesting client 40_126, the communication session between the requesting client 40_126 and the context daemon 40_102 will also be terminated. Thus, the context daemon 40_102 will not have a communication channel to deliver callback notifications to the requesting client 40_ 126.

At step 40_2008, the computing device may detect an event associated with the requesting client 40_ 126. For example, the context daemon 40_102 may detect a current context that matches a context callback predicate (e.g., corresponding to a condition specified by the predicate). Launch daemon 40_702 may detect a device state corresponding to the device state request received from context daemon 40_102 and associated with the client identifier of context client 40_ 126.

At step 40_2010, the computing device may restart the requesting client 40_ 126. For example, when context daemon 40_102 detects that the current context matches a context callback predicate, context daemon 40_102 may attempt to send a callback notification to requesting client 40_ 126. However, since the communication session between the context daemon 40_102 and the requesting client 40_126 is terminated, the context daemon 40_102 cannot send callback notifications to the requesting client 40_ 126. Thus, upon detecting that the communication channel with the requesting client 40_126 has been terminated, the context daemon 40_102 may send the client identifier received from the requesting client 40_126 to the launch daemon 40_702 that is requested to restart the requesting client 40_ 126. In some implementations, upon requesting the launch daemon 40_702 to restart the requesting client 40_126, the context daemon 40_102 may delete all callback request data associated with the client identifier of the requesting client 40_126 (e.g., stored in the callback registry 40_114 and/or the predicate database 40_ 116). Upon receiving the client identifier, the launch daemon 40_702 may restart the requesting client 40_ 126. Alternatively, launch daemon 40_702 may restart requesting client 40_126 upon detecting a device state corresponding to the device state request received from context daemon 40_102 and associated with the client identifier of context client 40_ 126.

At step 40_2012, the computing device can reestablish the communication session between the context daemon 40_102 and the requesting client 40_ 126. For example, upon reboot, the requesting client 40_126 may send a new callback request to the context daemon 40_102 to start a new communication session.

At step 40_2014, the computing device may receive a client callback request from the restarted requesting client 40_ 126. For example, the context daemon 40_102 may receive a callback request from the requesting client 40_126 specifying the same callback prediction corresponding to the current context, as described at step 40_ 2008. Upon receiving the callback request, context daemon 40_102 may determine that the callback request corresponds to the current context of the computing device.

At step 40_2016, the computing device may send a callback notification to the requesting client 40_ 126. For example, upon determining that the current context matches a callback request, the context daemon 40_102 may send a callback notification to the requesting client 40_126 using the re-established communication channel.

Fig. 40_21 is a flow diagram of an exemplary process 40_2100 for predicting future events based on historical context information. For example, the process 40_2100 may correspond to the event prediction mechanism described with reference to fig. 40_14 through 40_ 17.

At step 40_2102, the computing device may obtain historical context data for the context item. For example, the context daemon 40_102 (e.g., the usage history monitor 40_110) may generate a stream of historical events for each context item in the current context 40_112 that indicate changes in device context (e.g., device state) over time. For example, the history monitor 40_110 may generate a stream of historical events for a "locked" context item that indicates when the device is locked or unlocked, as described above.

At step 40_2104, the computing device may generate a historical context data vector of context items. For example, the context daemon 40_102 may analyze historical context data for context items in a 24 hour period within the previous 40_28 days. For example, context daemon 40_102 may generate 28 data vectors for each of the 40_28 previous 24 hour time periods. Each of the 28 data vectors may include 96 data entries corresponding to 96 15-minute slots in each 40_ 24-hour period (e.g., each vector may have a length of 96). The context daemon 40_102 may assign a probability value to each of the 96 15 minute time slots that corresponds to an observed value of the context item (e.g., device state) recorded during each of the 40_28 previous 24 hour time periods (e.g., the previous 40_28 days). For example, the context daemon 40_102 may assign a value (e.g., 0, 1, 0.45, etc.) to each of the 96 data slots in the 40_28 vectors that indicates a likelihood that the computing device will remain locked during each of the 96 15 minute slots in the previous 40_28 days.

At step 40_2106, the computing device may determine a short-term probability that a particular context value will be observed in each time slot. For example, the short-term probabilities may be calculated based on data collected over the previous days. For example, by calculating the average of 15 minute time slots within the first seven days, as described in the section "short term averaging" above, the context daemon 40_102 can calculate the short term Probability (PS) that the device will remain locked.

At step 40_2108, the computing device may determine a long-term probability that a particular context value will be observed in each time slot. For example, the long-term probability may be calculated based on data collected on the same day of each week (e.g., sunday, wednesday, etc.) within the previous few weeks. For example, the context daemon 40_102 may calculate the long term Probability (PL) that the device will remain locked by calculating the average of the 15 minute time slots over the previous four weeks as described above in the section "long term average".

At step 40_2110, the computing device may combine the short-term probability and the long-term probability to generate a combined probability. For example, context daemon 40_102 may combine the short-term Probability (PS) and the long-term Probability (PL) to generate a combined probability (P), as described above. In some implementations, the context daemon 40_102 may weight the long-term probabilities (or short-term probabilities) to adjust the impact of the long-term probabilities on the combined probabilities.

At step 40_2112, the computing device may generate a probability curve for the context term values. For example, the context daemon 40_102 may convert the slot wise probability values to probability curves as described above in the section "generate sleep curves".

At step 40_2114, the computing device may predict a future occurrence of a particular device context. For example, once a probability curve is generated, context daemon 40_102 may predict future occurrences of the same context item value based on the probability curve. For example, using the context embodiment of locking described above, context daemon 40_102 may predict that the device will remain locked for times of 11 pm and 7 am. Based on this locked context item prediction, context daemon 40_102 may infer that the user will fall asleep within the predicted time period.

FIG. 40_22 is a flow diagram of an exemplary process 40_2200 for servicing a sleep context callback request. For example, the process 40_2200 may correspond to at least the mechanism described in fig. 40_22 above. For example, the requesting client 40_126 (e.g., an application, utility, operating system tool, etc.) may send a callback request to the context daemon 40_102 specifying that the context daemon 40_102 should notify processes while the user sleeps. For example, the requesting client 40_126 may be configured to schedule maintenance activities while the user is sleeping, so that the user is not inconvenienced by these maintenance activities while using the computing device.

At step 40_2202, the computing device may receive a sleep context callback request. For example, the context daemon 40_102 may receive a callback request from the requesting client 40_126 specifying that the context daemon 40_102 should notify the requesting client 40_126 ten minutes after the user sleeps.

At step 40_2204, the computing device may initialize the sleep context monitor to service the callback request. For example, the sleep context monitor may be a monitor bank 40_106 that includes a context monitor 40_108 configured to monitor a lock state of the computing device. In some cases, the context monitor 40_108 may be configured to monitor the lock state of the computing device and the state of other components associated with the sleep context. For example, the context monitor 40_108 may monitor the status of navigation components, wireless network components (e.g., personal hotspots, bluetooth, etc.), device synchronization components, and/or device input/output components (e.g., headphone jack connectors, etc.).

At step 40_2206, the computing device may receive sleep context information from the context monitor. For example, the context monitor 40_108 may report the lock state of the computing device and/or the state of other monitored components to the context daemon 40_ 102.

At step 40_2208, the computing device may predict a future sleep period. For example, the context daemon 40_102 may predict future sleep periods, as described above with reference to fig. 40_ 21.

At step 40_2210, the computing device may schedule a sleep context callback. For example, if the predicted sleep period is from 11 pm to 7 am and the sleep context callback specifies that the requesting client 40_126 should be called back ten minutes after the start of the sleep period, the context daemon 40_102 may schedule a sleep callback at 11:10 pm.

At step 40_2212, the computing device may process the scheduled sleep context callback. For example, the context daemon 40_102 may detect when the current time equals the scheduled time of 11:10 am and determine whether to send a callback notification to the requesting client 40_ 126.

At step 40_2214, the computing device may determine whether the user is sleeping. For example, the context daemon 40_102 may analyze various context items (e.g., device states) to confirm that the user is sleeping. In some implementations, context daemon 40_102 may determine whether the current device lock state indicates that the device is locked. If the device is not locked, the context daemon 40_102 may cancel or delay sending the sleep callback notification to the requesting client 40_ 126.

In some implementations, context daemon 40_102 may determine whether the user is using the computing device passively. For example, a user may use (e.g., rely on) a device without providing user input or unlocking the device. If the user is passively using the computing device, the context daemon 40_102 may cancel or delay sending the sleep callback notification to the requesting client 40_ 126. For example, a user may be using a navigation feature of a computing device when the device is locked. Thus, the context daemon 40_102 may determine whether a navigation component (e.g., a GNSS system, Wi-Fi, and/or cellular data transceiver, etc.) is turned on. If the current context information indicates that these navigation components are powered, the context daemon 40_102 may determine that the user is passively using the computing device and is not asleep.

As another example of passive use, a user may be using personal hotspot functionality provided by a computing device when the computing device is locked. If the current context information indicates that the personal hotspot component is powered, the context daemon 40_102 may determine that the user is passively using the computing device and is not asleep.

As another example of passive use, the user may have initiated a synchronization operation with another device (e.g., a laptop, a tablet, a smart watch, etc.). The synchronization operation may be performed while the computing device is locked. If the current context information indicates that the computing device is performing a synchronization operation, the context daemon 40_102 may determine that the user is passively using the computing device and is not asleep.

At step 40_2216, the computing device may confirm that no imminent user activity needs to be scheduled to occur. For example, the user may actually fall asleep, but the computing device may have scheduled notifications visible to the user to occur soon that will wake the user and cause the user to use the computing device. For example, the computing device may have arranged an alert regarding an incoming communication (e.g., text message, instant message, email, phone call, etc.) that is arranged to occur soon after the sleep callback notification is arranged. The computing device may have scheduled an alarm clock alert that is scheduled to occur soon after the sleep callback notification is scheduled. The computing device may have scheduled a calendar reminder or alarm that is scheduled to occur soon after scheduling the sleep callback notification. The computing device may have scheduled departure time notifications that are scheduled to occur soon after the sleep callback notification is scheduled. The computing device may have scheduled an athletic reminder that is scheduled to occur soon after scheduling the sleep callback notification. If the context daemon 40_102 determines that user activity is scheduled to occur within a threshold period of time (e.g., 1 minute, 5 minutes, etc.), the context daemon 40_102 may cancel or delay sending the sleep callback notification to the requesting client 40_ 126.

In some implementations, the computing device may send a sleep callback notification to the client. For example, when the context daemon 40_102 confirms that the user is sleeping at step 40_2214 and confirms that no imminent user activity needs to be scheduled, the context daemon 40_102 may send a sleep callback notification to the requesting client 40_ 126.

Exemplary System architecture

FIG. 1A illustrates an exemplary device 100 having a system architecture for implementing the systems and processes described in this section.

Exemplary method for context monitoring/prediction

In some embodiments, a method of context monitoring comprises: receiving, by a context daemon executing on a computing device, values corresponding to one or more context items monitored by one or more context monitors; receiving, by a context daemon from a context client process, a request for context information corresponding to a first context item; determining, by the context daemon, that the context monitor is not currently monitoring the first context item; and initializing, by the context daemon, a new context monitor corresponding to the first context item. In some embodiments, initializing the new context monitor includes dynamically loading a new software package corresponding to the new context monitor into the context daemon. In some embodiments, initializing a new context monitor includes calling a new context monitor process separate from the context daemon, the new context monitor process corresponding to the new context monitor. In some embodiments, the one or more context item values describe a current state of one or more hardware components of the computing device. In some embodiments, the one or more context item values describe a current state of one or more software components of the computing device. In some embodiments, the method comprises: a stream of historical events corresponding to the first context item is generated by the new context monitor process. In some embodiments, the new context monitor process generates a historical event stream object that identifies the start time, duration, and context item values describing events in the historical event stream.

In some embodiments, a method of contextual notification includes: generating, by a context daemon executing on a computing device, information describing a current context of the computing device; receiving, by the context daemon from the context client process, a callback request specifying a predicate for sending notifications to the context client process, the predicate specifying context conditions for calling back the context client; detecting, by a context daemon, that a current context corresponds to a predicate; and in response to the detecting, sending, by the context daemon, a callback notification to the requesting client. In some embodiments, the context condition specifies a value of one or more context items that, when detected in the current context, cause the context daemon to send a callback notification to the context client. In some implementations, the callback request includes identifiers of the context client and the predicate; and further comprising: in some embodiments, the method comprises: generating, by a context daemon, a unique identifier of a callback request; and sending, by the context daemon, the unique identifier to the requesting client. In some embodiments, the method comprises: the association between the unique identifier, the context client identifier, and the predicate is stored by the context daemon. In some embodiments, the storing process includes storing the unique identifier, the context client identifier, and the predicate in memory associated with the context daemon. In some embodiments, the storing includes storing the unique identifier, the context client identifier, and the predicate in a predicate database. In some embodiments, the method comprises: receiving, by a context daemon, device state information describing current states of software and hardware components of a computing device; and generating, by the context daemon, current context information based on the received device state information.

In some embodiments, a method of context prediction comprises: obtaining, by a computing device, a stream of historical events corresponding to context items monitored by the computing device; calculating, by a computing device, a plurality of probabilities that a particular value of a context item will be observed within a respective time period in a historical event stream; generating, by the computing device, a probability curve based on the computed probability; and predicting, by the computing device, a future occurrence of the particular value of the context item based on the probability curve. In some embodiments, the method comprises: predicting a future occurrence of user activity based on the probability curve. In some embodiments, the predicted user activity corresponds to a predicted period of user sleep. In some embodiments, the method comprises: receiving, by the context daemon from the requesting client, a callback request requesting the context daemon to notify the callback client at a requested time before a predicted future occurrence of the user activity; scheduling, by the context daemon, delivery of the notification at the requested time; and sending, by the context daemon, a notification to the requesting client at a requested time prior to the predicted future occurrence of the user activity. In some embodiments, the method comprises: receiving, by the context daemon from the requesting client, a callback request requesting the context daemon to notify the callback client of a requested time during the predicted user sleep period; scheduling, by the context daemon, a delivery of the notification at a requested time during the predicted period of user sleep; determining that the requested time corresponds to a current time; determining, by the context daemon, whether the user is asleep at the current time; and sending, by the context daemon, a notification to the requesting client when the context daemon determines that the user is asleep at the current time. In some embodiments, determining whether the user is asleep at the current time comprises: it is determined whether a user-initiated operation is being performed by the computing device at a current time. In some embodiments, determining whether the user is asleep at the current time comprises: it is determined whether the user-visible operation is scheduled to be performed by the computing device within a threshold time period relative to a current time.

In some embodiments, a method of efficient context monitoring comprises: receiving, in a context daemon executing on a computing device, a first context callback request from a context client, the context callback request initiating a first communication session between the context client and the context daemon; receiving, by a context daemon, current context information; determining that the current context information corresponds to a context callback request; in response to the determination, detecting, by the context daemon, that the first communication session with the context client has terminated; and in response to the detection, sending, by the context daemon to the launch daemon, a restart message requesting the launch daemon to restart the context client. In some embodiments, the callback request includes a client identifier, and further comprising: the client identifier is sent by the context daemon to the launch daemon in the message. In some embodiments, the current context information includes one or more context items that describe a current state of the computing device. In some embodiments, the context callback request specifies conditions for notifying the context client based on current context information received by the context daemon. In some embodiments, the method comprises: the context client is terminated after the context daemon receives the first context callback request. In some embodiments, the method comprises: when a restart message is received, restarting the context client by the starting daemon; receiving, by the context daemon, a second context callback request from the context client after the client has been restarted; comparing the second context callback request with current context information; and in response to the comparison, notifying the context client that the current context information corresponds to the second context callback request. In some implementations, the second context callback requests establishment of a second communication session between the context client and the context daemon, and wherein the context daemon notifies the context client using the second communication session established by the context client.

Section 11: client, server, and Web aspects of search within applications

The content "client, server, and Web aspects of intra-application search" in this section describes client, server, and Web-based aspects of intra-application search, crowd-sourced application history search, and application view indexing and search according to some embodiments, and provides information that complements the disclosure provided herein. For example, some portions of this section describe methods of searching application states that supplement the disclosure provided herein, such as content related to creating and using deep links (as described below with reference to methods 600 and 800).

Concise summarization of client, server and Web aspects of intra-application search

Methods and apparatus for a device to perform a search using multiple application states are described. In an exemplary embodiment, a device receives a plurality of application states from a plurality of applications running on the device. The device also creates an index of a plurality of application states. In addition, the device receives a query for searching data stored on the device. In addition, the device searches for multiple application states using the index and the query. The device additionally determines a match of the query to one of the plurality of application states and returns a match of the matching application state.

In another embodiment, the device performs the query using multiple application states on the device. In this embodiment, the device executes the query using an index stored on the device. The device also receives a plurality of results that match the query. Further, the device determines a subset of the plurality of results corresponding to application states corresponding to the native application installed on the device. Further, the device presents, for each result in the subset of the plurality of results, the result and a representation of the native application corresponding to the result.

In another embodiment, the device selects an application state for multi-device searching. In this embodiment, the device detects on the device whether the application state has been selected as a result of a query for a device-level search on the device. The device also transmits application state to the server, where the application state is to be indexed with other application states from other devices.

In another embodiment, the device performs a search of the first device using the application state received from the second device. In this embodiment, a device receives a plurality of application states from a plurality of applications running on a plurality of devices. The device also creates an index of the plurality of application states. The device additionally receives a query for searching data stored on the device. In addition, the device searches multiple application states using the index and search query and returns a match of matching application states.

In another embodiment, the device performs a search. In this embodiment, the device transmits a query to the server and receives a plurality of results that match the query. The device also determines a subset of the plurality of results, the subset including an application state generated on another device, the application state corresponding to a native application state installed on the device. In addition, the device presents a link and a representation of the native application for each result in the subset of the plurality of results.

In another embodiment, the device indexes the application state in a search query index. In this embodiment, the application state of the application is received from another device coupled to the server. The device also generates a view of the application corresponding to the application state, where the view is a representation of a user interface of the application corresponding to the application state. In addition, the device indexes the view in a search query index.

In another embodiment, the device retrieves an application state having a view associated with the query result. In this embodiment, the device sends a query to the server. The device also receives a query result from the server, wherein the result includes a view of an application state of the application corresponding to the result, and the view is a representation of a user interface of the application corresponding to the application state. The device additionally presents the results with an indication of the view.

Other methods and apparatus are also described.

Detailed description of client, server, Web aspects of search within application

Methods and apparatus for a device to perform a search using multiple application states are described. In the following description, numerous specific details are set forth to provide a thorough explanation of embodiments of the present invention. It will be apparent, however, to one skilled in the art that embodiments of the invention may be practiced without these specific details. In other instances, well-known components, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

In the following description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. "coupled" is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, cooperate or interact with each other. "connect" is used to indicate the establishment of communication between two or more elements coupled to each other.

The processes depicted in the following figures are performed by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. Although the processes are described below in terms of certain sequential operations, it should be understood that certain operations described may be performed in a different order. Further, certain operations may also be performed in parallel rather than sequentially.

The terms "server," "client," and "device" are intended to refer to data processing systems in general, and not to specific form factors of the server, client, and/or device in particular.

Methods and apparatus for a device to perform a search using multiple application states are described. As described above, being able to search the web browser's history is useful because the user has digital routines to use the web browser. The digital routine may also include repeatedly accessing the same applications and using those applications to perform the same type of operation. As described above, smartphone users spend an average of 86% of their time using non-web browser applications. However, being able to search the history of a non-web browser application can present difficulties because the data of the application's usage history is difficult to access (if it occurs) and has a proprietary format. Therefore, the application history is difficult to search.

In one embodiment, a device generates and stores application states for an executing application. The device also indexes these application states so that a local search service running on the device can search the indexed application states to service the query results. In this embodiment, the application state is an instant snapshot of the application. The application state is similar to the web browser history. In one embodiment, the application state is for a non-web browser application. In one embodiment, the application state of an application may include the title of the state, the view, the date displayed in the view, associated metadata, and/or other state information. For example, in one embodiment, the application may be a review type application that displays reviews of different businesses and services for a geographic area. In this embodiment, each application state may be a set of comments and related information for a business or service (e.g., a name, address, contact information, open time, a description of the business or service, a set of comments submitted by a visitor or user of the service or business, and/or any other type of information associated with the business or service). Each application state may be displayed on one user interface page or across multiple user pages, where each page is content organized for display (in some embodiments, each page is a window of an application at a particular point in time). In one embodiment, each executing application derives one or more application states, where the device indexes the application states in an application state index.

By indexing the application state, the user may search the history of the application. This allows the user to search for and find previous application states. By looking up the application state, the user can use the application state to launch the corresponding application, thereby causing the application to indicate the location where the application was executing when the application derived the application state. The user may use the indexed application state to return the application to a previously used state via a common mechanism for multiple different applications. For example, in one embodiment, the application state may have a page of a transit application for a particular route of the transit system. In this embodiment, the user may navigate to a particular route, such as local bus route 7, in the transit application. By navigating to the particular route, the bus application exports the application state of the local bus route page to an application state index. With the indexed application state, a user may retrieve the application state via a query. For example, in one embodiment, the user may enter "bus route 7" in the query, and the application state of local bus route 7 will be displayed as the query result. Upon selection of the application state, the bus application will be loaded with the application state of the local bus route 7 and the page of the local bus route 7 in the bus application will be displayed to the user. Therefore, in this embodiment, the bus application is adopted to the same state as the previously executed state.

In another embodiment, the device may export application states to a remote application state indexer that may be used to support queries from devices that did not generate those application states. In this embodiment, the device outputs the application state that the user has occupied, where occupied application state means that the application state has been returned as a query result in response to a query by the user on the device, and the user has selected the application state. In addition, the device cleans up the application state by removing private information prior to exporting the application state. If the remote application state indexer has received the application state the necessary number of times, the remote application state indexer receives the application state and indexes the application state. In this embodiment, by indexing the application state after the necessary number of times, the application state has been crowd-sourced, such that many different users and/or devices have taken up the application state in a local search. In one embodiment, requiring a certain amount of application state occupancy increases the likelihood that the application state will be useful to other users. Once indexed, the remote search service may search the remote application state index to determine if there is an application state matching the query. For each match, the remote search service returns the matched application state to the client. On the client, the user may select an application state in which the corresponding application is launched and causes the application to indicate the location where the application was executing when the application exported the application state.

In another embodiment, the device generates an application state view of different application states. In this embodiment, the application state view is a representation of the user interface of the application corresponding to the application state. For example, in one embodiment, a comment-type application that may access the content of thousands or millions of comments for a business or service may have a view for each of thousands or millions of comments. These views can be used to preview application states and can also be used to preview applications in general. In one embodiment, these application state views may be used to preview the application state returned in a set of query results, or may generally be used to preview the application. In one embodiment, collecting multiple application state views of an application may be used to preview the application in the application store. For example, in one embodiment, a comment type application may have tens of application state views available for the application.

FIG. 41_1 is a block diagram of one embodiment of a system for indexing application state for use in a local device search index. In FIG. 41_1, a device 41_100 includes a plurality of applications 41_102 coupled to an application state indexer 41_ 104. In one embodiment, the device 41_100 is any type of device that can communicate network data with another device (e.g., a personal computer, laptop, server, mobile device (e.g., phone, smartphone, smartwatch, personal gaming device, etc.), other network element, etc.). In one embodiment, the appliance 41_100 may be a virtual machine, or may be an appliance hosting one or more virtual machines. In one embodiment, the device 41_100 additionally includes an application state search index 41_ 108. In one embodiment, each of the applications 41_102 is an executive that proceeds through a series of states while the application is running. For example, in one embodiment, the application 41_102 may be a word processing application, spreadsheet, contact, mail, phone, web browser, media player, commentary application, classified ads application, social network, efficiency, utility, game, real estate, photo, video, e-commerce, storefront, coupon, operating system, and/or any other type of application that may run on a device.

As described above, each of the applications 41_102 proceeds through a series of states while the application is being executed. In one embodiment, one of the application states is an instant snapshot of the application. In one embodiment, the application state of the application 41_102 may include a title, a user interface state, data displayed in the user interface, associated metadata, and/or other state information for the state. In another embodiment, the application state includes information describing how the state is presented in the search results. For example, in one embodiment, the application 41_102 may be a review type application that displays reviews of different businesses and services for a certain geographic area. In this embodiment, each application state may be a set of comments and related information for a business or service (e.g., a name, an address, contact information, an open time, a description of the business or service, a set of comments submitted by a visitor or user of the service or business, and/or any other type of information associated with the business or service). In one embodiment, in the case of a comment type application, the application state title is a title given for the application state, such as the name of the business or service. The user interface state of an application state may be a representation of the user interface of the application 41_102 corresponding to the application state. In this embodiment, the user interface state may include the following representation of the user interface: the location at which the user interface scrolls to the user interface, which component of the user interface is active, what mode the application may be in (e.g., the application 41_102 may have different modes for presenting information to the user). In one embodiment, the application may be small enough to include a title plus a uniform resource locator or application identifier and an application version number compatible with the state.

In one embodiment, each application state includes a title, searchable data and/or metadata, and application-specific opaque data. In this embodiment, the searchable data and/or metadata is data that: which is designated by the application 41_102 as data accessible by the search indexing service and/or the query search service, wherein the searchable data and/or metadata may be used to index the application state and may also be used to return the application state as a result of the query. For example, in one embodiment, the searchable data and/or metadata may be content in the application state (e.g., an application state title, content displayed in a user interface state, media data, location data, time data, or any other type of data or metadata that may be used for a search index). In one embodiment, the application-specific opaque data is application-specific data used to return the application to its previous state, and may or may not be searchable data. In this embodiment, the application is returned to the application state by loading the application state by the corresponding application 41_ 102. For example, in one embodiment, application-specific opaque data may include user interface state, user interface mode, and/or references to resources. The user interface mode may be a type of mode that the user is currently oriented to use. For example, in one embodiment, the word processor may be a draft layout view or a print layout view; and the image editing program may be in a library mode, an image editing mode, or a printing mode. In one embodiment, the referenced resource may be a field that is viewed or edited, a uniform resource locator of a resource that may be on a device or on another device (such as a server in a network). In one embodiment, data that is part of the application state may exist in pairs (keys, values) in the dictionary.

In one embodiment, one or more of the applications 41_102 each export one or more application states to the application state indexer 41_ 104. In this embodiment, the applications 41_102 may each export application states onto a fixed or variable schedule. For example, in one embodiment, the application 41_102 may derive the application state, derive the application state for each new user interface state, based on a fixed time, after one or more interactions with the user or some other metric. As another example, in another embodiment, the review application may navigate to a new review or review search. In this embodiment, by navigating to a new comment or comment search, a new view is generated and a new application state is created and exported to application state indexer 41_ 104. The application state indexer receives the application state and adds the application state to the application index 41_ 108. By adding the application state to the index, the new application state is available to the local search service to match queries received by the local search service. In another embodiment, the application state may be exported to a remote search application state index 41_108, which is described in FIGS. 41_6-11 below.

FIG. 41_2 is a block diagram of one embodiment of a system for searching an application state using an on-device application state search index. In FIG. 41_2, the device 41_200 includes an application 41_204 coupled to a local search service 41_ 208. The local search service 41_208 is also coupled to an application state search index 41_212 and a local search index 41_ 214. In one embodiment, the device 41_200 is a device as in fig. 41_ 1. In one embodiment, the application 41_204 includes a search input field 41_ 206. In this embodiment, the search input field is used to enter a query that may be used by a local search service to perform a search using the query. If a query is input to the search input 41_206, the application 41_204 sends the query to the local search service 41_ 208. The local search service 41_208 receives the query and determines a set of query results by searching the local search index 41_214 and/or the application state search index 41_212 to produce ranked results. In addition, the local search service 41_208 ranks the results and sends them back to the application 41_ 204.

In this embodiment, the search may include a search for objects stored on the device 41_ 200. For example, in one embodiment, the object may be a document, picture, music, application, email, calendar entry, and/or other object stored in a local search index. In one embodiment, the search is conducted based on an index maintained by the search module. In this embodiment, the index is an index of metadata stored in an object of the device. In an alternative embodiment, the local search service 41_208 may also apply the query to the application state search index 41_ 212. In this embodiment, the local search service 41_208 applies to queries to the application state search index 41_212 to determine if there are any application states that match the query. For example, in one embodiment, the local search service 41_208 applies a query to the searchable data for each application state stored in the index 41_ 212. In this embodiment, if there is a query match for one or more application states in the index 41_212, the local search service 41_208 returns a set of results including those one or more application states to the application 41_ 204. The application 41_204 displays the ranking result. If one of the displayed ranking results is an application state, the application may display an application icon, an application state title, and an application state summary. In one embodiment, upon selection of a displayed application state, an application corresponding to the application state is loaded with the application state. In this embodiment, by loading an application with an application state, the application is loaded into an execution state corresponding to the application state. For example, in one embodiment, if the application state is a particular coupon for a coupon application (e.g., rental car for weekends 5!), the coupon application is loaded with the application state and the application state displays the particular coupon as if the user had navigated to the coupon.

FIG. 41_3 is a block diagram of an embodiment of a user interface displaying application state query results among other query results. In fig. 41_3, three different possible user interfaces 41_300A-C are shown displaying application states on the device. In one embodiment, the user interface 41_300A includes a search input 41_302, an application status display 41_314A, other actions 41_310A, and an on-screen keyboard 41_ 312A. In one embodiment, the search input 41_302A is used to input a query for a user of the device. In this embodiment, a portion or the entire query may be entered and sent to a local search service to determine one or more sets of query results. In one embodiment, the results of the query are returned as one or more characters of the input search. Further, the application status display 41_314A includes an application icon 41_304A, an application status header 41_306A, and an application status summary 41_ 308A. In one embodiment, the application icon 41_304A is an icon representing an application corresponding to an application state. In this embodiment, the application icon 41_304A may be part of the application state returned from the query or retrieved based on information stored in the application state. In one embodiment, the application state header 41_306A is a header of the application state stored in the application state. In addition, the application state digest 41_308A is a digest of the application state. For example, in one embodiment, the application state summary 41_308A includes a description of the application state, such as a description of the contents of the application state. In this embodiment, the application state summary 41_308A may indicate to the user the content associated with the application state.

In one embodiment, the user interface 41_300A may include other actions 41_310A in addition to displaying the query results including the application state 41_ 314A. For example, in one embodiment, other actions 41_310A may include a link for searching a network using the query or searching an online encyclopedia using the query. The user interface 41_300A may also include an on-screen keyboard 41_312A used by the user to enter the search query. Alternatively, the query may be entered via other means (e.g., via a microphone coupled to the device, by another device coupled to the device, such as a smart watch coupled to the portable device). In one embodiment, the icon 41_304A may be an application state specific image thumbnail provided by an application. Further, the icon 41_304A may also be a video or a video preview. In another embodiment, the application state abstract may include "action" buttons, such as "phone call" icons, "play," direction, "" purchase.

In one embodiment, there are many different types of application states that can be displayed as query results. For example, in one embodiment, the application state may have a view of a transit application for a particular route of the transit system. In this embodiment, the user may navigate to a particular route, such as local bus route 7, in the transit application. By navigating to the particular route, the bus application exports the application state of the local bus route view to an application state index. With the indexed application state, a user may retrieve the application state via a query. For example, in one embodiment, the user may enter "bus route 7" in the query, and the application state of local bus route 7 will be displayed as the query result. Upon selection of the application state, the bus application will be loaded with the application state of the local bus route 7 and the user interface of the local bus route 7 in the bus application will be displayed to the user. Thus, in this embodiment, the bus application is adopted in the same state as seen previously.

As another example, in another embodiment, a user may use a food distribution application and the user only wants to reorder one of their previous orders. In this embodiment, a user may order Vietnam powder from a local restaurant using a local restaurant-specific application. In this order, the local restaurant application will export the application state corresponding to the Vietnam powder order. The application state will be indexed and accessed by the local search service. The user may later enter the name of the query "vietnam powder," "vietnam restaurant," or local restaurant, and the application state corresponding to the order may be one of the results. The application state may also be the highest hit of the result. Upon selection of this application state, the local restaurant application will be launched and display the previous Vietnam powder order so that the user can complete the Vietnam powder order.

In another example and embodiment, the user may maintain an image panel to plan their next field trip. In this embodiment, the user uses the image board application to link the pictures and comments about the next stroke. The user will use the image board application to go back to the particular image board to add a link from the device's clipboard. The image board application will export the application state of the image board for the field trip, which will be provided by the local search service. By searching for the application state, such as the name of a travel location, the user can quickly reach the field travel image panel within the image panel application by querying, rather than launching the image panel application and navigating to the particular image panel view.

In one embodiment, saving application state can be used to quickly access a particular view of a utility that may be difficult to navigate to. For example, in one embodiment, the device setup application may have many levels of multiple options. In this embodiment, the user may enter a battery usage page in the settings application to see which application is consuming the majority of the battery. The battery usage page may be four or more levels and thus difficult to access. By deriving the application state for the battery usage page of the set application, the user may be able to enter a query "battery usage," "battery," "batter," or other prefix of the term battery usage, resulting in the application state for the battery usage page of the set application being displayed as a result of the query. This would provide quick access to pages in the setup application that may be difficult to navigate to.

In another embodiment, the application state results for the query may be displayed along with other query results from other domains, such as the local search index described in FIG. 41_2 above. In FIG. 41_3, the user interface 41_300B displays the application state 41_314B along with other query results 41_ 310B. In this user interface 41_300B, the search input 41_302B is displayed along with the application state 41_314B, other query results 41_310B, and the on-screen keyboard 41_ 312B. In one embodiment, the search input 41_302B and the on-screen keyboard 41_312C are the same as described above for the user interface 41_ 300A. Further, the application state 41_314B includes an application icon 41_304B, an application state title 41_306B, and an application state summary 41_308B, which are the same as the application icon, application state title, and application state summary described above for the user interface 41_ 300A. In addition, the user interface 41_300B includes other query results 41_310B, which may be other query results from other domains or application states. For example, in one embodiment, other query results 41_310B for the query "battery" may include an object index in the local search index that matches the term "battery", other application states that match the term "battery", or other matching query results for a local or remote search index (e.g., a web search index) that matches the term "battery".

As described above, application states may also be saved for utility applications running on the device. For example, in one embodiment, these settings applications of the device for configuring the device may also export application states. The user interface 41_300C is an example of a query result including an application state of a setting application. In fig. 41_3, the user interface 41_300C, search input 41_302C are displayed with the application state 41_314B, other actions 41_310C, and the on-screen keyboard 41_ 312C. In one embodiment, the search input 41_302C and the on-screen keyboard 41_312C are the same as described above for the user interface 41_ 300A. Further, the application state 41_314C includes an application icon 41_304B, an application state header 41_306C that sets a component (e.g., battery usage) of the application, and an application state summary 41_308C that sets a component (e.g., battery usage) of the application.

As described above, to make the application state accessible to the local search service, the application state is added to an index accessible to the local search service. FIG. 4A is a flow diagram of one embodiment of a process 41_400 for indexing application state received from a plurality of different applications on a device. In one embodiment, process 41_400 is performed by an application state indexer (such as application state indexer 41_104) as described above in FIG. 41_ 1. In FIG. 4A, the process 41_400 begins with receiving a plurality of application states from a plurality of applications on a device at block 41_ 402. For example, in one embodiment, the process 41_400 may receive application states from various applications (such as word processing applications, spreadsheets, contacts, mail, phone, web browser, media player, commentary applications, classified advertising applications, social networks, efficiencies, utilities, games, real estate, photos, videos, e-commerce, storefronts, coupons, operating systems, and/or any other type of application that may run on a device). In one embodiment, the application may send application state to processes 41_400 simultaneously, serially, and/or in combinations thereof. At block 41_404, for each application state received by the process 41_400, the process 41_400 adds the application state to the application state index. In one embodiment, the process 41_400 adds application state (e.g., inverted index and completion attempt) to the application state index by adding an application state identifier, indexable text, application identifier, and/or insertion time to the search index data structure.

By adding multiple application states to the application state index, the indexed application states may be used for query searches of a local search service. FIG. 4B is a flow diagram of one embodiment of a process 450 for determining query results for a query using an application state index. In one embodiment, the process 450 is performed by a local search service to determine query results for a query using an application state index, such as the local search service 41_208 described in FIG. 41_2 above. In FIG. 4B, process 450 begins with receiving a query at block 452. In one embodiment, the query is a search string entered by the user in the application and sent to process 450. In one embodiment, the input may be entered through text, speaking a term, automatically generated, and/or some other means of entering a query prefix. For example, in one embodiment, a user may enter a query in a web browser or a file browser. At block 454, the process 450 determines a set of query results for the query using the local application state index. In one embodiment, the process 450 uses information in the query to determine a matching application state in the local application state index. At block 456, the process 450 ranks the set of query results. In one embodiment, the ranking is based on a score for each application state that matches the query. The process 450 returns the ranking of the set of query results at block 458. In one embodiment, the process 450 sends the ranked set of query results back to the application that sent the query to the process 450.

FIG. 41_5 is a flow diagram of one embodiment of a process 41_500 for receiving and presenting application state as part of a query result. In one embodiment, the process 41_500 is performed by an application to receive and present application state as part of a query result, such as the application 41_204 described in FIG. 41_2 above. In fig. 41_5, the process 41_500 begins at block 41_502 with sending a query to a local search service. In one embodiment, the query may be a search string entered by the user in the application and sent to the local search service. In this embodiment, the input may be entered by text, speaking words, automatically generated, received from a coupled device (e.g., a smart watch coupled to a portable device), and/or some other manner of entering a search string. In another embodiment, the search system may suggest a query, and the user may select one query from a plurality of choices. Alternatively, the query may be extracted from the context. For example, in one embodiment, a user is reading a text message and conducting a search, a query is extracted by the data detection system and automatically issued or suggested to the user. Further, a query may be issued by following a link from another application. At block 41_504, the process 41_500 receives a set of results, wherein the results include application state. In one embodiment, the set of results is ranked with the highest ranked result being the highest hit. At block 41_506, the process 41_500 presents the application state in the user interface. In this embodiment, the application state includes an application state title, a summary, and an indication of an icon corresponding to the application of the application state. In one embodiment, the process 41_500 displays the application state as described above in fig. 41_ 3. In response to the selected application, the process 41_500 launches the corresponding application using the selected application state. For example, in one embodiment, if the application state is a review for a local restaurant for a review type application, the process 41_500 launches a review type application having a local restaurant application state. In this embodiment, the comment type application will be launched such that the view presented to the user is one of the local restaurants in the comment type application. In one embodiment, if the application is not installed on the device, the process 41_500 may download the application from a remote source (such as an application store, web page, or other remote server). In this embodiment, the process 41_500 will install the application and launch the application using the selected application state.

As described above, multiple applications may export application state that is locally indexed on the device executing the applications. In one embodiment, these application states can further be exported to a remote application state indexer and used to support queries from devices that did not generate these application states. FIG. 41_6 is a block diagram of one embodiment of a system 41_618 that indexes application states for use in remote search indexing. In FIG. 41_6, the device 41_600 is coupled to a remote application state indexer 41_ 610. In one embodiment, each of the devices 41_600 includes a plurality of applications 41_602 executing on the device 41_600, and each of the applications 41_602 is coupled to an application state module 41_604 on the device. In this embodiment, each of the applications 41_602 exports one or more application states 41_612 to the application state module 41_ 604. In this embodiment, the application state module 41_604 indexes the received application state in the application state search index 41_608 as described in fig. 41_1 through fig. 41_5 above. In another embodiment, these application states may be sent to the remote application state indexer 41_ 610. By sending application states to the remote application state indexer 41_610, these application states can be made available to support queries from other devices not shown. Thus, in this embodiment, indexed application states from multiple applications running on multiple devices may be used for query results in response to queries sent by devices that do not generate these application states.

In one embodiment, each locally indexed application state may also be exported to a remote application state indexer 41_ 610. In this embodiment, thousands or millions of application states may be generated and sent to the remote application state indexer 41_ 610. However, as the many application states are exported and indexed, this may create an application state index that is too large and/or has many useless dummy entries. Further, some or all of the exported application state may include private information that is not intended to be included in the indexed application state 41_ 614.

In one embodiment, if the application states are already occupied on the device, the application state module 41_604 exports these application states to the remote application state indexer 41_ 610. In this embodiment, to occupy the application state, the application state module determines, in response to a query by a user on the device, whether the application state has been returned as a result of the query, and determines whether the user has selected the application state. In one embodiment, occupying the application state means that the user has sent the query to a local search service, the local search service has returned the application state in the form of a set of query results, and the user has selected or viewed the application state. In one embodiment, occupying the application state indicates to the application state module 41_604 that the particular application state may be more important than other application states generated by the device 41_ 600. For each occupied application state, the application state module 41_604 exports the application state to the remote application state indexer 41_ 610.

In another embodiment, the application state module 41_604 cleans up the application state by removing any possible private information that may be in the application state before exporting the occupied application state to the remote application state indexer 41_ 610. In one embodiment, the application state may include private information, such as a user name, private contact information, location, time of access, social security number, bank account number, and/or any other type of private information that may be in the application state. In one embodiment, the application that created the application state may mark certain information stored in the application state as private. In another embodiment, the device may add private information to the application state. Alternatively, the application status module 41_604 may know that certain information is private, whether or not the information is marked as private. In either of these embodiments, the application state module 41_604 will remove the private information.

In one embodiment, the remote application state indexer 41_610 receives application states from a plurality of devices 41_ 600. The remote application state indexer 41_610 may receive application state from several devices or up to thousands or millions of devices. Additionally, in one embodiment, the remote application state indexer 41_610 maintains two sets of application states. One set of application states is the indexed application state 41_ 614. These are a set of application states that have been indexed and made available to the search service. Another set of application states is the un-indexed application state 41_ 616. In one embodiment, the remote application state indexer 41_610 adds the application state to an index set of application states once the application state has been occupied by one or more devices the necessary number of times. For example, in one embodiment, if an application state is occupied 50 times, the application state is added to an index set of application states. In an alternative embodiment, an application state may be added to an index set of application states if the application state has been occupied as many times as possible. In one embodiment, the necessary number of times an application state will be occupied before being indexed into an application index may vary depending on the type of application state. For example, in one embodiment, an application state that includes geolocation information (e.g., an application state of a region coupon) may need to be occupied a fewer number of times than an application state that does not have geolocation information.

In this embodiment, indexing the application state after it has been occupied the necessary number of times increases the likelihood that the application state will be useful to other users. For example, in one embodiment, many different users on different devices use a local bus application and generate application states for the local bus route 7. In this embodiment, this is a common route, so the application state is occupied by a user accessing the application state through a local search service. The application state is indexed by the remote application state indexer 41_610 and is available for remote search services.

In one embodiment, remote application indexer 41_610 determines whether the application state has been previously sent by computing a hash for the application state. If the hash matches other hashes stored by remote application indexer 41_610, then remote application indexer 41_610 increments the number of times the remote application indexer has received the application state. If the necessary number of times has been received, the remote application indexer 41_610 indexes the application state. Indexing the application state is further described in FIG. 41_8 below.

Fig. 41_7 is a block diagram of one embodiment of a system for searching application states using a remote application state search index. In fig. 41_7, the device 41_702 is coupled to a remote application status search service 41_ 714. In one embodiment, the device 41_702 includes an application 41_704 coupled to a local search service 41_ 708. The local search service 41_708 is further coupled to an application state search index 41_ 716. In one embodiment, the device 41_702, application 41_704, local search service 41_708, and application state search index 41_716 are devices, applications, local search services, and application state search indexes as described in fig. 41_2 above. In another embodiment, the local search service 41_708 may forward the query to the remote application search service 41_714, where the remote application search service 41_714 determines whether there is a set of results for the query. The remote application search service 41_714 returns the set of results to the local search service 41_708, which in turn returns the set of results to the application 41_ 704. Alternatively, the application 41_704 may send the query to a remote application search service 41_714, which in turn sends the set of query results back to the application 41_ 704.

As described above, the remote application state search service 41_714 receives a query from the device 41_702 and returns a set of query results for the query to the device 41_ 702. In one embodiment, the remote application state search service 41_714 receives a query searching the index application state 41_712 to obtain application states that match the received query, scores each of the matching application states, ranks the set of results, and returns the ranked results to the application. The application 41_704 displays a result including the application state. In one embodiment, the application 41_704 displays the icon, application state title, and application state summary of the application as described above in FIG. 41_ 3. Upon selection of the displayed application state, the application is loaded with the application state. In one embodiment, if the application state is stored locally on the device, the application is in the same state as it would if the user had taken the application state.

Figure 41_8 is a flow diagram of one embodiment of a process 41_800 of adding application state to an application state index. In one embodiment, the process 41_800 is performed by an application state deriver module to add an application state to an application state index, such as the application state deriver module 41_606 described above in fig. 41_ 6. In fig. 41_8, the process 41_800 begins by receiving the application state at block 41_ 802. In one embodiment, the application state is received by the process 41_800 from one or more applications running on the device that is performing the process 41_ 800. At block 41_802, the process 41_800 determines whether the application state is occupied. In one embodiment, the application state is occupied if the application state is returned in a set of results that match the query and the user has selected the application state to load into the application. If the application state is not occupied, execution proceeds to block 41_802 above. If the application state is already occupied, the process 41_800 clears the application state of block 41_ 806. In one embodiment, the process 41_800 cleans up the application state by removing private information associated with and/or stored in the application state described in FIG. 41_6 above. At block 41_808, the process 41_800 sends the cleaned application state to the remote application state indexing service. In one embodiment, the remote application state index service may add the application state to the application state index.

Figure 41_9 is a flow diagram of one embodiment of a process 41_900 for indexing application states by an application state indexing service. In one embodiment, the process 41_900 is performed by a remote application state indexer, such as the remote application state indexer 41_610 described above in FIG. 41_6, to index the application state. In fig. 41_6, the process 41_900 begins with receiving an indication of an application state from a device at block 41_ 902. In one embodiment, the application state indication is a hash of the application state. By receiving a scatter of the application state rather than the complete application state, the process 41_900 does not receive the application state until the application state is common to multiple clients or has been occupied as many times as necessary. At block 41_904, the process 41_900 increments the number of occurrences of the application state. In one embodiment, the process 41_900 maintains a count of the application state hashes. If this is the first time the process 41_900 has received the indication, the count is 1. The process 41_900 determines whether the number of occurrences at block 41_906 is greater than a threshold. In one embodiment, the application state having been received by the process 41_900 the necessary number of times means that the application state has been occupied multiple times and is a candidate to be indexed and available to service queries. For example, in one embodiment, if the application state has been occupied 50 times, the application state of a particular coupon for the coupon application may be made available to the application state index. If the number of occurrences is greater than the threshold, the process 41_900 sends a request for full application state to the device that sent the last application state indication. The process 41_900 receives the application state at block 41_ 910. The process 41_900 indexes the application state at block 41_ 910. By indexing the application state, the process 41_900 makes this application state available as part of a set of results for the query.

In another embodiment, rather than requesting the full application state when the application state indication was last received, the process 41_900 begins building the application state gradually until the process 41_900 receives the last portion of the application state and indexes the application state. For example, in one embodiment, process 41_900 requires the last M clients to send 1/M of the application state to process 900. In this embodiment, the application states are the same application state since they generate the same application state hash. This means that these M parts of the application state can be connected through the process 41_ 900. This embodiment may provide additional privacy in that portions of the application state are transmitted each time the process 41_900 is allowed to build the complete application state.

FIG. 41_10 is a flow diagram of one embodiment of a process 41_1000 for performing a query search using an application state index. In one embodiment, the process 41_1000 is performed by a remote application state search service to perform a query search using an application state index, such as the remote application state search service 41_714 described above in fig. 41_ 7. In fig. 41_10, the process 41_1000 begins by receiving a query from a client at block 41_ 1002. In one embodiment, the query is a search string entered by the user in the application and sent to the remote search service as described above. At block 41_1004, the process 41_1000 searches the application state index using the query. In one embodiment, the process 41_1000 determines whether there are any application states that match the query. The process 41_1000 determines a set of results for the query at block 41_ 1006. In one embodiment, the set of results includes one or more application states that match some or all of the text in the query. At block 41_1008, the process 41_1000 ranks the set of results. In one embodiment, the process 41_1000 ranks the set of results by determining a score for each of the results and ranking the results using the scores. At block 41_1010, the process 41_1000 combines a set of results with results from other search domains. In one embodiment, if the search is a federated search, where different indexes are searched using the same query, the process 41_1000 combines results from other search domains with the set of results determined using the application state index. For example, in one embodiment, the query may be used to search an application state index, a general Web search index, and/or a different index (e.g., a media index, an application storage index, a map index, an online encyclopedia index, and/or another type of index). At block 41_1012, the process 41_1000 returns a set of ranked results to the client along with other results generated in block 41_ 1010.

FIG. 41_11 is a flow diagram of one embodiment of a process 41_1100 for receiving and presenting application state as part of a query result. In one embodiment, the process 41_1100 is performed by an application program to receive and present application program state as part of a query result, such as the application program 41_704 described above in FIG. 41_ 7. In fig. 41_11, the process 41_1100 begins at block 41_1102 with sending a query to a remote search service. In one embodiment, the query may be a search string entered by the user in the application and sent to the remote search service. In this embodiment, the input may be entered by text, speaking a word, automatically generated, received from a coupled device (e.g., a smart watch coupled to a portable device), and/or some other manner of entering a search string. At block 41_1104, the process 41_1100 receives a set of results, wherein the results include application state. In this embodiment, the application state is a cleaned application state that has been occupied by the user as many times as necessary, as described above in fig. 41_ 6. In one embodiment, the set of results is ranked with the highest ranked result being the highest hit. At block 41_1106, the process 41_1100 presents the application state in the user interface. In this embodiment, the application state includes an application state title, a summary, and an indication of an icon corresponding to the application of the application state. In one embodiment, the process 41_1100 displays the application state as described above in FIG. 41_ 2. In response to the selected application, the process 41_1100 launches the corresponding application using the selected application state. For example, in one embodiment, if the application state is a comment for a local restaurant for a comment type application, the process 41_1100 launches a comment type application having a local restaurant application state. In this embodiment, the comment type application will be launched such that the view presented to the user is one of the local restaurants in the comment type application. In one embodiment, the process 41_1100 may download an application from a remote source (such as an application store, web page, or other remote server) if the application is not installed on the device. In this embodiment, the process 41_1100 will install the application and launch the application using the selected application state.

FIG. 41_12 is a block diagram of one embodiment of a system 41_1200 to index application state views for use in remote search indexing. In fig. 41_12, the device 41_1202 is coupled to an application state memory 41_1206 and an application state index 41_ 1208. In one embodiment, the device 41_1202 retrieves the application states stored in the application state memory 41_1206 and, for each of the application states, the device 41_1202 generates a view for each of these application states. In one embodiment, a view of an application state is a representation of a user interface of an application corresponding to the application state. For example, in one embodiment, the user interface may include text, images, video, audio, animation, graphics, and/or other types of user interface components. In this embodiment, the corresponding view is a two-dimensional representation of the user interface. In one embodiment, an application may have many different views generated based on different application states associated with the application. For example, in one embodiment, a comment-type application that may access the content of thousands or millions of comments for a business or service may have a view for each of thousands or millions of comments. The view is further described in fig. 41_13 below.

In one embodiment, the application state stored in the application state memory 41_1206 may be the application state that the user has occupied the necessary number of times, as explained above in fig. 41_ 6. Alternatively, the application state memory 41_1206 may also include non-indexed application states. In addition, the application state index 41_1208 includes indexed application states with views generated for those application states. In this embodiment, these views may be returned with the application state as part of a set of results for the query. The search engine 41_1210 includes an application state search service 41_1212 that receives queries from the device 41_ 1214. The application state search service 41_1212 receives queries from the device 41_1214, uses the queries to search the application state index, determines matching application states for the queries with associated views, scores the matching application states, ranks the matching application states, and returns the matching application states as a set of results to the device that sent the original query.

As described above, an application state may have an associated view. FIG. 41_13 is a block diagram of one embodiment of an application state view 41_ 1302. In fig. 41_13, the device 41_1300 has an application program executed in a specific application program state. The application in this application state displays the application state user interface 41_ 1302. The application state user interface 41_1302 may include various components, such as icons, text, images, video, audio, animation, graphics, and/or other types of user interface components. For example, in one embodiment, the application state user interface 41_1302 includes an image 41_1304, text 41_1306, and an icon 41_ 1308. In one embodiment, a view may be generated from the application state user interface 41_ 1302. In this environment, the view is a representation of the application state user interface 41_1302, which may be saved and indexed along with the application state in the application state index. For example, in one embodiment, the view of the application state user interface 41_1302 is a two-dimensional image, such as a GIF, JPEG, PNG, and/or another type of two-dimensional image. In this embodiment, the two-dimensional image of the view may be stored with the application state in an application state index.

FIG. 41_14 is a flow diagram of one embodiment of a process 41_1400 for generating an application state view using application states. In one embodiment, process 41_1400 is performed by an application state view generator and indexer to generate an application state view using application state, such as the application state view generator and indexer 41_1204 described above in FIG. 41_ 12. In fig. 41_14, process 41_1400 begins with receiving the application state at block 41_ 1402. In one embodiment, the process 41_1400 receives the application state from an application state memory, such as the application state memory 41_1206 as described above in fig. 41_ 12. At block 41_1404, the process 41_1400 generates an application state view using the application state. In one embodiment, the process 41_1400 generates an application state view by simulating an application using the application state. In this embodiment, the application is executed in the simulator using the application state. The process 41_1400 may capture the application user interface for the application state using the private framework of the simulator. Alternatively, the process 41_1400 may load the application onto the virtual platform or device itself and use a mechanism to generate a view of the application state. At block 41_1406, the process 41_1400 adds the application state view to the application state index of the corresponding application state.

By generating these application state views, the process 41_1400 can generate multiple views for one or more applications. These views may be used to preview application state and may also be used to preview applications in general. In one embodiment, these application state views may be used to preview the application state returned in a set of query results, or may be used to preview applications generally. The use of the view with query is further described in FIG. 41_15 below. In one embodiment, collecting multiple application state views of an application may be used to preview the application. For example, in one embodiment, a comment type application may have tens of application state views available for the application. For those interested in this comment-type application (e.g., those viewing the application in an application store), these application state views may be made available so that the user can preview the application before purchasing and/or downloading the application. In this embodiment, the user may browse tens of views forward and backward to understand the appearance of the application.

FIG. 41_15 is a flow diagram of one embodiment of a process 41_1500 for receiving and presenting application state (including application state views) as part of a query result. In one embodiment, the process 41_1500 is performed by a device to receive and present an application state view as part of a query result, such as the device 41_1214 described in FIG. 41_12 above. In fig. 41_15, the process 41_1500 begins at block 41_1502 with sending a query to a remote search service. In one embodiment, the query may be a search string entered by the user in the application and sent to the remote search service. In this embodiment, the input may be entered through text, speaking words, automatically generated, received from a coupled device (e.g., a smart watch coupled to a portable device), and/or some other manner of entering a search string. At block 41_1504, the process 41_1500 receives a set of results, wherein the results include application state. In this embodiment, the application state is the cleaned application state that has been occupied by the user as many times as necessary, as described above in fig. 41_ 6. In one embodiment, the set of results is ranked with the highest ranked result being the highest hit. At block 41_1506, the process 41_1500 presents the application state in the user interface. In this embodiment, the application state includes an application state title, a summary, an indication of an icon corresponding to the application of the application state, and an indication of availability of a corresponding application view. In response to the selected application state view, the process 41_1500 retrieves and presents the application state view. In one embodiment, by displaying an application state view, a user may obtain a preview of an application executing in the application state. This may help the user decide whether to select the application state. In another embodiment, even if an application is installed on the device, the preview view may be faster than launching the application with the application state. For example, in one embodiment, if the application status view is a review of a local restaurant for a review type application, the process 41_1500 retrieves and displays the application status view.

Exemplary machine-readable media, methods, and systems for client, server, Web aspects of in-application searching

In one aspect, methods and apparatus are provided for a device that performs a search using multiple application states. In an exemplary embodiment, a device receives a plurality of application states from a plurality of applications running on the device. The device also creates an index of a plurality of application states. In addition, the device receives a query for searching data stored on the device. In addition, the device searches for multiple application states using the index and the query. The device additionally determines a match of the query to one of the plurality of application states and returns a match of the matching application state.

In some embodiments, there is provided a machine-readable medium having executable instructions to cause one or more processing units to perform a method to perform a search using a plurality of application states, the method comprising: receiving a plurality of application states from a plurality of applications running on a device; creating an index of a plurality of application states, the index stored on the device; receiving a query for searching data stored on a device; searching a plurality of application states using the index and query; a match of the query to one of the plurality of application states is determined and the match of the matching application state is returned. In some embodiments, each of the plurality of application states includes data representing an instant snapshot of an application for that application state. In some embodiments, there are multiple application states for one of the multiple applications. In some embodiments, in addition to searching for multiple application states, the query is used to search for files stored on the device. In some embodiments, an application state of the plurality of application states includes user interface information representing a view location of user interface data used by a corresponding one of the plurality of applications.

In some embodiments, there is provided a machine-readable medium having executable instructions to cause one or more processing units to perform a method to execute a query using a plurality of application states, the method comprising: performing a query on the device using an index stored on the device; receiving a plurality of results matching the query; determining a subset of the plurality of results corresponding to an application state corresponding to a native application installed on a device; and presenting, for each result in the subset of the plurality of results, the result and a representation of the native application corresponding to the result.

In another aspect, methods and apparatus are provided for selecting a device for application state for multi-device searching. In this embodiment, the device detects on the device whether the application state has been selected as a query result for a device-level search on the device. The device also transmits the application state to the server, wherein the application state is to be indexed with other application states from other devices. In some embodiments, a machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method to select an application state for multi-device searching, the method comprising: detecting, on a device, whether the application state has been selected as a query result for a device-level search on the device; and transmitting the application state to a server, wherein the application state is indexed with other application states from other devices.

In some embodiments, a machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method to perform a search for a first device using application state received from a second device, the method comprising: receiving a plurality of application states from a plurality of applications running on a plurality of devices; creating an index of a plurality of application states; receiving a query for searching data stored on a device; searching a plurality of application states using the index and search query; a match of the search query to one of the plurality of application states is determined, and the match of the matching application state is returned. In some embodiments, creating the index comprises: if the number of times an application state is received satisfies a threshold, the application state of the plurality of application states is added to the index. In some embodiments, a machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method for performing a search, the method comprising: transmitting a query from a device to a server; receiving a plurality of results that match the query; determining that each result of the plurality of results comprises a subset of application state generated on another device corresponding to a native application installed on the device; and presenting a link and a representation of the native application for each result in the subset of the plurality of results.

In yet another aspect, methods and apparatus are provided for a device that indexes application states in a search query index. In this embodiment, the application state of the application is received from another device coupled to the server. The device also generates a view of the application corresponding to the application state, where the view is a representation of a user interface of the application corresponding to the application state. In addition, the device indexes the view in a search query index. In some embodiments, a machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method to index an application state in a search query index, the method comprising: receiving, using a server, an application state of an application from a device coupled to the server; generating a view of the application corresponding to the application state, wherein the view is a representation of a user interface of the application corresponding to the application state; and index the view in the search query index. In some embodiments, further comprising: the view is linked to an index entry for the application state. In some embodiments, the index entry is part of an index, where the index includes a plurality of index entries for application states for a plurality of applications from a plurality of devices coupled to the server. In some embodiments, the application state includes data representing an instant snapshot of the application for the application state. In some embodiments, the index entry includes information selected from the group consisting of a title, searchable data, and application-specific opaque data. In some embodiments, the view is an image. In some embodiments, a view is an image having a plurality of frames. In some embodiments, generating comprises: executing an application on the virtual device; and capturing a screen image of a user interface of the application corresponding to the application state. In some embodiments, generating comprises: simulating the application with the application state; and capturing a screen image of a user interface of the application corresponding to the application state.

In some embodiments, a machine-readable medium is provided having executable instructions to cause one or more processing units to perform a method to retrieve application state having a view associated with a query result, the method comprising: sending a query to a server; receiving a query result from a server, wherein the result includes a view of an application state of an application corresponding to the result, and the view is a representation of a user interface of the application corresponding to the application state; and present the results with the indications of the views. In some embodiments, further comprising: the view is presented in response to a gesture of the user.

Functional block diagram of an exemplary electronic device

According to some embodiments, fig. 42 illustrates a functional block diagram of an electronic device 4200 configured according to principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 42 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4200 is implemented as the portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 42, electronic device 4200 includes a display unit 4201 configured to display information, such as touch-sensitive display system 112 (also referred to as a touchscreen and a touchscreen display) of fig. 1A, a touch-sensitive surface unit 4203 configured to receive contacts, gestures, and other user inputs on the touchscreen display, such as display controller 156 and touch-sensitive display system 112 of fig. 1A, and a processing unit 4205 coupled to display unit 4201 and touch-sensitive surface unit 4203. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 42 shows the display unit 4201 and the touch-sensitive surface unit 4203 integrated with the electronic device 4200, however, in some embodiments, one or both of these units are in communication with the electronic device while the units remain physically separate from the electronic device. In some embodiments, the processing unit includes an execution unit (e.g., execution unit 4207 of fig. 42), a collection unit (e.g., collection unit 4209 of fig. 42), an acquisition unit (e.g., acquisition unit 4211 of fig. 42), an association unit (e.g., association unit 4213 of fig. 42), a providing unit (e.g., providing unit 4215 of fig. 42), a transmission unit (e.g., transmission unit 4217 of fig. 42), a reception unit (e.g., reception unit 4219 of fig. 42), a display unit (e.g., display unit 4221 of fig. 42), a detection unit (e.g., detection unit 4223 of fig. 42), an execution unit (e.g., execution unit 4225 of fig. 42), a determination unit (e.g., determination unit 4227 of fig. 42), and a monitoring unit (e.g., monitoring unit 4229 of fig. 42).

In some embodiments, the processing unit 4205 (or one or more components thereof, such as units 1007-1029) is configured to: executing an application on the electronic device (e.g., with execution unit 4207) in response to an instruction from a user of the electronic device; while executing the application, collecting (e.g., with the collection unit 4209) usage data that includes at least one or more actions performed by a user within the application; automatically acquiring (e.g., with acquisition unit 4211) at least one trigger condition based on the collected usage data without human intervention; associate (e.g., with association unit 4213) the at least one trigger condition with a particular action of the one or more actions performed by the user within the application; and upon determining that the at least one trigger condition has been satisfied, provide an indication to the user (e.g., with providing unit 4215) indicating that a particular action associated with the trigger condition is available. In some embodiments of the electronic device 4200, the processing unit (or one or more components thereof, such as the units 4207-4229) is further configured to perform the methods in any of a 2-a 22 described above in the "summary of the invention" section.

According to some embodiments, fig. 43 illustrates a functional block diagram of an electronic device 4300 configured according to principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 43 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4300 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 43, electronic device 4300 includes a display unit 4301 configured to display information (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)), a touch-sensitive surface unit 4303 configured to receive contacts, gestures, and other user inputs on the touchscreen display (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A), and a processing unit 4305 coupled with display unit 4301 and touch-sensitive surface unit 4303. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, FIG. 43 shows the display unit 4301 and the touch-sensitive surface unit 4303 integrated with the electronic device 4300, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. In some implementations, the processing unit includes a display unit (e.g., display unit 4309 of fig. 43), a detection unit (e.g., detection unit 4307 of fig. 43), a retrieval unit (e.g., retrieval unit 4311 of fig. 43), a fill unit (e.g., fill unit 4313 of fig. 43), a scroll unit (e.g., scroll unit 4315 of fig. 43), a presentation unit (e.g., presentation unit 4317 of fig. 43), a selection unit (e.g., selection unit 4319 of fig. 43), a contact unit (e.g., contact unit 4321 of fig. 43), a reception unit (e.g., reception unit 4323 of fig. 43), and an execution unit (e.g., execution unit 4325 of fig. 43).

In some embodiments, the processing unit 4305 (or one or more components thereof, such as units 4307-4329) is configured to: detect a search activation gesture from a user of the electronic device on the touch-sensitive display (e.g., with the detection unit 4307 and/or the touch-sensitive surface unit 4303); in response to detecting the search activation gesture, a search interface is displayed (e.g., with display unit 4309 and/or display unit 4301) on the touch-sensitive display, the search interface including: (i) a search entry section; and (ii) a prediction portion displayed prior to receiving any user input at the search entry portion, the prediction portion populated by one or more of: (a) at least one affordance for contacting a person of a plurality of previously contacted people, the person being automatically selected from the plurality of previously contacted people (e.g., by selection unit 4319) based at least in part on a current time, and (b) at least one affordance for performing a predicted action in an application of a plurality of applications available on the electronic device, the predicted action being automatically selected (e.g., by selection unit 4319) based at least in part on application usage history associated with a user of the electronic device. In some embodiments of the electronic device 4300, the processing unit (or one or more components thereof, such as the units 4307-4325) is further configured to perform the methods in any of C2-C18 described above in the "summary of the invention" section.

Fig. 44 illustrates a functional block diagram of an electronic device 4400 configured in accordance with the principles of various described embodiments, according to some embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 44 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4400 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 44, electronic device 4400 includes a display unit 4401 configured to display information (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touch screen and a touch screen display)), a touch-sensitive surface unit 4403 configured to receive contacts, gestures, and other user inputs on the touch screen display (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A), and a processing unit 4405 coupled to display unit 4401 and touch-sensitive surface unit 4403. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 44 shows the display unit 4401 and the touch-sensitive surface unit 4403 integrated with the electronic device 4400, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the processing unit includes a detection unit (e.g., the detection unit 4407 of fig. 44), a display unit (e.g., the display unit 4409 of fig. 44), a retrieval unit (e.g., the retrieval unit 4411 of fig. 44), a search mode entry unit (e.g., the search mode entry unit 4412 of fig. 44), a padding unit (e.g., the padding unit 4413 of fig. 44), an acquisition unit (e.g., the acquisition unit 4415 of fig. 44), a determination unit (e.g., the determination unit 4417 of fig. 44), and a selection unit (e.g., the selection unit 4419 of fig. 44).

The processing unit 4405 (or one or more components thereof, such as units 4407-4419) is configured to: displaying content associated with an application that is executing on an electronic device on a display unit (e.g., display unit 4401) (e.g., with display unit 4409 and/or display unit 4401); detecting, by the touch-sensitive surface unit (e.g., touch-sensitive surface unit 4403) and (e.g., with detection unit 4407 and/or touch-sensitive surface unit 4403), a swipe gesture that, when detected, will cause the electronic device to enter a search mode other than the application; in response to detecting the swipe gesture, enter a search mode (e.g., with search mode entry unit 4411), the search mode including a search interface displayed on a display unit (e.g., display unit 4407); in connection with entering the search mode, determining at least one suggested search query based at least in part on information associated with the content (e.g., with determination unit 4417); and prior to receiving any user input at the search interface, use the at least one suggested search query and populate (e.g., with population unit 4413) the displayed search interface. In some embodiments of electronic device 4400, the processing unit (or one or more components thereof, such as units 4407-4419) is further configured to perform the method in any of D2-D16 described above in the "summary" section.

According to some embodiments, fig. 45 illustrates a functional block diagram of an electronic device 4500 configured according to the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 45 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4500 is implemented as the portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 45, electronic device 4500 includes a display unit 4501 (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)), which is configured to display information, a touch-sensitive surface unit 4503 (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A) that is configured to receive contacts, gestures, and other user inputs on the touchscreen display, and a processing unit 4505 coupled to display unit 4501 and touch-sensitive surface unit 4503. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, FIG. 45 shows display unit 4501 and touch-sensitive surface unit 4503 integrated with electronic device 4500, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the processing unit includes a detection unit (e.g., detection unit 4507 of fig. 45), a display unit (e.g., display unit 4509 of fig. 45), a fill unit (e.g., fill unit 4511 of fig. 45), and a search mode entry unit (e.g., search mode entry unit 4513 of fig. 45).

The processing unit 4505 (or one or more components thereof, such as units 1007-1029) is configured to: detecting, by the touch-sensitive surface unit (e.g., touch-sensitive surface unit 4503) and (e.g., with detection unit 4507 and/or touch-sensitive surface unit 4503), a swipe gesture on the user interface, wherein when the swipe gesture is detected, the electronic device is to be brought into a search mode; and in response to detecting the swipe gesture, enter a search mode (e.g., with search mode entry unit 4513), and entering the search mode includes: prior to receiving any user input within the search interface, the first content item is used and the search interface is populated differently from the user interface (e.g., with a population element 4511 and/or display element 4509 and/or 4501). In some embodiments, in accordance with a determination that the user interface includes content associated with an application, wherein the user interface is different from a home screen that includes selectable icons for invoking the application, populating the search interface with the first content item includes using the at least one suggested search query and populating the search interface (e.g., with the population element 4511), wherein the search query is based at least in part on the content associated with the application; and in accordance with a determination that the user interface is associated with the page of the home screen, populating the search interface with the first content item includes populating the search interface with affordances that include selectable depictions of at least one point of interest within a threshold distance of a current location of the electronic device (e.g., with a population element 4511). In some embodiments of electronic device 4500, the processing unit (or one or more components thereof, such as units 4507-4513) is further configured to perform the method of E2 described above in the "summary of the invention" section.

Fig. 46 illustrates a functional block diagram of an electronic device 4600, according to some embodiments, configured according to the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 46 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4600 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 46, electronic device 4600 includes a display unit 4601 configured to display information (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)), a touch-sensitive surface unit 4603 configured to receive contacts, gestures, and other user inputs on the touchscreen display (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A), a location sensor unit 4607 configured to obtain location information for the electronic device, and a processing unit 4605 coupled with display unit 4601, touch-sensitive surface unit 4603, and location sensor unit 4607. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 46 shows display unit 4601 and touch-sensitive surface unit 4603 integrated with electronic device 4600, however, in some embodiments, one or both of these units communicate with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the processing unit includes a display unit (e.g., display unit 4609 of fig. 46), a retrieval unit (e.g., retrieval unit 4611 of fig. 46), a determination unit (e.g., determination unit 4613 of fig. 46), a storage unit (e.g., storage unit 4615 of fig. 46), an identification unit (e.g., identification unit 4617 of fig. 46), a selection unit (e.g., selection unit 4619 of fig. 46), a reception unit (e.g., reception unit 4621 of fig. 46), a provision unit (e.g., provision unit 4623 of fig. 46), and a playback unit (e.g., playback unit 4625 of fig. 46).

The processing unit 4605 (or one or more components thereof, such as units 4609-4625) is configured to: automatically determining (e.g., with determination unit 4613) that the electronic device user is located within a vehicle that is parked at a geographic location without instructions from the user; upon determining that the user has left the vehicle located at the geographic location, determining (e.g., with determination unit 4613) whether positioning information retrieved (e.g., with identification unit 4617) from a location sensor unit (e.g., location sensor unit 4607) (e.g., with retrieval unit 4621) for identifying the geographic location satisfies an accuracy criterion; upon determining (e.g., with determining unit 4613) that the location information does not meet the accuracy criteria, providing a prompt to the user (e.g., with providing unit 4623) to enter information regarding the geographic location; and in response to providing the prompt, receive information about the geographic location from the user (e.g., using receiving unit 4623) and store the information as vehicle location information (e.g., using storage unit 4615). In some embodiments of the electronic device 4600, the processing unit (or one or more components thereof, such as the units 4609-4625) is further configured to perform the methods in any of F2-F16 described above in the "summary of the invention" section.

According to some embodiments, fig. 47 illustrates a functional block diagram of an electronic device 4700 configured according to the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 47 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4700 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 47, electronic device 4700 includes a display unit 4701 configured to display information (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)), a touch-sensitive surface unit 4703 configured to receive contacts, gestures, and other user inputs on the touchscreen display (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A), a position sensor unit 4707 configured to obtain positioning information of the electronic device, and a processing unit 4705 coupled to display unit 4701 and touch-sensitive surface unit 4703. In some embodiments, the electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 47 shows display unit 4701 and touch-sensitive surface unit 4703 integrated with electronic device 4700, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. In some embodiments, the processing unit includes a detection unit (e.g., the detection unit 4709 of fig. 47), a display unit (e.g., the display unit 4711 of fig. 47), a retrieval unit (e.g., the retrieval unit 4713 of fig. 47), a determination unit (e.g., the determination unit 4715 of fig. 47), an identification unit (e.g., the identification unit 4717 of fig. 47), an unlocking unit (e.g., the unlocking unit 4719 of fig. 47), and a search pattern entry unit (e.g., the search pattern entry unit 4721 of fig. 47).

The processing unit 4705 (or one or more components thereof, such as units 4709-4719) is configured to: without receiving any instruction from the user of the electronic device: monitoring a geographic location of the electronic device using a location sensor unit (e.g., location sensor unit 4707); determining (e.g., by the determining unit 4715) that the electronic device is within a threshold distance of a predetermined type of point of interest based on the monitored geographic location; in accordance with a determination that the electronic device is within a threshold distance of the point of interest: identifying (e.g., by the identifying means 4717) at least one activity currently prevalent at the point of interest; retrieving (e.g., by retrieving unit 4713) information about the point of interest, including retrieving information about at least one activity currently prevalent at the point of interest; detecting (e.g., by detecting unit 4709 and/or touch-sensitive surface unit 4703) a first input by a touch-sensitive surface unit (e.g., touch-sensitive surface unit 4703), the first input, when detected, causing the electronic device to enter a search mode; and in response to detecting the first input, entering a search mode (e.g., via search mode entry unit 4721), and entering the search mode includes, prior to receiving any user input at the search interface, presenting (e.g., via display unit 4711 and/or display unit 4701) an affordance including: (i) information about the at least one activity, and (ii) an indication that the at least one activity has been determined to be an activity that is currently popular at the point of interest. In some embodiments of the electronic device 4700, the processing unit (or one or more components thereof, e.g., units 4709-4719) is further configured to perform any of the methods G2-G10 in the "summary" section, as described above.

According to some embodiments, fig. 48 illustrates a functional block diagram of an electronic device 4800 configured according to the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 48 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4800 is implemented as the portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 48, electronic device 4800 includes a display unit 4801 (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)) configured to display information, a touch-sensitive surface unit 4803 (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A) configured to receive contacts, gestures, and other user inputs on the touchscreen display, and a processing unit 4805 coupled to display unit 4801 and touch-sensitive surface unit 4803. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 48 shows the display unit 4801 and touch-sensitive surface unit 4803 integrated with the electronic device 4800, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a voice communication receiving unit (e.g., the voice communication receiving unit 4807 of fig. 48), a content item extracting unit (e.g., the content item extracting unit 4809 of fig. 48), an availability determining unit (e.g., the availability determining unit 4811 of fig. 48), an application identifying unit (e.g., the application identifying unit 4813 of fig. 48), a display unit (e.g., the display unit 4815 of fig. 48), a content item storage unit (e.g., the content item storage unit 4817 of fig. 48), a feedback providing unit (e.g., the feedback providing unit 4819 of fig. 48), an input detecting unit (e.g., the input detecting unit 4821 of fig. 48), an application opening unit (e.g., the application opening unit 4823 of fig. 48), a filling unit (e.g., the filling unit 4825 of fig. 48), and a voice communication analyzing unit (e.g., the voice communication analyzing unit 4827 of fig. 48).

In some embodiments, the processing unit (or one or more components thereof, such as units 4807-4827) is configured to: at least a portion of the voice communication is received (e.g., by voice communication receiving unit 4807), the portion of the voice communication including voice provided by a remote user of a remote device different from the user of the electronic device. The processing unit is further configured to: the content item is extracted based at least in part on the speech provided by the remote user of the remote device (e.g., by content item extraction unit 4809) and it is determined whether the content item is currently available on the electronic device (e.g., by availability determination unit 4811). In accordance with a determination that the content item is not currently available on the electronic device, the processing unit is further configured to: (i) identify an application associated with the content item (e.g., by application identification unit 4813), and (ii) display an optional description of the content item on the display (e.g., by display unit 4815 and/or display unit 4801). In response to detecting selection of the selectable description (e.g., via the input detection unit 4821 and/or the touch-sensitive surface unit 4803), the processing unit is configured to: the content item is stored for presentation by the identified application (e.g., by content item storage unit 4817).

In some embodiments of the electronic device 4800, the content item is a new event.

In some embodiments of electronic device 4800, the content item is a new event detail for an event currently associated with a calendar application on the electronic device.

In some embodiments of electronic device 4800, the content item is a new contact.

In some embodiments of electronic device 4800, the content item is new contact information for an existing contact associated with a phone application on the electronic device.

In some embodiments of the electronic device 4800, the voice communication is a real-time telephone call.

In some embodiments of the electronic device 4800, the voice communication is a real-time FaceTime communication.

In some embodiments of the electronic device 4800, the voice communication is a recorded voice mailbox.

In some embodiments of electronic device 4800, displaying the selectable description includes displaying the selectable description within a user interface that includes recent calls made using a phone application (e.g., with display unit 4815 and/or display unit 4801).

In some embodiments of the electronic device 4800, the optional description is displayed with the following indication: the content item is associated with a voice communication (e.g., using display unit 4815 and/or display unit 4801).

In some embodiments of the electronic device 4800, detecting the selection includes receiving the selection (e.g., using the input detection unit 4821) while displaying the user interface including the most recent call.

In some embodiments of the electronic device 4800, the processing unit is further configured to: in conjunction with displaying the selectable description of the content item, feedback is provided (e.g., using feedback providing unit 4819) to the user of the electronic device that the content item has been detected.

In some embodiments of electronic device 4800, providing feedback includes sending information regarding the detection of the content item (e.g., by feedback providing unit 4819) to a different electronic device that is proximate to the electronic device.

In some embodiments of the electronic device 4800, the processing unit is further configured to: determining that the voice communication includes information about the first physical location; detect input (e.g., by input detection unit 4821); and in response to detecting the input, open (e.g., via application opening unit 4823) an application that is capable of accepting the location data and populating the application with information about the first physical location.

In some embodiments of the electronic device 4800, the application is a mapping application, and populating the mapping application with information related to the first physical location comprises: a map displayed within the mapping application is populated with a location identifier corresponding to the first physical location.

In some embodiments of the electronic device 4800, the processing unit is further configured to: determining that the voice communication includes information about the first physical location; an input is detected (e.g., by input detection unit 4821), and in response to detecting the input, the search interface is populated with information about the first physical location (e.g., by population unit 4825).

In some embodiments of the electronic device 4800, extracting the content item includes analyzing a portion of the voice communication to detect a predetermined type of content, and analyzing while the voice communication is output through an audio system in communication with the electronic device.

In some embodiments of electronic device 4800, analyzing the voice communication includes (e.g., using voice communication analysis unit 4827): (i) converting speech provided by a remote user of a remote device into text; (ii) applying a natural language processing algorithm to the text to determine whether the text includes one or more predefined keywords; and (iii) in accordance with a determination that the text includes the respective predefined keyword, determining that the voice communication includes speech describing the content item.

In some embodiments of electronic device 4800, receiving at least the portion of the voice communication includes receiving an indication (e.g., an instruction) from a user of the electronic device that the portion of the voice communication should be analyzed.

In some embodiments of electronic device 4800, the indication corresponds to a selection of a hardware button.

In some embodiments of electronic device 4800, the indication corresponds to a command from the user of the electronic device, including the word "hey, Siri".

In some embodiments of the electronic device 4800, the processing unit is further configured to: a second portion of the voice communication is received, the second portion including voice provided by the remote user of the remote device and voice provided by the user of the electronic device (e.g., the voice communication is a real-time telephone call, the second portion including discussions between the user and the remote user). The processing unit is further configured to: extract the second content item 4809 based at least in part on the speech provided by the remote user of the remote device and the speech provided by the user of the electronic device (e.g., using the content item extraction unit); in accordance with a determination that the second content item is not currently available on the electronic device: (i) identify a second application associated with the second content item (e.g., by application identification unit 4813), and (ii) display a second selectable description of the second content item on the display (e.g., by display unit 4815 and/or display unit 4801). In response to detecting selection of the second selectable description, the processing unit is configured to: the second content item is stored (e.g., using content item storage unit 4817) for presentation by the identified second application.

In some embodiments of electronic device 4800, the selectable description and the second selectable description are displayed within a user interface, wherein the user interface includes recent calls made using a phone application.

According to some embodiments, fig. 49 illustrates a functional block diagram of an electronic device 4900 configured in accordance with the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 49 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 4900 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 49, electronic device 4900 includes a display unit 4901 (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)), configured to display information, a touch-sensitive surface unit 4903 (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A) configured to receive contacts, gestures, and other user inputs on the touchscreen display, and a processing unit 4905 coupled to display unit 4901 and touch-sensitive surface unit 4903. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 49 shows display unit 4901 and touch-sensitive surface unit 4903 integrated with electronic device 4900, however, in some embodiments, one or both of these units communicate with the electronic device while they remain physically separate from the electronic device. The processing unit includes a voice communication receiving unit (e.g., a voice communication receiving unit 4907 of fig. 49), a content item extracting unit (e.g., a content item extracting unit 4909 of fig. 49), an indication providing unit (e.g., an indication providing unit 4911 of fig. 49), an input detecting unit (e.g., an input detecting unit 4913 of fig. 49), an application opening unit (e.g., an application opening unit 4915 of fig. 49), an application filling unit (e.g., an application filling unit 4917 of fig. 49), a feedback providing unit (e.g., a feedback providing unit 4919 of fig. 49), and a voice communication analyzing unit (e.g., a voice communication analyzing unit 4921 of fig. 49).

In some embodiments, the processing unit (or one or more components thereof, such as units 4907 to 4921) is configured to: at least a portion of the voice communication is received (e.g., by voice communication receiving unit 4907), the portion of the voice communication including voice provided by a remote user of a remote device different from the user of the electronic device. The processing unit is further configured to: determining the voice communication includes identifying the voice of the physical location (e.g., by the content item extraction unit 4909). In response to determining that the voice communication includes voice identifying the physical location, the processing unit is configured to: an indication is provided (e.g., by the content item extraction unit 4909) that information regarding the physical location has been detected. The processing unit is further configured to: an input is detected by the touch-sensitive surface unit (e.g., by the input detection unit 4911). In response to detecting the input, the processing unit 2506 is configured to: (i) open the application that accepts the geographic location data (e.g., by application opening unit 4913), and (ii) populate the application with information about the physical location (e.g., by application population unit 4915).

In some embodiments of the electronic device 4900, the voice communication is a real-time telephone call.

In some embodiments of the electronic device 4900, the voice communication is a real-time FaceTime call.

In some embodiments of the electronic device 4900, the voice communication is a recorded voice mailbox.

In some embodiments of the electronic device 4900, providing the indication includes displaying a selectable description of the physical location within a user interface, wherein the user interface includes recent calls made using the phone application.

In some embodiments of the electronic device 4900, the optional description indicates that the content item is associated with a voice communication.

In some embodiments of the electronic device 4900, detecting the input includes: an input is detected on the selectable depiction while the user interface including the most recent call is displayed.

In some embodiments of the electronic device 4900, providing the indication includes providing haptic feedback to a user of the electronic device (e.g., using the feedback providing unit 4919).

In some embodiments of the electronic device 4900, providing the indication includes transmitting information about the physical location (e.g., using the feedback providing unit 4919) to a different electronic device that is proximate to the electronic device.

In some embodiments of the electronic device 4900, determining that the voice communication includes voice describing the physical location includes: the portion of the voice communication is analyzed to detect information about the physical location (e.g., using voice communication analysis unit 4921), and the analysis is performed while the voice communication is output through an audio system in communication with the electronic device.

In some embodiments of the electronic device 4900, receiving at least the portion of the voice communication includes receiving instructions from a user of the electronic device that the portion of the voice communication should be analyzed.

In some embodiments of the electronic device 4900, the instruction corresponds to a selection of a hardware button.

In some embodiments of the electronic device 4900, the instruction corresponds to a command from a user of the electronic device, including the word "hey, Siri".

According to some embodiments, fig. 50 illustrates a functional block diagram of an electronic device 5000 configured according to the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 50 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 5000 is implemented as a portable multifunction device 100 (fig. 1A to 1B).

As shown in fig. 50, electronic device 5000 includes a display unit 5001 (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)) configured to display information, a touch-sensitive surface unit 5003 (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A) configured to receive contacts, gestures, and other user inputs on the touchscreen display, and a processing unit 5005 coupled to display unit 5001 and touch-sensitive surface unit 5003. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 50 shows display unit 5001 and touch-sensitive surface unit 5003 integrated with electronic device 5000, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing units include a presentation unit (e.g., the presentation unit 5007 of fig. 50), a next input determination unit (e.g., the next input determination unit 5009 of fig. 50), a content analysis unit (e.g., the content analysis unit 5011 of fig. 50), a selection reception unit (e.g., the selection reception unit 5013 of fig. 50), a key input monitoring unit (e.g., the key input monitoring unit 5015 of fig. 50), and a presentation stop unit (e.g., the presentation stop unit 5017 of fig. 50).

In some embodiments, the processing unit (or one or more components thereof, e.g., units 5007-5017) is configured to: the text entry field and the copy of the session are presented in the instant messaging application on the display (e.g., via presentation unit 5007 and/or display unit 5001). When presenting the instant messaging application on the display, the processing unit is further configured to: determining that the next possible input from the electronic device user is information about the physical location (e.g., by the next input determination unit 5009) the processing unit is further configured to: analyzing content associated with the text entry field and the session transcript, determining a suggested physical location based at least in part on a portion of the analyzed content (e.g., by the content analysis unit 5011); presenting a selectable user interface element identifying the suggested physical location within the instant messaging application on the display (e.g., via presenting unit 5007); receiving a selection of a selectable user interface element (e.g., by selecting the receiving unit 5013 and/or the touch-sensitive surface unit 5003); and in response to receiving the selection, a representation of the suggested physical location is presented in the text entry field (e.g., by presentation element 5007).

In some embodiments of the electronic device 5000, the instant messaging application includes a virtual keyboard and the selectable user interface element is displayed in a suggestion portion adjacent to and above the virtual keyboard.

In some embodiments of the electronic device 5000, determining that the next possible input from the user of the electronic device is information about the physical location comprises: content associated with the text entry field and the transcript of the session is processed to detect that the transcript of the session includes a question regarding the current location of the user.

In some embodiments of the electronic device 5000, processing the content includes applying a natural language processing algorithm to detect one or more predefined keywords that constitute the question.

In some embodiments of the electronic device 5000, the problem is included in a message received from a second user different from the user.

In some embodiments of the electronic device 5000, determining that the next possible input from the user of the electronic device is information about the physical location comprises: the typed input received from the user is monitored (e.g., using the typed input monitoring unit 5015) in the text input portion of the instant messaging application.

In some embodiments of the electronic device 5000, the processing unit is further configured to: in accordance with a determination that the user is typing and has not selected a selectable user interface element, presentation of the selectable user interface element is stopped (e.g., using the presentation stopping unit 5017).

In some embodiments of the electronic device 5000, the processing unit is further configured to: in accordance with a determination that the user has provided additional input indicating that the user will not select a selectable user interface element, presentation of the selectable user interface element is stopped (e.g., using the presentation stopping unit 5017).

In some embodiments of the electronic device 5000, the representation of the proposed physical location includes information for identifying a current geographic location of the electronic device.

In some embodiments of the electronic device 5000, the representation of the proposed physical location is an address.

In some embodiments of the electronic device 5000, the suggested physical location is a map object that includes an identifier of the suggested physical location.

In some embodiments of the electronic device 5000, the suggested physical location corresponds to a location that the user recently viewed in an application other than the instant messaging application.

In some embodiments of the electronic device 5000, the instant messaging application is an email application.

In some embodiments of the electronic device 5000, the instant messaging application is a text messaging application.

According to some embodiments, fig. 51 illustrates a functional block diagram of an electronic device 5100 configured in accordance with principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 51 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 5100 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 51, the electronic device 5100 includes a display unit 5101 configured to display information, such as the touch-sensitive display system 112 of fig. 1A (also referred to as a touch screen and a touch screen display), a touch-sensitive surface unit 5103 configured to receive contacts, gestures, and other user inputs on the touch screen display (e.g., the display controller 156 and the touch-sensitive display system 112 of fig. 1A), and a processing unit 5105 coupled with the display unit 5101 and the touch-sensitive surface unit 5103. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 51 shows the display unit 5101 and touch-sensitive surface unit 5103 integrated with the electronic device 5100, however, in some embodiments, while these units remain physically separate from the electronic device, one or both of these units are in communication with the electronic device. The processing units include an information acquisition unit (e.g., the information acquisition unit 5107 of fig. 51), an application exit unit (e.g., the application exit unit 5109 of fig. 51), a request reception unit (e.g., the request reception unit 5111 of fig. 51), an application capability determination unit (e.g., the application capability determination unit 5113 of fig. 51), an application presentation unit (e.g., the application presentation unit 5115 of fig. 51), an application filling unit (e.g., the application filling unit 5117 of fig. 51), an input detection unit (e.g., the input detection unit 5119 of fig. 51), an application switching user interface display unit (e.g., the application switching user interface display unit 5121 of fig. 51), an application association determination unit (e.g., the application association determination unit 5123 of fig. 51), and an access providing unit (e.g., the access providing unit 5125 of fig. 51).

In some embodiments, the processing unit (or one or more components thereof, e.g., units 5107-5125) is configured to: while the first application is displayed, information identifying the first physical location viewed by the user in the first application is acquired (e.g., by the information acquisition unit 5107). The processing unit is further configured to: the first application is exited (e.g., by the application exit unit 5109), and upon exiting the first application, a request is received from the user to open a second application different from the first application (e.g., by the request receiving unit 5111). In response to receiving the request and in accordance with a determination that the second application is able to accept the geographic location information (e.g., a determination processed or performed by the application capability determination unit 5113), presenting the second application (e.g., by the application presentation unit 5115), and presenting the second application includes populating the second application with information based at least in part on the first physical location identifying information (e.g., by the application population unit 5117).

In some implementations of the electronic device 5100, receiving the request to open the second application includes: upon exiting the first application, an input is detected (e.g., using input detection unit 5119) on the affordance of the second application.

In some embodiments of the electronic device 5100, the affordance for the second application is an icon displayed within the home screen of the electronic device.

In some implementations of the electronic device 5100, detecting the input includes: (i) detecting a double-click on the physical Home button, (ii) in response to detecting the double-click, (e.g., using application-switching user interface display unit 5121) displaying an application-switching user interface, and (iii) detecting selection of an affordance from within the application-switching user interface.

In some embodiments of the electronic device 5100, populating the second application includes displaying a user interface object that includes information based at least in part on the first physical location identifying information.

In some embodiments of the electronic device 5100, the user interface object includes a textual description that informs the user that the first physical location was recently viewed in the first application.

In some embodiments of the electronic device 5100, the user interface object is a map displayed in the second application, and populating the second application includes populating the map to include the identifier of the first physical location.

In some implementations of the electronic device 5100, the second application presents a virtual keyboard and the user interface object is displayed above the virtual keyboard.

In some embodiments of the electronic device 5100, acquiring the information includes acquiring information about a second physical location, and displaying the user interface object includes displaying the user interface object having information related to the second physical location.

In some embodiments of the electronic device 5100, determining that the second application is capable of accepting geographic location information includes performing one or more of the following (e.g., one or more determinations performed by the application association determination unit 5123 and/or the application capability determination unit 5113): (i) determining that the second application includes an input-receiving field capable of accepting and processing the geographic location data; (ii) determining that the second application is capable of displaying the geographic location information on the map; (iii) determining that the second application is capable of using the geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using the geographic location information.

In some embodiments of the electronic device 5100, determining that the second application is capable of accepting geographic location information includes: it is determined that the second application includes an input-receiving field capable of accepting and processing the geographic location data, and the input-receiving field is a search box that allows searching within a map displayed in the second application.

In some embodiments of the electronic device 5100, the processing unit is further configured to: in response to receiving the request, it is determined whether the second application is associated with the first application based on the user's application usage history (e.g., using the application association determining unit 5123).

In some embodiments of the electronic device 5100, the processing unit is further configured to: the second application is provided access rights to the information identifying the first physical location (e.g., using access providing unit 5125) prior to rendering the second application, and the second application cannot access the information identifying the first physical location until the access rights are provided.

Fig. 52 illustrates a functional block diagram of an electronic device 5200, according to some embodiments, configured in accordance with the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 52 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 5200 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 52, electronic device 5200 includes a display unit 5201 configured to display information (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touch screen and a touch screen display)), a touch-sensitive surface unit 5203 configured to receive contacts, gestures, and other user inputs on the touch screen display (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A), and a processing unit 5205 coupled to display unit 5201 and touch-sensitive surface unit 5203. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 52 shows the display unit 5201 and the touch-sensitive surface unit 5203 integrated with the electronic device 5200, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing units include an information acquisition unit (e.g., the information acquisition unit 5207 of fig. 52), an input detection unit (e.g., the input detection unit 5209 of fig. 52), an application identification unit (e.g., the application identification unit 5211 of fig. 52), an affordance presentation unit (e.g., the affordance presentation unit 5213 of fig. 52), an application opening unit (e.g., the application opening unit 5215 of fig. 52), an application filling unit (e.g., the application filling unit 5217 of fig. 52), an application switching user interface presentation unit (e.g., the application switching user interface presentation unit 5219 of fig. 52), and an application capability determination unit (e.g., the application capability determination unit 5221 of fig. 52).

In some embodiments, the processing unit (or one or more components thereof, such as units 5207 to 5221) is configured to: information for identifying a first physical location viewed by a user in a first application is acquired (e.g., by the information acquisition unit 5207) and a first input is detected (e.g., by the input detection unit 5209). In response to detecting the first input, the processing unit is configured to: identify (i) a second application that is capable of accepting geographic location information (e.g., via the application identification unit 5209), and (ii) present an affordance different from the first application on at least a portion of the display and suggest opening the second application having information about the first physical location (e.g., via the affordance presentation unit 5213). The processing unit is further configured to: a second input is detected (e.g., by input detection unit 5209) at the affordance. In response to detecting the second input at the affordance, the processing unit is configured to: (i) open the second application (e.g., by the application opening unit 5215), and (ii) populate the second application to include information based at least in part on the first physical location identifying information (e.g., by the application populating unit 5217).

In some embodiments of the electronic device 5200, the first input corresponds to a request to open an application-switching user interface (e.g., the first input is a double-click on a physical Home button of the electronic device)

In some embodiments of the electronic device 5200, the affordance is presented within an application-switching user interface.

In some embodiments of the electronic device 5200, presenting the affordance includes: presenting, in conjunction with presenting the affordance, a representation of an application executing on the electronic device within the application-switching user interface (e.g., using the application-switching user interface presentation unit 5219); and presenting the affordance in an area of the display that is below the application representation.

In some embodiments of the electronic device 5200, the first input corresponds to a request to open a Home screen of the electronic device (e.g., the first input is a single tap on a physical Home button of the electronic device).

In some embodiments of the electronic device 5200, the affordance is presented over a portion of the home screen.

In some embodiments of the electronic device 5200, the suggestion includes a textual description that is specific to a type associated with the second application.

In some embodiments of the electronic device 5200, populating the second application includes displaying a user interface object that includes information based at least in part on the first physical location identifying information.

In some embodiments of the electronic device 5200, the user interface object includes a textual description that informs the user that the first physical location was recently viewed in the first application.

In some embodiments of the electronic device 5200, the user interface object is a map displayed in the second application, and populating the second application includes populating the map to include an identifier of the first physical location.

In some embodiments of the electronic device 5200, the second application presents a virtual keyboard and the user interface object is displayed over the virtual keyboard.

In some embodiments of the electronic device 5200, identifying the second application that is capable of accepting the geographic location information includes performing one or more of the following (e.g., one or more determinations performed using the application capability determination unit 5221): (i) determining that the second application includes an input-receiving field capable of accepting and processing geographic location data; (ii) determining that the second application is capable of displaying the geographic location information on the map; (iii) determining that the second application is capable of using the geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing the transport service using the geographic location information.

In some embodiments of the electronic device 5200, identifying that the second application is capable of accepting the geographic location information includes: it is determined (e.g., using the application capability determination unit 5221) that the second application includes an input-receiving field that is capable of accepting and processing geographic location data, and that the input-receiving field is a search box that allows searching within a map displayed in the second application.

Fig. 53 illustrates a functional block diagram of an electronic device 5300 configured according to principles of various described embodiments, according to some embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 53 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 5300 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 53, electronic device 5300 includes a display unit 5301 (e.g., touch-sensitive display system 112 of fig. 1A (also referred to as a touch-screen and touch-screen display)), a touch-sensitive surface unit 5303 (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A) configured to receive contacts, gestures, and other user inputs on the touch-screen display, and a processing unit 5305 coupled with display unit 5301 and touch-sensitive surface unit 5303. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 53 shows the display unit 5301 and the touch-sensitive surface unit 5303 integrated with the electronic device 5300, however, in some embodiments, one or both of these units communicate with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes an information acquisition unit (e.g., information acquisition unit 5307 of fig. 53), a vehicle entry determination unit (e.g., vehicle entry determination unit 5309 of fig. 53), a presentation providing unit (e.g., presentation providing unit 5311 of fig. 53), an instruction receiving unit (e.g., instruction receiving unit 5313 of fig. 53), a route guidance promotion unit (e.g., route guidance promotion unit 5315 of fig. 53), and a message detection unit (e.g., message detection unit 5317 of fig. 53).

In some embodiments, the processing unit (or one or more components thereof, e.g., units 5307-5317) is configured to: information identifying a first physical location viewed by a user in the first application executing on the electronic device is obtained (e.g., by information obtaining unit 5307). The processing unit is further configured to: it is determined that the user has entered the vehicle (e.g., by the vehicle entry determination unit 5309). In response to determining that the user has entered the vehicle, the processing unit is configured to: a prompt is provided to the user prompting the user to use the first physical location as a destination for route guidance (e.g., by prompt providing unit 5311). In response to providing the prompt, an instruction is received from the user instructing the user to use the first physical location as a destination for route guidance (e.g., by the instruction receiving unit 5313). The processing unit is further configured to: route guidance to the first physical location is facilitated (e.g., by route guidance facilitation unit 5307).

In some embodiments of electronic device 5300, the processing unit is further configured to: detecting (e.g., by message detection unit 5317) that the electronic device has received a message, including detecting that the message contains information identifying a second physical location; and in response to the detection, provide a new prompt to the user (e.g., by prompt providing unit 5311) to use the second physical location as a new destination for route guidance.

In some embodiments of electronic device 5300, the processing unit is further configured to: in response to receiving an instruction from the user to use the second physical location as a new destination, route guidance to the second physical location is facilitated (e.g., by the route guidance facilitation unit 5315).

In some embodiments of electronic device 5300, detecting that the message contains information identifying the second physical location includes: the detection is performed when a virtual assistant available on the electronic device is reading a message to a user through an audio system in communication with the electronic device.

In some embodiments of electronic device 5300, determining that the user has entered the vehicle includes detecting that the electronic device has established a communication link with the vehicle.

In some embodiments of electronic device 5300, facilitating the route guidance includes providing the route guidance through a display of the electronic device.

In some embodiments of electronic device 5300, facilitating the route guidance includes sending, to the vehicle, information identifying the first physical location.

In some embodiments of electronic device 5300, facilitating route guidance includes providing route guidance through an audio system in communication with the electronic device (e.g., a car speaker or an internal speaker of the device itself).

According to some embodiments, fig. 54 illustrates a functional block diagram of an electronic device 5400 configured according to principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 54 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 5400 is implemented as the portable multifunction device 100 (fig. 1A to 1B).

As shown in fig. 54, the electronic device 5400 includes a display unit 5401 (e.g., the touch-sensitive display system 112 of fig. 1A (also referred to as a touchscreen and a touchscreen display)) configured to display information, a touch-sensitive surface unit 5403 (e.g., the display controller 156 and the touch-sensitive display system 112 of fig. 1A) configured to receive contacts, gestures, and other user inputs on the touchscreen display, and a processing unit 5405 coupled to the display unit 5401 and the touch-sensitive surface unit 5403. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 54 shows the display unit 5401 and touch-sensitive surface unit 5403 integrated with the electronic device 5400, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a presentation unit (e.g., the presentation unit 5407 of fig. 54), a request reception unit (e.g., the request reception unit 5409 of fig. 54), a user interface object providing unit (e.g., the user interface object providing unit 5411 of fig. 54), an active pasting unit (e.g., the active pasting unit 5413 of fig. 54), and a capability determination unit (e.g., the capability determination unit 5415 of fig. 54).

In some embodiments, the processing unit (or one or more components thereof, e.g., units 5407-5415) is configured to: presenting content in the first application (e.g., via presentation unit 5407 and/or display unit 5401); receiving, from a user, a request to open a second application different from the first application (e.g., via the request-receiving element and/or the touch-sensitive surface element 5403), the second application including an input-receiving field; in response to receiving the request, presenting the second application with the input-receiving field (e.g., via presentation unit 5407 and/or display unit 5401); prior to receiving any user input at the input-receiving field, providing a selectable user interface object to allow the user to paste at least a portion of the content into the input-receiving field (e.g., by the user interface object providing unit 5411 and/or the display unit 5401); and paste the portion of the content into the input-receiving field (e.g., via the active paste element 5413) in response to detecting selection of the selectable user interface object.

In some embodiments of the electronic device 5400, prior to providing the selectable user interface object, the processing unit is further configured to: the input reception field is identified (e.g., using the capability determination unit 5415) as a field that can accept the portion of the content.

In some embodiments of the electronic device 5400, in response to detecting selection of the input-receiving field, the input-receiving field is identified as a field that is capable of accepting the portion of the content.

In some embodiments of the electronic device 5400, the portion of the content corresponds to an image.

In some embodiments of the electronic device 5400, the portion of the content corresponds to textual content.

In some embodiments of the electronic device 5400, the portion of the content corresponds to textual content and an image.

In some embodiments of the electronic device 5400, the first application is a web browsing application and the second application is an instant messaging application.

In some embodiments of the electronic device 5400, the first application is a photo browsing application and the second application is an instant messaging application.

In some embodiments of the electronic device 5400, the processing unit is further configured to: a request to copy at least the portion of the content is received before receiving a request to open the second application.

In some embodiments of the electronic device 5400, the selectable user interface objects are displayed along with an indication of: the portion of the content was recently viewed in the first application.

Fig. 55 illustrates a functional block diagram of an electronic device 5500 configured in accordance with the principles of various described embodiments, according to some embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof which carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in fig. 55 are optionally combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein. For ease of discussion, the electronic device 5500 is implemented as a portable multifunction device 100 (fig. 1A-1B).

As shown in fig. 55, electronic device 5500 includes a display unit 5501 configured to display information (e.g., touch-sensitive display system 112 (also referred to as a touch screen and a touch screen display) of fig. 1A), a touch-sensitive surface unit 5503 configured to receive contacts, gestures, and other user inputs on the touch screen display (e.g., display controller 156 and touch-sensitive display system 112 of fig. 1A), and a processing unit 5505 coupled to display unit 5501 and touch-sensitive surface unit 5503. In some embodiments, an electronic device is configured according to any one of the computing devices shown in fig. 1E (e.g., computing devices a-D). For ease of illustration, fig. 55 shows display unit 5501 and touch-sensitive surface unit 5503 integrated with electronic device 5500, however, in some embodiments, one or both of these units are in communication with the electronic device, although these units remain physically separate from the electronic device. The processing unit includes a presentation unit (e.g., presentation unit 5507 of fig. 55), a determination unit (e.g., determination unit 5509 of fig. 55), an acquisition unit (e.g., acquisition unit 5511 of fig. 55), a search execution unit (e.g., search execution unit 5513 of fig. 55), an information preparation unit (e.g., information preparation unit 5515 of fig. 55), an affordance display unit (e.g., affordance display unit 5517 of fig. 55), and a detection unit (e.g., detection unit 5519 of fig. 55).

In some embodiments, the processing unit (or one or more components thereof, e.g., units 5507-5519) is configured to: present textual content associated with the application on the display (e.g., via presentation unit 5507 and/or display unit 5501); determining that a portion of the textual content is relevant to: (i) location, (ii) contact, or (iii) event (e.g., by determining unit 5509); upon determining that the portion of the text content is location-related, location information is acquired from a location sensor on the electronic device (e.g., by acquisition unit 5511), and the acquired location information is prepared for display as a predicted content item (e.g., by information preparation unit 5515); upon determining that the portion of the text content is related to a contact, performing a search on the electronic device, searching for contact information related to the portion of the text content (e.g., via search execution unit 5513), and preparing information associated with at least one contact retrieved via the search for display as a predicted content item (e.g., via information preparation unit 5515); upon determining that the portion of the textual content is related to an event, performing a new search on the electronic device, searching for event information related to the portion of the textual content (e.g., by search execution unit 5513), and preparing information based at least in part on at least one event retrieved by the search for display as a predicted content item (e.g., by information preparation unit 5515); displaying an affordance including a predicted content item in an application (e.g., via affordance display unit 5517 and/or display unit 5501); detecting selection of an affordance through the touch-sensitive surface (e.g., through detection unit 5519); and in response to detecting the selection, information associated with the predicted content item is displayed on the display adjacent to the textual content (e.g., by presentation unit 5507 and/or display unit 5501).

In some embodiments of the electronic device 5500, the portion of the textual content corresponds to the textual content that was most recently presented in the application.

In some embodiments of the electronic device 5500, the application is an instant messaging application, and the portion of the textual content is a question received in the instant messaging application from a remote user of a remote device different from the electronic device.

In some embodiments of the electronic device 5500, the portion of textual content is input provided by an input receiving field within an application by a user of the electronic device.

In some embodiments of the electronic device 5500, the portion of textual content is identified in response to a user input selecting a user interface object that includes the portion of textual content.

In some embodiments of the electronic device 5500, the application is an instant messaging application and the user interface object is a message bubble in a conversation displayed within the instant messaging application.

In some embodiments of the electronic device 5500, the processing unit is further configured to: detecting selection of a second user interface object; in response to detecting the selection: (I) ceasing to display the affordance having the predicted content item, and (ii) determining that textual content associated with the second user interface object is relevant to the location, contact, or event; and, in accordance with the determination, displaying the new predicted content item within the application.

In some embodiments of the electronic device 5500, the affordance is displayed adjacent to a virtual keyboard within an application.

In some embodiments of the electronic device 5500, information associated with the predicted content item is displayed in an input-receiving field, wherein the input-receiving field is a field that displays typing input received at the virtual keyboard.

The operations in any of the information processing methods described above are optionally implemented by running one or more functional modules in an information processing apparatus, such as a general purpose processor (e.g., as described above with respect to fig. 1A and 3) or an application specific chip.

The operations described above with reference to fig. 6A-6B and 8A-8B are optionally performed by the components depicted in fig. 1A-1B or fig. 42-55. For example, performing operation 602 and detecting operation 802 are optionally implemented by event sorter 170, event recognizer 180, and event handler 190. Event monitor 171 in event sorter 170 detects a contact on touch-sensitive display 112 and event dispatcher module 174 delivers the event information to application 136-1. A respective event recognizer 180 of application 136-1 compares the event information to respective event definitions 186 and determines whether a first contact at a first location on the touch-sensitive surface (or whether rotation of the device) corresponds to a predefined event or sub-event, such as selection of an object on a user interface, or rotation of the device from one orientation to another. When a respective predefined event or sub-event is detected, event recognizer 180 activates event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update application internal state 192. In some embodiments, event handler 190 accesses a respective GUI updater 178 to update the content displayed by the application. Similarly, it will be clear to those skilled in the art how other processes may be implemented based on the components depicted in fig. 1A-1B.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the exemplary discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments described, with various modifications as are suited to the particular use contemplated.

Claims (39)

1. A method, comprising:

at an electronic device with a touch-sensitive display:

acquiring information for identifying a first physical location viewed by a user in a first application;

detecting a first input;

in response to detecting the first input:

identifying a second application capable of accepting geographic location information; and

presenting an affordance different from the first application on at least a portion of the touch-sensitive display and suggesting opening of the second application, wherein the suggestion includes information about the first physical location;

Detecting a second input at the affordance; and

in response to detecting the second input at the affordance:

opening the second application; and

populating the second application to include information based at least in part on the first physical location identifying information.

2. The method of claim 1, wherein the first input corresponds to a request to open an application-switching user interface.

3. The method of claim 2, wherein the affordance is presented within the application-switching user interface.

4. The method of claim 3, wherein presenting the affordance comprises:

presenting, in conjunction with presenting the affordance, a representation of an application executing on the electronic device within the application-switching user interface; and

presenting the affordance in the touch-sensitive display area that is below the representation of the application.

5. The method of claim 1, wherein the first input corresponds to a request to open a home screen of the electronic device.

6. The method of claim 5, wherein the affordance is presented over a portion of the home screen.

7. The method of claim 1, wherein the suggestion includes a textual description that is specific to a type associated with the second application.

8. The method of claim 1, wherein populating the second application includes displaying a user interface object that includes information that is based at least in part on the first physical location identifying information.

9. The method of claim 8, wherein the user interface object includes a textual description that informs the user that the first physical location was recently viewed in the first application.

10. The method of claim 9, wherein:

the user interface object is a map displayed within the second application, and

populating the second application includes: populating the map to include an identifier of the first physical location.

11. The method of claim 9, wherein the second application is presented with a virtual keyboard and the user interface object is displayed above the virtual keyboard.

12. The method of claim 1, wherein identifying that the second application is capable of accepting geographic location information comprises one or more of: (i) determining that the second application includes an input-receiving field capable of accepting and processing geographic location data; (ii) determining that the second application is capable of displaying geographic location information on a map; (iii) determining that the second application is capable of using geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using geographic location information.

13. The method of claim 12, wherein:

identifying that the second application is capable of accepting geographic location information comprises: determining that the second application includes an input-receiving field capable of accepting and processing geographic location data; and is

The input-receiving field is a search box that allows searching within a map displayed within the second application.

14. A non-transitory computer-readable storage medium storing executable instructions that, when executed by an electronic device with a touch-sensitive display, cause the electronic device to:

acquiring information for identifying a first physical location viewed by a user in a first application;

detecting a first input;

in response to detecting the first input:

identifying a second application capable of accepting geographic location information; and

presenting an affordance different from the first application on at least a portion of the touch-sensitive display and suggesting opening of the second application, wherein the suggestion includes information about the first physical location;

detecting a second input at the affordance; and

in response to detecting the second input at the affordance:

Opening the second application; and

populating the second application to include information based at least in part on the first physical location identifying information.

15. The non-transitory computer-readable storage medium of claim 14, wherein the first input corresponds to a request to open an application-switching user interface.

16. The non-transitory computer-readable storage medium of claim 15, wherein the affordance is presented within the application-switching user interface.

17. The non-transitory computer-readable storage medium of claim 16, wherein presenting the affordance includes:

presenting, in conjunction with presenting the affordance, a representation of an application executing on the electronic device within the application-switching user interface; and

presenting the affordance in the touch-sensitive display area that is below the representation of the application.

18. The non-transitory computer-readable storage medium of claim 14, wherein the first input corresponds to a request to open a home screen of the electronic device.

19. The non-transitory computer-readable storage medium of claim 18, wherein the affordance is presented over a portion of the home screen.

20. The non-transitory computer-readable storage medium of claim 14, wherein the suggestion includes a textual description that is specific to a type associated with the second application.

21. The non-transitory computer-readable storage medium of claim 14, wherein populating the second application includes displaying a user interface object that includes information that is based at least in part on the first physical location identifying information.

22. The non-transitory computer-readable storage medium of claim 21, wherein the user interface object includes a textual description that informs the user that the first physical location was recently viewed in a first application.

23. The non-transitory computer-readable storage medium of claim 22, wherein:

the user interface object is a map displayed within the second application, and

populating the second application includes: populating the map to include an identifier of the first physical location.

24. The non-transitory computer readable storage medium of claim 22, wherein the second application is presented with a virtual keyboard and the user interface object is displayed above the virtual keyboard.

25. The non-transitory computer-readable storage medium of claim 14, wherein identifying that the second application is capable of accepting geographic location information comprises one or more of: (i) determining that the second application includes an input-receiving field capable of accepting and processing geographic location data; (ii) determining that the second application is capable of displaying geographic location information on a map; (iii) determining that the second application is capable of using geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using geographic location information.

26. The non-transitory computer-readable storage medium of claim 25, wherein:

identifying that the second application is capable of accepting geographic location information comprises: determining that the second application includes an input-receiving field capable of accepting and processing geographic location data; and is

The input-receiving field is a search box that allows searching within a map displayed within the second application.

27. An electronic device, comprising:

a touch-sensitive display;

one or more processors; and

memory storing one or more programs that, when executed by the one or more processors, cause the electronic device to:

Acquiring information for identifying a first physical location viewed by a user in a first application;

detecting a first input;

in response to detecting the first input:

identifying a second application capable of accepting geographic location information; and

presenting an affordance different from the first application on at least a portion of the touch-sensitive display and suggesting opening of the second application, wherein the suggestion includes information about the first physical location;

detecting a second input at the affordance; and

in response to detecting a second input at the affordance:

opening the second application; and

populating the second application to include information based at least in part on the first physical location identifying information.

28. The electronic device of claim 27, wherein the first input corresponds to a request to open an application-switching user interface.

29. The electronic device of claim 28, wherein the affordance is presented within the application-switching user interface.

30. The electronic device of claim 29, wherein presenting the affordance includes:

Presenting, in conjunction with presenting the affordance, a representation of an application executing on the electronic device within the application-switching user interface; and

presenting the affordance in the touch-sensitive display area that is below the representation of the application.

31. The electronic device of claim 27, wherein the first input corresponds to a request to open a home screen of the electronic device.

32. The electronic device of claim 31, wherein the affordance is presented over a portion of the home screen.

33. The electronic device of claim 27, wherein the suggestion includes a textual description that is specific to a type associated with the second application.

34. The electronic device of claim 27, wherein populating the second application includes displaying a user interface object that includes information that is based at least in part on the first physical location identifying information.

35. The electronic device of claim 34, wherein the user interface object includes a textual description that informs the user that the first physical location was recently viewed in a first application.

36. The electronic device of claim 35, wherein:

the user interface object is a map displayed within the second application, and

populating the second application includes: populating the map to include an identifier of the first physical location.

37. The electronic device of claim 35, wherein the second application is presented with a virtual keyboard and the user interface object is displayed above the virtual keyboard.

38. The electronic device of claim 27, wherein identifying that the second application is capable of accepting geographic location information comprises one or more of: (i) determining that the second application includes an input-receiving field capable of accepting and processing geographic location data; (ii) determining that the second application is capable of displaying geographic location information on a map; (iii) determining that the second application is capable of using geographic location information to facilitate route guidance; and (iv) determining that the second application is capable of locating and providing transportation services using geographic location information.

39. The electronic device of claim 38, wherein:

identifying that the second application is capable of accepting geographic location information comprises: determining that the second application includes an input-receiving field capable of accepting and processing geographic location data; and is

The input-receiving field is a search box that allows searching within a map displayed within the second application.

Images