JP7236585B2 - User interface for voice messages - Google Patents

Description

(Cross reference to related applications)
This application is related to U.S. Provisional Patent Application No. 63/023,225, filed May 11, 2020, entitled "USER INTERFACE FOR AUDIO MESSAGE," the contents of which are hereby incorporated by reference in its entirety. incorporated into the specification.

TECHNICAL FIELD This disclosure relates generally to computer user interfaces, and more specifically to techniques for managing voice messages.

Users use alphanumeric characters to send instant messages to communicate with other users. Such messaging techniques typically require the user to enter characters using a keyboard or the device to transliterate the sound into characters that are subsequently sent. A user uses a mobile device to interact with an external device. Such interactions are often cumbersome and require the user to individually interact with multiple external devices.

Current techniques for managing voice messages using electronic devices are generally cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may involve multiple key presses or keystrokes. Existing techniques take longer than necessary, wasting user time and device energy. The latter problem is particularly acute in battery operated devices.

Accordingly, the present technology provides electronic devices with faster, more efficient methods and interfaces for managing voice messages. Such methods and interfaces optionally supplement or replace other methods for managing messages. Such methods and interfaces reduce the cognitive burden placed on users and create more efficient human-machine interfaces. For battery-operated computing devices, such methods and interfaces conserve power and increase intervals between battery charges. Such methods and interfaces allow sending messages that contain the tone of the message (e.g., importance, seriousness, sarcasm, sadness, hilarity) rather than just the words of the message, allowing the recipient to , allows to recognize the speaker of a message by the speaker's unique voice. Such methods and interfaces also reduce or eliminate the need to use a keyboard to provide input or use input to correct phonetic transcriptions.

Current techniques for managing the state of external devices on mobile devices are generally cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may involve multiple key presses or keystrokes. Existing techniques take longer than necessary, wasting user time and device energy. The latter problem is particularly acute in battery operated devices.

Accordingly, the present technology provides electronic devices with faster, more efficient methods and interfaces for managing external devices. Such methods and interfaces optionally control external devices, such as allowing a user to manage individual external devices or manage external devices simultaneously via a single input. complements or replaces other methods of managing Such methods and interfaces reduce the cognitive burden placed on users and create more efficient human-machine interfaces. For battery-operated computing devices, such methods and interfaces conserve power and increase intervals between battery charges.

According to some embodiments, a method is described. The method is performed on a computer system in communication with a display generation component. The method comprises, via a display generation component, a first selectable user interface object target, wherein selection of the first selectable user interface object target causes a set of external devices to generate a first audio. Initiating a process for recording and transmitting messages, wherein the set of external devices comprises a first set of one or more external devices and a second set of one or more external devices , and a first accessory control user interface object target corresponding to the first external accessory device, wherein selection of the first accessory control user interface object target changes the state of the first external accessory device. displaying a first user interface, including initiating a process of sending an instruction to do so; and, while displaying the first user interface, an input corresponding to a request to display a second user interface and displaying the second user interface in response to receiving a request to display the second user interface, the second user interface being the second selectable user interface object target, the selection of the second selectable user interface object target is performed on the one or more external devices without transmitting a second voice message to the first set of one or more external devices; Initiating a process for recording and transmitting a second voice message to the second set; and a second accessory control user interface object object corresponding to the second external accessory device. , displaying, wherein selection of the second accessory control user interface object target includes initiating a process of sending an instruction to change the state of the second external accessory device.

According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display generating component, the one or more programs Via the generation component, selection of a first selectable user interface object target, the first selectable user interface object target recording a first voice message on a set of external devices; a set of external devices, a first set of one or more external devices and a second set of one or more external devices, and a first set of one or more external devices; Including a first accessory control user interface object target corresponding to the external accessory device, wherein selection of the first accessory control user interface object target sends an instruction to change the state of the first external accessory device. displaying a first user interface comprising initiating a process; receiving input corresponding to a request to display a second user interface while displaying the first user interface; displaying a second user interface in response to receiving a request to display a user interface, the second user interface being a second selectable user interface object subject to a second The selection of the selectable user interface object target communicates a second voice message to the second set of one or more external devices without transmitting the second voice message to the first set of one or more external devices. Initiating a process for recording and sending a voice message and a second accessory control user interface object object corresponding to a second external accessory device, the second accessory control user interface object The selection of the target includes an instruction to display, including initiating a process of sending an instruction to change the state of the second external accessory device.

According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display generating component, the one or more programs Via the component, a first selectable user interface object target, wherein selection of the first selectable user interface object target records and transmits a first voice message to a set of external devices. wherein the set of external devices comprises a first set of one or more external devices and a second set of one or more external devices and a first external A process including a first accessory control user interface object target corresponding to the accessory device, wherein selection of the first accessory control user interface object target sends an instruction to change the state of the first external accessory device. displaying a first user interface, while displaying the first user interface, receiving input corresponding to a request to display a second user interface; displaying a second user interface in response to receiving a request to display an interface, the second user interface being a second selectable user interface object subject to the second selection; A selection of possible user interface object targets is to send a second voice to a second set of one or more external devices without sending a second voice message to the first set of one or more external devices. Initiating a process for recording and sending a message and a second accessory control user interface object object corresponding to a second external accessory device, the second accessory control user interface object object. Selecting the display includes initiating a process of sending an instruction to change the state of the second external accessory device.

According to some embodiments, a computer system is described. The computer system comprises one or more processors in which the computer communicates with the display generating component and a memory storing one or more programs configured to be executed by the one or more processors. , one or more programs, via a display generating component, a first selectable user interface object target, wherein selection of the first selectable user interface object target is transmitted to a set of external devices; Initiating a process for recording and transmitting a first voice message, wherein the set of external devices includes a first set of one or more external devices and a first set of one or more external devices. 2 and a first accessory control user interface object target corresponding to the first external accessory device, wherein selection of the first accessory control user interface object target corresponds to the first external accessory device displaying a first user interface, and responding to a request to display a second user interface while displaying the first user interface, including initiating a process of sending an instruction to change the state of and displaying a second user interface in response to receiving a request to display the second user interface, the second user interface being a second selectable user interface object target, the selection of the second selectable user interface object target is performed on the one or more external devices without transmitting a second voice message to the first set of one or more external devices; Initiating a process for recording and transmitting a second voice message to the second set; and a second accessory control user interface object object corresponding to the second external accessory device. , the selection of the second accessory control user interface object target includes initiating a process of sending an instruction to change the state of the second external accessory device.

According to some embodiments, a computer system is described. The computer system, via a display generating component in communication with the computer system, selects a first selectable user interface object target, the selection of the first selectable user interface object target being controlled by a set of external devices. a first set of one or more external devices and one or more external devices A second set and a first accessory control user interface object target corresponding to the first external accessory device, wherein selection of the first accessory control user interface object target corresponds to the first external accessory device. Means for displaying a first user interface comprising initiating a process of sending an instruction to change the state of the device; and displaying a second user interface while displaying the first user interface. means for receiving input corresponding to a request to display; and means for displaying the second user interface in response to receiving a request to display the second user interface, wherein the second user The interface is a second selectable user interface object target, wherein selection of the second selectable user interface object target outputs a second voice message to the first set of one or more external devices Initiating a process for recording and transmitting a second voice message to a second set of one or more external devices without transmission and a second accessory corresponding to the second external accessory device. a control user interface object target, wherein selection of the second accessory control user interface object target initiates a process of sending an instruction to change the state of the second external accessory device; Prepare.

According to some embodiments, a method is described. The method is performed on a computer system in communication with a display generation component. The method includes receiving first data corresponding to a first media; displaying a notification; and receiving, after receiving the first data, second data corresponding to a second medium, the second medium being different than the first medium. and determining that the second media satisfies a set of one or more criteria, and (1) the time the second data was received, and (2) the first data displaying, via the display generating component, a second notification having an indication of the first media until the end of the second time period, according to the determination based on the time that the first displaying a second notification having a different time period end than the second time period end.

According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display component, the one or more programs first receiving a first data corresponding to the media of, displaying, via the display generating component, a first notification having an indication of the first media until the end of the first time period; after receiving data, receiving second data corresponding to a second medium, wherein the second medium is different from the first medium; media satisfies a set of one or more criteria, and according to a determination based on (1) the time the second data was received, and (2) the time the first data was received; Displaying, via a display generating component, a second notification having the first media suggestion until the end of the second time period, the end of the first time period being the second Displaying a second notification different from the end of the time period.

According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with the display component, the one or more programs receiving first data corresponding to media; displaying, via a display generating component, a first notification having an indication of the first media until the end of a first time period; receiving second data corresponding to a second medium, wherein the second medium is different from the first medium; and display according to a determination that the media satisfies a set of one or more criteria and according to a determination based on (1) the time the second data was received; and (2) the time the first data was received. Displaying, via the generating component, a second notification having the first media suggestion until the end of the second time period, the end of the first time period being the second time period. displaying a second notification different from the end of the cycle.

According to some embodiments, a computer system is described. The computer system comprises one or more processors, with which the computer system communicates with the display component, and memory storing one or more programs configured to be executed by the one or more processors. wherein the one or more programs receive first data corresponding to the first media and, via the display generation component, have suggestions of the first media until the end of the first time period After displaying the first notification and receiving the first data, receiving second data corresponding to the second media, wherein the second media is different from the first media; 2 data and determining that the second media satisfies a set of one or more criteria; and (1) the time the second data was received, and (2) the first displaying a second notification having an indication of the first media until the end of the second time period via the display generation component according to a determination based on the time the data was received; displaying a second notification in which the end of the one time period is different than the end of the second time period.

According to some embodiments, a computer system is described. The computer system, via means for receiving first data corresponding to the first medium, and a production component in communication with the computer system, until the end of the first time period, means for displaying a first notification having suggestions; and means for, after receiving the first data, receiving second data corresponding to second media, wherein the second media according to means different from the first media and determining that the second media meets a set of one or more criteria, and the time the second data is received; and (2) the first data is means for displaying, via the display generating component, a second notification having an indication of the first media until the end of the second time period according to the received time-based determination, the means for: and means for an end of one time period to differ from an end of the second time period.

According to some embodiments, a method is described. The method is performed on an electronic device having one or more microphones, one or more speakers, and one or more display devices. The method includes receiving audio input via one or more microphones and displaying a visual indication of the audio input, a portion of the audio input via one or more display devices. the visual indication includes displaying the visual indication having a first set of one or more colors, according to the determination that the visual indication corresponds to the audio input sent to the one or more devices for displaying. , the audio input does not include instructions for displaying, via one or more display devices, transmitting any portion of the audio input to another device, the visual instructions may include one or more and displaying a second set of one or more colors different from the first set of colors.

According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer readable storage medium configured to be executed by one or more processors of an electronic device having one or more microphones, one or more speakers, and one or more display devices. , one or more of the programs receiving audio input via one or more microphones and displaying visual indications of the audio input, a portion of the audio input being one The visual indication has a first set of one or more colors, according to a determination that it corresponds to audio input transmitted to one or more devices for display via one or more display devices. Visual, including displaying instructions, according to a determination that the audio input does not include instructions to transmit any portion of the audio input to another device for display via one or more display devices. The indication includes displaying a visual indication having a second set of one or more colors that is different than the first set of one or more colors. do, including instructions.

According to some embodiments, a temporary computer-readable storage medium is described. A temporary computer-readable storage medium is one or more processors configured to be executed by one or more processors of an electronic device having one or more microphones, one or more speakers, and one or more display devices. storing a program, one or more of which receive audio input via one or more microphones and display a visual indication of the audio input, wherein a portion of the audio input is The visual indication has a first set of one or more colors, according to a determination that it corresponds to voice input transmitted to one or more devices for display via the above display device. and according to a determination that the audio input does not include instructions to transmit any portion of the audio input to another device for display via one or more display devices. displays a visual indication of voice input, including displaying a visual indication having a second set of one or more colors different from the first set of one or more colors , including instructions.

According to some embodiments, an electronic device is described. The electronic device includes one or more microphones, one or more speakers, one or more display devices, one or more processors, and one or more processors configured to be executed by the one or more processors. one or more of which is to receive audio input via one or more microphones and display a visual indication of the audio input; corresponds to audio input transmitted to one or more devices for display via one or more display devices, the visual indication has a first set of one or more colors comprising displaying visual instructions, according to a determination that the voice input does not include instructions to transmit any portion of the voice input to another device for display via one or more display devices; The visual indication includes displaying a visual indication having a second set of one or more colors different from the first set of one or more colors. Contain instructions to display instructions.

According to some embodiments, an electronic device is described. The electronic device comprises one or more microphones, one or more speakers, one or more display devices, means for receiving audio input via the one or more microphones, and a visual representation of the audio input. Means for displaying instructions, the visual instructions according to a determination that a portion of the audio input corresponds to the audio input sent to one or more devices for display via one or more display devices. includes displaying a visual indication having a first set of one or more colors, and any portion of the voice input that the voice input displays via one or more display devices. the visual instruction to display a second set of one or more colors that is different from the first set of one or more colors. and means for containing.

According to some embodiments, a method is described. The method is performed in a computer system that communicates with a display generation component and one or more input devices. The method includes displaying, via a display generating component, a first state indicating that a plurality of accessory devices of a first accessory type are in a first state, wherein: The plurality of accessory devices are a first accessory device of a first accessory type and in a first state and a second accessory device of the first accessory type and in a first state, wherein displaying a first state including a second accessory device different than the first accessory device; detecting a gesture while displaying the first state indicator; in response to detecting the first accessory device and the second accessory device in a different state in accordance with determining that the gesture is a first type gesture and directed at the first state indicator. causing a change to a second state, and upon determining that the gesture is a second type gesture different from the first type gesture and directed at the first state indicator, via the display generation component, Displaying a first accessory control user interface object target corresponding to one accessory device, wherein selection of the first accessory control user interface object target causes a second accessory device to change to a second state. and displaying a first accessory control user interface object target that causes the first accessory device to change to the second state without the first accessory device.

According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system, the computer system comprising a display generation component and one In communication with the input devices, the one or more programs display, via a display generation component, a first state indicating that the plurality of accessory devices of the first accessory type are in the first state. wherein the plurality of accessory devices of the first accessory type are a first accessory device of the first accessory type and in a first state; and a first accessory device of the first accessory type and in a first state displaying a first state, including one in which the second accessory device is different than the first accessory device, and while displaying the first state indicator , detects the gesture, and in response to detecting the gesture, activates the first accessory device and the second accessory device in accordance with determining that the gesture is a first type gesture and directed at the first state indicator. causing a change to a second state different from the first state and generating a display according to determining that the gesture is a second type gesture different from the first type gesture and directed at the first state indicator; displaying, via a component, a first accessory control user interface object target corresponding to the first accessory device, wherein selection of the first accessory control user interface object target corresponds to the second accessory device; Instructions are included for displaying a first accessory control user interface object target that causes the first accessory device to change to the second state without causing the second state to change.

According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system, which includes a display generation component and one or more and the one or more programs displaying, via a display generating component, a first state indicating that the plurality of accessory devices of the first accessory type are in the first state. and wherein the plurality of accessory devices of the first accessory type are the first accessory device of the first accessory type and in the first state and the first accessory device of the first accessory type and in the first state while displaying a first state and displaying a first state indicator, including a second accessory device, wherein the second accessory device is different than the first accessory device; Detecting the gesture and, in response to detecting the gesture, activating the first accessory device and the second accessory device in accordance with determining that the gesture is a gesture of the first type and directed at the first state indicator. causing a change to a second state different from the one state, according to determining that the gesture is a second type gesture different from the first type gesture and directed at the first state indicator; displaying, via an element, a first accessory control user interface object target corresponding to the first accessory device, wherein selection of the first accessory control user interface object target causes the second accessory device to become the second accessory device; Instructions are included to display a first accessory control user interface object target that causes the first accessory device to change to a second state without causing the first accessory device to change to a second state.

According to some embodiments, a computer system is described. The computer system comprises one or more processors, with which the computer system communicates with display generating components and one or more input devices, and one or more configured to be executed by the one or more processors. wherein the one or more programs, via the display generation component, generate a first state indicating that the plurality of accessory devices of the first accessory type are in the first state. displaying, the plurality of accessory devices of a first accessory type are a first accessory device of the first accessory type and in a first state; displaying a first state, including a second accessory device in a state wherein the second accessory device is different than the first accessory device and displaying a first state indicator the first accessory device and the second accessory in response to detecting the gesture and determining, in response to detecting the gesture, that the gesture is a gesture of a first type and directed to a first state indicator; causing the device to change to a second state different from the first state, according to determining that the gesture is a second type gesture different from the first type gesture and directed to the first state indicator; displaying, via a display generating component, a first accessory control user interface object target corresponding to the first accessory device, wherein selection of the first accessory control user interface object target corresponds to the second accessory Instructions are included for displaying a first accessory control user interface object target that causes the first accessory device to change to the second state without causing the device to change to the second state.

According to some embodiments, a computer system is described. The computer system, via a display generating component, means for displaying a first state indicating that a plurality of accessory devices of a first accessory type are in a first state, the first accessory The plurality of accessory devices of type are a first accessory device in a first state and a second accessory device of the first accessory type in a first state. means for detecting a gesture while displaying the first status indicator; and responsive to detecting the gesture. and placing the first accessory device and the second accessory device in a second state different from the first state in accordance with determining that the gesture is a first type gesture and directed at the first state indicator. and according to a determination that the gesture is a second type gesture different from the first type gesture and directed at the first status indicator, via the display generating component, the first accessory device means for displaying a first accessory control user interface object target corresponding to a second accessory device without causing a second accessory device to change to a second state. , and means for causing the first accessory device to change to the second state.

According to some embodiments, a method is described. The method is performed in a computer system in communication with a display generation component and one or more input devices. The method includes, via a display generating component, a first representation of a first media message at a first location and a second representation of a second media message at a second location different from the first location. displaying a media user interface including the representation of 2; and responsive to displaying the media user interface, playing the first media message without being based on user input corresponding to the media user interface; In response to detecting completion of playback of the first media message, displaying a second representation of the second media message without displaying a representation corresponding to the first media message via the display generating component. displaying.

According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system, the computer system comprising a display generation component and one Communicating with the input devices, the one or more programs, via the display generating component, at a first location, a first representation of a first media message and a second representation different from the first location. displaying a media user interface including a second representation of a second media message at a location of and, in response to displaying the media user interface, the first and, in response to detecting completion of playback of the first media message, playing a second media message without displaying a representation corresponding to the first media message via the display generation component. Instructions are included to display a second representation of the message.

According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system, which includes a display generation component and one or more and one or more programs, via a display generating component, at a first location, a first representation of a first media message and a second representation different from the first location. at a location, displaying a media user interface including a second representation of a second media message; Playing the media message and, in response to detecting completion of playing the first media message, producing a second media message without displaying a representation corresponding to the first media message via the display generating component. includes instructions for displaying a second representation of .

According to some embodiments, a computer system is described. The computer system comprises one or more processors, with which the computer system communicates with display generating components and one or more input devices, and one or more configured to be executed by the one or more processors. wherein the one or more programs are a first representation of the first media message at a first location and different from the first location via the display generating component displaying, at a second location, a media user interface including a second representation of a second media message; and in response to displaying the media user interface, not based on user input corresponding to the media user interface; playing a first media message; and, in response to detecting completion of playing the first media message, a second media message without displaying a representation corresponding to the first media message via the display generation component; display a second representation of the media message of the.

According to some embodiments, a computer system is described. A computer system communicates with a display generation component and one or more input devices. A computer system, via a display generating component, outputs a first representation of a first media message at a first location and a second representation of a second media message at a second location different from the first location. means for displaying a media user interface including the representation of 2; and in response to displaying the media user interface, for playing a first media message without being based on user input corresponding to the media user interface. and, in response to detecting completion of playback of the first media message, generating the second media message without displaying a representation corresponding to the first media message via the display generation component. and means for displaying a representation of 2.

Executable instructions for performing these functions are optionally contained within a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. Executable instructions for performing these functions are optionally contained within a temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors.

Thus, devices are provided with faster, more efficient methods and interfaces for managing voice messages, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can complement or replace other methods for managing voice messages.

For a better understanding of the various described embodiments, reference should be made to the following detailed description in conjunction with the following drawings, where like reference numerals refer to the following figures. Refers to corresponding parts throughout.

1 is a block diagram illustrating a portable multifunction device having a touch sensitive display according to some embodiments; FIG.

FIG. 4 is a block diagram illustrating exemplary components for event processing according to some embodiments;

1 illustrates a portable multifunction device with a touch screen according to some embodiments;

1 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments; FIG.

4 illustrates an exemplary user interface of a menu of applications on a portable multifunction device according to some embodiments;

4 illustrates an exemplary user interface of a multifunction device having a touch sensitive surface separate from the display according to some embodiments;

1 illustrates a personal electronic device according to some embodiments;

1 is a block diagram illustrating a personal electronic device according to some embodiments; FIG.

4 illustrates exemplary components of a personal electronic device having a touch sensitive display and an intensity sensor according to some embodiments; 4 illustrates exemplary components of a personal electronic device having a touch sensitive display and an intensity sensor according to some embodiments;

1 illustrates exemplary components and user interface of a personal electronic device according to some embodiments; 1 illustrates exemplary components and user interface of a personal electronic device according to some embodiments; 1 illustrates exemplary components and user interface of a personal electronic device according to some embodiments; 1 illustrates exemplary components and user interface of a personal electronic device according to some embodiments;

1 illustrates an electronic device according to some embodiments;

1 is a block diagram illustrating an electronic device according to some embodiments; FIG.

FIG. 2 illustrates a set of physical structures and exemplary devices, according to some embodiments;

4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments; 4 illustrates an exemplary user interface for sending a voice message, according to some embodiments;

FIG. 4 is a flow diagram illustrating a method for sending a voice message, according to some embodiments; FIG. 4 is a flow diagram illustrating a method for sending a voice message, according to some embodiments;

4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments. 4 illustrates an exemplary user interface for displaying notifications, according to some embodiments.

FIG. 4 is a flow diagram illustrating a method for displaying notifications, according to some embodiments.

4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments; 4 illustrates an exemplary user interface for displaying visual instructions, according to some embodiments;

FIG. 4 is a flow diagram illustrating a method for displaying visual instructions, according to some embodiments;

4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments. 4 illustrates an exemplary user interface for managing external devices, according to some embodiments.

FIG. 4 is a flow diagram illustrating a method for managing external devices, according to some embodiments;

4 illustrates an exemplary user interface for playing a voice message, according to some embodiments; 4 illustrates an exemplary user interface for playing a voice message, according to some embodiments; 4 illustrates an exemplary user interface for playing a voice message, according to some embodiments; 4 illustrates an exemplary user interface for playing a voice message, according to some embodiments; 4 illustrates an exemplary user interface for playing a voice message, according to some embodiments; 4 illustrates an exemplary user interface for playing a voice message, according to some embodiments;

FIG. 4 is a flow diagram illustrating a method for playing a voice message, according to some embodiments; Description of the embodiment

The following description sets forth exemplary methods, parameters, and the like. It should be recognized, however, that such description is not intended as a limitation on the scope of the present disclosure, but is instead provided as a description of exemplary embodiments.

What is needed is an electronic device that provides an efficient method and interface for managing voice messages. Techniques for managing voice messages increase device efficiency (e.g., reduce unnecessary displays) and increase information provided around external devices (e.g., display source of voice messages). ), increasing privacy (by providing visual feedback to the user of the state of the device that received the audio input). Such techniques can reduce the cognitive burden of users using voice messaging, thereby increasing productivity. In addition, such techniques can reduce processor and battery power otherwise wasted on redundant user input.

1A-1B, 2, 3, 4A-4B, and 5A-5J below describe exemplary devices for implementing techniques for sending and receiving voice messages. FIG. 6 is a diagram illustrating a set of physical structures and exemplary devices for implementing techniques for sending and receiving voice messages, according to some embodiments.

7A-7AH illustrate exemplary user interfaces for sending voice messages, according to some embodiments. 8A-8B are flow diagrams illustrating methods for sending voice messages, according to some embodiments. The user interfaces of Figures 7A-7AH are used to illustrate the processes described below, including the processes of Figures 8A-8B.

9A-9X illustrate exemplary user interfaces for displaying notifications, according to some embodiments. FIG. 10 is a flow diagram illustrating a method for displaying notifications, according to some embodiments. The user interfaces of FIGS. 9A-9X are used to illustrate the processes described below, including the process of FIG.

11A-11R show exemplary user interfaces for displaying visual instructions. FIG. 12 is a flow diagram illustrating a method for displaying visual indications. The user interfaces of FIGS. 11A-11R are used to illustrate the processes described below, including the process of FIG.

13A-13N illustrate exemplary user interfaces for managing device status. FIG. 14 is a flow diagram illustrating a method for managing device state. The user interfaces of FIGS. 13A-13N are used to illustrate the processes described below, including the process of FIG.

15A-15F illustrate exemplary user interfaces for playing voice messages, according to some embodiments. FIG. 16 is a flow diagram illustrating a method for playing a voice message, according to some embodiments. The user interfaces of FIGS. 15A-15F are used to illustrate the processes described below, including the process of FIG.

Although the following description uses terms such as "first," "second," etc. to describe various elements, these elements should not be limited by those terms. These terms are only used to distinguish one element from another. For example, a first touch could be referred to as a second touch, and similarly a second touch could be referred to as a first touch, without departing from the scope of various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

The terminology used in the description of the various embodiments provided herein is for the purpose of describing particular embodiments only and is not intended to be limiting. In the description of the various embodiments described and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural as well, unless the context clearly dictates otherwise. Also, as used herein, the term "and/or" is understood to refer to and include any and all possible combinations of one or more of the associated listed items. want to be The terms "includes," "including," "comprises," and/or "comprising," as used herein, refer to the features mentioned, specifying the presence of an integer, step, operation, element, and/or component, but specifying the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof It will be further understood that no exclusions are made.

The term "if" is optionally used, depending on the context, "when" or "upon" or "in "response to determining" or "in response to detecting". Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are optional. Literally, depending on the context, "upon determining" or "in response to determining", or "when detecting [state or event] “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event]”.

Embodiments of electronic devices, user interfaces for such devices, and associated processes of using such devices are described. In some embodiments, the device is a portable communication device such as a mobile phone that also includes PDA functionality and/or other functionality such as music player functionality. Exemplary embodiments of portable multifunction devices include those manufactured by Apple Inc. of Cupertino, Calif. including, but not limited to, iPhone®, iPod Touch®, and iPad® devices from Apple Inc. Optionally, other portable electronic devices such as laptops or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touch pads) are also used. It should also be appreciated that in some embodiments the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (eg, a touchscreen display and/or touchpad).

The following discussion describes electronic devices that include displays and touch-sensitive surfaces. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, mouse and/or joystick.

Devices are typically used in drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, e-mail applications, instant messaging applications, training Various applications such as one or more of a support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application are supported.

Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and/or changed from application to application and/or within each application. In this way, the common physical architecture of the device (such as the touch sensitive surface) optionally supports various applications with a user interface that is intuitive and transparent to the user.

Attention is now directed to embodiments of portable devices with touch-sensitive displays. FIG. 1A is a block diagram illustrating a portable multifunction device 100 having a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience and is known or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110 , speaker 111 , microphone 113 , input/output (I/O) subsystem 106 , other input control devices 116 , and external ports 124 . Device 100 optionally includes one or more photosensors 164 . Device 100 optionally includes one or more contact strength sensors 165 that detect the strength of contact on device 100 (eg, a touch-sensitive surface such as touch-sensitive display system 112 of device 100). Device 100 optionally generates tactile output on device 100 (eg, on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). It includes one or more tactile output generators 167 . These components optionally communicate via one or more communication buses or signal lines 103 .

As used herein and in the claims, the term "strength" of contact on the touch-sensitive surface refers to the force or pressure (force per unit area) of contact (e.g., finger contact) on the touch-sensitive surface, or Refers to a proxy for the force or pressure of contact on a touch-sensitive surface. The intensity of contact has a range of values including at least four distinct values, and more typically including several hundred (eg, at least 256) distinct values. The strength of contact is optionally determined (or measured) using various techniques and various sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensitive surface are optionally used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine the estimated force of the contact. Similarly, the stylus' pressure-sensitive tip is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and/or change in the contact area detected on the touch-sensitive surface, the capacitance and/or change in the touch-sensitive surface proximate to the contact, and/or the resistance and/or change in the touch-sensitive surface proximate to the contact. /or its variation is optionally used as a substitute for contact force or pressure on the touch-sensitive surface. In some implementations, the alternate measure for contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is in units corresponding to the alternate measure). ). In some implementations, the surrogate measurement for contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether a strength threshold has been exceeded ( For example, the intensity threshold is the pressure threshold measured in units of pressure). to display affordances (e.g., on a touch-sensitive display) and/or measure user input (e.g., on a touch-sensitive display, touch-sensitive surface, or Not normally accessible by the user on reduced size devices with limited area for receiving (via physical/mechanical controls such as knobs or buttons) It allows the user access to additional device features that are available.

As used herein and in the claims, the term "tactile output" refers to the physical displacement of the device relative to its previous position, which is to be detected by the user in the user's sense of touch. Refers to the physical displacement of a component (eg, touch-sensitive surface) relative to another component of the device (eg, housing) or the displacement of a component relative to the device's center of mass. For example, in situations where a device or device component is in contact with a touch-sensitive surface of a user (e.g., fingers, palm, or other part of the user's hand), tactile sensations generated by physical displacement The output will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or device components. For example, movement of a touch-sensitive surface (eg, touch-sensitive display or trackpad) is optionally interpreted by the user as a "down-click" or "up-click" of a physical actuator button. In some cases, a user can "down-click" or "up-click" even when there is no movement of the physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's motion. I feel the tactile sensation such as As another example, movement of the touch-sensitive surface is optionally interpreted or perceived by the user as "roughness" of the touch-sensitive surface, even if there is no change in the smoothness of the touch-sensitive surface. be. Such a user's interpretation of touch depends on the user's personal sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "upclick", "downclick", "roughness"), the generated tactile A sensory output corresponds to the physical displacement of a device, or a component of a device, that produces the described sensory perception of a typical (or average) user.

Device 100 is only one example of a portable multifunction device; device 100 optionally has more or fewer components than those shown; It should be understood that there are combinations of more than one component, or optionally different configurations or arrangements of those components. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and/or application specific integrated circuits.

Memory 102 optionally includes high speed random access memory and optionally non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. include. Memory controller 122 optionally controls access to memory 102 by other components of device 100 .

Peripheral interface 118 may be used to couple input and output peripherals of the device to CPU 120 and memory 102 . One or more processors 120 operate or execute various software programs and/or instruction sets stored in memory 102 to perform various functions and process data for device 100 . In some embodiments, peripherals interface 118 , CPU 120 and memory controller 122 are optionally implemented on a single chip, such as chip 104 . In some other embodiments, they are optionally implemented on separate chips.

Radio frequency (RF) circuitry 108 transmits and receives RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to electromagnetic signals or vice versa to communicate with communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes well-known circuits for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, Tuner, one or more oscillators, digital signal processor, CODEC chipset, subscriber identity module (SIM) card, memory, and the like. The RF circuit 108 is optionally connected to the Internet, also known as the World Wide Web (WWW), an intranet, and/or a cellular telephone network, a wireless local area network (LAN), and/or a metropolitan area network. (metropolitan area network, MAN) and other devices via wireless communication. RF circuitry 108 optionally includes well-known circuitry for detecting a near field communication (NFC) field, such as by a short-range communication radio. Wireless communication optionally includes, but is not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high speed downlink high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), evolution, data-only, EV-DO, HSPA, HSPA+, dual cell HSPA (Dual-Cell HSPA, DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiplex code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth®, Bluetooth Low Energy (BTLE®), Wireless Fidelity, Wi-Fi (eg, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and/or IEEE 802.11ac), voice over Internet Protocol, VoIP), Wi-MAX®, protocols for email (e.g. Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g. extensible messaging and presence protocol XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and multiple communication standards, protocols, including Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this specification; and techniques.

Audio circuitry 110 , speaker 111 , and microphone 113 provide an audio interface between the user and device 100 . Audio circuitry 110 receives audio data from peripherals interface 118 , converts the audio data to electrical signals, and transmits the electrical signals to speaker 111 . The speaker 111 converts the electrical signal into human audible sound waves. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113 . Audio circuitry 110 converts the electrical signals to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data is optionally retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118 . In some embodiments, audio circuitry 110 also includes a headset jack (eg, 212 in FIG. 2). A headset jack is provided between the audio circuitry 110 and a detachable audio input/output peripheral, such as an output-only headphone or a headset with both an output (e.g., monaural or binaural headphones) and an input (e.g., a microphone). provides an interface for

I/O subsystem 106 couples input/output peripherals on device 100 , such as touch screen 112 and other input control devices 116 , to peripherals interface 118 . I/O subsystem 106 optionally includes one for display controller 156, light sensor controller 158, depth camera controller 169, intensity sensor controller 159, tactile feedback controller 161, and other input or control devices. The above input controller 160 is included. One or more input controllers 160 receive/send electrical signals to/from other input control devices 116 . Other input control devices 116 optionally include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, the input controller(s) 160 is optionally coupled (or not bound to either). One or more buttons (eg, 208 in FIG. 2) optionally include up and down buttons for volume control of speaker 111 and/or microphone 113 . The one or more buttons optionally include push buttons (eg, 206 in FIG. 2).

U.S. Patent Application No. 11/322,549, "Unlocking a Device by Performing Gestures on an Unlock Image," filed Dec. 23, 2005, which is incorporated herein by reference in its entirety, U.S. Patent No. 7, 657,849, a quick press of the push button optionally disengages the lock on the touch screen 112, or optionally using gestures on the touch screen. Start the process of unlocking the device. A longer press of a push button (eg, 206) optionally turns power to device 100 on or off. The functionality of one or more of the buttons is optionally customizable by the user. Touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

Touch-sensitive display 112 provides an input and output interface between the device and the user. Display controller 156 receives electrical signals from touchscreen 112 and/or transmits electrical signals to touchscreen 112 . Touch screen 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

Touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and/or tactile contact. Touchscreen 112 and display controller 156 (together with any associated modules and/or instruction sets in memory 102) detect contact (and any movement or interruption of contact) on touchscreen 112 and detect detected contact. Converts interaction with user interface objects (eg, one or more softkeys, icons, web pages, or images) displayed on touch screen 112 . In an exemplary embodiment, the contact point between touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 optionally uses LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, although other display technologies are also used in other embodiments. be. The touchscreen 112 and display controller 156 optionally use, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, and one or more contact points with the touchscreen 112. Any of a number of now known or later developed touch sensing technologies, including other proximity sensor arrays or other elements that determine, are used to detect contact and any movement or interruption thereof. In an exemplary embodiment, Apple Inc. of Cupertino, Calif. Projected mutual-capacitive sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple.

The touch-sensitive display in some embodiments of touch screen 112 is optionally disclosed in U.S. Pat. , 570,557 (Westerman et al.), and/or 6,677,932 (Westerman), and/or the multi-touch sensing touchpads described in US Patent Publication No. 2002/0015024 Al. are doing. However, the touchscreen 112 displays visual output from the device 100, whereas the touch-sensitive touchpad does not provide visual output.

The touch sensitive display in some embodiments of the touch screen 112 is described in (1) U.S. Patent Application Serial No. 11/381,313, "Multipoint Touch Surface Controller," filed May 2, 2006, and (2) May 2004. U.S. Patent Application Serial No. 10/840,862, "Multipoint Touchscreen," filed May 6, 2004; (4) U.S. Patent Application Serial No. 11/048,264, filed January 31, 2005; "Gestures For Touch Sensitive Input Devices"; 038,590, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices," (6) U.S. patent application Ser. User Interface", (7) U.S. Patent Application Serial No. 11/228,700, filed Sep. 16, 2005; No. 11/228,737, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," and (9) U.S. patent application Ser. Hand-Held Device" application. All of these applications are incorporated herein by reference in their entireties.

Touch screen 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user optionally contacts touch screen 112 using any suitable object or appendage, such as a stylus, finger, or the like. In some embodiments, user interfaces are designed to primarily deal with finger contacts and gestures, which are not as precise as stylus-based input due to the larger finger contact area on touch screens. there is a possibility. In some embodiments, the device converts coarse finger input into precise pointer/cursor positions or commands to perform actions desired by the user.

In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad for activating or deactivating certain functions. In some embodiments, a touchpad is a touch-sensitive area of a device that displays no visual output, unlike a touchscreen. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

Device 100 also includes a power system 162 that provides power to the various components. Power system 162 optionally includes a power management system, one or more power sources (eg, batteries, alternating current (AC)), a recharging system, power failure detection circuitry, a power converter or inverter, power status indicators (eg, lighting diodes (LEDs)) and any other component associated with the generation, management, and distribution of power within a portable device.

Device 100 also optionally includes one or more optical sensors 164 . FIG. 1A shows a photosensor coupled to photosensor controller 158 in I/O subsystem 106 . Optical sensor 164 optionally includes a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistor. Optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. Optical sensor 164, in conjunction with imaging module 143 (also called a camera module), optionally captures still images or video. In some embodiments, the optical sensors are located on the back of the device 100, as opposed to the touchscreen display 112 on the front of the device, so the touchscreen display is used for still and/or video image capture. It is effective to use it as a viewfinder for In some embodiments, the light sensor is placed on the front of the device so that the user's image is optionally obtained for video conferencing, and the user can see other video conference participants on the touch screen display. see above. In some embodiments, the position of the light sensor 164 can be changed by the user (e.g., by rotating the lens and sensor within the device housing), so that a single light sensor 164 can be used with the touch screen display. It is also used for teleconferencing and for capturing still and/or video images.

Device 100 also optionally includes one or more depth camera sensors 175 . FIG. 1A shows a depth camera sensor coupled to depth camera controller 169 within I/O subsystem 106 . Depth camera sensor 175 receives data from the environment and creates a three-dimensional model of an object (eg, face) in the scene from a viewpoint (eg, depth camera sensor). In some embodiments, in conjunction with imaging module 143 (also referred to as a camera module), depth camera sensor 175 optionally determines depth maps of various portions of the image captured by imaging module 143. used for In some embodiments, an image of the user with depth information is optionally acquired for the videoconference while the user is viewing other videoconference participants on the touch screen display. , and a depth camera sensor is placed on the front of the device 100 to capture a selfie image with depth map data. In some embodiments, depth camera sensor 175 is located on the back of the device, or on the back and front of device 100 . In some embodiments, the position of the depth camera sensor 175 can be changed by the user (eg, by rotating the lens and sensor within the device housing) so that the depth camera sensor 175 can be used in conjunction with the touch screen display. , for video conferencing as well as for capturing still and/or video images.

Device 100 also optionally includes one or more contact intensity sensors 165 . FIG. 1A shows a contact intensity sensor coupled to intensity sensor controller 159 within I/O subsystem 106 . Contact intensity sensor 165 is optionally one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch sensitive surfaces, or other intensity sensors (e.g. , sensors used to measure the force (or pressure) of contact on the touch-sensitive surface. Contact intensity sensor 165 receives contact intensity information (eg, pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed or proximate to a touch-sensitive surface (eg, touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back side of device 100 , opposite touch screen display 112 located on the front side of device 100 .

Device 100 also optionally includes one or more proximity sensors 166 . FIG. 1A shows proximity sensor 166 coupled to peripherals interface 118 . Alternatively, proximity sensor 166 is optionally coupled to input controller 160 within I/O subsystem 106 . Proximity sensor 166 is optionally disclosed in U.S. patent application Ser. 「Ｐｒｏｘｉｍｉｔｙ Ｄｅｔｅｃｔｏｒ Ｉｎ Ｈａｎｄｈｅｌｄ Ｄｅｖｉｃｅ」第１１／６２０，７０２号、「Ｕｓｉｎｇ Ａｍｂｉｅｎｔ Ｌｉｇｈｔ Ｓｅｎｓｏｒ Ｔｏ Ａｕｇｍｅｎｔ Ｐｒｏｘｉｍｉｔｙ Ｓｅｎｓｏｒ Ｏｕｔｐｕｔ」、第１１／５８６，８６２号、「Ａｕｔｏｍａｔｅｄ Ｒｅｓｐｏｎｓｅ Ｔｏ Ａｎｄ Ｓｅｎｓｉｎｇ Ｏｆ Ｕｓｅｒ Ａｃｔｉｖｉｔｙ Ｉｎ Ｐｏｒｔａｂｌｅ Ｄｅｖｉｃｅｓ」、及びSerial No. 11/638,251, "Methods And Systems For Automatic Configuration Of Peripherals," which applications are hereby incorporated by reference in their entireties. In some embodiments, the proximity sensor turns off and disables the touchscreen 112 when the multifunction device is placed close to the user's ear (e.g., when the user is making a phone call). .

Device 100 also optionally includes one or more tactile output generators 167 . FIG. 1A shows a tactile output generator coupled to haptic feedback controller 161 within I/O subsystem 106 . Tactile output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components, and/or motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other electromechanical devices that convert energy into linear motion, such as tactile output-producing components of (eg, components that convert electrical signals to tactile outputs on the device). Contact intensity sensor 165 receives tactile feedback generation instructions from haptic feedback module 133 and generates a tactile output on device 100 that can be sensed by a user of device 100 . In some embodiments, at least one tactile output generator is juxtaposed or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112) and, optionally, the touch-sensitive surface. Tactile output is generated by moving vertically (eg, in/out of the surface of device 100) or horizontally (eg, back and forth in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100 opposite touch screen display 112 located on the front of device 100 .

Device 100 also optionally includes one or more accelerometers 168 . FIG. 1A shows accelerometer 168 coupled to peripherals interface 118 . Alternatively, accelerometer 168 is optionally coupled to input controller 160 within I/O subsystem 106 . Accelerometer 168 is optionally disclosed in U.S. Patent Publication No. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices," and U.S. Patent Publication No. 20050190059, both of which are hereby incorporated by reference in their entirety. 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer". In some embodiments, information is displayed on the touch screen display in portrait or landscape view based on analysis of data received from one or more accelerometers. In addition to the accelerometer 168, the device 100 includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for obtaining information about the location and orientation (e.g., longitudinal or lateral) of the device 100. optionally included.

In some embodiments, the software components stored in memory 102 include operating system 126, communication module (or instruction set) 128, touch/motion module (or instruction set) 130, graphics module (or instruction set). ) 132 , a text input module (or instruction set) 134 , a Global Positioning System (GPS) module (or instruction set) 135 , and an application (or instruction set) 136 . Additionally, in some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) stores device/global internal state 157, as shown in FIGS. 1A and 3. FIG. The device/global internal state 157 is an active application state that indicates which, if any, applications are currently active, and which applications, views, or other information may be displayed in various areas of the touchscreen display 112 . display state indicating occupancy; sensor state, which includes information obtained from the device's various sensors and input control device 116; and positional information regarding the location and/or orientation of the device.

operating system 126 (e.g., Darwin®, RTXC®, LINUX®, UNIX®, OS X®, iOS®, WINDOWS®, or Embedded operating systems such as VxWorks®) include various software components and/or drivers that control and manage common system tasks (e.g., memory management, storage device control, power management, etc.). , facilitates communication between various hardware and software components.

Communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by RF circuitry 108 and/or external ports 124 . include. The external port 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) connects to other devices directly or indirectly through a network (e.g., Internet, wireless LAN, etc.). is adapted to bind to In some embodiments, the external port is a multi-pin (e.g., , 30 pin) connector.

Contact/motion module 130 optionally detects contact with touchscreen 112 (in conjunction with display controller 156) and other touch-sensitive devices (eg, touchpads or physical click wheels). The contact/motion module 130 determines whether contact has occurred (e.g., detecting a finger-down event), the strength of the contact (e.g., the force or pressure of the contact, or alternatives to the force or pressure of the contact). determining whether there is movement of the contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more events of dragging a finger); Contains various software components for performing various operations related to detecting contact, such as determining whether contact has stopped (e.g., detecting a finger-up event or interruption of contact) . Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the contact point, represented by the set of contact data, optionally includes speed (magnitude), velocity (magnitude and direction), and/or acceleration (magnitude and direction) of the contact point. and/or change of direction). These operations are optionally applied to single contacts (eg, single-finger contacts) or multiple simultaneous contacts (eg, “multi-touch”/multiple-finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect contact on the touchpad.

In some embodiments, the contact/motion module 130 is configured to: One or more sets of intensity thresholds are used. In some embodiments, at least one subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds are determined by the physical hardware of device 100, rather than being determined by activation thresholds of specific physical actuators). can be adjusted without changing the wear). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to any of a wide range of predefined thresholds without changing the trackpad or touchscreen display hardware. Additionally, in some implementations, the user of the device is provided with software settings that adjust one or more of a set of intensity thresholds (e.g., individual intensity by adjusting thresholds and/or adjusting multiple intensity thresholds at once).

Contact/motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different movements, timing, and/or strength of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture detects a finger down event followed by a finger up (lift off) event at the same (or substantially the same) position as the finger down event (e.g., icon position). Including detecting. As another example, detecting a finger swipe gesture on a touch-sensitive surface involves detecting a finger down event followed by one or more finger drag events followed by a finger up (lift off) event. including detecting

Graphics module 132 may be used on touch screen 112 or other display to include components that change the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of displayed graphics. includes a variety of known software components for rendering and displaying graphics. As used herein, the term "graphics" includes, but is not limited to, text, web pages, icons (such as user interface objects including softkeys), digital images, videos, animations, etc. Contains any object that can be displayed.

In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives, from an application or the like, one or more codes specifying the graphics to be displayed, along with coordinate data and other graphics characteristic data as appropriate, and then outputs to the display controller 156. Generates image data for the screen to be displayed.

Haptic feedback module 133 is used by tactile output generator(s) 167 to generate tactile output at one or more locations on device 100 in response to user interaction with device 100. Contains various software components for generating instructions.

Text input module 134 is optionally a component of graphics module 132 and is used for various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input). ) to provide a soft keyboard for entering text.

GPS module 135 can determine the location of the device and use this information in various applications (e.g., to phone 138 for use in location-based dialing, camera 143 as photo/video metadata). to and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map/navigation widgets).

Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof.
- the contacts module 137 (sometimes called an address book or contact list);
a telephone module 138;
a teleconferencing module 139,
- email client module 140;
an instant messaging (IM) module 141;
● training support module 142,
- a camera module 143 for still and/or video images;
an image management module 144;
● video player module,
● music player module,
the browser module 147,
a calendar module 148;
- Optionally weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other user-obtained and user-created widgets a widget module 149 including one or more of 149-6;
- a widget creation module 150 for creating user-created widgets 149-6;
a search module 151,
- a video and music player module 152 that integrates a video player module and a music player module;
● memo module 153,
- map module 154 and/or
• Online video module 155;

Examples of other applications 136 that are optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA enabled applications, cryptography, digital writing. Rights management, speech recognition, and speech replication.

In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, contact module 137 optionally adds name(s) to the address book. remove name(s) from address book, phone number(s), email address(es), physical address(es), or other information to name associating, associating images with names, sorting and sorting names, providing a phone number or email address to initiate communication by phone 138, videoconferencing module 139, email 140, or IM 141 and/or or to manage an address book or contact list (eg, stored within application internal state 192 of contact module 137 in memory 102 or memory 370).

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, telephone module 138 optionally includes , enter a character sequence corresponding to a telephone number, access one or more telephone numbers in the contacts module 137, modify the entered telephone number, dial the respective telephone number, conduct a conversation, and end the conversation. Used to stop a call or hang up when As previously mentioned, wireless communication optionally uses any of a number of communication standards, protocols, and techniques.

RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, light sensor 164, light sensor controller 158, touch/motion module 130, graphics module 132, text input module 134, contact module 137. , and telephony module 138, the videoconferencing module 139 provides executable instructions for initiating, conducting, and terminating a videoconference between the user and one or more other participants according to user instructions. include.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, email client module 140 composes and sends emails in response to user commands. , including executable instructions for receiving and managing. In conjunction with the image management module 144, the email client module 140 greatly facilitates the creation and sending of emails with still or video images captured by the camera module 143.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, instant messaging module 141 allows the input of character sequences corresponding to instant messages, previously modification of entered characters, transmission of respective instant messages (e.g. Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephony-based instant messages, or Internet (using XMPP, SIMPLE, or IMPS for instant messaging based), receiving instant messages, and viewing received instant messages. In some embodiments, instant messages sent and/or received are optionally graphics, photos, audio files, video files supported by MMS and/or Enhanced Messaging Service (EMS). , and/or other attachments. As used herein, "instant messaging" refers to telephony-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., using XMPP, SIMPLE, or IMPS). It refers to both

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, training support module 142: Create workouts (e.g., with time, distance, and/or calorie burn goals), communicate with training sensors (sports devices), receive training sensor data, and calibrate sensors used to monitor training , selecting and playing music for training, and displaying, storing, and transmitting training data.

In conjunction with touch screen 112, display controller 156, light sensor(s) 164, light sensor controller 158, touch/motion module 130, graphics module 132, and image management module 144, camera module 143 captures still images or Contains executable instructions for capturing video (including a video stream) and storing it in memory 102 , modifying the characteristics of a still image or video, or deleting a still image or video from memory 102 .

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 arranges, modifies (or modifies) still and/or video images. edit), or other manipulation, labeling, deletion, presentation (eg, in a digital slideshow or album), and storage.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, browser module 147 provides web pages or portions thereof and links to web pages. Contains executable instructions for browsing the Internet according to user instructions, including retrieving, linking, receiving, and displaying attachments and other files.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 can and executable instructions for creating, displaying, modifying, and storing calendars and data associated with calendars (eg, calendar items, to-do lists, etc.) in accordance with the .

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is optionally downloaded and used by the user. (eg weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or mini-applications created by users (eg , user created widget 149-6). In some embodiments, widgets include HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, widgets include XML (Extensible Markup Language) files and JavaScript files (eg, Yahoo! widgets).

Widget creator module 150, in conjunction with RF circuit 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, and browser module 147, optionally creates widgets. (eg, widgetize user-specified portions of web pages).

In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, search module 151 provides one or more search criteria (e.g., one or more user executable instructions for searching for text, music, sounds, images, videos, and/or other files in memory 102 that match a specified search term).

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 can play MP3 or AAC files. Executable instructions that enable a user to download and play pre-recorded music and other sound files stored in one or more file formats such as, and videos (e.g., on touch screen 112 or external port 124 contains executable instructions for displaying, presenting, or otherwise reproducing (on an external display connected via). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, notes module 153 is an execution module for creating and managing notes, to-do lists, etc., according to user instructions. Contains possible instructions.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 optionally: Receive, display, modify, and, in accordance with user instructions, maps and data associated with maps (e.g., data regarding driving directions, stores and other points of interest at or near a particular location, and other location-based data) used to remember.

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online Video module 155 allows users to access specific online videos, browse specific online videos, receive (eg, by streaming and/or downloading), and play (eg, on a touch screen or connected via external port 124). on an external display), sending e-mails with links to specific online videos; contains instructions that allow for other management of online video in one or more file formats, such as H.264. In some embodiments, instant messaging module 141 is used rather than email client module 140 to send links to specific online videos. Additional discussion of online video applications can be found in U.S. Provisional Patent Application No. 60/936,562, entitled "Portable Multifunction Device," filed Jun. 20, 2007, the contents of which are hereby incorporated by reference in their entirety. ，Ｍｅｔｈｏｄ，ａｎｄ Ｇｒａｐｈｉｃａｌ Ｕｓｅｒ Ｉｎｔｅｒｆａｃｅ ｆｏｒ Ｐｌａｙｉｎｇ Ｏｎｌｉｎｅ Ｖｉｄｅｏｓ」、及び２００７年１２月３１日出願の米国特許出願第１１／９６８，０６７号、「Ｐｏｒｔａｂｌｅ Ｍｕｌｔｉｆｕｎｃｔｉｏｎ Ｄｅｖｉｃｅ，Ｍｅｔｈｏｄ，ａｎｄ Ｇｒａｐｈｉｃａｌ Ｕｓｅｒ Ｉｎｔｅｒｆａｃｅ ｆｏｒ Ｐｌａｙｉｎｇ Ｏｎｌｉｎｅ Ｖｉｄｅｏｓ」においてcan see.

Each of the modules and applications identified above may be implemented to perform one or more of the functions described above and the methods described herein (e.g., the computer-implemented methods and other information processing methods described herein). Corresponds to the set of possible instructions. These modules (eg, instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are optionally combined in various embodiments. modified or otherwise reconfigured. For example, a video player module is optionally combined with a music player module into a single module (eg, video and music player module 152 of FIG. 1A). In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. In addition, memory 102 optionally stores additional modules and data structures not described above.

In some embodiments, device 100 is a device in which operation of a predefined set of functions is performed solely through a touchscreen and/or touchpad. number of physical input control devices (push buttons, dials, etc.) is reduced.

A default set of functions performed only through a touchscreen and/or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad navigates device 100 to the main menu, home menu, or root menu from any user interface displayed on device 100 when touched by a user. In such embodiments, the "menu button" is implemented using a touchpad. In some other embodiments, the menu buttons are physical push buttons or other physical input control devices rather than touchpads.

FIG. 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (eg, within operating system 126) and respective applications 136-1 (eg, the aforementioned applications 137-151, 155, any of 380-390).

Event sorter 170 receives event information and determines the application 136-1 to which the event information is transmitted and the application view 191 of application 136-1. Event sorter 170 includes event monitor 171 and event dispatcher module 174 . In some embodiments, application 136-1 includes application internal state 192 that indicates the current application view(s) displayed on touch-sensitive display 112 when the application is active or running. In some embodiments, device/global internal state 157 is used by event sorter 170 to determine which application(s) is currently active, and application internal state 192 is used to determine when event information is delivered. used by event sorter 170 to determine the application view 191 to be displayed.

In some embodiments, application internal state 192 indicates resume information to be used when application 136-1 resumes execution, information being displayed by application 136-1, or a user ready to display information. Additional features such as interface state information, one or more of a state queue that allows the user to return to a previous state or view of application 136-1, and a redo/undo queue of previous actions taken by the user. Contains information.

Event monitor 171 receives event information from peripherals interface 118 . Event information includes information about sub-events (eg, a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I/O subsystem 106 or sensors such as proximity sensor 166, accelerometer(s) 168, and/or microphone 113 (via audio circuitry 110). . Information that peripherals interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (eg, receipt of input above a predetermined noise threshold and/or exceeding a predetermined duration).

In some embodiments, event sorter 170 also includes hit view determination module 172 and/or active event recognizer determination module 173 .

Hit view determination module 172 provides software procedures for determining where a sub-event occurred within one or more views when touch-sensitive display 112 displays more than one view. Views are made up of controls and other elements that the user can see on the display.

Another aspect of the user interface associated with an application is the set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the respective application) in which the touch is detected optionally corresponds to a program level within the application's program hierarchy or view hierarchy. For example, the lowest level view in which touch is detected is optionally called the hit view, and the set of events recognized as appropriate input is optionally the initial view that initiates a touch gesture. The determination is based at least in part on the hit view of the touch.

The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process the sub-event. In most situations, the hit view is the lowest level view in which the initiating sub-event (eg, the first sub-event in the sequence of sub-events forming the event or potential event) takes place. Once a hit view is identified by the hit view determination module 172, this hit view typically receives all sub-events related to the same touch or input source that was identified as the hit view.

Active event recognizer determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only hit views should receive a particular sequence of sub-events. In another embodiment, the active event recognizer determination module 173 determines that all views containing the physical location of the sub-event are actively involved views, thus all actively involved views are: Determine that a particular sequence of sub-events should be received. In other embodiments, even if a touch sub-event is completely localized to an area associated with one particular view, views higher up in the hierarchy remain actively involved views.

Event dispatcher module 174 dispatches event information to an event recognizer (eg, event recognizer 180). In embodiments that include active event recognizer determination module 173 , event dispatcher module 174 transmits event information to event recognizers determined by active event recognizer determination module 173 . In some embodiments, the event dispatcher module 174 stores event information obtained by each event receiver 182 in an event queue.

In some embodiments, operating system 126 includes event sorter 170 . Alternatively, application 136 - 1 includes event sorter 170 . In still other embodiments, event sorter 170 is a stand-alone module or part of another module stored in memory 102 such as touch/motion module 130 .

In some embodiments, application 136-1 includes multiple event handlers 190 and one or more applications, each containing instructions for handling touch events that occur within respective views of the application's user interface. Contains view 191 . Each application view 191 of application 136 - 1 includes one or more event recognizers 180 . Typically, each application view 191 includes multiple event recognizers 180 . In other embodiments, any one or more of event recognizers 180 are separate modules, such as user interface kits or higher-level object objects from which application 136-1 inherits methods and other attributes. is part of In some embodiments, each event handler 190 processes one or more of data updater 176, object updater 177, GUI updater 178, and/or event data 179 received from event sorter 170. including. Event handler 190 optionally utilizes or calls data updater 176 , object updater 177 , or GUI updater 178 to update application internal state 192 . Alternatively, one or more of application views 191 include one or more respective event handlers 190 . Also, in some embodiments, one or more of data updater 176 , object updater 177 , and GUI updater 178 are included in each application view 191 .

Each event recognizer 180 receives event information (eg, event data 179) from event sorter 170 and identifies events from this event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184 . In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (optionally including sub-event delivery instructions).

The event receiver 182 receives event information from the event sorter 170 . Event information includes information about sub-events, such as touches or movement of touches. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes rotation of the device from one orientation to another (e.g., portrait to landscape or vice versa), and the event information is the device's current orientation (device (also called the attitude of the

The event comparator 184 compares the event information to predefined event or sub-event definitions and determines the event or sub-event or determines or updates the state of the event or sub-event based on the comparison. . In some embodiments, event comparator 184 includes event definitions 186 . Event definitions 186 include definitions of events (eg, predefined sequences of sub-events), eg, event 1 (187-1) and event 2 (187-2). In some embodiments, sub-events within event (187) include, for example, touch initiation, touch termination, touch movement, touch cancellation, and multiple touches. In one embodiment, the definition for event 1 (187-1) is a double tap on a displayed object. A double-tap is, for example, a first touch on a display object for a given step (touch start), a first lift-off for a given step (touch end), a second touch on a display object for a given step (touch start), and a second lift-off (end of touch) for a given stage. In another example, the definition of event 2 (187-2) is a drag on a displayed object target. Drag includes, for example, a touch (or contact) on a display object for a given step, movement of the touch across the touch-sensitive display 112, and lift-off of the touch (end of touch). In some embodiments, the event also includes information about one or more associated event handlers 190 .

In some embodiments, event definitions 187 include definitions of events for respective user interface objects. In some embodiments, event comparator 184 performs hit testing to determine which user interface objects are associated with sub-events. For example, when a touch is detected on touch-sensitive display 112 in an application view in which three user interface objects are displayed on touch-sensitive display 112, event comparator 184 performs a hit test to Determine which of the two user interface objects are associated with the touch (sub-event). If each displayed object is associated with a respective event handler 190, the event comparator uses the results of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and object that triggers the hit test.

In some embodiments, each event 187 definition also includes a delay action that delays delivery of event information until it is determined if the sequence of sub-events corresponds to the event recognizer's event type.

If the respective event recognizer 180 determines that the sequence of sub-events does not match any of the events in the event definition 186, the respective event recognizer 180 enters an event not possible, event failed, or event terminated state, After that, ignore the next sub-event of the touch-based gesture. In this situation, if there is another event recognizer that remains active for the hit view, that event recognizer will continue to track and process the ongoing touch gesture sub-events.

In some embodiments, each event recognizer 180 has configurable properties, flags, and/or properties that indicate to actively participating event recognizers how the event delivery system should perform sub-event delivery. or includes metadata 183 with a list. In some embodiments, metadata 183 includes configurable properties, flags, and/or a list. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate whether sub-events are delivered to various levels in the view hierarchy or program hierarchy.

In some embodiments, each event recognizer 180 activates the event processor 190 associated with the event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognizer 180 delivers event information associated with the event to the event processor 190 . Activating the event handler 190 is separate from sending (and deferring sending) sub-events to each hit view. In some embodiments, the event recognizer 180 populates a flag associated with a recognized event, and the event handler 190 associated with that flag catches the flag and performs a predetermined process.

In some embodiments, event delivery instructions 188 include sub-event delivery instructions to deliver event information about the sub-event without activating the event handler. Instead, sub-event delivery instructions deliver event information to event handlers associated with a sequence of sub-events or to actively participating views. An event handler associated with a sequence of sub-events or actively participating views receives event information and performs predetermined processing.

In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, the data updater 176 updates phone numbers used by the contacts module 137 or stores video files used by the video player module. In some embodiments, object updater 177 creates and updates objects used by application 136-1. For example, the object updater 177 creates new user interface objects or updates the positions of user interface objects. The GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to data updater 176 , object updater 177 , and GUI updater 178 . In some embodiments, data updater 176 , object updater 177 , and GUI updater 178 are included in a single module of respective application 136 - 1 or application view 191 . In other embodiments, they are contained in two or more software modules.

The foregoing description of event handling of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 100 using input devices, all of which are on touch screens. It should be understood that it does not start with . For example, mouse movement and mouse button presses, contact movement such as tapping, dragging, scrolling on touchpads, pen stylus input, devices optionally coupled with single or multiple presses or holds on the keyboard. movements, verbal commands, detected eye movements, biometric inputs, and/or any combination thereof are optionally utilized as inputs corresponding to sub-events that define the event to recognize.

FIG. 2 illustrates portable multifunction device 100 having touch screen 112, according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200 . In this embodiment, as well as other embodiments described below, the user may, for example, use one or more fingers 202 (not drawn to scale) or one or more styluses 203 (not shown to scale). are not drawn to scale) to select one or more of the graphics by performing a gesture on the graphics. In some embodiments, selection of one or more graphics is performed when the user breaks contact with the one or more graphics. In some embodiments, the gesture is one or more taps, one or more swipes (left to right, right to left, upward and/or downward), and/or contact with device 100. Finger rolling (right to left, left to right, upwards and/or downwards) is optionally included. In some implementations or situations, accidental contact with a graphic does not select that graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application if the gesture corresponding to selection is a tap.

Device 100 also optionally includes one or more physical buttons such as a “home” button or menu button 204 . As previously mentioned, menu button 204 is optionally used to navigate to any application 136 within the set of applications optionally running on device 100 . Alternatively, in some embodiments, menu buttons are implemented as softkeys within a GUI displayed on touch screen 112 .

In some embodiments, the device 100 includes a touch screen 112, a menu button 204, push buttons 206 to turn the device on/off and lock the device, volume control buttons 208, a subscriber identity module (SIM) card slot. 210, a headset jack 212, and an external port 124 for docking/charging. The push button 206 optionally powers on/off the device by depressing the button and holding the button depressed for a predetermined period of time such that depressing the button for a predetermined period of time elapses. It is used to lock the device by releasing the button before doing so and/or to unlock the device or initiate the unlocking process. In an alternative embodiment, device 100 also accepts verbal input via microphone 113 to activate or deactivate some functions. Device 100 also optionally includes one or more contact intensity sensors 165 that detect the intensity of contact on touch screen 112 and/or one or more tactile sensors that generate tactile outputs for the user of device 100 . An output generator 167 is included.

FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments; Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., home controller for commercial use or business use). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses interconnecting these components. 320 included. Communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects and controls communications between system components. Device 300 includes input/output (I/O) interface 330 including display 340, which is typically a touch screen display. The I/O interface 330 also optionally includes a keyboard and/or mouse (or other pointing device) 350, a touchpad 355, and a tactile output generator 357 ( e.g., analogous to the tactile output generator 167 described above with reference to FIG. 1A) and a sensor 359 proximity, touch sensing, and/or contact intensity sensors). Memory 370 includes high speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices, and optionally one or more magnetic disk storage devices, optical disk storage devices, flash memory devices. , or other non-volatile solid-state storage device. Memory 370 optionally includes one or more storage devices located remotely from CPU(s) 310 . In some embodiments, memory 370 stores programs, modules and data structures similar to, or a subset thereof, stored in memory 102 of portable multifunction device 100 (FIG. 1A). memorize Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100 . For example, memory 370 of device 300 optionally stores drawing module 380, presentation module 382, word processing module 384, website creation module 386, disc authoring module 388, and/or spreadsheet module 390. In contrast, memory 102 of portable multifunction device 100 (FIG. 1A) optionally does not store these modules.

Each of the above-identified elements of FIG. 3 are optionally stored in one or more of the memory devices previously described. Each of the modules identified above corresponds to a set of instructions that perform the functions previously described. The modules or programs (e.g., set of instructions) identified above need not be implemented as separate software programs, procedures, or modules; thus, in various embodiments, various subsets of these modules are optionally , combined or otherwise rearranged. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. In addition, memory 370 optionally stores additional modules and data structures not described above.

Attention is now directed to user interface embodiments that are optionally implemented, for example, on portable multifunction device 100 .

FIG. 4A illustrates an exemplary user interface of a menu of applications on portable multifunction device 100 according to some embodiments. A similar user interface is optionally implemented on device 300 . In some embodiments, user interface 400 includes the following elements, or subsets or supersets thereof.
- signal strength indicator(s) 402 for wireless communication(s) such as cellular and Wi-Fi signals;
● Time 404,
- Bluetooth(R) indicator 405,
- battery status indicator 406;
- A tray 408 with icons for frequently used applications, such as:
o an icon 416 of the phone module 138 labeled "phone", optionally including an indicator 414 of the number of missed calls or voicemail messages;
o Icon 418 of email client module 140 labeled "Mail", optionally including indicator 410 of number of unread emails;
o icon 420 for browser module 147, labeled "Browser"; and o icon 422 for video and music player module 152, also called iPod (trademark of Apple Inc.) module 152, labeled "iPod". , and other application icons, such as:
o IM module 141 icon 424, labeled "Message";
o Icon 426 of calendar module 148, labeled "Calendar";
o icon 428 of image management module 144, labeled "Photos";
o icon 430 for camera module 143, labeled "Camera";
o icon 432 for online video module 155, labeled "online video";
o icon 434 of Stocks widget 149-2, labeled "Stocks";
o icon 436 of map module 154, labeled "Map";
o icon 438 for weather widget 149-1, labeled "Weather";
o icon 440 of alarm clock widget 149-4, labeled "Clock";
o Icon 442 for training support module 142, labeled "Training Support";
o Icon 444 of notes module 153, labeled "Notes"; and o Settings application or module, labeled "Settings", which provides access to settings for device 100 and its various applications 136. icon 446 .

Note that the icon labels shown in FIG. 4A are merely exemplary. For example, the icon 422 for the video and music player module 152 is labeled "music" or "music player." Other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to each application icon. In some embodiments, the label of a particular application icon is different than the name of the application corresponding to that particular application icon.

FIG. 4B shows an exemplary display on a device (eg, device 300 of FIG. 3) having a touch-sensitive surface 451 (eg, tablet or touchpad 355 of FIG. 3) that is separate from display 450 (eg, touchscreen display 112). user interface. Device 300 also optionally includes one or more contact strength sensors (eg, one or more of sensors 359) that detect the strength of contact on touch-sensitive surface 451 and/or It includes one or more tactile output generators 357 that generate tactile outputs.

Although some of the examples below are given with reference to input on the touchscreen display 112 (which is a combined touch-sensitive surface and display), in some embodiments the device is as shown in FIG. 4B. Detects input on a touch-sensitive surface separate from the display shown. In some embodiments, the touch-sensitive surface (eg, 451 in FIG. 4B) has a primary axis (eg, 453 in FIG. 4B) that corresponds to the primary axis (eg, 453 in FIG. 4B) on the display (eg, 450). 452). According to these embodiments, the device interacts with touch-sensitive surface 451 at locations corresponding to respective locations on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in FIG. 4B). Touches (eg, 460 and 462 in FIG. 4B) are detected. In this way, user input (e.g., contacts 460 and 462, and their movements) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B) is detected by the touch-sensitive surface being separate from the display. When used by the device to operate the user interface on the multifunction device's display (eg, 450 in FIG. 4B). It should be appreciated that similar methods are optionally used for other user interfaces described herein.

Additionally, although the examples below are given primarily with reference to finger input (e.g., finger contact, finger tap gesture, finger swipe gesture), in some embodiments one or more of the finger inputs are , may be replaced by input from another input device (eg, mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (eg, instead of contact) followed by movement of the cursor along the path of the swipe (eg, instead of moving the contact). As another example, a tap gesture is optionally replaced with a mouse click while the cursor is positioned over the location of the tap gesture (e.g., detect contact and subsequently stop detecting contact). instead). Similarly, when multiple user inputs are detected simultaneously, it will be appreciated that multiple computer mice are optionally used simultaneously, or mouse and finger contact are optionally used simultaneously. sea bream.

FIG. 5A shows an exemplary personal electronic device 500. As shown in FIG. Device 500 includes body 502 . In some embodiments, device 500 can include some or all of the features described with respect to devices 100 and 300 (eg, FIGS. 1A-4B). In some embodiments, the device 500 has a touch-sensitive display screen 504, hereinafter touch screen 504. Alternatively, or in addition to touchscreen 504, device 500 has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more sensors that detect the strength of contact (e.g., touch) being applied. Includes intensity sensor. One or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on the intensity of the touch, meaning that different intensity touches can invoke different user interface operations on device 500 .

Exemplary techniques for detecting and processing touch intensity are disclosed, for example, in International Patent Application No. PCT/US2013/040061, "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application", and International Patent No. WO/2014/103712, published as See related application entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships," International Patent Application No. PCT/US2013/069483, filed May 11.

In some embodiments, device 500 has one or more input mechanisms 506 and 508 . Input mechanisms 506 and 508, if included, can be physical mechanisms. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, allow the device 500 to be attached to, for example, hats, eyewear, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, pants, belts, shoes, purses, backpacks, etc. can make it possible. These attachment mechanisms allow the device 500 to be worn by the user.

FIG. 5B shows an exemplary personal electronic device 500. As shown in FIG. In some embodiments, device 500 can include some or all of the components described with respect to FIGS. 1A, 1B, and 3. FIG. Device 500 has a bus 512 operatively coupling an I/O section 514 to one or more computer processors 516 and memory 518 . The I/O section 514 can be connected to a display 504, which can have a touch sensing component 522 and optionally an intensity sensor 524 (eg, a touch intensity sensor). In addition, the I/O section 514 communicates to receive application and operating system data using Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular, and/or other wireless communication techniques. It can be connected to unit 530 . Device 500 can include input mechanisms 506 and/or 508 . Input mechanism 506 is optionally a rotatable input device or a depressible and rotatable input device, for example. In some examples, input mechanism 508 is optionally a button.

In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors such as GPS sensor 532, accelerometer 534, orientation sensor 540 (e.g., compass), gyroscope 536, motion sensor 538, and/or combinations thereof, All of these sensors can be operatively connected to I/O section 514 .

Memory 518 of personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which when executed by one or more computer processors 516 . , for example, in a computer processor, including process 800 (FIGS. 8A-8B), process 1000 (FIG. 10), process 1200 (FIG. 12), process 1400 (FIG. 14), and process 1600 (FIG. 16), below. The techniques described can be implemented. A computer-readable storage medium is any capable of tangibly containing or storing computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. can be a medium of In some examples, the storage medium is a temporary computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic, optical, and/or semiconductor storage devices. Examples of such storage devices include magnetic discs, CDs, DVDs, or optical discs based on Blu-ray® technology, as well as resident solid state memories such as flash, solid state drives, and the like. Personal electronic device 500 is not limited to the components and configuration of FIG. 5B, but can include other or additional components in multiple configurations.

As used herein, the term “affordance” refers to a user-interacting graphical user interface optionally displayed on a display screen of device 100, 300, and/or 500 (FIGS. 1A, 3, and 5A-5B). Points to an interface object. For example, images (eg, icons), buttons, and text (eg, hyperlinks) each optionally constitute affordances.

As used herein, the term "focus selector" refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as a "focus selector," so that the cursor is positioned over a particular user interface element (e.g., button, window, slider, or other user interface element). When an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touch pad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B) while overlying, a particular user interface element Adjusted according to the detected input. In some implementations, including a touch screen display that enables direct interaction with user interface elements on the touch screen display (e.g., touch-sensitive display system 112 of FIG. 1A or touch screen 112 of FIG. 4A), Detected touch acts as a "focus selector", thus allowing input (e.g., by touch When a press input) is detected, certain user interface elements are adjusted according to the detected input. In some implementations, focus can be moved without corresponding cursor movement or contact movement on the touch screen display (e.g., by using the tab key or arrow keys to move focus from one button to another). is moved from one area of the user interface to another area of the user interface, and in these implementations the focus selector moves as focus moves between different areas of the user interface. Regardless of the specific form it takes, a focus selector is generally a user interface element (or contact on a touch screen display) that is controlled by a user to communicate the user's intended interaction through the user interface. (eg, by indicating to the device the elements of the user interface that the user is intended to interact with). For example, the position of a focus selector (e.g., cursor, touch, or selection box) over the corresponding button while a press input is detected on a touch-sensitive surface (e.g., touchpad or touchscreen) It indicates that the user intends to activate the corresponding button (as opposed to other user interface elements shown on the device's display).

As used herein and in the claims, the term "characteristic strength" of contact refers to the properties of that contact based on one or more strengths of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally a predefined number of intensity samples, i.e., a predefined event (e.g., after detecting a contact, before detecting lift-off of the contact, before or after detecting the start of movement of the contact, a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds). The characteristic strength of contact is optionally maximum strength of contact, mean strength of contact, average strength of contact, top 10 percentile strength of contact, strength of contact , a maximum 90 percent value of the intensity of contact, and so on. In some embodiments, the duration of contact is used in determining the characteristic strength (eg, when the characteristic strength is the average strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an action has been performed by the user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, a contact with a characteristic strength that does not exceed a first threshold results in a first action, and a contact with a characteristic strength that exceeds the first strength threshold but does not exceed a second strength threshold results in a second A contact that results in a motion of 2 and has a characteristic intensity exceeding a second threshold results in a third motion. In some embodiments, a comparison between the characteristic strength and one or more thresholds is used to determine whether to perform the first action or the second action. Instead, it is used to determine whether to perform one or more actions (e.g., whether to perform each action or not to perform each action).

FIG. 5C illustrates detecting multiple contacts 552A-552E on touch-sensitive display screen 504 by multiple intensity sensors 524A-524D. FIG. 5C additionally includes an intensity diagram showing current intensity measurements of intensity sensors 524A-524D versus intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are each 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are each 7 intensity units. In some implementations, the aggregate intensity is the sum of the intensity measurements of multiple intensity sensors 524A-524D, which in this example is 32 intensity units. In some embodiments, each contact is assigned a respective intensity that is a fraction of the aggregate intensity. FIG. 5D shows assigning aggregate strengths to contacts 552A-552E based on distance from center of force 554. FIG. In this example, contacts 552A, 552B, and 552E are each assigned a strength of contact of 8 intensity units of the aggregate strength, and contacts 552C and 552D are each assigned a strength of contact of 4 strength units of the aggregate strength. . More generally, in some implementations, each contact j is assigned a respective intensity Ij, which is a fraction of the total intensity A, according to a predetermined mathematical function Ij=A·(Dj/ΣDi); where Dj is the distance from the center of force to each contact j, and ΣDi is the sum of the distances from the center of force to all respective contacts (eg, from i=1 to the last). The operations described with reference to FIGS. 5C-5D may be performed using an electronic device similar or identical to device 100, 300 or 500. FIG. In some embodiments, the characteristic strength of contact is based on one or more strengths of contact. In some embodiments, an intensity sensor is used to determine a single characteristic intensity (eg, a single characteristic intensity of a single contact). Note that the intensity diagrams are not part of the display user interface and are included in FIGS. 5C-5D to aid the reader.

In some embodiments, a portion of the gesture is identified for purposes of determining characteristic strength. For example, the touch-sensitive surface optionally receives successive swipe contacts transitioning from a starting location to reaching an ending location, at which point the contact increases in strength. In this example, the characteristic strength of the contact at the end location is optionally based on only a portion of the continuous swipe contact (eg, only the portion of the swipe contact at the end location) rather than the entire swipe contact. In some embodiments, a smoothing algorithm is optionally applied to the strength of the swipe contact prior to determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate small increases or decreases in strength of swipe contacts for purposes of determining characteristic strength.

The intensity of contact on the touch-sensitive surface is optionally one or more intensities, such as a touch detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and/or one or more other intensity thresholds. Characterized against a threshold. In some embodiments, the light pressure intensity threshold corresponds to the intensity at which the device performs actions typically associated with clicking a physical mouse button or trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device performs a different action than is typically associated with clicking a physical mouse button or trackpad. In some embodiments, when a touch is detected with a characteristic strength below a light pressure intensity threshold (e.g., above a nominal touch detection intensity threshold below which no contact is detected), the device detects the light pressure intensity Moving the focus selector according to the movement of the contact on the touch-sensitive surface without performing the actions associated with the threshold or the deep pressure intensity threshold. In general, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface diagrams.

An increase in the characteristic strength of a contact from an intensity below the light pressure threshold to an intensity between the light pressure threshold and the deep pressure threshold is sometimes referred to as a "light pressure" input. An increase in the characteristic strength of a contact from an intensity below the deep pressure threshold to an intensity above the deep pressure threshold is sometimes referred to as a "deep pressure" input. An increase in the characteristic strength of a touch from an intensity below the touch detection intensity threshold to an intensity between the touch detection intensity threshold and the light pressure intensity threshold is sometimes referred to as detecting contact on the touch surface. A decrease in the characteristic strength of the contact from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting lift-off of the contact from the touch surface. In some embodiments, the contact detection strength threshold is zero. In some embodiments, the contact detection strength threshold is greater than zero.

In some embodiments described herein, in response to detecting a gesture comprising a respective pressure input or having detected a respective pressure input performed by a respective contact (or multiple contacts). One or more actions are performed in response to and each pressure input is detected based at least in part on detecting an increase in the intensity of the contact (or contacts) above the pressure input intensity threshold. . In some embodiments, the respective action is performed in response to detecting an increase in the strength of the respective contact above a pressure input strength threshold (eg, a "downstroke" of the respective pressure input). In some embodiments, the pressure input includes increasing the strength of each contact above the pressure input strength threshold followed by decreasing the strength of the contact below the pressure input strength threshold, and each action comprises a subsequent pressure Execution is in response to detecting a decrease in the intensity of the respective touch below the input threshold (eg, an "upstroke" of the respective pressure input).

FIGS. 5E-5H illustrate a contact from an intensity below the light press intensity threshold (eg, “IT _L ”) in FIG. 5E to an intensity above the deep press intensity threshold (eg, “IT _D ”) in FIG. 5H. 562 shows the detection of a gesture involving press input corresponding to an increase in strength. A gesture performed by contact 562 is detected on touch-sensitive surface 560 and displayed on display user interface 570, including application icons 572A-572D displayed within default area 574, over application icon 572B corresponding to app 2. A cursor 576 is displayed at . In some embodiments, gestures are detected on touch-sensitive display 504 . The intensity sensor detects the intensity of contact on touch-sensitive surface 560 . The device determines that the intensity of contact 562 has peaked above the deep press intensity threshold (eg, “IT _D ”). Contact 562 is maintained on touch sensitive surface 560 . In response to detection of the gesture, according to contact 562 having an intensity above the intensity threshold of a deep press (eg, "IT _D ") during the gesture, as shown in FIGS. A reduced-scale representation 578A-578C (eg, a thumbnail) of the document is displayed. In some embodiments, the strength compared to one or more strength thresholds is the characteristic strength of the contact. Note that the intensity diagram for contact 562 is not part of the display user interface and is included in FIGS. 5E-5H to aid the reader.

In some embodiments, the display of representations 578A-578C includes animation. For example, as shown in FIG. 5F, representation 578A is first displayed in proximity to application icon 572B. As the animation progresses, representation 578A moves upward and representation 578B is displayed closer to application icon 572B, as shown in FIG. 5G. Representation 578A then moves upward, representation 578B moves upward toward representation 578A, and representation 578C is displayed proximate application icon 572B, as shown in FIG. 5H. Representations 578A-578C form an array over icon 572B. In some embodiments, the animation progresses according to the intensity of contact 562, as shown in _FIGS . As it does so, representations 578A-578C appear and move upward. In some embodiments, the strength on which the animation progression is based is the characteristic strength of the contact. The operations described with reference to FIGS. 5E-5H may be performed using an electronic device similar or identical to device 100, 300, or 500. FIG.

In some embodiments, the device employs intensity hysteresis to avoid inadvertent inputs, sometimes referred to as "jitter," wherein the device has a hysteresis intensity threshold that has a predetermined relationship to the pressing input intensity threshold. (e.g., the hysteresis intensity threshold is X intensity units less than the pressure input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the pressure input intensity threshold ). Thus, in some embodiments, the pressure input includes an increase in strength of each contact above a pressure input strength threshold followed by a decrease in strength of the contact below a hysteresis strength threshold corresponding to the pressure input strength threshold, Each action is performed in response to detecting a subsequent decrease in the strength of the respective contact below the hysteresis strength threshold (eg, the "upstroke" of the respective pressure input). Analogously, in some embodiments, a pressure input causes the device to increase the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressure input intensity threshold, and optionally a subsequent hysteresis intensity. Detected only when detecting a decrease in the strength of contact to a strength less than or equal to each detected pressing input (e.g., increasing the strength of the contact or decreasing the strength of the contact, depending on the situation) is executed in response to

For ease of explanation, descriptions of actions performed in response to a press input associated with, or a gesture including, a press input associated with a pressure input intensity threshold optionally include the an increase in intensity, an increase in intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressure input intensity threshold, a decrease in intensity of the contact below the pressure input intensity threshold, and/or a hysteresis intensity corresponding to the pressure input intensity threshold. Triggered in response to detecting any decrease in the intensity of the touch below the threshold. Further, in examples in which actions are described as being performed in response to detecting a decrease in the intensity of contact below a pressure input intensity threshold, the actions optionally correspond to the pressure input intensity threshold, and in response to detecting a decrease in the intensity of the touch below a lower hysteresis intensity threshold.

FIG. 5I shows an exemplary electronic device 580. FIG. Device 580 includes body 580A. In some embodiments, device 580 can include some or all of the features described with respect to devices 100, 300, and 500 (eg, FIGS. 1A-5B). In some embodiments, device 580 has one or more speakers 580B (hidden in body 580A), one or more microphones 580C, one or more touch-sensitive surfaces 580D, and one or more displays 580E. Alternatively, or in addition to the display and touch-sensitive surface 580D, the device has a touch-sensitive display (also called a touchscreen). As with devices 100, 300, and 500, in some embodiments, touch-sensitive surface 580D (or touch screen) is optionally configured to detect the strength of contact (e.g., touch) being applied. including one or more intensity sensors of One or more intensity sensors on touch-sensitive surface 580D (or touch screen) may provide output data representing the intensity of the touch. The user interface of device 580 can respond to touches based on the intensity of the touch, meaning that different intensity touches can invoke different user interface actions on device 580 . In some embodiments, one or more displays 580E are one or more light emitting diodes (LEDs). For example, the display can be a single LED, an LED cluster (eg, red, green, and blue LEDs), multiple separated LEDs, multiple separated LED clusters, or other arrangement of one or more LEDs. For example, display 580E may be an array of nine separate LED clusters arranged in a circle (eg, ring). In some examples, the one or more displays are composed of any one or more of another type of light emitting element.

FIG. 5J illustrates an exemplary personal electronic device 580. As shown in FIG. In some embodiments, device 580 can include some or all of the components described with respect to FIGS. 1A, 1B, 3, and 5A-5B. Device 580 has a bus 592 operably coupling I/O section 594 to one or more computer processors 596 and memory 598 . The I/O section 594 can be connected to a display 582, which can have a touch-sensitive component 584 and optionally an intensity sensor 585 (eg, a contact intensity sensor). In some embodiments, touch-sensitive component 584 is a separate component from display 582 . Additionally, the I/O section 594 is a communication unit 590 for receiving application and operating system data using Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular, and/or other wireless communication technologies. can be connected to Device 580 may include input mechanism 588 . In some examples, input mechanism 588 is optionally a button. In some examples, input mechanism 588 is optionally a microphone. Input mechanism 588 is optionally a plurality of microphones (eg, a microphone array).

Electronic device 580 includes speaker 586 for outputting audio. Device 580 may include audio circuitry (eg, in I/O section 594 ) that receives audio data, converts the audio data to electrical signals, and transmits the electrical signals to speaker 586 . Speaker 586 converts the electrical signal into human audible sound waves. Audio circuitry (eg, in I/O section 594) also receives electrical signals converted from sound waves by a microphone (eg, input mechanism 588). Audio circuitry (eg, in I/O section 594) converts electrical signals to audio data. Audio data is optionally retrieved from memory 598 and/or RF circuitry 590 (eg, of communication unit 590) and/or transmitted to memory 102 and/or RF circuitry 108 by I/O section 594. be.

Memory 598 of personal electronic device 580 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which when executed by one or more computer processors 596 , for example, in a computer processor, including process 800 (FIGS. 8A-8B), process 1000 (FIG. 10), process 1200 (FIG. 12), process 1400 (FIG. 14), and process 1600 (FIG. 16), below. The techniques described can be implemented. A computer-readable storage medium is any capable of tangibly containing or storing computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. can be a medium of In some examples, the storage medium is a temporary computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic, optical, and/or semiconductor storage devices. Examples of such storage devices include magnetic discs, CDs, DVDs, or optical discs based on Blu-ray technology, as well as resident solid state memories such as flash, solid state drives, and the like. Personal electronic device 580 is not limited to the components and configuration of FIG. 5J, but can include other or additional components in multiple configurations.

Attention is now directed to embodiments of user interfaces (“UIs”) and associated processes implemented on an electronic device such as portable multifunction device 100 , device 300 or device 500 .

FIG. 6 includes a diagram 600 showing a set of physical structures and exemplary devices according to some embodiments. FIG. 6 is used to illustrate the processes described below, including the processes of FIGS. 8A-8B, 10 and 12. FIG.

Diagram 600 includes home 602 and a set of devices (eg, device 646, device 604, etc.). Home 602 includes multiple rooms, such as kitchen 602a, dining room 602b, bedroom 602c, living room 602d, and sunroom 602e. Kitchen 602a is in the upper left portion of home 602, and dining room 602b is between kitchen 602a and bedroom 602c, where bedroom 602c is to the right of dining room 602b. Kitchen 602a and dining room 602b do not have walls between them, but dining room 602b and bedroom 602c do. Living room 602 d is in the lower left portion of home 602 and sunroom 602 e is in the lower right portion of home 602 . In some embodiments, home 602 includes rooms other than the rooms illustrated in FIG. 6 and/or excludes the rooms illustrated in FIG. Although diagram 600 illustrates a home, this is merely an example and the techniques described herein can work with other types of physical structures such as office buildings, hotels, apartments, etc. want to be recognized.

As shown in FIG. 6, each room of home 602 has at least one device. However, in some embodiments, rooms in home 602 do not have devices.

As shown in FIG. 6, John's phone 604 is in the kitchen 602a located on the counter between the stove and the sink. In dining room 602b, dining room speakers 616 are located above the dining room table. In bedroom 602c, a personal tablet 634 is located on the bed, bedroom speakers 636 are located on the nightstand adjacent to the bed, and bedroom lights 632 are located on the opposite side of the bed. In living room 602d, living room speakers 646 and 648 are on opposite sides of the television, and living room lamp 642 is positioned in front of the television screen. In solarium 602e, solarium speaker 656 and solarium speaker 658 are located on opposite sides of the room.

As shown in FIG. 6, computing device 674 and Jane's phone 684 are outside home 602 . Computer device 674 is located in the dashboard of the car at distance 670 from home 602 and Jane's phone 684 is at distance 680 away from home 602 and is currently in Jane's possession. In some embodiments, computing device 674 is connected to an electronic device such as a mobile phone (eg, a personal phone similar to John's phone 604 and/or Jane's phone 684), which connects to computing device 674. It provides instructions to the computing device 674 to cause the visual content to be displayed. Distance 670 and distance 680 exceed the threshold distance away from home (eg, 1, 5, 10 meters), so computing device 674 and Jane's phone 684 are not considered to be at a location corresponding to home 602 . (eg, not within a geofence of home 602 or connected to a wireless network associated with (eg, established within) home 602).

In this example, the speakers shown in FIG. and living room lamps 642), and personal devices (e.g., John's phone 604, tablet 634, computing device 674, and Jane's phone 684) are connected to home 602 (e.g., John's user account or Jane's user account). assigned (eg, programmatically mapped) to home 602 (eg, to a corresponding group) by an associated user account.

Additionally, each individual speaker and light is also assigned to the room in which it is located (eg, mapped to a corresponding group). For example, dining room speakers 616 are assigned (eg, programmatically mapped to corresponding groups) dining room 602b. Bedroom speaker 636 and bedroom light 632 are designated for bedroom 602c. Living room lamps 642, living room speakers 646, and living room speakers 648 are designated living room 602d, and sunroom speakers 656 and sunroom speakers 658 are designated sunroom 602e. In some embodiments, each group corresponding to each room is also designated home 602 . Thus, devices mapped to rooms in home 602 are also mapped to home 602 at the same time.

An external device cannot be assigned to two different device groups corresponding to different rooms. For example, dining room speakers 616 cannot be assigned to kitchen 602a (eg, the group corresponding to kitchen 602a) and dining room 602b at the same time. However, two groups of devices, each assigned to a respective room, can be mapped to the (same) zone. In this example, entertainment zone 660 is designated living room 602d in FIG. 6 and sunroom 602e in FIG. In this example, entertainment zone 660 is not a physical room in home 602c. However, entertainment zones are programmatically defined as being designated as solarium living room 602d and solarium 602e. In some embodiments, two different zones share groups of devices (eg, including some overlapping devices, each including some non-overlapping devices). For example, a group of devices designated for solarium 602e may be mapped to an entertainment zone 660 and an eating zone (e.g., an eating zone may be mapped to a group of devices designated for dining room 602b and solarium 602e). It is specified).

In this example, personal devices (eg, John's phone 604, tablet 634, and Jane's phone 684) are not programmatically mapped to groups corresponding to rooms, but only to groups corresponding to home.

In some embodiments, the speakers, lights, and personal devices (eg, John's phone 604, tablet 634, Jane's phone 684) shown in FIG. 6 are connected to one or more networks (eg, wireless networks (eg, Bluetooth , NFC, Wi-Fi, 4G, etc.)). In some embodiments, other smart devices such as smart appliances, smart thermostats, smart plug outlets, etc. are connected to one or more networks. In some embodiments, one or more personal devices can also include other types of personal devices such as smartwatches, laptop computers, desktops, and the like.

7A-7AH illustrate exemplary electronic devices and user interfaces for sending voice messages, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in Figures 8A-8B.

FIG. 7A shows electronic device 700 displaying user interface 702 . 7A, electronic device 700 is a portable multifunction device having one or more components described above with respect to any one or more of devices 100, 200, 300, and 500. In FIG. With respect to the following figures, in some embodiments device 700 is operating less than a predetermined distance away from home 602 . However, in other embodiments, device 700 is operating more than a predetermined distance away from home 602 . In some embodiments, device 700 is connected to a network (e.g., wireless network) associated with home 602, so device 700 communicates with at least one device (e.g., speakers, lights) located in home 602. connected to the same wireless network to which the In some embodiments, more than one wireless network is associated with home 602 . In some embodiments, device 700 is connected to at least one of those networks. In some embodiments, device 700 is not connected to one or more networks associated with home 602 . Thus, device 700 is able to send audio multicast messages and communicate with devices within home 602 regardless of how far device 700 is from home 602 or the network to which device 700 is connected.

Although this description often refers to messages as multicast messages (or audio multicast messages), such messages optionally include (1) multicast communications, (2) broadcast communications, (3) unicast communications, (4) sent using or any combination thereof (e.g., (1) a first device (e.g., controller) sends a unicast communication to a second device (e.g., home hub) and (2) the second device transmits either multicast or broadcast communications (eg, including content from unicast communications) to one or more tertiary devices (eg, home and/or one or more to one or more accessory devices within a personal device). In some embodiments, a multicast communication is a communication sent to multiple recipients. In some embodiments, broadcast communications are communications sent to devices on a channel or network. In some embodiments, broadcast communications are communications sent to all devices on a channel or network. In some embodiments, unicast communications are communications sent to a single recipient. In such embodiments, multiple unicast communications are sent if the content needs to be sent to multiple devices.

In some embodiments, broadcast communications are communications sent to all devices on a channel or network. In such embodiments, multiple unicast communications are sent if the content needs to be sent to multiple devices.

As shown in FIG. 7A, user interface 702 includes application icons 704, such as home application icon 704a. Each of the application icons 704 corresponds to an application such that selection of the application icon causes the device 700 to display the user interface of the respective application. For example, home application icon 704a corresponds to the home application. In FIG. 7A, device 700 detects tap gesture 750a (eg, select) at a location corresponding to home application icon 704a.

As shown in FIG. 7B, in response to detecting tap gesture 750a, device 700 displays home application user interface 710 . User interface 710 is a user interface for controlling a plurality of external devices designated as home 602 . User interface 710 includes home location identifier 712, state information 714, macro controls 716, external device controls 718, navigation controls 722, home multicast instructions 724, and home multicast affordances 726 (eg, selectable user interface object targets). .

Home location identifier 712 is the name corresponding to home 602 . In this example, the location identifier is part of the home 602 address (eg, "123 Main Street"). In some embodiments, the location identifier is an alias, such as a home-designated alias. In some embodiments, the alias is a system-created alias. In some embodiments, the alias is a user-created alias. In some embodiments, the location identifier is set by a user of device 700 or an external device. In some embodiments, the location identifier is for a graphical user interface object, such as an image.

State information 714 includes information about the current state of the portion of the device designated Home 602 . Here, state information 714 indicates that at least one light is at home 602 . In some embodiments, status information 714 includes information about the status of multiple devices (eg, "bedroom fan on").

As shown in FIG. 7B, below the home location identifier 712 and state information 714 are two rows of macro controls 716 . Macro controls 716 are affordances (eg, selectable user interface object targets) for initiating respective macros. In some embodiments, each macro causes one or more external devices to perform actions. For example, a "good night" macro control, when selected, causes one or more lights to turn off, one or more speakers to stop playing media, and/or one or more deadbolt locks to lock.

Additionally, in FIG. 7B, user interface 710 includes a row for external device controls 718 below macro controls 716 . In FIG. 7B, the two rows of external device controls 718 extend over three columns, so there are a total of six displays (eg, three device controls on each corresponding row). In some embodiments, a user may have fewer external devices for controls that require fewer external device controls 718 . In other embodiments, the user may have more external devices for controls that require more external device controls 718 . In some embodiments, when more external device controls 718 are displayed than available screen space, the screen scrolls (eg, up and down or left and right) to display all external device controls. External device controls 718 include dining room speaker control 718a, living room lamp control 718b, living room speaker control 718c, sunroom speaker control 718d, bedroom speaker control 718e, and bedroom light control 718f. Additionally, each of the external device controls 718 includes state information representing the current state of the respective external device. For example, state information for dining room speaker control 718a indicates that dining room speaker 616 is currently playing media (not represented by state information 714 in this example), and state information for living room lamp control 718b indicates that the living room Indicates that lamp 642 is currently off (represented by status information 714 in this example).

Each of the external device controls 718 controls at least one device designated home 602 when selected. For example, selecting dining room speaker control 718a causes dining room speaker 616 to play or pause music. In some embodiments, when dining room speaker control 718a is selected, the state information for dining room speaker control 718a changes to indicate that dining room speaker 616 has paused playing media (e.g., bedroom speaker control 718e ), or to indicate that the media has stopped playing. Some of the external device controls 718 control multiple devices when selected. For example, sunroom speaker control 718d causes sunroom speakers 656 and 658 to play or pause media playback when selected, and the status information displayed on sunroom speaker control 718d changes accordingly. Some of the external device controls 718 are not speaker device controls. For example, the external device controls 718 include a living room lamp control 718b that, when selected, causes the living room lamp control 718b to turn on/off the living room lamps 642 and the state displayed on the living room lamp control 718b. Information changes accordingly.

In some embodiments, external device control 718 is selected from a group of other external controls that are also designated home 602 . For example, the 6 controls shown in FIG. 7B can be selected from 10, 20, or 50 (or any other number greater than 6) external controls that are also designated home 602 . In some embodiments, external device controls 718 are selected based on a determination that each of these external device controls is a favorite control. In some embodiments, the determination of whether the external device is a favorite control is based on the most recently used, most frequently used, newly added, favorite, external device control (or external device) controls. Designated as critical external device control (eg, via detection of one or more inputs), etc., or based on any combination thereof.

At the bottom of user interface 710 are navigation controls 722 . Navigation controls 722 include home navigation controls 722a, room navigation controls 722b, and automated navigation controls 722c. Each of navigation controls 722, when selected, causes device 700 to display a different user interface for the home application.

As shown in FIG. 7B, user interface 710 includes home multicast instructions 724 adjacent to home multicast affordances 726 . In FIG. 7B , home multicast instructions 724 require the user to select (eg, “tap”) home multicast affordance 726 to send voice messages that are multicast to one or more devices designated home 602 . indicates that there is In FIG. 7B, device 700 detects tap gesture 750b at a location corresponding to home multicast affordance 726. In FIG.

In response to detecting tap gesture 750 b , device 700 initiates a process to send an audio multicast message to one or more devices designated as home 602 . In some embodiments, the process includes displaying the user interface illustrated in Figures 7C, 7D, and 7E, as described below. In some embodiments, the process includes displaying the user interface illustrated in FIG. 7C1, as described below.

In response to detecting tap gesture 750b, user interface 730 includes home destination identifier 732, exit affordance 734, and multicast recording affordance 736, as shown in FIG. 7C. Home destination identifier 732 indicates that the voice multicast message was directed to a group of devices designated home 602 (eg, to receive the voice multicast message). Exit affordance 734 is an affordance for exiting user interface 730 . In some embodiments, when exit affordance 734 is selected, device 700 ceases displaying user interface 730 and redisplays user interface 710, as illustrated in FIG. 7B. 7C, device 700 detects tap gesture 750c at a location corresponding to multicast recording affordance 736. In FIG.

In response to detecting tap gesture 750c, device 700 displays user interface 740, as shown in FIG. 7D. User interface 740 includes home destination identifier 732 and exit affordance 734 , which were also included in user interface 730 . Additionally, the user interface 740 also includes an audio status bar 742 and an audio wave 744a.

Further, in FIG. 7D, device 700 also begins recording an audio multicast message in response to detecting tap gesture 750c. The audio of the audio multicast message is detected via one or more microphones of device 700 . Audio for the audio multicast message is detected from one or more external sources (eg, user audio, etc.). Here, recorded audio is represented by audio wave 744a, which fills audio status bar 742 over a period of time as more audio input is received by device 700. FIG.

As shown in FIG. 7D, in response to detecting tap gesture 750c, device 700 displays stop multicast recording affordance 746 at the location where multicast recording affordance 736 was previously displayed in FIG. 7C. 7D, device 700 detects tap gesture 750d at a location corresponding to multicast stop affordance 746. In FIG.

In response to detecting tap gesture 750d, device 700 displays user interface 750, as shown in FIG. 7E. In FIG. 7E, in response to detecting tap gesture 750d, device 700 has ceased recording or capturing audio from one or more external sources. User interface 750 thus includes an audio status bar 742 mostly filled with audio waves 744a representing at least a portion of a recorded audio multicast message. As shown in FIG. 7E, user interface 750 includes audio multicast message play affordance 752 and audio multicast message delete affordance 754 (eg, home destination identifier 732, end affordance 734, audio status bar 742, audio wave 744a). In addition to). In some embodiments, in response to detecting selection of audio multicast message playback affordance 752, device 700 initiates playback of the audio multicast message. In some embodiments, in response to detecting selection of audio multicast message deletion affordance 754, device 700 deletes the audio multicast message. In some embodiments, device 700 redisplays user interfaces 710 , 730 , and/or 740 in response to detecting selection of audio multicast message deletion affordance 754 .

As shown in FIG. 7E, in response to detecting tap gesture 750d, device 700 moves to the location where multicast stop affordance 746 was previously displayed in FIG. 7D (where multicast recording affordance 736 was displayed in FIG. 7C). to display the multicast transmission affordance 756 .

7C, 7D, and 7E, device 700 may display the user interface shown in FIG. 7C1 in response to detecting tap gesture 750b in FIG. 7B. As shown in FIG. 7C1, in response to detecting tap gesture 750b, device 700 displays user interface 741 and sends an audio multicast message (eg, without detecting any further gestures, such as tap gesture 750c). automatically start recording. As shown in FIG. 7C1, user interface 741 includes home destination identifier 732, exit affordance 734, displayed by device 700 using techniques similar to those described above with respect to FIGS. 7C, 7D, and 7E. , audio waves 744 a , and multicast transmission affordances 747 . For example, in FIG. 7C1, sound wave 744a is dynamic and changes over time with respect to one or more characteristics (eg, level, pitch, tone) of the currently recorded sound over the length of time the sound is being recorded. changes dramatically. In some embodiments, sound waves 744a move to the left as the sound is being recorded for the length of time the sound is being recorded.

In some embodiments, instead of passing audio through the flows described in 7C, 7D, and 7E and/or through the flow described in FIG. Recorded using G.741. For example, in response to detecting a hold gesture on home multicast affordance 726, device 700 begins recording an audio multicast message. In such an example, recording is stopped when the hold gesture ends (eg, is released), and replacement of home multicast affordance 726 with a send affordance (eg, analogous to multicast send affordance 756) occurs. In some embodiments, the audio is automatically sent as a multicast message when the holding gesture ends. In some embodiments, when the hold gesture ends, home multicast instruction 724 provides an affordance to play audio (eg, similar to audio status bar 742) and an affordance to remove audio (eg, voice multicast message (similar to Delete Affordance 754). In another example, in response to detecting a tap gesture on home multicast affordance 726, home multicast affordance 726 behaves similarly to multicast recording affordance 736 (e.g., stops recording (similar to multicast recording stop affordance 746)). replaced by affordances). In such an example, in response to detecting a tap gesture on a stop affordance, (1) the stop affordance is replaced with a send affordance (eg, similar to multicast send affordance 756), and (2) a home multicast instruction 724 is replaced with an affordance for playing audio (eg, similar to audio status bar 742) and an affordance for deleting audio (eg, similar to delete audio multicast message affordance 754). In FIG. 7E, device 700 detects tap gesture 750e at a location corresponding to multicast transmission affordance 756, or in FIG.

As shown in FIG. 7F, in response to detecting tap gesture 750e or tap gesture 750c1 (or a tap gesture on a send affordance as described in the previous paragraph), device 700 sends an audio multicast message to home 602. multicast (transmit) to a set of external devices specified in . Such transmission optionally includes (1) sending a message that includes an audio multicast message to another device (e.g., a device located within a home); Sending a message to a set of external devices (in some embodiments, the other device identifies the set of external devices on behalf of device 700). Table 706 is provided to indicate the external devices to which audio multicast messages are directed. For example, a checkmark in the "Audio Multicast Messages Received" column indicates that an audio multicast message was directed (e.g., sent) to an external device on the respective row of checkmarks; A dash in a column indicates that the audio multicast message was not directed (eg, not sent) to the external device on the respective column of dashes.

In FIG. 7F, table 706 shows that audio multicast messages are directed to each speaker (eg, dining room speaker 616, bedroom speaker 636, living room speaker 646, living room speaker 648, sunroom speaker 656, and sunroom speaker 658). (and received). Notably, voice multicast messages were also directed to personal devices (eg, John's phone 604, tablet 634, and Jane's phone 684). In some embodiments, the audio multicast message is directed to a personal device regardless of whether the personal device was at a location corresponding to home (eg, regardless of whether distances 670 and 680 exceed a threshold distance). For example, the voice multicast message was directed to John's phone 604 and tablet 634 located inside home 602 in FIG. In addition, the audio multicast message was also directed to computer device 674 and Jane's phone 684 outside home 602 (distances 670 and 680 from home 602, respectively). In some embodiments, audio multicast messages are directed to personal devices depending on whether the personal device is included in the home network (e.g., if the personal device is included in the home network, Audio multicast messages are directed to personal devices).

Table 706 of FIG. 7F also shows that the audio multicast message was not directed to some external devices (and therefore did not receive the audio multicast message), but they are designated home 602. . For example, table 706 shows that audio multicast messages are not directed to bedroom lights 632 and living room lights 642 . In some embodiments, the audio multicast message is not directed to some types of external devices, regardless of whether the audio multicast message is designated for the group of devices to which the audio multicast message is being sent. In some embodiments, these types of external devices include devices that do not have one or more audio input devices (eg, microphones) and/or one or more audio output devices (eg, speakers). In some embodiments, these types of external devices include devices that are inactive, unpowered, or non-functional at (or before) the time the voice message is sent.

As shown in FIG. 7F, at some point after detecting tap gesture 750e or tap gesture 750c1 (or the tap gesture on the transmit affordance, as described above), device 700 redisplays user interface 710. FIG. In some embodiments, the user interface 710 may detect any additional input after the device 700 detects the tap gesture 750e or the tap gesture 750c1 (e.g., tap gesture 750e or tap gesture 750c1 (or send affordance A predetermined period of time after detecting the above tap gesture)) (as described above, (eg, after a predetermined period of time after detecting the tap gesture)). In some embodiments, user interface 710 is displayed when one or more additional inputs are detected, such as the tap gesture input detected on exit affordance 734 of FIG. 7E.

In some embodiments, in response to detecting tap gesture 750f1 at a location corresponding to living room lamp control 718b, device 700 turns living room lamp 642 on (or off). In some embodiments, in response to detecting tap gesture 750f2 at a location corresponding to living room speaker control 718c, device 700 causes living room speakers 646 and 648 (or at least one of them) to: Send an instruction to start playing the media (or pause the playing of the media if the current state is playing). In FIG. 7F, device 700 detects tap gesture 750f3 at a location corresponding to room navigation affordance 722b.

As shown in FIG. 7G, in response to detection of tap gesture 750f3, device 700 displays home application user interface 760, and instead of home navigation control 722a (eg, not bolded in FIG. 7G), home . Navigation control 722b (eg, bold in FIG. 7G) is shown selected. User interface 760 is a user interface for controlling a plurality of external devices assigned to living room 602 d , as indicated by living room location identifier 762 . User interface 710 includes macro controls 766 , external device controls 768 , navigation controls 722 , living room multicast instructions 774 , and living room multicast affordances 776 . These elements of user interface 760 are specific controls for living room devices, in contrast to the elements of user interface 710, which are controls for home-wide devices. In some embodiments, any one or more of these elements of user interface 760 serve similar purposes to any one or more of the elements of user interface 710 described above (eg, in FIG. 7B). Or have functionality. In some embodiments, user interface 760 includes status information including the current status of devices assigned to living room 602d. In some embodiments, device 700 displays home application user interface 760 after receiving a swipe gesture on user interface 710 .

In particular, the home application's user interfaces (eg, user interfaces 710 and 760) are laid out in a hierarchy. For example, user interface 710 is a higher node in the hierarchy and contains controls for a group of external devices designated as home 602 . While displaying user interface 710, an input (eg, a single input (eg, tap gesture 750f)) can be detected that causes device 700 to display user interface 760, which is a lower node in the hierarchy. , including controls for groups of external devices assigned to each room of home 602 (eg, living room 602d) (and not other rooms).

As shown in FIG. 7G, external device controls 768 include living room speaker control 768a, living room lamp control 768b, living room switch (eg, light switch) control 768c, and living room television control 768d. In FIG. 7G, living room speaker control 768a and living room lamp control 768b, when selected, activate living room lamp device control 718b and living room speaker device control 718c to their respective external devices (e.g., in FIG. 7B ) causes each of the corresponding external devices to perform the same action, rather than the action you want it to perform. Thus, in this example, living room speaker control 768a and living room lamp control 768b are effectively the same controls as living room speaker device control 718c and living room lamp device control 718b, but they are part of their corresponding components. They are visually different from each other. In some embodiments, living room speaker control 768a and living room lamp control 768b do not appear visually different from living room speaker device control 718c and living room lamp device control 718b.

As compared to the external device controls 718 displayed in FIG. 7B, the external device controls 768 include at least one external device control displayed as part of the external device controls 718 in FIG. room lamp control 768b), and at least one external device control not displayed (eg, living room switch control 768c and living room television control 768d). In other words, at least one external device (eg, living room speakers 646 and 648) can be controlled from user interfaces 710 and 760, and at least one external device (eg, bedroom speakers 636 and living room speakers of user interface 760). room television) can only be controlled from only one of user interface 710 and user interface 760 . In some embodiments, in response to detecting tap gesture 750g1 at a location corresponding to living room lamp control 768b, device 700 turns living room lamp 642 on (or off). In some embodiments, in response to detecting tap gesture 750g2 at a location corresponding to living room speaker control 768a, device 700 causes living room speakers 646 and 648 (or at least one of them) to: Send an instruction to start playing the media (or pause the playing of the media if the current state is playing).

Additionally, external device controls 718 are designated as one set of favorite controls for home 602 in FIG. 7B, while external device controls 768 are designated for all controls designated for living room 602d in FIG. 7G. One set. In FIG. 7B, user interface 760 does not include any external devices not designated for living room 602d. For example, dining room speaker control 718 a , solarium speaker device control 718 d , bedroom speaker device control 718 e , and bedroom light device control 718 f are not displayed on user interface 760 . In some embodiments, a user interface for controlling multiple external devices for another room (eg, bedroom, solarium, kitchen, etc.) is displayed in response to detecting tap gesture 750f3. In some embodiments, a user for controlling multiple external devices for another room is displayed in response to one or more inputs (eg, swipe inputs) detected in FIG. 7G.

As shown in FIG. 7G, user interface 760 includes living room multicast instructions 774 adjacent to living room multicast affordances 776 . In FIG. 7G, living room multicast instructions 774 select living room multicast affordance 776 (e.g., “tap ”) indicates that you should Here, the living room multicast affordance 776 does not appear visually different from the home multicast affordance 726 (eg, in FIG. 7B), and the same location on the device 700 where the home multicast affordance 726 was displayed in FIG. 7B. to be displayed. In some embodiments, living room multicast affordance 776 appears visually different from home multicast affordance 726 . In FIG. 7G, device 700 detects tap gesture 750g3 at a location corresponding to living room multicast affordance 776. In FIG.

In response to detecting tap gesture 750g3, device 700 initiates a process to send an audio multicast message to one or more devices designated in living room 602d. In some embodiments, the process includes displaying a user interface 780a, as illustrated in Figure 7H. In some embodiments, the process includes displaying a user interface similar to user interface 741 of FIG. 7C1 displayed in response to device 700 detecting tap gesture 750g3. In some embodiments, the process causes device 700 to display a user interface similar to user interface 741 of FIG. 7C1 to record an audio multicast message for transmission to a group of devices designated in living room 602d. include. In some embodiments, the process includes device 700 displaying a user interface, as described below in FIGS. sent.

As shown in FIG. 7H, in response to detection of tap gesture 750g3, user interface 780a is similar to user interface 730 of FIG. 7C such that user interface 780a also includes exit affordance 734 and multicast recording affordance 736. However, user interface 780 a includes living room destination identifier 782 instead of home destination identifier 732 . Living room destination identifier 782 indicates that the group of devices designated for living room 602d will receive the multicast message sent via user interface 780a. In some embodiments, user interface 780a does not list individual names/identifiers of devices that are part of a group. In some embodiments, one or more of the techniques described above with respect to user interface 730 can be applied to user interface 780a.

7K, in some embodiments, user interface 780b, in response to detecting tap gesture 750g3, displays one or more (or all) external devices (e.g., speakers) assigned to living room 602d. ) is unavailable (eg, inactive or off). As shown in FIG. 7K, user interface 780b displays if any one or more of the external devices designated for living room 602d are unavailable (or if less than all are unavailable). is not), the multicast recording affordance 736 includes an unavailable indication 788 to indicate that it is displayed as grayed out or greyed out. In FIG. 7K, in response to detection of tap gesture 750k, device 700 continues to display user interface 780b as multicast recording affordance 736 is disabled (eg, one or more designated for living room 602d). until an external device becomes available). In some embodiments, a user interface similar to user interface 780b is displayed in response to device 700 detecting tap gesture 750c in FIG. 7C.

Returning to FIG. 7H, device 700 receives tap gesture 750 h at a location corresponding to multicast recording affordance 736 .

In response to receiving tap gesture 750h, device 700 begins recording audio from one or more external sources (eg, using device 700's microphone), as shown in FIG. A user interface 784 similar to user interface 740 is displayed. Like user interface 740 , user interface 784 includes end affordance 734 , audio status bar 742 , and stop multicast recording affordance 746 . However, user interface 784 includes audio wave 744b representing a different audio recorded from the audio recorded in FIG. 7D represented by audio wave 744a. User interface 784 also includes living room destination identifier 782 instead of home destination identifier 732 . In some embodiments, one or more of the techniques described above with respect to user interface 740 can be applied to user interface 784 . 7I, device 700 detects tap gesture 750i at a location corresponding to multicast stop affordance 746. In FIG.

As shown in FIG. 7J, in response to detecting tap gesture 750i, device 700 displays user interface 786 and stops recording or capturing audio from one or more external sources. User interface 786 is similar to user interface 750, except that audio wave 744b is displayed (instead of audio wave 744a), and living room destination identifier 782 is displayed (instead of home destination identifier 732). ing. In some embodiments, one or more of the techniques described above with respect to user interface 750 can be applied to user interface 786 . In FIG. 7J, the device 700 performs a tap gesture 750j at a location corresponding to the multicast transmission affordance 756 (or, as described above, a tap gesture on a user interface similar to user interface 741 of FIG. 7C1 (eg, tap gesture 750c1 and similar)).

In some embodiments, the device 700 performs a tap gesture 750j, as shown in FIG. 7L (or, as described above, a tap gesture on a user interface similar to the user interface 741 of FIG. 750c1), displays user interface 790. Device 700 indicates that the audio multicast message cannot be transmitted to one or more (or all) of the external devices designated in living room 602d. display user interface 790. User interface 790 does not transmit indication 791 to indicate that the audio multicast message was not transmitted to the targeted external device. , a user interface similar to user interface 790 is displayed in response to device 700 detecting tap gesture 750e of FIG. 7E or tap gesture 750c1 of FIG. 7C1.

In some embodiments, any one or more of user interfaces 730, 740, 750, 780a, 780b, 784, 786, and 790 are configured to swap the intended target of the recorded audio multicast message. (e.g. changing the destination identifier). For example, after initiating the process for sending an audio multicast message to a first group of devices (e.g., devices designated as Home 602) (during illustration of FIG. 7J, and user interface 741 of FIG. 7C1 as described above). (such as before receiving an input 750j on a user interface similar to or a tap gesture (eg, similar to tap gesture 750c1), device 700 responds to the intended target (eg, or intended A plurality of affordances for swapping objectives) from a first group of devices to a second group of devices (e.g., devices designated for dining room 602b). be able to.

In some embodiments, audio is recorded and transmitted from a user interface similar to user interface 741 of FIG. 7C1 that is displayed in response to detection of tap gesture 750g. In some embodiments, audio is recorded from user interface 760 of FIG. 7G instead of the flow described in FIGS. 7H-7J. For example, in response to detecting a hold gesture on living room multicast affordance 776, device 700 begins recording an audio multicast message. In such an example, recording stops when the hold gesture ends (eg, is released), causing replacement of living room multicast affordance 776 with a send affordance (eg, similar to multicast send affordance 756). . In such an example, living room multicast instructions 774 include an affordance for playing audio (eg, similar to audio status bar 742) and an affordance for deleting audio (eg, similar to delete audio multicast message affordance 754). be replaced by In another example, in response to detecting a tap gesture on living room multicast affordance 776, living room multicast affordance 776 operates similarly to multicast recording affordance 736 (e.g., similar to multicast recording stop affordance 746). ) is replaced by the Stop Affordance). In such an example, in response to detecting a tap gesture on a stop affordance, the stop affordance is replaced with a send affordance (eg, similar to multicast send affordance 756), and living room multicast instructions 774 turn audio off. Replaced by affordances to play (eg, similar to audio status bar 742) and affordances to delete audio (eg, similar to delete audio multicast message affordance 754).

As shown in table 706 of FIG. 7M, a tap gesture 750j (or a tap gesture on a transmit affordance as described in the previous paragraph or a tap on a user interface similar to user interface 741 of FIG. 7C1 as described above). In response to detecting a gesture (eg, similar to tap gesture 750c1), device 700 multicasts (sends or directs) an audio multicast message to a set of external devices designated in living room 602d. Such transmission optionally includes (1) sending a message containing an audio multicast message to another device (e.g., a device located in a living room); and (2) the other device Sending a message to a set of external devices (in some embodiments, the other device identifies the set of external devices on behalf of device 700). As shown in table 706, audio multicast messages are directed only to living room speakers 646 and 648, which receive audio multicast messages. In particular, audio multicast messages are not directed to personal devices (e.g., John's phone 604, tablet 634, and Jane's phone 684) because the personal devices are not assigned to the group corresponding to the room ( For example, personal devices are not assigned to the group corresponding to living room 602d). Additionally, the audio multicast messages are not directed to living room lamps 642, since living room lamps 642 can receive audio multicast messages for similar reasons discussed above (eg, in FIG. 7F). This is because it is not the type of external device.

In FIG. 7M, device 700 detects voice input 792a that includes the phrase "Hey Assistant Multicast Hello". Some time after detecting a portion of speech input 792a, device 700 displays suggestion 794a on user interface 794 (eg, on the left side of user interface 794). Here, suggestion 794a is a partial response to voice input 792a. Suggestion 794a is text that reads, "What is your business?" In some embodiments, device 700 outputs media that includes an audible version of suggestion 794a.

7N, after displaying user interface 794 of FIG. 7M, device 700 displays suggestion 794b on the right side of user interface 794. In FIG. In addition, device 700 displays suggestions 794c on the left side of user interface 794 and below suggestions 794b. Suggestion 794b is a letter written "Hey Assistant Multicast Hello" which is the letter corresponding to the audible phrase contained in voice input 792a. In addition, suggestion 794c is the text that says "OK, I sent your multicast to 123 Main Street." In particular, 123 Main Street is an identifier for home 602 (eg, home location identifier 712 in FIG. 7B). In some embodiments, device 700 outputs media including an audible version of response 794c.

As shown in table 706 of FIG. 7N, after detecting audio input 792a, device 700 directs an audio multicast message to one or more devices designated as home 602, as described above in connection with FIG. 7F. (e.g. send). Table 706 displays audio multicast messages ( "Hello") is directed to each speaker in the Home (e.g., dining room speaker 616, bedroom speaker 636, living room speaker 646, living room speaker 648, sunroom speaker 656, and sunroom speaker 658) and audio multicast Indicates that the message (eg, "hello") was directed to a home personal device (eg, John's phone 604, tablet 634, Jane's phone 684).

As shown in FIG. 7O, some time after displaying user interface 794 in FIG. 7N, device 700 displays user interface 710 . While displaying user interface 710, device 700 detects gesture 750o (eg, long press gesture) at the location corresponding to bedroom speaker control 718e in FIG. 7O.

As shown in FIG. 7P, in response to detecting gesture 750o, device 700 displays user interface 796 that includes multiple setting affordances for controlling bedroom speakers 636 (e.g., room settings (e.g., bedroom speakers)). 7P to indicate that 636 is designated as bedroom 602c), set favorites (e.g., locations included in the favorites are selected in FIG. 7P, thereby enabling bedroom speaker control). 718e can be displayed on the user interface 710)). As shown in FIG. 7P, one of the setting affordances is multicast setting affordance 796a, which includes an indication that bedroom speakers 636 are configured to receive multicast messages. In FIG. 7P, device 700 detects tap gesture 750p at a location corresponding to multicast setup affordance 796a.

As shown in FIG. 7Q, in response to detecting tap gesture 750p, device 700 displays user interface 798, including multicast settings toggle 798a. In FIG. 7Q, multicast settings toggle 798a indicates that bedroom speakers 636 are configured to receive multicast messages (eg, the circle displayed to the right of the settings toggle). In FIG. 7Q, device 700 detects tap gesture 750q at a location corresponding to multicast setup affordance 798a.

As shown in FIG. 7R, tap gesture 750q changes the state of multicast toggle 798a (eg, moves the circle to the left of the settings toggle) in response to detecting device 700, causing bedroom speakers 636 to receive the multicast message. Indicates that it is not configured to In some embodiments, the multicast setting affordance 796a of FIG. 7P is selectable to change the setting from on to off without navigating to a different user interface (eg, user interface 798).

In FIG. 7S, device 700 detects voice input 792b that includes the phrase "Hey Assistant, multicast to solarium, hello." Some time after detecting a portion of speech input 792b, device 700 displays suggestion 794d in user interface 794 (eg, left side of user interface 794). Here, suggestion 794d is a partial response to voice input 792b. Suggestion 794d is text that reads, "What is your business?" In some embodiments, device 700 outputs media containing an audible version of suggestion 794d.

In FIG. 7T, after displaying user interface 794 in FIG. 7S, device 700 displays suggestion 794e on the right side of user interface 794 and suggestion 794f on the left side of user interface 794 (and below suggestion 794e). Suggestion 794e is the letters that say "Hey Assistant, Multicast to Solarium, Hello" which are letters that correspond to the audible phrase contained in voice input 792b. Additionally, suggestion 794f is text that says "OK, your multicast has been sent to the solarium." In particular, "sunroom" is an identifier for a group of devices designated as sunroom 602e in FIG.

As shown in table 706 of FIG. 7T, after detecting audio input 792b, device 700 directs (eg, transmits) an audio multicast message to one or more devices designated to solarium 602e. Table 706 indicates that audio multicast messages (eg, "Hello") are directed to solarium speakers 656 and solarium speakers 658, and audio multicast messages (eg, "Hello") are not directed to other devices. show. In particular, voice input 792b included an identifier (eg, "sunroom"), while voice 792a did not. Accordingly, in some embodiments, when audio input is detected without an identifier, device 700 transmits an audio multicast message to a group of devices designated as a default group. In some embodiments, the default group is the group corresponding to Home (eg, with Home 602, etc.), as described above in connection with FIG. 7N.

In FIG. 7U, device 700 detects voice input 792c that includes the phrase "Hey Assistant, multicast to entertainment zone, hello." Some time after detecting a portion of speech input 792c, device 700 displays suggestion 794g in user interface 794 (eg, left side of user interface 794). Here, suggestion 794g is a partial response to voice input 792c. Suggestion 794g is a letter that reads, "What is your business?" In some embodiments, device 700 outputs media containing an audible version of suggestion 794g.

In FIG. 7V, some time after displaying user interface 794 of FIG. 7U, device 700 displays suggestion 794i on the left side of user interface 794 (and below suggestion 794h). Suggestion 794i is text that says "OK, your multicast has been sent to the entertainment zone." A zone is an identifier for a group of devices. Here, entertainment zone 660 is assigned to the group of devices corresponding to solarium 602e of FIG. 6 and the group of devices corresponding to living room 602d of FIG.

As shown in table 706 of FIG. 7V, after detecting audio input 792c, device 700 directs (eg, transmits) an audio multicast message to one or more devices designated as entertainment zones. Table 706 shows that an audio multicast message (eg, "Hello") is directed to living room speaker 646, living room speaker 648, sunroom speaker 656, and sunroom speaker 658, and an audio multicast message (eg, "Hello") is not directed to another device.

As shown in FIG. 7V, the device 700 also displays a multicast notification 708a indicating that a new audio multicast message has been received from the group of devices designated for solarium 602e. The new audio multicast message is a reply to the audio multicast message sent in Figure 7T.

In FIG. 7W, device 700 detects voice input 792d that includes the phrase "Hey Assistant Multicast I'm leaving soon."

In FIG. 7X, some time after detecting voice input 792d, device 700 displays suggestion 794j on the left side of user interface 794 and below suggestion 794i. Suggestion 794j is text that reads "Hey Assistant Multicast I'm leaving soon." This character corresponds to an audible phrase contained in voice input 792d. Here, voice input 792d does not include an identifier. Additionally, in FIG. 7X, device 700 also displays display 794k on the right side of user interface 794 and below suggestion 794j. Suggestion 794k is text that says "OK, your multicast has been sent to the solarium."

As shown in table 706 of FIG. 7X, after detecting audio input 792d, device 700 initiates a process (e.g., "I'm leaving soon") to send an audio multicast message to a group of devices designated for the solarium. ”). Specifically, the device 700 directs (eg, transmits) an audio multicast message to one or more devices designated to the solarium 602e without an audio input 792d that includes an identifier for the solarium. Therefore, because audio input 792d was detected within a predetermined time period after receiving multicast notification 708a, device 700 would instead select the default group (the group corresponding to home 602) for the next detected audio Intelligently direct multicast messages to solariums. In some embodiments, device 700 sends the next detected audio multicast message to the default group because audio input 702d was received within a predetermined amount of time after the previous multicast message was sent to the solarium. intelligently send it to the solarium instead. In some embodiments, two different zones can share at least one group of devices (eg, the same group of devices assigned to a particular room).

As shown in FIG. 7X, device 700 also displays multicast notification 708b indicating that an audio multicast message has been received from a group of devices designated for solarium 602e. Device 700 now receives this message at 10:19.

In FIG. 7Y, device 700 detects voice input 792e containing the phrase "Hey Assistant Multicast Let's go to dinner" and displays user interface 794 with suggestion 794l. Suggestion 794l reads, "What is your business?" In Figure 7Y, the time is now 10:30 (eg, 11 minutes have passed since the user interface of Figure 7X was displayed).

In FIG. 7Z, after displaying user interface 794 in FIG. 7Y, device 700 displays suggestion 794m on the right side of user interface 794 and suggestion 794n on the left side of user interface 794 (and below suggestion 794m). Suggestion 794m is the text that says "Hey, Assistant, Multicast, Let's go to dinner", which is the text that corresponds to the spoken phrase contained in voice input 792e. In addition, suggestion 794n is text that says "OK, I sent your multicast to 123 Main Street."

After detecting voice input 792e, device 700 directs a voice multicast message (eg, "Let's go to dinner") to one or more devices designated as home 602, as shown in table 706 of FIG. 7Z. (eg, transmit), but there is no voice input 792e containing the home 602 identifier. Now, more than the predetermined time period has passed since multicast notification 708b was received, so device 700 sends the next detected audio multicast message to the default group (home) instead of solarium 602e. 602) intelligently. In some embodiments, the device 700 sends the detected audio multicast message to the default group because no audio input 702e was received within a predetermined time after the previous multicast message was sent to the solarium 602e. intelligently send to

Further, as shown in table 706 of FIG. 7Z, the multicast message of FIG. 7Z (as opposed to FIG. 7N) indicates that the multicast setting of bedroom speaker 636 is turned off in response to detecting tap gesture 750q of FIG. 7Q. Therefore, it was not aimed at the bedroom speaker 636 .

Some time after displaying user interface 794 in FIG. 7Z, device 700 displays user interface 710 as shown in FIG. 7AA. In FIG. 7AA, device 700 detects tap gesture 750aa at a location corresponding to home setting affordance 928 above home location identifier 712 in FIG. 7AA.

In response to detecting tap gesture 750aa, device 700 displays user interface 797, as shown in FIG. 7AB. User interface 797 includes a group of people 797a who have one or more personal devices assigned to the Home application (eg, the group corresponding to Home 602a). User interface 797 also includes multicast grant affordance 797b. In FIG. 7AB, device 700 detects tap gesture 750ab at a location corresponding to multicast grant affordance 797b.

As shown in FIG. 7AC, in response to detecting the tap gesture 750ab, the device 700 displays a user interface 799 that includes multiple options for when audio multicast messages should be allowed to be sent to personal devices. do. In FIG. 7AC, the options include never option 779a, at home option 779b, and any time option 799c. In Figure 7AC, the optional time option 799c is selected. Accordingly, in some embodiments, device 700 is configured to receive multicast messages at any time regardless of device 700's location. In some embodiments, in response to detecting tap gesture 750c1 at a location corresponding to never option 779a, device 700 is configured to never receive multicast messages regardless of device 700's location. In some embodiments, in response to detecting tap gesture 750c2 at a location corresponding to at-home option 799b, device 700 may, while device 700 is at a location corresponding to or within home 602, Configured to receive only multicast messages (eg, within a threshold distance of home or whether the device is connected to one or more networks associated with home 602). In some embodiments, when device 700 is away from home at distance 670 and distance 680, device 700, such as computing device 674 and Jane's phone 684, if their distance is greater than a threshold distance, Do not receive audio multicast messages when selected in home option 799b. In some embodiments, one or more options are selectable and displayed, such as an option to only receive messages while the device is away from home. In some embodiments, when device 700 is connected to one or more networks associated with home 602 (eg, connected to a home Wi-Fi network), device 700 selects at-home option 799b. received an audio multicast message and device 700 is not connected to one or more networks associated with home 602 (eg, not connected to a home Wi-Fi network), device 700 , does not receive audio multicast messages even when at-home option 799b is selected (eg, regardless of device location).

7AD-7AE show exemplary user interfaces for displaying the user interfaces described above on a device having a screen larger than device 700, such as tablet 634. FIG. User interface 711 in FIG. 7AD corresponds to user interface 710 described above. Additionally, user interface 731 in FIG. 7AE corresponds to user interface 730 described above. However, any of the user interfaces described above can be displayed and transitioned on tablet 634 using one or more techniques similar to those described above for FIGS. 7A-7AC.

7AF-7AH show exemplary user interfaces for displaying the user interfaces described above on a device having a smaller screen than device 700, such as smartwatch 701. FIG. User interface 713 in FIG. 7AF corresponds to user interface 710 described above. Moreover, user interface 741 of FIG. 7AG corresponds to user interface 740 described above. Further, user interface 751 in FIG. 7AH corresponds to user interface 750 described above. However, any of the user interfaces described above can be displayed and transitioned on smartwatch 701 using one or more techniques similar to those described above for FIGS. 7A-7AC.

8A-8B are flow diagrams illustrating methods for sending voice messages, according to some embodiments. Method 800 is performed in a computer system (eg, 100, 300, 500, 634, 674, 700, 701, 850) in communication with a display generating component (eg, touch sensitive display). Some acts of method 800 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

As discussed below, method 800 provides an intuitive method for sending voice messages, according to some embodiments. The method reduces the cognitive burden of users sending voice messages, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, enabling users to send voice messages faster and more efficiently saves power and increases intervals between battery charges.

The computer system (802) displays, via a display generation component, a first user interface (eg, 730) (eg, a user interface for controlling a set of external devices). The first user interface includes a first selectable user interface object target (e.g., 726) (e.g., includes a display), and selection of the first selectable user interface object target comprises a first voice. A process for recording (e.g., via a microphone in a computer system) and transmitting (e.g., , user interfaces 730, 740, and 750) and the set of external devices is a first set of one or more external devices (e.g., 616) and a second set of one or more external devices. a set (eg, 646, 648). In some embodiments, while displaying the first selectable user interface object target, the computer system receives input corresponding to selection of the first selectable user interface object target; In response to receiving input corresponding to selection of one selectable user interface object target, the computer system generates a first set of one or more devices and a second set of one or more devices ( For example, displaying a user interface for recording and sending voice messages to a set of external devices. In some embodiments, the first set of one or more devices is different than the second set of one or more devices. In some embodiments, a user interface for controlling a set of external devices includes multiple representations corresponding to a first set of one or more devices and a second set of one or more devices. and/or displaying accessory control user interface object objects. In some embodiments, the plurality of accessory control user interface object targets corresponds to a first set of one or more devices and a second set of one or more devices is one or more selectable set of user interface object objects. In some embodiments, each of the plurality of accessory control user interface object targets, when selected, causes a respective accessory device of the set of external devices to perform one or more functions and/or a respective Cause an accessory device to change state. In some embodiments, the set of external devices is connected to the first set of one or more devices (eg, smart speakers) or the second set of one or more devices (eg, smart speakers). This includes any type of external electronic device (eg, cellular phone, tablet computer). In some embodiments, the set of external devices is a top-level location (or destination) (e.g., a first type of location) (e.g., a home location, a business, an office (a location at a top-level location ( For example, one or more regions (or zones) within an area and/or perimeter defined by a location (e.g., sub-location) defining one or more locations and/or a location of a top-level location; A location at the top level of the hierarchy, a location linked (eg, programmatically) to one or more locations (eg, sub-locations) and/or associated with one or more regions (or zones)). In some embodiments, the set of external devices includes a first group of devices (eg, a subgroup of devices) and a second group of devices (eg, a subgroup of devices). In configuration, sub-groups of devices are defined, identified, and/or linked to sub-locations of a top-level location (e.g., at the boundaries, regions, or perimeters of the top-level location and/or to the top-level location). (e.g., programmatically linked) and placed (e.g., by a user)) and/or within a top-level location (e.g., within a boundary, area, or perimeter of the top-level location and/or (e.g., a user defined, identified and/or linked) (e.g. a place defined by two or more sub-places) (within a top-level place (e.g. a living room within a top-level place and an entertainment zone that includes a dining room, a personal zone that includes two or more bedrooms, a living room, and a bathroom within a top-level location.In some embodiments, target groups are subgroups of devices. Associated and not a top-level group of devices.

The first user interface includes a first accessory control user interface object target (eg, one of 718) corresponding to the first external accessory device (eg, a selectable user interface object target); Accessory control user interface object target selection (e.g., 750f1, 750f2) commands to change the state of the first external accessory device (e.g., lock or unlock doors, turn lights on or off). (e.g. selection triggers the transmission of an instruction). In some embodiments, in response to receiving the input, the computer system initiates a process (e.g., selection causes transmission of the command) to send the command to change the state of the first external accessory device. and change the visual appearance of the first accessory control user interface object target from the first visual appearance to a second visual appearance that is different from the first visual appearance (e.g., pressed pressed state (or vice versa), from a first color to a second color, one or more difference characters (eg, "on to off" (or vice versa)).

The computer system, while displaying a first user interface (e.g., first user interface), receives an input (e.g., 750f3) corresponding to a request to display a second user interface (e.g., 760) ( For example, a single user input (e.g., a tap input on a selectable user interface object object or icon that navigates to a user interface for controlling a second set of one or more external devices) or a first Receive 804 a plurality of user inputs received while selectable user interface objects are displayed (e.g., a user interface for controlling a second set of one or more external devices). (eg, without controlling a first set of one or more external devices). In some embodiments, the computer system receives a request to display the second user interface when the computer system receives a selection of selectable user interface object targets for displaying the second user interface.

In response to receiving the request to display the second user interface, the computer system displays (806) the second user interface (eg, 760). The second user interface includes (e.g., including displaying) a second selectable user interface object target (812) (e.g., 776), wherein selection of the second selectable user interface object target is , sending the second voice message to the second one of the one or more external devices without sending the second voice message to the first set of one or more external devices (eg, 706 in FIG. 7M). Initiate a process (eg, user interfaces 780a, 784, and 786a) to record and transmit a set (eg, 646, 648). The second user interface is a second accessory control corresponding to a second external accessory device (eg, 642, 646, 648) (eg, not corresponding to the first set of one or more devices) A user interface object target (814) (e.g., one of 768) includes (e.g., a selectable user interface object target), and the selection of the second accessory control user interface object target (e.g., 750g1, 750g2) is , initiates the process of sending an instruction (eg, selection causes transmission of the instruction) to change the state of the second external accessory device. In some embodiments, in response to receiving the input, the computer system initiates a process (e.g., selection causes transmission of the command) to send the command to change the state of the second external accessory device. and change the visual appearance of the second accessory control user interface object target from the first visual appearance to a second visual appearance that is different from the first visual appearance (e.g., pressed pressed state (or vice versa), from a first color to a second color, one or more difference characters (eg, "on to off" (or vice versa)). In some embodiments, in response to receiving a request to display the second user interface, the computer system ceases displaying the first user interface and the first user interface object target. In some embodiments, the second selectable user interface object target replaces the first selectable user interface object target. In some embodiments, the second selectable user interface object target is displayed at the same location where the first selectable user interface object target was previously displayed. In some embodiments, the second user interface replaces the display of the first user interface. In some embodiments, the computer system selects a first set of one or more external devices from a second user interface (e.g., accessory control user interface object targets are not displayed, one or more devices corresponding to one of the first set) cannot be controlled (eg, an external device cannot change state). In some embodiments, the first user interface displays a set of one or more representations and/or accessory control user interface object objects corresponding to a first set of one or more external devices (one displaying (without displaying) a set of one or more representations corresponding to the second set of said external devices. In some embodiments, each of the accessory control user interface object targets causes the respective device to perform one or more functions or change the state of the respective device when selected. In some embodiments, while displaying the second selectable user interface object target, the computer system receives input corresponding to selection of the second selectable user interface object target; In response to receiving input corresponding to selection of two selectable user interface object targets, the computer system, without transmitting a voice message to the first set of one or more external devices, performs one or more display a user interface for recording and sending a voice message to a second set of external devices in . In some embodiments, while displaying a second selectable user interface object target, the computer system controls a first display of one or more external devices including displaying a selectable user interface object target. receive user input (e.g., swipe input) corresponding to a request to display a user interface for controlling one set of one (e.g., without controlling a second set of one or more external devices); , which, when selected, records and transmits a voice message to a first set of one or more external devices without transmitting the voice message to a second set of one or more external devices. start the process for In some embodiments, the voice message includes voice recorded by a computer system. In some embodiments, the process of recording and transmitting the voice message to the external device includes recording the voice message using a computer system (eg, using a microphone in communication with the computer system). In some embodiments, the voice message includes the audio of the voice of the user of the computer system. By providing a hierarchical user interface for controlling one set of controls, each user interface corresponding to a node in a respective hierarchy can be assigned to another user interface for controlling another respective set of devices. Provide visual feedback to the user regarding the set of devices in the hierarchy of controls that can be controlled via their respective user interfaces, along with visual feedback regarding how the user can intuitively navigate. . Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) It makes the user-system interface more efficient (by reducing user error and reducing user error), as well as reducing power usage by enabling users to use the device more quickly and efficiently. , to improve the battery life of computer systems.

In some embodiments, the first external accessory device (eg, 718b) is different than the second external accessory device (eg, 768c). In some embodiments, the first external accessory device is grouped under the "Home" group, but is not part of the "Dining Room" category (e.g., a light located in the bedroom), but is the second external accessory device. are external devices that are grouped under both the "Home" group, the "Dining Room" category (eg, a light located in the dining room). In contrast, in some embodiments, the first external accessory device is the same device as the second external accessory device (eg, they are the same lights placed in the dining room, or they are is the same smart speaker placed in the dining room).

In some embodiments, the first selectable user interface object target (eg, 726) (eg, while displayed as part of the first user interface) is via a display generation component ( on a display generating component), displayed at a first location (e.g., locations 726 and 776), and a second selectable user interface object target (e.g., 776) (e.g., of a second user interface). while being displayed as part of it) is displayed at a first location (eg, locations 726 and 776) via the display generating component (eg, on the display generating component). In some embodiments, input received while displaying the first user interface replaces display of the first selectable user interface object target with a second selectable user interface object target. Let By co-presenting the first selectable user interface object or the second selectable user interface object object when a predetermined condition is met, the user can select the selectable user interface object object. (e.g., which particular set of devices is selectable, which particular selectable user interface object object transmits the voice message of the record) of multicast can quickly locate selectable user interface object targets for improve the usability of computer systems by providing users with improved visual feedback (e.g., assisting users in providing appropriate input when operating/interacting with devices; reduce power usage by making the user-system interface more efficient (by reducing user error) as well as enabling users to use the computer system more quickly and efficiently; Improve computer system battery life.

In some embodiments, a first external accessory device (e.g., 646, 648) (e.g., a smart speaker) may be configured to play music using the first accessory control user interface object target. can be included in a set of external devices (e.g., a first external accessory device can be configured from a first user interface, audio (recorded by a computer system) can be a first selectable user interface transmitted to the first external accessory device using the object target). In some embodiments, the first user interface is a third accessory control user interface object target (eg, 718b) corresponding to a third external accessory device (eg, 642) (eg, a selectable user interface object). In some embodiments, selection of the third accessory control user interface object target changes the state of the third external accessory device (e.g., locks or unlocks doors, turns lights on or off). Initiate the process of sending an order (eg, selection triggering the sending of an order). In some embodiments, the third external accessory device (eg, 642) is not included in the set of external devices and selection of the first selectable user interface object target (eg, 776) is to a third external accessory device (eg, 706 at 7F). In some embodiments, the first user interface can be controlled (eg, turned on/off) but the first voice message is not sent (eg, an external accessory device plays the voice message). includes an accessory control user interface object object for some external accessory device (because it does not have a speaker for the device), and the first user interface can be controlled (e.g., turned on/off) and the first voice messages are sent (eg, because the external accessory device has a speaker for playback of the voice message). In some embodiments, the second user interface can be controlled (eg, turned on/off), but the second voice message is not sent (eg, an external accessory device plays the voice message). includes an accessory control user interface object target for some external accessory devices (because they do not have speakers for the device), and a second user interface can be controlled (e.g., turned on/off) and a second voice messages are sent (eg, because the external accessory device has a speaker for playback of the voice message). In some embodiments, a first set of one or more external devices (e.g., bedroom group devices to which voice messages are sent) are connected to a first external accessory device (e.g., bedroom group devices to which voice messages are not sent). group lights), and a second set of external devices (e.g., dining room devices to which voice messages are sent) includes a second external accessory device (e.g., dining room group lights to which voice messages are not sent). light). Therefore, in some embodiments, the voice message is not necessarily sent to all devices that can be controlled using the accessory control user interface object object; it is sent to each accessory device once it is selected. Cause one or more functions to be performed and/or change the state of the respective accessory device. In some embodiments, each accessory control user interface object target can be used to control (to perform one or more functions and/or change the state of the respective accessory device) The external device is included in a set of external devices (e.g., a first or second set of external devices), so that voice messages can be sent to the external device and separately control the external device be able to.

In some embodiments, a first user interface corresponds to a top level location (eg, 602) (eg, structure, home) and a second user interface corresponds to a first type (eg, 602a-602a). 602e) (eg, structure room, home dining room) corresponds to the first sub-location (eg, 602d). In some embodiments, the second set of one or more external devices (eg, 646, 648) corresponds to the first sub-location (eg, 602d) (eg, one or more external devices are designated as sub-locations of the first type). In some embodiments, the second set (eg, 646, 648) of one or more external devices (eg, any) is one or more external devices (eg, , 602a-602e), a second sub-location (eg, 602a, 602b, 602c, or 602e) of the first type (eg, any other room of the structure) (eg, , any sub-location other than the first sub-location). In some embodiments, the first user interface corresponds to a top level place and the second user interface corresponds to a sub-place (eg, a room in a home) of the top level place. In some embodiments, the first room and the second room are part of a zone (eg, a collection of rooms grouped together). In some embodiments, the second set of one or more external devices are devices identified (eg, by the user) located in the room to which the second user interface corresponds.

In some embodiments, the first user interface corresponds to a top-level location (eg, structure, home) and the second user interface corresponds to a first type (eg, 602a-602e) (eg, A third sub-location of a second type (entertainment zones (eg, 602d and 602e)) (eg, first zone, main floor) corresponding to multiple (eg, 602a-602e) sub-locations (eg, multiple rooms). correspond to the location. In some embodiments, the second set of one or more external devices is a second set of one or more external devices included in the plurality of sub-locations (eg, 602a-602e) of the first type. Based on the set, multiple rooms of the structure corresponding to a third sub-location (e.g., including a first room (e.g., dining room of the home) and a second room (e.g., bedroom of the home) ). In some embodiments, the first user interface corresponds to a top-level place and the second user interface corresponds to sub-places (eg, multiple rooms of a home) of the top-level place. In some embodiments, the second set of one or more external devices includes at least one device designated for the first room and at least one device designated for the second room. In some embodiments, the top-level location corresponds to at least one device that the third sub-location does not correspond to (e.g., Home includes some devices not contained within the third sub-location ).

In some embodiments, the second external accessory device is associated with a third sub-location of the second type (eg, entertainment zones (eg, 602d and 602e)) (eg, the first zone) ( corresponding). In some embodiments, the second external accessory device (e.g., 646, 648) is also a third sub-location of a second type (e.g., entertainment zones (e.g., 602d and 602e)) that is different than the third sub-location. 4 sub-locations (eg, second zone). In some embodiments, the second zone is a sub-location (eg, multiple rooms of a home). In some embodiments, the second zone is associated with an external accessory device not associated with the first zone. In some embodiments, the computer system can initiate processes for recording and sending messages to various zones. The first zone and the second zone can correspond to different sets of devices, some of which correspond to both the first zone and the second zone, some of which correspond to the first It corresponds to only one of the 1 zone or the 2nd zone.

In some embodiments, a second set of one or more external devices (eg, 646, 648) is associated with the first location (eg, 602d) (eg, room, sublocation, zone). recording the second voice message to the second set of one or more external devices without transmitting the second voice message to the first set of one or more external devices and the process for sending displays an indication (e.g., textual indication, graphical indication) of the first location (e.g., 782) (e.g., characters designating the name of the room or zone in which the voice message is to be sent) including doing In some embodiments, the process for recording and sending voice messages includes displaying a corresponding location indication to the external device to which each voice message is to be sent. In some embodiments, the first user interface corresponds to a top-level place (e.g., a structure such as Home) and the second user interface corresponds to a sub-place of the top-level place (e.g., a room in Home). , home zone). In some embodiments, upon displaying the second user interface, the computer system detects selection (eg, tapping on, pressing and holding) of a second selectable user interface object target. In some embodiments, in response to detecting selection of a second selectable user interface object target, the computer system selects a recording user interface object target (e.g., a selectable recording button) and a recording user Interface selection (e.g., tapping, pressing, and holding) causes recording of voice messages by recording sounds detected by the computer system, and sub-locations (e.g., rooms in which voice messages are sent). (eg, by replacing all or part of the second user interface) simultaneously with suggestions (eg, textual suggestions, graphical suggestions). In some embodiments, the second user interface includes sub-location suggestions. Provides users with visual feedback on the status of the process for recording and transmitting voice messages by displaying location suggestions associated with groups of devices to which voice messages are intended to be multicast. do. improve the usability of computer systems by providing users with improved visual feedback (e.g., assisting users in providing appropriate input when operating/interacting with devices; reduce power usage by making the user-system interface more efficient (by reducing user error) as well as enabling users to use the computer system more quickly and efficiently; Improve computer system battery life.

In some embodiments, as part of initiating the process for recording and transmitting a voice message to an external device, the computer system automatically responds to requests to initiate voice recording (e.g., Record a new voice message (without intervening user input) and display a recording user interface (eg, 741) that includes dynamic suggestions (eg, 744a) of the voice being recorded.

In some embodiments, the computer system detects selections (eg, taps, presses, and holds) of recorded user interface object targets. In response to detecting selection of the record user interface object subject, the computer system records the voice message. After recording the voice message (and before sending the voice message), the computer system offers the option of changing the external device to which the message is sent. In some embodiments, the computer system receives activation of a dropdown affordance and, in response, displays one or more sub-locations (each corresponding to a respective set of external devices). A computer system receives a selection of one of the one or more sublocations and a confirmation to send the voice message. In response, the computer system sends the voice message to each one of the one or more external devices corresponding to the selected sub-location rather than to the second set of one or more external devices corresponding to the second user interface. Send.

In some embodiments, the set of external devices includes personal mobile electronic devices (eg, 604, 634, 684) (eg, smart phones, cell phones, pocket watches). In some embodiments, personal mobile electronic devices are not included in the first set of one or more devices and are not included in the second set of one or more external devices. In some embodiments, the personal mobile electronic device is never assigned to any sub-location of the top-level location, but is a member of the top-level location. Thus, in some embodiments, selecting the first selectable user interface object target will cause the personal mobile electronic device (and any other personal mobile electronic device that is a member of the top-level place) to A process is initiated to record and send a voice message to a set of external devices, including:

In some embodiments, the second set of one or more external devices does not include a personal mobile electronic device (eg, 604, 634, 684) (eg, smart phone, cell phone, watch). In some embodiments, personal mobile electronic devices are not included in the first set of one or more devices and are not included in the second set of one or more devices. In some embodiments, the set of external devices includes a personal mobile electronic device, so voice messages sent from the first user interface are sent to the personal mobile electronic device (and optionally the home other mobile devices that it is part of). In some embodiments, voice messages sent from the second user interface (or any user interface corresponding to a sub-location) are not sent to the personal mobile electronic device (and optionally the home one). is not sent to other mobile devices that are part of it). In some embodiments, the computer system denies assigning a personal mobile electronic device to any sub-location of a top-level location, but allows a personal mobile electronic device to be assigned to a top-level location. do. Thus, in some embodiments, selecting the second selectable user interface object target will cause the personal mobile electronic device (or any other personal mobile electronic device that is a member of the top-level place) to A process of recording and transmitting a voice message to a second set of one or more devices, which is not included, is initiated.

In some embodiments, after displaying the second user interface, the computer system receives input (e.g., 750q) (e.g., user input, input on the settings user interface) and one or more external devices. disables sending voice messages to the second set (eg, 638) of . In some embodiments, in response to receiving input to disable sending voice messages to a second set of one or more external devices, the computer system selects a first selectable Sending a voice message to a first set of one or more external devices via selection of a user interface object target (e.g., 726) (e.g., via selection of a first selectable user interface object target) ), and sending voice messages to a second set of one or more external devices via selection of a second selectable user interface object target (e.g., 776). (eg 768a). In some embodiments, after disabling the transmission of voice messages to the second set of one or more external devices, navigating to the first user interface and selecting the first selectable user interface object Recording and transmitting a voice message to a first set of one or more external devices without transmitting the voice message to a second set of one or more external devices by receiving a selection of targets process is started. In some embodiments, in response to receiving input to disable sending voice messages to a second set of one or more external devices, the computer system causes the one or more external devices to from the set of external devices. In some embodiments, after disabling transmission of voice messages to a second set of one or more external devices, navigating to a second user interface and selecting a second selectable user interface object Receiving the subject selection does not initiate the process of recording and sending a voice message to any device. In some embodiments, after disabling transmission of voice messages to the second set of one or more external devices, the second user interface includes a second selectable user interface object target. (or including a non-selectable/disabled version of the user interface object object), thereby initiating the process of recording and transmitting voice messages to a second set of one or more external devices. Can not.

In some embodiments, the first user interface corresponds to a top-level place (eg, 602) (eg, a structure such as Home) and the second user interface corresponds to the top-level place (eg, 602 ) (eg, 602a-602d) (eg, Home Room, Home Zone). In some embodiments, the computer system (e.g., while not displaying the first user interface and not displaying the second user interface) communicates with the computer system (e.g., the computer system's microphone). At the communicating microphone, audio input (e.g., 792a-792e) including natural language requests of the digital assistant is received (808), where the audio input includes voice messages (e.g., including the term "multicast"), location identifiers (e.g., top-level location, sub-location), and voice messages (eg, received after an instruction to initiate the process of sending a voice message and/or received after a location identifier). In some embodiments, upon determining that the location identifier corresponds to a top-level location (eg, 782a), the computer system establishes a full-duplex voice communication channel without establishing a two-way voice communication channel (e.g., 782a). external devices for playback on a set of external devices, including a first set of one or more external devices and a second set of one or more external devices. Send 810 to a set of arbitrary devices (eg, multicast). In some embodiments, upon determining that the location identifier corresponds to a sub-location (eg, 792b, 792c), the computer system plays on a second set of one or more external devices, Transmit 812 a voice message (eg, multicast) without establishing a full-duplex voice communication channel with any device in the set of external devices in order not to play it back on the first set of external devices. . In some embodiments, the computer system enables a remote device (e.g., a service, a set of external devices) to distribute voice messages to external devices corresponding to each external device. Send the location identifier with the voice message.

In some embodiments, when part of initiating a process for recording and transmitting a voice message to an external device is included, the computer directs the recording user interface object target (eg, 736) (eg, selectable record button) to select a recording user interface object target (e.g., 750c) (e.g., tap and then press the end when the voice message is complete; while speaking the voice message); (press and hold, tap to allow the computer system to identify the end of the voice message) causes recording of the voice message by recording the sound detected by the computer system , and a send user interface object target (e.g., 756) (e.g., selectable send button), wherein selection of a second user interface object target (e.g., 750e) (e.g., tap) causes the computer system to Display (eg, by replacing all or part of the second user interface) a recording user interface (eg, 730, 740, 750) that also includes one that causes an external device to send a voice message. In some embodiments, the recording user interface also specifies the name of the location (e.g., top-level location, sub-location, room or zone where the voice message is sent) corresponding to the external device to which the voice message is sent. (e.g., textual suggestion, graphical suggestion) at the same time.

In some embodiments, the computer system communicates with one or more input devices including microphones. In some embodiments, as part of the process (e.g., user interface 730, 740, 750) for recording and transmitting the first voice message to a set of external devices, the computer system via a microphone. to receive a second audio input. In some embodiments, the computer system sends a first voice message to a set of external devices, the first voice message including a recording of the second audio input (eg, based on.

Note that the process details described above with respect to method 800 (eg, FIGS. 8A-8B) apply equally to the methods described below. For example, method 800 optionally includes any one or more of the various method features described below with respect to methods 1000 and 1200 . For example, the computer systems may be the same system. In another example, the voice message sent in method 800 can be media received in 1000 . In another example, the audio input sent in method 1200 may be the same media received in method 1000 . For the sake of brevity, these details are not repeated below.

9A-9X illustrate exemplary devices and exemplary user interfaces for displaying notifications, according to some embodiments. The user interfaces of these figures are used to illustrate the processes described below, including the process of FIG.

The phrases "multicast message" and/or "audio multicast message" are used to describe a device, using techniques similar to those described in FIGS. is generally used herein with respect to FIGS. 9A-9X to refer to multicast messages sent from. However, as used herein, multicast messages and/or audio multicast messages are not necessarily intended to include multicast communications sent using other features of device 700 .

FIG. 9A shows electronic device 700 displaying user interface 900 . 9A, electronic device 700 is a portable multifunction device having one or more components described above with respect to any one or more of devices 100, 200, 300, and 500. In FIG. With respect to the figures described below, in some embodiments device 700 is operating less than a threshold distance away from home 602 . However, in other embodiments, device 700 is operating more than a threshold distance away from home 602 .

As shown in FIG. 9A, user interface 900 is the settings user interface corresponding to the Home application, as described above (eg, FIGS. 6 and 7A-7AH). Here, the configuration user interface is a person (e.g., user) registered on device 700 and a person (e.g., user) recognized by the home application (e.g., group of people 797a in FIG. 7AB). This is the setting page of Kam Apple Seed. As shown in FIG. 9A, device 700 is configured to receive multicast notifications, indicated by multicast option 900a being enabled in FIG. 9A. Because multicast option 900a is enabled, multicast notifications are received by device 700 as shown in FIGS. 9B-9X. In some embodiments, device 700 detects tap gesture 950a at a location corresponding to multicast option 900a. In some embodiments, in response to detecting tap gesture 950a (or one or more subsequent tap gestures), device 700 changes multicast option 900a from enabled to disabled, and device 700 turns off this Not configured to receive multicast messages sent according to the flows described herein. Accordingly, in some embodiments, device 700 does not receive multicast messages when multicast option 900a is disabled.

Figures 9B-9H show multicast and non-multicast notifications displayed over a period of time while multicast option 900a is enabled. Specifically, FIGS. 9B-9H show that the display disappears or ceases (e.g., for a predetermined period of time) unless a new multicast notification directed to a particular target group of devices is received within a predetermined period of time. 12 shows that multicast announcement (eg, a multicast message sent using the methods described herein) directed to (eg, corresponding to) a particular target group of devices (eg, after). In some embodiments, a particular target group of devices may be a home (eg, home 602 in FIG. 6), a room (eg, dining room 602b, bedroom 602c in FIG. 6), or a zone (eg, in FIG. 7U). A group of devices designated as an "entertainment zone" as described in connection therewith.

For example, in FIG. 9B, at 10:00 AM, device 700 receives data indicating that updated content has been published by the news application.

In response to receiving the data, device 700 displays a user interface including news notification 902, as shown in FIG. 9B. In FIG. 9B, news notification 902 is not a multicast notification. Accordingly, news notification 902 is displayed regardless of whether multicast option 900a is enabled or disabled. In other words, the state of multicast option 900a does not affect news notification 902 because it is not a multicast notification.

At 10:02 AM, device 700 receives data (“home target group”) indicating that a new audio multicast message (eg, directed to devices including electronic device 700) has been sent to home 602 . In some embodiments, the data includes target group identifiers (eg, "home") as recipients of new audio multicast messages.

As shown in FIG. 9C, in response to receiving the data, device 700 displays multicast notification 904 representing the data received at 10:02 AM. In FIG. 9C, multicast notification 904 is displayed concurrently with news notification 902 . Multicast notification 904 includes application suggestion 904a, notification type suggestion 904b, sender suggestion 904c, goal suggestion 904d, and time of receipt suggestion 904e.

Application suggestion 904a indicates the application that provides the notification. Here, application suggestion 904a includes a graphical representation icon designated as the home application, similar to home icon 704a of FIG. 7A.

Notification type suggestion 904b indicates the notification type of the notification. where the notification type is multicast notification.

Sender suggestion 904c indicates the sender of the audio multicast message. Here, the sender is John, the person registered with the device that initiated the process of sending the audio multicast message (eg, the device in the home target group). In some embodiments, if the sender is a person, the identity of the sender is obtained from one or more devices that analyze audio input corresponding to the audio multicast message using one or more audio identification algorithms. established by In some embodiments, if the identity of the sender cannot be established as a person, the sender suggestion 904c is that the sender is a device (e.g., a "living speaker"), such as a device that transmits audio multicast messages. indicate. In some embodiments, when the sender is not established as a person, e.g., when the person is not registered with the sender device that initiated the process to send the audio multicast message, and/or when multiple When people are registered with the sender device that initiated the process for sending audio multicast messages, the person has not been established with the sender device.

Target suggestion 904d indicates the target group of devices to which the audio multicast message corresponds. Here, goal suggestion 904d is a home goal group (eg, "from home"). In some embodiments, the target group (corresponding to the audio multicast message) is determined by the sender of the audio multicast message, as described above in connection with FIGS. 7E-7F, 7J-7M, and 7S-7U A group that is identified. In some embodiments, the target group is based on one or more user inputs (eg, voice inputs (eg, 792a-729d) and/or gestures (eg, 750b and 750g3) to identify senders of voice multicast messages). In some embodiments, the target group is identified based on the initial audio multicast message when the return message is received at device 700. For example, device 700 determines that device 700 is in bedroom 602c. A return audio multicast message having a target group corresponding to bedroom 602c may be received in some circumstances, such as if the initial audio multicast message was previously directed to a group of devices designated as .

The time of receipt suggestion 904e indicates when the notification was displayed (or, in some embodiments, when data corresponding to the notification was received). Here, the time of reception suggestion 904e indicates that the notification was displayed at "NOW" or 10:02 AM.

At 10:09 AM, device 700 continues to display news notification 902 and ceases to display multicast notification 904, as shown in FIG. 9D. In some embodiments, notification 904 is not displayed on the user interface of device 700 at all after multicast notifications cease to be displayed. In some embodiments, notification 904 is not displayed on the user interface of FIG. 9D at any other time after multicast notifications cease to be displayed. In some embodiments, the device receives user input to review pending notifications (e.g., notifications that have not been affected by the user) and, in response, displays the multicast notification 904. Display new notifications 902 .

As shown in FIG. 9D, the device 700 may send the home target before the expiration (and/or within a predetermined time period) of the multicast notification 904 decay period (eg, 1 second, 30 seconds, 60 seconds, 300 seconds, etc.). Stop displaying the multicast notification 904 (even if the display is on) because no data corresponding to the new audio multicast message corresponding to the group has been received. On the other hand, the device 700 continues to display the news notification 902 if the news notification 902 is not a multicast notification and the predetermined period of time has elapsed (and the display is on).

Figures 9E-9H illustrate examples of techniques in addition to those described above in connection with Figures 9B-9D. In FIGS. 9B-9D, the decay period for multicast notifications (eg, predetermined time periods) is 5 minutes for illustrative purposes only. In some embodiments, the decay period is shorter or longer than 5 minutes.

At 10:10, device 700 receives three different sets of data. A first set of data indicates that a new audio multicast message corresponding to the home target group has been received. A second set of data indicates that a new audio multicast message has been received corresponding to the group of devices designated for bedroom 602c (the "bedroom target group"). A third set of data is a group of devices (“entertainment zone target group”) designated to an entertainment zone (“entertainment zone target group”) (e.g., devices designated to living room 602d and sunroom 602e). group combination) has been received.

In some embodiments, the first set of data is the initial audio multicast message or reply audio multicast message (eg, reply to the initial audio multicast message sent by device 700) corresponding to the home target group. represents In some embodiments, the second set of data and the third set of data are received in response to the first audio multicast message sent separately to the respective target groups; Represents a return audio multicast message, ie, an audio multicast message. In some embodiments, the second set of data and the third set of data are generated by device 700 as described above (eg, FIGS. 7M, 7T, 7V, and 7X). It is not the first message received from each target group because it has not been assigned to a zone or room target group and cannot receive the first message sent from those target groups.

As shown in FIG. 9E, device 700 displays multicast notifications 906a, 908a, and 910 in response to receiving a set of three data. Device 700 also continues to display news notification 902 . The notifications are displayed in the notification list, as shown in FIG. 9E. The oldest notifications are displayed at the bottom of the list (eg, news notifications 902) and the newest notifications are at the top of the list (eg, multicast notifications 910). Thus, although notifications 906a, 908a, and 910 were displayed approximately simultaneously, notifications 906a, 908a, and 910 were each displayed in the order shown in FIG. 9E. In some embodiments, the notifications are displayed in a different order or arrangement.

As shown in Figure 9E, notification 906a indicates that an audio multicast message was sent by "John" from the home target group. Notification 908a indicates that an audio multicast message was sent by "Jane" from the bedroom target group, and notification 910 indicates that an audio multicast message was sent by "Emma" from the entertainment zone target group. As shown in FIG. 9E, all three of the new multicast notifications contain a time receipt suggestion of "NOW" because the data was recently received (or recently displayed) at 10:10 AM. However, because the data was received (or the notification was displayed) at 10:00, the time receipt suggestion for news notification 902 is "10 minutes ago."

At 10:11 AM, device 700 receives data indicating that a new audio multicast message corresponding to the home target group has been received.

As shown in FIG. 9F, in response to receiving new data, device 700 receives data indicating a new audio multicast message corresponding to the home target group (eg, data received at 10:11), so the multicast Display multicast notification 906b before the decay period of notification 906a expires (eg, within a predetermined period of time after displaying multicast notification 906a or receiving the first data of FIG. 9E). In particular, device 700 has not updated its display of notifications 906a and 910, has not received data indicating new audio multicast messages corresponding to their respective target groups, and has not expired its decay period (e.g., , they have not been displayed for more than 5 minutes without receiving data corresponding to a new multicast message for their target group), so they continue to display these notifications.

Multicast notification 906b is an updated version of multicast notification 906a that device 700 discontinued in FIG. 9F. Looking again at FIG. 9E, multicast notification 906a indicates that John sent a multicast message corresponding to the home target group. Returning to FIG. 9F, multicast notification 906b also includes information (e.g., "John" in notification 906b) that John sent a previous multicast message corresponding to the home target group (as originally represented by notification 906a). include. In addition, multicast notification 906b includes information that the other has sent a multicast message corresponding to the home target group. Accordingly, multicast notification 906b contains information about two audio multicast messages: the sender ("John") of the first audio multicast message corresponding to the home target group; Contains information representing the sender (eg, "the other") of the corresponding second audio multicast message.

As shown in FIG. 9F, multicast notification 906b is displayed at the top of the list along with the current time reception suggestion since it is the most recent notification. In particular, although the time receipt indication of multicast notification 906b is changed in FIG. , multicast notification 906a was first displayed (or the data corresponding to multicast notification 906a was first received) at 10:10 in FIG. 9E. In some embodiments, multicast notification 906a is displayed concurrently with multicast notification 906b. In some embodiments, when multicast notification 906a is displayed concurrently with multicast notification 906b, the time-of-day receipt suggestion of multicast notification 906a is updated to read "NOW." In some embodiments, updating the time stamp of the multicast notification 906a extends the decay period of the multicast notification 906a. In some embodiments, receiving the data in FIG. 9F includes multicast notification 906a (or the content of multicast notification 906a, since multicast notification 906b includes content representing the audio multicast message represented by multicast notification 906a). Extends the decay period of

As shown in FIG. 9F, notifications 910, 908a, and 902 listed respectively from most recent to least recent are below multicast notification 906b. Additionally, the time receipt indications for notifications 910, 908a, and 902 are such that each reflects the length of time each was displayed (or the time elapsed since the data corresponding to each respective notification was received). , updated in FIG. 9F.

At 10:12 AM, device 700 receives data indicating that a new audio multicast message corresponding to the bedroom target group has been received.

As shown in FIG. 9G, in response to receiving the data, device 700 sends a multicast notification because data indicating a new audio multicast message corresponding to the bedroom target group was received (eg, data received at 10:12). Display multicast notification 908b before the decay period of 908b expires (eg, within a predetermined period of time after displaying multicast notification 906b or receiving the second data of FIG. 9E). In particular, device 700 has not updated notifications 908b and 910, no data has been received indicating new audio multicast messages corresponding to their respective target groups, and their decay periods have not expired, so notification 908b and 910 continue to be displayed. In some embodiments, multicast notification 908b is displayed using techniques similar to those described above in FIG. 9F for multicast notification 906b.

Multicast notification 908b is an updated version of multicast notification 908a that device 700 discontinued in FIG. 9G. Referring again to FIG. 9E, multicast notification 908a indicates that Jane sent a multicast message corresponding to the bedroom target group. Returning to FIG. 9G, multicast notification 908b indicates that two multicast messages were sent by Jane corresponding to the bedroom target group. Here, instead of multicast notification 908b containing new senders, multicast notification 908b displays content representing an increased amount of notifications sent by each sender compared to multicast notification 908a. In some embodiments, multicast notification 908a includes a representation of the amount of notifications received from the bedroom target group regardless of whether different senders have sent new multicast messages.

As shown in FIG. 9H, multicast notification 908b is displayed at the top of the list, and notifications 906b, 910, and 902 are displayed below multicast notification 908b in order from most recent to least recent.

As shown in FIG. 9H, at 10:15 AM, device 700 stops displaying multicast notification 910 because the decay period for multicast notification 910 has expired. In other words, the device 700 may detect the decay of the notification 910 within a predetermined time when the multicast notification 910 is first displayed (or when data indicating an audio multicast message corresponding to an entertainment target group is received). (before the end of the time period), it has not received data indicating a new audio multicast message corresponding to the target group (eg, "entertainment zone") of the multicast notification 910. Thus, referring to FIG. 9E, the device 700 receives data corresponding to respective target groups indicated in the first and second sets of data before each respective notification decay period expires, and On the other hand, since the data corresponding to the target group indicated in the third set of data was not received before the decay period of the multicast notification 910 expired, the content representing the first set of data (e.g., multicast notification 906b) and the content representing the second set of data (e.g., one of the multicasts of multicast notification 908b) and no longer displaying the content representing the third set of data. (eg, multicast notification 910).

9I-9P show user interfaces for playback of received audio multicast messages. Specifically, FIGS. 9I-9K illustrate scenarios in which multiple audio multicast messages are automatically played in response to a single input. Additionally, Figures 9L-9P illustrate scenarios in which one received audio multicast message is selected from a group of other received audio multicast messages and played. In FIG. 9H, device 700 detects tap gesture 950h at a location corresponding to multicast notification 906b.

In response to detecting tap gesture 950h, device 700 displays user interface 920, as shown in FIG. 9I. User interface 920 is a user interface for displaying the audio multicast message indicated in multicast notification 906b, which is the audio multicast message corresponding to the home target group (as indicated by "Multicast from Home" in FIG. 9I). be. User interface 920 includes graphical sender suggestions 924a and graphical sender suggestions 924b, each representing one of the audio multicast messages represented by multicast notification 906b. Here, graphical sender suggestion 924a is larger than graphical sender suggestion 924b, indicating in the center of the user interface that the audio multicast message corresponding to graphical sender suggestion 924a is currently selected for playback. Graphical sender suggestion 924a is an image of a speaker device representing data corresponding to the audio multicast message sent by "the other" received at 10:11 AM in FIG. 9F. Graphical sender suggestion 924b is John's image (i.e., 797a containing John A.'s image in Figure 7AB), and the data representative is the voice multicast sent by John received at 10:10 AM in Figure 9E. Respond to messages. In some embodiments, the voice messages are displayed in the order in which they were received (eg, graphical sender suggestion 924b and graphical sender suggestion 924b are toggled in FIG. 9E). User interface 920 further includes current sender suggestion 922a, which shows the name assigned to the speaker or device displayed in the center of the user interface (or maximum graphical sender suggestion). Here, the current sender suggestion 922a is that the speaker has the name "living room" and that an audio multicast message from the sender is currently playing or can be currently played by selection of playback affordance 926. indicate. In addition, the device 700 also indicates that the currently selected audio multicast message has a length of 10 seconds that can be played, and the remaining indication 928, and the selected audio multicast message is the home target. Display a count suggestion 930 representing two messages 1 out of the total number of audio multicast messages corresponding to the group. In some embodiments, when the setting is enabled to allow phonetic transcription of audio multicast messages, user interface 920 includes a textual representation of the currently selected audio multicast message.

As shown in FIG. 9I, device 700 receives data indicating that a new audio multicast message corresponding to the bedroom target group has been received, and displays multicast notification 940 in response to receiving the data. In some embodiments, device 700 updates the list of notifications based on this received data, as described in connection with FIG. 9L below. 9I, device 700 detects tap gesture 950i at a location corresponding to playback affordance 926. In FIG.

As shown in FIGS. 9J-9K, in response to detecting tap gesture 950i, device 700 begins playing media. As shown in FIG. 9J, device 700 plays the last received audio multicast message corresponding to the home target group (eg, received at 10:11 in FIG. 9F above). In FIG. 9J, while playing the currently selected audio multicast message (eg, the audio multicast message from “the other” received at 10:11), device 700 displays time display 932, Update suggestions 928 and 930. As shown in FIG. 9K, device 700 is currently playing the first audio multicast message corresponding to the home target group (eg, received at 10:10 from John in FIG. 9E above). In particular, after device 700 completes playing the last audio multicast message corresponding to the home target group (eg, without receiving additional input after detecting tap gesture 950i), device 700 automatically plays the first audio multicast message. Play a message.

As shown in FIG. 9L, some time after displaying the user interface in FIG. 9K, device 700 redisplays the list of notifications. Here, notification 902b has stopped being displayed. In some embodiments, the notification 902b indicates that a gesture was detected at a location corresponding to the notification and/or that any one or more of the audio multicast messages were played in FIGS. 9J-9K. has been discontinued. In some embodiments, notification 902b ceases to display because its decay period expired before the new multicast corresponding to the home target group was received (eg, at 10:16 AM).

In FIG. 9L, device 700 has also displayed multicast notification 908c and has ceased displaying multicast notification 908b. Multicast notification 908c is an updated version of multicast notification 908b updated to reference data corresponding to the audio multicast message received in FIG. 9I, represented by multicast notification 940 in FIG. 9I. Thus, multicast notification 908b contains content indicating that two multicast notifications corresponding to the bedroom target group were received, while multicast notification 908c contains content indicating that three multicast notifications corresponding to the bedroom target group were received. has been updated to include content indicating that In FIG. 9L, device 700 detects press and hold gesture 950l at a location corresponding to multicast notification 908c. In some embodiments, device 700 detects press and hold gesture 950l while the gesture continues to be detected (or held). In some embodiments, device 700 detects press and hold gestures after the gesture is released.

As shown in FIG. 9M, in response to detecting the press and hold gesture 950l, the device 700 displays the user interface 960 (eg, after the press and hold gesture 950l is completed while continuing to detect the press and hold gesture 950l). indicate. User interface 960 includes user interface elements similar to user interface 920 (as described above in connection with FIGS. 9I-9K). However, user interface 960 has a smaller area than user interface 920 . Device 700 can employ one or more techniques such as those described above with respect to user interface 920, while user interface 960 is the user interface for playing audio multicast messages from bedroom target groups. In some embodiments, in response to detecting user input 950m1 on open affordance 962, device 700 ceases displaying user interface 960 and displays a user interface similar in size to user interface 920. . In FIG. 9M, device 700 detects swipe gesture 950m2 at a location corresponding to user interface 960. In FIG.

As shown in FIG. 9N, in response to detecting the first gesture (eg, swipe gesture 950m2), device 700 sends the currently selected audio multicast message to the last received message corresponding to the bedroom target group. The second most recently received audio multicast message corresponding to the bedroom target group from the audio multicast message (eg, whose corresponding data was received in FIG. 9I) (eg, whose corresponding data was received in FIG. 9G). change to ). 9N, device 700 detects swipe gesture 950n at a location corresponding to user interface 960. In FIG.

As shown in FIG. 9O, in response to detecting a second gesture (eg, swipe gesture 950n), device 700 renders the currently selected audio multicast message to the second, last, and second corresponding to the bedroom target group. The first received audio multicast message corresponding to the bedroom target group (e.g., its corresponding data was received in FIG. 9E) from the received audio multicast message (e.g., its corresponding data was received in FIG. 9F). is changed). In FIG. 9O, device 700 detects tap gesture 950o at a location corresponding to user interface 960 (eg, on playback affordance 926).

As shown in FIG. 9P, in response to detecting the tap gesture 950o, the device 700 detects the first received audio message corresponding to the bedroom target group without initiating playback of another audio multicast message. Start playing a multicast message (eg, currently selected). In some embodiments, in FIG. 9P (or 9I-9J), device 700 detects tap gesture 950p at a location corresponding to multicast affordance 736 and responds to detection of tap gesture 950p to multicast affordance 736. At corresponding locations, device 700 begins recording media via one or more input devices as described above (eg, FIGS. 7C and 7I). In some embodiments, in FIG. 9P, device 700 detects one or more audio inputs, such as audio input 720a (as described above), to play to Jane corresponding to the bedroom target group. Send an audio multicast message.

9Q-9V illustrate exemplary user interfaces for displaying the user interfaces described above on a device having a smaller display than device 700, such as smartwatch 701. FIG. Any of the user interfaces described above may be implemented using one or more similar techniques to those described above for FIGS. and 980) (or tablet 634).

9W-9X illustrate exemplary user interfaces for displaying the user interfaces described above on a device such as computing device 674. FIG. As noted above, in some embodiments, computing device 674 communicates with one or more devices that transmit instructions to computing device 674 to display one or more user interfaces. Any of the user interfaces described above are displayed and transitioned on computing device 674 (eg, user interfaces 996 and 998) using one or more similar techniques to those described above for FIGS. 9A-9P. be able to.

FIG. 10 is a flow diagram illustrating a method for displaying notifications using a computer system, according to some embodiments. Method 1000 is executed in a computer system (eg, 100, 300, 500, 634, 674, 700, 701, 850) in communication with display generation components. Some acts of method 1000 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

As described below, method 1000 provides an intuitive way to display notifications. This method reduces the user's cognitive burden in reviewing notifications, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling users to access notifications faster and more efficiently saves power and extends the time between battery charges.

A computer system receives (1002) first data corresponding to a first medium.

The computer system, via the display generation component (eg, in response to receiving the first data), until the end of the first time period (eg, a predetermined first time period), displays the first A first notification (e.g., 904b, 904c) (e.g., a notification corresponding to first data) (e.g., a notification selectable to initiate a process of playing the first media) with an indication of media (e.g., 904b, 904c) For example, 906a, 908a, 910) are displayed (1004).

After receiving the first data (e.g., after initiating display of the first notification), the computer system receives second data corresponding to the second media (1006); (eg, 932 of 9J) is different from the first media (eg, 932 of 9K).

The computer system follows a determination that the second medium satisfies a set of one or more criteria (and, in some embodiments, the one or more criteria determine whether the second medium meets the first medium). and (2) determinations based on (1) the time the second data is received, and (2) the time the first data is received (e.g., the criteria that are met when directed to the same target group); 2 data is received within a predetermined time period after the first data is received), through the display generation component until the end of the second time period (e.g., the second display (1008) in response to receiving the data. In some embodiments, the second time period is based on receiving the second data. In some embodiments, the second time period is not based on receiving the first data. In some embodiments, the second time period is a predetermined second time period), the first media (e.g., 932 of 9K) (e.g., notifications corresponding to at least the first data) ( e.g., a notification selectable to initiate a process for playing at least the first media) (e.g., a notification indicating receipt of at least the first data) (e.g., an indication of 906b (e.g., "John '), a second notification (eg, 906b, 908b, 908c) with 908b (eg, '2'), where the end of the first time period is the end of the second time period. different. In some embodiments, the second notification is the first notification. In some embodiments, the second notification is different than the first notification (eg, the second notification includes (1) information corresponding to the first medium and (2) information corresponding to the second medium). It is a notification that combines the corresponding information). In some embodiments, the second notification is displayed after the first time period (eg, displayed first). In some embodiments, the length of time the information corresponding to the first media is displayed is extended when the reception of the second media is within a predetermined time period of the reception of the first media. In some embodiments, the second time period is determined based on the difference between when the first data is received and when the second data is received. In some embodiments, the media (e.g., first media, second media) are recorded on separate devices, and a group of devices including a computer system that receives the first data and the second data. A voice message sent to In some embodiments, the set of one or more criteria is the end of the display period of the first media (e.g., the duration for which the notification is valid, the duration for which the notification is displayed in the set of notifications) contains criteria that are met when the second media is received before . In some embodiments, the set of one or more criteria displays the first notification (eg, via activation of the first notification, via user input clearing the first notification) It contains criteria that are met when the second media is received before it stops playing. By dynamically displaying a second notification that includes a notification included in the first notification that is related to the second notification when certain criteria are met, the user is informed about the second notification and the first notification. It provides feedback on the relationship between the notifications, while also allowing the suggestions contained in the first notification to continue to be displayed even if the display of the first notification ceases. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use the computer system more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, the first notification (eg, 906a) is sent when the second data is received (eg, in accordance with a determination that the second media meets a set of one or more criteria). is displayed. In some embodiments, as part of displaying the second notification (eg, 906b), the computer system ceases displaying the first notification (eg, in FIG. 9E). In some embodiments, according to a determination that the second medium satisfies a set of one or more criteria (and in some embodiments, the one or more criteria are such that the second medium media), and determining that the second data was received within a predetermined time period after the first data was received. Stop showing notifications. By stopping displaying the first notification when the second notification contains information about the first notification, feedback is provided to the user regarding the information in the first notification without disrupting the user interface. . Improve device usability by providing users with improved visual feedback (e.g., assist users in providing appropriate input when operating/interacting with a computer system). It makes the user-system interface more efficient (by reducing user error), as well as reducing power usage by enabling users to use computer systems more quickly and efficiently. , to improve the battery life of computer systems.

In some embodiments, the first media is a first voice message (e.g., 932 of 9K) (e.g., recorded by an external device (using a microphone) and transmitted by the external device to a group of devices. voice messages directed to (eg, sent to) multiple external devices, including computer systems. In some embodiments, the second voice message is directed to (or sent to) a second plurality of external devices (eg, the same plurality of external devices, different from the plurality of external devices). 2 voice message. Displaying a notification associated with a multicast voice message provides visual feedback to the user as to the type of notification, indicating that the notification should decay and be responded to within a predetermined time period. is identified by the user. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use the computer system more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, the first notification (eg, 904) indicates that the first media (eg, 932 of 9K) is (eg, the first data corresponding to the media was sent) multiple external devices. group suggestions (eg, 904d). In some embodiments, the first data identifies a group of multiple external devices to which the first media is directed. In some embodiments, the plurality of external devices includes computer systems. By displaying a notification that includes an indication of the group of devices to which the first media is directed, the user is provided with feedback regarding the devices sending and/or receiving the playback message. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use the computer system more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, in response to receiving the first data, the computer system sends a first notification (e.g., 906a) a set of notifications (e.g., the notifications displayed in FIGS. 9B-9H) (e.g., a set of notifications displayed in response to user input requesting display of a set of notifications and accessible for display; list display of recent notifications), in some embodiments, upon reaching the end of the first time period, the computer system retrieves the first notification (e.g., 904) from the collection of notifications. Remove. In some embodiments, prior to the end of the first time period, according to receiving second data corresponding to the second medium, and the second medium is the set of one or more criteria , the computer system removes the first notification (eg, 906a) from the set of notifications and adds the second notification (eg, 906b) to the set of notifications. In some embodiments, according to receiving data corresponding to the media before the end of the first predetermined time period, and if the second media does not meet the set of one or more criteria. adding a second notification to the set of notifications without removing the first notification from the set of notifications according to the determination of . In some embodiments, the computer system removes the first notification from the set of notifications according to the display edge condition being met for the first notification. By choosing to delete a notification when no additional data related to the notification (e.g., no updates or no new media) has been received within a certain period of time, the device will allow the user to can be avoided from providing stale (e.g., old, irrelevant) notifications to users that clutter the UI without having to manually delete the . Improve the usability of a computer system (e.g., operate the computer system/computer system make the user-system interface more efficient (by helping the user to provide appropriate input and reduce user error when interacting with the Reduce power usage and improve computer system battery life by enabling the use of

In some embodiments, the first data identifies a target group of multiple external devices to which the first media is directed (eg, transmitted). In some embodiments, correspond to the first media (eg, and initiate a process to play the first media (eg, the first voice message) when the notification is activated) While displaying the notifications (eg, 906a, 908a, 910) (eg, the first notification), the computer system receives (1010) third data corresponding to a third medium. In some embodiments, in response to receiving 1012 third data corresponding to the third media, and in accordance with a determination that the third media satisfies a set of one or more criteria (some In an embodiment of , the one or more criteria include criteria that are met when the media is directed to the same target group of devices as the first media), the computer system controls the first media (e.g., and Upon activation of the notification, a corresponding notification (eg, first notification, second notification) is displayed, which initiates a process for playing the first media (eg, first voice message). Extend the duration (1014) while (eg, 906b in FIG. 9F). Some embodiments respond 1012 to receiving third data corresponding to the third media, and in accordance with a determination that the third media does not meet a set of one or more criteria (e.g. , the third media was directed to a different set of target devices as compared to the first media), the computer system will display the notification corresponding to the first media (e.g., FIG. 9F 908a, 910) stop extending the duration (1016). In some embodiments, when the computer system receives additional data corresponding to media meeting the set of one or more criteria, the device responds by displaying the media meeting the set of one or more criteria. Extends the duration while the notification is displayed via the display generation component, unless the previous notification that was sent has already ceased to be displayed based on reaching the end of the time period. continue. Thus, when a voice message is received directed to the same set of target devices (e.g., external devices), the notification for that chain of voice messages indicates that the most recently received voice message is the corresponding one for display. It will continue to be displayed until the predetermined time period is reached. Eliminates the need for users to manually update/delete notifications by choosing whether to extend the notification's decay period based on whether updated information associated with the notification has been received for a given period of time , the device can provide new (eg, new, relevant) notifications to the user while avoiding providing stale notifications (eg, old, irrelevant) to the user that clutter the UI. Improve the operability of a computer system (e.g., operate the computer system/computer system make the user-system interface more efficient (by assisting the user in providing appropriate input and reducing user error when interacting with the Reduce power usage and improve computer system battery life by enabling the use of

In some embodiments, upon determining that the second media satisfies the set of one or more criteria, the second notification (eg, satisfies the set of one or more criteria) is sent to the second Includes an indicator (eg, number) based on (eg, showing) the number of media (eg, 908a and 908b), including the first media represented by the notification (eg, the number of voice messages). In some embodiments, each subsequent notification after the first notification in a chain of notifications includes an indication of the number of media (eg, number of voice messages) represented by the displayed notification. For example, if the notification is the fifth notification in the chain, the notification indicates that the notification corresponds to 5 received media (eg, 5 voice messages). In some embodiments, following a determination that the second media does not meet the set of one or more criteria, the second notification (eg, meeting the set of one or more criteria) is sent to the second does not include an indicator (eg, a number) based on (eg, indicating) the number of media that includes the first media (eg, the number of voice messages) represented by the notification. By updating indications of the number of media items (audio multicast messages (e.g., audio messages multicast to one or more devices)) received from the device's target group, the current state, the device's adhered goals Feedback is provided to the user regarding the number of media items received for the group. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use computer systems more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, the second notification includes the first media suggestion. In some embodiments, while displaying the second notification (eg, 602b), the computer system receives input directed to the second notification (eg, 950h). In some embodiments, in response to receiving input directed to the second notification (eg, and in accordance with determining that the second media meets a set of one or more criteria), the computer system: , initiate a process for playing the first media and the second media (eg, 920, 922 in FIGS. 9I-9J) (eg, playing the first media and the second media, the first display a user interface for initiating playback of media and then secondary media). In some embodiments, initiating the process for playing the first and second media automatically (e.g., in response to receiving input directed to the second notification without user input intervention). ), playing the first media and then the second media, and displaying a user interface (eg, 741) for playing the first media. In some embodiments, playing the first media in response to receiving input directed to the second notification and in accordance with determining that the second media does not meet the set of one or more criteria. As part of initiating a process to play the second media without playing (e.g., displaying a user interface that plays the second media and does not play the first media), the computer system: Begin playing the second media, then do not automatically play the first media. In some embodiments, the computer system is directed to the first notification while displaying the first notification following a determination that the second media does not meet the set of one or more criteria. receiving the input directed to the first notification and, in response to receiving input directed to the first notification, initiating a process for playing the first media without playing the second media.

In some embodiments, the second media satisfies a set of one or more criteria (eg, the second media is directed to the same external device as the first media). In some embodiments, while displaying the second notification (eg, 908c), the computer system receives input directed to the second notification (eg, 950l). In some embodiments, in response to receiving input directed to a second notification (eg, in accordance with determining that the second media meets a set of one or more criteria), the computer system: displaying a user interface for playing the first media (e.g., 960) (e.g., as part of initiating a process for playing the first media and the second media); , the user interface for playing the first media includes a record user interface object target (eg, a selectable record button). In some embodiments, a recording (eg, 736) user interface object target selection (eg, 950p) causes the first data to be directed (eg, transmitted) to a plurality of external devices. following a determination to identify a first group (e.g., bedroom devices) of a plurality of external devices for sending a voice message to a first group (e.g., and not a second group) of a plurality of external devices. Start a process (eg, 730, 740, 750). In some embodiments, selection of a recording (eg, 736) user interface object target (eg, 950p) causes the first data to be directed (eg, transmitted) to the first media. Upon determining to identify a second group of external devices (e.g., home devices) different from the group, the computer system transmits the voice message to the second group of external devices (e.g., the first start a process (e.g., 780a, 784, 786a) to send to a non-group). In some embodiments, the computer system detects selection (e.g., tapping, pressing, and holding) of a recording user interface object target and, in response to detecting selection of a recording user interface object target, , to initiate the corresponding action. In some embodiments, the set of external devices is a top-level location (or destination) (e.g., a first type of location) (e.g., a home location, a business, an office (a location at a top-level location ( For example, one or more regions (or zones) within an area and/or perimeter defined by a location (e.g., sub-location) defining one or more locations and/or a location of a top-level location; A location at the top level of the hierarchy, a location linked (eg, programmatically) to one or more locations (eg, sub-locations) and/or associated with one or more regions (or zones)). In some embodiments, initiating the process for sending the voice message includes recording the voice message by recording sound detected by the computer system. The user interface for playing the first media includes suggestions (eg, textual suggestions, graphical suggestions) of groups (eg, top-level locations, sub-locations) (eg, names of rooms or zones in which voice messages are sent). characters that specify ).

In some embodiments, while displaying a user interface (e.g., 960) for playing the first media (e.g., a process for playing the first media and the second media is initiated) while displaying), the computer system receives user input (e.g., via one or more input devices using a touch-sensitive surface) (e.g., 950i, 950m2 , 950n). In some embodiments, in response to receiving user input and in accordance with determining that the user input (e.g., 950i) was directed to selecting a playback user interface object target (e.g., to play the first media). A tap on a play button displayed as part of the user interface), the computer system plays the first media (and, optionally, automatically thereafter plays the second media and switch to the user interface to play the media). In some embodiments, in response to receiving user input, and the user input is a swipe gesture (eg, 950m2, 950n) directed to a user interface for playing the first media (eg, the first A left swipe gesture on the user interface for playing the media), the computer system displays the user interface for playing the first media, and displays the user interface for playing the second media. (eg, including a play button to play the second media without playing the first media). In some embodiments, in response to receiving user input and in accordance with determining that the user input (e.g., 950i) was directed to selecting a playback user interface object target (e.g., to play the first media). tap on a play button displayed as part of the user interface), the computer system plays the second media (e.g., the most recently received media corresponding to the notification), then plays the first media do.

In some embodiments, a user interface (e.g., FIG. 15B 720 of ~15E) and while playing the first media, the computer system receives input. In some embodiments, in response to receiving user input and directing the user input to a first location on the user interface (eg, location 736 on user interface 720) for playing the first media. the computer system to transmit and record new media (e.g., one or more audio multicast messages) (and, optionally, automatically record audio for transmission). display the user interface (eg, 741). In some embodiments, in response to receiving user input and in accordance with determining that the user input was directed to a second location (e.g., location 1534) on the user interface for playing the first media. , the computer system displays a user interface for playing the first media (e.g., corresponding to 1524a) to a display of a user interface for playing the second media (e.g., corresponding to 1524b). As a replacement, the second location on the user interface for playing the first media is different than the first location on the user interface for playing the first media. Upon determining that user input is directed to a first location and is a tap input, the computer system detects subsequent input (e.g., at the same location as (or near) the first location). Automatically record audio (eg, via one or more microphones) until subsequent input) is detected. Upon determining that the user input is directed to the first location and is a press and hold gesture, the computer system outputs sound (eg, via one or more microphones) until an end of the user input is detected. Record automatically. In some embodiments, while displaying the second notification, the computer system receives input directed to the second notification (eg, 950h). In some embodiments, in response to receiving input directed to the second notification, the computer system ceases displaying the second notification. In some embodiments, when the second notification is activated, the second notification responds to user input (e.g., requesting to display a set of notifications accessible to the user for display). A listing of recent notifications received by the computer system displayed in response) removed from the set of notifications. In some embodiments, the collection of notifications continues to include other notifications (eg, email notifications, instant messaging notifications). By opting to remove notifications when they are interacted with, the device can send stale notifications (e.g., old, irrelevant) to the user cluttering the UI without requiring the user to manually delete the notification. allow you to avoid providing Improve the operability of a computer system (e.g., operate the computer system/computer system make the user-system interface more efficient (by assisting the user in providing appropriate input and reducing user error when interacting with the Reduce power usage and improve computer system battery life by enabling the use of

In some embodiments, a computer system includes multiple external devices (e.g., home to display a user interface for configuring whether a computer system is included in a group of devices that receive voice messages directed to external devices of the The user interface of a number of external devices (e.g., A first option (e.g., 799c) that includes (e.g., always) the computer system in a group of (e.g., devices that receive voice messages directed to external devices in the home), and the current physical location of the computer system. Multiple external devices (e.g., voice messages directed to home external devices) that receive data corresponding to location-directed media regardless of location (e.g., as determined by the computer system's location sensor). a second option (e.g., 799a) that does not (e.g., never) include the computer system in the group of receiving devices).In some embodiments, the user registers to receive data corresponding to the media. The computer system can be configured to enable/disable the display of notifications based on received data corresponding to/unregistering or media.

In some embodiments, a computer system includes multiple external devices (e.g., home display a user interface for configuring whether the computer system is included in a group of external devices receiving data corresponding to a group of devices receiving voice messages directed to the external devices of the , the user interface for configuring whether a computer system is included in a group indicates that the computer system (e.g., 700) currently corresponds to a location (e.g., within a geofenced perimeter of the home, a specific A plurality of external devices that receive data corresponding to location-directed media (e.g., home external devices) that receive data corresponding to location-directed media when at a physical location within a room (as determined by the computer system's location sensor). computer system (e.g., 700) in a group of devices (devices that receive voice messages that receive voice messages), and the computer system (e.g., 700) currently corresponds to a location (e.g., within a geofenced perimeter of a home, A plurality of external devices (e.g., outside the home) that receive data corresponding to location-directed media when not at a physical location within a particular room of the home (as determined by the computer system's location sensor). a third option (eg, 77b) that does not include the computer system (eg, 700) in the group of devices that receive voice messages directed to the device (eg, 77b). is at the location, input can be provided to configure the system to only receive data (and/or display only notifications) corresponding to media directed to a group of devices at the location (e.g., a computer system is at home, get only messages destined for the home; when the computer system is in the room, get only messages destined for the room).

Note that the process details described above with respect to method 1000 (eg, FIG. 10) are also applicable to the methods described below/above in a similar manner. For example, method 1000 optionally includes any one or more of the various method features described above/below with respect to methods 800 and 1200 . For example, the computer systems may be the same system. In another example, the voice message sent in method 800 can be media received in method 1000 . In another example, the audio input sent in method 1200 may be the same media received in method 1000 . For the sake of brevity, these details are not repeated below.

11A-11R illustrate electronic devices and exemplary user interfaces for managing visual displays. The user interfaces of these figures are used to illustrate the processes described below, including the process of FIG.

11A-11R illustrate that when device 1100 determines that a portion of the audio input should be sent to one or more external devices as an audio multicast message, device 1100 may select a first type (eg, single color and the device 1100 cannot determine that a portion of the audio input should be sent as an audio multicast message to one or more external devices (e.g., determines that no portion of the audio input should be sent as an audio multicast message to one or more external devices), device 1100 has visual content of the second type (e.g., multi-colored lights) Demonstrate a scenario for displaying visual instructions. The first and second types of visual content are described below for illustrative purposes only. The first and second types of visual content provide a visual indication to the user whether spoken audio input is potentially multicast to one or more external devices.

FIG. 11A shows electronic device 1100 in position on a table in front of a person. Device 1100 is a smart speaker and includes one or more features of device 580 . For example, device 1100 includes one or more displays 1102 that include one or more components of one or more displays 580E. Display 1102 includes a plurality of LEDs. In some embodiments, device 1100 includes one or more features of devices 100, 300, or 500. FIG. In FIG. 11A, device 1100 is associated with home 602 and living room 602d for illustrative purposes only. Thus, in FIG. 11A, device 1100 is one of living room speakers 646 and 648 as described above in connection with FIG. In some embodiments, device 1100 is associated with another room in home 602 (or another home).

As shown in FIG. 11A, device 1100 is in an inactive state (eg, sleep state, low power state, state in which device 1100 is not displaying visual content via display 1102), thereby causing display 1102 does not display visual content (eg, visual instructions). In some embodiments, the LEDs of display 1102 are not illuminated when device 1100 is not displaying visual content.

11A-11L illustrate device 1100 displaying visual content in response to receiving audio input including information multicast to one or more target groups of devices. In some embodiments, one or more similar techniques described above in connection with device 700 for receiving audio input of FIGS. 7M-7N and 7S-7Z receive audio input, as described below. It can be applied to device 1100 (or vice versa).

11B-11F illustrate device 1100 receiving audio input from a person standing in front of device 1100. FIG. The voice input received by the device 1100 of FIGS. 11B-11F is "Hey Assistant Multicast what music would you like to play in the living room?" The speech input received in FIGS. 11B-11F may be received spontaneously or otherwise. 11B-11F are shown to provide exemplary embodiments of how device 1100 may respond to different portions of audio input.

11B, device 1100 receives audio input portion 1150b from a person standing in front of device 1100 via one or more microphones of device 1100. In FIG. Here, the person is uttering the phrase "Hey Assistant".

As shown in FIG. 11B, device 1100 is in an active state (e.g., awake state, power state, display 1102 displays visual content). In FIG. 11B, the device 1100 displays visual content that is multi-colored (e.g., display a visual indication) (e.g., light). In FIG. 11B, device 1100 does not determine that some of the audio input should be sent as an audio multicast message to one or more external devices, so device 1100 displays visual content that is in multiple colors.

11C, device 1100 receives audio input portion 1150c from a person standing in front of device 1100 via one or more microphones of device 1100. In FIG. Here, the person is uttering the phrase "multicast".

In response to receiving audio input portion 1150c, device 1100 remains active, as shown in FIG. 11C. In FIG. 11C, device 1100 determines that some of the audio input should be sent as an audio multicast message to one or more external devices, so device 1100 sends a single color (eg, any other 1102 black disk in FIG. 11B) without the type of . Here, the device 1100 detects an instruction (eg, “multicast”) phrase indicating that the person is issuing a multicast command (eg, a command to multicast a voice message), so the device 1100 detects the voice input. Determine that the part should be sent (multicast) as an audio multicast message.

11D, device 1100 receives a voice input portion 1150d from a person standing in front of device 1100 via one or more microphones of device 1100. In FIG. Here, the person is uttering the phrase "what kind of music."

In response to receiving audio input portion 1150d, device 1100 remains active, as shown in FIG. 11D. In FIG. 11D, device 1100, based on a determination that a portion of the audio input should be sent to one or more external devices as an audio multicast message (eg, as described above with respect to FIG. 11C ) continue to display visual content that is a single color. In some embodiments, device 1100 directs some of the audio input to one or more devices designated as home 602 (eg, for reasons similar to those described above in connection with FIGS. 7M-7N). Determine that it should be sent. In some embodiments, device 1100 is a device based on voice input where some of the voice input corresponds to a group identifier (e.g., "home", "living room", "entertainment zone", "to dining room"). determines one or more devices to transmit a portion of the audio input based on whether they are designated in the group of .

11C, the pattern of visual content displayed by the device 1100 of FIG. 11D is different than the pattern of visual content displayed by the device 1100 of FIG. 11C. As shown in FIGS. 11C-11D, device 1100 displays different visual content via display 1102 in response to a person's voice. In other words, device 1100 changes the pattern of visual content because "multicast" and "what kind of music" are different phrases. In some embodiments, device 1100 adjusts the speed, sharpness, volume, pitch, tone, and/or rhythm of portions of audio content such that different portions of audio content generate different patterns of visual content. change the pattern of the visual content based on (e.g. beats); In some embodiments, the device 1100 adjusts the saturation, brightness, brightness, and/or intensity of visual content based on the speed, sharpness, volume, pitch, tone, and/or rhythm (e.g., beats) of a portion of audio content. Change size or position. In some embodiments, the device 1100 does not change patterns of visual content and/or react to human speech.

11E, device 1100 continues to receive audio input portion 1150e via one or more microphones of device 1100 from a person standing in front of device 1100. In FIG. Here, the person is uttering the phrase "what you want to play".

In response to receiving audio input portion 1150e, device 1100 remains active, as shown in FIG. 11E. In FIG. 11E, device 1100 continues to display visual content that is a single color based on the determination that a portion of the audio input should be sent to one or more external devices. Additionally, because audio input portion 1150e is different from audio input portion 1150d, device 1100 changes the pattern of visual content.

11F, device 1100 continues to receive audio input portion 1150f from a person standing in front of device 1100 via one or more microphones of device 1100. In FIG. Here the person is uttering the phrase "In the living room?"

As shown in FIG. 11F, in response to receiving audio input portion 1150f, device 1100 remains active. In FIG. 11F, device 1100 continues to display visual content that is a single color based on the determination that a portion of the audio input should be sent to one or more external devices. In addition, device 1100 changes the pattern of visual content because audio input portion 1150f is different from audio input portion 1150e.

In FIG. 11F, device 1100 detects the edges of the audio input received in FIGS. 11B-11F. 11F, in response to detecting the end of the audio input, device 1100 transmits one or more portions of the audio input as an audio multicast message to one or more external devices. In some embodiments, the audio multicast messages include 1150c-1150d (eg, "What music do you want to play in your living room?"). In some embodiments, due to the determination that voice input portion 1150b and/or 1150c corresponds to the command portion of the voice input, the voice multicast message includes voice input portion 1150b (“Hey, Assistant”) and/or 1150c ( "multicast"). In some embodiments, when the input includes a portion corresponding to a target group identifier (e.g., "home" or "living room"), the target group identifier (not the portion of the speech input corresponding to the target group identifier) is , is transmitted (eg, as plain text) with an audio multicast message, but is not output as audio by one or more external devices. In some embodiments, audio multicast messages output by one or more external devices sound like a person (which is a played recording of the person speaking the audio input) and are computer generated via the device. not a speech.

As shown in FIG. 11G, device 1100 sends a portion of the voice input to a target group identifier corresponding to home 602 (eg, "123 Main Street" as described above with respect to home location identifier 712). provides (eg, outputs) an output 1154g indicating that Upon outputting output 1154g, device 1100, upon determining that a portion of the audio input (eg, received in FIGS. 11B-11F) should be sent as an audio multicast message to one or more external devices, causes device 1100 to , to display visual content that is a single color. In some embodiments, device 1100 displays multi-color visual content when outputting audio confirmation 1154g regardless of this determination.

As shown in FIG. 11H, device 1100 changes to an inactive state. 11I, device 1100 receives data corresponding to an audio multicast message from Jane via any one or more of the external devices designated as home 602. In FIG.

As shown in FIG. 11I, in response to receiving an audio multicast message (and without (independently) receiving user input), device 1100 provides output 1154i which is an introductory tone or chime. do. As shown in FIG. 11J, after providing output 1154i, device 1100 provides output 1154j corresponding to the audio multicast message received in FIG. 11I.

As shown in FIGS. 11I-11J, device 1100 displays visual content that is a single color when providing output 1150i and output 1150j. Device 1100 displays visual content that is a single color because device 1100 is outputting an audio multicast message. As shown in FIGS. 11I-11J, the device 1100 also displays visual content that is the same pattern. Thus, in FIGS. 11I-11J, device 1100 does not change visual content based on the different content of output 1150i and output 1150j. In some embodiments, device 1100 changes visual content based on different content of output 1150i and output 1150j. In some embodiments, device 1100 provides output 11541150i and/or output 1150j at the current volume level of device 1100 (eg, system volume level). In some embodiments, one of output 1154i and/or output 1150j is provided at the volume level of device 1100, while the other is not. In some embodiments, device 1100 receives multiple audio multicast messages within a period of time and outputs the same (or different) introductory tone before each of the audio multicast messages is played. do. In some embodiments, the device 1100 receives multiple audio multicast messages within a period of time, and prior to each group (eg, grouped by the device's target group) of audio multicast messages being played. , outputs the same (or different) introductory gradation.

As shown in FIG. 11J, the voice message provided by device 1100 says, "Jane replied from Home: How's Jazz?" Here, the voice message provided by device 1100 includes the target identifier, and in some embodiments does not.

11K, device 1100 receives a voice input portion 1150k from a person standing in front of device 1100 via one or more microphones of device 1100. In FIG. Here, the person is uttering the phrase "Hey Assistant Reply, Exactly".

As shown in FIG. 11K, device 1100 determines that a portion of audio input portion 1150k should be sent as an audio multicast message to one or more external devices, so device 1100 displays a single color (eg, 11K 1102 black disc). In some embodiments, after device 1100 provides output 1150j (eg, the audio multicast message received in FIG. 11J), audio input portion 1150k is within a predetermined time (eg, 1, 2, 30, 60 seconds). , the device 1100 makes this determination. In some embodiments, voice input portion 1150k includes the phrase "reply" and/or includes the phrase "reply" within a second predetermined time period after device 1100 provides output 1150j. , the device 1100 makes this determination. In some embodiments, the second predetermined period of time is greater than the first predetermined period of time.

As shown in FIG. 11L, device 1100 changes to an inactive state.

11M-11R illustrate device 1100 displaying visual content in response to receiving audio input that does not include information that is multicast to one or more target groups of devices.

11M-11O illustrated device 1100 receiving voice input from a person standing in front of device 1100. FIG. The voice input received by the device 1100 of FIGS. 11M-11O is "Hey Assistant, play jazz music in the living room." The speech input received in FIGS. 11M-11O may be received spontaneously or otherwise. Figures 11M-11O are shown to provide exemplary embodiments of how device 1100 may respond to different portions of audio input.

11M, device 1100 receives audio input portion 1150m from a person standing in front of device 1100 via one or more microphones of device 1100. In FIG. Here, the person is uttering the phrase "Hey Assistant". As shown in FIG. 11M, in response to receiving audio input portion 1150m, device 1100 changes to an active state using one or more techniques similar to those described above in FIG. 11B. Additionally, because the device 1100 does not determine that some of the audio input should be sent as an audio multicast message to one or more external devices, the device 1100 may use visual content that is multi-colored (e.g., lights). display.

11N, the device 1100 receives an audio input portion 1150n from a person standing in front of the device 1100 via one or more microphones of the device 1100. In FIG. Here, the person is uttering the phrase "play jazz music". As shown in FIG. 11N, in response to receiving audio input portion 1150n, device 1100 does not determine that the portion of the audio input should be sent as an audio multicast message to one or more external devices. Device 1100 displays visual content (eg, lights) that are multiple colors.

11O, the device 1100 receives an audio input portion 1150o from a person standing in front of the device 1100 via one or more microphones of the device 1100. In FIG. Here, the person is uttering the phrase "in the living room." As shown in FIG. 11O, in response to receiving audio input portion 1150o, device 1100 does not determine that the portion of the audio input should be sent as an audio multicast message to one or more external devices. Device 1100 displays visual content (eg, lights) that are multiple colors.

11M-11O, device 1100 visual content is not responsive to audio input. In some embodiments, device 1100 does not change the pattern of visual content when device 1100 does not determine that a portion of the audio input should be sent to one or more external devices as an audio multicast message. . In some embodiments, device 1100 changes patterns of visual content that are responsive to different audio input portions regardless of this determination.

As shown in FIG. 11P, the device 1100 provides (eg, outputs) an output 1154p indicating that the device 1100 is playing the current jazz radio. Upon providing output 1154g, device 1100 causes device 1100 to determine that a portion of the audio input (eg, received in FIGS. 11M-11O) should not be sent as an audio multicast message to one or more external devices. , to display visual content (eg, a visual indication) that is multi-colored. In FIG. 11P, device 1100 also outputs jazz music.

While outputting jazz music, the device 1100 changes to an inactive state, as shown in FIG. 11Q.

As shown in FIG. 11R, device 1100 receives audio input portion 1150r from a person standing in front of device 1100 via one or more microphones of device 1100 . Here, the person is uttering the phrase "Pause." As shown in FIG. 11R, in response to receiving audio input portion 1150r, device 1100 determines that the portion of the audio input should be sent as an audio multicast message to one or more external devices (e.g., , it is determined that no part of the audio input should be so), the device 1100 displays visual content (eg, lights) that is multi-colored. In FIG. 11R, device 1100 also stops outputting jazz music and pauses media corresponding to jazz music in response to receiving a “pause” command.

FIG. 12 is a flow diagram illustrating a method for displaying visual instructions using an electronic device, according to some embodiments. Method 1200 is performed on a device (eg, 100, 300, 500, 634, 674, 700, 701, 850) having one or more microphones, one or more speakers, and one or more display devices. Some acts of method 1200 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

As discussed below, method 1200 provides an intuitive way to display visual instructions. The method relieves the user of the cognitive burden of determining the type of command given to the device, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling users to recognize command types faster and more efficiently saves power and extends the time between battery charges.

An electronic device receives audio input (eg, 1150a-1150r) via one or more microphones (1202).

The electronic device displays 1204 a visual representation of the audio input (e.g., light displayed via 1102), where visual instructions (e.g., 11B-11F, 1102 of 11K) is transmitted (eg, and/or transmitted (eg, multicast) to one or more devices (eg, multicast) to display 1206 (eg, and some In an embodiment of the first type of command (e.g., at least some of the voice input) to a plurality of external devices (e.g., personal devices (cellular phones), external accessory devices (e.g., smart home accessory devices (e.g., smart ( (e.g. intelligent) light bulbs, thermostats, door locks, door openers (e.g. garage door openers), speakers, televisions, electrical outlets, power strips, cameras, appliances (e.g. refrigerators, stoves)) another part of the voice input corresponding to the command) (e.g. part of the voice input not sent))))) one or more parts of the voice input corresponding to the voice input (e.g. 1150b-1150f, 1150r) and via one or more display devices, visual instructions (e.g., in FIGS. 11M-11O, 11R 1102) to be displayed 1210 (e.g., for outputting part of the audio input via speakers of other devices (e.g., multiple other devices), etc.) (e.g., another part of the audio input 2 type commands) to send any part of the audio input to other devices (e.g. instructions to multicast). including displaying a visual indication having a second set of one or more colors different from the first set of . In some embodiments, the first type of command is a command that initiates a process for sending at least some of the speech input (eg, portions of the speech input to be sent) to multiple external devices. In some embodiments, sending at least some of the audio inputs to multiple devices causes two or more of the multiple external devices to output some of the audio inputs (e.g., automatically output (e.g., without intervening user input). In some embodiments, the plurality of external devices includes different types of external electronic devices (eg, personal devices, external accessory devices). In some embodiments, a plurality of devices may be connected to a top-level location (or destination) (e.g., a first type of location) (e.g., a home location, a business, an office (e.g., a location at a top-level location (e.g., One or more regions (or zones) within an area and/or perimeter defined by a location (e.g., sub-location) defining one or more locations and/or locations of a top-level location, of a hierarchy of locations A first group associated with a place at the top level, a place linked (e.g. programmatically) to one or more places (e.g. sub-locations) and/or one or more regions (or zones)) In some embodiments, the first group of devices can include multiple subgroups of devices, hi some embodiments, the subgroups of devices are associated with the top-level location sub-locations (e.g., defined, identified, and/or linked (e.g., programmatically linked) to the top-level location (e.g., defined, identified, and/or linked to the top-level location, e.g., by a user) ) placed room) and/or within the top-level location (e.g., within the boundaries, areas, or perimeters of the top-level location and/or defined, identified, and/or linked (e.g., by a user) ) (e.g., a place defined by two or more sub-places) (within a top-level place (e.g., an entertainment zone containing a living room and dining room within a top-level place, within a top-level place personal zone, including two or more bedrooms, living rooms, and bathrooms.In some embodiments, the first user interface is a control user interface.In some embodiments, multiple devices. is associated with multiple subgroups of devices, hi some embodiments, the visual instructions continue to be displayed while receiving audio input via one or more microphones. In the form, visual instructions are initially displayed and then cease to be displayed while receiving audio input via one or more microphones, hi some embodiments, the second type of command is not a command that initiates a process for sending at least some of the audio input to multiple external devices (e.g. speakers, personal devices (e.g. cellular phones) . In some embodiments, the visual instructions continue to be displayed while receiving audio input via one or more microphones. In some embodiments, the visual instructions cease to display after being initially displayed while receiving audio input via one or more microphones. By displaying a visual indication having a first set of colors when it is determined that a portion of the speech input should be sent to one or more devices, the speech is Feedback is provided to the user as to whether a portion of the voice input will be played on the external electronic device so that the user is aware that the audio input can potentially be heard by the user. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve. one or more (e.g., different from the first set of one or more colors) when it is determined that the user's voice input does not contain instructions to send any portion of the voice input to another device By displaying a visual indication having a second set of colors, the user is provided with reassurance that what the user is saying cannot be heard by individuals on other external devices. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, the electronic device obtains electronic device information (e.g., obtains information such as the weather, answers to one or more queries, etc.) or changes one or more states (e.g., Any one or more of audio state (phone call answer/placement, volume adjustment), media playback state (e.g. media pause, play, fast forward, rewind, skip, shuffle), on/off state A voice input (eg, 1150m-1150o) that requests to change the device is a voice input that does not contain instructions to send any part of the voice input to another device.

In some embodiments, audio inputs (eg, 1150b-1150f, 1150k) corresponding to audio inputs sent to one or more devices are identifiers of groups of devices (eg, location, sub-location, group (“ home") or categories (e.g., locations corresponding to "dining room", "living room", "sunroom" within the group).In some embodiments, the voice input is the group of devices and a voice message to be sent to a group of devices.In some embodiments, voice inputs (eg, 1150m-1150o, 1150r) that do not contain instructions to send any part of the voice input to other devices are , does not include a device group identifier 1212. In some embodiments, the voice input does not include a device group identifier and the voice message is not sent to the device group.

In some embodiments, a first set of one or more colors (eg, 1102 in FIGS. 11C-11F) is a solid color and a second set of one or more colors (eg, 1102 in FIGS. 1102) of 11M-11O are multiple colors. In some embodiments, solid colors are not included in multiple colors. By applying different visual effects to the electronic device when different predetermined conditions are met, the user can quickly recognize whether their utterances are transmitted and played by the external electronic device. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device; reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, as part of displaying a visual indication (eg, 1102 in FIGS. 11C-11F) having a first set of one or more colors, the electronic device based on voice input ( For example, based on variations in the values of the input parameters (such as volume, tone, and/or pitch) of the voice input) for a period of time (eg, 1, 5, 10 seconds) (eg, at the time the voice input is received). display an animation (eg, 1102 in FIGS. 11C-11F) (of light patterns (eg, spatial patterns)) that varies (eg, in size, brightness, saturation, and/or position) over time).

In some embodiments, as part of the animated display (eg, 1102 in FIGS. 11C-11F), the electronic device displays a visual prompt based on any one or more of volume or pitch of the audio input. (eg, 1102 in FIGS. 11C-11F) is changed (eg, increased or decreased) in any one or more of the brightness, size, color saturation, or position. In some embodiments, the volume and/or pitch of the audio input increases/decreases any one or more of brightness, size, color saturation of the visual indication. In some embodiments, the rhythm (or beat) of the animation increases/decreases as the rhythm (eg, beat) of the audio input increases/decreases. In some embodiments, the visual indication occupies more/less area of the display device as the volume and/or pitch of the audio input increases/decreases. User speech by updating visual properties (e.g. brightness, size, color saturation or position) of visual instructions to reflect changes in user speech input (e.g. changes in volume or pitch of speech input) Visual feedback is provided to the user regarding various characteristics of the input. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, the electronic receives data corresponding to multiple voice messages for playback. In some embodiments, the data corresponding to the plurality of voice messages for playback includes an identifier for a group of devices (e.g., group ("home") or category (e.g., "dining room", "living room" within the group). locations corresponding to "room", "sunroom" categories).In some embodiments, the multiple voice messages are for multiple identifiers, each identifier corresponding to a different group or category. In some embodiments, in response to receiving data corresponding to a plurality of voice messages for playback, the electronic device prior to outputting a corresponding voice message for each of the plurality of voice messages (e.g., and (before outputting an identifier for a group of devices), an introductory tone (e.g. 1154i) (e.g. a dinging or ringing sound, an introductory tone or chime for all voice messages). In embodiments, the identifier is output before the group of voice messages corresponding to the identifier (eg, identifying the target group of devices) is output Multicast alert recipients where the voice messages are multicast type voice messages Output the same introductory tone before outputting a different voice message.By providing improved voice feedback, the usability of the device is enhanced (e.g., operating the device/with the device User device interfaces become more efficient (by helping users make appropriate inputs and reducing user errors when interacting), which in turn allows users to use devices more quickly and efficiently. By enabling this, the power usage of the device is reduced and the battery life is improved.

In some embodiments, an electronic device receives data corresponding to an audio message (eg, an audio multicast message) for playback. In some embodiments, the data corresponding to the voice message for playback is an identifier associated with a group of devices (e.g., group ("home") or category (e.g., "dining room", " locations corresponding to the "living room", "sunroom" categories).In some embodiments, in response to receiving data, the electronic device automatically, without user input, sends a voice message (e.g., , through the speaker of the electronic device) (e.g., 1154j).In some embodiments, outputting a voice message may include an identifier of a group of devices (e.g., group ("Home") or category (e.g., , locations corresponding to the "dining room", "living room", and "sunroom" categories within the group).

In some embodiments, a second visual indication (eg, 1102 of 11I) is displayed in response to receiving data prior to outputting the voice message. In some embodiments, the second visual indication is the first visual appearance (e.g., color , size, brightness). In some embodiments, the second visual indication is a second visual appearance that differs from the first visual appearance when the voice message is a voice message to be sent to the electronic device and not to other devices. has a visual appearance of

In some embodiments, an electronic device receives data corresponding to an audio message (eg, an audio multicast message) for playback. In some embodiments, the data corresponding to the voice message for playback is an identifier associated with a group of devices (e.g., group ("home") or category (e.g., "dining room", " locations corresponding to the "living room", "sunroom" categories).In some embodiments, voice messages are output (e.g., automatically in response to a user request).In some embodiments, , while outputting an audio message, the electronic device provides visual instructions (e.g., For example, 1102) in FIG. 11J are displayed at the same time.

In some embodiments, the voice message is output (eg, 1154j) based on the electronic device's volume level (eg, system level audio level) (eg, current volume level). In some embodiments, the electronic device displays visual content (eg, 1102 in FIG. 11A) (eg, any light (eg, , no LED is turned on)). In some embodiments, the electronic device is in an inactive display state (eg, no LEDs lit) before the visual indication is displayed. In some embodiments, the electronic device is in an active display state (eg, some LEDs are illuminated) while displaying visual instructions. By not displaying visual content when no audio input is being received, the user is provided with visual feedback that no audio input has been received and any utterances by the user have not been transmitted and played by other external devices. provide to Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, the electronic device stops displaying 1214 the visual instructions (eg, 1102 in FIG. 11G) and displays the visual content (eg, FIG. 11H) via one or more display devices. of 1102). In some embodiments, the electronic device is in an inactive display state after stopping displaying visual content. By not displaying visual content when no audio input is being received, the device is in an inactive state and no audio input is being received, providing the user with visual feedback that extends the battery life of the device. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device; reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, the visual indication (eg, 1102 in FIG. 11G) is a different state or process (eg, multicast (eg, or visual content) corresponding to the process or processes for sending/receiving audio multicast messages). By providing only one visual indication at a time, it provides the user with clarity about the state of the electronic device and whether the user's utterances are being transmitted and played back by other external devices. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device; reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, the visual indication (eg, 1102 in FIG. 11G) communicates with the electronic device (eg, transmits data on or to the electronic device or receives data from the electronic device 1 state of one or more applications (e.g., state for multicast processes, or processes for sending/receiving audio multicast messages, processes associated with the state of an electronic device, running on one or more external applications); It is suggested that the light (eg, LED at 1102) provide different light based on a process associated with the state of the electronic device.

Note that the details of the processes described above with respect to method 1200 (eg, FIG. 12) are also applicable in an analogous manner to the methods described above. For example, method 1200 optionally includes any one or more of the various method features described above with respect to methods 800 and 1000 . For example, the computer systems may be the same system. In another example, the voice message sent in method 800 can be media received in method 1000 . In another example, the audio input sent in method 1200 may be the same media received in method 1000 . For the sake of brevity, these details are not repeated below.

13A-13N illustrate exemplary electronic devices and user interfaces for managing external devices, according to some embodiments. The user interfaces of these figures are used to illustrate the processes described below, including the process of FIG.

FIG. 13A shows electronic device 700 displaying user interface 710 . In some embodiments, device 700 may be associated with home 602 (e.g., located at an associated physical location, designated communicating with one or more networks, such as interacting with one or more devices, or not). User interface 710 is a user interface for controlling a plurality of external devices designated as home 602 . The user interface 710 has one or more properties and may be responded to in a manner such as described above in connection with FIGS. 7A-7AH: home location identifier 712; macro controls 716; external device controls 718; 722, home multicast instructions 724, and home multicast affordances 726 (eg, selectable user interface object targets).

User interface 710 also includes status indicator 1314 . Each of the status indicators includes information about the current status of some of the devices currently designated as home 602 (eg, some of which was described above with respect to FIG. 6). As shown in FIG. 13A, status indicators 1314 include carbon monoxide status indicator 1314a, alarm status indicator 1314b, account error status indicator 1314c, update available status indicator 1314d, and part of timer status indicator 1314e (respectively). , from left to right). Carbon monoxide status indicator 1314a indicates that one or more carbon monoxide detectors or sensors have detected the presence of carbon monoxide in home 602 and therefore one or more carbon monoxide detectors or sensors A carbon detection state, eg, alert "on", indicates detection of a carbon monoxide leak. Alarm status indicator 1314b indicates that one or more alarms associated with home 602 have been tripped, thus indicating that one or more alarms are active, e.g., in a triggered or alerted state. show. Account error status indicator 1314 c indicates whether device 700 is not currently logged into the account associated with home 602 or device 700 connects to one or more known external devices designated home 602 . Indicates that you have a problem with the account associated with Home 602, such as having a problem. Update available status indicator 1314d indicates that updated software may be installed on one or more external devices and/or devices 700 (eg, software that includes one or more features corresponding to the Home application). . Timer status indicator 1314e indicates that one or more timers are currently active. Additionally, timer status indicator 1314e indicates that at least one timer (or the aggregate amount of time of multiple timers) has reached approximately 0% of the time since the timer (or multiple timers) was originally set. indicates that Specifically, this state is represented by remaining time display 1314e1 covering approximately 100% of the perimeter of timer state indicator 1314e.

Each status indicator includes a name (eg, "carbon monoxide)" associated with the status indicator (eg, carbon monoxide status indicator 1314a) and at least one particular visual characteristic (eg, a color (eg, red color)). , greyed out, etc.), patterns, shapes, etc. In some embodiments, any one or more of the state indicators may be displayed in the state represented by the state indicators (e.g., active, warning, (e.g. carbon monoxide detector or sensor), each status indicator may also include a quick visual representation of the status of the status indicator .

In FIG. 13A, each state indicator is within (e.g., is part of or belongs to) a particular state indicator category, and each state indicator is in the respective state indicator category to which each of the respective state indicators corresponds. displayed in order based on In FIG. 13A, a particular status indicator category of status indicators indicates the urgency (or importance) of the status indicator. A status indicator within one status indicator category may have one or more characteristics that differ from one or more characteristics of status indicators within a different status indicator category. As shown in FIG. 13A, carbon monoxide status indicator 1314a and alarm status indicator 1314b are in a first status indicator category (eg, most urgent category, alarm category) and a first color (eg, vertical hatching). in red), as indicated by In some embodiments, the first status indicator category indicates that the external device associated with the particular status indicator should be addressed as soon as possible. 13A, account error status indicator 1314c and update available status indicator 1314d are placed in a second status indicator category (eg, second most urgent category, more urgent than first status indicator category). are in the low status indicator category, action required category) and are displayed in a different color than the color of the status indicators in the first category (eg, shown without hatching). In FIG. 13A, timer status indicator 1314e is in a third status indicator category (eg, a third most urgent category, a status indicator category less urgent than the second status indicator category). Timer state indicator 1314e includes one or more dynamic components (eg, remaining time indicator 1314e1) related to the state of the timer corresponding to timer state indicator 1314e. In some embodiments, one or more dynamic components (eg, remaining time display 1314e1) are updated in real time (eg, as further described below with reference to FIG. 13B). In some embodiments, the status indicators in the first and/or second status indicator categories include one or more dynamic It has no components.

In some embodiments, when it is determined that a new state indicator should be displayed, the device 700 displays the new state indicator after (or before or between) the state indicators in the same state indicator category as the new state indicator. Show indicators. For example, if it is determined that a new status indicator is to be displayed and the new status indicator is in the second status indicator category, device 700 displays a display between update available status indicator 1314d and timer status indicator 1314e. Display a new state indicator (eg, instead of appending a new state indicator at the start and/or end of every displayed state indicator 1314). Thus, the new status indicator is displayed on the device 700 in a position relevant to the urgency of the status indicator, and the user is able to perceive the relative urgency of the new status indicator compared to the relative urgency of other displayed status indicators. Allows new status indicators to be identified and/or interacted with based on urgency. 13A, device 700 detects tap gesture 1350a at a location corresponding to state indicator 1314. In FIG.

In FIG. 13B, device 700 determines that gesture 1350a is a left swipe gesture. As shown in FIG. 13B, device 700 determines that gesture 1350a is a swipe gesture, so device 700 displays more state indicators than can be displayed on device 700 (eg, in the same area as state indicator 1314). If exists, display an additional status indicator. Specifically, device 700 moves status indicator 1314 previously displayed in FIG. 13A to the left (eg, toward gesture 1350a), thereby causing carbon monoxide status indicator 1314a, alarm status indicator 1314b, and account error status indicator 1314c are no longer displayed, update available status indicator 1314d and timer status indicator 1314e are replaced by carbon monoxide status indicator 1314a and alarm status indicator 1314b respectively, climate status indicator 1314f, door lock status Indicator 1314g and a portion of light status indicator 1314h are newly displayed in FIG. 13B.

As shown in FIG. 13B, timer status indicator 1314e includes a dynamic component, remaining time display 1314e1. 13A-13B, the remaining time display 1314e1 is around timer status indicator 1314e in FIG. 13A, as the timer starts at 13 minutes in FIG. from approximately 100% of the portion to approximately 75% of the periphery of timer status indicator 1314e of FIG. 13B (eg, in response to the passage of time, but not in response to user input).

In FIG. 13B, climate state indicator 1314f is in the third state indicator category, which is the same state category that timer state indicator 1314e was described above with respect to FIG. 13A. Climate status indicator 1314f represents status indicators corresponding to the status of multiple different types of external devices (eg, air quality sensors, temperature, humidity sensors). Climate state indicator 1314f indicates that the average temperature detected by one or more climate control devices (or where the one or more climate control devices are set) is 68 degrees and the average temperature detected by one or more humidity sensors is 68 degrees. Humidity is 40%, indicating good average air quality as detected by one or more air quality sensors. In addition, status indicator 1314f also indicates that the thermostat control is activating the heating system (eg, the room is being heated by the device the thermostat control is controlling), which is indicated by the upward arrow of status indication 1314f. shown. In some embodiments, when the status indicator 1314f includes a downward pointing arrow, the status indicator 1314f indicates that the thermostat control is activating the air conditioning system (eg, the room is being cooled by the device that the thermostat control is controlling). indicates that In some embodiments, when the status indicator 1314f includes dashes, the status indicator 1314f indicates that the thermostat control will not activate the air conditioning or heating system (e.g., the room is at the specified temperature at which the thermostat is set). is detected). 13B, device 700 detects tap gesture 1350b at a location corresponding to state indicator 1314. In FIG.

In FIG. 13C, device 700 determines that gesture 1350b is a left swipe gesture. As shown in FIG. 13C, device 700 has determined that gesture 1350b is a swipe gesture, so device 700 displays an additional status indicator. Specifically, device 700 moves status indicator 1314 previously displayed in FIG. 13B to the left, thereby displaying update available status indicator 1314d, timer status indicator 1314e, and climate status indicator 1314f. 13C, door lock status indicator 1314g, light status indicator 1314h slide to the left, and window open status indicator 1314i, door sensor status indicator 1314j, and motion sensor status indicator 1314k are newly displayed in FIG. 13C. In embodiments in which gesture 1350b is determined to be a rightward swipe gesture, device 700 moves the state indicators to the right to display any one or more of the previously displayed state indicators (e.g., FIG. 13A display carbon monoxide status indicator 1314a, alarm status indicator 1314b).

In FIG. 13C, door lock status indicator 1314g, light status indicator 1314h, and open window status indicator 1314i are placed in a fourth status indicator category (e.g., fourth most urgent category, more urgent than third status indicator category). state indicators category, quick action state indicators category). As noted, categories of particular condition types are predefined, but in some embodiments, categories of condition indicators of particular types may be specified via one or more user inputs. In some embodiments, the device 700 may be a specific type of device (eg, light) and/or a user may indicate a specific type of device and/or a specific type of status indicator (eg, most important, second most Displays one or more settings for a particular type of status indicator that allows setting status categories of status indicators to be associated with (important, third most urgent, etc.).

A status indicator in the fourth status indicator category indicates that a number of external devices of a particular type are in a particular state. For example, as shown in FIG. 13B, door lock status indicator 1314g indicates that two door locks are unlocked. As another example, in FIG. 13C, light status indicator 1314h has five lights active (or on) and open window status indicator 1314i has seven window blinds (e.g., covers, shades) open. indicates that In particular, the status indicators of the fourth status indicator category exclude information about devices of the same type in different states. For example, the living room desk lamp designated home 602 is off (eg, as indicated by living room desk lamp control 718g). In FIG. 13C, the living room desk lamp is off, so light status indicator 1314h does not include information about the living room desk (discussed in more detail below with respect to FIGS. 13D-13F).

Figures 13C-13H illustrate different gestures detected by device 700 aimed at a state indicator (eg, a state indicator of the fourth state indicator category). Specifically, FIGS. 13C-13H show different gestures directed at light status indicator 1314h. However, similar techniques described herein can be applied to gestures directed at other status indicators, such as door lock status indicator 1314g and window open status indicator 1314i. In FIG. 13C, device 700 detects gesture 1350c corresponding to (eg, at a corresponding location) light status indicator 1314h.

In FIG. 13D, device 700 determines that gesture 1350c is a press and hold gesture (eg, a gesture detected over a predetermined period of time). As shown in FIG. 13D, because device 700 determined that gesture 1350c was a press and hold gesture, device 700 may display an additional status indicator (e.g., when it determined that gesture 1350b was a swipe gesture). display the user interface 1340 (as opposed to doing). As shown in FIG. 13D, user interface 1340 has been replaced by user interface 710 previously displayed in FIG. 13C.

As shown in FIG. 13D, light status indicator 1314h indicates the status of one or more lights designated home 602, and user interface 1340 provides a display for lights (e.g., lighting devices) designated home 602. of external device control 718 . As shown in FIG. 13D, the external device controls 718 displayed on the user interface 1340 are the rooms or portions of the home 602 (e.g., "bedroom", "dining room") to which their respective external device controls are assigned. , “living room”, “porch”). For example, bedroom light control 718f and bedroom nightlight control 718h are assigned to the bedroom (e.g., 602c) of home 602 and are under the "Bedroom" heading on user interface 1340, and dining room light control 718i is the dining room of home 602. A living room lamp control 718b and a living room desk lamp control 718g are assigned to the living room (e.g., 602d) of home 602, and are located under the "Dining Room" heading on user interface 1340. and is under the "Living Room" heading on user interface 1340, and porch light control 718j is assigned to the porch of home 602 and is under the "Porch" heading on user interface 1340.

In particular, as shown in FIG. 13D, bedroom light control 1718f, bedroom nightlight control 718h, dining room light control 718i, living room lamp control 718b, and porch light control 718j are controlled when the respective corresponding device (e.g., light bulb) is turned on. or indicate that it is working. In other words, the devices corresponding to bedroom light control 1718f, bedroom nightlight control 718h, dining room light control 718i, living room lamp control 718b, and porch light control 718j are on, as indicated by light status indicator 1314h in FIG. 13C. One of such "five lights" suggestions is shown. Further, looking at light status indicator 1314h, six lights are shown on user interface 1340, but there is no indication of a sixth light and/or a sixth light status. Even though only five lights are on or active, identified by light status indicator 1314h, Home may have more lights, such as lights that are inactive or off with which Home is associated. In some implementations, the user interface 1340 shows the detail status of all lights when displaying the detail status of the lights. In FIG. 13D, the living room desk lamp is displayed with a different visual indication than the other lights (eg, indicating that the living room desk lamp control 718g is off). In some embodiments, user interface 1340 includes living room desk lamp control 718g in FIG. 13D because the living room desk lamp is not in the same state as the other light controls represented by light status indicator 1314h in FIG. 13C. do not have. In some embodiments, when the user interface 1340 does not include the living room lamp control 718g of FIG. Living room lamp control 718g may be displayed on user interface 1340 in response to receipt of an indication of when physically turned on, or the same state of the other light control represented by light state indicator 1314h. . In FIG. 13D, device 700 detects tap gesture 1350d directed at bedroom light control 718f.

In FIG. 13E, in response to detecting tap gesture 1350d, device 700 sends an instruction to change the state of bedroom light 632 (eg, turn off), updating the visual appearance of bedroom light control 718f. As shown in FIG. 13E, the bedroom light control 718f is in the active state, as opposed to the bedroom light (e.g., bedroom light 632) corresponding to the bedroom light control 718f, as shown in FIG. 13D, for example. OFF, for example indicating an inactive state. In particular, the other light controls displayed on user interface 1340 are unchanged in FIGS. 13D-13E. In response to detecting tap gesture 1350d, device 700 sends a command to change bedroom light 632 without other lights represented by other light controls changing state. Thus, user interface 1340 allows the user to cause each device represented by each corresponding control to individually perform an action (e.g., change the state of the device) without causing other devices to perform the action. enable. In FIG. 13E, device 700 detects tap gesture 1350e directed at completion affordance 1342. In FIG.

In response to detecting tap gesture 1350e, device 700 ceases displaying user interface 1340 and redisplays user interface 710 with status indicator 1314, as shown in FIG. 13F. As shown in FIG. 13F, light status indicator 1314h indicates that 4 lights are on at home 602, as opposed to 5 lights on, as shown by light status indicator 1314h in FIG. 13C. updated to reflect Accordingly, some state indicators are used to indicate the number of devices currently in a particular state (e.g., whether a device has changed from that particular state, or whether a device has changed to that particular state). Updated. Further, in FIG. 13F, the bedroom light control 718f also shows other light controls displayed on the user interface 710 (e.g., living room lamp control 718b and living room desk lamp control 718g), as shown in FIG. 13C. While unchanged when compared to their respective appearances, they have changed on the user interface 710 to reflect that the bedroom lights are off. Accordingly, the device controls displayed on user interface 710 are also updated in response to device 700 detecting a gesture directed to the device controls displayed via user interface 1340 (eg, gesture 1350d). be done. In FIG. 13F, device 700 detects gesture 1350f directed at light status indicator 1314h.

In FIG. 13G, device 700 determines that gesture 1350f is a tap gesture (eg, a gesture detected for less than a predetermined period of time, a gesture different from gestures 1350a-1350c). In FIG. 13G, device 700 determines that gesture 1350f is a tap gesture, so device 700 turns off (or changes state) the four lights indicated as on by light state indicators 1314h in FIG. 13F. Send orders. Thus, bedroom nightlight control 718h, dining room light control 718i, living room lamp control 718b, and porch light control 718j are off in FIG. 13G. As shown in FIG. 13G, device 700 determines that gesture 1350f is a tap gesture, so device 700 changes the visual appearance of light status indicator 1314h (eg, in FIG. 13G) and updates the light status indicator 1314h to show that the four lights are turned off. In particular, as shown in FIG. 13G, living room lamp control 718b and living room desk lamp control 718g are also changed to indicate that their respective corresponding devices are turned off. Thus, in FIG. 13G, device 700 changes the state of multiple devices (e.g., devices designated in different rooms) in response to a single gesture (e.g., tap) on the state indicator, which is , which only changes the state of one device in response to the same type of gesture (e.g., tap) on an external device control (e.g., as described above with respect to gesture 1350d in FIG. 13D). It is a thing. Further, in FIG. 13G, the device 700 performs the same action (eg, turn on the light, turn on the door, turn on the door, etc.) based on the type of device (eg, light, door lock, window blind adjuster) indicated by the state indicator to which the tap gesture was directed. locks, closing blinds) to change the state of multiple devices. As shown in FIG. 13G, gesture 1350f in FIG. 13H is a tap gesture directed at light state indicator 1314h, so device 700 does not change the state of the other state indicator and responds to the other state indicator. Other devices do not send commands to change state and maintain the appearance of other state indicators (eg, when comparing FIGS. 13F-13G).

In FIG. 13H, device 700 determines that the predetermined period of time has elapsed since gesture 1350f was detected without device 700 detecting another gesture directed at light status indicator 1314h. As shown in FIG. 13H, it is determined that the predetermined period of time has elapsed since gesture 1350f was detected, so device 700 ceases displaying light status indicator 1314h and causes light status indicator 1314h to stop displaying previously displayed light status indicator 1314h. Open window status indicator 1314i, door sensor status indicator 1314j, and motion sensor status indicator 1314k are moved to the left to fill the occupied space. For example, open window status indicator 1314i is adjacent to light status indicator 1314h in FIG. 13G, and open window status indicator 1314i occupies the space vacated by light status indicator 1314h in FIG. 13H. Since some of the status indicators have been shifted to the left, an additional status indicator, photosensor status indicator 1314k, is newly displayed. Door sensor status indicator 1314j, motion sensor status indicator 1314k, and light sensor status indicator 1314k are included in a fifth status indicator category (e.g., fifth most urgent category, less urgent status indicator than fourth status indicator category). category, Sensor Status Indicators category). Door sensor status indicator 1314j indicates that the door sensor assigned to home 602 detected that the door was ajar. Motion sensor status indicator 1314 k indicates that motion has been detected by one or more motion sensors designated as home 602 . Light sensor status indicator 1314l indicates that one or more door sensors have detected that one or more doors designated to home 602 are open. In some embodiments, when a gesture directed at light status indicator 1314h is detected before a predetermined period of time has elapsed, device 700 changes the four lights to the on state and displays light status indicator 1314h. to indicate that four lights are on (eg, as shown in FIG. 13F). This ability to continue to look at the status indicator for a predetermined period of time after the accessory associated with the status indicator changes from an emergency or active state provides the user with an opportunity to easily undo the accessory's status change. . In some embodiments, if a gesture directed at light status indicator 1314h is detected before a predetermined period of time elapses, device 700 will display the , maintain the display of light status indicator 1314h. In some of these embodiments, when light status indicator 1314h is first displayed on user interface 710 (eg, FIG. 13B), one or more lights were initially in any of the lights. Device 700 maintains the display of light status indicators 1314h because light status indicators 1314h indicate that one or more of . In FIG. 13H, device 700 detects tap gesture 1350h on room navigation control 722b.

As shown in FIG. 13I, in response to detecting tap gesture 1350h, device 700 displays Home application user interface 1360 with room navigation control 722b selected (eg, as shown in bold). indicates that User interface 1360 is a user interface for controlling a plurality of external devices assigned to bedroom 602 c , as indicated by bedroom identifier 1368 . In some embodiments, device 700 uses one or more techniques such as those described above in connection with displaying user interface 760 of FIG. 7G to display user interface 1360 of FIG. 13I. can be done. In some embodiments, user interface 1360 may alternatively be displayed in response to detection of tap gesture 750f3 of FIG. 7F and/or user interface 760 may alternatively be displayed in response to detection of tap gesture 1350h of FIG. 13H. Alternate displays can be provided in response.

As shown in FIG. 13I, user interface 1360 includes a reduced set of status indicators than those displayed in FIGS. 13A-13C and 13F-13H. User interface 1360 includes a reduced set of status indicators as it is determined that each status indicator displayed on user interface 1360 corresponds to at least one device designated in bedroom 602 . In other words, when displaying user interface 1360 for controlling a plurality of external devices designated in bedroom 602c, device 700 ceases displaying status indicators that do not correspond to at least one device designated in bedroom 602c. do.

As shown in FIG. 13I, the reduced set of condition indicators includes carbon monoxide condition indicator 1314a, climate condition indicator 1314f, and open window condition indicator 1314i. In particular, climate state indicator 1314f of FIG. 13I indicates the temperature associated with a thermostat that is assigned to the bedroom but does not indicate the humidity and/or air quality information indicated by climate state indicator 1314f of FIG. 13B. Here, the climate condition indicator 1314f of FIG. 13I does not indicate humidity and/or air quality information because no humidity and air quality sensors have been assigned to the bedroom. However, climate status indicator 1314f in FIG. 13I indicates temperatures associated with thermostat control, since at least one thermostat is assigned to the bedroom. Additionally, the status indicator 1314f of FIG. 13I also continues to indicate that the thermostat control is activating the heating system (eg, the room is being heated by the device the thermostat control is controlling) (eg, 1314f of FIG. 13B). ), this is indicated by the upward pointing arrow in state suggestion 1314f in FIG. 13I. Accordingly, status indicators corresponding to multiple devices of the same type may provide a display of information regarding any one or more of the device types when navigating to user interfaces corresponding to different portions or rooms of home 602. can be discontinued. Additionally, the open window status indicator 1314i of FIG. 13I has been modified to indicate that one window is open, as opposed to indicating that seven windows are open as shown in FIG. 13H. ing. The open window status indicator 1314i of FIG. 13I has been modified to indicate that one window is open because only one bedroom window has a window blind adjuster assigned to bedroom 602c. In FIG. 13I, device 700 detects a press and hold gesture 1350i directed at open window status indicator 1314i.

In response to detecting the press and hold gesture 1350i, the device 700 displays a user interface 1370 including window blind adjustable controls 1318i, as shown in FIG. 13J. In FIG. 13J, device 700 detects gesture 1350j (swipe down gesture) on window blind adjustable control 1318i. As shown in FIG. 13K, in response to detecting gesture 1350j, device 700 instructs the window blind adjuster to change from a 100% open state (as indicated by suggestion 1372 in FIG. 31J) to (e.g., Send a command to the window blind adjuster device assigned to bedroom 602b to change it to 58% open (as indicated by suggestion 1372 in FIG. 13K).

In particular, in FIG. 13J, device 700 displays window blind adjustable control 1318i, where selection of window blind adjustable control 1318i causes the state of the window blind adjuster to be indicated by window blind adjustable control 1318i. (e.g., "open" 100% open) and opposite to the state indicated by the window blind adjustable control 1318i (e.g., "open") (e.g., "closed" 0%). open). Thus, selecting the window blind selectable control 1318i is different from allowing the user to change the window blind adjuster by directing a tap gesture in the direction of the open window status indicator 1314i of FIG. 13I. Allows the user to change the window blind adjuster to any state. In some embodiments, device 700 displays a user interface similar to user interface 1340 of FIG. 13D when open window status indicator 1314i of FIG. 13I indicates more than a predetermined number of devices. Thus, if there are more than a predetermined number of devices, as indicated by window state indicator 1314i in FIG. Change the state (eg, on/off state, open/closed state, locked/unlocked state). In some embodiments, the device 700 may indicate that the user is on one or more devices when there are less than a predetermined number of devices than when there are more than a predetermined number of devices indicated by the status indicator. to display controls that allow you to have more (or different) controls. In some embodiments, when there are less than a predetermined number of devices indicated by the state indicator and a tap-and-hold gesture is detected on the state indicator, device 700 detects that more than the predetermined number of devices are identified by the state indicator. state, display more controls for controlling a particular device than the number of controls displayed for a particular device.

13L-13N illustrate exemplary user interfaces displayed in response to detecting gestures directed at other status indicators. In some embodiments, gestures directed at other state indicators are the same type of gestures or gestures as gesture 1350c (eg, press and hold gestures) to display the respective user interfaces of FIGS. 13L-13N. It can be the same type of gesture as 1350f (eg, tap gesture). Accordingly, in some embodiments, device 700 displays the user interface of each of FIGS. 13L-13N in response to detecting any type of gesture (eg, press and hold gesture or tap gesture).

FIG. 13L shows device 700 displaying user interface 1382 . User interface 1370 is displayed in response to device 700 detecting a gesture directed at carbon monoxide status indicator 1314a of FIG. 13A. As shown in FIG. 13L, user interface 1382 includes external device controls 718, where each external device control indicates the status of a carbon monoxide device designated home 602. FIG. As shown by user interface 1382, the bedroom carbon monoxide sensor is triggered (e.g., as shown by bedroom carbon monoxide control 718k), and the kitchen carbon monoxide sensor and living room carbon monoxide sensor are not triggered (as indicated by sensors 718l-718m). Thus, in response to the device 700 detecting a gesture directed at a status indicator of a status indicator category other than the fourth status indicator category (eg, carbon monoxide status indicator 1314a), the device 700 will detect the status of the respective category. A particular type of device control corresponding to the indicator is displayed, where the device control has one or more visual appearances that indicate the state in which the device corresponding to the respective control is. In some embodiments, user interface 1370 is displayed using techniques similar to those described above in connection with FIGS. 13D-13E.

FIG. 13M shows device 700 displaying user interface 1384 . User interface 1384 is displayed in response to device 700 detecting a gesture directed at climate state indicator 1314f of FIG. 13B. As shown in FIG. 13L, user interface 1384 provides controls corresponding to multiple types of devices because climate status indicator 1314f indicates the status of multiple types of devices (eg, as described above with respect to FIG. 13B). include. For example, in FIG. 13L, user interface 1384 provides controls for controlling thermostat devices (eg, bedroom thermostat control 718n and living room thermostat control 718r), controls for controlling air quality sensors (eg, kitchen air quality sensor living room air quality sensor control 718o and living room air quality sensor control 718r), and controls for controlling humidity sensors (eg, kitchen humidity sensor control 718p).

FIG. 13N shows device 700 displaying user interface 1386 . User interface 1386 is displayed in response to device 700 detecting a gesture directed at motion sensor status indicator 1314k of FIG. 13H. As shown at 13N, user interface 1386 includes controls corresponding to the motion sensor designated home 602 . User interface 1386 is displayed using techniques similar to those described above in connection with FIGS. 13D-13E and 13L.

FIG. 14 is a flow diagram illustrating a method for managing external devices using a computer system, according to some embodiments. The method 1400 is a computer system (eg, 100, 300, 500, 634, 674, 700, 701, 700, 701, 700, 701, 700, 700, 701, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, 700, ... 850). Some acts of method 1400 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

As described below, method 1400 provides an intuitive way to manage external devices. The method reduces the user's cognitive burden for managing external devices, thereby creating a more efficient human-machine interface. For battery-operated computing devices, it conserves power and extends the time between battery charges by enabling users to perceive device status more quickly and efficiently.

A computer system determines that a plurality of accessory devices of a first accessory type (e.g., light bulbs, windows, switches, outlets, locks (e.g., door locks, window locks), televisions) are in a first state (e.g., on, off, Open, Closed, Locked, Unlocked (e.g., graphical representation and corresponding lettering (e.g., two bulbs are on)) a first status indicator (e.g., 1314g, 1314h, 1314i) , display (e.g., as part of a display of a first user interface (e.g., a user interface for controlling a set of accessory devices) including (e.g., including a display)) via a display generation component (e.g., 1402). The plurality of accessory devices of a first accessory type are a first accessory device (1404) of a first accessory type and in a first state, and a first accessory device (1404) of a first accessory type and in a first state. A second accessory device (1406) is included, wherein the second accessory device is different than the first accessory device. In some embodiments, the first state indicator is the accessory device of the first accessory type that is not in the first state (eg, the second state) (eg, the first state indicator is the do not show or control (eg, when selected) one or more devices of the first accessory type (eg, in a second state). In some embodiments, the first status indicator does not indicate the accessory device of the accessory type that is not the first accessory type, regardless of the current status of the accessory device of the accessory type that is not the first accessory type.

The computer system detects (eg, via one or more input devices) gestures (eg, 1350a-1350c, 1350f) while the computer system is displaying the first status indicator (1408).

In response to detecting (1410) a gesture, the gesture is a first type of gesture (e.g., 1350f) (e.g., tap gesture) to a first state indicator (e.g., 1350h) (e.g., at a corresponding location). In accordance with the determination that it is directed, the computer system directs the first accessory device (e.g., to perform an action that changes the state of the first accessory device) and the second accessory device (e.g., the second accessory device Cause the device to change 1412 to a second state that is different from the first state (e.g., as opposed to (e.g., on/off, locked/unlocked, open/closed)) to perform an action that changes the state of the device (1412). (e.g., via initiating a process (and in some embodiments, a process may send one or more instructions to one or more devices (e.g., a server, a first accessory device and a second a hub device, a first accessory device, a second accessory device that communicates with a plurality of devices, including the device), in some embodiments the first accessory device and the second accessory device; As part of entering the second state, the computer system causes one or more states to be in the second state (or any state different from the first state) (e.g., before the gesture is detected). accessory device to perform any action and/or change state.

A second type of gesture (eg, 1350c) (eg, a long press gesture) in response to 1410 detecting a gesture, where the gesture is different from the first type of gesture, causes a first state indicator (eg, 1314h) ), the computer system, via the display generation component, directs a first accessory control user interface object target (eg, 718b, 718f-718j) corresponding to the first accessory device (eg, as part of the display of the second user interface) is displayed 1414, where the selection of the first accessory control user interface object target (eg, 1350d) (a first type of gesture (eg, tap gesture)). ) causes a first accessory device to change a second accessory device to a second state (eg, to perform an action that changes the state of the first accessory device). (e.g., a second accessory device continues to be in the first state after completing placing the first accessory device in the first state) without changing to a second state (e.g., starting a process). (and in some embodiments, the process communicates instructions to one or more devices (e.g., a server, multiple accessory devices (e.g., including first and second accessory devices) the hub device, the first accessory device (not the second accessory device).In some embodiments, while displaying the first accessory control user interface object target corresponding to the first accessory device Second, the computer system detects a gesture directed at the first accessory control user interface object target (e.g., a first type gesture directed at the first accessory control user interface object target) and performs a first In response to detecting a gesture directed at an accessory control user interface object target, the computer system moves the first accessory device (e.g., to perform an action that modifies the first accessory device) to the second accessory device. Cause an action to change to a second state without placing the accessory device in the second state (e.g., placing the first accessory device in the first state). After completing, the second accessory device remains in the first state). In some embodiments, in response to detecting a gesture directed at the first state indicator, and following determining that the gesture is a second gesture that is different from the first gesture (eg, a long press gesture): The computer system displays, via the display generating component, a third accessory control user interface object object corresponding to the third accessory device, where the third accessory device is of the first accessory device type and the Not in one state (e.g., in a second state) and in response to detecting a gesture directed at a third accessory control user interface object target, the computer system causes the third accessory device to: Without changing the state of the first accessory device and the second accessory device (e.g., without putting the first accessory device and the second accessory device into the second state) (e.g., the third accessory (to perform an action that modifies the device) to a first state. In some embodiments, the second user interface corresponds (eg, or any) to an accessory device that is not of the first type (eg, of a second accessory type that differs from the first accessory type). Does not include accessory control user interface object targets. In some embodiments, in response to detecting a gesture directed at the first state indicator, and following determining that the gesture is a second gesture (e.g., a long press gesture) that is different from the first gesture. , the computer system does not display an accessory control user interface object object that is of a different accessory type than the first accessory type and is displayed via the second user interface). In some embodiments, in response to detecting a gesture directed at the first state indicator, and following determining that the gesture is a second gesture that is different from the first gesture (eg, a long press gesture): The computer system ceases displaying the first status indicator. By performing different actions based on the type of gesture detected when a gesture is directed at the first state indicator, the user is provided with more control options without cluttering the user interface. By allowing additional control of the system without cluttering the UI with additional displayed controllers, the usability of the system is enhanced and the user-system interface (e.g., operating/interacting with the system by helping the user to provide the correct input at times and by reducing user error), as well as enabling the user to use the system more quickly and efficiently, Reduces system power usage and improves battery life. providing a user with one control over which multiple devices change states by causing the first accessory device and the second accessory device to change to a second state that is different from the first state, thereby: The number of inputs required to cause multiple devices to change state is reduced. Improves the usability of the system by reducing the number of inputs required to perform an action (e.g., prompting the user to provide appropriate input when operating/interacting with the system). It makes the user-system interface more efficient (by assisting and reducing user error), as well as reducing power usage by enabling users to use the system more quickly and efficiently. , to improve system battery life.

In some embodiments, determining that the gesture is a second type gesture (eg, 1350c) directed at the first state indicator (eg, 1314h), unlike the first type gesture Accordingly, the computer system, via the display generation component, concurrently displays a first accessory control user interface object object (eg, 718b, 718f-718j), and a second accessory control user interface object corresponding to the second accessory device. Display objects (eg, 718b, 718f-718j). In some embodiments, selection of a second accessory control user interface object target (e.g., via a first type of gesture (e.g., tap gesture)) causes (e.g., via a process (and In some embodiments, the process includes one or more devices (e.g., a server, a hub device communicating with a plurality of accessory devices (e.g., including first and second accessory devices), a second accessory device (e.g., a second sending a command to a second accessory device to change to a second state without causing the first accessory device to change to the second state; In an embodiment, while displaying a second accessory control user interface object target corresponding to a second accessory device, the computer system detects a gesture directed at the first accessory control user interface object target. (e.g., a first type gesture directed at a first accessory control user interface object target), in response to detecting a gesture directed at a second accessory control user interface object target, the computer system: causing a second accessory device to perform an action to change to a second state without bringing the first accessory device to the second state (e.g., to perform an action that changes the second accessory device); (For example, the first accessory device remains in the first state after completing placing the second accessory device in the first state.) In some embodiments, a second accessory control user interface. In response to the detection of a gesture directed at the object target, the second accessory control user interface object target displays the first visual appearance (e.g., character, size, color, pressed state (e.g., depressed , pressed)), is changed to a second visual appearance (e.g., character, size, color, pressed state (e.g., depressed, pressed)) that is different from the first visual appearance. Displaying a second accessory control user interface object target corresponding to a second accessory device concurrently with a first accessory control corresponding to a first accessory device (eg, the gesture is a first type of gesture) (as opposed to a gesture of the second type directed at the first status indicator), giving the user more control options for controlling individual accessory devices without disrupting the user interface. offer. By allowing additional control of the system without cluttering the UI with additional displayed controllers, the usability of the system is enhanced and the user-system interface (e.g., operating/interacting with the system by helping the user to provide the correct input at times and by reducing user error), as well as enabling the user to use the system more quickly and efficiently, Reduces system power usage and improves battery life.

In some embodiments, the gesture along with the first state indicator is in a second state indicating that the plurality of accessory devices of a second accessory type, different from the first accessory type, are in a third state. A plurality of accessory devices of a second accessory type detected while simultaneously displaying indicators (eg, 1314g, 1314i) are of the second accessory type and are in a third state (eg, the first state and a state different from the second state) and a second accessory type, including a fourth accessory device in the third state. In some embodiments, the third accessory device of the second accessory type and in the second state and the fourth accessory device of the second accessory type and in the second state are the first and a second accessory device. In some embodiments, the gesture is a first type of gesture (e.g., similar to 1350f) (e.g., a tap gesture) to a second state indicator (e.g., 1314g, 1314i) (e.g., corresponding location ), the computer system places the third accessory device and the fourth accessory device in a different state than the third state (e.g., places the first accessory device and the second accessory device in a different state). change to a fourth state (eg process (and in some embodiments, the process sends one or more instructions to one or more devices (e.g., server, first accessory device, second accessory device, third and a hub device that communicates with multiple devices, including a fourth accessory device).In some embodiments, the gesture is a second type gesture (e.g., similar to 1350c ) (e.g., press and hold gestures) directed (e.g., at corresponding locations) to the second state indicators (e.g., 1314g, 1314i), the computer system, via the display generation component, , displays a third accessory control user interface object target corresponding to a third accessory device (eg, as part of displaying the second user interface) In some embodiments, the third accessory By selecting (e.g., via a process (and in some embodiments, a process the above devices (e.g., a server; a hub device that communicates with a plurality of accessory devices (e.g., including first, second, third, and fourth accessory devices); a third accessory device (e.g., a fourth accessory device); a third accessory device (e.g., to perform an action that changes the state of the first accessory device), a fourth accessory device to a fourth state change to a fourth state (and In some embodiments, a first accessory control user corresponding to a first accessory device and/or a second accessory control user interface object target (eg, as part of displaying the second user interface) without displaying the interface object target via a display generation component (e.g., a fourth accessory device enters a third state after causing a third accessory device to be in a fourth state). continue to exist). In some embodiments, upon determining that the gesture is a second type of gesture (e.g., press and hold gestures) and directed at a second state indicator (e.g., at a corresponding location), the computer system: A fourth accessory control user interface object object is displayed concurrently with the third accessory control user interface object object. In some embodiments, when a fourth accessory control user interface object target is selected, the computer system (e.g., via a process (and in some embodiments, the process is one or more devices (e.g., a server, a hub device that communicates with a plurality of accessory devices (e.g., including first, second, third, and fourth accessory devices), a fourth accessory device (but not a third accessory device) sending a command) causes a fourth accessory device to change to a fourth state without causing a third accessory device to change to a fourth state, a state indicator to which the gesture was directed and a detected state indicator; By performing different actions based on the type of gesture, the user is provided with more control options for controlling different accessory device types without cluttering the user interface. By allowing additional control of the system without clutter, the usability of the system is enhanced and the user-system interface (e.g., to provide appropriate input when operating/interacting with the system). (by assisting the user and reducing user error), and in addition reducing the power usage of the system by enabling the user to use the system more quickly and efficiently, Battery life is improved.By having a third accessory device and a fourth accessory device change to a fourth state, the user is provided with one control for multiple devices to change state, allowing multiple devices to change states. The number of inputs required to change state is reduced.By reducing the number of inputs required to perform an action, the operability of the system is improved (e.g., operating the system/ make the user-system interface more efficient (by assisting the user in providing appropriate input and reducing user error when interacting with the Reduce power usage and improve system battery life by enabling the use of

In some embodiments, the fifth accessory device (eg, the device represented by 718b in FIG. 13A) is of the first accessory device type and the fifth accessory device (eg, represented by 718b in FIG. 13A) device) is in the first state, the plurality of accessory devices of the first accessory type (eg, as indicated at 1314h) (eg, the first accessory device and the second accessory device). different) includes a fifth accessory device. In some embodiments, the fifth accessory device (eg, the device represented by 718g in FIG. 13A) is of the first accessory device type and the fifth accessory device (eg, represented by 718g in FIG. 13A) device) is not in the first state, the plurality of accessory devices of the first accessory type does not include a fifth accessory device (eg, as indicated at 1314h). In some embodiments, following determining that the fifth accessory device is not of the first accessory device type and that the fifth accessory device is in the first state, the plurality of accessory devices of the fifth accessory type Does not include a fifth accessory device. In some embodiments, the plurality of accessory devices of the first accessory type according to determining that the fifth accessory device is not of the first accessory device type and the fifth accessory device is not in the first state. does not include the fifth accessory device. By providing a state indicator to control accessory devices of the first accessory device type that are in the first state and not to control accessory devices of the first accessory device type that are not in the first state, a specific predetermined A user is provided with a status indicator to control the accessory device that meets the specified condition. System operability is enhanced by performing actions when a set of conditions are met without the need for further user input, allowing the user-system interface (e.g., operating/interacting with the system (by helping users to provide the appropriate input when they do this, by reducing user error), as well as enabling users to use the system more quickly and efficiently. , which reduces system power usage and improves battery life.

In some embodiments, the first type of gesture is a tap gesture (eg, 1350f) (eg, a gesture detected (eg, via one or more input devices) in less than a predetermined period of time). is. In some embodiments, the second type of gesture is a press and hold gesture (eg, 1350c) (eg, detected (eg, via one or more input devices) for longer than a predetermined period of time). gesture).

In some embodiments, in response to detecting the gesture, the gesture is a third type of gesture (e.g., 1350a) (e.g., swipe gesture, gesture that moves in a direction (e.g., sideways)) (e.g., and directed to the first state indicator), the computer system displays (e.g., by scrolling through the state indicators) the previously undisplayed state indicator (e.g., 1314f in FIG. 13B). (eg, a previously undisplayed status indicator indicates that multiple accessory devices of accessories different than the first accessory device type are in the same status). In some embodiments, display of the first state indicator is stopped (eg, by scrolling the first state indicator out of view) upon determining that the gesture is the third type of gesture. In some embodiments, a plurality of displayed state indicators (e.g., moving in the direction of the gesture (e.g., left, right, up, down, etc.) in accordance with determining that the gesture is the third type of gesture so that after the plurality of state indicators have been moved, the first state indicator ceases to be displayed, while one or more state indicators that were not displayed before detecting the third type gesture are In some embodiments, one or more controls that are not state indicators (e.g., scene controls, external accessory device controls) remain displayed after the gesture is detected. By performing a different action when the gesture is directed at the first state indicator, the user is provided with more control options (eg, more state indicators) without cluttering the user interface. By allowing additional control of the system without cluttering the UI with additional displayed controllers, the usability of the system is enhanced and the user-system interface (e.g., operating/interacting with the system by helping the user to provide the correct input at times and by reducing user error), as well as enabling the user to use the system more quickly and efficiently, Reduces system power usage and improves battery life.

In some embodiments, as part of causing the first accessory device and the second accessory device to change to the second state, the computer system causes the first accessory device and the second accessory device to change the first state. perform a type of action based on the type of accessory (e.g. lights, door locks, blinds, temperature control) (e.g. turn off lights, unlock door locks, open blinds a certain percentage, turn off AC/heat) . By having the first accessory device and the second accessory device perform a type of action based on the first accessory type, the computer system can perform a particular action based on particular predetermined conditions. be possible. System operability is enhanced by performing actions when a set of conditions are met without the need for further user input, allowing the user-system interface (e.g., operating/interacting with the system (by helping users to provide the appropriate input when they do this, by reducing user error), as well as enabling users to use the system more quickly and efficiently. , which reduces system power usage and improves battery life.

In some embodiments, prior to detecting the gesture, the computer system detects a third state indicator (eg, 1314b) adjacent to the first state indicator (eg, second accessory type (eg, first accessory type)). and a fourth status indicator (eg, 1314c) adjacent to the third status indicator (eg, 1314e) (eg, a fourth status indicator (eg, 1314c) indicating that a plurality of accessory devices of different accessory types are in a third status) (eg, 1314e). 3 accessory types (e.g., indicating that a plurality of accessory devices of a second accessory type and a different than the first accessory type are in a fourth state). In some embodiments, the first state indicator, the third state indicator, and the fourth state indicator each have a corresponding state indicator (e.g., a respective state indicator category in which each respective state indicator falls). corresponding to each status indicator category (e.g. alarm (e.g. co, co2, smoke, leak, alarm), action required (e.g. accessory not responding, account error, low battery, updated available) , critical status (important: home temperature, door or window open), quick actions (e.g., reversing accessory status, door unlock, window open, 5 lights on, 2 fans on category), sensor (e.g. humidity level, light level "34 lux", etc.)). In some embodiments, while sequentially displaying the first state indicator, the third state indicator, and the fourth state indicator, the computer system determines that the fifth state indicator should be displayed. any one of a different accessory device type (e.g., from the accessory device type indicated by the first, second, and third status indicators) In response to receiving an indicator that the above conditions have changed) and an indication that a fifth condition indicator should be displayed: (1) the fifth condition indicator is in the first condition indicator category; (e.g., the same state indicator category as the first state indicator), the computer system displays the fifth state indicator in a first position (e.g., adjacent to the first state indicator) and ( 2) according to determining that the fifth status indicator is in a second status indicator category (eg, a different status indicator category than the first status indicator category) (eg, the same status indicator category as the third status indicator); , displaying a fifth status indicator at a second location that is different from the first location (e.g., a location adjacent to the third status indicator); and (3) the fifth status indicator is in the third state. A first location according to a determination that it is in an indicator category (eg, a different status indicator category than the first status indicator category and the third status indicator category) (eg, the same status indicator category as the fourth status indicator category). and displaying the fifth status indicator at a third location (eg, adjacent to the fourth status indicator) different from the second location. Displaying the status indicators in order based on their respective categories provides the user with improved feedback regarding the importance or category of a particular status indicator, allowing the user to quickly identify the category to which a particular status indicator corresponds. make it possible to recognize Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use computer systems more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, the computer system detects a change in state of each accessory device of the first accessory type. In some embodiments, in response to detecting a change in state of each accessory device of the first accessory type, following determining that the state of each accessory device has changed to the first state, the computer system comprises: , adding each accessory device to a plurality of accessory devices of a first accessory type (e.g., light bulb, window, switch, outlet, lock (e.g., door lock, window lock), television) in a first state. . In some embodiments, the state of the respective accessory device of the first accessory type is changed to the second state in response to detecting a change in state of the respective accessory device (or alternatively, the second state). 1 state), the computer system sets each accessory device to a first accessory type (e.g., light bulb, window, switch, outlet, lock (e.g., light bulb, window, switch, outlet, lock)) in the first state. e.g. door locks, window locks), televisions) from multiple accessory devices. Thus, when additional accessories of each type are detected to be in their respective states, the additional accessories are added to their respective state indicators, thereby allowing the additional accessories to activate their state indicators. An accessory is removed from its respective status indicator when it is no longer detected that some accessories of each type are controlled and in their respective states, whereby the removed accessory activates the status indicator. not controlled by By updating multiple accessory devices based on accessory status, the system allows users to quickly change the status of accessories of the same type and status, allowing the accessory to (e.g., via press and hold) allow individual access. By allowing additional control of the system without cluttering the UI with additional displayed controllers, the usability of the system is enhanced and the user-system interface (e.g., operating/interacting with the system by helping the user to provide the correct input at times and by reducing user error), as well as enabling the user to use the system more quickly and efficiently, Reduces system power usage and improves battery life

In some embodiments, the first state indicator is displayed (or updated) based on the number of current accessory devices of the first accessory type that are in the first state. including. In some embodiments, prior to detecting the gesture, the first state indicator includes an indication of a first number of accessory devices (eg, of a first accessory type and in a first state). . In some embodiments, upon determining that a tenth accessory device of the first accessory device type has changed to the first state, the computer system outputs a first accessory device type different than the first number of accessory device indications. Update the first status indicator to include an indication of accessory devices number two (eg, of a second accessory type and in a second state). In some embodiments, upon determining that the second accessory has not changed to the first state, the computer system suggests a first number of accessory devices and suggests a second number of accessory devices and updates the first status indicator to include an indication of a different third number of accessory devices. In some embodiments, the second number of accessory devices is greater than the first number of accessory devices and the third number of accessory devices is less than the first number of accessory devices. Displaying the status indicator with suggestions based on the current number of accessory devices allows the user to quickly recognize the number of devices to which the status indicated by the status indicator corresponds, and the number of devices entered on the status indicator. Provide users with improved feedback on how many devices can potentially be affected. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use computer systems more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, upon determining that the gesture is a first type of gesture (e.g., 1350f) and directed at a first state indicator (e.g., 1350h), the computer system performs a predetermined period of time (e.g., 1350h). (e.g., 1 second, 5 seconds, 30 seconds) (e.g., a predetermined period of time after the gesture is detected) before discontinuing display of the first status indicator (e.g., 1350h in FIG. 13G is the same as in FIG. 13H). has been discontinued). In some embodiments, according to determining that the second gesture of the first type gesture is received before a predetermined amount of time after receiving the gesture of the first type, the first Stops displaying the status indicator. In some embodiments, following determining that the gesture is a first type of gesture and directed at a first state indicator, following determining that the state indicator is in the first category, the computer system: after a second predetermined period of time and/or after a predetermined period of time (e.g., lights are off, doors are locked, alarms are not active, etc.). Stop displaying the state indicator when represented by the state indicator changed to state. In some embodiments, according to determining that the gesture is a first type of gesture and directed at the first state indicator, according to determining that the state indicator is in a second category different from the first category. , the computer system determines whether the accessory device represented by the status indicator is activated after a second predetermined period of time and/or after a predetermined period of time (e.g., lights off, door locked, alarm Do not stop displaying the state indicator when represented by the state indicator changed to a particular state (such as not active). Stopping displaying the state indicator after detecting input of the first type of gesture provides visual feedback that an action has been performed that changes the state of the accessory device controlled via the state indicator. Provide to users. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use computer systems more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, a first state indicator (eg, 1314h) is displayed at a first location when the gesture is detected. In some embodiments, upon determining that the gesture is a first type of gesture (eg, 1350f) and directed at the first state indicator (eg, 1314h), the computer system performs the first Stop displaying one status indicator (eg, 1314h) and display a sixth status indicator (eg, 1314i in FIG. 13H) (eg, different from the first status indicator) in the first location. In some embodiments, the sixth state indicator was adjacent to the first state indicator before the gesture was detected. In some embodiments, displaying a state indicator that is different from the first state indicator at the first location includes displaying the state indicator that is different from the first state indicator from the third location to the third location via animation. Including moving to one location. Remove the state indicator by stopping displaying the state indicator and displaying a state indicator adjacent to its place after detecting the input of the first type of gesture, and remove the state indicator on the display (e.g., the display generating component The number of inputs required to display a new status indicator (eg, an adjacent status indicator) is reduced. Improves the usability of the system by reducing the number of inputs required to perform an action (e.g., prompting the user to provide appropriate input when operating/interacting with the system). It makes the user-system interface more efficient (by assisting and reducing user error), as well as reducing power usage by enabling users to use the system more quickly and efficiently. , to improve system battery life.

In some embodiments, upon determining that the gesture is a first type of gesture (eg, 1350i) and directed at the first state indicator (eg, 1314h), the computer system determines that the first state indicator is , that a plurality of accessory devices of the first accessory type have changed state (e.g., not the first state, a second state (e.g., or a second visual appearance that differs from the first visual appearance). changed from indicating that a plurality of accessory devices of a first accessory type are in a first state (e.g., or displayed in a first visual appearance) to indicate Visually updates the first status indicator (eg, 1314h in FIG. 13G) so that the In some embodiments, the first status indicator ceases display for a predetermined period of time after being visually updated. In some embodiments, upon determining that the gesture is a second type gesture (or a third type gesture) and directed at the first status indicator, the computer system controls the indicate that the plurality of accessory devices are in the second state (and/or maintain a first state indicator (eg, indicate that the plurality of accessory devices of the first accessory type are in the first state); d)) stop updating the first state indicator. The action of changing the state of the accessory device controlled via the state indicator by updating the state indicator to indicate that the plurality of accessory devices have changed state after detecting input of the first type of gesture. Provide visual feedback to the user that it has been done. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use computer systems more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, the first status indicator (eg, 1314h) is displayed on the first user interface (eg, 710). In some embodiments, the computer system sends a request (eg, 1350h) to display a second user interface (eg, 1360) that is different from the first user interface (eg, 710) (eg, the first while displaying the first status indicator on the user interface). In some embodiments, in response to receiving a request to display a second user interface (eg, 1350h), the computer system causes one or more accessory devices of the first accessory type to enter the first state. display a second user interface (eg, 1360) including a status indicator (eg, 1350i in FIG. 13I) indicating that the second user interface (eg, 1360) (eg, currently displayed user interface) represents a first group of accessory devices (e.g., rooms (e.g., parts of a building (e.g., home) (e.g., rooms (e.g., bedroom, living room, office, garage), porch)). group of devices specified in the device), and the first accessory device corresponds to the first group of accessory devices (e.g., while the first accessory device is in the first state), The one or more accessory devices of the first accessory type include the first accessory device, and the second user interface (eg, 1360) (eg, the currently displayed user interface) is of the first group. corresponding accessory devices, the first accessory device not corresponding to the first group of accessory devices (eg, while the first accessory device is in the first state), and one or more of the first accessory type does not include a first accessory device, hi some embodiments, in response to receiving a request to display a second user interface, the computer system comprises a first user interface including a first user interface display a status indicator that indicates the status of multiple devices when certain pre-determined conditions are met (e.g., a user interface that includes status for a group of accessory devices). associated with a user interface corresponding to a particular group of accessory devices, depending on whether they correspond and whether a particular device associated with the status indicator (e.g., same state and accessory type) corresponds to a group of accessory devices. A user is automatically provided with status indicators to indicate multiple devices, and the system is highly operable by performing actions when a set of conditions are met without requiring further user input. make the user-system interface more efficient (e.g., by helping the user to provide appropriate input and reducing user error when operating/interacting with the system); By enabling users to use the system more quickly and efficiently, system power usage is reduced and battery life is improved. Based on whether the user interface containing the state corresponds to a group of accessory devices, and whether the particular device (e.g., same state and accessory type) associated with the state indicator corresponds to the group of accessory devices. By displaying status indicators of the device status of the user, visual feedback is provided to the user regarding the status indicators that indicate the devices associated with the particular user interface being displayed. Improve the usability of the computer system by providing the user with improved visual feedback (e.g., assist the user in providing appropriate input when operating/interacting with the computer system) reduce power usage by making the user-system interface more efficient (by reducing user errors), as well as enabling users to use the computer system more quickly and efficiently and improve the battery life of your computer system.

In some embodiments, one or more accessory devices of the third accessory type are in a seventh state (eg, on/off, open, closed) before detecting the gesture (eg, 1350a-1350c and 1350f). ) is displayed. In some embodiments, the one or more accessory devices include an eighth accessory device of the third accessory type. In some embodiments, the gesture (eg, 1350f, 1350i) is a second type of gesture (eg, press and hold gesture) to the eighth state indicator (eg, 1314h) (eg, at a corresponding location). ), the eighth state indicator (e.g., 1314h in FIG. 13I) is in the seventh state (e.g., only one accessory of the third accessory type is in the seventh state). Upon determining that less than a predetermined number (e.g., less than two) of accessory devices of three accessory types are supported (and/or indicated), the computer system directs an eighth accessory control user corresponding to an eighth accessory device. Display an interface object target (eg, 1318) (eg, having more control and/or definition than the ninth accessory control user interface object target). In some embodiments, selection of an eighth accessory control user interface object target causes the eighth accessory device to enter an eighth state (e.g., 80% brightness, 20% brightness, 40% open, 20 % open, a state that is not completely opposite to the seventh state, a state that corresponds to a different property that can be exhibited by the device than the property that corresponds to the seventh state (e.g., on/off) color (e.g., blue, red, green, yellow, white).In some embodiments, the gestures (e.g., 1350f, 1350i) are second type gestures (e.g., press and hold gestures). , the third state indicator (eg, 1314h in FIG. 13I) is in the seventh state, according to the determination that the eighth state indicator (eg, 1314h) is directed (eg, at a corresponding location). Upon determining (and/or indicating) that it supports (and/or indicates) a predetermined number or more (e.g., two or more) of accessory devices of the accessory type, the computer system creates a ninth accessory control user interface object corresponding to an eighth accessory device. Display a target (e.g., control as in 718h) In some embodiments, selection of a ninth accessory control user interface object target causes an eighth accessory device (e.g., eighth accessory control change to a ninth state (e.g., on/off, open/closed, opposite to the seventh state) that is different from the eighth state (without displaying a user interface object object). In a form, the number of accessory user interface control object targets corresponding to the eighth accessory device is in the seventh state if the eighth state indicator is less than a predetermined number (eg, two) of the third accessory type A determination is made that the eighth state indicator indicates that less than a predetermined number (e.g., two) of the third accessory type is in the seventh state than a determination is made that a In some embodiments, the eighth accessory control user interface object target is larger than the ninth accessory control user interface object target When selected, the control represented by the state indicator By displaying controls that change the accessory device differently based on the number of without having to provide additional or different controls in some situations. By allowing additional control of the system without cluttering the UI with additional displayed controllers, the usability of the system is enhanced and the user-system interface (e.g., operating/interacting with the system by helping the user to provide the correct input at times and by reducing user error), as well as enabling the user to use the system more quickly and efficiently; Reduces system power usage and improves battery life.

In some embodiments, along with each status indicator (eg, eighth status indicator) corresponding to a single accessory device, each status indicator includes the accessory device's name (eg, alphanumeric characters). Following each status indicator (eg, eighth status indicator) corresponding to multiple accessory devices, each status indicator includes an indication (eg, a numeric representation) of the number of accessory devices represented.

In some embodiments, the computer system controls the operation of another type of status indicator (eg, 1314a-1314f and 1314j-1314l) (eg, climate control, sensor (eg, different types of sensors combined in a single status indicator). user interface (e.g., 1382, 1384, 1386 ). In some embodiments, device 700 does not stop displaying other types of controls after detecting a tap.

Note that the details of the process (eg, FIG. 14) described above with respect to method 1400 are also applicable in an analogous manner to the methods described above. For example, method 1400 optionally includes any one or more of the various method features described above with respect to methods 800 and 1000 . In another example, the voice message sent in method 800 can be a change or more status indicator, as described above with respect to method 1400 . For the sake of brevity, these details are not repeated below.

15A-15F illustrate exemplary user interfaces for playing voice messages, according to some embodiments. The user interfaces of these figures are used to illustrate the processes described below, including the process of FIG.

FIG. 15A shows device 700 displaying user interface 702 including affordance 704 . As shown in FIG. 15A, multicast notification 1502 is displayed over a portion of user interface 702 . Multicast notification 1502 indicates that an audio multicast message has been sent from John and three others to a group of devices designated Home 602 , including device 700 . In other words, because device 700 is designated home 602, device 700 is receiving audio multicast messages from John and three others. Multicast notification 1502 includes playback representation 1504 . In some embodiments, playback representation 1504 is selectable. In some embodiments, multicast notification 1502 may be combined with other multicast messages described above (e.g., multicast notification 708a, multicast notification 708, multicast notification 904, multicast notification 906a, multicast notification 906b, multicast notification 908a, multicast notification 908b, multicast Notification 908c, multicast notification 910) can include one or more features as described above. In some embodiments, such other multicast messages may include one or more features of multicast notification 1502 (eg, playback representation 1504). 15A, device 700 detects tap gesture 1550a on multicast notification 1502. In FIG.

15B-15F illustrate exemplary user interfaces for playing (eg, automatically) audio multicast messages. In some embodiments, the user interfaces described in connection with FIGS. 15B-15F replace one or more of the user interfaces described above, such as the user interfaces of 9I-9J and/or the user interfaces of 9M-9P. can be displayed in

As shown in FIG. 15B, in response to detecting tap gesture 1550a, device 700 displays user interface 1520 overlaid on a portion of user interface 702 (e.g., user input after detecting tap gesture 1550a). ), begins playing the audio multicast message received from John, as indicated by the current sender suggestion 1322a. In some embodiments, as part of initiating playback of audio, device 700 outputs audio through one or more speakers electronically coupled to device 700 .

As shown in FIG. 15B, user interface 1520 includes multicast home identifier 1532, which indicates that an audio multicast message sent to a group of devices associated with home 602 is being played. User interface 1520 also includes multicast recording affordance 736 , current sender suggestion 1522 a , graphical sender suggestion 1524 , and count suggestion 930 . In some embodiments, device 700 displays and operates in view of multicast recording affordance 736, current sender suggestions 922a-922b, graphical sender suggestions 924a-924b, and counting suggestions 930, respectively, described above. Device 700 uses one or more of the techniques described above to display and operate in view of multicast recording affordances 736, current sender suggestions 1522a, graphical sender suggestions 1524, and counting suggestions 930 used to do.

15A, notification 1502 indicates that four audio multicast messages were received (eg, "John and 3 others") and played. Accordingly, in FIG. 15B, user interface 1520 includes graphical sender suggestions 1524 corresponding to a portion of the received message. In FIG. 15B, graphical sender suggestions 1524 include graphical sender suggestions 1524a-1524c. Graphical suggestion 1524a corresponds to the first audio multicast message received from "John." Graphical suggestion 1524b corresponds to a second audio multicast message received (or sent) after the first audio multicast message. Graphical suggestion 1524c corresponds to a third audio multicast message received (or sent) after the second audio multicast message. In particular, in FIG. 15A, user interface 1520 does not include a graphical suggestion corresponding to the fourth audio multicast message because it was determined that a predetermined number (eg, 3) of graphical suggestions should be displayed simultaneously.

Graphical suggestions 1524a-1524c correspond to different multicast messages sent by different users (or devices) indicated by graphical suggestions 1524a-1524c depicting different representations. Graphical suggestion 1524a includes a visual representation of the sender (eg, "John") of the first audio multicast message (eg, the determined sender). Graphical suggestion 1524b includes a visual representation of the device (eg, speaker) that recorded the second audio multicast message. Graphical suggestion 1524c corresponds to the sender of a third audio multicast message (eg, "EA" for "Emma Appleseed" and associated with Home 602 as shown in FIG. 7AB). Contains visual representations of characters. Graphical suggestion 1524b includes a visual representation of the device instead of a visual representation of the sender, as the person who generated the second audio multicast message cannot be determined. Graphical suggestion 1524c includes a visual representation of the character instead of the user's visual representation because device 700 does not have access to the user's visual representation (eg, Emily). In some embodiments, multiple graphical suggestions include the same visual representation upon determination that the audio multicast messages corresponding to the graphical suggestions were sent by the same sender.

As shown in FIG. 15B, graphical suggestion 1524a is the largest graphical suggestion displayed on user interface 1520 and is displayed along with audio suggestion 1524a1 (eg, a circle surrounding graphical suggestion 1524a). The first audio multicast message from John is currently being played, as indicated by current sender suggestion 1522a. Thus, graphical suggestion 1524a is the full graphical representation and is displayed with audio suggestion 1524a1 indicating that an audio multicast message from John is currently being played. In some embodiments, audio suggestion 1524a1 is dynamic, varying in size according to the first audio multicast message being played. Graphical suggestion 1524 b is the second largest graphical suggestion displayed on user interface 1520 because the second audio multicast message is the next audio multicast message in queue to be played by device 700 . Graphical suggestion 1524c displays a third maximum audio multicast message displayed on user interface 1520 because the third audio multicast message is the next audio multicast message in queue to be played by device 700 after the second audio multicast message. is a graphical suggestion of In FIG. 15B, device 700 indicates that the first audio multicast message ends within a predetermined time period (eg, 0 seconds, 1 second) (or the next message (eg, second message) start within the period).

In response to determining that the first audio multicast message ends (or has ended) within a predetermined period of time (eg, based on a tap gesture on skip affordance 1534, without user input), the device 700 rotates graphical suggestions 1524b-1524c counterclockwise and stops displaying graphical suggestion 1524a. As shown in FIG. 15C, in response to determining that the first audio multicast message has ended (or has ended) within a predetermined period of time, graphical indication 1524b changes the graphical representation 1524a to what it was previously shown in FIG. 15B. Displayed where displayed, graphical suggestion 1524c is displayed where graphical representation 1524b was previously displayed in FIG. 15B. As shown in FIG. 15C, graphical suggestion 1524a is displayed at the same size that graphical suggestion 1524b was previously displayed in FIG. 15B, and graphical suggestion 1524c is displayed at the same size that graphical suggestion 1524b was previously displayed in FIG. 15B. Is displayed.

As shown in FIG. 15C, current sender suggestion 1522b has been replaced with current sender suggestion 1522a previously displayed in FIG. 15B. Thus, in FIG. 15C, the device 700 is now playing a second audio multicast message because the first audio multicast message has completed playing by the time user interface 1520 of FIG. 15C is displayed. As shown in FIG. 15C, graphical suggestion 1524b is now displayed with audio suggestion 1524b1 displayed using techniques similar to those described above with respect to audio suggestion 1524a1. As shown in FIG. 15C, graphical suggestion 1524c has audio surrounding graphical suggestion 1524c because device 700 is not currently outputting a third audio multicast message (eg, an audio multicast message corresponding to graphical suggestion 1524c). No suggestions. In some embodiments, the device 600 displays an animation that gradually increases the size of the graphical indications 1524b-1524c while transitioning between the display of the user interface 1520 of 15B and the display of the user interface 1520 of FIG. 15C. display.

As shown in FIG. 15C , device 700 has ceased displaying graphical indication 1524a (eg, the first audio multicast message has ended), so device 700 no longer receives the fourth message sent by Jane. A new graphical suggestion 1524d (as shown by current sender suggestion 1522d in FIG. 15E) corresponding to the audio multicast message is displayed. Graphical suggestion 1524d is the same size and displayed in the same location as graphical suggestion 1524c of FIG. 15B (eg, because graphical suggestion 1524d is now second in the queue for audio playback). In FIG. 15C, device 700 indicates that the second audio multicast message ends within a predetermined time period (eg, 0-1 seconds) (or the next message (eg, third audio multicast message) start (or end) within a period of time.

In response to determining that the second audio multicast message ends (or has ended) within the predetermined period of time, device 700 rotates graphical suggestions 1524c-1524d counter-clockwise to graphical suggestion 1524b. stop displaying . As shown in FIG. 15D, in response to determining that the second audio multicast message will expire within a predetermined time period, graphical suggestion 1524c displays the location and size at which graphical representation 1524b was previously displayed in FIG. 15C. , and graphical suggestion 1524d is displayed at the location and size that graphical representation 1524c was previously displayed in FIG. 15C.

As shown in FIG. 15D, the current sender suggestion 1522c has been replaced with the current send instruction 1522b previously displayed in FIG. 15C. Thus, in FIG. 15D, the device 700 is now playing the third audio multicast message because the second audio multicast message has completed playing by the time the user interface 1520 of FIG. 15D is displayed. Device 700 automatically plays a third audio multicast message without input intervention for a predetermined period of time after the end of playing the second audio multicast message. In some embodiments, device 700 plays predetermined sounds (eg, chimes, buzzes, tones) between played multicast messages. For example, the device 700 can play a first sound before automatically starting playing a second audio multicast message, and before automatically starting playing a third audio multicast message: A second sound (eg, the same chime) can be played. In some embodiments, the device 600 transitions the user interface (e.g., transitions the user interface 1520 of FIG. 15C to the user interface 1520 of FIG. 15D) while playing (after or before) the predetermined sound. do). In some embodiments, in response to detecting an input on skip affordance 1534 of FIG. 15C, device 700 ends playing the second voice message (eg, played in FIG. 2), start playing a third voice message and transition the user interface (e.g., transition user interface 1520 of FIG. 15C to user interface 1520 of FIG. 15D). . In FIG. 15D, device 700 indicates that the third audio multicast message ends within a predetermined time period (eg, 0-1 seconds) (or the next message (eg, fourth audio multicast message) start (or end) within a period of time.

In response to determining that the third audio multicast message ends (or has ended) within a predetermined period of time, device 700 moves (eg, changes the position of) graphical indication 1524d. , expand counterclockwise and stop displaying graphical suggestion 1524c. As shown in FIG. 15E, since graphical suggestion 1524d is the only displayed graphical suggestion, device 700 increases the size of graphical suggestion 1524d to a size greater than the size previously displayed of graphical suggestion 1524c. Let In some embodiments, graphical suggestion 1524d is displayed at a different location (eg, in the center of 1524) in FIG. 15E compared to the location of graphical suggestion 1524a in FIG. 15B. As shown in FIG. 15E, current sender suggestion 1522d has been replaced with current send instruction 1522c previously displayed in FIG. 15D. Thus, in FIG. 15E, the device 700 is now playing the fourth audio multicast message (eg, at graphical suggestion 1524d) because playback of the third audio multicast message was completed by the time user interface 1520 of FIG. 15E was displayed. corresponding audio multicast message). Device 700 may play a fourth audio multicast message, without input intervention, for a predetermined period of time after completion of playback of the third audio multicast message (e.g., after completion of the third audio multicast message). automatically play.

As shown in FIG. 15F, after determining that playback of the fourth voice message has ended, device 700 may proceed to include graphical suggestions 1524a-1524c without playing additional media (or multicast voice messages). Update the user interface 1520. In addition to redisplaying graphical suggestions 1524 a - 1524 c , device 700 displays playback affordance 926 . In some embodiments, in response to detecting tap gesture 1550f1 on playback affordance 926, device 700 begins playing the first audio multicast message, following the flow described in FIGS. 15B-15E. repeat. In some embodiments, in response to detecting a tap gesture 1550f2 (or a press and hold gesture) on multicast affordance multicast recording affordance 736, device 700 (eg, multicast voice message (eg, first a user interface for recording and sending voice messages to a group of devices associated with home 602 (to reply to the group of devices to which the second, third, and fourth voice multicast messages) were sent; indicate. Accordingly, in some embodiments, in response to detecting a tap gesture on multicast recording affordance 736, device 700 displays user interface 741 of FIG. 7C1, user interface 730 of FIG. 7C, or user interface 740 of FIG. 7D. display. In some embodiments, a tap gesture on multicast recording affordance 736 initiates audio recording until an edge recording input (eg, another tap gesture) is detected. When an edge recording input is detected, the recorded voice message is sent to the group of devices. In some embodiments, press and hold gestures are detected on the multicast recording affordance 736 . Audio recording is initiated when a press and hold gesture is detected. When the end of the press and hold gesture (eg, lift-off) is detected, the voice recording ends and the recorded voice message is sent to the group of devices. In some embodiments, detecting activation of the multicast recording affordance 736 (eg, tap gesture, tap and hold gesture) in any of the user interfaces of FIGS. 15B-15E plays back the current playback message. and start a process to record a voice message for sending to a group of devices.

FIG. 16 is a flow diagram illustrating a method for playing a voice message using a computer system, according to some embodiments. The method 1600 can be performed by a computer system (eg, 100, 300, 500, 634, 634, 634, 634, 634, 634, 634, 634, 634, 634, 634, 634, 634) in communication with a display generating component (eg, integrated display, connected display) and one or more input devices (eg, touch sensitive surface). 674, 700, 701, 850). Some acts of method 1600 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

As discussed below, method 1600 provides an intuitive way to play media content. The method reduces the cognitive burden on users of playing media messages, thereby creating a more efficient human-machine interface. For battery-operated computing devices, it conserves power and extends the time between battery charges by enabling users to play media content faster and more efficiently.

A computer system (e.g., 700) displays (1602) a media user interface (e.g., 1520) via a display generation component that includes, at a first location, a first media message (e.g., voice media, visual media, audiovisual media), and a second representation (1604) (e.g., 1524a in FIG. 15B) of the second media message at a second location different from the first location. A representation (1606) (eg, 1524b in FIG. 15B) is included. In some embodiments, the media user interface includes a representation of multiple (e.g., some, but not all) media messages of a group of media (e.g., the group of media is in an ordered set). .

In response to displaying the media user interface and not based on user input corresponding to the media user interface (eg, 1520), the computer system (eg, 700) plays a first media message ( 1608) (eg, output audio and/or video of the first media message).

In response to detecting completion of playback of the first media message (e.g., reaching the end of the first media message by playing to the end of the first media message), the computer system (e.g., 700) via the display generation component (eg, as in FIG. 15C) without displaying a representation (eg, 1524a) corresponding to the first media message (eg, any representation) (eg, , FIG. 15C) displays 1610 a second representation (eg, 1524b) of the second media message.

Displaying representations of various media messages at different locations provides the user with feedback on the order of the media messages and, for example, which media message is currently playing and which media message will play next. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, multiple (eg, some, but not all, three) representations of media messages are displayed in varying sizes based on an ordered set of orders. In some embodiments, the earlier a media message is in the ordered set, the larger the corresponding representation of the media message (e.g., the representation of the currently playing media message is the largest, then The representation of the media message that is played is the next largest, etc.). Therefore, the order of media messages is reflected in the relative size of their corresponding display images.

According to some embodiments, displaying a first representation (eg, 1524a in FIG. 15B) of a first media message (eg, audio media, visual media, audiovisual media) at a first location. places a first representation (eg, 1524a) of a first media message (eg, audio media, visual media, audiovisual media) in a first location and a first size (eg, as in FIG. 15B). Including displaying in According to some embodiments, displaying the second representation of the second media message at the second location (eg, 1524b of FIG. 15B) is performed at the second location and (eg, 1524b of FIG. 15B). displaying a second representation (eg, 1524b) of the second media message at a second size that is different (eg, smaller) than the first size (such as).

Displaying representations of various media messages at different sizes provides the user with feedback regarding the order of the media messages (e.g., larger size is earlier in an ordered set) and e.g. which media message is currently playing. In it, provide which media message to play next. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device; reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

According to some embodiments, a computer system (eg, 700), via a display generation component, renders a first representation of a first media message at a first location and a second representation at a second location. 2 media message simultaneously with a third size different from the first size (and optionally different from the second size), e.g. 2 size) and at a third location different from the first and second locations, displaying a third representation (eg, 1524c) of the third media message. In response to detecting completion of playback of the first media message (e.g., reaching the end of the first media message by playing to the end of the first media message), the computer system (e.g., 700) via the display generating component, without displaying the first representation (eg, 1524a) of the first media message in the first size and first location (eg, in FIG. 15C). The computer system (eg, 700) displays a second representation (eg, 1524b) of the second media message (eg, 1524b) in a second size and in a second location (eg, FIG. 15C). ) displays a third representation (eg, 1524c) of the third media message.

By displaying representations of various media messages in different locations and different sizes, feedback is provided to the user regarding the order in which the media messages should be played. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

According to some embodiments, a computer system (eg, 700) receives a target group of devices (eg, home multiple devices (eg, home automation system ))), and from a second remote user different from the first remote user (e.g., a device, including a computer system). A media message (e.g., a group of media) including a second media message that is a second (e.g., voice) message directed to a target group (e.g., a plurality of devices in a home (e.g., a home automation system)) , first, second, third, and fourth media messages) is received (eg, via a wireless network). A computer system (e.g., 700), via a display generation component, sends a notification (e.g., 1502) corresponding to a media message (e.g., data to the end of a time period (as described in further detail above)). display) in response to receipt of While displaying the notification (eg, 1502), the computer system (eg, 700) may, via one or more input devices, receive the notification (eg, activation of the notification displayed on the lock screen, media , a selection (eg, 1550a) corresponding to a group of media, the group of media being in an ordered set. In some embodiments, displaying the media user interface occurs in response to receiving the selection (eg, 1550a) of the notification (eg, 1502). In some embodiments, playing the first media message occurs in response to receiving the selection (eg, 1550a) of the notification (eg, 1502).

Displaying a media user interface and automatically playing a first media message in response to notification selection, thereby causing the computer system to perform two functions (display and playback) in response to the same input. , thereby reducing the need for the computer system to receive additional input to initiate playback. Improve device usability by reducing the number of inputs required to perform an action (e.g., prompt the user to provide appropriate input when manipulating/interacting with the device). It reduces power usage by making the user-device interface more efficient (by assisting and reducing user error), as well as enabling users to use the device more quickly and efficiently, thereby reducing power usage and improve battery life.

In some embodiments, the first representation (eg, 1524a), the second representation (eg, 1524b), and the third representation (eg, 1524c) are not displayed in one line (eg, the first , second and third places are not all on one line). In some embodiments, representations of displayed media are not displayed in one line (eg, as representations are deleted and new representations are added).

According to another embodiment, the first size is larger than the second size. In some embodiments, the second size is larger than the third size. In some embodiments, the second size and the third size are the same size. In some embodiments, a representation of the currently playing media file (eg, 1524a in FIG. 15B, 1524b in FIG. 15C) is larger than the size of multiple other (eg, all) display representations of the media message. displayed in size.

By displaying representations of various media messages in larger and larger sizes, feedback is provided to the user regarding the order of the media messages. For example, the earlier the corresponding media message, the larger the representation is in the ordered set of media messages, where (for example) the largest representation is the currently playing media item. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

According to some embodiments, completion of playback of the first media message (e.g., reaching the end of the first media message by playing to the end of the first media message) is detected In response, play the second media message (eg, output the audio and/or video of the second media) without receiving (or requiring) further user input. In some embodiments, the second media message is played with displaying a second representation of the second media message at the first size and at the first location.

By automatically playing the second media message in response to completion of the first media message, the computer system does not need to receive additional input to begin playing the second media message. Improve device usability by reducing the number of inputs required to perform an action (e.g., prompt the user to provide appropriate input when manipulating/interacting with the device). It reduces power usage by making the user-device interface more efficient (by assisting and reducing user error), as well as enabling users to use the device more quickly and efficiently, thereby reducing power usage and improve battery life.

In some embodiments, in response to detecting completion of playback of the second media message (e.g., skipping the remainder of the first media upon reaching the end of the second media message), The computer system plays the third media message (eg, outputs audio and/or video of the third media message) without receiving (or requiring) further user input. In some embodiments, the third media message is played with displaying a third representation of the third media message at the first size and at the first location. In some embodiments, the fifth representation of the fifth media is not displayed after detecting completion of playing the second media message.

According to some embodiments, a fourth representation (e.g., audio media, visual media, audiovisual media) of a fourth media message (e.g., audio media, visual media, audiovisual media) prior to detecting completion of playback of the first media message. 1524d) is not displayed concurrently with the first display of the first media (eg, in the media user interface, as shown in FIG. 15B). after (eg, in response to) detection of completion of playback of the first media message (eg, reaching the end of the first media message by playing to the end of the first media message); In some embodiments, in response to detecting completion of playback of the first media message, the computer system (e.g., 700), via the display generation component, (e.g., media user interface (e.g., , 1520)) of the fourth media message (e.g., 1524d) (e.g., in a third size and in a third location) (in some embodiments, the fourth media message is displayed simultaneously with the second and third representations of the media message).

playing a first predetermined number of media messages of a group of media messages without displaying additional media message representations of the group of media until the media message is played and the representation of the played media message is removed; Displaying the representation provides feedback to the user that additional media messages are available for playback without disrupting the user interface. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

In some embodiments, when the media user interface is first displayed (e.g., in response to selecting a displayed notification), the media user interface displays the first representation without including the fourth representation. , a second representation, and a third representation.

In some embodiments, in response to detecting completion of playback of the third media message (eg, reaching the end of the third media message, skipping the remainder of the third media message). , displaying a fourth representation (eg, 1524d) of the fourth media message (eg, 1524d) at a different (eg, larger) size than the first size and at a different location than the first, and (eg, FIG. 15E ) without receiving (or requiring) user input.

According to some embodiments, while playing the second media message, the computer system (eg, 700) prompts the media user interface (eg, 1520) for the next media user interface object target (eg, 1534). ) and, when activated (e.g., detecting a tap-on), causes playback of the second media message to complete (e.g., stops playing the currently playing media message, remaining of the second media message). Stop playing, skip past the rest of the second media message) (and optionally start playing a third media message). While playing the second media message, the computer system (eg, 700) activates (eg, tap on). In response to receiving activation of the next media user interface object target (eg, 1534), the computer system (eg, 700) renders a second representation of the second media message via the display generation component. Without displaying, a third representation of the third media message is displayed at the first location (and optionally at the first size) via the display generating component. In some embodiments, the third media message is played with displaying a third representation of the third media message at the first location at the first size. In some embodiments, the next media user interface object target completes the currently playing media message when activated. In some embodiments, by detecting the completion of the currently playing media message (e.g., unless the currently playing media message is the last media message of a group of media), the technology: Proceed to play the next media message in the ordered set. In some embodiments, in response to receiving activation of the next media user interface object object (eg, 1534), the computer system (eg, without playing to the end of the second media message) Stop playing the second media message and play a third media message.

By displaying the next media user interface object target, the computer system receives user input to skip and play the next media message without spending additional time playing the remainder of the currently playing media message. enable the reception of Providing the user with this additional control improves the usability of the device (e.g., assists the user in providing appropriate input when operating/interacting with the device, and reduces the risk of user error). reduce power usage and device battery life by making the user-device interface more efficient by reducing improve.

In some embodiments, the computer system does not display (e.g., in the media user interface) previous media user interface object targets (e.g., while playing the second media) and upon activation (e.g., tapping on detection), complete playback of the (eg, second) media message and begin playback of the previous (eg, first) media message. In some embodiments, the computer system prompts the next media user interface when the last (eg, fourth) media message (eg, of a group of media) is playing (eg, as in FIG. 15E). Stop displaying the object target.

In some embodiments, the first media message, the second media message and the third media message are part of a group of media. According to some embodiments, after playing the final media message (e.g., in response to detecting completion) (e.g., after playing or skipping all media messages of a group of media), the computer system (e.g., , 700), via the display generation component, renders the playback user interface object object of the media user interface (e.g., 926, at the first location, at the location where the fourth representation of the fourth media message was displayed). ) and, when activated (e.g., 1550f1, tap-on detection), before proceeding to playback of subsequent media messages (e.g., second, third, and fourth media messages). Play the message (eg, output audio and/or video of the first media message). In some embodiments, activation of the play user interface object target restarts the process again to play the media message of the group of media. In some embodiments, after playing a final media message (eg, of a group of media), the computer system retrieves first, second, and third media messages (and optionally a third media message) from the computer system. 4 media message) (eg, after a predetermined duration) (or otherwise render the media message inaccessible to the user).

Play media messages (and re-listen, re-view) becomes possible. By providing additional control options without cluttering the UI with additional controllers being displayed, the usability of the device is enhanced and the user-device interface (e.g., when operating/interacting with the device making the device more efficient (by helping the user to give the correct input and reducing user error), as well as by enabling the user to use the device more quickly and efficiently Reduced usage and improved battery life.

According to some embodiments, the first media message, the second media message, and the third media message (and optionally the fourth media message) are transmitted from a device comprising a computer system (e.g., Aimed at a target group of multiple devices in the home (eg, home automation systems). According to some embodiments, a computer system (eg, 700) displays a multicast affordance (eg, 736) on a media user interface via a display generation component and when activated (eg, 1550f2, tap-on detection), which initiates the process of sending an audio multicast message to a target group of devices. In some embodiments, the process of sending an audio multicast message includes displaying the user interface of FIG. 7C1 and is described with respect to FIG. 7C1. In some embodiments, the one or more devices are electronic devices that are part of a group of electronic devices to which the first, second, and third media messages are directed.

Respond to media messages containing messages by presenting a multicast affordance that initiates the process of sending an audio multicast message to one or more devices, without having to individually select which users/devices will receive the message It provides the user with capabilities, thereby reducing the number of inputs required by the user to provide a response. Improve device usability by reducing the number of inputs required to perform an action (e.g., prompting the user to provide appropriate input when manipulating/interacting with the device). It reduces power usage by making the user-device interface more efficient (by assisting and reducing user error), as well as enabling users to use the device more quickly and efficiently, thereby reducing power usage and improve battery life.

In some embodiments, the process of sending the audio multicast message includes recording the audio message with a computer system. In some embodiments, voice messages are made available for playback on a computer system (eg, as part of a group of media). In some embodiments, the computer system transmits the audio multicast message and also sends the first, second, and third media messages (and optionally a fourth media message) from the computer system to the computer. Delete from the system (or otherwise render the media message inaccessible to the user).

According to some embodiments, while playing respective media messages (eg, first media message, second media message), computer system (eg, 700) causes the display generation component to via the media user interface to display the name of each user (eg, 1524a, 1524b, 1524c, 1524d) from whom each media message was received. In some embodiments, each user's name is replaced when the playing media message changes (eg, previous name, current name, next name are displayed in the same place).

Displaying the name of each user from whom the currently playing media was received provides feedback to the user as to the source from which the computer system received the currently playing media message. Improve the usability of the device by providing improved visual feedback to the user (e.g., assist the user in providing appropriate input when operating/interacting with the device, reduce power usage and device battery life by making the user-device interface more efficient (by reducing errors in the improve.

According to some embodiments, the representation of the currently playing media message (e.g., 1524a in FIG. 15B, 1524b in FIG. 15C) is the background color (e.g., 1524b and 1524c in FIG. 15B and 1524c and 1524d in FIG. 15C) in a first background color (eg, green) different from the background color.

According to some embodiments, the media message is an audio message (eg, a multicast audio message). In some embodiments, a voice message (e.g., the first media message) is recorded (using a microphone) with an external device and sent by the external device to a group of devices and/or a computer system. is a voice message intended (eg, sent) to multiple external devices, including In some embodiments, the second media message is directed to (or sent to) a second plurality of external devices (eg, the same plurality of external devices, different from the plurality of external devices). 2 voice message.

By automatically playing (e.g., in sequence) multiple voice messages of a conversation, a computer system can play a conversation (e.g., multiple voice conversations between participants), thereby reducing the number of inputs required. Improve device usability by reducing the number of inputs required to perform an action (e.g., prompt the user to provide appropriate input when manipulating/interacting with the device). It reduces power usage by making the user-device interface more efficient (by assisting and reducing user error), as well as enabling users to use the device more quickly and efficiently, thereby reducing power usage and improve battery life.

In some embodiments, the total count of media messages for the group of media (eg, "4" in "1 of 4" at 930 in FIG. 15B) is displayed. In some embodiments, the number of media currently playing (e.g., along with the total count) (e.g., "1" in "1 of 4" at 930 in FIG. 15B, the ordered 1 of the group of media) in pairs) (eg, “3 of 4” as in FIG. 15D).

In some embodiments, the media message (eg, the first media message) corresponds to a multi-user device (eg, a device configured for multiple users, a device not configured for a particular user). do. In some embodiments, the media message (eg, the second media message) corresponds to a single user device (eg, a device configured for a single user). In some embodiments, the corresponding user visual indicators (eg, 1522a, 1524a, 1522c, 1524c) of the media message are displayed to the user concurrently with playing the corresponding media message. In some embodiments, visual indicators (eg, 1522b, 1524b) are displayed to indicate the multi-user devices while playing media messages corresponding to the multi-user devices.

Note that the details of the processes described above with respect to method 1600 (eg, FIG. 16) are also applicable in an analogous manner to the methods described above. For example, method 1600 optionally includes any one or more of the various method features described above with respect to methods 800 , 1000 and 1400 . In another example, the voice message of method 800 can be the same as the voice message sent in method 1600 . In another example, the displayed notification described with respect to method 800 may be the same notification displayed in method 1600 . In another example, the audio input sent to one or more devices for method 1200 can be the same audio message received and played by the computer system of method 1600 . For the sake of brevity, these details are not repeated here.

The foregoing, for purposes of explanation, has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. These embodiments were chosen and described in order to best explain the principles of the technology and its practical application. This enables others skilled in the art to best utilize the technology and various embodiments, with various modifications as appropriate for the particular application intended.

Having fully described the present disclosure and examples with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are intended to be included within the scope of this disclosure and examples as defined by the claims.

As noted above, one aspect of the present technology is the ability to collect and use data available from various sources to provide visual content (e.g., visual instructions) or voice input to users on one or more devices. To improve the transmission of any other content that provides the user with information as to whether it will be sent as a voice message. This disclosure indicates that, in some instances, this collected data includes personal information data that uniquely identifies a particular person or that can be used to contact or locate a particular person. I am thinking of getting.

This disclosure recognizes that the use of personal information data in the technology may be a use to the benefit of the user. For example, personal information data can be used to improve the display of suggestions so that each user can determine whether voice input is sent as a voice message. The use of such personal information data thus allows calculated control over the transmission of voice messages by the user. Furthermore, other uses for personal information data that benefit users are also contemplated by the present disclosure.

The present disclosure contemplates that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will adhere to robust privacy policies and/or practices. Specifically, such entities implement privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements to keep personal data confidential and strictly maintained. and should be used consistently. Such policies should be readily accessible by users and should be updated as data collection and/or use changes. Personal information from users should be collected for the entity's lawful and legitimate use and should not be shared or sold except for those lawful uses. Moreover, such collection/sharing should be done after informing and obtaining consent from the user. In addition, such entities protect and secure access to such Personal Data and ensure that others who have access to that Personal Data comply with their privacy policies and procedures. should consider taking all necessary steps to Further, such entities may themselves be evaluated by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be adapted to the specific types of personal information data collected and/or accessed and comply with applicable laws and standards, including jurisdiction-specific considerations. should be adapted. For example, in the United States, collection or access to certain health data may be governed by federal and/or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). However, health data in other countries may be subject to other regulations and policies and should be dealt with accordingly. Different privacy practices should therefore be maintained for different types of personal data in each country.

Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively prevents use of or access to personal information data. That is, the present disclosure contemplates that hardware and/or software elements may be provided to prevent or block access to such personal information data. For example, when receiving voice input from a user, the technology is configured to allow the user to choose to "opt-in" or "opt-out" of participation in the collection of personal information data during registration for the service or at any time thereafter. can be In another example, the user may choose not to provide voice input that is sent as a voice message based on the displayed suggestions. In yet another example, the user may choose to limit the content of the voice message corresponding to the voice input based on the displayed suggestions. In addition to providing "opt-in" and "opt-out" options, this disclosure contemplates providing notice regarding access or use of personal information. For example, the user may be notified upon download of an app that will access the user's personal data, and then reminded again just before the personal data is accessed by the app.

Furthermore, it is the intent of this disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. Additionally, where applicable, data de-identification can be used to protect user privacy in certain health-related applications. De-identification may include, where appropriate, removing certain identifiers (e.g. date of birth, etc.), controlling the amount or specificity of data stored (e.g. location data rather than address level). city level), controlling how data is stored (eg, aggregating data across users), and/or in other ways.

Thus, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure also covers the use of such personal information data. It is contemplated that it is also possible to implement the various embodiments without requiring access to the . That is, various embodiments of the present technology are not rendered inoperable by the lack of all or part of such personal information data. For example, visual instructions may be non-personal data or information, such as content requested by a device associated with a user, other non-personal information available for voice messaging services, or publicly available information. Only a minimal amount of personal information can be displayed to the user.

Claims (18)

a method,
In a computer system in communication with a display generation component,
Via said display generation component,
A first selectable user interface object object, wherein selection of said first selectable user interface object object records and transmits a first audio message to be output by a set of external devices. wherein the set of external devices includes a first set of one or more external devices and a second set of one or more external devices a selectable user interface object target;
A first accessory control user interface object target corresponding to a first external accessory device, wherein selection of said first accessory control user interface object target causes instructions to change the state of said first external accessory device. displaying a first user interface including a first accessory control user interface object object that initiates the sending process;
receiving input corresponding to a request to display a second user interface while displaying the first user interface;
displaying the second user interface in response to receiving the request to display the second user interface, the second user interface comprising:
a second selectable user interface object target, wherein a selection of said second selectable user interface object target is to be output by said first set of one or more external devices; a second selectable to initiate a process for recording and transmitting said second voice message to be output by said second set of one or more external devices without transmitting a voice message; user interface object targets, and
a second accessory control user interface object target corresponding to a second external accessory device, wherein selection of said second accessory control user interface object target causes instructions to change the state of said second external accessory device; displaying a second user interface including a second accessory control user interface object object that initiates the sending process;
A method, including 2. The method of claim 1, wherein the first external accessory device is different than the second external accessory device. The first selectable user interface object target is displayed at a first location via the display generation component, and the second selectable user interface object target is displayed via the display generation component. 3. A method according to any one of claims 1 to 2, wherein a message is displayed at the first location. said first external accessory device being included in said set of external devices;
The first user interface includes a third accessory control user interface object target corresponding to a third external accessory device, wherein selection of the third accessory control user interface object target corresponds to the third start a process that sends commands to change the state of external accessory devices in
The third external accessory device is not included in the set of external devices, and the selection of the first selectable user interface object target is the first external accessory device to be output by the third external accessory device. do not start the process of sending voice messages,
4. A method according to any one of claims 1-3. The first user interface corresponds to a top-level place, the second user interface corresponds to a first sub-place of a first type, and the one or more external devices. A second set of one or more external devices corresponds to the first sub-location, and the second set of one or more external devices corresponds to the first sub-location. 5. A method according to any one of claims 1 to 4, wherein it cannot be created to correspond to the second sub-location of said first type based on a set of two. The first user interface corresponds to a top-level location and the second user interface corresponds to a third sub-location of a second type corresponding to multiple sub-locations of a first type. and the second set of one or more external devices is based on the second set of one or more external devices included in the plurality of sub-locations of the first type , corresponding to the third sub-location. The second external accessory device is associated with the third sub-location of the second type, wherein the second external accessory device is different from the third sub-location. 7. The method of claim 6, also associated with a fourth sub-location of type 2. The second set of one or more external devices is associated with a first location, and the second set of one or more external devices is to be output by the first set of one or more external devices. said process for recording and transmitting said second voice message to be output by said second set of one or more external devices without transmitting a voice message at said first location; 8. The method of any one of claims 1-7, comprising displaying a suggestion. 9. The method of any one of claims 1-8, wherein the set of external devices comprises a personal mobile electronic device. 10. The method of any one of claims 1-9, wherein the second set of one or more external devices does not include a personal mobile electronic device. receiving input to disable sending voice messages to be output by the second set of one or more external devices after displaying the second user interface;
said first selectable user interface object target in response to receiving said input to disable sending voice messages to be output by said second set of one or more external devices. and via selection of said second selectable user interface object subject to disabling transmission of voice messages to be output by said second set of one or more external devices. and
11. The method of any one of claims 1-10, further comprising: the first user interface corresponds to a top-level location, the second user interface corresponds to a sub-location of the top-level location;
receiving, at a microphone in communication with the computer system, audio input comprising natural language requests of a digital assistant, the audio input comprising:
instructions for initiating a process of sending a voice message;
a location identifier;
receiving, including voice messages;
According to determining that the location identifier corresponds to the top-level location,
for playback on said set of one or more external devices, including said first set of one or more external devices and said second set of one or more external devices; sending the voice message without establishing a full-duplex voice communication channel with any device of the set;
According to determining that the location identifier corresponds to the sub-location,
to play on said second set of one or more external devices and not play on said first set of one or more external devices; transmitting the voice message without establishing a voice communication channel;
12. The method of any one of claims 1-11, further comprising: initiating respective processes for recording and transmitting respective voice messages to be output by the set of external devices;
starting said recording of a new voice message without further user input;
displaying a recording user interface including one or more dynamic suggestions of recorded speech;
13. A method according to any one of claims 1 to 12, comprising said computer system communicating with one or more input devices including a microphone, said process for recording and transmitting said first voice message to be output by said set of external devices; ,
receiving a second audio input via the microphone;
sending said first voice message to said set of external devices, said first voice message being based on said second audio input;
14. A method according to any one of claims 1 to 13, comprising The first set of one or more external devices includes a first speaker capable of playing the first voice message and the second voice message; 15. A method according to any preceding claim, wherein a second set comprises a second speaker capable of reproducing said first voice message and said second voice message. A computer program for causing a computer to perform the method according to any one of claims 1 to 15. a memory storing the computer program according to claim 16 ;
and one or more processors capable of executing said computer programs stored in said memory. A computer system comprising means for performing the method according to any one of claims 1 to 15.

Images