KR102026523B1 - Intelligent task discovery - Google Patents

Abstract

The present invention is directed to a system and process for operating messages and processing messages. In one example process, the electronic device receives a communication comprising a text string and determines whether a portion of the text string is associated with a predetermined data type of the plurality of data types. The data type is associated with at least one task. In accordance with a determination that a portion of the text string is associated with a data type, the electronic device receives a user input representing a predetermined one of the at least one task and, in response, causes the task to be performed based on the portion of the text string. Upon determining that a portion of the text string is not associated with the data type, the electronic device stops causing the task to be performed based on the portion of the text string.

Description

Intelligent Task Discovery {INTELLIGENT TASK DISCOVERY}

FIELD The present disclosure generally relates to intelligent automated assistants, and more particularly, to intelligent automated assistants for intelligent task discovery.

The intelligent automation assistant can help the user consume content on the electronic device. For example, the user may ask the intelligent automation assistant to read a text message received at the electronic device, so that the user may consume the text message while engaging in other activities (eg, driving). However, in order to perform subsequent operations using certain portion (s) of the content (eg, address, phone number, and URL mentioned in the text message), the user must provide the necessary inputs, for example, manually You have to start the service and manually recreate the relevant parts of the content in the service.

For example, if a user receives a text message containing a real address and wants to go to the real address, the user must manually launch the navigation application and provide (eg, type) the address to the navigation application. This experience is not only cumbersome but also potentially dangerous because it makes the user unable to focus on other ongoing activities (eg, driving). In other words, if the electronic device cannot conveniently initiate and / or perform the desired tasks using the relevant portions of the content, it will cause user frustration or negatively impact the user experience.

Example methods are disclosed herein. An example method for operating an automated assistant on an electronic device includes receiving, at the electronic device, a communication comprising a text string and determining whether a portion of the text string is associated with a predetermined data type of the plurality of data types; The data type has at least one task associated with it. In response to determining that a portion of the text string is associated with a data type of the plurality of data types, the electronic device receives a user input representing a task of at least one task associated with the data type, and in response to the user input representing the task, the task Is performed based on a portion of a text string. Upon determining that a portion of the text string is not associated with a data type of the plurality of data types, the electronic device stops causing the task to be performed based on the portion of the text string.

An example method includes, at an electronic device, receiving a communication comprising a text string and determining whether a portion of the text string is associated with a predetermined data type of the plurality of data types. In accordance with the determination that a portion of the text string is associated with a data type of the plurality of data types, the electronic device infers a task having one or more parameters based on the data type, wherein a predetermined parameter of the one or more parameters includes a portion of the text string. -Prints the inferred task. In accordance with the determination that the portion of the text string is not associated with the data type of the plurality of data types, the electronic device stops inferring the task having one or more parameters based on the data type.

Exemplary non-transitory computer readable storage media are disclosed herein. An exemplary non-transitory computer readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of the electronic device, cause the device to: receive a communication comprising a text string; Instructions for determining if a portion of the text string is associated with a predetermined data type of the plurality of data types, the data type having at least one task associated with it. The instructions further cause the electronic device to receive a user input indicating a task of at least one task associated with the data type, in accordance with determining that the portion of the text string is associated with the data type of the plurality of data types, the user representing the task. In response to the input, cause the task to be performed based on a portion of the text string. The instructions further cause the electronic device to stop causing the task to be performed based on the portion of the text string in accordance with the determination that the portion of the text string is not associated with the data type of the plurality of data types.

An exemplary non-transitory computer readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of the electronic device, cause the device to: receive a communication comprising a text string; Instructions for determining if a portion of the text string is associated with a predetermined data type of the plurality of data types. The instructions further cause the electronic device to infer a task having one or more parameters based on the data type in accordance with determining that a portion of the text string is associated with a data type of the plurality of data types, wherein the predetermined one of the one or more parameters is Contains a portion of a text string-causes the inferred task to be printed The instructions further cause the electronic device to stop inferring a task having one or more parameters based on the data type in accordance with determining that a portion of the text string is not associated with the data type of the plurality of data types.

Example devices are disclosed herein. An example electronic device includes one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors. One or more programs receive a communication comprising a text string; Instructions for determining if a portion of the text string is associated with a predetermined data type of the plurality of data types, the data type having at least one task associated with it. The one or more programs further receive user input indicative of the task of the at least one task associated with the data type in accordance with the determination that the portion of the text string is associated with the data type of the plurality of data types; In response to user input representing the task, instructions for causing the task to be performed based on the portion of the text string. The one or more programs further include instructions for stopping the task from being performed based on the portion of the text string in accordance with the determination that the portion of the text string is not associated with the data type of the plurality of data types.

An example electronic device includes one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors. One or more programs receive a communication comprising a text string; Instructions for determining if a portion of the text string is associated with a predetermined data type of the plurality of data types. The one or more programs further infer a task having one or more parameters based on the data type, in accordance with the determination that the portion of the text string is associated with the data type of the plurality of data types, wherein the predetermined one of the one or more parameters is the text string. Contains part of-contains instructions for outputting the inferred task. The one or more programs further include instructions to stop inferring a task having one or more parameters based on the data type in accordance with determining that a portion of the text string is not associated with the data type of the plurality of data types.

An example electronic device includes means for receiving a communication comprising a text string; Means for determining if a portion of the text string is associated with a predetermined data type of the plurality of data types, the data type having at least one task associated with it; In response to determining that a portion of the text string is associated with a data type of the plurality of data types, the user receives a user input representing a task of at least one task associated with the data type, and in response to the user input representing the task, the task returns the text string. Means for being performed based on a portion of the; And means for stopping the task to be performed based on the portion of the text string in accordance with the determination that the portion of the text string is not associated with the data type of the plurality of data types.

An example electronic device includes means for receiving a communication comprising a text string; Means for determining if a portion of the text string is associated with a predetermined data type of the plurality of data types; Inferring a task having one or more parameters based on the data type, in accordance with a determination that the portion of the text string is associated with the data type of the plurality of data types, wherein the predetermined one of the one or more parameters includes a portion of the text string; Means for outputting the inferred task; And determining to stop inferring a task having one or more parameters based on the data type in accordance with the determination that the portion of the text string is not associated with the data type of the plurality of data types.

For a better understanding of the various described embodiments, reference should be made to the following detailed description of the invention in connection with the following drawings in which like reference numerals indicate corresponding parts throughout the figures thereof.
1 is a block diagram illustrating a system and environment for implementing a digital assistant, in accordance with various examples.
2A is a block diagram illustrating a portable multifunction device implementing the client side portion of a digital assistant in accordance with some embodiments.
2B is a block diagram illustrating example components for event processing in accordance with various examples.
3 illustrates a portable multifunction device implementing a client side portion of a digital assistant according to various examples.
4 is a block diagram of an example multifunction device having a display and a touch-sensitive surface in accordance with various examples.
5A illustrates an example user interface for a menu of applications on a portable multifunction device, in accordance with various examples.
5B illustrates an example user interface for a multifunction device with a touch-sensitive surface that is separate from the display in accordance with various examples.
6A illustrates a personal electronic device according to various examples.
6B is a block diagram illustrating a personal electronic device according to various examples.
7A is a block diagram illustrating a digital assistant system or server portion thereof in accordance with various examples.
FIG. 7B illustrates the functions of the digital assistant shown in FIG. 7A, in accordance with various examples.
7C illustrates a portion of the ontology, in accordance with various examples.
8A-8E illustrate an example user interface of an electronic device in accordance with some embodiments.
9A-9C illustrate an example user interface of an electronic device in accordance with some embodiments.
10A-10C illustrate an example process for intelligent task search according to various examples.
11 illustrates an example process for intelligent task search according to various examples.
12 shows a functional block diagram of an electronic device according to various examples.
13 shows a functional block diagram of an electronic device according to various examples.

In the following description of the disclosure and embodiments, reference is made to the accompanying drawings, in which specific embodiments are shown by way of example in the drawings. It is to be understood that other embodiments and examples may be practiced and changes may be made without departing from the scope of the present disclosure.

Techniques for intelligent task discovery are desirable. As shown herein, the described techniques enhance productivity by improving a user's ability to interact with an electronic device. In addition, these techniques can reduce the computational requirements and battery power consumed in other ways as a result of the redundant response of the electronic device to user input.

Although the following description uses terms such as "first", "second", and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, the first input may be referred to as a second input and similarly, the second input may be referred to as a first input. The first input and the second input can both be outputs and in some cases can be separate and different inputs.

The terminology used in the description of the various described examples herein is for the purpose of describing particular examples and is not intended to be limiting. As used in the description of the various examples described and the appended claims, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the listed related items. The terms “include”, “including”, “comprise”, and / or “comprising”, as used herein, refer to features, integers, Specify presence of steps, steps, actions, elements, and / or components, but presence of one or more other features, integers, steps, actions, elements, components, and / or groups thereof Or it will be further understood that no addition is excluded.

The term "if" refers to "when" or "upon" or "in response to determining" or "depending upon," depending on the context. In response to detecting ”. Similarly, the phrase "if determined to" or "if a [condition or event mentioned] is detected" means "determined by" or "in response to determining by" or "[mentioned condition] depending on the context. Or "in detecting an event" or "in response to detecting a [referred condition or event]".

1. System and environment

1 shows a block diagram of a system 100 in accordance with various examples. In some examples, system 100 may implement a digital assistant. The terms "digital assistant", "virtual assistant", "intelligent automated assistant", or "automatic digital assistant" are used to interpret natural language input in colloquial and / or written forms to infer user intentions and infer a user. It may refer to any information processing system that performs operations based on intent. For example, to act as inferred user intent, the system may perform one or more of the following: identify a task flow with steps and parameters designed to achieve inferred user intent, and infer user intent Entering specific requirements from the task flow; Executing a task flow by calling a program, method, service, API, or the like; And generating an output response to the user in audio (eg, speech) and / or visual form.

Specifically, the digital assistant may accept the user request at least in part in the form of natural language commands, requests, statements, descriptions, and / or questions. Typically, a user request can obtain either an informative answer or the performance of a task by a digital assistant. The satisfactory response to the user request may be the provision of the requested informative answer, the performance of the requested task, or a combination of both. For example, the user may ask the digital assistant a question such as "Where am I now?" Based on the user's current location, the digital assistant may reply, "You are in Central Park near the preface." The user may also request to perform a task, for example, "Invite my friends to my girlfriend's birthday party next week." In response, the Digital Assistant can acknowledge the request by saying "Yes, I'll do it right away," and then invite each of his friends listed in the user's electronic address book with an appropriate event invitation on behalf of the user. Can be transmitted. During the performance of the requested task, the digital assistant may interact with the user in a continuous conversation that sometimes involves multiple exchanges of information over an extended period of time. There are a number of different ways to interact with the digital assistant to request information or to perform various tasks. In addition to providing verbal responses and taking programmed actions, the digital assistant can also provide responses in other visual or audio forms, such as text, notifications, music, video, animation, and the like.

As shown in FIG. 1, in some examples, the digital assistant may be implemented according to a client-server model. The digital assistant may comprise a client side portion 102 (hereinafter referred to as "DA client 102") running on user device 104, and a server side portion 106 (hereinafter referred to as "DA server) running on server system 108; (106) "). The DA client 102 can communicate with the DA server 106 via one or more networks 110. DA client 102 may provide client-side functions such as user-facing input and output processing, and communication with DA server 106. DA server 106 may provide server-side functions to any number of DA clients 102 each present on an individual user device 104.

In some examples, DA server 106 includes a client facing I / O interface 112, one or more processing modules 114, data and model 116, and an I / O interface 118 for external services. can do. The client facing I / O interface 112 may facilitate client facing input and output processing for the DA server 106. One or more processing modules 114 may use the data and models 116 to process speech input and determine a user's intention based on natural language input. In addition, one or more processing modules 114 perform task execution based on inferred user intentions. In some examples, DA server 106 may communicate with external services 120 via network (s) 110 for task completion or information acquisition. I / O interface 118 to external services may facilitate such communications.

User device 104 may be any suitable electronic device. For example, a user device may be a portable multifunction device (eg, device 200, described below with reference to FIG. 2A), a multifunction device (eg, device 400, described below with reference to FIG. 4). Or a personal electronic device (eg, device 600, described below with reference to FIGS. 6A and 6B). The portable multifunction device may be a mobile phone that also includes other functions, such as, for example, a PDA and / or music player function. Specific examples of portable multifunction devices include iPhone®, iPod Touch®, and iPad® devices from Apple Inc., Cupertino, California. Other examples of portable multifunction devices may include, without limitation, a laptop or tablet computer. Further, in some examples, user device 104 may be a non-portable multifunction device. In particular, the user device 104 can be a desktop computer, a game console, a television, or a television set top box. In some examples, user device 104 may include a touch-sensitive surface (eg, a touch screen display and / or touchpad). In addition, user device 104 may optionally include one or more other physical user-interface devices, such as a physical keyboard, mouse, and / or joystick. Various examples of electronic devices, such as multifunction devices, are described in more detail below.

Examples of communication network (s) 110 may include a local area network (LAN) and a wide area network (WAN), such as the Internet. The communication network (s) 110 may include, for example, Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE). ), Code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or It may be implemented using any known network protocol, including various wired or wireless protocols, such as any other suitable communication protocol.

Server system 108 may be implemented on one or more standalone data processing devices or a distributed computer network. In some examples, server system 108 may utilize various virtualization of third party service providers (eg, third party cloud service providers) to provide basic computing resources and / or infrastructure resources of server system 108. Devices and / or services may also be employed.

In some examples, user device 104 may communicate with DA server 106 via second user device 122. The second user device 122 can be similar or identical to the user device 104. For example, the second user device 122 may be similar to the devices 200, 400, or 600 described below with reference to FIGS. 2A, 4, and 6A and 6B. User device 104 may be configured to communicatively couple to second user device 122 via a direct connection, such as Bluetooth, NFC, BTLE, or the like, or via a wired or wireless network, such as a local Wi-Fi network. In some examples, the second user device 122 may be configured to act as a proxy between the user device 104 and the DA server 106. For example, the DA client 102 of the user device 104 is configured to send information (eg, a user request received at the user device 104) to the DA server 106 via the second user device 122. Can be. The DA server 106 may process the information and return related data (eg, data content in response to the user request) via the second user device 122 to the user device 104.

In some examples, user device 104 may be configured to communicate an abbreviated request for data to second user device 122 to reduce the amount of information sent from user device 104. The second user device 122 can be configured to determine to add supplemental information to the abbreviated request to generate a complete request and forward it to the DA server 106. This system architecture advantageously allows user device 104 (eg, a watch or similar small electronic device) with limited communication capabilities and / or limited battery power to have stronger communication capabilities and / or as a proxy for DA server 106. The use of a second user device 122 (eg, a mobile phone, laptop computer, tablet computer, etc.) with battery power may enable access to the services provided by the DA server 106. Although only two user devices 104, 122 are shown in FIG. 1, the system 100 may include any number and any type of user device configured in this proxy configuration to communicate with the DA server system 106. I will understand that.

Although the digital assistant shown in FIG. 1 may include both a client side portion (eg, DA client 102) and a server side portion (eg, DA server 106), in some examples, the functions of the digital assistant are: It can be implemented as a standalone application installed on a user device. In addition, the sharing of functions between the client portion and the server portion of the digital assistant may be different in different implementations. For example, in some examples, the DA client may be a thin-client that only provides user-facing input and output processing functions and delegates all other functions of the digital assistant to the back-end server.

2. Electronic device

It is now directed to embodiments of an electronic device for implementing a client-side portion of a digital assistant. 2A is a block diagram illustrating a portable multifunction device 200 having a touch-sensitive display system 212, in accordance with some embodiments. Touch-sensitive display 212 is sometimes referred to as "touch screen" for convenience and sometimes referred to as or referred to as "touch-sensitive display system". Device 200 may include memory 202 (optionally including one or more computer readable storage media), memory controller 222, one or more processing units (CPU) 220, peripheral interface 218, RF Circuitry 208, audio circuitry 210, speaker 211, microphone 213, input / output (I / O) subsystem 206, other input control devices 216, and external port 224 It includes. Device 200 optionally includes one or more optical sensors 264. Device 200 may optionally include one or more contact intensity sensors for detecting the intensity of contacts on device 200 (eg, a touch-sensitive surface, such as touch-sensitive display system 212 of device 200). 265). Device 200 optionally generates tactile outputs on device 200 (eg, a touch-sensitive surface, such as touch-sensitive display system 212 of device 200, or a touchpad of device 400). One or more tactile output generators 267 for generating tactile outputs on 455. These components optionally communicate over one or more communication buses or signal lines 203.

As used in the specification and claims, the term "strength" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of the contact (eg, finger contact) on the touch-sensitive surface, or touch-sensitive Refers to a substitute (proxy) for the force or pressure of contact on the surface. The intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds (eg, at least 256) distinct values. The intensity of the contact is optionally determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are used to measure force at various points on the touch-sensitive surface, optionally. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine an estimated force of the contact. Similarly, a pressure sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch sensitive surface. Alternatively, the size and / or changes in the detected contact area on the touch-sensitive surface, the capacitance of the touch-sensitive surface near the contact and / or the changes therein, and / or the touch-sensitive surface in the vicinity of the contact Resistance and / or variations thereon are optionally used as a substitute for the force or pressure of contact on the touch-sensitive surface. In some implementations, alternative measurements for contact force or pressure are directly used to determine whether the intensity threshold has been exceeded (eg, the intensity threshold is described in units corresponding to the alternative measurements). In some implementations, alternative measurements for contact force or pressure are converted to estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (eg, the intensity threshold of the pressure Pressure threshold measured in units). Using the intensity of the contact as an attribute of the user input, otherwise, displays the affordances (eg on a touch-sensitive display) and / or displays the user input (eg, touch-sensitive display, touch-sensitive). Allow user access to a surface or additional device functionality that may not be accessible by the user on a reduced size device with limited real estate to receive (via physical or mechanical controls such as knobs or buttons) do.

As used in the specification and claims, the term “tactile output” refers to the physical displacement of the device relative to its previous position, the component of the device relative to other components of the device (eg, a housing) (eg, a touch-sensitive surface). ), Or the displacement of the component with respect to the center of mass of the device to be detected by the user using the user's tactile sense. For example, in situations where a device or component of the device is in contact with a user's surface that is sensitive to touch (eg, a finger, palm, or other area of the user's hand), the tactile output generated by the physical displacement is generated by the user. It will be interpreted as tactile sensation corresponding to a perceived change in the physical properties of the device or component of the device. For example, movement of a touch-sensitive surface (eg, a touch-sensitive display or trackpad) may optionally be performed by a user to "down click" or "up click" of a physical actuator button. Interpreted as In some cases, the user may feel tactile, such as “down click” or “up click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that is physically pressed (eg, displaced) by the user's movement. Will feel. As another example, the movement of the touch-sensitive surface is optionally interpreted or sensed by the user as a "roughness" of the touch-sensitive surface even when there is no change in the flatness of the touch-sensitive surface. While these interpretations of touch by the user will be susceptible to the user's sensory perception, there are many touch sensory perceptions that are common to the majority of users. Thus, when tactile output is described as corresponding to a user's specific sensory perception (eg, "up click", "down click", "roughness"), unless otherwise stated, the generated tactile output is typical (or On average) corresponds to the physical displacement of the device or component thereof that will produce the described sensory perception of the user.

Device 200 is only one example of a portable multifunction device and device 200 optionally has more or fewer components than shown, optionally, combines two or more components, or optionally a component. It will be understood that they have different configurations or arrangements. The various components shown in FIG. 2A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application-specific integrated circuits (ASICs).

Memory 202 may include one or more computer readable storage media. Computer readable storage media may be tangible and non-transitory. Memory 202 may include fast random access memory, and may also include nonvolatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other nonvolatile solid state memory devices. Memory controller 222 may control access to memory 202 by other components of device 200.

In some examples, the non-transitory computer readable storage medium of memory 202 may be instructions from an instruction execution system, apparatus or device, such as a computer based system, a processor-containing system, or an instruction execution system, apparatus or device. Can be used to store instructions (eg, to perform aspects of processes 1000 and 1100 described below) for use by or in connection with other systems capable of executing instructions. In other examples, the instructions (eg, for performing aspects of the processes 1000, 1100 described below) may be stored on a non-transitory computer readable storage medium (not shown) of the server system 108 or in memory. And may be shared between the non-transitory computer readable storage medium of 202 and the non-transitory computer readable storage medium of server system 108. In the context of this specification, a “non-transitory computer readable storage medium” can be any medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

Peripheral interface 218 may be used to couple input and output peripherals of the device to CPU 220 and memory 202. One or more processors 220 drives or executes various software programs and / or sets of instructions stored in memory 202 to perform various functions for device 200 and to process data. In some embodiments, peripheral interface 218, CPU 220, and memory controller 222 may be implemented on a single chip, such as chip 204. In some other embodiments, they may be implemented on separate chips.

Radio frequency (RF) circuitry 208 receives and transmits RF signals, also referred to as electromagnetic signals, RF circuitry 208 converts electrical signals to / from electromagnetic signals and communicates via communications signals. And RF circuitry 208. Optionally, the RF circuitry 208 optionally includes an antenna system, an RF transceiver, one or more amplifiers, tuners, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM). It includes well-known circuitry for performing these functions, including without limitation, cards, memory, etc. The RF circuitry 208 is optionally also referred to as the World Wide Web (WWW) by wireless communication. Internet, intranet and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and / or networks such as metropolitan area networks (MAN), and Communicate with other devices RF circuitry 208 optionally includes well-known circuitry for detecting near field communication (NFC) fields, such as by a short-range communication radio. Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), evolution, data-only (EV-DO), HSPA, HSPA +, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), NFC, wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), voice over Internet Protocol (VoIP) , Wi-MAX, e-mail protocols (eg, Internet message access protocol (IMAP) and / or post office protocol (POP)), instances Messaging (eg, extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS), or this document) Any one of a plurality of communication standards, protocols and techniques, including but not limited to any other suitable communication protocol, including but not yet developed as of the filing date of the application, is used.

The audio circuitry 210, the speaker 211, and the microphone 213 provide an audio interface between the user and the device 200. The audio circuitry 210 receives audio data from the peripheral interface 218, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 211. The speaker 211 converts an electrical signal into a sound wave that can be heard by a person. Audio circuitry 210 also receives electrical signals converted from sound waves by microphone 213. The audio circuitry 210 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 218 for processing. Audio data may be retrieved from and / or transmitted to the memory 202 and / or the RF circuitry 208 by the peripheral interface 218. In some embodiments, audio circuitry 210 also includes a headset jack (eg, 312 of FIG. 3). The headset jack is provided between the audio circuitry 210 and the detachable audio input / output peripherals, such as a headset having output-only headphones or an output (eg one or both ear headphones) and an input (eg a microphone). Provide an interface.

I / O subsystem 206 couples input / output peripherals on device 200, such as touch screen 212 and other input control devices 216, to peripheral interface 218. I / O subsystem 206 may optionally include one or more input controllers for display controller 256, light sensor controller 258, intensity sensor controller 259, haptic feedback controller 261, and other input or control devices. 260). One or more input controllers 260 receive / transmit electrical signals from / to other input control devices 216. Other input control devices 216 optionally include physical buttons (eg, push button, rocker button, etc.), dial, slider switch, joystick, click wheel, and the like. In some alternative embodiments, input controller (s) 260 is optionally coupled to (or not coupled to) any of a pointer device such as a keyboard, an infrared port, a USB port, and a mouse. One or more buttons (eg, 308 of FIG. 3) optionally include up / down buttons for volume control of speaker 211 and / or microphone 213. One or more buttons optionally include a push button (eg, 306 of FIG. 3).

Quick press of a push button is described in US Patent Application No. 11 / 322,549, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," filed December 23, 2005 (US Pat. No. 7,657,849). Likewise, the touch screen 212 can be unlocked, or the process of unlocking the device using a gesture on the touch screen can be started, which is hereby incorporated by reference in its entirety. Longer pressing of a push button (eg, 306) may power on or off device 200. The user can customize the function of one or more of the buttons. Touch screen 212 is used to implement virtual or soft buttons and one or more soft keyboards.

Touch-sensitive display 212 provides an input interface and an output interface between the device and the user. Display controller 256 receives and / or transmits electrical signals from / to touch screen 212. The touch screen 212 displays visual output to the user. The visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output may correspond to user interface objects.

Touch screen 212 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. The touch screen 212 and the display controller 256 may be connected to the touch screen 212 (and any movement of the contact, or with any associated modules and / or sets of instructions in the memory 202) or Interrupt) and convert the detected contact into an interaction with user interface objects (eg, one or more soft keys, icons, webpages or images) displayed on touch screen 212. In an exemplary embodiment, the point of contact between the touch screen 212 and the user corresponds to the user's finger.

The touch screen 212 may use liquid crystal display (LCD) technology, light emitting polymer display (LPD) technology, or light emitting diode (LED) technology, but other display technologies may be used in other embodiments. Touch screen 212 and display controller 256 include capacitive, resistive, infrared and surface acoustic wave techniques, as well as other proximity sensor arrays or other elements, to determine one or more contact points with touch screen 212. Any, but not limited to, any of the currently known or later developed touch sensing techniques can be used to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology such as found in iPhone® and iPod Touch® from Apple Inc., Cupertino, Calif., Is used.

Touch sensitive displays in some embodiments of touch screen 212 are described in the following U.S. Pat. It may be similar to the multi-touch sensitive touchpads described in US Patent Publication No. 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, touch screen 212 displays visual output from device 200, while touch-sensitive touchpads do not provide visual output.

In some embodiments a touch-sensitive display of touch screen 212 may be as described in the following applications: (1) Filed May 2, 2006 and entitled “Multipoint Touch Surface Controller” US Patent Application No. 11 / 381,313; (2) US Patent Application No. 10 / 840,862, filed May 6, 2004 and entitled "Multipoint Touchscreen"; (3) US patent application Ser. No. 10 / 903,964, filed Jul. 30, 2004, entitled “Gestures For Touch Sensitive Input Devices”; (4) US patent application Ser. No. 11 / 048,264, filed Jan. 31, 2005 and entitled "Gestures For Touch Sensitive Input Devices"; (5) US patent application Ser. No. 11 / 038,590, filed Jan. 18, 2005, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices”; (6) US patent application Ser. No. 11 / 228,758, filed Sep. 16, 2005 and entitled "Virtual Input Device Placement On A Touch Screen User Interface"; (7) US patent application Ser. No. 11 / 228,700, filed Sep. 16, 2005 and entitled "Operation Of A Computer With A Touch Screen Interface"; (8) US patent application Ser. No. 11 / 228,737, filed Sep. 16, 2005 and entitled "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard"; (9) US patent application Ser. No. 11 / 367,749, filed March 3, 2006, entitled "Multi-Functional Hand-Held Device." All of these applications are incorporated herein by reference in their entirety.

The touch screen 212 can have a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. The user may contact the touch screen 212 using any suitable object or appendage, such as a stylus, a finger, or the like. In some embodiments, the user interface is designed to operate primarily using finger based contacts and gestures, which may be less precise than stylus based input due to the wider contact area of a finger on the touch screen. In some embodiments, the device translates the rough finger-based input into precise pointer / cursor positioning or command to perform the actions desired by the user.

In some embodiments, in addition to the touch screen, device 200 may include a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad, unlike the touch screen, is a touch sensitive area of the device that does not display visual output. The touchpad may be a touch-sensitive surface separate from the touch screen 212 or an extension of the touch-sensitive surface formed by the touch screen.

Device 200 also includes a power system 262 for providing power to various components. The power system 262 includes a power management system, one or more power sources (eg, battery, alternating current (AC)), recharging system, power failure detection circuit, power converter or inverter, power status indicator (eg, light emitting diode (LED) ), And any other components associated with the generation, management, and distribution of power within portable devices.

Device 200 may also include one or more optical sensors 264. 2A shows an optical sensor coupled to an optical sensor controller 258 in I / O subsystem 206. The optical sensor 264 may include a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Optical sensor 264 receives light from the environment, which is projected through one or more lenses, and converts the light into data representing an image. In conjunction with imaging module 243 (also referred to as a camera module), optical sensor 264 may capture still images or video. In some embodiments, the optical sensor is located on the back side of the device 200 opposite the touch screen display 212 on the front of the device so that the touch screen display is a viewfinder for still and / or video image acquisition. To be used. In some embodiments, an optical sensor is located on the front of the device to allow the user's image to be acquired for video conferencing while the user views other video conferencing participants on the touch screen display. In some embodiments, the position of the light sensor 264 is changed by the user (eg, by rotating the lens and sensor in the device housing) such that the single light sensor 264 can be used for video conferencing and freezing with the touch screen display, and And / or both sides of the video image acquisition.

Device 200 also optionally includes one or more contact intensity sensors 265. 2A shows a contact intensity sensor coupled to an intensity sensor controller 259 in the I / O subsystem 206. Contact intensity sensor 265 may optionally include one or more piezo-resistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (eg, Sensors used to measure the force (or pressure) of the contact on the touch-sensitive surface. The contact intensity sensor 265 receives contact intensity information (eg, pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is located with or close to a touch-sensitive surface (eg, touch-sensitive display system 212). In some embodiments, at least one contact intensity sensor is located on the back side of the device 200 that is opposite the touch screen display 212 located on the front side of the device 200.

Device 200 may also include one or more proximity sensors 266. 2A shows proximity sensor 266 coupled to peripherals interface 218. Alternatively, proximity sensor 266 may be coupled to input controller 260 in I / O subsystem 206. Proximity sensor 266 is described in US patent application Ser. No. 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; And 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals," which are hereby incorporated by reference in their entirety. In some embodiments, the proximity sensor turns off and disables touch screen 212 when the multifunction device is located near the user's ear (eg, when the user is making a phone call).

Device 200 also optionally includes one or more tactile output generators 267. 2A shows a tactile output generator coupled to haptic feedback controller 261 in I / O subsystem 206. The tactile output generator 267 may optionally include one or more electroacoustic devices, such as speakers or other audio components, and / or a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output. Electromechanical devices that convert energy into linear motion, such as generating components (eg, components that convert electrical signals into tactile outputs on the device). The contact intensity sensor 265 receives tactile feedback generation instructions from the haptic feedback module 233 to generate tactile outputs on the device 200 that can be sensed by a user of the device 200. In some embodiments, the at least one tactile output generator is located with or close to a touch-sensitive surface (eg, touch-sensitive display system 212), and optionally, perpendicularly (eg, to) the touch-sensitive surface. To generate a tactile output by moving in or out of the surface of the device 200 or laterally (eg, back and forth in the same plane as the surface of the device 200). In some embodiments, at least one tactile output generator sensor is located on the back side of the device 200 opposite the touch screen display 212 located on the front side of the device 200.

Device 200 may also include one or more accelerometers 268. 2A shows accelerometer 268 coupled to peripherals interface 218. Alternatively, accelerometer 268 may be coupled to input controller 260 in I / O subsystem 206. Accelerometer 268 is described in US Patent Publication No. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices" and US Patent Publication No. 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer". As may be, both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or landscape view on a touch screen display based on analysis of data received from one or more accelerometers. The device 200 optionally includes a magnetometer (not shown), in addition to the accelerometer (s) 268, and a GPS for obtaining information regarding the position and orientation (eg, vertical or horizontal) of the device 200. Or a GLONASS or other global navigation system) receiver (not shown).

In some embodiments, software components stored in memory 202 may include operating system 226, communication module (or set of instructions) 228, contact / motion module (or set of instructions) 230, graphics module ( Or set of instructions) 232, a text input module (or set of instructions) 234, a global positioning system (GPS) module (or set of instructions) 235, a digital assistant client module 229, and an application. (Or sets of instructions) 236. In addition, the memory 202 may store data and models, such as user data and models 231. In addition, in some embodiments, memory 202 of FIG. 2A or 470 of FIG. 4 stores device / global internal state 257 as shown in FIGS. 2A and 4. The device / global internal state 257 includes one or more of the following: active application state indicating which applications, if present, are currently active; Display status, indicating what applications, views, or other information occupy various regions of touch screen display 212; Sensor status including information obtained from various sensors of the device and input control devices 216; And location information regarding the location and / or attitude of the device.

Operating system 226 (e.g., an embedded operating system such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or VxWorks) provides general system tasks (e.g., memory management, storage device control, power management, etc.). Various software components and / or drivers for controlling and managing the network, and facilitating communication between various hardware and software components.

The communication module 228 facilitates communication with other devices via one or more external ports 224, and may also be used to process data received by the RF circuitry 208 and / or external ports 224. Contains software components. External ports 224 (e.g., Universal Serial Bus (USB), FireWire, etc.) may be coupled to other devices either directly or indirectly through a network (e.g., the Internet, wireless LAN, etc.). It is composed. In some embodiments, the external port is a multi-pin (eg, 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used in iPod® (trademark of Apple Inc.) devices.

The contact / motion module 230 optionally detects contact with the touch screen 212 (along with the display controller 256), and other touch-sensitive devices (eg, a touchpad or physical click wheel). The contact / motion module 230 determines whether a contact has occurred (eg, detecting a finger-down event), the intensity of the contact (eg, the force or pressure of the contact, or the contact's Determining a substitute for force or pressure, determining if there is a movement of contact and tracking movement across the touch-sensitive surface (eg, detecting one or more finger-dragging events). And various software components for performing various operations related to the detection of the contact, such as determining if the contact has been interrupted (eg, detecting a finger-up event or contact interruption). Include them. The contact / motion module 230 receives contact data from the touch sensitive surface. Determining the movement of a contact point represented by a series of contact data optionally determines the speed (size), velocity (size and direction), and / or acceleration (change in size and / or direction) of the contact point. It involves doing. These operations optionally apply to single contacts (eg, one finger contacts) or to multiple simultaneous contacts (eg, “multi-touch” / multiple finger contacts). In some embodiments, contact / motion module 230 and display controller 256 detect a contact on the touchpad.

In some embodiments, contact / motion module 230 may determine one or more sets of intensity thresholds to determine whether an action was performed by a user (eg, to determine whether the user “clicked” on an icon). I use it. In some embodiments, at least a subset of the intensity thresholds is determined in accordance with the software parameters (eg, the intensity thresholds are not determined by the activation thresholds of specific physical actuators, and changing the physical hardware of the device 200 is Can be adjusted without). For example, the mouse “click” threshold of a trackpad or touch screen display can be set to any of a wide range of predefined thresholds without changing the trackpad or touch screen display hardware. In addition, in some implementations, a user of the device may be configured to perform a predetermined set of intensity thresholds (eg, by adjusting individual intensity thresholds and / or by adjusting a plurality of intensity thresholds all at once with a system level click “strength” parameter). Software settings for adjusting one or more are provided.

The contact / motion module 230 optionally detects a gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different motions, timings, and / or intensities of detected contacts). Thus, the gesture is optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture may detect a finger-down event and then finger-up at the same location (or substantially the same location) (eg, the location of the icon) as the finger-down event. Detecting a (lift off) event. As another example, detecting a finger swipe gesture on a touch-sensitive surface detects a finger-down event followed by one or more finger-dragging events, followed by finger-up (liftoff). ) Detecting the event.

Graphics module 232 renders and displays graphics on touch screen 212 or other displays, including components for changing the visual effects (eg, brightness, transparency, saturation, contrast, or other visual properties) of the displayed graphics. It includes various known software components for display. As used herein, the term “graphics” includes without limitation text, web pages, icons (eg, user interface objects including soft keys), digital images, videos, animations, and the like. , Any object that can be displayed to the user.

In some embodiments, graphics module 232 stores data representing a graphic to be used. Each graphic is optionally assigned a corresponding code. Graphics module 232 receives one or more codes from applications or the like that specify graphics to be displayed, along with coordinate data and other graphics attribute data, if necessary, and then generates screen image data to display controller 256. Output

The haptic feedback module 233 is used by the haptic output generator (s) 267 to generate tactile outputs at one or more locations on the device 200 in response to user interactions with the device 200. And various software components for generating the data.

Text input module 234, which may be a component of graphics module 232, requires various applications (eg, contact 237, email 240, IM 241, browser 247, and text input). Soft keyboards for entering text in any other application).

The GPS module 235 determines the location of the device and provides this information to provide various applications (e.g., phone 238 for use in location-based dialing; camera 243 as photo / video metadata and location-based services, For example, applications that provide weather widgets, local phone book widgets, and map / navigation widgets.

The digital assistant client module 229 may include various client-side digital assistant instructions to provide client-side functions of the digital assistant. For example, the digital assistant client module 229 may include various user interfaces of the portable multifunction device 200 (eg, microphone 213, accelerometer (s) 268, touch-sensitive display system 212, optical sensors) 229, other input control devices 216, etc.) may receive voice input (eg, speech input), text input, touch input, and / or gesture input. The digital assistant client module 229 is hearing through various output interfaces of the portable multifunction device 200 (eg, speaker 211, touch-sensitive display system 212, tactile output generator (s) 267, etc.). It may provide output in an enemy (eg, speech output), visual, and / or tactile form. For example, the output may be provided as voice, sound, alert, text message, menu, graphic, video, animation, vibration, and / or combinations of two or more thereof. During operation, the digital assistant client module 229 may communicate with the DA server 106 using the RF circuitry 208.

User data and models 231 may be configured to provide various data associated with the user (eg, user specific vocabulary data, user preference data, user specific name pronunciation, user's electronic address book, to-do list, shopping) to provide client-side functions of the digital assistant. Data from lists, etc.). In addition, the user data and models 231 may include various models for processing user input and determining user intention (eg, speech recognition model, statistical language model, natural language processing model, ontology, task flow model, service model, etc.). ) May be included.

In some examples, digital assistant client module 229 uses various sensors, subsystems, and peripheral devices of portable multifunction device 200 to establish a situation, current user interaction, and / or current user input associated with the user. To collect additional information from the environment of the portable multifunction device 200. In some examples, digital assistant client module 229 may provide contextual information or a subset thereof with user input to DA server 106 to help infer user intent. In some examples, the digital assistant can also use the contextual information to determine how to prepare the outputs and deliver them to the user. Situation information may be referred to as situation data.

In some examples, the contextual information accompanying the user input may include sensor information such as lighting, ambient noise, ambient temperature, an image or video of the surrounding environment, and the like. In some examples, the contextual information may also include the physical state of the device, such as device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, and the like. In some examples, information related to the software status of the DA server 106, such as running processes, installed programs, past and current network activity, background services, error logs, resource usage, etc., of the portable multifunction device 200 may be used. May be provided to the DA server 106 as contextual information associated with the input.

In some examples, digital assistant client module 229 may optionally provide information (eg, user data 231) stored on portable multifunction device 200 in response to requests from DA server 106. . In some examples, digital assistant client module 229 may also derive additional input from the user via natural language conversation or other user interfaces upon request by DA server 106. The digital assistant client module 229 may pass additional input to the DA server 106 to assist the DA server 106 in deducing and / or fulfilling the intention of the user's intentions expressed in the user request.

A more detailed description of the digital assistant is described below with reference to FIGS. 7A-7C. It should be appreciated that the digital assistant client module 229 may include any number of submodules of the digital assistant module 726 described below.

Applications 236 can include the following modules (or sets of instructions), or a subset or superset thereof:

연락처 모듈(237)(때때로 주소록 또는 연락처 목록으로 지칭됨);

A contact module 237 (sometimes referred to as an address book or contact list);

전화 모듈(238);

Telephone module 238;

화상 회의 모듈(239);

Video conferencing module 239;

이메일 클라이언트 모듈(240);

Email client module 240;

인스턴트 메시징(IM) 모듈(241);

Instant messaging (IM) module 241;

운동 지원 모듈(242);

Exercise support module 242;

정지 및/또는 비디오 이미지들을 위한 카메라 모듈(243);

Camera module 243 for still and / or video images;

이미지 관리 모듈(244);

Image management module 244;

비디오 플레이어 모듈;

Video player module;

음악 플레이어 모듈;

Music player module;

브라우저 모듈(247);

Browser module 247;

캘린더 모듈(248);

Calendar module 248;

날씨 위젯(249-1), 주식 위젯(249-2), 계산기 위젯(249-3), 알람 시계 위젯(249-4), 사전 위젯(249-5), 및 사용자에 의해 획득되는 다른 위젯들뿐만 아니라 사용자-생성 위젯들(249-6) 중 하나 이상을 포함할 수 있는 위젯 모듈들(249);

Weather widget 249-1, stock widget 249-2, calculator widget 249-3, alarm clock widget 249-4, dictionary widget 249-5, and other widgets obtained by the user Widget modules 249 that may include one or more of user-created widgets 249-6 as well;

사용자-생성 위젯들(249-6)을 만들기 위한 위젯 생성기 모듈(250);

Widget generator module 250 for creating user-created widgets 249-6;

검색 모듈(251);

Search module 251;

비디오 플레이어 모듈 및 음악 플레이어 모듈을 통합하는 비디오 및 음악 플레이어 모듈(252);

A video and music player module 252 integrating the video player module and the music player module;

메모 모듈(253);

A memo module 253;

지도 모듈(254); 및/또는

Map module 254; And / or

온라인 비디오 모듈(255).

Online video module 255.

Examples of other applications 236 that may be stored in memory 202 are other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-operated applications, encryption, digital rights management. , Speech recognition, and speech replication.

In addition to the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, the contact module 237 may include an address book or a contact list (eg, Stored in the application internal state 292 of the contact module 237 in the memory 202 or the memory 470, including: adding name (s) to the address book; ; Deleting name (s) from the address book; Associating telephone number (s), email address (es), physical address (es) or other information with a name; Associating an image with a name; Categorizing and distinguishing names; Providing telephone numbers or email addresses to initiate and / or facilitate communication by telephone 238, video conferencing module 239, email 240 or IM 241, and the like.

RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, and text In conjunction with input module 234, telephone module 238 enters a sequence of characters corresponding to the telephone number, accesses one or more telephone numbers within contact module 237, modifies the entered telephone number, and It can be used to dial a phone number, to have a conversation, and to disconnect or disconnect when the conversation is complete. As mentioned above, wireless communication may use any of a plurality of communication standards, protocols, and techniques.

RF circuitry 208, audio circuitry 210, speaker 211, microphone 213, touch screen 212, display controller 256, light sensor 264, light sensor controller 258, contact / motion In conjunction with module 230, graphics module 232, text input module 234, contact module 237, and telephone module 238, video conferencing module 239 may be used by the user and one or more other users according to user instructions. Executable instructions for initiating, executing, and terminating a video conference between the participants.

Together with the RF circuitry 208, the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, the email client module 240 is instructed by the user. Executable instructions to respond to, create, send, receive, and manage email. In conjunction with the image management module 244, the email client module 240 makes it very easy to create and send emails with still or video images taken with the camera module 243.

The instant messaging module 241, along with the RF circuitry 208, the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, is instantaneous. Input a sequence of characters corresponding to the message, modify previously entered characters, for example, a short text message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messages Or use XMPP, SIMPLE or IMPS for Internet-based instant messages) to send respective instant messages, receive instant messages, and view received instant messages. In some embodiments, instant messages sent and / or received may be graphics, photos, audio files, video files, and / or as supported by MMS and / or Enhanced Messaging Service (EMS). Other attachments may be included. As used herein, "instant messaging" refers to telephone-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, messages sent using XMPP, SIMPLE or IMPS). S) both.

RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, map module 254 And, in conjunction with the music player module, exercise support module 242 devises exercises (eg, with time, distance, and / or calorie consumption goals); Communicate with athletic sensors (sports devices); Receive exercise sensor data; Calibrate sensors used to monitor exercise; Select and play music for a workout; Commands that are executable to display, store, and transmit athletic data.

With touch screen 212, display controller 256, optical sensor (s) 264, optical sensor controller 258, contact / motion module 230, graphics module 232, and image management module 244. Camera module 243 captures still images or videos (including video streams) and stores them in memory 202, modifies the characteristics of still images or videos, or captures still images or videos from memory 202. Executable instructions for deleting the video.

Together with the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, the text input module 234, and the camera module 243, the image management module 244 is stationary and And / or executable instructions for arranging, modifying (eg, editing) video images, or otherwise manipulating, labeling, deleting, presenting and storing (eg, in a digital slide show or album).

The browser module 247, along with the RF circuitry 208, the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, is a web page. Executable instructions for browsing the Internet in accordance with user instructions, including searching for, linking to, receiving, and displaying attachments and other files linked to web pages as well as their or portions thereof Include them.

RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, email client module 240, and browser module ( 247, together with calendar module 248, generates, displays, modifies, and stores calendars and data associated with calendars (eg, calendar entries, to-do lists, etc.) in accordance with user instructions. Includes executable instructions to execute.

Widget modules, along with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234 and browser module 247. 249 may be downloaded and used by a user (eg, weather widget 249-1, stock widget 249-2, calculator widget 249-3, alarm clock widget 249-4 and dictionary widget) (249-5)), or mini-applications that may be generated by the user (eg, user generated widget 249-6). In some embodiments, the widget includes a Hypertext Markup Language (HTML) file, a Cascading Style Sheets (CSS) file, and a JavaScript file. In some embodiments, the widget includes an Extensible Markup Language (XML) file and a JavaScript file (eg, Yahoo! widgets).

Widget generator module, with RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234 and browser module 247 250 may be used by the user to create widgets (eg, changing a user specific portion of the web page to a widget).

In addition to the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, the search module 251 may be based on one or more search criteria in accordance with user instructions. Executable instructions to retrieve text, music, sound, image, video, and / or other files in memory 202 that match (eg, one or more user specific search terms).

With touch screen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208 and browser module 247 , Video and music player module 252 may execute executable instructions that enable a user to download and play recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and videos (eg, And executable instructions to display, display, or otherwise play back on the touch screen 212 or on an externally connected display through an external port 224. In some embodiments, device 200 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

In addition to the touch screen 212, the display controller 256, the contact / motion module 230, the graphics module 232, and the text input module 234, the memo module 253 is capable of taking notes, to do according to user instructions. Executable instructions for creating and managing to-do lists and the like.

RF circuitry 208, touch screen 212, display controller 256, contact / motion module 230, graphics module 232, text input module 234, GPS module 235, and browser module 247 In conjunction with user instructions, map module 254 may be used to map maps and data associated with maps (eg, driving directions; data relating to shops and other points of interest at or near a particular location; and other Location-based data) can be used to receive, display, modify, and store.

Touch screen 212, display controller 256, contact / motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208, text input module 234, In conjunction with email client module 240 and browser module 247, online video module 255 allows a user to access, browse and browse online videos in one or more file formats such as H.264 (eg, streaming and Receive and / or play (eg, on a touch screen or on an externally connected display via external port 224), send an email with a link to a specific online video, and otherwise manage Contains commands that allow you to do so. In some embodiments, instant messaging module 241 rather than email client module 240 is used to send a link to a particular online video. Further discussion of online video applications is described in U.S. Provisional Patent Application 60 / 936,562, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," filed June 20, 2007 and 31 December 2007. See, for example, US Patent Application No. 11 / 968,067, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," which is hereby incorporated by reference, the contents of which are hereby incorporated by reference in their entirety. .

Each of the above identified modules and applications are executable for performing one or more of the functions described above and the methods described herein (eg, computer-implemented methods and other information processing methods described herein). Corresponds to a set of instructions. These modules (eg, sets of instructions) need not be implemented as separate software programs, procedures or modules, so that various subsets of these modules may be combined or otherwise rearranged in various embodiments. . For example, the video player module may be combined with the music player module to become a single module (eg, the video and music player module 252 of FIG. 2A). In some embodiments, memory 202 may store a subset of the identified modules and data structures. In addition, memory 202 may store additional modules and data structures not described above.

In some embodiments, device 200 is a device in which the operation of a predefined set of functions on the device is performed exclusively through the touch screen and / or touchpad. By using the touch screen and / or touchpad as the primary input control device for operation of device 200, the number of physical input control devices (such as push buttons, dials, etc.) on device 200 can be reduced.

The predefined set of functions performed solely via the touch screen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 200 from any user interface displayed on the device 200 to the main, home, or root menu. In these embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of the touchpad.

2B is a block diagram illustrating example components for event processing in accordance with some embodiments. In some embodiments, memory (202 of FIG. 2A or 470 of FIG. 4) includes event classifier 270 (eg, in operating system 226) and respective application 236-1 (eg, the application described above). 237-251, 255, 480-490.

The event classifier 270 receives the event information, and determines an application 236-1 to deliver the event information, and an application view 291 of the application 236-1. The event classifier 270 includes an event monitor 271 and an event dispatcher module 274. In some embodiments, application 236-1 includes an application internal state 292 that represents the current application view (s) displayed on touch sensitive display 212 when the application is active or running. In some embodiments, device / global internal state 257 is used by event classifier 270 to determine which application (s) are currently active, and application internal state 292 is event classifier 270. Is used to determine application views 291 to convey event information.

In some embodiments, application internal state 292 is a user representing resume information to be used when application 236-1 resumes execution, information being displayed by or ready to be displayed by application 236-1. One of the interface status information, a status queue for allowing the user to return to the previous state or view of the application 236-1, and a redo / undo queue of previous actions taken by the user. Include additional information as above.

Event monitor 271 receives event information from peripherals interface 218. The event information includes information about the subevent (eg, user touch on touch sensitive display 212 as part of a multi-touch gesture). Peripheral interface 218 is received from I / O subsystem 206 or a sensor, such as proximity sensor 266, accelerometer (s) 268, and / or microphone 213 (via audio circuitry 210). Send the information. The information that peripheral interface 218 receives from I / O subsystem 206 includes information from touch-sensitive display 212 or touch-sensitive surface.

In some embodiments, event monitor 271 sends requests to peripherals interface 218 at predetermined intervals. In response, peripherals interface 218 transmits event information. In other embodiments, peripheral interface 218 transmits event information only when there is a significant event (eg, receiving input above a predetermined noise threshold and / or above a predetermined duration).

In some embodiments, event classifier 270 also includes a hit view determination module 272 and / or an active event recognizer determination module 273.

The hit view determination module 272 provides software procedures for determining where a subevent occurred within one or more views when the touch-sensitive display 212 displays more than one view. Views consist of controls and other elements that a user can see on the display.

Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of each application) where a touch is detected may correspond to the program levels in the program or view hierarchy of the application. For example, the lowest level view in which a touch is detected may be referred to as a hit view, and the set of events recognized as appropriate inputs may be determined based at least in part on the hit view of the initial touch initiating a touch based gesture.

The hit view determination module 272 receives information related to subevents of a touch based gesture. If the application has multiple views organized in a hierarchy, the hit view determination module 272 identifies the hit view as the lowest view in the hierarchy that should handle the subevent. In most situations, the hit view is the lowest level view in which the initiating subevent (ie, the first subevent in the sequence of subevents forming the event or potential event) occurs. Once a hit view is identified by hit view determination module 272, the hit view typically receives all subevents related to the same touch or input source that cause it to be identified as a hit view.

The active event recognizer determination module 273 determines which view or views within the view hierarchy should receive a particular sequence of subevents. In some embodiments, active event recognizer determination module 273 determines that only the hit view should receive a particular sequence of subevents. In other embodiments, active event recognizer determination module 273 determines that all views including the physical location of the subevent are actively involved views, such that all actively involved views are specific to the subevents. Determine that you need to receive the sequence. In other embodiments, even if touch subevents are entirely confined to an area associated with one particular view, the parent views in the hierarchy will still remain as actively participating views.

Event dispatcher module 274 delivers the event information to an event recognizer (eg, event recognizer 280). In embodiments that include an active event recognizer determination module 273, the event dispatcher module 274 delivers the event information to the event recognizer determined by the active event recognizer determination module 273. In some embodiments, event dispatcher module 274 stores event information in an event queue, which is retrieved by respective event receiver 282.

In some embodiments, operating system 226 includes an event classifier 270. Alternatively, application 236-1 includes event classifier 270. In still other embodiments, the event classifier 270 is a standalone module or part of another module stored in the memory 202 such as the contact / motion module 230.

In some embodiments, application 236-1 includes a plurality of event handlers 290 and one or more application views 291, each of which is a touch event occurring within a respective view of an application's user interface. Instructions for processing them. Each application view 291 of application 236-1 includes one or more event recognizers 280. Typically, each application view 291 includes a plurality of event recognizers 280. In other embodiments, one or more of the event recognizers 280 may be a separate module, such as a user interface kit (not shown) or a higher level object to which the application 236-1 inherits methods and other attributes. It is part. In some embodiments, each event handler 290 is one of the data updater 276, the object updater 277, the GUI updater 278, and / or the event data 279 received from the event classifier 270. It includes the above. The event handler 290 may use the data updater 276, the object updater 277, or the GUI updater 278 to call or update the application internal state 292. Alternatively, one or more of the application views 291 include one or more respective event handlers 290. Further, in some embodiments, one or more of data updater 276, object updater 277, and GUI updater 278 are included within respective application view 291.

Each event recognizer 280 receives event information (eg, event data 279) from the event classifier 270 and identifies the event from the event information. Event recognizer 280 includes an event receiver 282 and an event comparator 284. In some embodiments, event recognizer 280 also includes at least a subset of metadata 283 and event delivery instructions 288 (which may include subevent delivery instructions).

The event receiver 282 receives event information from the event classifier 270. The event information includes information regarding sub-events, for example touch or touch movement. Depending on the subevent, the event information also includes additional information such as the location of the subevent. When the subevent relates to the motion of the touch, the event information may also include the speed and direction of the subevent. In some embodiments, the event includes rotation of the device from one orientation to another (eg, from vertical orientation to landscape orientation or vice versa), and the event information is the current orientation of the device (also referred to as the device attitude). It contains the corresponding information.

Event comparator 284 compares the event information with predefined event or subevent definitions, and based on the comparison, determines the event or subevent or determines or updates the status of the event or subevent. In some embodiments, event comparator 284 includes event definitions 286. Event definitions 286 include definitions of events (eg, predefined sequences of subevents), eg, event 1 287-1, event 2 287-2, and the like. In some embodiments, subevents within event 287 include, for example, touch start, touch end, touch move, touch cancel, and multiple touch. In one example, the definition for event 1 287-1 is a double tap on the displayed object. The double tap is, for example, a first touch on the displayed object during the predetermined phase (start of touch), a first liftoff during the predetermined phase (end of touch), the displayed object during the predetermined phase. A second touch on the touch (start of touch), and a second liftoff (touch end) during the predetermined phase. In another example, the definition for event 2 287-2 is dragging on the displayed object. Dragging includes, for example, a touch (or touch) on the displayed object during a predetermined phase, the movement of the touch across the touch-sensitive display 212, and the liftoff (touch end) of the touch. In some embodiments, the event also includes information about one or more associated event handlers 290.

In some embodiments, event definition 287 includes a definition of an event for a respective user interface object. In some embodiments, event comparator 284 performs a hit test to determine which user interface object is associated with the subevent. For example, in an application view in which three user interface objects are displayed on touch-sensitive display 212, if a touch is detected on touch-sensitive display 212, event comparator 284 may display three user interface objects. A hit test is performed to determine which of these are associated with the touch (subevent). When each displayed object is associated with its own event handler 290, the event comparator uses the results of the hit test to determine which event handler 290 should be activated. For example, event comparator 284 selects an event handler associated with the object and subevent that triggers the hit test.

In some embodiments, the definition for each event 287 also includes delayed actions that delay delivery of event information until after determining whether the sequence of subevents corresponds to or does not correspond to the event type of the event recognizer. do.

If each event recognizer 280 determines that the series of subevents does not match any of the events in the event definitions 286, the respective event recognizer 280 is unable to event, event failure, or event termination. After entering the state, each event recognizer ignores subsequent subevents of the touch-based gesture. In this situation, other event recognizers, if any, remain active for the hit view to continue tracking and processing subevents of the ongoing touch-based gesture.

In some embodiments, each event recognizer 280 may configure configurable attributes, flags, and / or lists that indicate how the event delivery system should perform subevent delivery for actively participating event recognizers. It has metadata 283 having. In some embodiments, metadata 283 includes configurable attributes, flags, and / or lists that indicate how event recognizers can or will interact with each other. In some embodiments, metadata 283 includes configurable attributes, flags, and / or lists that indicate whether subevents are delivered at various levels in the view or program hierarchy.

In some embodiments, each event recognizer 280 activates an event handler 290 associated with the event when one or more specific subevents of the event are recognized. In some embodiments, each event recognizer 280 delivers event information associated with the event to event handler 290. Activating event handler 290 is separate from sending (and delaying sending) subevents to their hit views. In some embodiments, event recognizer 280 sends a flag associated with the recognized event, and event handler 290 associated with the flag catches the flag and performs a predefined process.

In some embodiments, event delivery instructions 288 include subevent delivery instructions that convey event information about a subevent without activating the event handler. Instead, subevent delivery instructions deliver event information to event handlers associated with a series of subevents or to actively participating views. Event handlers associated with a series of subevents or actively participating views receive event information and perform a predetermined process.

In some embodiments, data updater 276 generates and updates data used in application 236-1. For example, the data updater 276 updates the phone number used in the contact module 237 or stores the video file used in the video player module. In some embodiments, object updater 277 creates and updates objects used in application 236-1. For example, object updater 277 creates a new user interface object or updates the location of the user interface object. GUI updater 278 updates the GUI. For example, the GUI updater 278 prepares the display information and sends it to the graphics module 232 for display on the touch-sensitive display.

In some embodiments, event handler (s) 290 include or access data updater 276, object updater 277, and GUI updater 278. In some embodiments, data updater 276, object updater 277, and GUI updater 278 are included within a single module of respective application 236-1 or application view 291. In other embodiments, they are included in two or more software modules.

The foregoing discussion regarding event processing of user touches on touch-sensitive displays also applies to other forms of user inputs for operating multifunction devices 200 having input devices, all of which are disclosed on touch screens. It should be understood that it is not. For example, mouse movements and mouse button presses may optionally include pressing or holding a single or multiple keyboards; Contact movements such as tap, drag, scroll, and the like on the touchpad; Pen stylus inputs; Movement of the device; Verbal instructions; Detected eye movements; Biometric inputs; And / or any combination thereof optionally used as inputs corresponding to sub-events defining the event to be recognized.

3 illustrates a portable multifunction device 200 having a touch screen 212 in accordance with some embodiments. The touch screen optionally displays one or more graphics within the user interface (UI) 300. In this embodiment as well as in other embodiments described below, the user may, for example, have one or more fingers 302 (not shown to scale in the figure) or one or more stylus 303 (scale in the drawing). May be used to select one or more of the graphics by performing a gesture on the graphics. In some embodiments, the selection of one or more graphics occurs when the user stops contacting the one or more graphics. In some embodiments, the gesture may optionally include one or more tabs of a finger in contact with device 200, one or more swipes (from left to right, right to left, up and / or down) and / or (from right) Rolling left, right to left, up and / or down). In some implementations or situations, unintended contact with the graphic does not select the graphic. For example, when the gesture corresponding to the selection is tapping, the swipe gesture passing over the application icon, optionally, does not select the corresponding application.

Device 200 may also include one or more physical buttons, such as “home” or menu button 304. As discussed above, the menu button 304 may be used to navigate to any application 236 in the set of applications that may run on the device 200. Alternatively, in some embodiments, the menu button is implemented as a soft key in the GUI displayed on the touch screen 212.

In one embodiment, the device 200 includes a touch screen 212, a menu button 304, a push button 306 to power on / off the device and lock the device, volume control button (s) 308, Subscriber Identity Module (SIM) card slot 310, headset jack 312, and docking / charging external port 224. Push button 306 selectively turns on / off power at the device by pressing the button and keeping the button pressed for a predetermined time interval; It can be used to lock a device and / or unlock a device or initiate an unlocking process by pressing a button and releasing the button before a predefined time interval has elapsed. In an alternative embodiment, device 200 also receives verbal input for activation or deactivation of some functions via microphone 213. The device 200 also optionally generates tactile outputs for the user of the device 200 and / or one or more contact intensity sensors 265 to detect the intensity of the contacts on the touch screen 212. One or more tactile output generators 267 for the purpose of operation.

4 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, in accordance with some embodiments. The device 400 need not be portable. In some embodiments, device 400 may be a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as children's learning toy), game system, or control device (eg, home or industrial). Controller). Device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, memory 470, and one or more communication buses 420 for interconnecting these components. It includes. Communication buses 420 optionally include circuitry (sometimes called a chipset) that interconnects system components and controls communication between them. Device 400 includes an input / output (I / O) interface 430 that includes a display 440, which is typically a touch screen display. I / O interface 430 is also optionally a tactile output for generating tactile outputs on keyboard and / or mouse (or other pointing device) 450 and touchpad 455, device 400. Generator 457 (eg, similar to tactile output generator (s) 267 described above with reference to FIG. 2A), and sensors 459 (eg, contact intensity sensor (s) described above with reference to FIG. 2A Light, acceleration, proximity, touch-sensitive, and / or contact intensity sensors) similar to 265). Memory 470 includes fast random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; Optionally include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 470 optionally includes one or more storage devices located remotely from CPU (s) 410. In some embodiments, memory 470 is programs, modules, and data structures similar to programs, modules, and data structures stored in memory 202 of portable multifunction device 200 (FIG. 2A). Or a subset thereof. Also, memory 470 optionally stores additional programs, modules, and data structures that are not present in memory 202 of portable multifunction device 200. For example, the memory 470 of the device 400 may optionally include a drawing module 480, a presentation module 482, a word processing module 484, a website authoring module 486, and a disk authoring module 488. ) And / or spreadsheet module 490, while memory 202 of portable multifunction device 200 (FIG. 2A), optionally, does not store such modules.

Each of the above identified elements in FIG. 4 may be stored in one or more of the previously mentioned memory devices. Each of the above identified modules corresponds to a set of instructions for performing the function described above. The identified modules or programs (eg, sets of instructions) need not be implemented as separate software programs, procedures or modules, so that various subsets of these modules may be combined in various embodiments or Can be rearranged in other ways. In some embodiments, memory 470 may store a subset of the identified modules and data structures. In addition, memory 470 may store additional modules and data structures not described above.

We now look at embodiments of user interfaces that may be implemented, for example, on portable multifunction device 200.

5A illustrates an example user interface for a menu of applications on portable multifunction device 200, in accordance with some embodiments. Similar user interfaces may be implemented on device 400. In some embodiments, user interface 500 includes the following elements, or a subset or superset thereof:

셀룰러 및 Wi-Fi 신호들과 같은 무선 통신(들)에 대한 신호 강도 표시자(들)(502);

Signal strength indicator (s) 502 for wireless communication (s), such as cellular and Wi-Fi signals;

시간(504);

Time 504;

블루투스 표시자(505);

A Bluetooth indicator 505;

배터리 상태 표시자(506);

Battery status indicator 506;

다음과 같은, 빈번하게 사용되는 애플리케이션용 아이콘들을 갖는 트레이(508):

A tray 508 with icons for frequently used applications, such as:

부재 중 전화들 또는 음성메일 메시지들의 개수의 표시자(514)를 선택적으로 포함하는 "전화"라고 라벨링된 전화 모듈(238)에 대한 아이콘(516);

Icon 516 for telephony module 238 labeled “telephone” optionally including an indicator 514 of the number of missed calls or voicemail messages;

읽지 않은 이메일들의 개수를 나타내는 표시자(510)를 선택적으로 포함하는, "메일"이라고 라벨링된 이메일 클라이언트 모듈(240)에 대한 아이콘(518);

An icon 518 for the email client module 240 labeled “mail”, optionally including an indicator 510 indicating the number of unread emails;

"브라우저"라고 라벨링된, 브라우저 모듈(247)에 대한 아이콘(520); 및

An icon 520 for the browser module 247, labeled “Browser”; And

"아이팟"이라고 라벨링된, 아이팟(애플 인크의 등록상표) 모듈(252)로도 지칭되는 비디오 및 음악 플레이어 모듈(252)에 대한 아이콘(522); 및

An icon 522 for the video and music player module 252, also referred to as an iPod (trademark of Apple Inc.) module 252, labeled “iPod”; And

다음과 같은, 다른 애플리케이션용 아이콘들:

Icons for other applications, such as:

"메시지"라고 라벨링된, IM 모듈(241)에 대한 아이콘(524);

An icon 524 for the IM module 241, labeled “Message”;

"캘린더"라고 라벨링된, 캘린더 모듈(248)에 대한 아이콘(526);

An icon 526 for the calendar module 248, labeled "Calendar";

"사진"이라고 라벨링된, 이미지 관리 모듈(244)에 대한 아이콘(528);

An icon 528 for image management module 244, labeled “Photo”;

"카메라"라고 라벨링된, 카메라 모듈(243)에 대한 아이콘(530);

An icon 530 for camera module 243, labeled “camera”;

"온라인 비디오"라고 라벨링된, 온라인 비디오 모듈(255)에 대한 아이콘(532);

An icon 532 for the online video module 255, labeled "online video";

"주식"이라고 라벨링된, 주식 위젯(249-2)에 대한 아이콘(534);

An icon 534 for stock widget 249-2, labeled “Stock”;

"지도"라고 라벨링된, 지도 모듈(254)에 대한 아이콘(536);

An icon 536 for the map module 254, labeled “Map”;

"날씨"라고 라벨링된, 날씨 위젯(249-1)에 대한 아이콘(538);

An icon 538 for the weather widget 249-1, labeled “Weather”;

"시계"라고 라벨링된, 알람시계 위젯(249-4)에 대한 아이콘(540);

An icon 540 for the alarm clock widget 249-4, labeled “Clock”;

"운동 지원"이라고 라벨링된, 운동 지원 모듈(242)에 대한 아이콘(542);

Icon 542 for workout support module 242, labeled “Workout Support”;

"메모"라고 라벨링된, 메모 모듈(253)에 대한 아이콘(544); 및

An icon 544 for the memo module 253, labeled “Notes”; And

"설정"이라고 라벨링된, 설정 애플리케이션 또는 모듈에 대한 아이콘(546)으로서, 설정 애플리케이션 또는 모듈은 디바이스(200) 및 그것의 다양한 애플리케이션들(236)을 위한 설정에 대한 액세스를 제공한다.

As an icon 546 for a configuration application or module, labeled "configuration," the configuration application or module provides access to the configuration for device 200 and its various applications 236.

It should be noted that the icon labels shown in FIG. 5A are merely illustrative. For example, the icon 522 for the video and music player module 252 may optionally be labeled "music" or "music player." Other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label for a particular application icon is separate from the name of the application corresponding to the particular application icon.

5B illustrates a device (eg, device 400) having a touch-sensitive surface 551 (eg, tablet or touchpad 455, FIG. 4) separate from display 550 (eg, touch screen display 212). ), An exemplary user interface on FIG. 4). Device 400 may also optionally include one or more contact intensity sensors (eg, one or more of sensors 457) and / or a device for detecting the intensity of contacts on touch-sensitive surface 551. One or more tactile output generators 459 for generating tactile outputs for the user of 400.

Some examples that follow will be provided with reference to inputs on the touch screen display 212 (combined with the touch-sensitive surface and display), but in some embodiments, the device is separate from the display as shown in FIG. 5B. Detect inputs on a personal touch-sensitive surface. In some embodiments, the touch-sensitive surface (eg, 551 of FIG. 5B) has a major axis (eg, 552 of FIG. 5B) corresponding to the major axis (eg, 553 of FIG. 5B) on the display (eg, 550). According to these embodiments, the device may interact with the touch-sensitive surface 551 at locations corresponding to respective locations on the display (eg, in FIG. 5B, 560 corresponds to 568 and 562 corresponds to 570). Detects contacts (eg, 560, 562 of FIG. 5B). In this way, user inputs (eg, contacts 560, 562 and their movements) detected by the device on a touch-sensitive surface (eg, 551 of FIG. 5B) are such that the touch-sensitive surface is separate from the display. When used by the device to manipulate the user interface on the display of the multifunction device (eg, 550 of FIG. 5B). It should be understood that similar methods are optionally used for the other user interfaces described herein.

In addition, the following examples are given primarily with reference to finger inputs (eg, finger contacts, finger tapping gestures, finger swipe gestures), while in some embodiments, one or more of the finger inputs are input differently. It should be understood that it is replaced by input from the device (eg, mouse based input or stylus input). For example, the swipe gesture is optionally replaced with a mouse click (eg, instead of a contact) followed by a movement of the cursor along the path of the swipe (eg, instead of the movement of the contact). As another example, the tap gesture is optionally replaced with a mouse click while the cursor is positioned over the location of the tap gesture (eg, instead of stopping detection of the contact and detecting subsequent contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are selectively used simultaneously, or that mouse and finger contacts are selectively used simultaneously.

6A illustrates an example personal electronic device 600. Device 600 includes a body 602. In some embodiments, device 600 may include some or all of the features described with respect to devices 200, 400 (eg, FIGS. 2A-4B). In some embodiments, device 600 has a touch-sensitive display screen 604 (hereinafter touch screen 604). Alternatively or in addition to touch screen 604, device 600 has a display and a touch-sensitive surface. As in devices 200 and 400, in some embodiments, touch screen 604 (or touch-sensitive surface) is one or more intensity sensors for detecting the intensity of the contacts (eg, touches) being applied. You can have One or more intensity sensors of touch screen 604 (or touch-sensitive surface) may provide output data indicative of the intensity of the touches. The user interface of device 600 may respond to touches based on the intensity of the touches, meaning that touches of different intensities may invoke different user interface operations on device 600.

Techniques for detecting and processing touch intensity can be found, for example, in related applications: filed May 8, 2013, and entitled "Device, Method, and Graphical User Interface for Displaying User". Interface Objects Corresponding to an Application, filed in International Patent Application PCT / US2013 / 040061, and November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output”. International Patent Application PCT / US2013 / 069483, "Relationships", the contents of each of which are hereby incorporated by reference in their entirety.

In some embodiments, device 600 has one or more input mechanisms 606, 608. The input mechanisms 606, 608, if included, may be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 600 has one or more attachment mechanisms. These attachment mechanisms (if included) are such that the device 600 may be, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watches, chains, pants, belts, shoes, wallets. It can be attached to a backpack, etc. These attachment mechanisms can cause the device 600 to be worn by a user.

6B illustrates an example personal electronic device 600. In some embodiments, device 600 may include some or all of the components described with respect to FIGS. 2A, 2B, and 4. Device 600 has a bus 612 that operatively couples I / O section 614 with one or more computer processors 616 and memory 618. I / O section 614 may be connected to display 604, which may have touch sensitive component 622, and, optionally, touch intensity sensitive component 624. In addition, I / O section 614 provides communication unit 630 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication techniques. ) Can be connected. Device 600 may include input mechanisms 606 and / or 608. The input mechanism 606 may be, for example, a rotatable input device or a pushable and rotatable input device. The input mechanism 608 may be a button in some examples.

The input mechanism 608 may be a microphone, in some examples. The personal electronic device 600 may include a GPS sensor 632, an accelerometer 634, a direction sensor 640 (eg, a compass), a gyroscope 636, a motion sensor 638, and / or combinations thereof. And various sensors, all of which may be operatively connected to the I / O section 614.

The memory 618 of the personal electronic device 600 may, for example, be executed by one or more computer processors 616 to cause the computer processors to include the processes described below (eg, 10 and 11 may include one or more non-transitory computer readable storage media for storing computer executable instructions that may cause the processes 1000, 1100 of FIGS. 10 and 11 to be performed. Personal electronic device 600 is not limited to the components and configurations of FIG. 6B, and may include other or additional components in a number of configurations.

As used herein, the term “affordance” refers to a user interactive graphic that can be displayed on the display screen of devices 200, 400, and / or 600 (FIGS. 2, 4, and 6). Refers to a user-interactive graphical user interface object. For example, images (eg, icons), buttons, and text (eg, links) may each constitute affordances.

As used herein, the term "focus selector" refers to an input element that represents the current portion of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the touch-sensitive surface (eg, FIG. 4) while the cursor is over a particular user interface element (eg, a button, window, slider or other user interface element). When an input (eg, a press input) is detected on the touchpad 455 of FIG. 5 or the touch-sensitive surface 551 of FIG. 5B, the cursor is “focus selector” such that a particular user interface element is adjusted according to the detected input. Function as. Some implementations include a touch screen display (eg, touch-sensitive display system 212 of FIG. 2A or touch screen 212 of FIG. 5A) that enables direct interaction with user interface elements on the touch screen display. In particular, when an input (eg, a press input by contact) is detected on a touch screen display at a location of a particular user interface element (eg, a button, window, slider or other user interface element), the particular user interface element is detected. The detected contact on the touch screen functions as a "focus selector" to be adjusted according to the input input. In some implementations, focus can be applied to one of the user interfaces without moving the corresponding cursor or moving the contact on the touch screen display (eg, by using a tab key or an arrow key to move the focus from one button to another). From a zone to another zone of the user interface, in such implementations, the focus selector moves in accordance with the movement of focus between different zones of the user interface. Regardless of the particular form of the focus selector, the focus selector is generally used to convey the user's intended interaction with the user interface (eg, by presenting to the device an element of the user interface that the user wishes to interact with). Is a user interface element (or contact on a touch screen display) controlled by the. For example, while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of the focus selector (e.g., cursor, contact, or selection box) on the respective button is displayed on the display of the device. Unlike the other user interface elements shown, this will indicate that the user is trying to activate their own button.

As used in the specification and claims, the term "characteristic intensity" of a contact refers to the characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity may optionally be determined for a predefined number of intensity samples, or for a predefined event (eg, after detecting a contact, before detecting a liftoff of the contact, before detecting the start of the movement of the contact). Or later, before detecting the end of the contact, before or after detecting an increase in the intensity of the contact, and / or before or after detecting a decrease in the intensity of the contact) a predetermined period of time (eg, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) based on the set of intensity samples collected. The characteristic intensity of a contact may optionally include a maximum value of the intensities of the contact, a mean value of the intensities of the contact, an average value of the intensities of the contact, and a top 10 percentile value of the intensities of the contact. ), Half of the maximum value of the intensities of the contact, 90 percent of the maximum value of the intensity of the contact, and the like. In some embodiments, the duration of the contact is used to determine the characteristic strength (eg, when the characteristic strength is an average of the strength of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether the operation was performed by the user. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, as a result of the contact having a characteristic strength that does not exceed the first threshold, the first operation is performed and as a result of the contact having a characteristic strength that exceeds the first intensity threshold but does not exceed the second intensity threshold, the second An operation is performed, and a third operation is performed as a result of the contact having a characteristic strength above the second threshold. In some embodiments, whether to perform one or more operations rather than being used to determine whether to perform a first action or a second action (eg, to perform each action or to perform its own action). The comparison between the characteristic strength and one or more thresholds is used to determine whether to ignore).

In some embodiments, a portion of the gesture is identified to determine the characteristic strength. For example, the touch-sensitive surface can receive a continuous swipe contact that moves from the starting position to the ending position, at which point the strength of the contact increases. In this example, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact (eg, only on the portion of the swipe contact at the end position), not the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensities of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one of an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. It includes one or more. In some situations, these smoothing algorithms eliminate narrow spikes or dips in the strengths of the swipe contact to determine the characteristic strength.

The intensity of the contact on the touch-sensitive surface can be characterized for one or more intensity thresholds, such as contact detection intensity thresholds, light press intensity thresholds, deep press intensity thresholds, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device will perform operations typically associated with clicking a button or track pad of a physical mouse. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device will perform operations different from those typically associated with clicking a button or trackpad of a physical mouse. In some embodiments, when the contact is detected with a characteristic strength below the light click intensity threshold (eg, and above the nominal touch detection intensity threshold (contact no longer detected)), the device is pressed lightly. The focus selector will move in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the threshold or the deep press intensity threshold. In general, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface figures.

The increase in the characteristic intensity of the contact from the intensity below the light click intensity threshold to the intensity between the light click intensity threshold and the deep click intensity threshold is sometimes referred to as a "light click" input. An increase in the characteristic intensity of a contact from an intensity below the deep click intensity threshold to an intensity above the deep click intensity threshold is sometimes referred to as a "deep click" input. An increase in the characteristic intensity of a contact from an intensity below the touch detection intensity threshold to an intensity between the touch detection intensity threshold and the light press intensity threshold is sometimes referred to as detection of a contact on the touch surface. The reduction in the characteristic intensity of a contact from an intensity above the touch detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detection of a liftoff of the contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

In some embodiments described herein, one or more operations detect a respective press input performed in response to detecting a gesture including a respective press input or performed by a respective contact (or a plurality of contacts). And wherein each press input is detected based at least in part on detecting an increase in intensity of a contact (or a plurality of contacts) above the press input intensity threshold. In some embodiments, the respective action is performed in response to detecting an increase in the strength of the respective contact above the click input intensity threshold (eg, a “down stroke” of the respective click input). In some embodiments, the press input includes an increase in the strength of the respective contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the press input intensity threshold, with each action being a self below the press input threshold. Is performed in response to detecting a subsequent decrease in the intensity of the contact of (e.g., an " up stroke " of the respective press input).

In some embodiments, the device uses intensity hysteresis to avoid accidental inputs, sometimes referred to as “jitter,” where the device has a hysteresis intensity threshold having a predefined relationship to the click input intensity threshold ( For example, the hysteresis intensity threshold is a unit of X intensity lower than the click input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or any suitable ratio of the click input intensity threshold). As such, in some embodiments, the press input includes an increase in the strength of the respective contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, respectively. The operation of is performed in response to detecting a subsequent decrease in the strength of each contact below the hysteresis intensity threshold (eg, an "up stroke" of the respective push input). Similarly, in some embodiments, the press input causes the device to increase the contact strength from an intensity below the hysteresis intensity threshold to an intensity above the press input threshold, and optionally, subsequent contact intensity to an intensity below the hysteresis intensity. Detected only when a decrease is detected, each action is performed in response to detecting a press input (eg, increase in contact strength or decrease in contact intensity depending on the environment).

For convenience of description, the description of the operations performed in response to a press input associated with the press-input intensity threshold or in response to a gesture comprising a press input, optionally, increases the intensity of the contact above the press-input intensity threshold, hysteresis. An increase in the intensity of the contact from an intensity below the intensity threshold to an intensity above the press-input intensity threshold, a decrease in the intensity of the contact below the press-input intensity threshold, and / or below a hysteresis intensity threshold corresponding to the press-input intensity threshold. Is triggered in response to detecting any one of the decreases in the intensity of the contact. In addition, in the examples in which the operation is described as being performed in response to detecting a decrease in the intensity of the contact below the click input intensity threshold, the action is optionally below the hysteresis intensity threshold that corresponds to and is lower than the press input intensity threshold. Is performed in response to detecting a decrease in the intensity of the contact.

3. Digital Assistant System

7A shows a block diagram of a digital assistant system 700, in accordance with various examples. In some examples, digital assistant system 700 may be implemented on a standalone computer system. In some examples, digital assistant system 700 may be distributed across multiple computers. In some examples, some of the modules and functions of the digital assistant may be divided into a server portion and a client portion, where the client portion is one or more user devices (eg, devices 104, 122, 200, 400 or 600). And, for example, communicate with a server portion (eg, server system 108) via one or more networks as shown in FIG. 1. In some examples, digital assistant system 700 may be an implementation of server system 108 (and / or DA server 106) shown in FIG. 1. Digital assistant system 700 is merely one example of a digital assistant system, and digital assistant system 700 may have more or fewer components than shown, or may combine two or more components, or Or it may have components of different configurations or arrangements. The various components shown in FIG. 7A may be implemented in hardware, software instructions for execution by one or more processors, firmware, or a combination thereof, including one or more signal processing and / or application specific integrated circuits.

The digital assistant system 700 can include a memory 702, one or more processors 704, an input / output (I / O interface 706), and a network communication interface 708. It may communicate with each other via a bus or signal line 710.

In some examples, memory 702 may be a non-transitory computer readable medium, such as fast random access memory and / or nonvolatile computer readable storage medium (eg, one or more magnetic disk storage devices, flash memory devices, or other nonvolatile solids). State memory device).

In some examples, I / O interface 706 connects input / output devices 716 of digital assistant system 700, such as displays, keyboards, touch screens, and microphones, to user interface module 722. Can be combined. I / O interface 706, in conjunction with user interface module 722, receives user inputs (eg, voice inputs, keyboard inputs, touch inputs, etc.) and processes them accordingly. In some examples, for example when the digital assistant is implemented on a standalone user device, the digital assistant system 700 may include the components described for the devices 200, 400, or 600 of FIGS. 2A, 4, 6A, and 6B, respectively. And any of the I / O communication interfaces. In some examples, digital assistant system 700 may represent a server portion of a digital assistant implementation and communicate with a user through a client-side portion residing on a user device (eg, devices 104, 200, 400, or 600). Can interact.

In some examples, network communication interface 708 may include wired communication port (s) 712 and / or wireless transmit and receive circuitry 714. The wired communication port (s) may receive and transmit communication signals via one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), FireWire, or the like. The wireless circuitry 714 can receive and transmit RF signals and / or optical signals with communication networks and other communication devices. The wireless communication may use any of a plurality of communication standards, protocols, and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. have. Network communication interface 708 is digital assistant system 700 and other utilizing networks such as the Internet, intranets, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs). It may enable communication between devices.

In some examples, memory 702 or computer readable storage media of memory 702 may include operating system 718, communication module 720, user interface module 722, one or more applications 724, and digital assistant. Programs, modules, instructions, and data structures may be stored, including all or a subset of module 726. In particular, the memory 702 or the computer readable storage medium of the memory 702 may store instructions for performing the processes 1000 and 1100 described below. One or more processors 704 may execute these programs, modules, and instructions and read / write from / to data structures.

Operating system 718 (e.g., an embedded operating system such as Darwin, RTXC, LINUX, UNIX, iOS, OS X, WINDOWS, or VxWorks) is a common system task (e.g., memory management, storage). Various software components and / or drivers for controlling and managing device control, power management, etc.), and may facilitate communications between various hardware, firmware, and software components.

The communication module 720 may facilitate communication between the digital assistant system 700 and other devices via the network communication interface 708. For example, the communication module 720 may communicate with the RF circuitry 208 of electronic devices, such as the devices 200, 400, 600 shown in FIGS. 2A, 4, 6A, and 6B, respectively. The communication module 720 may also include various components for processing data received by the wireless circuitry 714 and / or the wired communication port 712.

The user interface module 722 receives commands and / or inputs from the user (eg, from a keyboard, touch screen, pointing device, controller, and / or microphone) via the I / O interface 706, and displays on the display. You can create user interface objects. The user interface module 722 also prepares outputs (eg, speech, sound, animation, text, icons, vibrations, haptic feedback, lighting, etc.) and via the I / O interface 706 (eg, display). , Audio channels, speakers, touchpads, and the like).

Applications 724 may include programs and / or modules configured to be executed by one or more processors 704. For example, when the digital assistant system is implemented on a standalone user device, the applications 724 can include user applications such as games, calendar applications, navigation applications, or email applications. When the digital assistant system 700 is implemented on a server, the applications 724 may include, for example, resource management applications, diagnostic applications, or scheduling applications.

The memory 702 may also store the digital assistant module 726 (or the server portion of the digital assistant). In some examples, digital assistant module 726 may include the following submodule, or a subset or superset thereof: input / output processing module 728, speech-to-text (STT) ) Processing module 730, natural language processing module 732, conversation flow processing module 734, task flow processing module 736, service processing module 738, and speech synthesis module 740. Each of these modules may have access to one or more of the following systems or data and models, or a subset or superset thereof, of digital assistant module 726: ontology 760, lexical index 744, user Data 748, task flow models 754, service models 756, and ASR systems.

In some examples, using the models implemented in processing modules, data, and digital assistant module 726, the digital assistant can perform at least some of the following: converting speech input into text; Identifying the user's intention expressed in natural language input received from the user; Actively drawing and obtaining the information needed to fully infer a user's intentions (eg, by disambiguating words, games, intentions, etc.); Determining a task flow for performing the inferred intent; And executing a task flow to perform the inferred intent.

In some examples, as shown in FIG. 7B, the I / O processing module 728 may interact with the user via the I / O devices 716 of FIG. 7A or may connect the network communication interface 708 of FIG. 7A. Interact with a user device (eg, devices 104, 200, 400, or 600) to obtain user input (eg, speech input) and provide responses (eg, as speech outputs) to the user input. Can be. The I / O processing module 728 may optionally obtain contextual information associated with the user input from the user device, either with or immediately after receipt of the user input. The contextual information may include user-specific data, vocabulary, and / or preferences associated with user input. In some examples, the contextual information also includes information related to the software and hardware status of the user device at the time the user request is received and / or the user's surrounding environment at the time the user request is received. In some examples, I / O processing module 728 may also send subsequent questions to the user regarding the user request, and receive answers therefrom. If a user request is received by I / O processing module 728 and the user request can include speech input, I / O processing module 728 sends the speech input for STT processing module 730 for speech-to-text conversion. (Or speech recognizer).

The STT processing module 730 may include one or more ASR systems. One or more ASR systems may process speech input received through I / O processing module 728 to generate a recognition result. Each ASR system may include a front-end speech pre-processor. The front-end speech preprocessor can extract representative features from the speech inputs. For example, the front-end speech preprocessor may Fourier transform a speech input to extract spectral features that characterize the speech input as a representative multi-dimensional vector sequence. In addition, each ASR system may include one or more speech recognition models (eg, acoustic models and / or language models), and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian-Mixture Models, Deep Neural Network Models, n-gram language models, and other statistical models. . Examples of speech recognition engines may include dynamic time distortion based engines and weighted finite state converter (WFST) based engines. One or more speech recognition models and one or more speech recognition engines may be used to determine intermediate recognition results (eg, phonemes, phoneme strings, and subwords), and ultimately text recognition results (eg, words, word strings, or tokens). Can be used to process the extracted representative features of the front-end speech preprocessor. In some examples, speech input may be processed at least in part by a third party service or on a user's device (eg, device 104, 200, 400, or 600) to generate a recognition result. If the STT processing module 730 generates a recognition result that includes a text string (eg, words, or sequence of words, or sequence of tokens), the recognition result is a natural language processing module 732 for inference of intent. Can be delivered.

More details about speech-text processing are described in US patent application Ser. No. 13 / 236,942, filed September 20, 2011, entitled "Consolidating Speech Recognition Results." The disclosure is incorporated herein by reference.

/ 및 /

/의 후보 발음과 연관된 "tomato"라는 단어를 포함할 수 있다. 또한, 어휘 단어는 사용자로부터의 이전 스피치 입력에 기초한 맞춤형 후보 발음과 연관될 수 있다. 이러한 맞춤형 후보 발음은 STT 처리 모듈(730)에 저장될 수 있고, 디바이스 상의 사용자의 프로필을 통해 특정 사용자와 연관될 수 있다. 일부 예들에서, 단어에 대한 후보 발음은 단어의 철자 및 하나 이상의 언어 및/또는 음성 규칙에 기초하여 결정될 수 있다. 일부 예들에서, 후보 발음은 예를 들어, 알려진 정준 발음(canonical pronunciations)에 기초하여 수동으로 생성될 수 있다.In some examples, the STT processing module 730 may include and / or access a vocabulary of words recognizable through the phonetic symbol conversion module 731. Each lexical word may be associated with one or more candidate pronunciations of the word represented by a speech recognition phonetic symbol. In particular, the vocabulary of the recognizable word may include a word associated with a plurality of candidate pronunciations. For example, the vocabulary is /

/ And /

May include the word "tomato" associated with a candidate pronunciation of /. In addition, lexical words may be associated with a tailored candidate pronunciation based on previous speech input from the user. Such customized candidate pronunciations may be stored in the STT processing module 730 and may be associated with a particular user through a profile of the user on the device. In some examples, the candidate pronunciation for a word may be determined based on the spelling of the word and one or more language and / or phonetic rules. In some examples, the candidate pronunciation may be generated manually based on known canonical pronunciations, for example.

/는 /

/보다 높은 순위가 될 수 있는데, 그 이유는 이전 발음이 더 일반적으로 사용되는 발음이기 때문이다(예를 들어, 모든 사용자 중에서, 특정 지리적 지역에 있는 사용자의 경우, 또는 임의의 다른 적절한 사용자들의 서브셋의 경우). 일부 예들에서, 후보 발음은 후보 발음이 사용자와 연관된 맞춤형 후보 발음인지 여부에 기초하여 순위가 매겨질 수 있다. 예를 들어, 맞춤형 후보 발음은 정준 후보 발음보다 높은 순위가 매겨질 수 있다. 이는 정준 발음에서 벗어나는 고유한 발음을 가진 고유 명사를 인식하는 데 유용할 수 있다. 일부 예들에서, 후보 발음은 지리적인 기원, 국적, 또는 민족성과 같은 하나 이상의 스피치 특성과 연관될 수 있다. 예를 들어, 후보 발음 /

/는 미국과 연관될 수 있는 반면, 후보 발음 /

/는 영국과 연관될 수 있다. 또한, 후보 발음의 순위는 디바이스 상의 사용자 프로필에 저장된 사용자의 하나 이상의 특성(예를 들어, 지리적 기원, 국적, 민족성 등)에 기초할 수 있다. 예를 들어, 사용자가 미국과 연관되어 있다는 사용자의 프로필로부터 결정될 수 있다. 미국과 연관되어 있는 사용자에 기초하여, 후보 발음 /

/(미국과 연관됨)는 후보 발음 /

/(영국과 연관됨)보다 높은 순위가 매겨질 수 있다. 일부 예들에서, 순위가 매겨진 후보 발음들 중 하나는 예측된 발음(예를 들어, 가장 가능성이 있는 발음)으로서 선택될 수 있다.In some examples, the candidate pronunciation may be ranked based on the commonality of the candidate pronunciations. For example, candidate pronunciation /

/ Is /

This can be higher than / because the previous pronunciation is the more commonly used pronunciation (for example, among all users, for users in a particular geographic region, or for any other appropriate subset of users). In the case of). In some examples, the candidate pronunciation may be ranked based on whether the candidate pronunciation is a custom candidate pronunciation associated with the user. For example, the customized candidate pronunciation may be ranked higher than the canonical candidate pronunciation. This may be useful for recognizing proper nouns with their own pronunciations that deviate from the canonical pronunciation. In some examples, candidate pronunciation may be associated with one or more speech characteristics, such as geographical origin, nationality, or ethnicity. For example, candidate pronunciation /

/ May be associated with the United States, while candidate pronunciation /

/ Can be associated with the United Kingdom. In addition, the ranking of candidate pronunciations may be based on one or more characteristics of the user (eg, geographic origin, nationality, ethnicity, etc.) stored in the user profile on the device. For example, it may be determined from the user's profile that the user is associated with the United States. Based on the user associated with the United States, candidate pronunciation /

/ (Associated with the United States) is a candidate pronunciation /

It can be ranked higher than / (associated with the UK). In some examples, one of the ranked candidate pronunciations may be selected as the predicted pronunciation (eg, the most likely pronunciation).

/를 먼저 식별할 수 있는 경우, 그것은 이어서 어휘 인덱스(744)에 기초하여 이 시퀀스가 단어 "tomato"에 대응한다고 결정할 수 있다.When a speech input is received, the STT processing module 730 may be used to determine the phonemes corresponding to the speech input (eg, using an acoustic model) and then (eg, using a language model). Try to determine a word that matches the phoneme. For example, STT processing module 730 may be a sequence of phonemes corresponding to a portion of speech input.

If / can be identified first, it can then determine that the sequence corresponds to the word "tomato" based on lexical index 744.

/가 단어 "tomato"에 대응한다고 결정할 수 있다.In some examples, STT processing module 730 may determine the word being spoken using an approximation matching technique. Thus, for example, the STT processing module 730 may determine the sequence of phonemes / even if the particular sequence of phonemes is not one of the candidate sequences of the phonemes for the word.

We can determine that / corresponds to the word "tomato".

The digital assistant's natural language processing module 732 ("natural language processor") takes a sequence of words or tokens ("token sequence") generated by the STT processing module 730 and converts the token sequence by the digital assistant. One may attempt to associate with one or more "actable intentions" that are recognized. An “actible intent” can represent a task that can be performed by a digital assistant and have an associated task flow implemented in task flow models 754. The associated task flow can be a series of programmed operations and steps that the digital assistant takes to perform a task. The category of capabilities of the digital assistant may depend on the number and type of task flows implemented and stored in the task flow models 754, or in other words, the number and type of “actible intentions” that the digital assistant recognizes. However, the efficiency of the digital assistant may also depend on the assistant's ability to infer the correct "actable intention (s)" from user requests expressed in natural language.

In some examples, in addition to the words or token sequence obtained from the STT processing module 730, the natural language processing module 732 may also include contextual information associated with a user request, for example, from the I / O processing module 728. Can be received. The natural language processing module 732 may optionally use contextual information to clarify, supplement, and / or further define information included in the token sequence received from the STT processing module 730. The contextual information may include, for example, user preferences, hardware and / or software states of the user device, sensor information collected before, during, or immediately after the user request, previous interactions between the digital assistant and the user (eg , Conversations). As described herein, contextual information may be dynamic and may vary with time, location, content of the conversation, and other factors.

In some examples, natural language processing may be based, for example, ontology 760. Ontology 760 can be a hierarchical structure that includes many nodes, each node having a " property " associated with one or more of " actable intentions " Represents any of the "attributes". As mentioned above, “actable intent” may represent a task that the digital assistant can perform, that is, a task that may be “actable” or affect. An "attribute" can represent a parameter that is associated with an actable intent or a sub aspect of another attribute. The linkage between the actable intent node and the attribute node in the ontology 760 may define how the parameters represented by the attribute node relate to the task represented by the actable intent node.

In some examples, ontology 760 may be comprised of actible intent nodes and attribute nodes. Within ontology 760, each actible intent node may be connected to one or more attribute nodes, either directly or through one or more intermediate attribute nodes. Similarly, each attribute node may be connected to one or more actible intent nodes either directly or through one or more intermediate attribute nodes. For example, as shown in FIG. 7C, ontology 760 may include a “restaurant reservation” node (ie, actable intent node). The attribute nodes "restaurant", "date / time" (for reservation), and "number of people" may each be directly linked to the actionable intent node (ie, "restaurant reservation" node).

In addition, the attribute nodes "cook", "price", "telephone number", and "location" may be subnodes of the attribute node "restaurant", and the "reserve restaurant" node (through the intermediate attribute node "restaurant" That is, each may be linked to an actionable intent node). For another example, as shown in FIG. 7C, ontology 760 may also include a “set reminder” node (ie, another actable intent node). The attribute nodes "Date / Time" (for setting reminders) and "Topic" (for reminders) may be linked to the "Set reminder" node, respectively. Since the attribute "date / time" may be related to both the task of making a restaurant reservation and the task of setting a reminder, the attribute node "date / time" is the "restaurant reservation" node and the "set reminder" in ontology 760. It can be linked to both nodes.

The actable intent node, along with its connected concept nodes, may be described as a "domain." In this discussion, each domain may be associated with an individual actable intent, and refers to a group of nodes (and relationships between them) associated with a particular actable intent. For example, the ontology 760 shown in FIG. 7C may include an example of a restaurant reservation domain 762 and an reminder domain 764 in the ontology 760. The restaurant reservation domain includes the actionable intent node "restaurant reservation", attribute nodes "restaurant", "date / time", and "number of people", and sub-attribution nodes "cooking", "price", "telephone number", and "Position". The reminder domain 764 may include the actible intent node "reminder setup", and the attribute nodes "topic" and "date / time". In some examples, ontology 760 may consist of many domains. Each domain may share one or more attribute nodes with one or more other domains. For example, a “date / time” attribute node may be associated with many different domains (eg, scheduling domain, travel reservation domain, movie ticket domain, etc.) in addition to restaurant reservation domain 762 and reminder domain 764. have.

Although FIG. 7C shows two example domains in ontology 760, other domains, for example, “find a movie”, “initiate a phone call”, “find directions”, “schedule meeting” "," Send a message "," provide an answer to a question "," read a list "," provide navigation commands "," provide commands for a task ", and the like. The "send message" domain may be associated with a "send message" actionable intent node and may further include attribute nodes such as "receiver (s)", "message type", and message body ". The attribute node "receiver" may be further defined by sub-attribute nodes such as, for example, "recipient name" and "message address".

In some examples, ontology 760 may include all domains (and hence actionable intentions) that the digital assistant can understand and operate on. In some examples, ontology 760 may be modified, such as by adding or removing entire domains or nodes, or by modifying relationships between nodes within ontology 760.

In some examples, nodes associated with multiple related actionable intentions may be clustered under “parent domain” within the ontology 760. For example, a “travel” parent domain may include a cluster of attribute nodes and actible intent nodes related to travel. The actionable intent nodes related to the trip may include "air booking", "hotel reservation", "car rental", "direction", "point of interest", and the like. Actionable intent nodes under the same parent domain (eg, “travels” parent domain) may have many attribute nodes in common. For example, the actionable intent nodes for "Airline Booking", "Hotel Booking", "Car Rental", "Get Directions", and "Find Point of Interest" are the attribute nodes "Start Location", "Destination", " One or more of "departure date / time", "arrival date / time", and "number of people".

In some examples, each node in ontology 760 may be associated with a set of words and / or phrases related to an attribute or actable intent represented by the node. The individual set of words and / or phrases associated with each node may be a so-called "vocabulary" associated with the node. Individual sets of words and / or phrases associated with each node may be stored in the lexical index 744 in relation to the attributes or actionable intentions represented by the node. For example, returning to FIG. 7B, the vocabulary associated with the node for the attribute of "Restaurant" is "food", "drink", "cooking", "hungry", "eat", "pizza", "fast food", Words such as "meal" and the like. For another example, the vocabulary associated with the node for the actionable intent of "initiate a phone call" is "call", "telephone", "dial", "ring", "call this number", "to Call "and the like. The lexical index 744 may optionally include words and phrases of different languages.

The natural language processing module 732 may receive a token sequence (eg, a text string) from the STT processing module 730 and determine which nodes are involved by the words in the token sequence. In some examples, if a word or phrase in the token sequence is found to be associated with one or more nodes in ontology 760 (via vocabulary index 744), the word or phrase “triggers” or “activates” these nodes. You can. Based on the amount and / or relative importance of the active nodes, the natural language processing module 732 may select one of the intentions that can act as a task the user intended to perform by the digital assistant. In some examples, the domain with the most "triggered" nodes may be selected. In some examples, the domain with the highest confidence value (eg, based on the relative importance of its various triggered nodes) may be selected. In some examples, the domain may be selected based on a combination of the number and importance of triggered nodes. In some examples, additional factors such as whether the digital assistant has previously correctly interpreted a similar request from the user are also considered in selecting the node.

User data 748 may contain user-specific information such as user-specific vocabulary, user preferences, user address, user's preferred language and second language, user's contact list, and other short or long term information for each user. It may include. In some examples, natural language processing module 732 may further define user intent by supplementing information included in user input using user-specific information. For example, for a user request "invite my friends to my birthday party", the natural language processing module 732 may determine who "friends" are and when and where the "birthday party" will be held. Instead of requiring the user to explicitly provide such information in his / her request, the user data 748 may be accessible.

Other details for searching ontologies based on token strings are described in US patent application Ser. No. 12 / 341,743, filed December 22, 2008, entitled "Method and Apparatus for Searching Using An Active Ontology. The entire disclosure of is incorporated herein by reference.

In some examples, once the natural language processing module 732 identifies the actionable intent (or domain) based on the user request, the natural language processing module 732 may construct a structured query to express the identified actionable intent ( structured query). In some examples, the structured query may include parameters for one or more nodes in the domain for actionable intent, with at least some of the parameters appended with specific information and requirements specified in the user request. For example, a user may say, "Make a meal reservation at a sushi store at seven." In such a case, the natural language processing module 732 may correctly identify the actable intent as "restaurant reservation" based on the user input. Depending on the ontology, the structured query for the "restaurant reservation" domain may include parameters such as {cooking}, {time}, {date}, {number of people} and the like. In some examples, based on the speech input, and the text derived from the speech input using the STT processing module 730, the natural language processing module 732 may generate a partially structured query for the restaurant reservation domain, The partially structured query here includes parameters {cuisine = "sushi"} and {time = "7pm". However, in this example, the user's speech contains insufficient information to complete a structured query associated with the domain. Thus, other mandatory parameters such as {number of people} and {date} may not be specified in the structured query based on the information currently available. In some examples, natural language processing module 732 may append the received contextual information to some parameters of the structured query. For example, in some examples, if a user requests a sushi restaurant "near me," natural language processing module 732 may append GPS coordinates from the user device to the {location} parameter in the structured query.

In some examples, natural language processing module 732 may pass the generated structured query (including any completed parameters) to task flow processing module 736 (“task flow processor”). The task flow processing module 736 receives the structured query from the natural language processing module 732, completes the structured query if necessary, and performs the operations required to "complete" the user's ultimate request. Can be configured. In some examples, various procedures needed to accomplish these tasks may be provided in task flow models 754. In some examples, task flow models 754 may include procedures for obtaining additional information from a user, and task flows for performing actions associated with actable intent.

As mentioned above, to complete a structured query, task flow processing module 736 may need to initiate additional conversations with the user to obtain additional information and / or to clearly distinguish potentially ambiguous utterances. . If such interaction is required, the task flow processing module 736 can call the conversation flow processing module 734 to participate in the conversation with the user. In some examples, conversation flow processing module 734 may determine how (and / or when) to ask the user for additional information, and receive and process the user response. The questions may be provided to users through the I / O processing module 728 and answers may be received from them. In some examples, the conversation flow processing module 734 may present the conversation output to the user via audio and / or visual output and receive input from the user via verbal or physical (eg, clicking) responses. Following the example above, when the task flow processing module 736 calls the conversation flow processing module 734 to determine the "number of people" and "date" information for the structured query associated with the domain "restaurant reservation", the conversation flow The processing module 734 is "How many people?" And questions such as "when is it?" When an answer is received from the user, conversation flow processing module 734 may then append the missing information to the structured query, or pass information to task flow processing module 736 to complete the missing information from the structured query. .

Once task flow processing module 736 has completed a structured query for actionable intent, task flow processing module 736 can proceed to perform the ultimate task associated with the actionable intent. Thus, task flow processing module 736 can execute steps and instructions in the task flow model in accordance with certain parameters included in the structured query. For example, the task flow model for the actible intent of "restaurant reservation" may include steps and instructions for contacting the restaurant and actually requesting a reservation for a particular number of people at a particular time. For example, using a structured query, such as: {Restaurant Reservation, Restaurant = ABC Cafe, Date = 3/12/2012, Time = 7:00 PM, Number of Persons = 5}, the task flow processing module 736 would then: The steps may be performed: (1) logging on to the ABC Cafe's server or restaurant reservation system such as OPENTABLE®, (2) entering date, time, and headcount information in a form on the website, (3 ) Submitting the form, and (4) creating a calendar entry for the reservation in the user's calendar.

In some examples, task flow processing module 736 may assist in service processing module 738 (“service processing module”) to complete a task requested at user input or to provide an informative answer requested at user input. Can be adopted. For example, the service processing module 738 can call other user applications installed on the user device or call them to set up a calendar entry, call a map search, to call a phone on behalf of the task flow processing module 736, or with them. Interact and call or interact with third party services (eg, restaurant reservation portals, social networking websites, banking portals, etc.). In some examples, the protocols and application programming interfaces (APIs) required by each service may be specified by a separate service model of service models 756. The service processing module 738 can access the appropriate service model for the service and generate requests for the service according to the protocols and APIs required by the service according to the service model.

For example, if a restaurant has enabled an online booking service, the restaurant may submit a service model that specifies APIs for passing parameters for making reservations and values of parameters for the online booking service. When requested by the task flow processing module 736, the service processing module 738 can establish a network connection with the online booking service using the web address stored in the service model, and format according to the API of the online booking service. May send the necessary parameters of the reservation (eg, time, date, number of people) to the online booking interface.

In some examples, natural language processing module 732, dialog flow processing module 734, and task flow processing module 736 may use information to further clarify and refine the user's intent to infer and define the user's intention. It can be used collectively and repeatedly to obtain and generate a response (ie, output to the user, or completion of a task) to finally fulfill the user's intent. The generated response may be a conversational response to speech input that at least partially fulfills the user's intent. In addition, in some examples, the generated response may be output as a speech output. In these examples, the generated response may be sent to speech synthesis module 740 (eg, speech synthesizer), where speech synthesis module 740 may be processed to synthesize a conversational response in speech form. In yet other examples, the generated response may be data content associated with satisfying a user request in speech input.

Speech synthesis module 740 may be configured to synthesize speech output for presentation to a user. Speech synthesis module 740 synthesizes the speech output based on the text provided by the digital assistant. For example, the generated chat response may be in the form of a text string. Speech synthesis module 740 may convert the text string into an audible speech output. Speech synthesis module 740 can use any suitable speech synthesis technique to generate speech output from text, which includes concatenative synthesis, unit selection synthesis, diphone synthesis, domain-specific synthesis, and po. Formant synthesis, Articulatory synthesis, hidden Markov model (HMM) based synthesis, and sinewave synthesis are not limited thereto. In some examples, speech synthesis module 740 may be configured to synthesize individual words based on a phoneme string corresponding to the words. For example, the phoneme string may be associated with a word of the generated conversation response. The phoneme string may be stored in metadata associated with the word. The speech synthesis model 740 may be configured to directly process phoneme strings in the metadata to synthesize words in speech form.

In some examples, instead of (or in addition to) using speech synthesis module 740, speech synthesis may be performed on a remote device (eg, server system 108), and the synthesized speech may be output to a user. May be sent to the user device for this purpose. For example, this may occur in some implementations in which output for the digital assistant is generated at the server system. And since server systems generally have more processing power or resources than user devices, it may be possible to obtain speech output of higher quality than actually in client-side synthesis.

Further details regarding the digital assistant are filed on January 10, 2011 and filed with the name "Intelligent Automated Assistant" US Patent Application No. 12 / 987,982, and filed September 30, 2011, US Patent Application No. 13 / 251,088 entitled "Generating and Processing Task Items That Represent Tasks to Perform", the entire disclosures of which are incorporated herein by reference.

4. Example Techniques for Intelligent Task Discovery

8A-8E illustrate example techniques including an example user interface (“UI”) for intelligent task navigation in accordance with some embodiments. These figures are also used to describe the processes described below, including the processes in FIGS. 10A-10C and 11.

8A shows electronic device 800. Electronic device 800 may, in some embodiments, devices 104, 122, 200, 400, 600, 1200, 1300 (FIGS. 1, 2A, 3, 4, 5A, 6A, and 6B, 12, 13). In the example shown, the electronic device 800 is a mobile phone. The illustrated example is described herein with reference to electronic device 800, although various electronic devices, such as laptops, desktops, tablets, wearable devices, and consumer electronics, such as televisions and speakers, in other implementations. It will be appreciated that it can be used.

The electronic device 800 provides a user interface so that a user can interact with the intelligent automation assistant. The user interface may be visual and / or audio. In some examples, electronic device 800 displays a graphical user interface on a display screen, such as a touch-sensitive display screen 802. In some examples, electronic device 800 samples audio user input through a microphone of the electronic device. Referring to FIG. 8A, the electronic device 800 receives a user input “Read My Messages”. User input may be speech, physical touch, active gesture, or a combination thereof. In response, the electronic device 800 displays a text transfer 804 of the user input “Read My Messages” on the display screen 802.

In response to the user input, the electronic device 800 infers the user's intention (eg, reading one or more text messages most recently received and not yet read on the electronic device) and accordingly receives the unread text message received on the electronic device. Acquire. The electronic device 800 converts the text message into speech and provides an audio output (eg, via a speaker) that includes an audio representation of the text message. In some cases, the audio output also includes relevant information (eg, caller name, time stamp, etc.) associated with the text message and a prompt for further input. In the example shown, the electronic device 800 provides an audio output 812: "When Tony says, come to Cupertino Infinite Loop 1, call Pepper at 123-456-7890. Would you like to reply? ? "

Based on the received text message, the electronic device 800 identifies one or more tasks. In some examples, electronic device 800 determines whether any portion of a text message (eg, a portion of a text string) is associated with a predetermined data type of the plurality of data types. The plurality of data types include: address (s) (eg, "Cupintino Infinite Loop 1"), phone number (s) (eg, "123-456-7890"), Uniform Resource Locator (URL) (eg " http://www.apple.com/ "), date (s) (e.g.," tomorrow "," 6/1/2000 "), interested person (s) (" Tony Stark "), Including but not limited to place (s) of interest, package tracking number (s), flight number (s), and the like. It will be appreciated that any information of interest may be defined as relating to one or more data types, eg, based on word recognition, patterns of information, and / or metadata.

In some examples, at electronic device 800, one or more detectors are implemented to analyze content (eg, text messages, emails, etc.) to detect each of one or more predefined data types. The detector includes one or more software programs capable of recognizing data of a predefined data type (s) in the content. In some examples, the detector may include text (eg, letters), numbers (eg, integers), text and / or numbers (eg, xx / xx / xxxx, (xxx) xxx-xxxx), predetermined keywords arranged in a particular format. ("http: //", "Apt."), information of interest (eg, names in a user's address book), or a combination thereof. In some examples, a detector is implemented to detect a single data type. For example, a phone number detector may be implemented to detect a phone number, a URL detector may be implemented to detect a URL, and an address detector may be implemented to detect an address. In some examples, each of the detectors on the electronic device 800 is run to analyze the content to detect the presence of data of each data type.

The electronic device 800 associates each data type with at least one task, and in some cases, each data type has one or more parameters. For example, the "address" data type may be associated with a task of navigation with a parameter of address, or with a task that updates a contact entry, or both. As another example, the “URL” data type may include a first task that opens a browser having a parameter of a website link, a second task of creating a bookmark of a linked address, and a third that adds a parameter of the website link to a read list. Associated with the task. As another example, the "flight number" data type is associated with a task of setting up a reminder using parameters of date and time.

In some examples, different types of electronic devices associate the same data type with different sets of tasks. For example, on a mobile phone, such as electronic device 800, the "URL" data type is associated with two tasks: a task that opens a URL in a browser and a task that adds the URL to the read list. In comparison, on a navigation device, the "URL" data type is associated only with the task of adding a URL to the read list. The task of opening a URL in a browser may be undesirable, for example, in a use case (eg, driving) of a navigation device.

In some examples, electronic device 800 associates a data type with one or more tasks based on contextual information. For example, based on the speed of movement of the electronic device (eg, measured by the GPS coordinates of the device over time), or when the electronic device is paired with a vehicle, the electronic device may be in a vehicle in which the electronic device moves. Can be determined. If the electronic device determines that the electronic device is in a moving vehicle, the "URL" data type is not associated with a task opening in the browser, but only with a task that adds a URL to the reading list, for example.

In some examples, after detecting data of a particular data type, the electronic device 800 displays (eg, on a display screen) one or more of the tasks associated with the data type along with corresponding parameters. With reference to FIG. 8B, multiple detectors are executed to analyze the text string of the text message described above. The telephone detector detects the telephone number "123-456-7890", and the address detector detects the address "Cupertino Infinite Loop 1". Based on the detected data types, the electronic device 800 may assign a first task (ie, dial a number) associated with the "phone number" data type and a second task (ie, address) associated with the "address" data type. Guiding directions). Accordingly, the electronic device 800 represents a selectable affordance 808 and a second task (dialing 123-456-7890) representing the first task (Directing to Cupertino Infinite Loop 1). Selectable affordances 810 are displayed on the display screen.

In some examples, if multiple tasks are displayable (eg, because the detected data type is associated with multiple tasks and / or multiple data types are detected), the electronic device 800 may be, for example, the type of electronic device. Based on, prioritize the display of some tasks over the display of other tasks. For example, on a navigation device, the display of a task of navigation takes precedence over the display of a dialing task. Prioritization of the indication can be implemented in a number of ways. For example, the affordances representing the first task are displayed more prominently than the affordances representing the second task (eg, different fonts, different colors, different setting positions). As another example, the affordances representing the second task are displayed after the affordances representing the first task are displayed (eg, temporarily). As another example, no affordances representing the second task are displayed. The second task, although not indicated, can still be initiated by speech input, as discussed in more detail below with respect to FIG. 8C.

It will be appreciated that the display of tasks on the electronic device 800 may vary based on other factors, such as the amount of available display area and contextual information. For example, the size and number of affordances (eg, buttons) displayed may vary based on the amount of available display area, which in turn results in the type of electronic device (eg, phone, tablet, Television) and / or the operating system of the electronic device. In some examples, visual enhancement (s) may be provided based on the amount of available display area. For example, on an electronic device (eg, a television) with relatively large display capabilities, the electronic device may pin an address (eg, for the task of navigation) as a pin on a map as part of an intelligent automated assistant GUI. As another example, if a phone number in a user's contact is detected, the electronic device obtains additional information related to the contact (via an index in the user's contact list) and as part of the intelligent automated assistant GUI for the task of dialing. Information may be provided (eg, displayed).

Referring to FIG. 8C, the electronic device 800 receives a user input that initiates a task of navigation and causes the task to be performed. As shown, the electronic device 800 displays the text string "Directing Direction to Cupertino Infinite Loop 1". In some examples, electronic device 800 calls one or more software service providers (eg, application (s), API (s), and / or function (s) for navigation) to perform the selected task. In some examples, the electronic device 800 causes the task to be performed on a second electronic device that is different from the electronic device 800. For example, upon receiving a user input, the electronic device 800 causes the navigation device to initiate the navigation, for example by sending an address to the navigation device.

In some examples, the user input initiating the task of the navigation is a physical touch on the selectable affordance 808 (“Directing to Cupertino Infinite Loop 1”) as shown in FIG. 8B. In some examples, electronic device 800 allows a user to initiate one or more tasks associated with data types detected through speech input, such as “take me there”. In response to receiving the speech input, the electronic device 800 determines whether the speech input represents one or more tasks associated with the detected data type. If the speech input indicates a task associated with the detected data type, the electronic device causes the task to be performed, as discussed in more detail below.

In some examples, the electronic device 800 identifies the task (s) and parameter (s) from speech input based on the contextual information. In some examples, the contextual information may include what is currently being displayed on the electronic device. For example, if affordance 808 (“Directing to Cupertino Infinite Loop 1”) in FIG. 8B is displayed when the user says “Take me there”, the electronic device is at least as displayed. Infer the user's intention to lead to "Cupintino Infinite Loop 1" based in part. In some examples, the contextual information may include data types (eg, address, phone number) and corresponding data (eg, “Cupintino Infinite Loop 1”, “123-456-7890”) detected by the electronic device. It may include. For example, if a user provides an audio input “take me there” when affordance 808 is not displayed, the electronic device intends to drive to “Cupintino Infinite Loop 1” based on contextual information. Can still be inferred.

In some examples, upon receiving a speech input (eg, “take me there”) initiating a task to perform, the electronic device 800 may process the data corresponding to the speech input, and selected portions of contextual information for processing. Send to one or more servers. In some examples, one or more servers convert the data into a textual representation (eg, a text string), infer at least a portion of the user intent based on the textual representation and the contextual information received and / or operate the inferred user intent as a task. Make up. In some examples, personal data of the user is not transmitted to one or more servers. In some examples, contextual information such as text messages and emails, and information inferred from the text messages and emails (eg, detected types) constitute personal data and thus are not transmitted to one or more servers. In some examples, contextual information, including personal data of a user, is used locally by the electronic device to assist in the processing of speech input and the identification of task (s) and parameter (s). For example, the detected phone number "123-456-7890" and the detected address "Cupino Tino Infinite Loop 1" are not sent to the server, but the phone number and address are used locally to make calls from the electronic device, respectively. Infers the parameters for the task of the navigation and the task of the navigation. In some examples, identification of task (s) and parameter (s) based on speech input is performed locally on the electronic device.

Data of certain data types detected by the electronic device 800 may be automatically stored in the electronic device. For example, after processing the text message described above, the electronic device retrieves the detected address "Cuptintino Infinite Loop 1" and the detected telephone number "123-456-7890" from each of the data types (i.e. , Address and phone number) in the electronic device 800. In some examples, metadata associated with the text message is stored. For example, the sender of the text message ("Tony Stark") and the time stamp of the text message are stored in association with the data and data types. In some examples, stored data, data types, and metadata are portions of short-term memory that last only for a limited period of time (eg, during the current user session of the intelligent automation assistant, for several minutes, until the device enters an inactive state). to be. In some examples, the stored data, data types, and metadata are part of long term memory associated with the user.

In some examples, stored contextual information is used to interpret user input. Referring to FIG. 8D, the user provides (eg, speaks) the input “Take me to the latest address Tony sent me”, which is transcribed and displayed on the display screen 802 as a text string 818. . Based on the phrase "Recent address that Tony sent me," the electronic device 800 refers to the stored contextual information, which associates "Tony" and "address" with "Cupintino Infinite Loop 1". . Accordingly, the electronic device 800 can infer the parameter "Cuptintino Infinite Loop 1" for the task of navigation. Referring to FIG. 8E, the electronic device launches a service provider (eg, a navigation application, a function, and / or an API) to perform a task of navigation with the parameter “Cupinotino Infinite Loop 1”.

It will be appreciated that the stored contextual information can be used to process user input regarding any software application on the electronic device 800. For example, if a user calls a software application (e.g. a navigation application) and says "Recent address that Tony sent me," the software application will infer "Cupintino Infinite Loop 1" with reference to the stored contextual information. It is possible. It should further be appreciated that the stored contextual information may be used for purposes other than interpreting user input by the electronic device. For example, when a user invokes a navigation application, the navigation application may provide (eg, display) relevant portions of contextual information, such as the address "Cupinotino Infinite Loop 1," as the proposed destination without a user request. Can be.

After searching for one or more tasks, the electronic device may present (eg, display) the discovered tasks to the user at multiple points in time during operation. Example entry points for discovered tasks are discussed with respect to FIGS. 9A-9C. 9A shows an electronic device 900. Electronic device 900 may, in some embodiments, devices 104, 122, 200, 400, 600, 1200, 1300 (FIGS. 1, 2A, 3, 4, 5A, 6A, and 6B, 12, 13). In the example shown, the electronic device 900 is an electronic device with navigation capabilities. The illustrated example is described herein with reference to electronic device 900, but various electronic devices, such as laptops, desktops, tablets, wearable devices (eg, watches, headphones), and consumer electronics, such as televisions And the speaker may be used in other implementations.

In some examples, the electronic device 900 may enter an entry for some of the tasks found within the content after presenting the particular content (eg, text message, email, document) to the user and / or while presenting the content to the user. Provide points. As shown in FIG. 9A, the electronic device 900 provides an audio output 918 corresponding to the text message received at the electronic device 900: "Tony said, to come to Cupertino Infinite Loop 1. When you call Pepper, call 123-456-7890. Do you want to reply? " The electronic device 900 also includes, on the display screen, an affordance 910 (“guiding”), an affordance 912 (“reading”), an affordance 914 (“no reply”), and Provides an appearance 916 (" reply "). In particular, affordance 910 (“guiding directions”) represents a task of navigation, the task of navigation being searched based on a text message referring to audio input 918. In some examples, affordance 910 is indicated after electronic device 900 provides audio output 918 and / or electronic device 900 finishes providing audio output 918. When affordance 910 is selected by the user (eg, via physical touch input, via speech input “Directing Directions”), the task of navigation to “Cupinotino Infinite Loop 1” is initiated (not shown). ).

In the example shown, after the electronic device 900 has finished providing the audio output 918, the electronic device 900 receives an input "Yes, goodbye" from the user. As shown in FIG. 9B, the electronic device 900 provides an audio output 926 ("Your reply to Tony, hello. Would you like to send it?"). Electronic device 900 also provides affordance 920 (“read”), affordance 922 (“change”), affordance 924 (“send”) while providing audio output 926. Display. When the user selects affordance 924 (“send”), the electronic device 900, in some cases, sends the user's reply using a service, such as a text message application.

The electronic device allows the user to make changes after composing the reply message, for example by specifying different service providers (eg, from different domains). For example, upon receiving the user input “Yes, hello” (in response to the prompt “Reply?”), The electronic device 800 may identify a default service provider for sending a reply. In some examples, the service provider is identified based on the original content (eg, original text message). That is, if the reply responds to the text message, the electronic device identifies the text message application as the default service provider. However, if the user provides subsequent input, such as "email" or "generate reminder," the electronic device identifies the second service provider based on the user input and generates a message (eg, "hello"). ), And related contextual information (“Tony Stark”) may be provided to a second service provider (eg, an email application or reminder API).

In some examples, the electronic device 900 provides additional entry points for some of the tasks discovered during the user session using the intelligent automation assistant. As shown in FIG. 9C, the electronic device 900 provides an audio output 930 (“send”) and in some cases transmits a text message in response to user input (not shown). The electronic device 900 also displays an affordance 928 (“guiding directions”), which represent the task of the navigation and are functionally identical to the affordance 910 (FIG. 9A) previously displayed. When affordance 928 is selected by the user (eg, via physical touch input, via speech input “Directing Directions”), the task of navigation to “Cupinotino Infinite Loop 1” is initiated (not shown). ). In some examples, if the affordances representing the task of the searched navigation are not displayed, the user can initiate the task via speech input, as described above.

In particular, each of the electronic device 800 (FIGS. 8A-8E) and the electronic device 900 (FIGS. 9A-9C) is directed to Pepper 123-456 when it comes to the same text string (“Cupintino Infinite Loop 1”). Phone 800-8 ", but the devices 800 and 900 each provide different visual outputs corresponding to the discovered tasks (e.g., the affordances 808 of FIG. 8B). 810 and affordance 910 of FIG. 9A). In some examples, the two electronic devices 800, 900 may have different numbers and types of tasks as a result of their different device type (eg, mobile phone and navigation device) and contextual information (eg, whether the device is in a moving vehicle). And as described above. In some examples, the two electronic devices display different numbers and types of tasks as a result of the amount of different available display areas, as described above.

Although the illustrated examples of FIGS. 8A-8E and 9A-9C are described herein with reference to text messages, various content such as email, audio messages, video, images, etc. may be analyzed to identify specific data types (and associated Will be able to detect tasks).

It will further be appreciated that the detection of data types and the identification of associated tasks can be triggered by or without a user request. For example, the electronic device 800 may initiate intelligent task search upon receipt of a text message, without a user request (eg, “Read My Messages”). When the electronic device 800 detects one or more specific data types, the electronic device may automatically output (eg, provide an indication or speech output) some of the tasks associated with the detected data types. The display and selection of the identified tasks may be implemented in a manner similar to those described above.

5. Example Processes for Intelligent Task Discovery

10A-10C show a process 1000 for operating a digital assistant according to various examples. Process 1000 is performed using, for example, one or more electronic devices (eg, devices 104, 106, 200, 400, 600, 1200, or 1300) that implement a digital assistant. In some examples, the process is performed at a client-server system (eg, system 100) that implements a digital assistant. The blocks of the process may be divided in any manner between the server (eg, DA server 106) and the client (eg, user device 104). In process 1000, some blocks are optionally combined, and the order of some blocks is optionally changed, and some blocks are optionally omitted. In some examples, only a subset of the features or blocks described below with reference to FIGS. 10A-10C are performed.

As described below, the method 1000 provides an efficient way for doing task navigation in content such as text messages and emails on an electronic device. The method creates a more efficient human-machine interface by manually specifying and invoking the task or service (s) and manually reproducing portions of the content to offload the user from performing the task (s). . In battery operated computing devices, the user conserves power and extends the battery charging cycle by allowing the tasks to be completed more efficiently.

At block 1002, the electronic device receives a communication. In some examples, the communication can be, but is not limited to, any content, such as a text message, an email, a document, an audio clip, an image, or a combination thereof. In some examples, the communication includes a text string (eg, inferred using a text recognition technique or a speech-to-text transformation technique). Optionally, at block 1003, the electronic device uses an automated assistant to provide a speech output corresponding to a text string of the communication (eg, “Come to Cupertino Infinite Loop 1”). In some examples, speech output corresponding to the text string of the communication is provided in response to a user request (eg, “Read My Messages”). In some examples, providing a speech output corresponding to the text string of the communication includes converting the text string of the communication to speech at block 1050.

At block 1006, the electronic device determines whether a portion of the text string is associated with a predetermined data type of the plurality of data types. The data type has at least one task associated with it. Optionally, at block 1008, the electronic device detects data of a data type, using a detector, in a portion of the text string. The detector corresponds to the data type of the plurality of data types. Optionally, at block 1010, the electronic device identifies at least one task from the plurality of tasks based on the type of electronic device.

In some examples, the data type of the plurality of data types includes at least one of the following: an address, telephone number, Uniform Resource Locator (URL), date, person of interest, place of interest, package tracking number, or flight number .

Optionally, at block 1012, the electronic device displays the affordance corresponding to the task using the display of the electronic device. Optionally, at block 1016, the electronic device prioritizes the indication of the affordance of the first task over the indication of the affordance of the second task based on the type of the electronic device. Optionally, at block 1017, the electronic device prioritizes the display of the affordance of the first task by stopping the display of the affordance of the second task. In some examples where the affordance of the second task is not displayed, when the electronic device receives a user input (eg, speech) representing the second task, the electronic device causes the second task to be performed.

At block 1014, the electronic device stops causing the task to be performed based on the portion of the text string in accordance with the determination that the portion of the text string is not associated with the data type of the plurality of data types.

At block 1018, the electronic device receives user input indicating a task of at least one task associated with the data type according to determining that a portion of the text string is associated with a data type of the plurality of data types. Optionally at block 1020, the user input includes a selection of displayed affordances.

Optionally at block 1022, the user input representing the task is speech input. In some examples, the speech input does not include a portion of the text. At block 1025, user intent is inferred from speech input based on contextual information. The contextual information includes a portion of the text.

In some examples, at block 1023, the electronic device sends data corresponding to the speech input to a second electronic device that is different from the electronic device. The data corresponding to the speech input does not include a text string of the communication. At block 1024, the electronic device receives, from the second electronic device, a user intent corresponding to the task.

In response to user input representing the task at block 1018, the electronic device causes the task to be performed based on the portion of the text string. In some examples, the task is performed at a second electronic device different from the electronic device.

In some examples, the speech output corresponding to the text string of the communication is a first speech output and the speech input representing the task is a first speech input. At block 1004, the electronic device provides a second speech output (eg, “Reply?”) Indicating a prompt to respond to the communication after providing the first speech output at block 1003. At block 1005, the electronic device receives a second speech input (eg, “Yes, goodbye”) from the user after providing the second speech output at block 1004. In some examples, the second speech input is received before the first speech input (eg, “take me there”) is received.

In some examples, at block 1054, the electronic device identifies a first service provider (eg, a text message software application) based on the communication. The first service provider is associated with a first domain of the plurality of domains. At block 1056, after receiving the second speech input (“yes, bye”), the electronic device receives a third speech input (eg, “send by email”) indicating a second one of the plurality of domains. do. The second domain is different from the first domain. In response, at block 1058, the electronic device identifies a second service provider (eg, an email software application) based on the third speech input, and at block 1060 collects data associated with the second speech input. Provide to service provider.

In some examples, two or more data types are detected in a text string. In some examples, the portion of the text string is the first portion of the text string, the data type is the first data type, and the task is the first task. At block 1070, the electronic device determines whether a second portion of the text string is associated with a second data type of the plurality of data types. At block 1072, in accordance with a determination that the second portion of the text string is associated with a second data type of the plurality of data types, the electronic device identifies the second task based on the second data type. At block 1074, according to the determination that the second portion of the text string is not associated with the second data type of the plurality of data types, the electronic device stops identifying the second task based on the second data type.

Optionally at block 1030, the electronic device stores information associated with the communication on the electronic device. In some examples, the stored information associated with the communication includes a portion of the text string. In some examples, the stored information associated with the communication includes a data type. In some examples, the stored information associated with the communication includes metadata associated with the communication. Optionally at block 1032, the electronic device receives an input. Optionally at block 1034, the electronic device identifies the task based on the input. Optionally at block 1036, the electronic device performs the task based on at least a portion of the stored information.

Optionally at block 1040, the electronic device stores information associated with the communication on the electronic device. In some examples, block 1040 is functionally similar and / or identical to block 1030. Optionally at block 1042, the electronic device invokes a software application and, at block 1044, performs a task based at least in part on the stored information.

11 shows a process 1100 for operating a digital assistant according to various examples. Process 1100 is performed using, for example, one or more electronic devices (eg, devices 104, 106, 200, 400, 600, 1200, or 1300) that implement a digital assistant. In some examples, the process is performed at a client-server system (eg, system 100) that implements a digital assistant. Blocks of processes may be divided in any manner between the server (eg, DA server 106) and the client (eg, user device 104). In process 1100, some blocks are optionally combined, and the order of some blocks is optionally changed, and some blocks are optionally omitted. In some examples, only a subset of the features or blocks described below with reference to FIG. 11 is performed.

As described below, the method 1100 provides an efficient way for doing task navigation in content such as text messages and emails on an electronic device. The method creates a more efficient human-machine interface by manually specifying and invoking the task or service (s) and manually reproducing portions of the content to offload the user from performing the task (s). . In battery operated computing devices, the user conserves power and extends the battery charging cycle by allowing the tasks to be completed more efficiently.

At block 1102, the electronic device receives a communication comprising a text string. In block 1104, the electronic device determines whether a portion of the text string is associated with a predetermined data type of the plurality of data types. In some examples, the data type of the plurality of data types includes at least one of the following: an address, telephone number, Uniform Resource Locator (URL), date, person of interest, place of interest, package tracking number, or flight number .

In accordance with the determination that the portion of the text string is associated with a data type of the plurality of data types, the electronic device infers a task having one or more parameters based on the data type at 1108 and outputs the inferred tasks at 1110. . Certain of the one or more parameters includes a portion of the text string.

Optionally, at block 1112, the electronic device displays an affordance corresponding to the inferred task using the display of the electronic device, and the user input includes a selection of affordances. Optionally at 1114, the electronic device outputs an audio output.

Optionally, at block 1116, the electronic device receives user input indicating a selection of the inferred task. Optionally, at block 1118, in response to user input indicating a selection of the inferred task, the electronic device causes the task to be performed based on a portion of the text string.

At block 1106, in accordance with the determination that the portion of the text string is not associated with the data type of the plurality of data types, the electronic device stops inferring the task having one or more parameters based on the data type.

According to some embodiments, FIG. 12 shows a functional block diagram of an electronic device 1200 constructed in accordance with the principles of the various embodiments described, wherein the various embodiments described are FIGS. 8A-8E, 9A- 9. 9c and those described with reference to FIGS. 10A-10C. The functional blocks of the device are, optionally, implemented by hardware, software, or a combination of hardware and software for carrying out the principles of the various described embodiments. It is understood by those skilled in the art that the functional blocks described in FIG. 12 may be selectively combined or separated into subblocks to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

As shown in FIG. 12, the electronic device 1200 includes a processing unit 1208, and optionally, a display unit 1202 and an output unit 1204. Optionally, the processing unit 1208 is coupled to the display unit 1202 and the output unit 1204. In some embodiments, the processing unit 1208 is the receiving unit 1210, the determination 1212, the ministry unit 1214, and optionally, the transformation unit 1216, the identification unit 1218, the providing unit 1220. , Detection unit 1222, inference unit 1224, transmission unit 1226, storage unit 1228, execution unit 1230, and calling unit 1232.

In some examples, processing unit 1208 may include: receiving a communication including a text string (eg, using receiving unit 1210) (eg, block 1002 of FIG. 10A); Determine whether a portion of the text string is associated with a predetermined data type of the plurality of data types (eg, using decision unit 1212), the data type having at least one task associated with it (eg, in FIG. 10B). Block 1006); In accordance with the determination that the portion of the text string is associated with a data type of the plurality of data types: (eg, using receiving unit 1210) receives a user input indicating a task of at least one task associated with the data type (eg, using receiving unit 1210); , Block 1018 of FIG. 10B; In response to user input representing the task, causing the task to be performed based on a portion of the text string (eg, using ministry unit 1214) (eg, block 1026 of FIG. 10C); In response to determining that a portion of the text string is not associated with a data type of the plurality of data types, to stop the task from being performed based on the portion of the text string (eg, using ministry unit 1214) (eg, Block 1014 of FIG. 10B).

In some examples, processing unit 1208 may: (eg, use output unit 1204) to use an automated assistant to provide a speech output corresponding to a text string of the communication (eg, using the block of FIG. 1003)) further configured.

In some examples, speech output corresponding to a text string of the communication is provided in response to a user request.

In some examples, providing a speech output corresponding to the text string of the communication is to convert the text string of the communication to speech (eg, using conversion unit 1216) (eg, block 1050 of FIG. 10A). It includes.

In some examples, the speech output corresponding to the text string of the communication is a first speech output and the speech input representing the task is a first speech input, and the processing unit 1208: after providing the first speech output (eg, Provide a second speech output indicating a prompt to respond to the communication (using output unit 1204) (eg, block 1004 of FIG. 10A); After providing the second speech output, the second speech input is further configured to receive a second speech input from the user (eg, using the receiving unit 1210), the second speech input being received before the first speech input is received. (Eg, block 1005 of FIG. 10A).

In some examples, processing unit 1208: identifies a first service provider based on the communication (eg, using identification unit 1218) —the first service provider is associated with a first domain of the plurality of domains. (Eg, block 1054 of FIG. 10A); After receiving the second speech input, receiving a third speech input indicating a second one of the plurality of domains (eg, using the receiving unit 1210), the second domain being different from the first domain; (Eg, block 1056 of FIG. 10A); Identify the second service provider (eg, using identification unit 1218) based on the third speech input (eg, block 1058 of FIG. 10A); It is further configured to provide data associated with the second speech input to the second service provider (eg, using the providing unit 1220) (eg, block 1060 of FIG. 10A).

In some examples, the data type of the plurality of data types includes at least one of the following: an address, telephone number, Uniform Resource Locator (URL), date, person of interest, place of interest, package tracking number, or flight number .

In some examples, determining whether a portion of a text string is associated with the data type of a plurality of data types: in the portion of the text string (eg, using detection unit 1222), using a detector, using the data type Detecting the presence of data in the detector, and the detector corresponds to the data type of the plurality of data types (eg, block 1008 of FIG. 10B).

In some examples, processing unit 1208 is configured to identify at least one task from the plurality of tasks based on the type of electronic device 1200 (eg, using identification unit 1218) (eg, the block of FIG. 10B). 1010) further configured.

In some examples, the portion of the text string is the first portion of the text string, the data type is the first data type, the task is the first task, and the processing unit 1208: (eg, uses the determining unit 1212). Determine whether the second portion of the text string is associated with a second data type of the plurality of data types (eg, block 1070 of FIG. 10C); In accordance with a determination that the second portion of the text string is associated with a second data type of the plurality of data types, identify a second task based on the second data type (eg, using the identification unit 1218) and (eg, , Block 1072 of FIG. 10C), in accordance with a determination that the second portion of the text string is not associated with a second data type of the plurality of data types (eg, using identification unit 1218) Is further configured to stop identifying the second task based on (eg, block 1074 of FIG. 10C).

In some examples, processing unit 1208 is further configured to display an affordance corresponding to the task (eg, using display unit 1202) (eg, block 1012 of FIG. 10B), and the user input is Selection of affordance (eg, block 1020 of FIG. 10B).

In some examples, the task is a first task and processing unit 1208 is: a first rather than an indication of the affordance of the second task based on the type of electronic device 1200 (eg, using display unit 1202). It is further configured to prioritize the indication of the affordance of the task (eg, block 1016 of FIG. 10B).

In some examples, prioritizing the display of the affordance of the first task over the display of the affordance of the second task may include: stopping display of the affordance of the second task (eg, using display unit 1202). (Eg, block 1017 of FIG. 10B).

In some examples, processing unit 1208 may: receive a user input indicating a second task (eg, using receiving unit 1210); It is further configured to cause the second task to be performed (eg, using the ministry unit 1214).

In some examples, the user input representing the task is speech input (eg, block 1022 of FIG. 10B). In some examples, the speech input does not include a portion of the text.

In some examples, processing unit 1208 is further configured to infer user intent from speech input based on contextual information (eg, using inference unit 1224), the contextual information comprising a portion of text ( For example, block 1025 of FIG. 10B).

In some examples, electronic device 1200 is a first electronic device, and processing unit 1208: uses the first electronic device to: use the first electronic device to perform data corresponding to the speech input (eg, using transmission unit 1226). Transmit to a second electronic device different from the first electronic device, wherein the data corresponding to the speech input does not include a text string of the communication (eg, block 1023 of FIG. 10B); It is further configured to receive a user intent corresponding to the task (eg, using the receiving unit 1210) (eg, block 1024 of FIG. 10B).

In some examples, processing unit 1208 is further configured to store information associated with the communication (eg, using storage unit 1228) on the electronic device 1200 (eg, with the block of FIG. 10C). (1030)).

In some examples, the stored information associated with the communication includes a portion of the text string.

In some examples, the stored information associated with the communication includes a data type.

In some examples, the stored information associated with the communication includes metadata associated with the communication.

In some examples, processing unit 1208 may: receive an input (eg, using receive unit 1210) (eg, block 1032 of FIG. 10C); Identify the task based on the input (eg, using identification unit 1218) (eg, block 1034 of FIG. 10C); It is further configured to perform the task based on at least a portion of the stored information (eg, using the execution unit 1230) (eg, block 1036 of FIG. 10C).

In some examples, processing unit 1208 may include: calling a software application (eg, using calling unit 1230) (eg, block 1042 of FIG. 10C); It is further configured to perform the task based on at least a portion of the stored information (eg, using the execution unit 1230) (eg, block 1044 of FIG. 10C).

In some examples, the electronic device 1200 is a first electronic device and the task is performed on the second electronic device.

The operations described above with respect to FIGS. 10A-10C are optionally implemented by the components shown in FIGS. 1-4, 6A- 6B, 7A, and 12. For example, receive operation 1002, decision operation 1006, and ministry operation 1026 are optionally implemented by processor (s) 120. It will be apparent to those skilled in the art how other processes may be implemented based on the components shown in FIGS. 1-4, 6A and 6B, 7A, and 12. will be.

According to some embodiments, FIG. 13 shows a functional block diagram of an electronic device 1300 constructed in accordance with the principles of the various embodiments described, wherein the various embodiments described are FIGS. 8A-8D, 9A- 9. 9c and those described with reference to FIG. 11. The functional blocks of the device are, optionally, implemented by hardware, software, or a combination of hardware and software for carrying out the principles of the various described embodiments. It is understood by those skilled in the art that the functional blocks described in FIG. 13 may be selectively combined or separated into subblocks to implement the principles of the various described embodiments. Thus, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

As shown in FIG. 13, the electronic device 1300 includes an output unit 1304, a processing unit 1308, and optionally a display unit 1302. Processing unit 1308 is coupled to output unit 1304 and, optionally, display unit 1302. In some embodiments, processing unit 1308 includes receiving unit 1310, determining unit 1312, inference unit 1314, and optionally, ministry unit 1316.

In some examples, processing unit 1308 may include: receiving a communication including a text string (eg, using receiving unit 1310) (eg, block 1102 of FIG. 11); Determine (eg, using decision unit 1312) that a portion of the text string is associated with a predetermined data type of the plurality of data types (eg, block 1104 of FIG. 11); In accordance with the determination that a portion of the text string is associated with a data type of the plurality of data types: inferring a task having one or more parameters based on the data type (eg, using inference unit 1314) —of one or more parameters The predetermined parameter includes a portion of a text string—output the inferred task (eg, using output unit 1304) (eg, block 1108 of FIG. 11); To stop inferring a task having one or more parameters based on the data type (eg, using inference unit 1314) in accordance with a determination that a portion of the text string is not associated with the data type of the plurality of data types. (Eg, block 1106 of FIG. 11).

In some examples, processing unit 1300 may include: receiving a user input indicating a selection of the inferred task (eg, using receiving unit 1310) (eg, block 1116 of FIG. 11); In response to user input indicative of selection of the inferred task, the task is further configured (eg, using ministry unit 1316) to be performed based on a portion of the text string (eg, block 1118 of FIG. 11). )).

In some examples, outputting the inferred task includes displaying an affordance corresponding to the inferred task (eg, using display unit 1302), and the user input includes a selection of affordance (eg, FIG. Block 1112 of 11).

In some examples, outputting the inferred task includes outputting audio output (eg, using output unit 1304) (eg, block 1114 of FIG. 11).

In some examples, the data type of the plurality of data types includes at least one of the following: an address, telephone number, Uniform Resource Locator (URL), date, person of interest, place of interest, package tracking number, or flight number .

The operations described above with respect to FIG. 11 are optionally implemented by the components shown in FIGS. 1-4, 6A and 6B, 7A, and 13. For example, receive operation 1102, decision operation 1104, and inference operation 1108 are optionally implemented by processor (s) 120. It will be apparent to those skilled in the art how other processes may be implemented based on the components shown in FIGS. 1-4, 6A and 6B, 7A, and 13. will be.

According to some embodiments, a computer readable storage medium (eg, a non-transitory computer readable storage medium) for storing one or more programs for execution by one or more processors of an electronic device is provided, wherein the one or more programs Includes instructions for performing any of the methods or processes described herein.

According to some embodiments, an electronic device (eg, portable electronic device) is provided that includes means for performing any of the methods or processes described herein.

According to some implementations, an electronic device (eg, portable electronic device) is provided that includes a processing unit configured to perform any of the methods or processes described herein.

According to some embodiments, an electronic device (eg, portable electronic device) is provided that includes a memory that stores one or more processors and one or more programs for execution by one or more processors, wherein the one or more programs are described herein. Instructions for executing any of the methods or processes described in the specification.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. Embodiments are selected and described in order to best explain the concept of the techniques and their practical application. Thus, it would be possible for those skilled in the art to best utilize the techniques and various embodiments using various modifications as appropriate for the particular use contemplated.

Although the present disclosure and examples have been described fully with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosure and examples as defined by the claims.

As mentioned above, one aspect of the present technology is the collection and use of data available from various sources for improving the navigation of tasks in content. The present disclosure contemplates that in some cases such collected data may include personal information data that can be used to uniquely identify, contact, or locate a particular individual. Such personal information data may include demographic data, location based data, telephone numbers, email addresses, home addresses, or any other identifying information.

The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data may be used to deliver targeted content of greater interest to a user. Thus, the use of such personal information data enables the calculated control of the delivered content. In addition, other uses for personal information data that benefit the user are also contemplated by the present disclosure.

The present disclosure further contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with well established privacy policies and / or privacy practices. In particular, such entities should generally implement and continue to use privacy policies and practices that are perceived to meet or exceed industrial or administrative requirements for keeping personal information data private and secure. For example, personal information from users should be collected for legitimate and proper use of the entity and should not be shared or sold outside these legitimate uses. In addition, such collection should only occur after receiving informed consent from users. In addition, such entities will take any necessary steps to protect and secure access to such personal data and to ensure that others having access to the personal data adhere to their privacy policies and procedures. . In addition, such entities may themselves be evaluated by third parties to prove their adherence to widely recognized privacy policies and practices.

Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of ad delivery services, the present technology may be configured to allow users to select "I agree" or "don't agree" to participate in the collection of personal information data during registration for the service. . In another example, users may choose not to provide location information for target content delivery services. In another example, users may choose not to provide accurate location information but allow delivery of location zone information.

Thus, while the invention broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the invention also provides that various embodiments may also be implemented without the need for access to such personal information data. Consider. In other words, various embodiments of the present technology are not rendered inoperable due to the lack of all or part of such personal information data. For example, the content may be a basic minimum amount of personal information or non-personal information data such as content requested by a device associated with the user, other non-personal information available for content delivery services, or publicly available information. Can be selected and communicated to users by inferring a preference based on.

Claims (72)

A method (1000) for operating an automated assistant on an electronic device, the method comprising:
In the electronic device,
Receiving (1002) a communication comprising a text string;
Determining (1004) that a portion of the text string is associated with a predetermined data type of a plurality of data types, the data type having at least one task associated with it;
In accordance with a determination that said portion of said text string is associated with said data type of said plurality of data types:
Determining a task among the at least one task;
Simultaneously using the display of the electronic device, displaying an affordance corresponding to the determined task and an affordance corresponding to a second task, wherein an indication of the affordance corresponding to the determined task is based on the type of the electronic device Prioritizing an indication of an affordance corresponding to the second task;
Receiving (1018) a user input associated with an affordance corresponding to the determined task; And
In response to receiving the user input associated with an affordance corresponding to the determined task, causing the determined task to be performed based on the portion of the text string; And
In response to determining that the portion of the text string is not associated with the data type of the plurality of data types, stopping 1010 from causing a task to be performed based on the portion of the text string.
How to include. The method of claim 1, further comprising providing speech output corresponding to the text string of the communication using the automated assistant. The method of claim 2, wherein the speech output corresponding to the text string of the communication is provided in response to a user request. 3. The method of claim 2, wherein providing the speech output corresponding to the text string of the communication comprises converting the text string of the communication to speech. The method of claim 2,
The speech output corresponding to the text string of the communication is a first speech output and the user input associated with an affordance corresponding to the determined task is a first speech input and the method includes:
After providing the first speech output, using the electronic device to provide a second speech output indicative of a prompt to respond to the communication; And
After providing the second speech output, further using the electronic device to receive a second speech input from the user,
And the second speech input is received before the first speech input is received. The method of claim 5,
Using the electronic device:
Identifying a first service provider based on the communication, the first service provider associated with a first domain of a plurality of domains;
After receiving the second speech input, receiving a third speech input representing a second one of the plurality of domains, wherein the second domain is different from the first domain;
Identifying a second service provider based on the third speech input; And
Providing the data associated with the second speech input to the second service provider. The method of claim 1, wherein the data type of the plurality of data types is:
address,
Phone number,
Uniform Resource Locator (URL),
date,
Interested Person,
Places of interest,
Package tracking number, or
And at least one of the flight number. 7. The method of claim 1, wherein determining whether the portion of the text string is associated with the data type of a plurality of data types comprises:
Using a detector, detecting the presence of data of the data type in the portion of the text string, the detector corresponding to the data type of the plurality of data types. The method according to any one of claims 1 to 6,
Using the electronic device, further comprising identifying the at least one task from a plurality of tasks based on the type of the electronic device. The method of claim 1, wherein the portion of the text string is a first portion of the text string, the data type is a first data type, the determined task is a first task, and the Way:
Identifying, using the electronic device, the second portion of the text string is associated with a second data type of the plurality of data types, identifying the second task based on the second data type. Included as, method. The method according to any one of claims 1 to 6,
And the user input associated with an affordance corresponding to the determined task comprises a selection of affordance corresponding to the determined task. delete delete The method according to any one of claims 1 to 6,
Using the electronic device, receiving a user input representing the second task; And
And causing the second task to be performed. The method of claim 1, wherein the user input associated with an affordance corresponding to the determined task is a speech input. The method of claim 15, wherein the speech input does not include the portion of the text string. The method of claim 15, further comprising inferring a user intent from the speech input based on context information, wherein the context information includes the portion of the text string. The method of claim 15, wherein the electronic device is a first electronic device, the method comprising:
Using the first electronic device:
Transmitting data corresponding to the speech input to a second electronic device different from the first electronic device-
The data corresponding to the speech input does not include the text string of the communication; And
Receiving from the second electronic device a user intent corresponding to the determined task. The method according to any one of claims 1 to 6,
Storing, on the electronic device, information associated with the communication. The method of claim 19, wherein the stored information associated with the communication includes the portion of the text string. The method of claim 19, wherein the stored information associated with the communication includes the data type. The method of claim 19, wherein the stored information associated with the communication includes metadata associated with the communication. The method of claim 19,
Using the electronic device:
Receiving an input;
Identifying a third task based on the input; And
And performing the third task based on at least a portion of the stored information. The method of claim 19,
Using the electronic device,
Invoking a software application;
And performing the determined task based on at least a portion of the stored information. The method of any one of claims 1 to 6, wherein the electronic device is a first electronic device and the determined task is performed on a second electronic device. As an electronic device,
One or more processors;
Memory; And
One or more programs stored in the memory and configured to be executed by the one or more processors, wherein the one or more programs
Receive a communication comprising a text string;
Determine if a portion of the text string is associated with a predetermined data type of a plurality of data types, the data type having at least one task associated with it;
In accordance with a determination that said portion of said text string is associated with said data type of said plurality of data types:
Determine a task of the at least one task;
Using the display of the electronic device, simultaneously displaying an affordance corresponding to the determined task and an affordance corresponding to a second task, wherein an indication of the affordance corresponding to the determined task is based on the type of the electronic device. Prioritized over an indication of an affordance corresponding to the second task;
Receive a user input associated with an affordance corresponding to the determined task;
In response to receiving the user input associated with an affordance corresponding to the determined task, causing the determined task to be performed based on the portion of the text string;
Cease to cause a task to be performed based on the portion of the text string in accordance with a determination that the portion of the text string is not associated with the data type of the plurality of data types.
And instructions for the electronic device. A non-transitory computer readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of an electronic device, cause the device to:
Receive a communication comprising a text string;
Determine whether a portion of the text string is associated with a predetermined data type of a plurality of data types, the data type having at least one task associated with it;
In accordance with a determination that said portion of said text string is associated with said data type of said plurality of data types:
Determine a task of the at least one task;
Using the display of the electronic device to simultaneously display an affordance corresponding to the determined task and an affordance corresponding to a second task, wherein the indication of the affordance corresponding to the determined task is based on the type of the electronic device Prioritizing an indication of an affordance corresponding to the second task;
Receive user input associated with an affordance corresponding to the determined task;
In response to receiving the user input associated with an affordance corresponding to the determined task, causing the determined task to be performed based on the portion of the text string;
In response to determining that the portion of the text string is not associated with the data type of the plurality of data types, causing the task to cease to be performed based on the portion of the text string.
A non-transitory computer readable storage medium comprising instructions. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images