KR102190856B1 - Identification of voice inputs that provide credentials - Google Patents

Abstract

Systems and processes are provided for identifying a voice input providing one or more user credentials. In one exemplary process, voice input may be received. The first character, the phrase identifying the second character, and the word may be identified based on the voice input. In response to the identification, the first letter, second letter, and word may be converted to text. The text may be displayed in a sequence corresponding to an order of a first character, a second character, and a word in the voice input using the display.

Description

Identification of voice inputs that provide credentials

Cross-reference to related applications

This application claims priority to US Patent Application No. 15/268,360, filed September 16, 2016 and entitled "IDENTIFICATION OF VOICE INPUTS PROVIDING CREDENTIALS", filed March 14, 2016 And claims priority to US Provisional Patent Application No. 62/308,090 entitled “IDENTIFICATION OF VOICE INPUTS PROVIDING CREDENTIALS”. This application claims priority to Danish patent application PA 2017 70128, filed on February 21, 2017 and entitled "IDENTIFICATION OF VOICE INPUTS PROVIDING CREDENTIALS". The content of each of these applications is hereby incorporated by reference in its entirety for all purposes.

Technical field

BACKGROUND OF THE INVENTION The present invention relates generally to speech recognition and, more particularly, to the identification of speech inputs that provide credentials.

Typically, authentication often requires the user to provide credentials such as a user name and password. Authentication may be performed by devices such as computers, tablets, mobile devices, televisions, multimedia devices, set top boxes, and the like. Speech recognition is becoming more popular in dictation and control devices. Credentials may include a mixture of letters, words, phrases, commands, and the like. Such mixed input makes it difficult to accurately recognize user speech. Therefore, it is important to improve the correct input of credentials using speech.

Systems and processes are provided for identifying speech inputs that provide one or more credentials. According to one or more examples, a method includes receiving, at a user device having a display, one or more processors and memory, a voice input. The method also includes identifying, based on the spoken input, the first letter, the phrase identifying the second letter, and the word. The method includes, in response to the identification, converting a first character, a second character, and a word into text; And using the display to display the text in a sequence corresponding to the order of the first character, the second character, and the word in the voice input.

For a better understanding of the various described embodiments, reference should be made to the specific details for carrying out the following invention in connection with the following drawings in which like reference numerals indicate corresponding parts throughout the drawings.
1 is a block diagram illustrating a system and environment for implementing a digital assistant, according to various examples.
2A is a block diagram illustrating a portable multifunction device implementing a client side portion of a digital assistant, in accordance with some embodiments.
2B is a block diagram illustrating example components for event processing according to various examples.
3 shows a portable multifunction device implementing a client side portion of a digital assistant, according to various examples.
4 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, according to various examples.
5A shows an exemplary user interface for a menu of applications on a portable multifunction device, according to various examples.
5B shows an exemplary user interface for a multifunction device with a touch-sensitive surface separate from a display, according to various examples.
6A illustrates a personal electronic device according to various examples.
6B is a block diagram illustrating a personal electronic device according to various examples.
7A is a block diagram illustrating a digital assistant system or a server portion thereof according to various examples.
7B illustrates the functions of the digital assistant shown in FIG. 7A, according to various examples.
7C shows a portion of an ontology according to various examples.
8A and 8B show block diagrams of example architectures for devices according to various examples.
9 shows a block diagram of an exemplary automatic speech recognition module according to various examples.
10A-10F illustrate the function of voice input identification for input of credentials according to various examples.
11A to 11D illustrate a function of voice input identification for input of credentials according to various examples.
12A-12D show a flow diagram of an exemplary process for identifying voice inputs according to various examples.
13 is a block diagram of an electronic device according to various examples.

In the following description of the present invention and embodiments, reference is made to the accompanying drawings, and specific embodiments that may be practiced are shown by way of example in the drawings. It is to be understood that other embodiments and examples may be practiced and changes may be made without departing from the scope of the invention.

Techniques for identification of voice inputs are required. As described herein, techniques for identifying spoken inputs, such as entering credentials that typically include a mixture of letters, words, phrases, and/or commands, are required for a variety of purposes. Such techniques are advantageous by allowing the user to enter credentials without the need for a hand. Additionally, such techniques for identifying spoken inputs can reduce the effort of a user to manually enter credentials or any other information into devices on a textual basis, which can be time consuming and cumbersome.

Although the following description uses terms such as “first”, “second”, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first character may be referred to as a second character, and similarly, a second character may be referred to as a first character, without departing from the scope of the various described examples. The first character and the second character can both be characters, and in some cases, separate and different characters.

Terms used in the description of various examples described herein are for the purpose of describing specific examples only, and are not intended to be limiting. As used in the description of various described examples and in the appended claims, the singular forms ("a", "an", and "the") are intended to include the plural as well, unless the context clearly indicates otherwise. Is intended. Further, it will be understood that the term “and/or” as used herein represents and encompasses any and all possible combinations of one or more of the associated items listed. The terms “include”, “including”, “comprise”, and/or “comprising”, as used herein, refer to the recited features, integers Specifies the presence of elements, steps, actions, elements, and/or components, but the presence of one or more other features, integers, steps, actions, elements, components, and/or groups thereof Or it will be further understood that it does not exclude addition.

The term "if" means "when" or "upon" or "in response to determining" or "in response to determining," depending on the context. It can be interpreted as meaning "in response to detecting." Similarly, the phrase "if it is determined to" or "if [the mentioned condition or event] is detected" means "when determining ..." or "in response to determining ... Or upon detecting an event] or “in response to detecting [the mentioned condition or event]”.

1. System and environment

1 shows a block diagram of a system 100 according to various examples. In some examples, system 100 may implement a digital assistant. The terms "digital assistant", "virtual assistant", "intelligent automation assistant", or "automatic digital assistant" deduce user intention by interpreting natural language input in spoken and/or written form, and inferred user It may refer to any information processing system that performs operations based on intent. For example, to operate according to the inferred user intent, the system may perform one or more of the following: identifying a task flow with steps and parameters designed to fulfill the inferred user intent; Entering specific requirements from the inferred user intent into the task flow; Executing a task flow by invoking programs, methods, services, APIs, etc.; And generating output responses to the user in an audible (eg, speech) and/or visual form.

Specifically, the digital assistant may accept user requests, at least in part, in the form of natural language commands, requests, statements, statements, and/or questions. Typically, the user request may seek either an informative response or the performance of the task by a digital assistant. A satisfactory response to a user request may be providing a requested informational response, performing a requested task, or a combination of both. For example, a user may ask a digital assistant a question such as "Where am I right now?" Based on the user's current location, the digital assistant may respond "You are in Central Park near the west gate". The user may also request the performance of a task, for example "Please invite my friends to my girlfriend's birthday party next week". In response, the digital assistant can acknowledge the request by saying "Yes, right away," and then send an appropriate calendar invitation to each of the user's friends listed in the user's electronic address book. invite). During the performance of the requested task, the digital assistant can sometimes interact with the user in a continuous conversation involving the exchange of multiple information over an extended period of time. There are a number of different ways to interact with the digital assistant to request information or to perform various tasks. In addition to providing verbal responses and taking programmed actions, the digital assistant may also provide other visual or audible forms of responses, such as text, notifications, music, videos, animations, and the like.

As shown in FIG. 1, in some examples, the digital assistant may be implemented according to a client-server model. The digital assistant includes a client-side portion 102 running on the user device 104 (hereinafter “DA client 102”), and a server-side portion 106 running on the server system 108 (hereinafter “DA server”). (106)") may be included. DA client 102 may communicate with DA server 106 through one or more networks 110. DA client 102 may provide client side functions such as user-facing input and output processing, and communication with DA server 106. DA server 106 may provide server-side functions to any number of DA clients 102 each residing on a separate user device 104.

In some examples, DA server 106 includes a client-facing I/O interface 112, one or more processing modules 114, data and models 116, and an I/O interface 118 to external services. It may include. The client-facing I/O interface 112 may facilitate client-facing input and output processing to the DA server 106. One or more processing modules 114 may use data and models 116 to process speech input and determine a user's intent based on natural language input. Further, one or more processing modules 114 perform task execution based on the inferred user intent. In some examples, DA server 106 may communicate with external services 120 via network(s) 110 to complete a task or obtain information. The I/O interface 118 to external services can facilitate such communications.

User device 104 can be any suitable electronic device. For example, a user device may be a portable multifunction device (e.g., device 200, described below with reference to FIG. 2A), a multifunction device (e.g., device 400, described below with reference to FIG. 4), Or a personal electronic device (eg, device 600, described below with reference to FIGS. 6A and 6B). The portable multifunction device may be, for example, a mobile phone, which also includes other functions such as PDA and/or music player functions. Specific examples of portable multifunctional devices may include iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, CA, USA. have. Other examples of portable multifunction devices may include, without limitation, laptop or tablet computers. Also, in some examples, user device 104 may be a non-portable multifunction device. In particular, the user device 104 may be a desktop computer, a game console, a television, or a television set top box. In some examples, user device 104 may include a touch-sensitive surface (eg, touch screen displays and/or touchpads). Further, user device 104 may optionally include one or more other physical user interface devices such as a physical keyboard, mouse, and/or joystick. Various examples of electronic devices, such as multifunctional devices, are described in more detail below.

Examples of communication network(s) 110 may include a local area network (LAN) and a wide area network (WAN), such as the Internet. The communication network(s) 110 is, for example, Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE). ), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or It can be implemented using any known network protocol, including various wired or wireless protocols, such as any other suitable communication protocol.

The server system 108 may be implemented on one or more standalone data processing devices or distributed computer networks. In some examples, the server system 108 provides various virtualization of third-party service providers (e.g., third-party cloud service providers) to provide the basic computing resources and/or infrastructure resources of the server system 108. Devices and/or services may also be employed.

In some examples, user device 104 can communicate with DA server 106 via second user device 122. The second user device 122 may be similar or identical to the user device 104. For example, the second user device 122 may be similar to the devices 200, 400, or 600 described below with reference to FIGS. 2A, 4, 6A, and 6B. The user device 104 may be configured to be communicatively coupled to the second user device 122 via a direct communication connection, such as Bluetooth, NFC, BTLE, or the like, or via a wired or wireless network, such as a local Wi-Fi network. I can. In some examples, the second user device 122 may be configured to act as a proxy between the user device 104 and the DA server 106. For example, the DA client 102 of the user device 104 is configured to transmit information (e.g., a user request received at the user device 104) to the DA server 106 via the second user device 122. Can be. DA server 106 may process the information and return relevant data (eg, data content in response to a user request) to user device 104 via second user device 122.

In some examples, user device 104 may be configured to communicate abbreviated requests for data to second user device 122 to reduce the amount of information transmitted from user device 104. The second user device 122 may be configured to determine supplemental information to add to the shortened request to generate a complete request to be sent to the DA server 106. Such a system architecture advantageously allows a user device 104 (e.g., a watch or similar small electronic device) with limited communication capabilities and/or limited battery power to have greater communication capabilities and/or battery power. 2 By using the user device 122 (eg, a mobile phone, laptop computer, tablet computer, etc.) as a proxy to the DA server 106, it is possible to access the services provided by the DA server 106. Although only two user devices 104, 122 are shown in FIG. 1, system 100 may include any number and types of user devices configured in this proxy configuration to communicate with DA server 106. It should be understood that there is.

Although the digital assistant shown in FIG. 1 may include both a client side portion (e.g., DA client 102) and a server side portion (e.g., DA server 106), in some examples, the functions of the digital assistant It can be implemented as a standalone application installed on the user device. Furthermore, the sharing of functions between the client portion and the server portion of the digital assistant may differ in different implementations. For example, in some examples, the DA client may be a thin-client that provides only user-facing input and output processing functions and delegates all other functions of the digital assistant to the backend server.

2. Electronic device

Attention is now directed to embodiments of electronic devices for implementing the client side portion of the digital assistant. 2A is a block diagram illustrating a portable multifunction device 200 with a touch-sensitive display system 212, in accordance with some embodiments. The touch-sensitive display 212 is sometimes referred to as a “touch screen” for convenience, and is sometimes known or referred to as a “touch-sensitive display system”. The device 200 includes a memory 202 (optionally including one or more computer-readable storage media), a memory controller 222, one or more processing units (CPU) 220, a peripheral interface 218, RF Circuitry 208, audio circuitry 210, speaker 211, microphone 213, input/output (I/O) subsystem 206, other input control devices 216, and external ports 224 Includes. Device 200 optionally includes one or more optical sensors 264. Device 200 optionally includes one or more contact intensity sensors for detecting the intensity of contacts on device 200 (e.g., a touch-sensitive surface, such as touch-sensitive display system 212 of device 200). 265). Device 200 optionally generates tactile outputs on device 200 (e.g., a touch-sensitive surface such as touch-sensitive display system 212 of device 200 or a touchpad of device 400). And one or more tactile output generators 267 for generating tactile outputs on 455. These components optionally communicate via one or more communication buses or signal lines 203.

As used in the specification and claims, the term "strength" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (eg, finger contact) on a touch-sensitive surface, or touch-sensitive Refers to a substitute (proxy) for the force or pressure of contact on a surface. The intensity of the contact has a range of values that includes at least 4 distinct values and more typically includes several hundred (eg, at least 256) distinct values. The strength of the contact is selectively determined (or measured) using various approaches, and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine the estimated force of the contact. Similarly, the pressure sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and/or changes thereto, the capacitance of the touch-sensitive surface near the contact and/or changes thereto, and/or the touch-sensitive surface near the contact. The resistance and/or changes thereto are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, alternate measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (eg, intensity threshold is described in units corresponding to the alternate measurements). In some implementations, alternative measurements for contact force or pressure are converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is It is the pressure threshold measured in units). By using the intensity of the contact as an attribute of the user input, the user would otherwise display affordances (e.g., on a touch-sensitive display) and/or (e.g., touch-sensitive display, touch-sensitive surface, or knob). ) Or via a physical/mechanical control such as a button) to receive user input) access to additional device functions that may not be accessible by the user on a reduced device with a limited footprint.

As used in the specification and claims, the term “tactile output” refers to the physical displacement of the device relative to its previous position, a component of the device (eg, a touch-sensitive surface) relative to another component of the device (eg, a housing). ), or the displacement of the component with respect to the center of mass of the device to be detected by the user using the user's tactile sense. For example, in a situation in which a device or component of the device is in contact with a user's surface that is sensitive to the touch (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement may be It will be interpreted as a tactile sensation corresponding to a perceived change in physical properties of a device or a component of a device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may optionally be directed to the user as a “down click” or “up click” of a physical actuator button. Interpreted by In some cases, the user may feel a tactile sensation such as a “down click” or “up click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that is physically pressed (eg, displaced) by the user's movement You will feel it. As another example, the movement of the touch-sensitive surface is, optionally, interpreted or sensed by the user as the "roughness" of the touch-sensitive surface even when there is no change in the flatness of the touch-sensitive surface. Although such interpretations of touch by the user will be susceptible to the user's individualized sensory perceptions, there are many touch sensory perceptions common to the majority of users. Thus, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "click up", "click down", "roughness"), unless otherwise stated, the generated tactile output is typical (or It corresponds to the physical displacement of the device or its components that will produce the described sensory perception for the user (average).

Device 200 is only an example of a portable multifunction device, and device 200 is, optionally, having more or fewer components than shown, optionally, combining two or more components, or optionally It should be appreciated that the components have a different configuration or arrangement. The various components shown in FIG. 2A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application-specific integrated circuits (ASICs).

Memory 202 may include one or more computer-readable storage media. Computer-readable storage media may be tangible and non-transitory. The memory 202 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. The memory controller 222 can control access to the memory 202 by other components of the device 200.

In some examples, the non-transitory computer-readable storage medium of memory 202 may be an instruction execution system, apparatus, or device, such as a computer-based system, a processor-containing system, or an instruction execution system, apparatus, or Fetches instructions from the device and can be used to store instructions for use by or in connection with another system capable of executing instructions (e.g., to perform aspects of process 1200 described below) . In other examples, instructions (eg, to perform aspects of process 1200 described below) may be stored on a non-transitory computer-readable storage medium (not shown) of server system 108 or memory 202 ) And the non-transitory computer-readable storage medium of the server system 108. In the context of this specification, a "non-transitory computer-readable storage medium" may be any medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The peripherals interface 218 may be used to couple the input and output peripherals of the device to the CPU 220 and memory 202. One or more processors 220 drive or execute various software programs and/or sets of instructions stored in memory 202 to perform various functions for device 200 and process data. In some embodiments, peripherals interface 218, CPU 220, and memory controller 222 may be implemented on a single chip, such as chip 204. In some other embodiments, they may be implemented on separate chips.

The radio frequency (RF) circuit unit 208 receives and transmits RF signals, also referred to as electromagnetic signals. The RF circuitry 208 converts electrical signals to/from electromagnetic signals, and communicates with communication networks and other communication devices via the electromagnetic signals. The RF circuit unit 208 optionally includes an antenna system, an RF transceiver, one or more amplifiers, tuners, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, etc. It includes, but not limited to, well-known circuitry for performing these functions. The RF circuitry 208 optionally includes networks such as the Internet, an intranet, and/or a wireless network, also referred to as the World Wide Web (WWW), such as a cellular telephone network, a wireless local area network (LAN), and /Or a metropolitan area network (MAN), and communicates with other devices by wireless communication. The RF circuitry 208 optionally includes well-known circuitry for detecting a near field communication (NFC) field, such as a short range radio. Wireless communication, optionally, GSM (Global System for Mobile Communications), EDGE (Enhanced Data GSM Environment), HSDPA (high-speed downlink packet access), HSUPA (high-speed uplink packet access), EV-DO (Evolution, Data-Only), HSPA, HSPA+, DC-HSPDA (Dual-Cell HSPA), LTE (long term evolution), NFC (near field communication), W-CDMA (wideband code division multiple access), CDMA (code division multiple access) ), TDMA (time division multiple access), Bluetooth, BTLE (Bluetooth Low Energy), Wi-Fi (Wireless Fidelity) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and/or IEEE 802.11ac ), voice over Internet Protocol (VoIP), Wi-MAX, protocol for email (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP)) ), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or communication protocols not yet developed at the filing date of this document , Including any other suitable communication protocol Use any of a number of communication standards, protocols and technologies, but not limited to these.

The audio circuit unit 210, the speaker 211, and the microphone 213 provide an audio interface between the user and the device 200. The audio circuit unit 210 receives audio data from the peripheral device interface 218, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 211. The speaker 211 converts the electrical signal into sound waves that can be heard by humans. The audio circuit unit 210 also receives an electrical signal converted from sound waves by the microphone 213. The audio circuitry 210 converts the electrical signal into audio data and transmits the audio data to the peripherals interface 218 for processing. Audio data may be retrieved from and/or transmitted to the memory 202 and/or RF circuitry 208 by the peripherals interface 218. In some embodiments, the audio circuitry 210 also includes a headset jack (eg, 312 in FIG. 3 ). The headset jack is between the audio circuitry 210 and removable audio input/output peripherals such as output-only headphones, or a headset having both an output (eg, one or both ear headphones) and an input (eg, a microphone). Provides an interface.

I/O subsystem 206 couples input/output peripherals on device 200 to peripherals interface 218, such as touch screen 212 and other input control devices 216. I/O subsystem 206 optionally includes one or more input controllers for display controller 256, light sensor controller 258, intensity sensor controller 259, haptic feedback controller 261 and other input or control devices. 260). One or more input controllers 260 receive/transmit electrical signals to/from other input control devices 216. Other input control devices 216 optionally include physical buttons (eg, push button, rocker button, etc.), dial, slider switch, joystick, click wheel, and the like. In some alternative embodiments, the input controller(s) 260 is optionally coupled (or not coupled to any) to any of a pointer device such as a keyboard, infrared port, USB port, and mouse. One or more buttons (eg, 308 in FIG. 3) optionally include up/down buttons for controlling the volume of the speaker 211 and/or microphone 213. The one or more buttons optionally include a push button (eg, 306 in FIG. 3).

The quick press of the push button is US Patent Application No. 11/322,549, filed on December 23, 2005 and entitled "Unlocking a Device by Performing Gestures on an Unlock Image", ie, US Patent No. As described in 7,657,849, the touch screen 212 may be unlocked, or the process of using gestures on the touch screen to unlock the device may be initiated, which application is hereby incorporated by reference in its entirety. . A longer press of a push button (eg, 306) may turn the device 200 on or off. The user can customize the function of one or more of the buttons. The touch screen 212 is used to implement virtual or soft buttons and one or more soft keyboards.

The touch-sensitive display 212 provides an input interface and an output interface between the device and the user. Display controller 256 receives and/or transmits electrical signals from/to touch screen 212. The touch screen 212 displays a visual output to the user. The visual output may include graphics, text, icons, video, and any combination thereof (collectively referred to as “graphic”). In some embodiments, some or all of the visual output may correspond to user interface objects.

The touch screen 212 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and/or tactile contact. The touch screen 212 and the display controller 256 (along with any associated modules and/or sets of instructions in the memory 202) contact (and any movement of the contact) on the touch screen 212 or Interrupt) and converts the detected contact into interactions with user interface objects (eg, one or more soft keys, icons, webpages or images) displayed on the touch screen 212. In an exemplary embodiment, the point of contact between the touch screen 212 and the user corresponds to a finger of the user.

The touch screen 212 may use liquid crystal display (LCD) technology, light emitting polymer display (LPD) technology, or light emitting diode (LED) technology, but other display technologies may be used in other embodiments. . The touch screen 212 and display controller 256 use capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements to determine one or more points of contact with the touch screen 212. Contact and any movement or interruption thereof can be detected using any of a plurality of touch-sensing technologies, including but not limited to, currently known or to be developed in the future. In an exemplary embodiment, a projected mutual capacitance sensing technology such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, CA is used.

The touch-sensitive display in some embodiments of the touch screen 212 includes the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and/or 6,677,932 (Westerman), and/or Or similar to the multi-touch sensitive touchpads described in US Patent Publication No. 2002/0015024A1, each of which is incorporated herein by reference in its entirety. However, touch screen 212 displays visual output from device 200, while touch-sensitive touchpads do not provide visual output.

The touch-sensitive display in some embodiments of the touch screen 212 may be as described in the following applications: (1) Filed May 2, 2006 and entitled "Multipoint Touch Surface Controller" US patent application Ser. No. 11/381,313; (2) US Patent Application Ser. No. 10/840,862 filed May 6, 2004 and entitled "Multipoint Touchscreen"; (3) US patent application Ser. No. 10/903,964 filed July 30, 2004 and entitled "Gestures For Touch Sensitive Input Devices"; (4) US patent application Ser. No. 11/048,264, filed Jan. 31, 2005 and entitled “Gestures For Touch Sensitive Input Devices”; (5) U.S. Patent Application No. 11/038,590, filed January 18, 2005 and entitled "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices"; (6) U.S. Patent Application No. 11/228,758, filed September 16, 2005 and entitled "Virtual Input Device Placement On A Touch Screen User Interface"; (7) US patent application Ser. No. 11/228,700, filed September 16, 2005 and entitled "Operation Of A Computer With A Touch Screen Interface"; (8) U.S. Patent Application No. 11/228,737, filed September 16, 2005 and entitled "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard"; And (9) US Patent Application No. 11/367,749, filed on March 3, 2006 and entitled "Multi-Functional Hand-Held Device". All of these applications are incorporated herein by reference in their entirety.

The touch screen 212 may have a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user may contact the touch screen 212 using any suitable object or accessory such as a stylus, finger, or the like. In some embodiments, the user interface is primarily designed to operate using finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of the finger on the touch screen. In some embodiments, the device converts the coarse finger-based input into a precise pointer/cursor position or command to perform the user's desired actions.

In some embodiments, in addition to the touch screen, the device 200 may include a touchpad (not shown) to activate or deactivate certain functions. In some embodiments, the touchpad, unlike a touch screen, is a touch sensitive area of the device that does not display a visual output. The touch pad may be a touch-sensitive surface separate from the touch screen 212 or an extension of the touch-sensitive surface formed by the touch screen.

Device 200 also includes a power system 262 for supplying power to various components. Power system 262 is a power management system, one or more power sources (e.g., battery, alternating current (AC)), recharge system, power failure detection circuit, power converter or inverter, power status indicator (e.g., luminous Diodes (LEDs)), and any other components associated with the generation, management, and distribution of power within portable devices.

Device 200 may also include one or more optical sensors 264. 2A shows an optical sensor coupled to the optical sensor controller 258 in the I/O subsystem 206. The optical sensor 264 may include a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. The optical sensor 264 receives light from the surrounding environment, projected through one or more lenses, and converts the light into data representing an image. In conjunction with the imaging module 243 (also referred to as a camera module), the optical sensor 264 can capture still images or video. In some embodiments, the optical sensor is located on the back of the device 200 opposite the touch screen display 212 on the front of the device, such that the touch screen display is used as a viewfinder for still and/or video image acquisition. To be used. In some embodiments, an optical sensor is located on the front of the device, allowing the user's image to be acquired for the video conference while the user is viewing other video conference participants on the touch screen display. In some embodiments, the position of the light sensor 264 is changed by the user (e.g., by rotating the lens and sensor within the device housing) so that a single light sensor 264 can be used for video conferencing and pause and /Or can be used for both video image acquisition.

Device 200 also optionally includes one or more contact intensity sensors 265. 2A shows a contact intensity sensor coupled to intensity sensor controller 259 in I/O subsystem 206. Contact intensity sensor 265 may optionally be one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., Sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). The contact intensity sensor 265 receives contact intensity information (eg, pressure information or a substitute for pressure information) from the surrounding environment. In some embodiments, at least one contact intensity sensor is collocated with or proximate to the touch-sensitive surface (eg, touch-sensitive display system 212). In some embodiments, at least one contact intensity sensor is located on the back of the device 200 opposite to the touch screen display 212 located on the front of the device 200.

Device 200 may also include one or more proximity sensors 266. 2A shows the proximity sensor 266 coupled to the peripherals interface 218. Alternatively, proximity sensor 266 may be coupled to input controller 260 in I/O subsystem 206. Proximity sensor 266 is described in US patent application Ser. No. 11/241,839 entitled "Proximity Detector In Handheld Device"; No. 11/240,788 entitled "Proximity Detector In Handheld Device"; No. 11/620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11/586,862 entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; And 11/638,251 entitled "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated herein by reference in their entirety. In some embodiments, the proximity sensor turns off and disables the touch screen 212 when the multifunction device is positioned near the user's ear (eg, when the user is making a phone call).

Device 200 also optionally includes one or more tactile output generators 267. 2A shows a tactile output generator coupled to a haptic feedback controller 261 in an I/O subsystem 206. The tactile output generator 267 may optionally generate one or more electroacoustic devices such as speakers or other audio components and/or a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output. It includes electromechanical devices that convert energy into linear motion, such as a component (eg, a component that converts electrical signals into tactile outputs on the device). The contact intensity sensor 265 receives tactile feedback generation instructions from the haptic feedback module 233 to generate tactile outputs on the device 200 that can be sensed by the user of the device 200. In some embodiments, the at least one tactile output generator is located with or proximate a touch-sensitive surface (e.g., touch-sensitive display system 212), and optionally, vertically (e.g., , Creating a tactile output by moving in/out of the surface of the device 200) or laterally (eg, back and forth in the same plane as the surface of the device 200). In some embodiments, at least one tactile output generator sensor is located on the back of the device 200 opposite to the touch screen display 212 located on the front of the device 200.

Device 200 may also include one or more accelerometers 268. 2A shows the accelerometer 268 coupled to the peripherals interface 218. Alternatively, the accelerometer 268 may be coupled to the input controller 260 in the I/O subsystem 206. The accelerometer 268 is described in US Patent Publication No. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices" and US Patent Publication No. 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer". May operate as described, both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on analysis of data received from one or more accelerometers. The device 200 optionally includes a magnetometer (not shown), in addition to the accelerometer(s) 268, and a GPS for acquiring information about the position and orientation (e.g., vertical or horizontal) of the device 200. Global Positioning System) (or GLONASS or other global navigation system) receiver (not shown).

In some embodiments, software components stored in memory 202 include operating system 226, communication module (or set of instructions) 228, contact/motion module (or set of instructions) 230, graphics module ( Or a set of instructions) 232, a text input module (or set of instructions) 234, a GPS module (or set of instructions) 235, a digital assistant client module 229, and applications (or a set of instructions). S) (236). Further, the memory 202 may store data and models, such as user data and models 231. In addition, in some embodiments, the memory (202 in FIG. 2A or 470 in FIG. 4) stores the device/global internal state 257 as shown in FIGS. 2A and 4. The device/global internal state 257 includes: an active application state indicating which applications are currently active, if present; A display state indicating which applications, views, or other information occupy various areas of the touch screen display 212; Sensor status, including information obtained from the input control devices 216 and various sensors of the device; And one or more of location information regarding a location and/or posture of the device.

Operating system 226 (e.g., an embedded operating system such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or VxWorks) includes general system tasks (e.g., memory management, storage device control, power Management, etc.) to control and manage various software components and/or drivers, and facilitates communication between various hardware and software components.

The communication module 228 facilitates communication with other devices through one or more external ports 224, and also provides various methods for processing data received by the RF circuitry 208 and/or external ports 224. Contains software components. External port 224 (e.g., Universal Serial Bus (USB), FireWire, etc.) to couple directly to other devices or indirectly through a network (e.g., Internet, wireless LAN, etc.) Is composed. In some embodiments, the external port is a multi-pin (eg, 30-pin) connector that is the same or similar and/or compatible with the 30-pin connector used in iPod® (trademark of Apple Inc.) devices.

The contact/motion module 230 optionally detects contact with the touch screen 212 (in conjunction with the display controller 256) and other touch sensitive devices (eg, a touchpad or a physical click wheel). The contact/motion module 230 determines whether a contact has occurred (e.g., detecting a finger-down event), the strength of the contact (e.g., force or pressure of the contact, or Determining a force or a substitute for pressure), determining if there is movement of a contact, and tracking movement across a touch-sensitive surface (e.g., detecting one or more finger-dragging events) A), and various software components for performing various operations related to the detection of a contact, such as determining whether a contact has been stopped (e.g., detecting a finger-up event or a contact break). Includes them. The contact/motion module 230 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by a series of contact data, optionally, determines the speed (magnitude), velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the contact point Includes doing. These actions, optionally, apply to single contacts (eg, one finger contacts) or to multiple simultaneous contacts (eg “multi-touch”/multiple finger contacts). In some embodiments, the contact/motion module 230 and display controller 256 detect a contact on the touchpad.

In some embodiments, the contact/motion module 230 generates a set of one or more intensity thresholds to determine whether an action was performed by a user (eg, to determine whether the user “clicked” an icon). Use. In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators, it is possible to change the physical hardware of the device 200). Can be adjusted without). For example, the mouse “click” threshold of a trackpad or touch screen display can be set to any of a wide range of predefined threshold values without changing the trackpad or touch screen display hardware. Additionally, in some implementations, the user of the device may have one or more of a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and/or by adjusting multiple intensity thresholds at once with a system level click “intensity” parameter). Software settings are provided to adjust the intensity threshold.

The contact/motion module 230 optionally detects a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different motions, timings, and/or intensities of detected contacts). Thus, the gesture is detected by selectively detecting a specific contact pattern. For example, detecting a finger tap gesture can result in the detection of a finger-down event and then a finger-up at the same location (or substantially the same location) as the finger-down event (e.g., the location of the icon). It involves detecting (liftoff) events. As another example, detecting a finger swipe gesture on a touch-sensitive surface detects a finger-down event followed by one or more finger-drag events, followed by a finger-up (liftoff). ) Including detecting the event.

Graphics module 232, including components for changing the visual effect (e.g., brightness, transparency, saturation, contrast, or other visual property) of the displayed graphic, rendering and It includes various known software components for display. As used herein, the term “graphic” includes, without limitation, text, web pages, icons (eg, user interface objects including soft keys), digital images, videos, animations, etc. , Includes any object that can be displayed to the user.

In some embodiments, the graphics module 232 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphic module 232 receives one or more codes specifying a graphic to be displayed together with coordinate data and other graphic property data, if necessary, from applications, etc., and then generates screen image data and outputs it to the display controller 256 do.

The haptic feedback module 233 is an instruction used by the tactile output generator(s) 267 to generate tactile outputs at one or more locations on the device 200 in response to user interactions with the device 200. It contains various software components for creating them.

The text input module 234, which may be a component of the graphics module 232, can be used for various applications (e.g., contacts 237, email 240, IM 241, browser 247, and text input). Soft keyboards for entering text in any other application).

The GPS module 235 determines the location of the device and uses this information for use in various applications (e.g., to the phone 238 for use in location based dialing; the camera 243 as photo/video metadata). And to applications that provide location-based services such as weather widgets, local yellow page widgets, and map/navigation widgets.

The digital assistant client module 229 may include various client-side digital assistant instructions to provide the client-side functions of the digital assistant. For example, the digital assistant client module 229 includes various user interfaces of the portable multifunction device 200 (e.g., a microphone 213, accelerometer(s) 268), a touch-sensitive display system 212, an optical sensor. Voice input (eg, speech input), text input, touch input, and/or gesture input through (s) 229, other input control devices 216, etc. The digital assistant client module 229 also includes various output interfaces of the portable multifunction device 200 (e.g., speaker 211, touch-sensitive display system 212, tactile output generator(s) 267, etc.). Through may provide output of audible (eg, speech output), visual, and/or tactile forms. For example, the output may be provided as voice, sound, warning, text message, menu, graphic, video, animation, vibration, and/or combinations of two or more of these. During operation, digital assistant client module 229 may communicate with DA server 106 using RF circuitry 208.

User data and models 231 are from various data associated with the user (e.g., user-specific vocabulary data, user preference data, user-specific name pronunciations, user's electronic address book to provide client-side functions of the digital assistant). Data, to-do lists, shopping lists, etc.). In addition, the user data and models 231 are various models (e.g., speech recognition models, statistical language models, natural language processing models, ontology, task flow models) to process user input and determine user intent. , Service models, etc.).

In some examples, the digital assistant client module 229 may be configured with various sensors, subsystems, and peripheral devices of the portable multifunction device 200 to establish a context associated with the user, current user interaction, and/or current user input. Additional information may be collected from the surrounding environment of the portable multifunctional device 200 by using them. In some examples, the digital assistant client module 229 can provide context information, or a subset thereof, along with user input to the DA server 106 to help infer the user's intent. In some examples, the digital assistant can also use contextual information to determine how to prepare and deliver outputs to the user. The context information may be referred to as context data.

In some examples, context information accompanying the user input may include sensor information, such as lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, and the like. In some examples, the context information may also include a physical state of the device, such as device orientation, device location, device temperature, power level, speed, acceleration, motion patterns, cellular signal strength, and the like. In some examples, information related to the software state of DA server 106, such as running processes of portable multifunction device 200, installed programs, past and present network activities, background services, error logs, resources. Usage or the like may be provided to the DA server 106 as context information associated with a user input.

In some examples, digital assistant client module 229 may selectively provide information stored on portable multifunction device 200 (eg, user data 231) in response to requests from DA server 106. . In some examples, digital assistant client module 229 can also derive additional input from the user via natural language conversations or other user interfaces upon request by DA server 106. Digital assistant client module 229 may pass additional inputs to DA server 106 to assist DA server 106 in intent inference and/or fulfillment of the user's intention expressed in the user request.

A more detailed description of the digital assistant will be described later with reference to FIGS. 7A to 7C. It should be appreciated that the digital assistant client module 229 may include any number of submodules of the digital assistant module 726 described below.

Applications 236 may include the following modules (or sets of instructions), or a subset or superset thereof:

연락처 모듈(237)(때때로 주소록 또는 연락처 목록으로 지칭됨);

Contacts module 237 (sometimes referred to as an address book or contact list);

전화 모듈(238);

Telephone module 238;

화상 회의 모듈(239);

Video conferencing module 239;

이메일 클라이언트 모듈(240);

Email client module 240;

인스턴트 메시징(IM) 모듈(241);

Instant messaging (IM) module 241;

운동 지원 모듈(242);

Exercise support module 242;

정지 및/또는 비디오 이미지들을 위한 카메라 모듈(243);

A camera module 243 for still and/or video images;

이미지 관리 모듈(244);

Image management module 244;

비디오 재생기 모듈;

Video player module;

음악 재생기 모듈;

Music player module;

브라우저 모듈(247);

Browser module 247;

캘린더 모듈(248);

A calendar module 248;

날씨 위젯(249-1), 주식 위젯(249-2), 계산기 위젯(249-3), 알람 시계 위젯(249-4), 사전 위젯(249-5), 및 사용자에 의해 획득되는 다른 위젯들뿐 아니라 사용자-생성 위젯들(249-6) 중 하나 이상을 포함할 수 있는 위젯 모듈들(249);

Weather widget 249-1, stock widget 249-2, calculator widget 249-3, alarm clock widget 249-4, dictionary widget 249-5, and other widgets acquired by the user As well as widget modules 249, which may include one or more of user-generated widgets 249-6;

사용자-생성 위젯들(249-6)을 만들기 위한 위젯 생성기 모듈(250);

A widget generator module 250 for making user-generated widgets 249-6;

검색 모듈(251);

Search module 251;

비디오 재생기 모듈 및 음악 재생기 모듈을 통합하는 비디오 및 음악 재생기 모듈(252);

A video and music player module 252 incorporating a video player module and a music player module;

메모 모듈(253);

Memo module 253;

지도 모듈(254); 및/또는

Map module 254; And/or

온라인 비디오 모듈(255).

Online video module (255).

Examples of other applications 236 that may be stored in memory 202 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management. , Speech recognition, and speech replication.

Along with the touch screen 212, the display controller 256, the contact/motion module 230, the graphics module 232 and the text input module 234, the contact module 237 is an address book or contact list, including: It may be used to manage (eg, stored in memory 202 or in application internal state 292 of contacts module 237 in memory 470): adding name(s) to the address book; Deleting name(s) from the address book; Associating phone number(s), email address(s), physical address(s), or other information with a name; Associating an image with a name; Sorting and sorting names; Providing phone numbers or email addresses to initiate and/or facilitate communication by phone 238, video conferencing module 239, email 240, or IM 241, and the like.

RF circuit part 208, audio circuit part 210, speaker 211, microphone 213, touch screen 212, display controller 256, contact/motion module 230, graphic module 232, and text Together with the input module 234, the phone module 238 inputs a sequence of characters corresponding to the phone number, accesses one or more phone numbers in the contact module 237, modifies the entered phone number, and It can be used to dial the phone number of, have a conversation, and disconnect or disconnect when the conversation is complete. As noted above, wireless communication may utilize any of a number of communication standards, protocols and technologies.

RF circuit unit 208, audio circuit unit 210, speaker 211, microphone 213, touch screen 212, display controller 256, optical sensor 264, optical sensor controller 258, contact/motion Along with module 230, graphics module 232, text input module 234, contact module 237, and phone module 238, the video conferencing module 239 can be used with one or more other Contains executable instructions for initiating, conducting and ending a video conference between participants.

Along with RF circuitry 208, touch screen 212, display controller 256, contact/motion module 230, graphics module 232 and text input module 234, email client module 240 is user-instructed It contains executable instructions for creating, sending, receiving, and managing emails in response to emails. In conjunction with the image management module 244, the email client module 240 makes it very easy to create and send emails with still or video images taken with the camera module 243.

With the RF circuitry 208, the touch screen 212, the display controller 256, the contact/motion module 230, the graphics module 232 and the text input module 234, the instant messaging module 241 Enter a sequence of characters corresponding to a message, modify previously entered characters (e.g., Short Message Service (SMS) or Multimedia Message Service for phone-based instant messages) Includes executable instructions to send each instant message, receive instant messages, and view received instant messages (using the MMS) protocol or using XMPP, SIMPLE or IMPS for Internet-based instant messages. . In some embodiments, instant messages sent and/or received are graphics, photos, audio files, video files, and/or as supported by MMS and/or Enhanced Messaging Service (EMS). Other attachments may be included. As used herein, "instant messaging" refers to phone-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, messages sent using XMPP, SIMPLE or IMPS). S) refers to both.

RF circuit unit 208, touch screen 212, display controller 256, contact/motion module 230, graphic module 232, text input module 234, GPS module 235, map module 254 , And with the music player module, the exercise support module 242 designs exercises (eg, with time, distance, and/or calorie expenditure goals); Communicates with athletic sensors (sports devices); Receive motion sensor data; Calibrate sensors used to monitor exercise; Select and play music for exercise; Contains executable instructions to display, store and transmit athletic data.

With touch screen 212, display controller 256, light sensor(s) 264, light sensor controller 258, contact/motion module 230, graphics module 232 and image management module 244 , The camera module 243 captures still images or video (including a video stream) and stores them in the memory 202, modifies the properties of the still image or video, or Contains executable instructions that cause the video to be deleted.

With the touch screen 212, display controller 256, contact/motion module 230, graphics module 232, text input module 234 and camera module 243, the image management module 244 is /Or include executable instructions to arrange, modify (e.g., edit) video images, or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and save. .

Along with the RF circuit unit 208, the touch screen 212, the display controller 256, the contact/motion module 230, the graphic module 232 and the text input module 234, the browser module 247 is a web page Executable instructions to browse the Internet according to user instructions, including retrieving, linking to, receiving, and displaying attachments and other files linked to web pages, as well as files or portions thereof. Includes them.

RF circuitry 208, touch screen 212, display controller 256, contact/motion module 230, graphics module 232, text input module 234, email client module 240, and browser module ( In conjunction with 247, the calendar module 248 is an executable that allows the creation, display, modification, and storage of calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.) according to user instructions. Contains instructions.

Widget modules, along with RF circuitry 208, touch screen 212, display controller 256, contact/motion module 230, graphics module 232, text input module 234 and browser module 247 249 can be downloaded and used by the user (e.g., weather widget 249-1, stock widget 249-2, calculator widget 249-3, alarm clock widget 249-4) and dictionary widgets. (249-5)), or mini-applications that can be created by the user (eg, user-generated widget 249-6). In some embodiments, the widget includes a Hypertext Markup Language (HTML) file, a Cascading Style Sheets (CSS) file, and a JavaScript file. In some embodiments, the widget includes an Extensible Markup Language (XML) file and a JavaScript file (eg, Yahoo! widgets).

Along with RF circuitry 208, touch screen 212, display controller 256, contact/motion module 230, graphics module 232, text input module 234 and browser module 247, a widget generator module 250 may be used to create widgets by a user (eg, change a user-specific portion of a web page into a widget).

Along with the touch screen 212, the display controller 256, the contact/motion module 230, the graphics module 232 and the text input module 234, the search module 251 provides one or more search criteria according to user instructions. Contains executable instructions to search for text, music, sound, image, video, and/or other files in memory 202 that match (eg, one or more user-specified search terms).

With touch screen 212, display controller 256, contact/motion module 230, graphics module 232, audio circuitry 210, speaker 211, RF circuitry 208 and browser module 247 , Video and music player module 252, executable instructions that allow a user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, and videos (e.g. , On the touch screen 212 or on an externally connected display through an external port 224), to display, screen, or otherwise play. In some embodiments, device 200 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

Along with the touch screen 212, the display controller 256, the contact/motion module 230, the graphic module 232, and the text input module 234, the memo module 253 can take notes and do notes according to user instructions. Contains executable instructions that allow you to create and manage to-do lists and the like.

RF circuit unit 208, touch screen 212, display controller 256, contact/motion module 230, graphic module 232, text input module 234, GPS module 235, and browser module 247 ), the map module 254 may provide maps and data associated with the maps according to user instructions (e.g., driving directions, data about stores and other points of interest at or near a specific location, and Other location-based data), can be used to receive, display, modify, and store.

Touch screen 212, display controller 256, contact/motion module 230, graphic module 232, audio circuit part 210, speaker 211, RF circuit part 208, text input module 234, Along with email client module 240 and browser module 247, online video module 255 allows a user to access online videos in one or more file formats such as H.264, browse them, (e.g., stream and / Or download), play (e.g., on a touch screen or on an externally connected display via an external port 224), send an email with a link to a specific online video, and otherwise manage Contains instructions. In some embodiments, instant messaging module 241 rather than email client module 240 is used to send a link to a particular online video. For further description of online video applications, see US Provisional Patent Application No. 60/936,562, filed June 20, 2007, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos" and December 31, 2007. It can be found in US Patent Application No. 11/968,067 filed on date, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos", the contents of which are hereby incorporated by reference in their entirety. .

Each of the above-identified modules and applications is executable for performing one or more functions described above and methods described herein (e.g., computer-implemented methods and other information processing methods described herein). Corresponds to a set of instructions. These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures or modules, so various subsets of these modules may be combined or otherwise rearranged in various embodiments. . For example, the video player module may be combined with a music player module to form a single module (eg, video and music player module 252 in FIG. 2A). In some embodiments, memory 202 may store a subset of the modules and data structures identified above. Also, the memory 202 may store additional modules and data structures not described above.

In some embodiments, device 200 is a device in which operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad. By using the touch screen and/or touchpad as the primary input control device for operation of the device 200, the number of physical input control devices (such as push buttons, dials, etc.) on the device 200 can be reduced.

The predefined set of functions performed entirely through the touch screen and/or the touchpad, optionally, includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 200 from any user interface displayed on the device 200 to a main, home or root menu. In these embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.

2B is a block diagram illustrating example components for event processing, in accordance with some embodiments. In some embodiments, the memory (202 in FIG. 2A or 470 in FIG. 4) is an event classifier 270 (e.g., in operating system 226) and a respective application 236-1 (e.g., application described above). S 237-251, 255, 480-490).

The event classifier 270 receives event information and determines the application 236-1 to which the event information is to be delivered, and the application view 291 of the application 236-1. The event classifier 270 includes an event monitor 271 and an event dispatcher module 274. In some embodiments, the application 236-1 includes an application internal state 292 that indicates the current application view(s) displayed on the touch-sensitive display 212 when the application is active or running. In some embodiments, device/global internal state 257 is used by event classifier 270 to determine which application(s) are currently active, and application internal state 292 is used by event classifier 270. It is used to determine application views 291 to deliver event information by.

In some embodiments, the application internal state 292 is the user indicating resume information to be used when the application 236-1 resumes execution, information being displayed or ready to be displayed by the application 236-1. Interface state information, such as one or more of a state queue to allow the user to return to the previous state or view of the application 236-1, and a redo/undo queue of previous actions taken by the user. Includes additional information.

The event monitor 271 receives event information from the peripheral device interface 218. The event information includes information about a sub-event (eg, a user touch on the touch-sensitive display 212 as part of a multi-touch gesture). Peripheral interface 218 is received from I/O subsystem 206 or a sensor such as proximity sensor 266, accelerometer(s) 268, and/or microphone 213 (via audio circuitry 210). Information is transmitted. The information that the peripherals interface 218 receives from the I/O subsystem 206 includes information from the touch-sensitive display 212 or touch-sensitive surface.

In some embodiments, event monitor 271 sends requests to peripherals interface 218 at predetermined intervals. In response to this, the peripheral device interface 218 transmits event information. In other embodiments, the peripherals interface 218 transmits event information only when there is a significant event (e.g., receiving an input exceeding a predetermined noise threshold and/or receiving an input over a predetermined duration). .

In some embodiments, event classifier 270 also includes a hit view determination module 272 and/or an active event recognizer determination module 273.

The hit view determination module 272 provides software procedures to determine where a subevent has occurred within one or more views when the touch-sensitive display 212 displays more than one view. Views are made up of controls and other elements that the user can see on the display.

Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, where information is displayed and touch-based gestures occur. Application views (of each application) for which a touch is detected may correspond to a program of the application or program levels within the view hierarchy. For example, the lowest level view in which a touch is detected may be referred to as a hit view, and the set of events recognized as appropriate inputs may be determined based at least in part on the hit view of the initial touch initiating the touch-based gesture.

The hit view determination module 272 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, the hit view determination module 272 identifies the hit view as the lowest-level view in the hierarchy that should handle sub-events. In most situations, the hit view is the lowest level view in which the subevent that is initiated (eg, the first subevent in the sequence of subevents forming the event or potential event) occurs. Once the hit view is identified by the hit view determination module 272, the hit view typically receives all sub-events related to the same touch or input source that caused it to be identified as a hit view.

Active event recognizer determination module 273 determines which views or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 273 determines that only the hit view should receive a particular sequence of subevents. In other embodiments, the active event recognizer determination module 273 determines that all views including the physical location of the sub-event are actively involved views, so that all actively involved views are It decides that the sequence should be received. In other embodiments, even if touch sub-events were entirely confined to an area associated with one particular view, higher views in the hierarchy would still remain as actively engaged views.

The event dispatcher module 274 delivers event information to an event recognizer (eg, event recognizer 280). In embodiments including an active event recognizer determination module 273, the event dispatcher module 274 communicates event information to the event recognizer determined by the active event recognizer determination module 273. In some embodiments, event dispatcher module 274 stores event information in an event queue, which event information is retrieved by respective event receivers 282.

In some embodiments, operating system 226 includes event classifier 270. Alternatively, application 236-1 includes an event classifier 270. In still other embodiments, event classifier 270 is a standalone module, or part of another module that is stored in memory 202, such as contact/motion module 230.

In some embodiments, application 236-1 includes a plurality of event handlers 290 and one or more application views 291, each of which is a touch event occurring within each view of the application's user interface. Contains instructions for processing them. Each application view 291 of application 236-1 includes one or more event recognizers 280. Typically, each application view 291 includes a plurality of event recognizers 280. In other embodiments, one or more of the event recognizers 280 may be part of a separate module, such as a user interface kit (not shown) or a higher level object for which the application 236-1 inherits methods and other properties. to be. In some embodiments, each event handler 290 is one of the data updater 276, the object updater 277, the GUI updater 278, and/or the event data 279 received from the event classifier 270. It includes the above. The event handler 290 may update the application internal state 292 by using or calling the data updater 276, the object updater 277, or the GUI updater 278. Alternatively, one or more of the application views 291 include one or more respective event handlers 290. Further, in some embodiments, one or more of the data updater 276, the object updater 277, and the GUI updater 278 are included within the respective application view 291.

Each event recognizer 280 receives event information (eg, event data 279) from event classifier 270 and identifies an event from the event information. The event recognizer 280 includes an event receiver 282 and an event comparator 284. In some embodiments, event recognizer 280 also includes at least metadata 283 and a subset of event delivery instructions 288 (which may include sub-event delivery instructions).

The event receiver 282 receives event information from the event classifier 270. The event information includes information about a sub-event, for example, a touch or a touch movement. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event relates to the motion of the touch, the event information may also include the speed and direction of the sub-event. In some embodiments, events include rotation of the device from one orientation to another (e.g., from portrait to landscape orientation, or vice versa), and the event information is the current orientation of the device (also referred to as device orientation). It contains corresponding information about).

The event comparator 284 compares the event information with predefined event or sub-event definitions, and based on the comparison, determines an event or sub-event, or determines or updates a state of an event or sub-event. In some embodiments, event comparator 284 includes event definitions 286. The event definitions 286 include definitions of events (eg, predefined sequences of sub-events), eg, event 1 287-1, event 2 287-2, and the like. In some embodiments, sub-events within event 287 include, for example, touch start, touch end, touch move, touch cancel, and multiple touches. In one example, the definition for event 1 287-1 is a double tap on the displayed object. The double tap is, for example, a first touch on a displayed object during a predetermined phase (touch start), a first liftoff during a predetermined phase (touch end), the displayed object during a predetermined phase. A second touch on the image (touch start), and a second liftoff during a predetermined phase (touch end). In another example, the definition for event 2 287-2 is dragging on the displayed object. Dragging includes, for example, a touch (or touch) on a displayed object during a predetermined phase, movement of the touch across the touch-sensitive display 212, and lift-off of the touch (touch termination). In some embodiments, the event also includes information about one or more associated event handlers 290.

In some embodiments, event definition 287 includes a definition of an event for a respective user interface object. In some embodiments, event comparator 284 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on the touch-sensitive display 212, when a touch is detected on the touch-sensitive display 212, the event comparator 284 is the three user interface objects. A hit test is performed to determine which one is associated with the touch (sub-event). If each displayed object is associated with a separate event handler 290, the event comparator uses the results of the hit test to determine which event handler 290 should be activated. For example, the event comparator 284 selects the object triggering the hit test and the event handler associated with the sub-event.

In some embodiments, the definition for individual event 287 also includes delayed actions that delay delivery of event information until after it is determined whether the sequence of sub-events corresponds to or does not correspond to the event type of the event recognizer.

If the individual event recognizer 280 determines that the series of sub-events do not match any of the events in the event definitions 286, the individual event recognizer 280 is in an event impossible, event failure, or event end state. Upon entry, the individual event recognizer then ignores subsequent sub-events of the touch-based gesture. In this situation, other event recognizers that remain active for the hit view, if any, continue to track and process sub-events of the ongoing touch-based gesture.

In some embodiments, individual event recognizer 280 has configurable attributes, flags, and/or lists that indicate how the event delivery system should perform sub-event delivery to actively engaged event recognizers. Includes metadata 283. In some embodiments, metadata 283 includes configurable properties, flags, and/or lists that indicate how event recognizers can interact with each other, or may be allowed to interact with each other. In some embodiments, metadata 283 includes configurable attributes, flags, and/or lists that indicate whether sub-events are delivered to various levels in the view or program hierarchy.

In some embodiments, each event recognizer 280 activates an event handler 290 associated with the event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognizer 280 communicates event information associated with the event to the event handler 290. Activating the event handler 290 is separate from sending (and delaying sending) sub-events to the respective hit view. In some embodiments, event recognizer 280 sends a flag associated with the recognized event, and event handler 290 associated with the flag catches the flag and performs a predefined process.

In some embodiments, the event delivery instructions 288 include subevent delivery instructions that deliver event information about the subevent without activating the event handler. Instead, subevent delivery instructions deliver event information to event handlers associated with a series of subevents or to actively participating views. Event handlers associated with a series of sub-events or actively participating views receive event information and perform a predetermined process.

In some embodiments, data updater 276 creates and updates data used by application 236-1. For example, the data updater 276 updates the phone number used in the contact module 237 or stores a video file used in the video player module. In some embodiments, object updater 277 creates and updates objects used in application 236-1. For example, the object updater 277 creates a new user interface object or updates the location of the user interface object. The GUI updater 278 updates the GUI. For example, GUI updater 278 prepares display information and sends it to graphics module 232 for display on a touch-sensitive display.

In some embodiments, the event handler(s) 290 includes or accesses a data updater 276, an object updater 277, and a GUI updater 278. In some embodiments, data updater 276, object updater 277, and GUI updater 278 are contained within a single module of each application 236-1 or application view 291. In other embodiments, they are contained within two or more software modules.

The above discussion regarding event handling of user touches on touch-sensitive displays also applies to other types of user inputs for operating multifunction devices 200 with input devices, but all of which are initiated on touch screens. It should be understood that it is not possible. For example, mouse movement and mouse button presses selectively coordinated with single or multiple keyboard presses or hold; Contact movements such as tap, drag, scroll, etc. on the touch pad; Pen stylus inputs; Device movement; Verbal instructions; Detected eye movements; Biometric inputs; And/or any combination thereof are optionally used as inputs corresponding to subevents defining the event to be recognized.

3 shows a portable multifunction device 200 having a touch screen 212 in accordance with some embodiments. The touch screen, optionally, displays one or more graphics within the user interface (UI) 300. In this embodiment as well as other embodiments described below, the user may, for example, have one or more fingers 302 (not drawn to scale in the drawings) or one or more stylus 303 (not drawn to scale in the drawings). ) To make it possible to select one or more of the graphics by making a gesture on the graphic. In some embodiments, the selection of one or more graphics occurs when the user stops contacting the one or more graphics. In some embodiments, the gesture may optionally be one or more taps of a finger in contact with the device 200, one or more swipes (left to right, right to left, up and/or down), and/or (right). To left, left to right, up and/or down). In some implementations or situations, if the graphic is inadvertently contacted, the graphic is not selected. For example, when the gesture corresponding to the selection is a tap, the swipe gesture of sweeping over the application icon selectively does not select the corresponding application.

Device 200 may also include one or more physical buttons, such as “home” or menu button 304. As previously described, the menu button 304 can be used to navigate to any application 236 in the set of applications that can run on the device 200. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 212.

In one embodiment, the device 200 includes a touch screen 212, a menu button 304, a push button 306 for turning on/off the device and locking the device, a volume control button(s) 308, It includes a subscriber identification module (SIM) card slot 310, a headset jack 312, and an external docking/charging port 224. The push button 306 optionally powers on/off the device by pressing the button and holding the button pressed for a predefined time interval; It is used to lock/lock the device, unlock the device or initiate the unlock process by pressing the button and releasing the button before a predefined time interval has elapsed. In an alternative embodiment, device 200 also receives verbal input for activation or deactivation of some functions via microphone 213. Device 200 also optionally generates tactile outputs for a user of device 200 and/or one or more contact intensity sensors 265 to detect the intensity of contacts on touch screen 212 One or more tactile output generators 267 for doing so.

4 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, in accordance with some embodiments. Device 400 need not be portable. In some embodiments, device 400 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as children's learning toys), game system, or control device (e.g., home or industrial Controller). The device 400 typically includes one or more processing units (CPUs) 410, one or more network or other communication interfaces 460, a memory 470, and one or more communication buses 420 for interconnecting these components. Includes. Communication buses 420 optionally include circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between them. Device 400 includes an input/output (I/O) interface 430 that includes a display 440, which is typically a touch screen display. The I/O interface 430 also optionally includes a keyboard and/or mouse (or other pointing device) 450 and a touchpad 455, a tactile output for generating tactile outputs on the device 400. Generator 457 (e.g., similar to tactile output generator(s) 267 described above with reference to FIG. 2A), and sensors 459 (e.g., contact intensity sensor(s) described above with reference to FIG. ), an optical sensor similar to 265, an acceleration sensor, a proximity sensor, a touch sensitive sensor, and/or a contact intensity sensor). Memory 470 includes high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; Optionally one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other nonvolatile solid state storage devices. Memory 470 optionally includes one or more storage devices located remotely from CPU(s) 410. In some embodiments, memory 470 includes programs, modules, and data structures similar to programs, modules, and data structures stored in memory 202 of portable multifunction device 200 (FIG. 2A). , Or a subset of them. In addition, memory 470 optionally stores additional programs, modules and data structures not present in memory 202 of portable multifunction device 200. For example, the memory 470 of the device 400, optionally, a drawing module 480, a presentation module 482, a word processing module 484, a website creation module 486, a disk authoring module 488 ), and/or the spreadsheet module 490, while the memory 202 of the portable multifunction device 200 (FIG. 2A), optionally, does not store these modules.

Each of the previously identified elements in FIG. 4 may be stored in one or more of the previously mentioned memory devices. Each of the previously identified modules corresponds to a set of instructions for performing the functions described above. The modules or programs (e.g., sets of instructions) identified above need not be implemented as separate software programs, procedures, or modules, so that various subsets of these modules may be combined or different in various embodiments. Can be rearranged in a way. In some embodiments, memory 470 may store a subset of the modules and data structures identified above. Also, the memory 470 may store additional modules and data structures not described above.

Hereinafter, embodiments of user interfaces that may be implemented on the portable multifunction device 200 will be described.

5A shows an exemplary user interface for a menu of applications on portable multifunction device 200, in accordance with some embodiments. Similar user interfaces may be implemented on device 400. In some embodiments, user interface 500 includes the following elements, or a subset or superset thereof:

셀룰러 및 Wi-Fi 신호들과 같은 무선 통신(들)에 대한 신호 강도 표시자(들)(502);

Signal strength indicator(s) 502 for wireless communication(s) such as cellular and Wi-Fi signals;

시간(504);

Time 504;

블루투스 표시자(505);

Bluetooth indicator 505;

배터리 상태 표시자(506);

Battery status indicator 506;

다음과 같은, 빈번하게 사용되는 애플리케이션들에 대한 아이콘들을 갖는 트레이(508):

Tray 508 with icons for frequently used applications, such as:

부재 중 전화들 또는 음성메일 메시지들의 개수의 표시자(514)를 선택적으로 포함하는 "전화"라고 라벨링된 전화 모듈(238)에 대한 아이콘(516);

An icon 516 for a phone module 238 labeled "call" optionally including an indicator 514 of the number of missed calls or voicemail messages;

읽지 않은 이메일들의 개수의 표시자(510)를 선택적으로 포함하는 "메일"이라고 라벨링된 이메일 클라이언트 모듈(240)에 대한 아이콘(518);

An icon 518 for an email client module 240 labeled “mail” that optionally includes an indicator 510 of the number of unread emails;

"브라우저"라고 라벨링된 브라우저 모듈(247)에 대한 아이콘(520);및

An icon 520 for a browser module 247 labeled "browser"; and

아이팟(애플 인크.의 상표) 모듈(252)로도 지칭되는, "아이팟"이라고 라벨링된 비디오 및 음악 재생기 모듈(252)에 대한 아이콘(522);및

An icon 522 for a video and music player module 252 labeled “iPod”, also referred to as an iPod (trademark of Apple Inc.) module 252; and

다음과 같은, 다른 애플리케이션들에 대한 아이콘들:

Icons for different applications, such as:

"메시지"라고 라벨링된 IM 모듈(241)에 대한 아이콘(524);

An icon 524 for the IM module 241, labeled "Message";

"캘린더"라고 라벨링된 캘린더 모듈(248)에 대한 아이콘(526);

An icon 526 for the calendar module 248, labeled "calendar";

"사진"이라고 라벨링된 이미지 관리 모듈(244)에 대한 아이콘(528);

An icon 528 for the image management module 244 labeled "Photo";

"카메라"라고 라벨링된 카메라 모듈(243)에 대한 아이콘(530);

Icon 530 for camera module 243, labeled "camera";

"온라인 비디오"라고 라벨링된 온라인 비디오 모듈(255)에 대한 아이콘(532);

Icon 532 for online video module 255, labeled "Online Video";

"주식"이라고 라벨링된 주식 위젯(249-2)에 대한 아이콘(534);

Icon 534 for stock widget 249-2, labeled "stock";

"지도"라고 라벨링된 지도 모듈(254)에 대한 아이콘(536);

An icon 536 for the map module 254 labeled "Map";

"날씨"라고 라벨링된 날씨 위젯(249-1)에 대한 아이콘(538);

Icon 538 for weather widget 249-1, labeled "Weather";

"시계"라고 라벨링된 알람 시계 위젯(249-4)에 대한 아이콘(540);

Icon 540 for alarm clock widget 249-4, labeled "Clock";

"운동 지원"이라고 라벨링된 운동 지원 모듈(242)에 대한 아이콘(542);

Icon 542 for exercise support module 242, labeled "Exercise Support";

"메모"라고 라벨링된 메모 모듈(253)에 대한 아이콘(544);및

An icon 544 for the notes module 253, labeled “Notes”; and

디바이스(200) 및 그의 다양한 애플리케이션들(236)에 대한 설정으로의 액세스를 제공하는, "설정"이라고 라벨링된, 설정 애플리케이션 또는 모듈에 대한 아이콘(546).

An icon 546 for a settings application or module, labeled "Settings," that provides access to settings for device 200 and its various applications 236.

It should be noted that the icon labels shown in FIG. 5A are merely exemplary. For example, the icon 522 for the video and music player module 252 may optionally be labeled “music” or “music player”. Other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to each application icon. In some embodiments, the label for a specific application icon is separate from the name of the application corresponding to the specific application icon.

5B shows a device (e.g., device 400) having a touch-sensitive surface 551 (e.g., a tablet or touchpad 455, FIG. 4) that is separate from a display 550 (e.g., touch screen display 212). ), Fig. 4) illustrates an exemplary user interface. Device 400 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 457) and/or device 400 to detect the intensity of contacts on the touch-sensitive surface 551. And one or more tactile output generators 459 for generating tactile outputs for a user.

While some examples that follow will be provided with reference to inputs on the touch screen display 212 (a combination of a touch-sensitive surface and display), in some embodiments, the device is separate from the display as shown in FIG. 5B. Detect inputs on a touch-sensitive surface. In some embodiments, the touch-sensitive surface (eg, 551 in FIG. 5B) has a major axis (eg, 552 in FIG. 5B) corresponding to a major axis (eg, 553 in FIG. 5B) on the display (eg, 550 ). According to these embodiments, the device is in contact with the touch-sensitive surface 551 at locations corresponding to respective locations on the display (e.g., in FIG. 5B, 560 corresponds to 568, 562 corresponds to 570). Detect contacts (eg, 560, 562 in FIG. 5B). In this way, user inputs (e.g., contacts 560, 562 and their movements) detected by the device on the touch-sensitive surface (e.g., 551 in FIG. 5B) are not subject to the touch-sensitive surface being separate from the display. When used by the device to manipulate the user interface on the display of the multifunction device (eg, 550 in FIG. It should be understood that similar methods are, optionally, used for other user interfaces described herein.

Additionally, while the following examples are given primarily with reference to finger inputs (e.g., finger contacts, finger tap gestures, finger swipe gestures), in some embodiments, one or more of the finger inputs is a different input device. It should be understood that input from (eg, mouse-based input or stylus input) is substituted. For example, a swipe gesture is optionally replaced with a mouse click (eg, instead of a contact) and subsequent movement of the cursor along the path of the swipe (eg, instead of moving the contact). As another example, the tap gesture is optionally replaced by a mouse click while the cursor is positioned over the location of the tap gesture (eg, instead of stopping detection of a contact and detecting subsequent contact). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice are selectively used simultaneously, or mouse and finger contacts are selectively used simultaneously.

6A shows an exemplary personal electronic device 600. The device 600 includes a body 602. In some embodiments, device 600 may include some or all of the features described with respect to devices 200 and 400 (eg, FIGS. 2A-4B ). In some embodiments, device 600 has a touch-sensitive display screen 604, hereinafter a touch screen 604. Alternatively, or in addition to the touch screen 604, the device 600 has a display and a touch-sensitive surface. As with devices 200 and 400, in some embodiments, touch screen 604 (or touch-sensitive surface) uses one or more intensity sensors to detect the intensity of applied contacts (eg, touches). Can have. One or more intensity sensors of the touch screen 604 (or touch-sensitive surface) may provide output data representing the intensity of the touches. The user interface of device 600 may respond to touches based on the intensity of the touches, meaning that different intensities of touches may invoke different user interface operations on device 600.

Techniques for detecting and processing touch intensity are, for example, related applications: filed on May 8, 2013, and the name of the invention is "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application" "International patent application PCT/US2013/040061, and PCT, filed on November 11, 2013 and entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships" /US2013/069483, each of these applications is hereby incorporated by reference in its entirety.

In some embodiments, device 600 has one or more input mechanisms 606 and 608. The input mechanisms 606, 608, if included, may be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 600 has one or more attachment mechanisms. These attachment mechanisms, if included, the device 600, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watches, chains, pants, belts, shoes, wallets, It can be attached to a backpack, etc. These attachment mechanisms may allow device 600 to be worn on a user.

6B shows an exemplary personal electronic device 600. In some embodiments, device 600 may include some or all of the components described with respect to FIGS. 2A, 2B, and 4. Device 600 has a bus 612 operatively coupling an I/O section 614 with one or more computer processors 616 and memory 618. The I/O section 614 can be connected to a display 604, which can have a touch sensitive component 622 and, optionally, a touch intensity sensitive component 624. Moreover, the I/O section 614 is provided with the communication unit 630 to receive application and operating system data using Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular and/or other wireless communication techniques. Can be connected. Device 600 may include input mechanisms 606 and/or 608. The input mechanism 606 can be, for example, a rotatable input device or a pushable and rotatable input device. The input mechanism 608 can be a button, in some examples.

The input mechanism 608 may be a microphone, in some examples. Personal electronic device 600, such as GPS sensor 632, accelerometer 634, direction sensor 640 (e.g., compass), gyroscope 636, motion sensor 638, and/or combinations thereof. , May include various sensors, all of which may be operatively connected to the I/O section 614.

The memory 618 of the personal electronic device 600, for example, when executed by one or more computer processors 616, causes the computer processors to include a process 1200 (FIGS. 12A-12D ). , May be a non-transitory computer-readable storage medium for storing computer-executable instructions, capable of performing the techniques described below. Computer-executable instructions can also be executed by or with an instruction execution system, apparatus, or device capable of fetching and executing instructions from an instruction execution system, apparatus or device (computer-based system, processor-containing system, or other system) May be stored and/or transferred within any non-transitory computer readable storage medium for use in connection with it. For the purposes of this document, a "non-transitory computer-readable storage medium" is any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. I can. Non-transitory computer-readable storage media can include, but are not limited to magnetic, optical, and/or semiconductor storage. Examples of such storage include magnetic disks, optical disks based on CD, DVD, or Blu-ray technologies, as well as persistent solid state memory such as flash, solid state drives, and the like. The personal electronic device 600 is not limited to the components and configuration of FIG. 6B, and may include other or additional components in a number of configurations.

As used herein, the term "affordance" refers to a user interactive (user) that can be displayed on the display screen of devices 200, 400, and/or 600 -interactive) Refers to a graphical user interface object. For example, images (eg, icons), buttons, and texts (eg, links) may each constitute an affordance.

As used herein, the term “focus selector” refers to an input element representing the current portion of the user interface that the user is interacting with. In some implementations that include a cursor or other location marker, a touch-sensitive surface (e.g., diagram) while the cursor is over a particular user interface element (e.g., button, window, slider, or other user interface element). When an input (e.g., a pressing input) is detected on the touchpad 455 of 4 or the touch-sensitive surface 551 of FIG. 5B, the cursor is “focused” so that a specific user interface element is adjusted according to the detected input. Functions as "selector". Some implementations including a touch screen display (e.g., touch-sensitive display system 212 of FIG. 2A or touch screen 212 of FIG. 5A) that allows direct interaction with user interface elements on the touch screen display. In the field, when an input (e.g., a pressing input by contact) is detected on a touch screen display at the location of a particular user interface element (e.g., button, window, slider or other user interface element), a particular user interface element The detected contact on the touch screen functions as a "focus selector" so that is adjusted according to the detected input. In some implementations, the focus is transferred to the user interface without movement of the corresponding cursor or movement of the contact (e.g., by moving the focus from one button to another using the tab key or arrow keys). Moved from one area of the user interface to another area of the user interface; In such implementations, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector has, the focus selector is generally used to convey the user's intended interaction with the user interface (e.g., by indicating to the device an element of the user interface the user wishes to interact with). Is a user interface element (or touch on a touch screen display) controlled by. For example, while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of the focus selector (e.g., cursor, contact, or selection box) on the respective button is (on the display of the device). Unlike other user interface elements shown in Figure 2), it will indicate that the user is intending to activate their respective buttons.

As used in the specification and claims, the term “characteristic strength” of a contact refers to a property of a contact based on one or more strengths of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is, optionally, a predefined number of intensity samples, or a predefined event (e.g., after detecting a contact, before detecting liftoff of the contact, before or after detecting the start of movement of the contact). , Before or after detecting the end of the contact, before or after detecting an increase in the intensity of the contact, and/or before or after detecting a decrease in the intensity of the contact, for a predetermined period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds). The characteristic strength of the contact is, optionally, a maximum value of the contact intensities, a mean value of the contact intensities, an average value of the contact intensities, and a top 10 percentile value of the contact intensities. ), a value of half the maximum value of the intensities of the contact, a value of 90 percent of the maximum value of the intensities of the contact, and the like. In some embodiments, the duration of the contact is used to determine the feature strength (eg, when the feature strength is the average of the strength of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether the operation was performed by the user. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, as a result of a contact having a characteristic strength not exceeding a first threshold, a first operation is performed, and a result of a contact having a characteristic strength that exceeds the first intensity threshold but not exceeding a second intensity threshold, a second The operation is performed, and as a result of the contact having a characteristic intensity exceeding the second threshold, a third operation is performed. In some embodiments, the comparison between the characteristic strength and the one or more thresholds is whether to perform one or more actions (e.g., whether to perform a respective action, rather than being used to determine whether to perform a first action or a second action). It is used to determine whether to perform or stop performing the respective actions).

In some embodiments, a portion of the gesture is identified to determine a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that moves from a start position and reaches an end position (the intensity of the contact increases at this point). In this example, the characteristic strength of the contact at the end position may be based on only a portion of the continuous swipe contact (eg, only the portion of the swipe contact at the end position), not the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the strengths of the swipe contact prior to determining the characteristic strength of the contact. For example, the smoothing algorithm may optionally be one of an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. Contains one or more. In some situations, these smoothing algorithms remove narrow spikes or dips in the strengths of the swipe contact to determine the feature strength.

The intensity of a contact on the touch-sensitive surface may be characterized for one or more intensity thresholds, such as a contact-detection intensity threshold, a light tap intensity threshold, a deep tap intensity threshold, and/or one or more other intensity thresholds. In some embodiments, the light tap intensity threshold corresponds to the intensity at which the device will perform actions typically associated with clicking a button or trackpad of a physical mouse. In some embodiments, the deep click intensity threshold corresponds to the intensity at which the device will perform actions that are different from actions typically associated with clicking a button or trackpad of a physical mouse. In some embodiments, when a contact is detected with a characteristic strength below the tapping intensity threshold (e.g., and above the nominal contact-detection intensity threshold (less than the contact is no longer detected)), the device is lightly pressed. It will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing an action associated with the intensity threshold or the deep press intensity threshold. In general, unless stated otherwise, these intensity thresholds are consistent among different sets of user interface drawings.

The increase in the characteristic strength of a contact from an intensity below the light tap intensity threshold to an intensity between the light tap intensity threshold and the deep tap intensity threshold is sometimes referred to as a "light tap" input. The increase in the characteristic strength of a contact from an intensity below the deep pressing intensity threshold to an intensity above the deep pressing intensity threshold is sometimes referred to as a "deep pressing" input. The increase in the characteristic strength of a contact from an intensity below the contact-detection intensity threshold to an intensity between the touch-detection intensity threshold and the lightly press intensity threshold is sometimes referred to as detection of a contact on the touch surface. The decrease in the characteristic strength of a contact from an intensity above the contact-detection intensity threshold to an intensity below the contact-detection intensity threshold is sometimes referred to as detection of a liftoff of the contact from the touch surface. In some embodiments, the contact-detection intensity threshold is zero. In some embodiments, the contact-detection intensity threshold is greater than zero.

In some embodiments described herein, one or more actions may be performed in response to detecting a gesture including the respective pressing input or detecting a respective pressing input performed with the respective contact (or a plurality of contacts). In response to, wherein each pressing input is detected based at least in part on detecting an increase in the intensity of a contact (or a plurality of contacts) above a pressing-input intensity threshold. In some embodiments, the respective action is performed in response to detecting an increase in the strength of the respective contact above the press-input strength threshold (eg, a “down stroke” of the respective press input). . In some embodiments, the push input comprises an increase in the strength of the respective contact above the push-input strength threshold and a subsequent decrease in the strength of the contact below the push-intensity threshold, the respective action being the push-input threshold. It is performed in response to detecting a subsequent decrease in the intensity of the respective contact less than (eg, the “up stroke” of the respective pressing input).

In some embodiments, the device uses intensity hysteresis to avoid sudden inputs, sometimes referred to as "jitter," wherein the device uses a hysteresis intensity threshold (a hysteresis intensity threshold that has a predefined relationship to the push-input intensity threshold). For example, the hysteresis intensity threshold is in units of X intensity that are lower than the press-input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some suitable ratio of the push-intensity threshold). As such, in some embodiments, the push input comprises an increase in the intensity of the respective contact above the push-input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the push-input intensity threshold, and , The respective action is performed in response to detecting a subsequent decrease in the intensity of the respective contact below the hysteresis intensity threshold (eg, the “up stroke” of the respective pressing input). Similarly, in some embodiments, the push input is an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the push-intensity threshold and, optionally, of the intensity of the contact to an intensity below the hysteresis intensity. It is detected only when the device detects a subsequent decrease, the respective action being performed in response to detecting a pressing input (eg, depending on the situation, increasing the intensity of the contact or decreasing the intensity of the contact).

For convenience of explanation, the description of actions performed in response to a press input associated with a press-intensity threshold or in response to a gesture including a press input may be, optionally, an increase in the strength of a contact above the press-intensity threshold, hysteresis. An increase in the intensity of the contact from an intensity below the intensity threshold to an intensity above the press-input intensity threshold, a decrease in the intensity of the contact below the tap-intensity threshold, and/or below the hysteresis intensity threshold corresponding to the push-input intensity threshold. Triggered in response to detecting any one of the decrease in the intensity of the contact. Further, in examples where the operation is described as being performed in response to detecting a decrease in the intensity of the contact below the push-intensity threshold, the operation optionally corresponds to the push-intensity threshold and has a lower hysteresis strength. It is performed in response to detecting a decrease in the intensity of the contact below a threshold.

3. Digital Assistant System

7A shows a block diagram of a digital assistant system 700 according to various examples. In some examples, digital assistant system 700 can be implemented on a standalone computer system. In some examples, digital assistant system 700 may be distributed across multiple computers. In some examples, some of the digital assistant's modules and functions may be divided into a server portion and a client portion, wherein the client portion is one or more user devices (e.g., devices 104, 122, 200, 400, or 600 )) and communicates with the server portion (e.g., server system 108) via one or more networks, such as shown in FIG. 1. In some examples, digital assistant system 700 may be an implementation of server system 108 (and/or DA server 106) shown in FIG. 1. Digital assistant system 700 is merely an example of a digital assistant system, and digital assistant system 700 may have more or fewer components than shown, or may combine two or more components. It should be noted that, or may have components of different configurations or arrangements. The various components shown in FIG. 7A may be implemented in hardware, including one or more signal processing and/or application specific integrated circuits, software instructions for execution by one or more processors, firmware, or a combination thereof.

The digital assistant system 700 may include a memory 702, one or more processors 704, an input/output (I/O) interface 706, and a network communication interface 708. These components may communicate with each other via one or more communication buses or signal lines 710.

In some examples, memory 702 is a non-transitory computer-readable medium, such as a high-speed random access memory and/or non-volatile computer-readable storage medium (e.g., one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid State memory device).

In some examples, the I/O interface 706 connects the input/output devices 716 of the digital assistant system 700, such as displays, keyboards, touch screens, and microphones, to the user interface module 722. Can be combined. The I/O interface 706, together with the user interface module 722, may receive user inputs (eg, voice input, keyboard inputs, touch inputs, etc.) and process them accordingly. In some examples, for example, where the digital assistant is implemented on a standalone user device, the digital assistant system 700 is a component described for devices 200, 400, or 600 of FIGS. 2A, 4, 6A and 6B, respectively. And I/O communication interfaces. In some examples, the digital assistant system 700 may represent the server portion of a digital assistant implementation, and the user through a client-side portion residing on a user device (e.g., devices 104, 200, 400, or 600). Can interact with.

In some examples, the network communication interface 708 may include wired communication port(s) 712 and/or wireless transmit and receive circuitry 714. The wired communication port(s) may receive and transmit communication signals via one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), Firewire, or the like. The wireless circuitry 714 may receive and transmit RF signals and/or optical signals to/from communication networks and other communication devices. Wireless communication may use any of a number of communication standards, protocols, and technologies such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. have. Network communication interface 708 is a digital assistant system 700 and others using networks such as the Internet, intranet, and/or wireless networks, such as cellular telephone networks, wireless local area networks (LAN), and/or metropolitan networks (MANs) It may enable communication between devices.

In some examples, the memory 702 or the computer-readable storage media of the memory 702 includes an operating system 718, a communication module 720, a user interface module 722, one or more applications 724, and a digital assistant. Programs, modules, instructions, and data structures, including all of module 726 or a subset thereof, may be stored. In particular, the memory 702 or the computer-readable storage medium of the memory 702 may store instructions for performing the process 1200 described below. One or more processors 704 may execute such programs, modules, and instructions, and read/write to/from data structures.

Operating system 718 (e.g., an embedded operating system such as Darwin, RTXC, LINUX, UNIX, iOS, OS X, WINDOWS, or VxWorks) includes general system tasks (e.g., memory management, storage device control, Power management, etc.), and may include various software components and/or drivers for controlling and managing, and facilitate communications between various hardware, firmware, and software components.

The communication module 720 may facilitate communication between the digital assistant system 700 and other devices via a network communication interface 708. For example, the communication module 720 may communicate with the RF circuitry 208 of electronic devices, such as the devices 200, 400 and 600 shown in FIGS. 2A, 4, 6A, and 6B, respectively. Communication module 720 may also include various components for processing data received by wireless circuitry 714 and/or wired communication port 712.

The user interface module 722 receives commands and/or inputs from the user (e.g., from a keyboard, touch screen, pointing device, controller, and/or microphone) via the I/O interface 706 and on the display. User interface objects can be created. User interface module 722 also prepares outputs (e.g., speech, sound, animation, text, icons, vibrations, haptic feedback, lighting, etc.), and via I/O interface 706 (e.g., display Fields, audio channels, speakers, and touchpads, etc.) to the user.

Applications 724 may include programs and/or modules configured to be executed by one or more processors 704. For example, when the digital assistant system is implemented on a standalone user device, the applications 724 may include user applications such as games, calendar applications, navigation applications, or email applications. When the digital assistant system 700 is implemented on a server, the applications 724 may include resource management applications, diagnostic applications, or scheduling applications, for example.

The memory 702 may also store a digital assistant module 726 (or the server portion of the digital assistant). In some examples, digital assistant module 726 may include the following submodules, or a subset or superset thereof: input/output processing module 728, speech-to-text (STT). ) Processing module 730, natural language processing module 732, conversation flow processing module 734, task flow processing module 736, service processing module 738, and speech synthesis module 740. Each of these modules may access one or more of the following systems or data and models of digital assistant module 726, or a subset or superset of them: ontology 760, vocabulary index 744, user Data 748, task flow models 754, service models 756, and ASR system.

In some examples, using models, data, and processing modules implemented in digital assistant module 726, the digital assistant can perform at least some of the following: converting speech input to text; Identifying the user's intent expressed in natural language input received from the user; Actively eliciting and obtaining the information necessary to fully infer the user's intentions (eg, by disambiguating words, games, intentions, etc.); Determining a task flow to fulfill the inferred intent; And executing the task flow to fulfill the inferred intention.

In some examples, as shown in FIG. 7B, the I/O processing module 728 interacts with the user through the I/O devices 716 of FIG. 7A or through the network communication interface 708 of FIG. 7A. Interact with a user device (e.g., devices 104, 200, 400, or 600) via to obtain user input (e.g., speech input) and provide responses to the user input (e.g., as speech outputs). I can. The I/O processing module 728 may selectively acquire context information associated with the user input from the user device together with or immediately after receiving the user input. Context information may include user-specific data, vocabulary, and/or preferences associated with user input. In some examples, the contextual information also includes information related to the software and hardware state of the user device at the time the user request was received and/or information related to the user's surroundings at the time the user request was received. In some examples, the I/O processing module 728 may also send follow-up questions to the user regarding the user request and receive answers therefrom. If the user request is received by the I/O processing module 728 and the user request may include speech input, the I/O processing module 728 sends the speech input to the STT processing module 730 for speech-to-text conversion. ) (Or speech recognizer).

The STT processing module 730 may include one or more ASR systems. One or more ASR systems may process speech input received through the I/O processing module 728 to generate a recognition result. Each ASR system may include a front-end speech pre-processor. The front-end speech preprocessor can extract representative features from speech input. For example, a front-end speech preprocessor can Fourier transform the speech input to extract spectral features that characterize the speech input as a representative sequence of multidimensional vectors. In addition, each ASR system may include one or more speech recognition models (eg, acoustic models and/or language models), and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian-Mixture Models, Deep Neural Network Models, n-gram language models, and other statistical models. . Examples of speech recognition engines may include dynamic time distortion based engines and weighted finite state transformer (WFST) based engines. One or more speech recognition models and one or more speech recognition engines may be used for intermediate recognition results (e.g., phonemes, phoneme strings, and sub-words), and ultimately text recognition results (e.g., words, word strings, or tokens). Sequence) can be used to process the extracted representative features of the front-end speech preprocessor. In some examples, speech input may be processed at least in part by a third party service or on a user's device (eg, device 104, 200, 400, or 600) to generate a recognition result. When the STT processing module 730 generates a recognition result comprising a text string (e.g., words, or a sequence of words, or a sequence of tokens), the recognition result is the natural language processing module 732 Can be delivered to. An exemplary ASR system is described in more detail in FIG. 9.

Further details on speech-text processing are described in US Patent Application No. 13/236,942 for "Consolidating Speech Recognition Results" filed September 20, 2011, the entire disclosure of which is incorporated herein by reference. Included as.

/ 및 /

/의 후보 발음과 연관된 "tomato"라는 단어를 포함할 수 있다. 또한, 어휘 단어는 사용자로부터의 이전 스피치 입력에 기초한 맞춤형 후보 발음과 연관될 수 있다. 이러한 맞춤형 후보 발음은 STT 프로세싱 모듈(730)에 저장될 수 있고, 디바이스 상의 사용자의 프로필을 통해 특정 사용자와 연관될 수 있다. 일부 예들에서, 단어에 대한 후보 발음은 단어의 철자 및 하나 이상의 언어 및/또는 음성 규칙에 기초하여 결정될 수 있다. 일부 예들에서, 후보 발음은 예를 들어, 알려진 정준 발음(canonical pronunciations)에 기초하여 수동으로 생성될 수 있다.In some examples, the STT processing module 730 may contain and/or access a vocabulary of recognizable words via the phonetic symbol conversion module 731. Each vocabulary word may be associated with one or more candidate pronunciations of a word represented by a speech recognition phonetic symbol. In particular, the vocabulary of the recognizable word may include words associated with a plurality of candidate pronunciations. For example, the vocabulary is /

/ And /

It may include the word "tomato" associated with the candidate pronunciation of /. In addition, the vocabulary words may be associated with a customized candidate pronunciation based on previous speech input from the user. Such customized candidate pronunciation may be stored in the STT processing module 730 and may be associated with a specific user through a user's profile on the device. In some examples, a candidate pronunciation for a word may be determined based on the spelling of the word and one or more language and/or phonetic rules. In some examples, the candidate pronunciation may be manually generated based on known canonical pronunciations, for example.

/는 /

/보다 높은 순위가 될 수 있는데, 그 이유는 이전 발음이 더 일반적으로 사용되는 발음이기 때문이다(예를 들어, 모든 사용자 중에서, 특정 지리적 지역에 있는 사용자의 경우, 또는 임의의 다른 적절한 사용자들의 서브세트의 경우). 일부 예들에서, 후보 발음은 후보 발음이 사용자와 연관된 맞춤형 후보 발음인지 여부에 기초하여 순위가 매겨질 수 있다. 예를 들어, 맞춤형 후보 발음은 정준 후보 발음보다 높은 순위가 매겨질 수 있다. 이는 정준 발음에서 벗어나는 고유한 발음을 가진 고유 명사를 인식하는 데 유용할 수 있다. 일부 예들에서, 후보 발음은 지리적인 기원, 국적, 또는 민족성과 같은 하나 이상의 스피치 특성과 연관될 수 있다. 예를 들어, 후보 발음 /

/는 미국과 연관될 수 있는 반면, 후보 발음 /

/는 영국과 연관될 수 있다. 또한, 후보 발음의 순위는 디바이스 상의 사용자 프로필에 저장된 사용자의 하나 이상의 특성(예를 들어, 지리적 기원, 국적, 민족성 등)에 기초할 수 있다. 예를 들어, 이는 사용자가 미국과 연관되어 있다는 사용자의 프로필로부터 결정될 수 있다. 사용자가 미국과 연관되어 있음에 기초하여, 후보 발음 /

/(미국과 연관됨)는 후보 발음 /

/(영국과 연관됨)보다 높은 순위가 매겨질 수 있다. 일부 예들에서, 순위가 매겨진 후보 발음들 중 하나는 예측된 발음(예를 들어, 가장 가능성이 있는 발음)으로서 선택될 수 있다.In some examples, the candidate pronunciation may be ranked based on the commonality of the candidate pronunciation. For example, the candidate pronunciation /

/ Is /

It can be ranked higher than / because the previous pronunciation is the more commonly used pronunciation (e.g., among all users, for users in a specific geographic area, or for any other suitable user For sets). In some examples, the candidate pronunciation may be ranked based on whether the candidate pronunciation is a customized candidate pronunciation associated with the user. For example, the customized candidate pronunciation may be ranked higher than the canonical candidate pronunciation. This may be useful for recognizing proper nouns with a unique pronunciation that deviates from the canonical pronunciation. In some examples, the candidate pronunciation may be associated with one or more speech characteristics, such as geographic origin, nationality, or ethnicity. For example, the candidate pronunciation /

/ Can be associated with the United States, while the candidate pronunciation /

/ Can be associated with the UK. Further, the ranking of the candidate pronunciations may be based on one or more characteristics of the user (eg, geographic origin, nationality, ethnicity, etc.) stored in a user profile on the device. For example, this may be determined from a user's profile that the user is associated with the United States. Based on the user's association with the United States, the candidate pronunciation /

/ (Associated with the United States) is the candidate pronunciation /

May be ranked higher than / (relevant to the UK). In some examples, one of the ranked candidate pronunciations may be selected as the predicted pronunciation (eg, most likely pronunciation).

/를 먼저 식별할 수 있는 경우, 그것은 이어서 어휘 인덱스(744)에 기초하여 이 시퀀스가 단어 "tomato"에 대응한다고 결정할 수 있다.When speech input is received, the STT processing module 730 can be used to determine the phoneme corresponding to the speech input (e.g., using an acoustic model), and then (e.g., using a language model). ) You can try to determine a word that matches the phoneme. For example, the STT processing module 730 is a sequence of phonemes corresponding to a portion of the speech input /

If / can be identified first, it can then determine that this sequence corresponds to the word "tomato" based on the vocabulary index 744.

/가 단어 "tomato"에 대응한다고 결정할 수 있다.In some examples, the STT processing module 730 can determine words in the spoken input using an approximate matching technique. Thus, for example, the STT processing module 730 may have a sequence of phonemes / even if a specific sequence of phonemes is not one of the candidate sequences of phonemes for the word.

We can determine that / corresponds to the word "tomato".

The digital assistant's natural language processing module 732 ("natural language processor") takes a sequence of words or tokens ("token sequence") generated by the STT processing module 730 and converts the token sequence by the digital assistant. It may attempt to associate with one or more perceived “actionable intents”. “Actionable intent” can represent a task that can be performed by the digital assistant, and can have an associated task flow implemented in task flow models 754. The associated task flow can be a series of programmed actions and steps that the digital assistant takes to perform the task. The scope of the digital assistant's capabilities may depend on the number and type of task flows implemented and stored in the task flow models 754, or in other words, the number and type of “actionable intents” recognized by the digital assistant. However, the effectiveness of the digital assistant may also depend on the assistant's ability to infer the correct "actionable intention(s)" from user requests expressed in natural language.

In some examples, in addition to the sequence of words or tokens obtained from the STT processing module 730, the natural language processing module 732 also includes a context associated with the user request, for example, from the I/O processing module 728. You can receive information. The natural language processing module 732 may optionally use context information to clarify, supplement, and/or further define information included in the token sequence received from the STT processing module 730. Context information may include, for example, user preferences, hardware and/or software states of the user device, sensor information collected before, during, or immediately after the user request, previous interactions between the digital assistant and the user (e.g. , Conversation), etc. As described herein, contextual information can be dynamic and can change with time, location, content of the conversation, and other factors.

In some examples, natural language processing may be based on ontology 760, for example. Ontology 760 may be a hierarchical structure including many nodes, each node being a “property” related to one or more of “actionable intent” or “actionable intent”, or other Express any of the "attributes". As noted above, “actionable intent” can represent a task that the digital assistant can perform, ie, a task that it can “actionable” or has an effect on. “Attribute” may represent a parameter associated with an actionable intent or other attribute of a lower aspect. The linkage between the actionable intent node and the property node in the ontology 760 may define how a parameter represented by the property node relates to a task expressed by the actionable intent node.

In some examples, the ontology 760 may be comprised of actionable intention nodes and attribute nodes. Within ontology 760, each actionable intent node may be linked to one or more attribute nodes directly or through one or more intermediate attribute nodes. Similarly, each attribute node may be linked to one or more actionable intent nodes either directly or through one or more intermediate attribute nodes. For example, as shown in FIG. 7C, the ontology 760 may include a “restaurant reservation” node (ie, an actionable intention node). The attribute nodes “restaurant”, “date/time” (for reservation), and “number of people” can each be linked directly to an actionable intention node (ie, “restaurant reservation” node).

Additionally, the attribute nodes "cooking", "price range", "telephone number", and "location" may be subnodes of the attribute node "restaurant", and the "restaurant reservation" node ( That is, they may be linked to each actionable intention node). For another example, as shown in FIG. 7C, the ontology 760 may also include a “set reminder” node (ie, another actionable intention node). The attribute nodes "Date/Time" (for setting a reminder) and "Subject" (for setting a reminder) may be linked to the "Set Reminder" node, respectively. Since the attribute "Date/Time" can be related to both the task of making a restaurant reservation and the task of setting a reminder, the attribute node "Date/Time" is the "Reserve Restaurant" node and "Set Reminder" in the ontology 760 It can be linked to both nodes.

An actionable intent node, along with its linked concept nodes, may be described as a “domain”. In this discussion, each domain may be associated with an individual actionable intent, and refers to a group of nodes (and relationships between them) associated with a particular actionable intent. For example, the ontology 760 illustrated in FIG. 7C may include an example of a restaurant reservation domain 762 and an example of a reminder domain 764 in the ontology 760. The restaurant reservation domain includes an actionable intent node "restaurant reservation", attribute nodes "restaurant", "date/time", and "persons", and sub-attribute nodes "cooking", "price range", "telephone number", and Includes "location". The reminder domain 764 may include an actionable intent node “reminders setup”, and attribute nodes “subject” and “date/time”. In some examples, ontology 760 can be composed of many domains. Each domain may share one or more attribute nodes with one or more other domains. For example, the "Date/Time" attribute node may be associated with many different domains (eg, scheduling domain, travel reservation domain, movie ticket domain, etc.), in addition to restaurant reservation domain 762 and reminder domain 764. .

Although FIG. 7C shows two exemplary domains in the ontology 760, other domains are, for example, "find a movie", "initiate a phone call", "find directions", "schedule a meeting". , "Send a message", "Provide an answer to a question", "Read a list", "Provide navigation commands", "Provide commands for a task", and the like. The “send message” domain may be associated with the “send message” actionable intent node, and may further include attribute nodes such as “recipient(s)”, “message type”, and message body. The attribute node "recipient" can be further defined by sub-attribute nodes, for example "recipient name" and "message address".

In some examples, ontology 760 may include all domains (and thus actionable intents) that the digital assistant can understand and operate in accordance with. In some examples, the ontology 760 may be modified, for example, by adding or removing entire domains or nodes, or by modifying the relationship between nodes within the ontology 760.

In some examples, nodes associated with a number of relevant actionable intents may be clustered under a “top domain” within ontology 760. For example, the "travel" higher-level domain may include a cluster of actionable intention nodes and attribute nodes related to the travel. Actionable intention nodes related to the trip may include "airline reservation", "hotel reservation", "car rental", "direction search", "finding a point of interest", and the like. Actionable intent nodes under the same parent domain (eg, "travel" parent domain) may have many attribute nodes in common. For example, actionable intent nodes for "Airline Reservation", "Hotel Reservation", "Car Rental", "Get Directions", and "Find Point of Interest" are attribute nodes "Start Location", "Destination", " One or more of "Date/Time of Departure", "Date/Time of Arrival", and "Number of Persons" may be shared.

In some examples, each node in ontology 760 may be associated with a set of words and/or phrases related to an attribute or actionable intent represented by the node. The individual set of words and/or phrases associated with each node may be a so-called “vocabulary” associated with the node. An individual set of words and/or phrases associated with each node may be stored in a vocabulary index 744 with respect to an attribute or actionable intent represented by the node. For example, returning to Fig. 7B, the vocabulary associated with the node for the attribute of "restaurant" is "food", "beverage", "cooking", "hungry", "eat", "pizza", "fast food", Words such as "meal" may be included. For another example, the vocabulary associated with the node for the actionable intent of "initiate a phone call" is "call", "call", "dial", "ringtone", "call this number", "to It may include words and phrases such as "call me". The vocabulary index 744 may optionally include words and phrases of different languages.

The natural language processing module 732 may receive a token sequence (eg, a text string) from the STT processing module 730 and determine which nodes are involved by words in the token sequence. In some examples, if a word or phrase in the token sequence is found to be associated with one or more nodes in ontology 760 (via vocabulary index 744), the word or phrase “triggers” or “activates” these nodes. I can make it. Based on the relative importance and/or amount of active nodes, the natural language processing module 732 may select one of the intents to act as a task the user intended the digital assistant to perform. In some examples, the domain with the most “triggered” nodes may be selected. In some examples, the domain with the highest confidence value (eg, based on the relative importance of its various triggered nodes) may be selected. In some examples, the domain may be selected based on a combination of importance and number of triggered nodes. In some examples, additional factors such as whether the digital assistant previously correctly interpreted a similar request from the user are likewise considered in selecting a node.

User data 748 contains user-specific information, such as user-specific vocabulary, user preferences, user address, user's default language and second language, user's contact list, and other short or long term information for each user. Can include. In some examples, the natural language processing module 732 may supplement information included in a user input using user-specific information to further define user intent. For example, for a user request "Invite my friends to my birthday party", the natural language processing module 732 determines who the "friends" are and when and where the "birthday party" will be held. Instead of requiring the user to provide such information explicitly in his/her request, the user data 748 may be accessed.

Other details of searching an ontology based on a token string are described in US Patent Application No. 12/341,743 for "Method and Apparatus for Searching Using An Active Ontology" filed on December 22, 2008. The entire disclosure of is incorporated herein by reference.

In some examples, once the natural language processing module 732 identifies an actionable intent (or domain) based on the user request, the natural language processing module 732 uses a structured query to express the identified actionable intent. structured query) can be created. In some examples, the structured query may include parameters for one or more nodes in the domain for actionable intent, with at least some of the parameters appended with specific information and requirements specified in the user request. For example, the user may say "Make me a dinner reservation at a sushi place at 7". In this case, the natural language processing module 732 can accurately identify that the actionable intention is "restaurant reservation" based on the user input. According to the ontology, the structured query for the "restaurant reservation" domain may include parameters such as {cooking}, {time}, {date}, and {persons}. In some examples, based on speech input and text derived from speech input using STT processing module 730, natural language processing module 732 can generate a partially structured query for the restaurant reservation domain, Here, the partially structured query includes parameters {cooking = "Sushi"} and {time = "7pm"}. However, in this example, the user's speech input contains insufficient information to complete the structured query associated with the domain. Thus, other essential parameters such as {persons} and {date} may not be specific to a structured query based on currently available information. In some examples, the natural language processing module 732 can append the received context information to some parameters of the structured query. For example, in some examples, if the user requested a sushi restaurant “near me”, the natural language processing module 732 may append GPS coordinates from the user device to the {location} parameter in the structured query.

In some examples, the natural language processing module 732 can pass the generated structured query (including any completed parameters) to the task flow processing module 736 ("task flow processor"). The task flow processing module 736 receives the structured query from the natural language processing module 732, completes the structured query, if necessary, and performs the actions necessary to "complete" the user's ultimate request. Can be configured. In some examples, various procedures required to complete these tasks may be provided in task flow models 754. In some examples, task flow models 754 may include procedures for obtaining additional information from a user, and task flows for performing actions associated with an actionable intent.

As described above, in order to complete the structured query, the task flow processing module 736 needs to initiate additional conversations with the user to obtain additional information and/or to disambiguate the potentially ambiguous speech inputs. There may be. When such an interaction is required, the task flow processing module 736 may call the conversation flow processing module 734 to participate in a conversation with the user. In some examples, the conversation flow processing module 734 can determine how (and/or when) to ask the user for additional information, and receive and process the user response. Questions can be provided to users through I/O processing module 728 and answers can be received from them. In some examples, the conversation flow processing module 734 can present the conversation output to the user via audio and/or visual output, and receives input from the user via spoken or physical (eg, clicking) responses. Following the above example, if the task flow processing module 736 calls the conversation flow processing module 734 to determine the "number of people" and "date" information for the structured query associated with the domain "restaurant reservation", the conversation flow The processing module 734 is "how many?" And “When is it?” can be created and forwarded to the user. When answers are received from the user, the dialog flow processing module 734 may then add the missing information to the structured query, or transmit the information to the task flow processing module 736 to complete the missing information from the structured query. .

Once the task flow processing module 736 has completed the structured query for the actionable intent, the task flow processing module 736 can proceed to perform the ultimate task associated with the actionable intent. Accordingly, the task flow processing module 736 may execute steps and instructions in the task flow model according to specific parameters included in the structured query. For example, a task flow model for actionable intent of “restaurant reservation” may include steps and instructions for contacting a restaurant and actually requesting a reservation for a specific number of people at a specific time. For example, using a structured query such as {Restaurant reservation, restaurant = ABC cafe, date = 3/12/2012, time = 7pm, number of people = 5}, the task flow processing module 736 can: (1) Logging on to a restaurant reservation system such as OPENTABLE® or the server of ABC Cafe, (2) entering date, time, and number of people information in a certain form on the website, (3) submitting the form, and ( 4) You can perform the step of creating a calendar entry for reservation in the user's calendar.

In some examples, task flow processing module 736 assists in service processing module 738 ("service processing module") to complete a requested task in user input or provide an informative answer requested in user input. You can employ a stance. For example, the service processing module 738 may call or interact with other user applications installed on the user device, to make a phone call on behalf of the task flow processing module 736, to set up a calendar entry, to invoke a map search, It can interact and act to invoke or interact with third party services (eg, restaurant reservation portal, social networking website, banking portal, etc.). In some examples, the protocols and application programming interfaces (APIs) required by each service may be specified by a respective one of the service models 756. The service processing module 738 may access an appropriate service model for a service, and generate requests for a service according to protocols and APIs required by the service according to the service model.

For example, if the restaurant has enabled an online reservation service, the restaurant may submit a service model specifying APIs for conveying parameters required for making a reservation and values of parameters required for the online reservation service. When requested by the task flow processing module 736, the service processing module 738 can establish a network connection with the online reservation service using the web address stored in the service model, and format according to the API of the online reservation service. The necessary parameters (eg, time, date, number of people) of the reservation can be transmitted to the online reservation interface.

In some examples, the natural language processing module 732, the conversation flow processing module 734, and the task flow processing module 736 can infer and define the user's intent, to further clarify and refine the user's intent. It can be used holistically and repeatedly to generate a response (ie, output to the user, or completion of a task) to obtain and finally fulfill the user's intent. The generated response may be a conversational response to a speech input that at least partially fulfills the user's intent. Also, in some examples, the generated response can be output as a speech output. In these examples, the generated response may be sent to a speech synthesis module 740 (eg, a speech synthesizer), where the speech synthesis module may be processed to synthesize a conversational response in a speech form. In still other examples, the generated response may be data content related to satisfying a user request in a speech input.

The speech synthesis module 740 may be configured to synthesize speech output for presentation to a user. The speech synthesis module 740 synthesizes the speech output based on the text provided by the digital assistant. For example, the generated conversation response may be in the form of a text string. The speech synthesis module 740 may convert a text string into an audible speech output. The speech synthesis module 740 can use any suitable speech synthesis technique to generate speech output from text, which is concatenative synthesis, unit selection synthesis, diphon synthesis, domain-specific synthesis, Formant synthesis, articulatory synthesis, hidden Markov model (HMM) based synthesis, and sinewave synthesis. In some examples, the speech synthesis module 740 may be configured to synthesize individual words based on a phonetic string corresponding to the words. For example, the phoneme string may be associated with a word in the generated conversation response. The phoneme string may be stored in metadata associated with a word. The speech synthesis module 740 may be configured to directly process phoneme strings in metadata to synthesize speech-type words.

In some examples, instead of using (or in addition to) speech synthesis module 740, speech synthesis may be performed on a remote device (e.g., server system 108), and the synthesized speech is output to the user. May be transmitted to the user device for the purpose. For example, this may occur in some implementations where the output to the digital assistant is generated in the server system. And since the server system generally has more processing power or resources than the user device, it may be possible to obtain a speech output of higher quality than is actually realized in the client-side synthesis.

Additional details on the digital assistant are filed on January 10, 2011 and entitled "Intelligent Automated Assistant", US Patent Application No. 12/987,982, filed September 30, 2011 and entitled "Generating and Processing Task Items That Represent Tasks to Perform" can be found in US Patent Application Ser. No. 13/251,088, the entire disclosures of which are incorporated herein by reference.

4. Example architecture of the device

8A shows a block diagram of an exemplary architecture for device 800 according to various examples of the present invention. In some examples, device 800 and remote control 810 may implement a digital assistant system (eg, digital assistant system 700). In some examples, device 800, remote control 108, and server (eg, server 108) may implement a digital assistant system (eg, digital assistant system 700). In the embodiment of FIG. 8A, media or other content is selectively received by device 800 via a network interface 802, optionally a wired or wireless connection. One or more processors 804 may optionally include a memory 806 that optionally includes instructions to perform one or more of the methods and/or processes described herein (e.g., process 1200), or Any number of programs stored in the storage can be executed.

In some embodiments, display controller 808 causes one or more user interfaces to be displayed on display 814. Further, input to device 800 is optionally provided by remote control 810 via remote interface 812 using a wireless or wired connection. It is understood that the embodiment of FIG. 8A is not intended to limit the features of the device of the present invention, and that other components that enable other features described herein are likewise selectively included in or omitted from the architecture of FIG. 8A. do. In some embodiments, device 800 optionally corresponds to one or more of portable multifunction device 200 in FIGS. 2A and 3 and device 400 in FIG. 4; The network interface 802 optionally corresponds to one or more of the RF circuitry 208 in FIGS. 2A and 4, the external port 224 and the peripherals interface 218, and the network communication interface 460 in FIG. 4, and ; Processor 804 optionally corresponds to one or more of processor(s) 220 in FIG. 2A and CPU(s) 410 in FIG. 4; Display controller 808 optionally corresponds to one or more of display controller 256 in FIG. 2A and I/O interface 430 in FIG. 4; Memory 806 optionally corresponds to one or more of memory 202 in FIG. 2A and memory 470 in FIG. 3; The remote interface 812 optionally includes one or more of the peripherals interface 218 and I/O subsystem 206 (and/or components thereof) in FIG. 2A, and the I/O interface 430 in FIG. 4. Correspond to; The remote control 812 is optionally a speaker 211 in FIG. 2A, a touch-sensitive display system 212, a microphone 213, a light sensor(s) 264, a contact intensity sensor(s) 265, a tactile sense. Red output generator(s) 267, other input control devices 216, accelerometer(s) 268, proximity sensor 266, and I/O subsystem 206, and keyboard/ Corresponding to one or more of the mouse 450, the touchpad 455, the tactile output generator(s) 457 and the contact intensity sensor(s) 459, and the touch-sensitive surface 551 in FIG. 5B, and / Or include him; Display 814 optionally corresponds to one or more of the touch-sensitive display system 212 in FIGS. 2A and 3, and the display 440 in FIG. 4.

8B shows an exemplary structure of a remote controller 810 according to various examples of the present invention. The remote control 810 optionally includes a touch-sensitive surface 551. In some embodiments, the touch-sensitive surface 551 is edge-to-edge (e.g., as shown in FIG. 8B, the touch-sensitive surface 551 is Extending to opposite edges, so that there is little or no surface of the remote control 810 between the touch-sensitive surface 551 and one or more edges of the remote control 810). The touch-sensitive surface 551 can optionally detect contact intensities (eg, clicks on the touch-sensitive surface 551) as well as contacts according to the examples described herein. The remote control 810 also optionally includes buttons 816, 818, 820, 822, 824, 826. Buttons 816, 818, 820, 822, 824, 826 are optionally, for example, mechanical buttons for initiating corresponding action(s) on device 800 or contacting or pressing of such buttons. These are mechanical button alternatives that can detect. In some embodiments, selection of the “menu” button 816 by the user causes the device 800 to be reversed (e.g., again, before the currently displayed user interface) in the currently running application or the currently displayed user interface. To the user interface that has been displayed), or to the device 800 to a user interface of one higher level than the currently displayed user interface. In some embodiments, selection of the “home” button 818 by the user is from any user interface displayed on the display 814 of the device 800 to the main, home, or root user interface (e.g., The device 800 is navigated to the home screen of the device 800, which optionally includes one or more applications accessible on the device 800. In some embodiments, selection of the “play/pause” button 820 by the user toggles between pause and play of the content item currently playing on the device 800 (eg, “play/pause” When the content item is being played on the device 800 when the "stop" button 820 is selected, the content item is selectively paused, and when the "play/pause" button 820 is selected, the content item is If paused on (800), the content item is selectively played). In some embodiments, selection of the "+" button 822 or the "-" button 824 by the user, respectively, the volume of audio being played by device 800 (e.g., current on device 800 Increases or decreases the volume of the content item being played. In some embodiments, selection of the “voice input” button 826 by the user causes the digital assistant (eg, digital assistant system 700) to receive the user's voice input. The remote control 810 may also transmit the received voice input or representation thereof to another device (eg, device 800 or server) for further processing. In some embodiments, the remote control 810 includes one or more accelerometers for detecting information about the motion of the remote control; Accelerometers are optionally used in the manner provided in the description of the embodiments described below.

9 shows an exemplary schematic block diagram of an automatic speech recognition (ASR) system 900 in accordance with some embodiments. In some examples, the ASR system 900 may correspond to the ASR system(s) 731 of the digital assistant system 700. In some embodiments, the ASR system 900 may be implemented using one or more multifunctional devices including, but not limited to, devices 200, 400, 800 (FIGS. 2A, 4, 8A). Multifunctional devices may include devices such as servers, personal computers, mobile devices, remote controls, and the like. In particular, the ASR system 900 may be implemented in memory (e.g., memory 202 or 470) and/or processor(s) (e.g., processor(s) 220 or CPUs 410) of one or more devices. I can. The ASR system 900 can be included in a digital assistant and can provide speech recognition capabilities to a multifunctional device. In particular, the ASR system 900 may be configured to perform any of the processes or methods (eg, process 1200) described below. The ASR system 900 may include a feature extractor 902 and a recognition engine 904 described further below.

In general, the ASR system 900 receives voice input (eg, speech input); Identify letters, phrases, commands, words, or combinations thereof (collectively, "content") based on the spoken input; The identified content of the voice input can be converted into text. The ASR system 900 may additionally cause text to be displayed in a sequence corresponding to the order of contents included in the voice input.

In some embodiments, characters may include letters, numbers, control characters, or spaces. The letter may be, for example, any lowercase or uppercase letter of the alphabet (eg, "a", "b", ... "z", or "A", "B", ... "Z"). The number may be any number ("0", "1", "2", ... "9") having a value of 0 to 9, for example. The control character may be a punctuation character (eg, "@", "#", "%", "!"). In some examples, punctuation marks can be identified by punctuation words. For example, the punctuation mark “@” can be identified by the punctuation word “at”; The punctuation mark "#" can be identified by the punctuation word "pound-sign"; The punctuation "%" can be identified by the punctuation word "percent" or "percent sign"; The punctuation mark "!" can be identified by the punctuation word "exclamation-point". Other examples of punctuation marks and their corresponding punctuation words are, for example, the punctuation "/" identified by the punctuation word "slash", the punctuation "-" identified by the punctuation word "hyphen" or "dash", the punctuation word The punctuation "." identified by "period" or "dot", the punctuation "," identified by the punctuation word "comma", the punctuation ":" identified by the punctuation word "colon", and the punctuation word "semicolon" Punctuation marks identified by ";", punctuation marks identified by the punctuation word "question-mark", punctuation marks "?" identified by the punctuation word "apostrophe", punctuation marks identified by the punctuation word "open-parenthesis" "(", the punctuation ")" identified by the punctuation word "close-parenthesis", the punctuation "$" identified by the punctuation word "dollar-sign", the punctuation "*" identified by the punctuation word "asterisk", The punctuation may include the punctuation “&” and the like identified by the word “ampersand”. The control character may also be a non-punctuation character. For example, the control character may be a carriage return, a tab, or the like.

The phrase can be used to identify the corresponding letter, and can include, for example, a spelled word used for identification of the letter. For example, a spelled word for identifying the letter "a" may be "apple", "alpha", or the like. Similarly, the spelled words to identify the letter "b" may be "boy", "bravo", and the like. In some examples, the phrase for identifying the corresponding letter may have a predetermined format such as "a as in apple", "b as in boy", "n like Nancy" and "m like Mary". It will be appreciated that any spelled word can be used to identify the corresponding letter, and that the phrase for identifying the corresponding letter can have any desired format.

The command may enable editing of displayed text and/or performing one or more other functions relating to the displayed text. Exemplary commands may include "upper case", "lower case", "insert", "delete", "replace", "copy", "paste", "select", "search", and the like. The command may include one or more accompanying parameters. For example, the command "insert" may be accompanied by parameters indicating a character to be inserted (eg, "insert c"), an insertion position, and the number of times of insertion. The command may be used additionally in combination with other commands in some examples. The command "insert" may be used in combination with the command "upper case", for example (eg, "insert upper case c").

A word may contain a plurality of letters. In some examples, a word may include any combination of a plurality of letters, such as natural language words “Lincoln” and “center”. In some situations, a word may include a predetermined combination of letters. As an example, the word following the punctuation “@” may include a predetermined combination of letters forming a domain extension (eg, “com,” “net,” “org”), acronyms (eg, FBI, IRS), etc. have. This type of word can have a predetermined number of letter combinations.

In an exemplary operation of the ASR system 900, upon receiving a speech input, the feature extractor 902 may preprocess the speech input and provide the preprocessed speech input to the recognition engine 904. In some examples, preprocessing the speech input may include extracting one or more acoustic features from the speech input. Accordingly, the feature extractor 902 may extract acoustic features from the speech input and provide the extracted acoustic features to the recognition engine 904. Upon receiving the pre-processed speech input, the recognition engine 904 can identify the content of the speech input. Recognition engine 904, for example, vocabulary 912, speech recognition models 914, context data 916, and recognized credential models 918 (collectively, knowledge repository 910). Content can be identified using one or more of. For example, the recognition engine 904 may compare the content of the speech input with characters, phrases, commands, and/or words (collectively, such as known types) stored in the knowledge repository 910. have. If one or more matches are found, the recognition engine 904 can identify the content as any combination of known types. Recognition engine 904, for example, a first character (eg, the letter "a"), the second character identified by the phrase (for example, the character identified by the phrase "b as in boy" "b"), and/or a word (eg, "Lincoln").

Once the content has been identified, the recognition engine 904 may convert the content of the spoken input to text and cause the text to be displayed in a sequence corresponding to the order of the content of the spoken input. For example, the order of the displayed text may be "a" "b" "r" corresponding to the order of characters in the voice input (FIGS. 10A to 10C ).

In some examples, identifying the content of the spoken input may include identifying one or more utterances included in the spoken input. As described, acoustic features extracted by feature extractor 902 may be provided to recognition engine 904 for content identification. Based on the acoustic characteristics, the recognition engine can identify one or more utterances in the corresponding speech input. In some examples, acoustic characteristics can be used to determine the start and end of one or more utterances. As an example, the recognition engine 904 may determine that one or more acoustic features correspond to (eg, match) a long silence or pause between utterances, and thus identify separate or individual utterances. As another example, the frequency spectrum of pause or silence may differ from the frequency spectrum of speech of a letter, phrase, command, or word. As a result, analysis of the frequency spectrum can be used for identification of one or more utterances in the speech input.

In some embodiments, identifying the content of the spoken input may include determining types of identified utterances. For example, for each identified utterance, the recognition engine 904 can determine the type of utterance. The recognition engine 904 may determine, for example, whether the type of speech corresponds to a letter, a phrase for identifying a corresponding letter, a command, or a word. In some examples, previously recognized characters, phrases, words, and commands may be stored in one or more portions of knowledge store 910. As an example, characters, phrases, commands, and words stored in the knowledge repository 910 can be found in commonly used characters, phrases, commands, and words and/or in one or more previous spoken inputs. Such characters, phrases, and commands received by recognition engine 904 may be included. The knowledge store 910 may be stored on a device 800, a server (eg, server 108), a plurality of servers (eg, a server farm), or a combination thereof.

As discussed above, knowledge repository 910 may include vocabulary 912, speech recognition models 914, context data 916, and/or recognized credential models 918. The vocabulary 912 includes commonly used letters (e.g., 26 English characters), phrases (e.g., phrases containing the phonetic symbol of NATO), commands (e.g., editing commands), and words ( For example, natural language words, domain extensions, and/or acronyms). As described above, speech recognition models 914 may include acoustic models and/or language models, for example, and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian hybrid models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines may include dynamic time distortion based engines and weighted finite state transformer (WFST) based engines.

In some embodiments, speech recognition models 914 may be user-specific and may be generated by speech recognition and language model generator 913 using user-specific data. The speech recognition and language model generator 913 may be configured to receive or obtain user-specific data, such as user input, user usage data, user profile information, and the like. The speech recognition and language model generator 913 may be further configured to generate one or more user-specific speech recognition models 914 using user-specific data.

Context data 916 may include, for example, user preferences, user speech profile, hardware and/or software states, sensor information collected, previous interactions between the digital assistant and the user, and the like. Context data 916 may also include information indicative of the characteristics of the secure link or device for which the user is providing credentials. For example, context data 916 may indicate that the user is providing credentials to a web email account (eg, Yahoo, Gmail), a cloud (eg, Apple iCloud) account, or device.

The recognized credential models 918 may include commonly used credential models and/or credential models based on a user's past interaction with the digital assistant. For example, the recognized credential models 918 may include a username model and a password model. Exemplary user name models include First Name.LastName@Domain_Name.com (e.g. Abraham.Lincoln@Lincolncenter.com), Firstname Initial.Lincoln@Lincolncenter.com (e.g. ALincoln@Lincolncenter.com), FirstNameLastName_ Numbers@domain_name.com (eg, AbrahamLincoln_123@Lincolncenter.com) may be included.

As described, the recognition engine 904 may determine the type of utterance of the speech input based, for example, on acoustic features provided by the feature extractor 902. In some examples, the recognition engine 904 (eg, using an acoustic model) intermediate recognition results (eg, phonemes, phonetic strings, and sub-words) corresponding to one or more identified utterances of the speech input. Can be identified. Intermediate recognition results can be used, for example, to determine the type of each utterance. Recognition engine 904 may determine whether the type of utterance corresponds to a letter, a phrase, a command, or a word, in some embodiments. For example, the recognition engine 904 can compare intermediate recognition results (eg, using a language model) to known types in the knowledge store 910.

In some examples, the recognition engine 904 may compare a sequence of phonemes of speech (eg, pronunciation of a letter or word) with one or more sequences of known types of phonemes included in the knowledge store 910. The sequence of phonemes can be identified for speech (eg, using acoustic features). If the sequence of phonemes of the utterance matches a phoneme sequence of a known type in the knowledge store 910 (eg, substantially matches), the recognition engine 904 determines the type of utterance accordingly. The degree to which phoneme sequences are matched can be used to determine the confidence level in some examples. For example, when phoneme sequences are perfect matches, the reliability level may be 100%. As another example, when 5 out of 6 phonemes of the sequences match, the reliability level may be 90%. When a sequence of phonemes matches a letter, phrase, command, or word (eg, substantially matches), the type of utterance is determined to be a letter, phrase, command, or word, respectively.

In some examples, upon determining that the type of utterance of the speech input corresponds to a letter, phrase, word, or combination thereof, the recognition engine 904 may obtain a representation of the speech input content corresponding to the utterance. The representation may be an audio representation, a text representation, a space/vector representation, or a combination thereof.

In some examples, if the sequence of phonemes matches more than one type in the knowledge store 910, the recognition engine 904 determines a confidence level for each matched type, and the type with the highest confidence level You can choose. For example, if a sequence of phonemes matches both a word (e.g., word "insert") and a command (e.g., command "insert"), the recognition engine 904 is You can determine the level of confidence for the type. In some embodiments, the confidence level may be determined based on additional content in the voice input or other content related to the voice input. For example, a command may carry one or more parameters for performing actions according to the command. As a result, when the recognition engine 904 identifies one or more parameters associated with a command in the speech input, it may determine that the confidence level for the type of command is higher than the confidence level for the type of word. In some examples, the recognition engine 904 can determine the confidence level based on the context data 916. In some examples, if no match is found, the recognition engine 904 may be configured to determine that the utterance corresponds to a word (eg, a natural language word) as a default configuration. In some examples, if no match is found, recognition engine 904 may also be configured to determine one or more candidate types. Candidate types can be determined, for example, based on the reliability levels of each type. In some examples, candidate types may be used to obtain candidate representations of the content of the spoken input. Candidate representations may be in the form of audio representations, text representations, space/vector representations, or a combination thereof. Candidate expressions can be presented to the user for selection.

It will be appreciated that the determination of the type of utterance may be performed for a letter, phrase, word, or command in any combination or in any order. For example, the recognition engine 904 may first determine whether the type of speech corresponds to a character. If the type of speech does not correspond to a letter, it further determines whether the type corresponds to a phrase, word, etc. As another example, the recognition engine 904 may also first determine whether the type of speech corresponds to a phrase. If the type of speech does not correspond to a phrase, it further determines whether the type corresponds to a letter, word, etc. Decisions can also be performed in parallel for letters, phrases, words, or commands.

Upon obtaining the expression, the recognition engine 904 may convert the expression into text. Upon conversion to text, the recognition engine 904 may cause the text to be displayed.

In some examples, identifying the content of the voice input may include determining a confidence level. The confidence level may indicate, for example, a probability that the content of the voice input will be correctly identified. In some examples, the confidence level may be associated with the identified content of the spoken input. As described, in some examples, identifying the content of the spoken input includes identifying the utterance of the spoken input, determining the type of utterance, and/or the content of the spoken input, described in more detail below. It may include obtaining an expression of Accordingly, the confidence level may be associated with the identification of the utterance, the determined type of utterance, the obtained representation of the content of the speech input, or a combination thereof, accordingly. For example, the recognition engine 904 may recognize the content as including a specific type of utterance when the associated confidence level is greater than or equal to a threshold. Conversely, the recognition engine 904 may not recognize the content as including any type of utterance if the associated confidence level is not above the threshold. Instead, the recognition engine 904 may ignore the identified utterance type and provide one or more candidate types of utterance, an error message, or a combination thereof. Candidate types can be determined, for example, based on the reliability levels of each type. Candidate types may be presented to the user in a visual form, an audible form, or a combination thereof. In some examples, the recognition engine 904 may also cause an error message (eg, “Sorry, I missed that. Can you say it again?”) to be provided to the user.

In some examples, the recognition engine 904 can obtain a confidence level for the representation of the speech input content. As previously described, the confidence level represents the degree of certainty that the determination of the type of speech is correct. Upon determining that the confidence level is above the threshold, the recognition engine 904 converts the representation to text. In some examples, upon determining that the confidence level is below a threshold, the recognition engine 904 can identify one or more candidate types for obtaining corresponding candidate representations. Candidate expressions may be provided to the user for selection. In some examples, the recognition engine 904 identifies candidate expressions, an error message (eg, “I do not recognize what you said, please say a letter or a phrase such as “a as in apple””), or a combination thereof. In some examples, candidate expressions are intermediate recognition results (eg, phonemes, phonetic strings, and sub-words) and a known type stored in knowledge repository 910 (eg, using a language model). It may be generated based on matching between them.

In some examples, the recognition engine 904 can identify the command based on the spoken input. As described above, in some embodiments, the recognition engine 904 may first identify the utterance of the speech input and determine whether the type of utterance corresponds to the command. Upon determining that the type of utterance corresponds to the command, the recognition engine 904 may cause the action to be initiated according to the command. For example, the recognition engine 904 may cause previously displayed text to be edited (eg, insert or delete characters).

In some examples, the recognition engine 904 may also cause one or more utterances to be stored for learning and/or training purposes. The utterances may be stored, for example, in memory (eg, memory 202) and/or in a database. One or more utterances correspond to characters identified by recognition engine 904, phrases, commands, and/or words that identify the corresponding characters in some examples. These utterances (eg, acoustic characteristics of the utterances) may be used to update at least a portion of the knowledge repository 910 so that the type of subsequent user utterances can be more accurately determined.

5. Examples of voice input identification

10A-10F illustrate the function of voice input identification according to various examples. In accordance with one or more examples described herein, an electronic device, such as device 800 and/or remote control 810 of FIG. 8A, provides colloquial user inputs (eg, voice inputs, speech inputs) provided by a user Can be controlled using The device 800, remote control 810, server (eg, server 108), or a combination thereof, may implement a digital assistant system (digital assistant system 700). Referring to FIGS. 10A-10F, in some examples, the device 800 may receive one or more voice inputs (eg, 1024A to 1024F) from the user 1030. User 1030 may, for example, provide voice inputs for the purpose of dictating or otherwise entering user credentials (eg, username and/or password). Display 814 may be used to prompt and/or allow user 1030 to enter a user name and password for authentication purposes. A user may be prompted to enter credentials to log in to a device, account, service, etc., for example. The device 800 may receive a voice input directly from the user 1030 or indirectly through another device such as the remote control 810. The voice inputs 1024A to 1024F may be received through the remote control 810 or the microphone of the device 800 (eg, the microphone 213). In some examples, the device 800 is capable of identifying the content (e.g., a character, a phrase identifying a character, a command, a word, or a combination thereof) of the voice inputs 1024A-1024F according to the examples described herein. I can. The identification of the content of the voice inputs 1024A-1024F may be performed by a digital assistant including, for example, an automatic speech recognition system (eg, ASR system 900).

Referring to FIG. 10A, in some examples, device 800 may identify a character based on a voice input. As an example, upon receiving voice input 1024A, device 800 may identify one or more utterances of voice input 1024A. As shown, the identified utterance of voice input 1024A may include the letter “a”.

Upon identifying one or more utterances, device 800 may determine the type of utterance. In some embodiments, the determination may include characters stored in the knowledge repository, such as commonly used alphabetic characters, numbers, and control characters, and/or a character previously received by the device 800 from the user 1030. Can be based on Characters included in the knowledge store 910 may be stored on the device 800 or a server (eg, server 108). For example, the device 800 may determine whether the type of speech corresponds to the character by matching a sequence of phonemes of speech (eg, pronunciation of a character) with a phoneme sequence of a known character.

In some examples, upon determining that the type of utterance corresponds to a character, device 800 identifies a character (eg, character “a”) as included in the content of speech input 1024A, and converts the character to text. Convert to

In response to the conversion to text, the device 800 may cause the text of the content of the voice input 1024A (eg, the text of the letter “a”) to be displayed, for example, using the display 814. .

Referring to FIG. 10B, in some examples, device 800 may identify a character identified by a phrase. As an example, upon receiving voice input 1024B, device 800 may identify one or more utterances of voice input 1024B. As shown, the voice input 1024B may include a phrase (eg, "b as in boy") for identifying a corresponding letter (eg, "b") of the phrase.

Upon identifying one or more utterances of speech input 1024B, device 800 may determine the type of utterance. In some embodiments, the decision may be based on phrases stored in the knowledge repository, such as commonly used phrases. Commonly used phrases for identifying characters are spelled words as specified in NATO phonetic symbols (e.g. Alpha, Bravo, Charlie, Delta, Echo, Foxtrot, Golf, Hotel, India, Juliet, Kilo, Lima. , Mike, November, Oscar, Papa, Quebec, Romeo, Sierra, Tango, Uniform, Victor, Whiskey, X-ray, Yankee, Zulu).

In some embodiments, passages in the knowledge repository may also include previously received passages. For example, user 1030 may not be able to use NATO's phonetic symbols as spelled words to identify characters. Instead, user 1030 may have previously used custom phrases such as "a as in Aaron" or "l like Lincoln". Device 800 may also store such custom phrases that user 1030 previously provided to the knowledge repository. As a result, when the user 1030 utters a custom phrase that has been used previously, the device 800 may also determine that the type of utterance corresponds to the phrase for identifying the corresponding character. Phrases for identifying corresponding characters in the knowledge store may be stored on device 800 or server (eg, server 108).

As described, the type of utterance may be determined in response to the user providing the voice input. For example, the device 800 may determine whether the type of speech corresponds to the phrase by matching a sequence of phonemes (eg, pronunciation of the phrase) of the utterance to the phoneme sequence of the phrase in the knowledge repository.

In some examples, upon determining that the type of speech corresponds to the phrase for identifying the corresponding character, the device 800 identifies the phrase as included in the content of the spoken input 1024B, and identifies the phrase identified by the phrase. Convert characters to text. In response to the conversion to text, the device 800 may cause the text of the content of the voice input 1024B (eg, the text of the letter “b”) to be displayed, eg, using the display 814. .

Referring to FIG. 10C, in some examples, device 800 may identify additional characters and phrases based on the additional voice input. As an example, upon receiving voice input 1024C, device 800 may identify one or more utterances of voice input 1024C. As shown, the voice input 1024C may include additional characters (eg, “r”, “a”, “h” and “a”) and additional phrases (eg, “m as in Mary”). have. As described, in response to receiving the voice input 1024C, the device 800 identifies one or more utterances of the voice input 1024C, determines the type of utterances, and determines the content of the voice input 1024C. The identified characters may be converted into text for display on the display 814.

In some examples, device 800C can identify a punctuation word based on the spoken input. Upon receiving speech input 1024C, for example, device 800 may identify one or more utterances of speech input 1024C. As shown, spoken input 1024C may include the punctuation word “at” that identifies (eg, corresponds to) the punctuation “@”.

Upon identifying one or more utterances of speech input 1024C, device 800 may determine the type of utterance. In some embodiments, the determination may include punctuation words stored in the knowledge repository, such as commonly used punctuation words (eg, “at”, “pound-sign”, “dollar-sign”, etc.) and/or a device ( 800) may be based on custom punctuation words previously received from user 1030. For example, instead of the punctuation word "asterisk" corresponding to the punctuation mark "*", the user 1030 may have uttered the punctuation word "star" in advance, also intending to correspond to the punctuation mark "*". Such custom punctuation words can also be stored in the knowledge repository. Punctuation words included in the knowledge store may be stored on the device 800 or a server (eg, server 108).

As described, the type of utterance may be determined in response to the user providing the voice input. For example, the device 800 may match a sequence of phonemes of speech (eg, pronunciation of a character) to a phoneme sequence of a punctuation word included in the knowledge storage 910 to determine whether the type of speech corresponds to a punctuation word. Can be determined. Under some circumstances, there may be more than one substantial match. Thus, the device 800 may disambiguate matches using confidence levels and/or based on the context of the punctuation word. As an example, device 800 may determine which match is appropriate based on one or more other utterances identified from a voice input provided by user 1030. As another example, the device 800 may determine a reliability level for each matched type, and select a type having the highest reliability level. For example, if a sequence of phonemes matches both a punctuation word (e.g., a punctuation word "at") and a word (e.g., a natural language word "at"), then the device 800 has confidence in the type as a punctuation word. It is possible to determine the level of confidence in the level and type as a word. In some embodiments, the confidence level may be determined based on additional utterances at speech input 1024C. For example, using the recognized credential models, the device 800 may determine that the user intends to provide an email address as the username in voice input 1024C. Accordingly, the device 800 may determine that the confidence level for the type of the punctuation word is higher than the confidence level for the type of the natural language word.

In some examples, upon determining that the type of utterance corresponds to a punctuation word, device 800 identifies a punctuation word (eg, the word “at”) in the content of the spoken input 1024C and corresponds to the punctuation word. Convert to text in punctuation. For example, the device 800 may convert the punctuation word "at" into text of the punctuation mark "@". Referring to FIG. 10C, in response to the conversion, the device 800 causes the text of the content of the voice input 1024C (eg, the text of the punctuation “@”) to be displayed, for example, using the display 814. can do.

Referring to FIG. 10D, in some examples, device 800 may identify one or more words based on the spoken input, for example by identifying one or more utterances of spoken input 1024D. As shown, voice input 1024D may include one or more words, each of which may include any combination of a plurality of letters. The words may be natural language words such as "Lincoln" and "center", for example.

Upon identifying the utterances of the speech input 1024D, the device 800 may determine the type of utterances. In some examples, the decision is made by words stored in the knowledge repository, such as commonly used words for entering credentials (eg, "yahoo", "gmail", and "aol") and/or by the user 1030. Can be based on the words spoken For example, the user 1030 may have previously uttered the words "Lincoln" and "center" to enter the user name. Thus, the words "Lincoln" and "center" are stored in the knowledge repository. Words included in the knowledge store may be stored on the device 800 or a server (eg, server 108).

The device 800 may determine whether the type of speech corresponds to a word by matching a sequence of phonemes (eg, pronunciation of a word) of the speech to a phoneme sequence of a word included in the knowledge storage.

In some examples, device 800 can determine whether the type of utterance corresponds to a word based on a predetermined composition. For example, the device 800 may be preconfigured to determine the type of utterance so that the default type of utterance corresponds to a word. For example, upon receiving a voice input, the device 800 identifies one or more utterances, and whether the type of each identified utterance corresponds to a character (e.g., a letter, number, or control character, such as punctuation). , It may be determined whether it corresponds to a phrase identifying the corresponding character or a command. If device 800 determines that the type of utterance does not correspond to any of these types, device 800 may be configured to determine, by default, that the type of utterance corresponds to a word.

In some examples, upon determining that the type of utterance corresponds to a word, device 800 identifies a word (eg, word “Lincoln”) and converts the identified word to text. Referring to FIG. 10D, upon conversion to text, the device 800 may use the display 814 to display the text of the content of the voice input 1024D (eg, the text of the word “Lincoln”) have.

Referring to FIG. 10E, in some examples, device 800 may identify additional punctuation words, for example by identifying one or more utterances in spoken input 1024E. As shown, the spoken input 1024E may include additional punctuation words (eg, punctuation words “dot”). In response to receiving speech input 1024E, device 800 can identify the utterance, determine the type of utterance as a punctuation word, and convert the utterance into text for display on display 814.

In some examples, device 800 can also identify a word having a predetermined combination of letters based on the spoken input. As an example, upon receiving voice input 1024E, device 800 can identify the utterance of voice input 1024E. As shown, the voice input 1024E may include words having a predetermined combination of characters (eg, domain extension). As described, words with a predetermined combination of letters are, for example, a plurality of domain extensions (eg, "com", "net", and "org"), acronyms (eg, FBI, IRS), etc. May contain characters.

Upon identifying the utterance based on the voice input 1024E, the device 800 may determine the type of utterance. In some examples, the decision may be based on words previously recognized and stored in a knowledge repository (eg, knowledge repository 910 ). As an example, these words may include some commonly used words for entering credentials (eg, domain extensions “com”, “net”, and “org”). As another example, these words may include words that the user 1030 has previously uttered, such as the acronyms “FBI”, “IRS”, and “USPTO”. Thus, words that have been previously spoken can also be stored in the knowledge repository. Words included in the knowledge store may be stored on the device 800 or a server (eg, server 108).

As described, in response to the user providing the voice input, the type of utterance may be determined. For example, as described above, the device 800 may determine whether the type of utterance corresponds to a word having a predetermined letter combination by matching the sequence of phonemes of the utterance with the phoneme sequence of the word in the knowledge repository. have.

In some examples, upon determining that the type of utterance corresponds to a word with a predetermined letter combination, device 800 identifies a word with a predetermined letter combination (eg, "com") and converts the word into text. Convert. Referring to FIG. 10E, in response to conversion to text, the device 800 uses the display 814 to cause the text of the content of the voice input 1024E (eg, the text of the word “com”) to be displayed. I can.

Referring to FIG. 10F, in some examples, device 800 may identify additional characters, punctuation words, and phrases. Device 800 can also identify the command. As an example, upon receiving speech input 1024F, device 800 may identify one or more utterances in speech input 1024F. As shown, the voice input 1024F includes letters, phrases, and commands (e.g., letter "a", punctuation word "asterisk", letter "b", command "upper case", letter "c", And a mixture of the punctuation words "dollar sign", the numbers "1", "2", and "3", and the punctuation word "percent-sign"). In response to receiving the voice input 1024F, the device 800 is configured with the letter "a", the punctuation word "asterisk", the letter "b", the punctuation word "dollar sign", the numbers "1", "2", And "3", and the utterances of the punctuation word "percent sign". Device 800 also determines the type of these utterances and

It can be converted into text for display on the display 814.

Referring to FIG. 10F, the device 800 may also identify a command based on the voice input, for example, by identifying a speech in the voice input 1024F. As shown, the voice input 1024F may include a command (eg, command “upper case”). The command makes it possible to edit the displayed text or perform any desired functions with respect to the displayed text. Exemplary commands include "upper case", "lower case", "insert", "delete", "replace", "copy", "paste", "select", "search", and the like.

Upon identifying the utterance based on the voice input 1024F, the device 800 may determine the type of utterance. In some examples, the decision may be based on commands previously recognized and stored in a knowledge store (eg, knowledge store 910 ). In some embodiments, these commands may include commonly used commands such as “insert”, “delete”, “upper case”, “lower case”, and the like. These commands may also include commands previously received by the device 800 from the user 1030. Commands included in the knowledge store may be stored on the device 800 or a server (eg, server 108). The device 800 may determine whether the type of utterance corresponds to the command by matching a sequence of phonemes (eg, pronunciation of the command) of the utterance to the phoneme sequence of the command in the knowledge storage.

In some examples, upon determining that the type of utterance corresponds to the command, device 800 may cause the action to be initiated according to the command. For example, referring to FIG. 10F, according to a determination that the type of utterance of the “upper case” corresponds to the command, the device 800 displays the contents of the voice input 1024F (eg, the utterance of the letter “c”). It can be converted to text, and an action can be initiated to convert the text of "c" to its upper case form "C".

11A-11E illustrate the function of voice input identification according to various examples. Referring to FIG. 11A, in some examples, device 800 can identify additional characters and phrases. Upon receiving speech input 1144A, for example, device 800 may identify one or more utterances in speech input 1144C. As shown, voice input 1144A includes letters, phrases, and commands (e.g., the letter “a”, the phrase “b as in boy”, the letters “r”, “a”, and “h”). ) May contain a mixture of. As described, in response to receiving voice input 1144A, device 800 identifies the utterances of the letters "a", "r", and "a", and the phrase "b as in boy" , Convert them to text, and cause the text to be displayed on the display 814.

Device 800 may also identify additional commands based on the voice input. As an example, device 800 may inadvertently identify the letter “h” as the letter “s”, thus causing the letter “s” to be displayed on the display 814 instead of the letter “h”. As shown in Fig. 11A, the user 1130 wishes to correct such an error, and thus, in the voice input 1144A, may utter “delete” intent to delete the letter “s”. In some examples, user 1130 may alternatively utter “h as in Henry” to more clearly emphasize the user's intended input.

In some examples, upon determining that the type of utterance in the voice input 1144A corresponds to the command, the device 800 may initiate an action according to the command. For example, referring to FIG. 11A, according to a determination that the type of utterance in the voice input 1144A corresponds to a command (eg, command “delete”), the device 800 is displayed on the display 814. May cause the immediately preceding character or the current character (eg, the character “s”) to be deleted. In some examples, if more than one “delete” command is identified, the device 800 The preceding characters and the current character can be deleted. For example, if the voice input 1144A includes two consecutive commands “delete”, the device 800 causes the two immediately preceding characters (eg, characters “s” and “a”) to be deleted. Can do; When the voice input 1144A includes three consecutive commands “delete”, the device 800 deletes the three immediately preceding characters (eg, characters “s”, “a”, and “r”). Can be made, and so on.

In some examples, after providing a voice input comprising the command "delete", the user 1030 adds, for example, to clarify the user's intent using a phrase (eg, "h as in Henry"). Voice input can be provided. Upon receiving the voice input 1144A, the device 800 can identify the correct character (eg, “h”) based on the voice input 1144A. For example, device 800 may identify the utterance of a phrase, convert the correct character into text based on the identification, and cause display of the text on display 814. As shown in FIG. 11A, the device 800 causes the display 814 to display the voice input 1144A as a result of performing an action according to a command (eg, deleting “s” and displaying “h”). The text of the content can be displayed.

11B, in some examples, device 800 may identify additional commands. As an example, the user 1130 may wish to insert the letter "a" and thus may utter "insert'a".

In some examples, upon determining that the type of utterance corresponds to a command (eg, “insert”), device 800 may cause an action to be initiated according to the command. For example, as shown in FIG. 11B, according to a determination that the type of speech of the voice input 1144B corresponds to the command, the device 800 may display a character (eg, “a”) following the command “insert”. It may be inserted at a position immediately before the current character displayed on the display 814 (eg, before the character “m”). As shown in FIG. 11B, the device 800 is as a result of performing an action according to a command (eg, inserting the letter “a” before the letter “m”) (eg, using the display 814) The text of the voice input 1144B may be displayed.

Referring to FIG. 11C, in some examples, commands may be provided in combination with one or more parameters. As an example, the user 1130 may not recognize an inappropriate input until after most of the voice input 1144C has been provided by the user 1130. For example, when the user 1130 tries to input a user name, the user 1130 may not be aware that the letter “a” is missing until after uttering “dot” and “com”. As a result, the user 1130 may wish to insert the letter “a” between the letters “h” and “m” in the word “abraham”. Accordingly, the user 1130 may provide a voice input 1144C that includes the command “insert” and further specifies “'a' between'h' and'm' before'at'”. In some examples, the voice input 1144C may include a command such as "insert'a' between'h' and'm' in the word'abraham'". As described, upon receiving speech input 1144C, device 800 can identify utterances in speech input 1144C.

Upon identifying the utterances, the device 800 may also determine whether the types of utterances correspond to a command (eg, “insert”) and one or more associated parameters. In some examples, device 800 may first determine whether one of the utterances corresponds to a command (eg, “insert”). Upon determining that the type of utterance corresponds to the command, the device 800 may further determine whether the command carries one or more parameters for performing an action according to the command. Such parameters are, for example, the character to be inserted or deleted (eg, "a"), and the edit position (eg, "'between''h' and'm' before'at'", "between'h' and ' m'in the word'abraham'"). In some examples, determining whether a command carries one or more parameters may enable actions to be initiated according to the user's intention, rather than according to a default configuration. For example, as described below, the device 800 does not depend on a command and one or more default configurations (e.g., default configurations such as deleting the previous character, inserting after the current character, etc.) Actions can be initiated according to parameters.

Upon determining that the command carries one or more parameters, device 800 may cause an action to be initiated in response to the command and parameters. For example, as shown in FIG. 11C, according to a determination that the type of utterance of "insert" corresponds to a command and a determination that there are parameters accompanying the "insert" command, the device 800 may receive a command (eg, a letter To insert a corresponding letter such as "a") and parameters (eg, between "h" and "m" in the word "abraham") may cause an action to be initiated. Referring to FIG. 11C, the device 800 voices (e.g., using the display 814) as a result of performing an action according to a command (e.g., inserting the letter "a" in the correct position) and parameters. The text of the content of the input 1144C may be displayed.

Referring to FIG. 11D, the device 800 may identify one or more additional commands. For example, the user 1130 may repeat the previous character. For example, in the voice input 1144D, in order to input the letter "b" twice in the password, the user 1130 may utter "b as in boy" and the following command "repeat".

As described, upon determining that the type of utterance corresponds to the command, the device 800 may cause the action to be initiated according to the command. For example, as shown in FIG. 11D, the device 800 may cause an action to be initiated according to a command (eg, to repeat a previous character). Referring to FIG. 11D, the device 800 may cause the text of the content of the voice input 1144D to be displayed as a result of performing an action according to a command (eg, displaying an additional letter “b”).

10A-10F and 11A-11D are described using device 800 as an example, but one or more of the digital assistant functions and processes described herein are alternatively a server (e.g., a server ( 108)) or a combination of device 800 and server. For example, upon receiving one or more of the voice inputs 1024A to 1024F, the device 800 may provide data representing the voice inputs 1024A to 1024F to the server 108, and the server ( 108) may identify the content of the voice inputs 1024A to 1024F. As an example, server 108 may identify one or more utterances, determine types of one or more utterances, and obtain representations of the content of speech inputs 1024A-1024F. In some examples, server 108 may provide representations of the content of voice inputs 1024A-1024F to device 800. Upon receiving the expressions of the content of the voice input 1024A-1024F from the server 108, the device 800 may convert the obtained expressions to text and cause the text to be displayed on the display 814. . In some examples, server 108 may convert representations of spoken inputs 1024A-1024F into text and provide the text to device 800. Device 800 may then cause text to be displayed on display 814.

6. Process to identify voice input

12A-12D show a flow diagram of an exemplary process 1200 for voice input identification in accordance with some embodiments. Process 1200 uses one or more devices 104, 108, 200, 400, 600, 800, or 810 (FIGS. 1, 2A, 4, 6A and 6B, or 8A and 8B). Can be done by The operations in process 1200 are selectively combined or divided and/or the order of some operations is selectively changed.

Referring to FIG. 12A, at block 1202, a voice input is received. Voice input can include one or more utterances corresponding to letters, phrases, commands, and/or words. The voice input may correspond to at least one of a first user credential indicating a user name or a second user credential indicating a password.

At block 1204, a first letter, a phrase identifying the second letter, and a word are identified based on the spoken input. In some examples, the first character includes at least one of letters, numbers, control characters, or spaces. Control characters may include punctuation, or non-punctuation characters (eg, tabs, carriage returns) identified by punctuation words. The phrase for identifying the second letter may include a spelled word for helping to identify the second letter. A word may comprise any combination of a plurality of letters representing, for example, a natural language word. Words may also include predetermined letter combinations indicating domain extensions and/or acronyms, for example.

In some examples, the first character based on the speech, the phrase identifying the second character, and the word are identified based on at least one of context data or one or more recognized credential models associated with the user. For example, the context data may include a user's calendar, contacts, user speech profile, or any other user-specific data. In some examples, the recognized credential models are recognized credential formats, such as a username format (eg, first_last@domain_name.com, first_name_last@domain_name.org), or a password format, For example, it can represent a mixture of letters, upper and lower case letters, control letters, and numbers.

In some examples, the first character based on the speech, the phrase identifying the second character, and the word are identified based on gaze detection. For example, while receiving a voice input from a user, it may be determined whether the user is looking at a certain direction or area (eg, by using a camera). If it is determined that the direction or area corresponds to the keyboard or key for entering credentials, the information in the direction or area the user is looking at helps to identify the first letter, the phrase identifying the second letter, and the word. Can be

At block 1206, the content of the speech input may be identified based on one or more speech recognition models stored on a user device (eg, device 800 in FIGS. 10A-10F and 11A-11D ). In some examples, identification may be performed on a user device without communicating with a server (eg, server 108).

At block 1208, in some examples, identification of the content of the voice input may be performed using a user device and a server. For example, identification may include providing data indicative of voice input from a user device to a server. The server may identify at least one of a first character, a phrase for identifying the second character, and a word based on one or more language models stored in the server. Then, the user device receives, from the server, data representing at least one of the identified first character, the identified phrase identifying the second character, or the identified word.

Referring to FIG. 12B, at block 1212, identifying the content of the voice input includes identifying the utterance of the voice input. At block 1214, the identification of the first letter, the phrase that identifies the second letter, and the word includes determining the type of speech.

At block 1216, it is determined whether the type of speech corresponds to a recognized character. At block 1218, for the utterance, it is determined whether the utterance corresponds to a control character. At block 1220, according to a determination that the utterance corresponds to a control letter, for the utterance, a determination is made whether the utterance corresponds to a punctuation word.

In block 1221, upon determining that the utterance corresponds to the punctuation word, a representation of the punctuation mark corresponding to the punctuation word is obtained. At block 1222, upon determining that the utterance does not correspond to a punctuation word, a representation of a control character other than the punctuation word is obtained.

At block 1224, it is determined whether the type of utterance corresponds to the recognized phrase identifying the second letter. At block 1226, it is determined whether the type of speech corresponds to a word.

At block 1228, based on the determined type of utterance, a representation of the first character corresponding to the utterance, a second character identified by the phrase, or a word is obtained. In some examples, the representation can include an audio representation, a text representation, a space/vector representation, or a combination thereof.

At block 1230, a confidence level for the presentation is obtained. The confidence level represents the degree of accuracy of the identification. At block 1232, it is determined whether the confidence level is equal to or greater than a threshold. In block 1234, upon determining that the confidence level is equal to or greater than the threshold, the first character corresponding to the utterance, the second character identified by the phrase, or the representation of the word is converted to text. At block 1236, upon determining that the confidence level is below a threshold, one or more candidate expressions, an error message, or a combination thereof may be provided.

At block 1238, at least one command may be identified. The command enables editing of the displayed text. At block 1240, at least one utterance is identified to identify the command. At block 1242, it is determined whether the type of utterance corresponds to the recognized command. At block 1244, upon determining that the type of utterance corresponds to the recognized command, an action is initiated according to the recognized command to edit the displayed text.

At block 1246, the utterance corresponding to the at least one command and information associated with the utterance corresponding to the at least one command are stored. The information associated with the utterance corresponding to the at least one command may include, for example, the number of times the user utters the command “delete”. That information can indicate the degree of accuracy of the identification.

At block 1248, in response to the identification, the first character, the second character, and the word are converted to text. At block 1250, the text is displayed on the display in a sequence corresponding to the order of the first letter, the second letter, and the word in the voice input. For example, the voice input may be provided in a sequence of a first letter, a word that follows, and a phrase that identifies the second letter that follows. Thus, the displayed text is in the order of the first character, followed by the word, and followed by the second character.

At block 1252, one or more identified utterances of the spoken input may be stored, ie, the first characters, the phrase identifying the second character, and one or more identified utterances corresponding to the word. Storing the utterances can enable learning and training of the digital assistant to improve the accuracy of identification.

7. Electronic device

13 shows a functional block diagram of an electronic device 1300 configured according to the principles of various described examples, including those described with reference to FIGS. 9, 10A-10F, and 11A-11D. . The functional blocks of the device may, optionally, be implemented by hardware, software, or a combination of hardware and software for carrying out the principles of the various described examples. Those skilled in the art will appreciate that the functional blocks described in FIG. 13 may, optionally, be combined or separated into sub-blocks to implement the principles of the various described examples. Accordingly, the description herein supports, optionally, any possible combination, separation or further definition of the functional blocks described herein.

As shown in FIG. 10, electronic device 1300 may include a microphone 1302 and a processing unit 1308. In some examples, the processing unit 1308 includes a receiving unit 1310, an identification unit 1312, a conversion unit 1314, and a causing unit 1316, and optionally, a providing unit 1318, determining It includes a unit 1320, an acquisition unit 1322 and a storage unit 1324.

The processing unit 1308 is configured to receive a voice input (eg, to the receiving unit 1310 ); And based on the voice input, (eg, with the identification unit 1312), the first character, the phrase identifying the second character, and the word are identified. The processing unit, in response to the identification, converts the first character, the second character, and the word into text (eg, to the conversion unit 1314 ); And using the display (eg, to the ministry unit 1316) to display the text in a sequence corresponding to the order of the first character, the second character, and the word in the voice input.

In some examples, the first character includes at least one of letters, numbers, control characters, or spaces.

In some examples, the control character includes a punctuation mark identified by a punctuation word.

In some examples, the phrase includes a spelled word to aid in the identification of the second letter.

In some examples, the word includes a plurality of letters.

In some examples, the plurality of characters represents at least one of a domain extension or an acronym.

In some examples, identifying the first letter, the phrase identifying the second letter, and the word is based on one or more speech recognition models stored in the user device (e.g., with the identification unit 1312), the first letter, And identifying at least one of a phrase or word that identifies the second letter.

In some examples, identifying the first character, the phrase identifying the second character, and the word may include providing data indicative of the spoken input to the server (eg, to the providing unit 1318 ); And receiving data representing at least one of the identified first character, the identified phrase identifying the second character, or the identified word, from the server (eg, to the receiving unit 1310).

In some examples, identifying the first letter, the phrase identifying the second letter, and the word is based on one or more language models stored in the server (e.g., with the identification unit 1312). And identifying at least one of a phrase or word that identifies a letter.

In some examples, based on the spoken input, identifying the first character, the phrase identifying the second character, and the word may include (eg, with identification unit 1312) identifying a utterance of the spoken input; Determining the type of utterance (eg, with determining unit 1320); And (e.g., to the obtaining unit 1322), based on the determined type of utterance, obtaining a first letter corresponding to the utterance, a second letter identified by the phrase, or a representation of the word.

In some examples, determining the type of utterance includes (eg, with determining unit 1320) determining whether the type of utterance corresponds to a recognized character; Determining whether the type of utterance corresponds to the recognized phrase identifying the second character (eg, with determining unit 1320); And (e.g., with determining unit 1320) determining whether the type of speech corresponds to a word.

In some examples, determining whether the type of utterance corresponds to a recognized character includes, for the utterance (eg, with determining unit 1320 ), determining whether the utterance corresponds to a control character; And, according to the determination that the utterance corresponds to the control letter, for the utterance (eg, to the determining unit 1320), determining whether the utterance corresponds to a punctuation word.

In some examples, based on the determined type of utterance, acquiring the representation of the punctuation corresponding to the punctuation word (e.g., to the acquiring unit 1322) according to the determination that the utterance corresponds to the punctuation word Acquiring, and in accordance with the determination that the utterance does not correspond to a punctuation word, (e.g., to the acquiring unit 1322) acquiring a representation of a control character other than the punctuation word.

In some examples, based on the spoken input, identifying the first letter, the phrase identifying the second letter, and the word may include obtaining a confidence level for the expression (eg, with the obtaining unit 1322 ); And (e.g., with determining unit 1320) determining whether the confidence level is equal to or greater than a threshold.

In some examples, converting a first character, a second character, and a word to text is a first character, phrase corresponding to the utterance (e.g., with conversion unit 1314), depending on the determination that the confidence level is above a threshold. Converting the expression of the second character, or word identified by, into text; And providing one or more candidate expressions, an error message, or a combination thereof, depending on the determination that the confidence level is less than the threshold value (eg, to the providing unit 1318 ).

In some examples, processing unit 1308 is further configured to identify at least one command (eg, with identification unit 1312 ), the at least one command enabling editing of the displayed text.

In some examples, identifying the at least one command includes (eg, with identification unit 1312) identifying the at least one utterance; Determining whether the type of utterance corresponds to the recognized command (e.g., to the determining unit 1320); And upon determining that the type of utterance corresponds to the recognized command, causing the action to be initiated according to the recognized command to edit the displayed text (eg, with the ministry unit 1316 ).

In some examples, the processing unit 1308 is to store the utterance corresponding to the at least one command (e.g., to the storage unit 1324) and the utterance corresponding to the at least one command (e.g., to the storage unit 1324). It is further configured to store information associated with.

In some examples, based on the spoken input, identifying the first letter, the phrase identifying the second letter, and the word is based on at least one of context data associated with the user or one or more recognized credential models.

In some examples, the context data associated with the user includes a user speech profile.

In some examples, the one or more recognized credential models represent one or more recognized credential formats.

In some examples, based on the voice input, identifying the first character, the phrase identifying the second character, and the word is based on gaze detection.

In some examples, the voice input corresponds to at least one of a first user credential representing a user name or a second user credential representing a password.

In some examples, the processing unit 1308 (e.g., to the storage unit 1324) is one or more utterances of the spoken input, i.e., the first characters, the phrase identifying the second character, and one or more corresponding to the word. It is further configured to store the utterances.

The operation described above with respect to FIG. 13 is, optionally, implemented by the components shown in FIGS. 1, 2A, 4, 6A and 6B, 7A, 8A and 8B, or 9. For example, receive operation 1310, identification operation 1312, transform operation 1314, and ministry operation 1316 are optionally implemented by processor(s) 220. It will be apparent to those skilled in the art how other processes may be implemented based on the components shown in FIGS. 1, 2A, 4, 6A and 6B, 7A, 8A and 8B, or 9.

Those skilled in the art will appreciate that the functional blocks described in FIG. 13 may, optionally, be combined or separated into sub-blocks to implement the principles of the various described embodiments. Accordingly, the description herein, optionally, supports any possible combination or separation or further definition of the functional blocks described herein. For example, processing unit 1308 may have an associated “controller” unit operatively coupled with processing unit 1308 to enable operation. Such a controller unit is not separately shown in FIG. 13, but it will be appreciated that it is within the understanding of those skilled in the art of designing a device having a processing unit 1308, such as device 1300. As another example, one or more units, such as receiving unit 1310, may be hardware units external to processing unit 1308 in some embodiments. Accordingly, the description herein optionally supports combinations, separations, and/or further definitions of the functional blocks described herein.

Exemplary methods, non-transitory computer-readable storage media, systems, and electronic devices are described in the following items:

Item 1. A method for identifying a spoken input providing one or more user credentials, comprising:

In a user device with a display, one or more processors and memory:

Receiving a voice input;

Identifying a first character, a phrase identifying the second character, and a word based on the voice input;

In response to the identification, converting the first character, the second character, and the word into text; And

Using the display to display text in a sequence corresponding to an order of the first letter, the second letter and the word in the spoken input.

Item 2. The method of item 1, wherein the first character comprises at least one of letters, numbers, control characters, or spaces.

Item 3. The method of item 2, wherein the control characters include punctuation marks identified by punctuation words.

Item 4. The method of any of items 1 to 3, wherein the phrase comprises a spelled word to aid in the identification of the second letter.

Item 5. The method of any of items 1 to 4, wherein the word comprises a plurality of letters.

Item 6. The method of item 5, wherein the plurality of characters represents at least one of a domain extension or an acronym.

Item 7. According to any one of items 1 to 6, the step of identifying the first letter, the phrase identifying the second letter, and the word, based on the voice input, comprises:

Identifying at least one of a first character, a phrase identifying a second character, or a word based on one or more speech recognition models stored on the user device.

Item 8. According to any one of items 1 to 7, the step of identifying the first character, the phrase identifying the second character, and the word, based on the voice input, comprises:

Providing data representing voice input to a server; And

Receiving, from the server, data representing at least one of a first character, a phrase identifying a second character, or a word,

Based on the voice input, the step of identifying a first character, a phrase identifying the second character, and a word,

And identifying at least one of a first letter, a phrase that identifies a second letter, or a word based on one or more language models stored on the server.

Item 9. According to any one of items 1 to 8, the step of identifying the first letter, the phrase identifying the second letter, and the word, based on the voice input, comprises:

Identifying a speech in the speech input;

Determining the type of utterance; And

Based on the determined type of utterance, obtaining a first letter corresponding to the utterance, a second letter identified by the phrase, or a representation of a word.

Item 10. In item 9, the step of determining the type of utterance comprises:

Determining whether the type of speech corresponds to a character;

Determining whether the type of speech corresponds to a phrase identifying the character; And

Performing at least one of determining whether the type of speech corresponds to a word.

Item 11. The method of item 10, the step of determining whether the type of speech corresponds to a character,

For the speech, determining whether the speech corresponds to a control character; And

In accordance with the determination that the utterance corresponds to the control letter, for the utterance, determining whether the utterance corresponds to a punctuation word,

Based on the determined type of speech, obtaining a representation of the character,

According to a determination that the utterance corresponds to the punctuation word, obtaining a representation of the punctuation corresponding to the punctuation word, and

In accordance with the determination that the utterance does not correspond to a punctuation word, obtaining a representation of a control character other than the punctuation word.

Item 12. The method of any one of items 9 to 11, wherein, based on the voice input, identifying the first letter, the phrase identifying the second letter, and the word,

Obtaining a confidence level for the expression; And

Further comprising the step of determining whether the confidence level is greater than or equal to a threshold,

Converting the first character, the second character, and the word into text,

Converting a first character corresponding to the utterance, a second character identified by the phrase, or an expression of a word into text according to a determination that the reliability level is equal to or greater than the threshold value; And

Providing one or more candidate expressions, an error message, or a combination thereof in response to a determination that the confidence level is below a threshold.

Item 13. According to any one of items 1 to 12,

Further comprising the step of identifying the at least one command based on the spoken input, wherein the at least one command enables editing of the displayed text.

Item 14. The method of item 13, wherein identifying the at least one command comprises:

Identifying at least one utterance;

Determining whether the type of speech corresponds to the command; And

Upon determining that the type of utterance corresponds to the command, causing to initiate an action corresponding to the command to edit the displayed text.

Item 15. According to item 14,

The method of claim 1, further comprising storing the utterance corresponding to the at least one command, and storing information associated with the utterance corresponding to the at least one command.

Item 16. The method of any one of items 1 to 15, wherein, based on the voice input, identifying the first letter, the phrase identifying the second letter, and the word comprises: context data associated with the user or one or more recognition Based on at least one of the established credential models.

Item 17. The method of item 16, wherein the context data associated with the user comprises a user speech profile.

Item 18. The method of item 16 or 17, wherein the one or more recognized credential models represent one or more recognized credential formats.

Item 19. The method of any of items 1 to 18, wherein based on the voice input, identifying the first character, the phrase identifying the second character, and the word is based on gaze detection.

Item 20. The method of any of items 1 to 19, wherein the voice input corresponds to at least one of a first user credential representing a user name or a second user credential representing a password.

Item 21. According to any one of items 1 to 20,

The method further comprising storing one or more utterances of the spoken input, wherein the one or more utterances correspond to first characters recognized based on the identification, a phrase identifying the second character, and a word.

Item 22. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by one or more processors of the electronic device, cause the electronic device to:

To receive voice input;

Identify a first letter, a phrase identifying the second letter, and a word based on the voice input;

In response to the identification, convert the first letter, second letter, and word into text;

A non-transitory computer-readable storage medium comprising instructions for causing, using a display, to display text in a sequence corresponding to an order of a first character, a second character, and a word in the spoken input.

Item 23. An electronic device comprising:

One or more processors;

Memory; And

One or more programs stored in memory and configured to be executed by one or more processors, the one or more programs,

For receiving voice input;

Based on the voice input, for identifying a first character, a phrase identifying the second character, and a word;

In response to the identification, for converting the first character, the second character, and the word into text; And

The electronic device comprising instructions for causing, using the display, to display text in a sequence corresponding to an order of a first character, a second character and a word in the speech input.

Item 24. An electronic device comprising:

Means for receiving voice input;

Means for identifying a first character, a phrase identifying the second character, and a word based on the spoken input;

Means for converting the first letter, second letter, and word to text in response to the identification; And

Means for causing, using the display, to display text in a sequence corresponding to an order of a first character, a second character and a word in the speech input.

Item 25. An electronic device, comprising:

One or more processors;

Memory; And

An electronic device comprising one or more programs stored in a memory, the one or more programs comprising instructions for performing the method of any one of items 1 to 21.

Item 26. A non-transitory computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs, when executed by one or more processors, cause the electronic device to: A non-transitory computer-readable storage medium comprising instructions for causing to perform the method of any of items 1 to 21.

Item 27. An electronic device, comprising:

An electronic device comprising means for performing any of the methods of items 1 to 21.

Item 28. An electronic device comprising:

It includes a processing unit, the processing unit,

To receive voice input;

Based on the spoken input, to identify a first letter, a phrase identifying the second letter, and a word;

In response to the identification, convert the first letter, the second letter, and the word into text; And

The electronic device, configured to display text in a sequence corresponding to an order of the first character, the second character and the word in the speech input.

Item 29. The electronic device of item 28, wherein the first character comprises at least one of letters, numbers, control characters, or spaces.

Item 30. The electronic device of item 29, wherein the control characters include punctuation marks identified by punctuation words.

Item 31. The electronic device of any of items 28 to 30, wherein the phrase comprises a spelled word to aid in identification of the second letter.

Item 32. The electronic device of any of items 28 to 31, wherein the word comprises a plurality of characters.

Item 33. The electronic device of item 32, wherein the plurality of characters represents at least one of a domain extension or an acronym.

Item 34. The item of any of items 28 to 33, wherein, based on the voice input, identifying the first letter, the phrase identifying the second letter, and the word,

The electronic device comprising identifying at least one of a first character, a phrase identifying a second character, or a word based on one or more speech recognition models stored in the user device.

Item 35. The method of any of items 28 to 34, wherein, based on the voice input, identifying the first letter, the phrase identifying the second letter, and the word,

Providing data indicative of voice input to the server; And

Receiving, from the server, data representing at least one of a first character, a phrase identifying a second character, or a word,

Based on the voice input, identifying the first character, the phrase identifying the second character, and the word,

The electronic device comprising identifying at least one of a first letter, a phrase identifying a second letter, or a word based on one or more language models stored on the server.

Item 36. The method of any of items 28 to 35, wherein, based on the voice input, identifying the first letter, the phrase identifying the second letter, and the word,

Identifying utterances of speech input;

Determining the type of utterance; And

The electronic device comprising obtaining, based on the determined type of utterance, a first character corresponding to the utterance, a second character identified by the phrase, or a representation of a word.

Item 37. For item 36, determining the type of utterance is:

Determining whether the type of speech corresponds to a character;

Determining whether the type of speech corresponds to the phrase identifying the letter; And

The electronic device comprising performing at least one of determining whether a type of speech corresponds to a word.

Item 38. In item 37, determining whether the type of speech corresponds to a letter,

For the utterance, determining whether the utterance corresponds to a control character; And

In accordance with the determination that the utterance corresponds to the control letter, for the utterance, determining whether the utterance corresponds to a punctuation word;

Based on the determined type of utterance, obtaining a representation of the character,

In accordance with the determination that the utterance corresponds to the punctuation word, obtaining an expression of the punctuation corresponding to the punctuation word, and

The electronic device comprising obtaining a representation of a control character other than the punctuation word upon determining that the utterance does not correspond to a punctuation word.

Item 39. The method of any one of items 36 to 38, wherein, based on the voice input, identifying the first letter, the phrase identifying the second letter, and the word,

Obtaining a level of confidence in the representation; And

Further comprising determining whether the confidence level is above a threshold,

Converting a first character, a second character, and a word to text,

Converting the first character corresponding to the utterance, the second character identified by the phrase, or the expression of the word into text according to a determination that the reliability level is equal to or greater than a threshold; And

The electronic device comprising providing one or more candidate expressions, an error message, or a combination thereof in response to a determination that the confidence level is below a threshold.

Item 40. The method of any one of items 28 to 39, wherein the processing unit:

The electronic device, further configured to identify the at least one command based on the spoken input, the at least one command enabling editing of the displayed text.

Item 41.The method of item 40, wherein identifying the at least one command,

Identifying at least one utterance;

Determining whether the type of speech corresponds to the command; And

And upon determining that the type of speech corresponds to the command, causing initiating an action corresponding to the command to edit the displayed text.

Item 42. The method of item 41, wherein the processing unit,

The electronic device, further configured to store the utterance corresponding to the at least one command, and to store information associated with the utterance corresponding to the at least one command.

Item 43. The method of any one of items 28 to 42, wherein, based on the spoken input, identifying the first letter, the phrase identifying the second letter, and the word comprises context data associated with the user or one or more recognized The electronic device, based on at least one of the credential models.

Item 44. The electronic device of item 43, wherein the context data associated with the user comprises a user speech profile.

Item 45. The electronic device of item 43 or item 44, wherein the one or more recognized credential models represent one or more recognized credential formats.

Item 46. The electronic device of any of items 28 to 45, wherein based on the voice input, identifying the first character, the phrase identifying the second character, and the word is based on gaze detection.

Item 47. The electronic device of any of items 28 to 46, wherein the voice input corresponds to at least one of a first user credential representing a user name or a second user credential representing a password.

Item 48. The method of any one of items 28 to 47,

The electronic device, further comprising storing one or more utterances of the speech input, wherein the one or more utterances correspond to first characters recognized based on the identification, a phrase identifying the second character, and a word.

The foregoing description has been described with reference to specific embodiments, for purposes of explanation. However, the exemplary discussions above are not intended to clarify or limit the invention to the precise forms disclosed. Many modifications and variations are possible in terms of the teaching material. The embodiments have been selected and described in order to best explain the principles of the techniques and their practical applications. Accordingly, those skilled in the art can be adapted to make the best use of the techniques, and various embodiments modified in various ways are suitable for the specific application considered.

While the present disclosure and examples have been sufficiently described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications are to be understood to be included within the scope of the disclosure and examples as defined by the claims.

Claims (48)

A method for identifying a voice input providing one or more user credentials, comprising:
In an electronic device with one or more processors and memory:
Receiving a voice input;
Identifying a first character, a phrase identifying a second character, and a word based on the voice input;
In response to the identification, converting the first character, the second character, and the word into text; And
Using a display to display the text in a sequence corresponding to the order of the first character, the second character and the word in the speech input. The method of claim 1, wherein the first character comprises at least one of letters, numbers, control characters, or spaces. 3. The method of claim 2, wherein the control characters include punctuation marks identified by punctuation words. The method of claim 1, wherein the phrase comprises a spelled word to aid in the identification of the second letter. The method of claim 1, wherein the word comprises a plurality of letters. 6. The method of claim 5, wherein the plurality of characters represents at least one of a domain extension or an acronym. The method of claim 1, wherein, based on the voice input, identifying a first character, a phrase identifying the second character, and a word,
Identifying at least one of the first character, the phrase identifying the second character, or the word based on one or more speech recognition models stored in the electronic device. The method of claim 1, wherein, based on the voice input, identifying a first character, a phrase identifying the second character, and a word,
Providing data representing the voice input to a server; And
Receiving, from the server, data representing at least one of the first character, the phrase identifying the second character, or the word,
Based on the voice input, identifying a first character, a phrase identifying the second character, and a word,
Identifying at least one of the first letter, the phrase identifying the second letter, or the word based on one or more language models stored on the server. The method of claim 1, wherein, based on the voice input, identifying a first character, a phrase identifying the second character, and a word,
Identifying an utterance of the voice input;
Determining the type of utterance; And
Based on the determined type of utterance, obtaining the first character corresponding to the utterance, the second character identified by the phrase, or a representation of the word. The method of claim 9, wherein determining the type of utterance,
Determining whether the type of speech corresponds to a character;
Determining whether the type of speech corresponds to a phrase identifying a character; And
And performing at least one of determining whether the type of utterance corresponds to a word. The method of claim 10, wherein determining whether the type of speech corresponds to a character,
For the utterance, determining whether the utterance corresponds to a control character; And
In accordance with the determination that the utterance corresponds to a control letter, determining, for the utterance, whether the utterance corresponds to a punctuation word,
Based on the determined type of speech, obtaining a representation of the character,
In accordance with a determination that the utterance corresponds to a punctuation word, obtaining a representation of a punctuation mark corresponding to the punctuation word, and
In accordance with a determination that the utterance does not correspond to a punctuation word, obtaining a representation of the control character other than a punctuation word. The method of claim 9, wherein, based on the voice input, identifying a first character, a phrase identifying the second character, and a word,
Obtaining a confidence level for the expression; And
Further comprising the step of determining whether the reliability level is equal to or greater than a threshold,
Converting the first character, the second character, and the word into text,
Converting the first character corresponding to the utterance, the second character identified by the phrase, or the expression of the word into text according to a determination that the reliability level is equal to or greater than the threshold; And
Providing one or more candidate expressions, an error message, or a combination thereof in response to a determination that the confidence level is less than the threshold. The method of claim 1,
Further comprising the step of identifying at least one command based on the spoken input, the at least one command enabling editing of the displayed text. The method of claim 13, wherein identifying the at least one command comprises:
Identifying at least one utterance;
Determining whether the type of utterance corresponds to a command; And
Upon determining that the type of utterance corresponds to a command, causing to initiate an action corresponding to the command to edit the displayed text. The method of claim 14,
The method of claim 1, further comprising storing the utterance corresponding to the at least one command, and storing information associated with the utterance corresponding to the at least one command. The method of claim 1, wherein the step of identifying a first character, a phrase for identifying the second character, and a word based on the voice input comprises: at least one of context data associated with a user or one or more recognized credential models. Based on, how. 17. The method of claim 16, wherein the context data associated with the user comprises a user speech profile. 17. The method of claim 16, wherein the one or more recognized credential models represent one or more recognized credential formats. The method of claim 1, wherein, based on the spoken input, identifying a first character, a phrase identifying the second character, and a word is based on gaze detection. The method of claim 1, wherein the voice input corresponds to at least one of a first user credential representing a user name or a second user credential representing a password. The method of claim 1,
And storing one or more utterances of the speech input, wherein the one or more utterances correspond to the first character recognized based on the identification, the phrase identifying the second character, and the word How to. A non-transitory computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs, when executed by the one or more processors, cause the electronic device to: A non-transitory computer-readable storage medium containing instructions for causing to perform the method of any of claims 1-21. As an electronic device,
One or more processors;
Memory; And
An electronic device comprising one or more programs stored in a memory, the one or more programs comprising instructions for performing the method of claim 1. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images