TW201506759A - Device and method for generating user interfaces from a template - Google Patents

Abstract

An electronic device is configured to receive a first request to display a user interface of a first third-party application on a respective display that is in communication with the device. The device is further configured to, in response to receiving the request, obtain a first user-interface template configured to be used by a plurality of third-party applications, and request, from the first application, one or more values for populating the first template. The device is also configured to receive, from the first application, while the first application is running on the device, a first set of values for populating the first template; populate the first template with the first set of values; generate a first user interface for the first application using the first template populated with the first set of values; and send information to the respective display that enables the first user interface to be displayed.

Description

Apparatus and method for generating a user interface from a same board Cross-reference to related applications

This application is related to the following application: U.S. Provisional Application No. 61/793,924, filed on March 15, 2013, entitled "Voice and Touch User Interface"; entitled "Pushing a Graphical User Interface to a Remote Device with US Application No. 13/032,614, filed on February 22, 2011, entitled "Pushing a User Interface to a Remote Device", filed on January 6, 2010 US Patent Application No. 12/119,960, filed on May 13, 2008, entitled "Pushing a User Interface to a Remote Device", entitled "Pushing a User Interface to a Remote Device", 2011 U.S. Application Serial No. 13/175,581, filed on Jul. 1, the entire disclosure of the U.S. Application Serial No. 13/2011, entitled "Pushing a Graphical User Interface to a Remote Device with Display Rules Provided by the Remote Device" U.S. Application Serial No. 13/250,947, entitled "Automatically Adapting User Interfaces for Hands-Free Interaction", September 30, 2011; entitled "In U.S. Patent Application Serial No. 12/987,982, filed on January 10, 2011, entitled "Intelligent Automated Assistant", US Provisional Application No. 61/295,774, filed on January 18, 2010, entitled "Intelligent Automated Assistant"; "Generating and Processing Data Items that Represent Tasks to US Provisional Application No. 61/493,201, filed on June 3, 2011, entitled "Perfect Application Adaptation User Interface for Hands-Free Interaction", US Provisional Application No. 61/, filed on June 9, 2012 657,744; US Application No. 12/207,316, filed on September 9, 2008, entitled "Radio with Personal DJ"; 2012, entitled "System and Method for Negotiating Control of a Shared Audio or Visual Resource" U.S. Provisional Application No. 61/727,554, filed on June 16, the entire disclosure of which is incorporated herein by reference. US Application No. 13/913,428, filed on June 8, 2013, entitled "Device, Method, and Graphical User Interface for Synchronizing Two of More Displays", of the User Interfaces for Limited Distraction and Non-Limited Distraction Contexts U.S. Provisional Application No. 61/832,842, filed on June 8, 2013, the entire contents of which are hereby incorporated by reference. In.

The present invention generally relates to an electronic device for generating a user interface, including but not limited to providing a user interface for a third party application.

In recent years, the use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has increased significantly. Exemplary touch sensitive surfaces include touch pads and touch screen displays. These surfaces are widely used to manipulate user interface objects on the first display associated with the first electronic device.

Exemplary manipulations include adjusting the position and/or size of one or more user interface objects, activating a button or opening a file/application represented by a user interface object, or otherwise manipulating a user interface. Exemplary user interface objects include digital images, views Messages, text, icons, control elements such as buttons, and other graphics.

The device sometimes has a first party (or inherent) device software (eg, an operating system of the device, and an application developed by the same entity that develops the operating system or under the direction of the entity), and includes the first party (or Third-party software that is inherently developed by the device software. Third party applications are sometimes loaded on the device to provide additional functionality to the device.

However, the user interface of third-party applications is often custom and/or application specific. In addition, application-specific user interfaces sometimes include words or controls that are too small to be safely used while driving a vehicle. In addition, different custom and/or application specific user interfaces for different applications sometimes place controls with similar functions in different locations, so that users will have to spend extra time and attention to locate and determine The function of these different controls. Therefore, having different customization and/or application specific user interfaces for different applications is tedious and inefficient. Moreover, in some cases, such user interfaces spend more time than is necessary for user manipulation, thereby wasting energy. This latter consideration is particularly important in battery operated devices.

Accordingly, there is a need for an electronic device having a fast and more efficient method for providing access to third party applications and eliminating driver distraction while using third party applications. A template for third-party applications that creates readability and ease of use for user interfaces that interact with third-party applications creates a user interface that eliminates driver distraction and improves usage efficiency. These methods supplement or replace, as needed, conventional methods for generating a user interface and providing access to third party applications. These methods reduce the perceived burden on the user and result in a more efficient human interface. For battery operated devices, these methods save power and increase the time between battery charges.

The above deficiencies and other problems associated with user interfaces for electronic devices having touch-sensitive surfaces are reduced or eliminated by the disclosed devices. In some embodiments, the device is Desktop computer. In some embodiments, the device is portable (eg, a notebook, tablet, or handheld device). In some embodiments, the device is a touchpad. In some embodiments, the device has a touch sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing multiple functions. . In some embodiments, the user interacts with the GUI primarily via finger contact and gestures on the touch-sensitive surface. In some embodiments, the functions include image editing, drawing, rendering, word processing, website creation, disc creation, spreadsheet creation, playing games, making phone calls, video conferencing, sending and receiving emails, instant messaging, and fitness as needed. Support, digital photography, digital video capture, web browsing, digital music playback, and/or digital video playback. Executable instructions for performing such functions are optionally included in a non-transitory computer readable storage medium or other computer program product configured for execution by one or more processors.

According to some embodiments, a method is performed at an electronic device having one or more processors and memory. The method includes receiving a first display request to display a user interface of a first third party application on a respective display in communication with the electronic device. In response to receiving the first display request, the method includes: obtaining a first user interface template configured to be used by a plurality of third party applications; and requesting for filling from the first third party application One or more values of the first user interface template. The method further includes receiving, from the first third party application, a first set of values for populating the first user interface template while the first third party application is executing on the electronic device; receiving The first user interface template is populated from the first set of values of the first third-party application; the first user interface template populated by the first set of values is used to generate the first third-party application a first user interface; and information that enables the first user interface for the first third party application to be displayed on the respective display to be sent to the respective display.

According to some embodiments, an electronic device includes a memory and a processing unit coupled to the memory. The processing unit is configured to receive a first display request to display a user interface of a first third party application on a respective display unit in communication with the electronic device. The processing unit is further configured to receive the first display request in response to receiving the first user interface template configured to be used by a plurality of third party applications; and from the first third party application Requesting one or more values to populate the first user interface template. The processing unit is also configured to receive a first value for populating the first user interface template from the first third party application while the first third party application is executing on the electronic device The first user interface template is populated by the first set of values received from the first third-party application; the first user interface template populated by the first set of values is used for the first a first user interface of the three-party application; and information that enables the first user interface for the first third-party application to be displayed on the respective display unit to be sent to the respective display unit.

According to some embodiments, an electronic device includes one or more processors, a memory, and one or more programs; the one or more programs are stored in the memory and configured to be by the one or more The processor is operative, and the one or more programs include instructions for performing the operations of any of the methods described herein. In accordance with some embodiments, a non-transitory computer readable storage medium stores therein instructions that, when executed by an electronic device, cause the apparatus to perform the operations of any of the methods described herein. According to some embodiments, an electronic device includes means for performing the operations of any of the methods described herein. According to some embodiments, an information processing device for use in a first electronic device includes means for performing the operations of any of the methods described herein.

Thus, electronic devices with displays have faster and more efficient methods for providing user interfaces for third party applications, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. These methods and interfaces may complement or replace conventional methods for generating a user interface.

100‧‧‧ portable multi-function device

102‧‧‧ memory

103‧‧‧Communication bus/signal line

104‧‧‧ wafer

106‧‧‧Input/Output (I/O) Subsystem

108‧‧‧RF circuit

110‧‧‧Optical circuit

111‧‧‧Speakers

112‧‧‧Touch-sensitive display/touch screen

113‧‧‧ microphone

116‧‧‧Other input or control devices

118‧‧‧ peripheral device interface

120‧‧‧Processing Unit (CPU)

122‧‧‧ memory controller

124‧‧‧External 埠//Connect/Charge external 埠

126‧‧‧ operating system

128‧‧‧Communication Module/Instruction Set

130‧‧‧Contact/Motion Module // Instruction Set

132‧‧‧Graphics Module/Instruction Set

133‧‧‧Tactile feedback module

134‧‧‧Text Input Module/Instruction Set

135‧‧‧Global Positioning System (GPS) Modules/Instructions

136‧‧‧Application/Instruction Set

136-1‧‧‧ individual applications

137‧‧‧Contact/Contact Module

138‧‧‧Phone module

139‧‧‧Video Conferencing Module

140‧‧‧Email/email client module

141‧‧‧ Instant Messaging (IM) Module / IM

142‧‧‧Internet Radio Module

143‧‧‧ camera module

144‧‧‧Image Management Module

147‧‧‧Browser/Browser Module

148‧‧‧ calendar module

149‧‧‧Interface Tool Set Module

149-1‧‧‧Weather Interface Tool Set

149-2‧‧‧ Stock Interface Tool Set

149-3‧‧‧Calculator Interface Tool Set

149-4‧‧‧Alarm interface tool set

149-5‧‧‧ dictionary interface tool set

149-6‧‧‧User-created interface toolset

150‧‧‧Digital Assistant Modules

151‧‧‧Vehicle Integration Module

152‧‧‧Video and music player module

153‧‧‧Memorandum module

154‧‧‧Map module

155‧‧‧record library module

156‧‧‧Display Controller/Display System Controller

157‧‧‧Device/general internal status

158‧‧‧ Optical Sensor Controller

159‧‧‧Intensity sensor controller

160‧‧‧Input controller

161‧‧‧Tactile feedback controller

162‧‧‧Power system

164‧‧‧Optical sensor

165‧‧‧Contact intensity sensor

166‧‧‧ proximity sensor

167‧‧‧Tactile output generator

168‧‧‧Accelerometer

170‧‧‧ Event Classifier

171‧‧‧ event monitor

172‧‧‧ hit view decision module

173‧‧‧Active event recognizer decision module

174‧‧‧Event Dispatcher Module

176‧‧‧data update program

177‧‧‧Object update program

178‧‧‧GUI update program

179‧‧‧Event data

180‧‧‧Event recognizer

182‧‧‧ Event Receiver Module

183‧‧‧Subsequent information

184‧‧‧ event comparator

186‧‧‧ Event definition

187-1‧‧‧Event 1

187-2‧‧‧Event 2

188‧‧‧ Event Delivery Instructions

190‧‧‧ event processing routine

191‧‧‧Application View

192‧‧‧Application internal state

200‧‧‧User Interface (UI)

202‧‧‧ fingers

203‧‧‧ stylus

204‧‧‧ "Start" or menu button/Menu button

206‧‧‧ Push button

208‧‧‧Volume adjustment button

210‧‧‧Subscriber Identification Module (SIM) card slot

212‧‧‧ headphone jack

300‧‧‧ device

310‧‧‧Processing Unit (CPU)

320‧‧‧Communication bus

330‧‧‧Input/Output (I/O) interface

340‧‧‧ display

350‧‧‧Keyboard/mouse/other indicator device

355‧‧‧ Trackpad

357‧‧‧Tactile output generator

359‧‧‧ sensor

360‧‧‧Network/Other Communication Interface

370‧‧‧ memory

380‧‧‧Drawing module

382‧‧‧ Presentation module

384‧‧‧Document Processing Module

386‧‧‧Website building module

388‧‧‧ disc creation module

390‧‧‧Trial Balance Module

400‧‧‧User interface

402‧‧‧Signal strength indicator

404‧‧‧Time

406‧‧‧Battery status indicator

408‧‧‧System匣

410‧‧‧ indicator

414‧‧‧ indicator

416‧‧‧ icon

418‧‧‧ icon

420‧‧‧ icon

422‧‧‧ icon

424‧‧‧ icon

426‧‧‧ icon

428‧‧‧ icon

430‧‧‧ icon

432‧‧‧ icon

434‧‧‧ icon

436‧‧‧ icon

438‧‧‧ icon

440‧‧‧ icon

442‧‧‧ icon

444‧‧‧ icon

446‧‧‧ icon

450‧‧‧ display

451‧‧‧Touch-sensitive surface

452‧‧‧ Spindle

453‧‧‧ Spindle

460‧‧‧Contact/Location

462‧‧‧Contact/location

468‧‧‧ position

470‧‧‧ position

500‧‧‧Working environment

505‧‧‧Communication network

510‧‧‧Server

512‧‧‧Processing Unit (CPU)

520‧‧‧Network or other communication interface

522‧‧‧Communication bus

524‧‧‧ memory

526‧‧‧ operating system

528‧‧‧Network Communication Module

540‧‧‧External information presentation system

542‧‧‧Processing Unit (CPU)

546‧‧‧User interface

547‧‧‧Display device

548‧‧‧Controls

550‧‧‧Network or other communication interface

552‧‧‧Communication bus

554‧‧‧ memory

556‧‧‧Operating system

558‧‧‧Network Communication Module

601‧‧‧User interface template

602‧‧‧App title bar

604‧‧‧Default main menu

606‧‧‧Content selection view

608‧‧‧Customized menu

610‧‧‧Active project column

612‧‧‧End button

700‧‧‧ method

800‧‧‧Electronic devices

802‧‧‧Processing unit

804‧‧‧Communication interface unit

806‧‧‧different display units

807‧‧‧ receiving unit

808‧‧‧Acquisition unit

810‧‧‧Request unit

812‧‧‧fill unit

814‧‧‧Interface generating unit

816‧‧‧Send unit

818‧‧‧Information Providing Unit

820‧‧‧Decision unit

822‧‧‧ index generation unit

824‧‧‧Voice providing unit

826‧‧‧Detection unit

828‧‧‧Digital assistant units

For a better understanding of the various described embodiments, reference should be made to the following embodiments, in which the same reference numerals are used throughout the drawings.

1A is a block diagram illustrating a portable multifunction device with a touch sensitive display, in accordance with some embodiments.

FIG. 1B is a block diagram illustrating an illustrative component for event handling in accordance with some embodiments.

2 illustrates a portable multifunction device with a touch screen in accordance with some embodiments.

3 is a block diagram of an illustrative multi-function device having a display and a touch-sensitive surface, in accordance with some embodiments.

4A illustrates an exemplary user interface for an application function table on a portable multifunction device, in accordance with some embodiments.

4B illustrates an exemplary user interface of a multi-function device having a touch-sensitive surface separate from the display, in accordance with some embodiments.

5 is a block diagram illustrating a work environment in which a portable multi-function device communicates with an external information presentation system and/or server in accordance with some embodiments.

6A-6C illustrate an exemplary user interface generated from a template in accordance with some embodiments.

7A-7I are flow diagrams illustrating a method of generating a user interface from a template in accordance with some embodiments.

FIG. 8 is a functional block diagram of an electronic device in accordance with some embodiments.

Many electronic devices use a graphical user interface. Many graphical user interfaces are custom designed or specifically targeted to individual applications. The method and apparatus described below improve the user interface by creating a user interface from a user interface template for use by third party applications method. In some embodiments, the user interface template is a generic or application-agnostic user interface template used by a plurality of different applications. These user interfaces are displayed on separate displays. Using the same generic user interface template for several different third-party applications helps standardize the user interface for third-party applications. This standardization makes it easier, easier, and more convenient for third-party applications to provide users with functional access to third-party applications while interacting with the user.

Such templates are particularly helpful in situations where the device is providing access to third party applications on the information presentation system external to the device, such as smart phones providing wisdom to the touch screen display in the car. Access to the primary application on the phone. The information presentation system optionally has a display having one of a plurality of different sets of operational characteristics (eg, display size, type of control, etc.). In some embodiments, the respective display is a display in a vehicle information display system that is viewable from a driver's seat of the vehicle to improve the use of third party applications by drivers and/or occupants in the vehicle Ease of use.

Hereinafter, FIGS. 1A-1B, 2, and 3 provide a description of an exemplary device. 4A-4B and 6A-6C illustrate an exemplary user interface generated from a template. 7A-7I are flow diagrams illustrating methods of providing a user interface for a third party application. 8 is a functional block diagram of an electronic device. The user interface in Figures 6A through 6C is used to illustrate the procedures in Figures 7A through 7I.

Exemplary device

Reference will now be made in detail to the embodiments, embodiments, In the following detailed description, numerous specific details are set forth It will be apparent to those skilled in the art, however, that the various embodiments described herein may be practiced without the specific details. In other instances, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the embodiments.

It should also be understood that, although the terms "first," "second," etc. are used in the context of the various elements, the elements are not limited by such terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact.

The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments, and is not intended to be limiting. The singular forms "a", "the" and "the" It is also understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It is to be understood that the terms "comprises" or "comprises" or "comprising" or "comprising" or "comprises" The presence or addition of features, integers, steps, operations, components, components, and/or groups thereof.

As used herein, the term "if" depends on the context of the context and is interpreted as if it is meant to mean "when..." or "after" or "in response to judgment..." or "in response to detection... …and". Similarly, the phrase "if judged..." or "if the [condition or event stated] is detected" depends on the context of the content as needed to mean "after the decision..." or "in response to the decision... And "or after the detection of [conditions or events stated]" or "in response to the detection of [conditions or events stated]".

Embodiments of electronic devices, user interfaces of such devices, and associated programs for using such devices are described. In some embodiments, the device is a portable communication device such as a mobile phone, which device also contains other functions such as PDA and/or music player functionality. Illustrative embodiments of portable multifunction devices include, but are not limited to, iPhone®, iPod Touch®, and iPad® devices available from Apple Inc. (Cupertino, California). Vision Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (eg, touch screen displays and/or trackpads), are needed. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer having a touch-sensitive surface (eg, a touch screen display and/or a touch pad).

In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and/or joystick.

The device typically supports a variety of applications such as one or more of the following applications: drawing applications, rendering applications, word processing applications, website building applications, disc authoring applications, spreadsheet applications, gaming applications Programs, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music playback Application, and / or digital video player application.

The various applications executing on the device optionally use at least one common physical user interface device, such as a touch sensitive surface. One or more of the touch-sensitive surfaces and corresponding information displayed on the device may be adjusted and/or changed between applications as needed, and/or adjusted and/or changed within the respective application. In this manner, the common physical architecture of the device, such as a touch-sensitive surface, supports a variety of applications as needed with a user interface that is intuitive and transparent to the user.

Attention is now directed to embodiments of portable devices having touch-sensitive displays. FIG. 1A is a block diagram illustrating a portable multifunction device 100 having a touch-sensitive display 112 in accordance with some embodiments. For convenience, touch sensitive display 112 is sometimes referred to as a "touch screen" and is sometimes referred to as or as a touch sensitive display system. The device 100 includes a memory 102 (which optionally includes one or more computer readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, and audio. Circuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem 106, other input or control devices 116, and external ports 124. Device 100 includes one or more optical sensors 164 as desired. Device 100 includes one or more intensity sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100) as desired. Device 100 optionally includes one or more for generating a tactile output on device 100 (eg, generating a tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). Tactile output generators 167. These components communicate via one or more communication busses or signal lines 103 as needed.

As used in the specification and claims, the term "strength" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact (eg, finger contact) on a touch-sensitive surface, or Refers to a substitute (agent) of the force or pressure of contact on a touch-sensitive surface. The intensity of the contact has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (eg, at least 256). The strength of the contact is determined (or measured) using various methods and combinations of various sensors or sensors as needed. For example, one or more force sensors beneath or adjacent to the touch-sensitive surface can be used to measure forces at various points on the touch-sensitive surface as needed. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine the estimated contact force. Similarly, the pressure sensitive tip of the stylus is used to determine the pressure of the stylus on the touch-sensitive surface as needed. Alternatively, the size of the contact area detected on the touch-sensitive surface and/or its change, the change in capacitance and/or capacitance of the touch-sensitive surface in contact with and/or the proximity of the touch-sensitive surface in contact with The change in resistance and/or resistance is used as a substitute for the force or pressure on the touch-sensitive surface as needed. In some implementations, the contact force or pressure surrogate measurement is used directly to determine if the intensity threshold has been exceeded (eg, the intensity threshold is described in units corresponding to the surrogate measurement). In some implementations, the contact force or pressure substitute measurement is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (eg, the intensity threshold is a unit of pressure) Measured pressure threshold).

As used in the specification and claims, the term "tactile output" refers to a physical displacement of a device that is detected by a user by the user's sense of touch with respect to the previous position of the device, such as a touch sensitive component (eg, touch sensitive). The physical displacement of the surface relative to another component of the device (eg, the outer casing), or the displacement of the component relative to the center of mass of the device. For example, where a device or component of a device is in contact with a user's touch-sensitive surface (eg, a finger, palm, or other portion of the user's hand), the tactile output produced by the physical displacement will be used by the user. Interpreted as a tactile sensation that is perceived to correspond to a perceived change in the physical characteristics of the device or device. For example, the movement of a touch-sensitive surface (eg, a touch-sensitive display or a trackpad) needs to be interpreted by the user as a "downstroke" or "upstroke" of the physical actuator button. In some situations, the user will feel a tactile sensation such as "down" or "up", even if there is no touch-sensitive surface that is physically pressed (eg, displaced) due to user movement. The same is true for the movement of the associated physical actuator button. As another example, the movement of the touch-sensitive surface needs to be interpreted or sensed by the user as the "roughness" of the touch-sensitive surface, even in the absence of a change in the smoothness of the touch-sensitive surface. While such interpretations of the touches made by the user will be subject to the individualized sensory perception of the user, there are many sensory perceptions of touches common to a large number of users. Thus, when the haptic output is described as corresponding to a particular sensory perception of the user (eg, "upstroke", "downstroke", "roughness"), unless otherwise stated, the haptic output produced corresponds to the device or its The component will produce a physical displacement of the described sensory perception for a typical (or average) user.

It should be appreciated that the device 100 is only one example of a portable multi-function device, and that the device 100 has more or less components than those shown, optionally combining two or more components, or having different components as needed. Configuration or configuration. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits and/or special application integrated circuits.

The memory 102 optionally includes a high speed random access memory and includes, if necessary, a non- Volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to memory 102 by other components of device 100, such as CPU 120 and peripheral device interface 118, is controlled by memory controller 122 as desired.

The peripheral device interface 118 can be used to couple the input and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 execute (execute, execute) various software programs and/or sets of instructions stored in the memory 102 to perform various functions of the device 100 and process the data.

In some embodiments, peripheral device interface 118, CPU 120, and memory controller 122 are implemented on a single wafer, such as wafer 104, as desired. In some other embodiments, peripheral device interface 118, CPU 120, and memory controller 122 are implemented on separate wafers as desired.

An RF (Radio Frequency) circuit 108 receives and transmits an RF signal, also referred to as an electromagnetic signal. The RF circuit 108 converts the electrical signal into an electromagnetic signal/converts the electromagnetic signal into an electrical signal and communicates with the communication network and other communication devices via the electromagnetic signal. RF circuitry 108 includes well-known circuitry for performing such functions, including but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, Codec (CODEC) chipset, Subscriber Identity Module (SIM) card, memory, etc. The RF circuit 108 is optionally connected by wireless communication to an Internet such as the World Wide Web (WWW), an intranet, and/or a wireless network (such as a cellular telephone network or a wireless local area network (LAN). And/or the network of the Metropolitan Area Network (MAN) and other devices communicate. Wireless communication, if desired, uses any of a number of communication standards, protocols, and technologies including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Data Evolution (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (eg, IEEE 802.11a, IEEE 802.11b, IEEE) 802.11g and/or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (eg, Internet Message Access Protocol (IMAP) and/or Post Office Protocol (POP)), Instant messaging (eg, Extensible Messaging and Presentation Protocol (XMPP), instant messaging and presence using extended Session Initiation Protocol (SIMPLE), Instant Messaging and Presence Services (IMPS)) and/or Short Message Service (SMS) , or any other suitable communication agreement that has not yet been developed as of the filing date of this document.

The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into a sound wave that is audible to humans. The audio circuit 110 also receives an electrical signal that is self-sound converted by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is retrieved from the memory 102 and/or the RF circuit 108 by the peripheral device interface 118 as needed, and/or the audio data is transmitted to the memory 102 and/or the RF circuit 108. In some embodiments, the audio circuit 110 also includes a headphone jack (eg, 212 of FIG. 2). The headphone jack provides an audio circuit 110 and a removable audio input/output peripheral (such as an output only headset, or an output (eg, a monaural or binaural headset) and an input (eg, , the microphone) the interface between the two headphones.

I/O subsystem 106 couples input/output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripheral device interface 118. The I/O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more inputs for other input or control devices. Controller 160. The one or more input controllers 160 receive/transmit electrical signals from other input or control devices 116 to other input or control devices 116. Other loss The access control device 116 optionally includes physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, and the like. In some alternative embodiments, the input controller(s) 160 are coupled to any of the following (or are not coupled to) as needed: keyboard, infrared cymbal, USB port, and such as sliding Mouse indicator device. One or more buttons (e.g., 208 of FIG. 2) include an up/down button for volume control of speaker 111 and/or microphone 113 as desired. One or more buttons include a push button as desired (eg, 206 of Figure 2).

The touch sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and/or transmits the electrical signals to the touch screen 112. The touch screen 112 displays a visual output to the user. Visual output includes graphics, text, graphics, video, and any combination thereof (collectively referred to as "graphics") as needed. In some embodiments, some or all of the visual output corresponds to a user interface item.

Touch screen 112 has a touch-sensitive surface, sensor or sensor set that accepts input from a user based on tactile and/or tactile contact. The touch screen 112 and the display controller 156 (along with any associated modules and/or sets of instructions in the memory 102) detect contact (and any movement or interruption of contact) on the touch screen 112 and will The detected contact is converted into an interaction with a user interface object (eg, one or more screen buttons, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology as needed, but other display technologies are used in other embodiments. Touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, and Other proximity sensor arrays or other components for determining one or more contact points with the touch screen 112 to detect contact and any How to move or interrupt. In an exemplary embodiment, projected mutual capacitance sensing techniques are used, such as projected mutual capacitance sensing techniques found in iPhone®, iPod Touch®, and iPad® available from Apple Inc. (Cupertino, California). .

The touch screen 112 has a video resolution of more than 100 dpi as needed. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. The user contacts the touch screen 112 using any suitable item or accessory (such as a stylus, finger, etc.) as needed. In some embodiments, the user interface is designed to function primarily by finger-based contact and gestures due to the larger contact area of the finger on the touch screen, as compared to the stylus-based input. The accuracy of finger-based contact and gestures is low. In some embodiments, the device translates the rough finger-based input into a precise indicator/cursor position or command to perform the desired action by the user.

In some embodiments, in addition to the touch screen, the device 100 may also include a touchpad (not shown) for activating or deactivating a specific function, as needed. In some embodiments, the touchpad is a touch-sensitive area of the device, and unlike the touchscreen, the touch-sensitive area does not display a visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

Device 100 also includes a power system 162 for powering various components. The power system 162 includes a power management system, one or more power sources (eg, batteries, alternating current (AC)), a recharging system, power fault detection circuitry, power converters or inverters, power status indicators (eg, , a light emitting diode (LED), and any other component associated with the generation, management, and distribution of power in a portable device.

Device 100 also includes one or more optical sensors 164 as desired. FIG. 1A shows an optical sensor coupled to optical sensor controller 158 in I/O subsystem 106. Optical sensor 164 optionally includes a charge coupled device (CCD), or a complementary metal oxide semiconductor (CMOS) optoelectronic crystal. Optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into information representative of the image. Combined with imaging module 143 (also known as camera module) Group), optical sensor 164 captures still images or video as needed. In some embodiments, the optical sensor is located on the back of the device 100 opposite the touch screen display 112 on the front of the device such that the touch screen display is enabled for use as a still image and/or video image capture. viewfinder. In some embodiments, another optical sensor is located on the front side of the device, such that when the user views other video conference participants on the touch screen display, the user's image is obtained for video viewing as needed. meeting.

Device 100 also includes one or more contact intensity sensors 165 as desired. FIG. 1A shows a contact intensity sensor coupled to intensity sensor controller 159 in I/O subsystem 106. The contact intensity sensor 165 includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric power sensors, optical force sensors, and capacitive touch-sensitive surfaces as needed. , or other intensity sensors (eg, sensors that measure the force (or pressure) that is in contact with the touch-sensitive surface). The contact intensity sensor 165 receives contact intensity information (eg, a proxy for pressure information or pressure information) from the environment. In some embodiments, at least one contact intensity sensor is co-located or in close proximity to the touch-sensitive surface (eg, touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on a back side of device 100 opposite touch screen display 112 located on the front side of device 100.

Device 100 also includes one or more proximity sensors 166 as desired. FIG. 1A shows a proximity sensor 166 coupled to a peripheral device interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I/O subsystem 106. In some embodiments, when the multi-function device is placed near the user's ear (eg, when the user is on a phone call), the proximity sensor turns off the touch screen 112 and disables the touch screen. 112.

Device 100 also includes one or more haptic output generators 167 as desired. FIG. 1A shows a haptic output generator coupled to a haptic feedback controller 161 in I/O subsystem 106. The haptic output generator 167 optionally includes one or more electro-acoustic devices, such as speakers or other audio components; and/or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerization. Object, piezoelectric actuator, electrostatic actuator, or other The haptic output produces components (eg, components that convert electrical signals into haptic outputs on the device). The touch intensity sensor 165 receives the haptic feedback generation command from the haptic feedback module 133 and generates a haptic output that can be sensed by the user of the device 100 on the device 100. In some embodiments, at least one haptic output generator is co-located or proximate to the touch-sensitive surface (eg, touch-sensitive display system 112), and as needed by vertical (eg, into/ The touch-sensitive surface is moved to produce a tactile output, either laterally or laterally (e.g., reciprocating in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on a back side of device 100 opposite touch screen display 112 located on the front side of device 100.

Device 100 also includes one or more accelerometers 168 as desired. FIG. 1A shows an accelerometer 168 coupled to a peripheral device interface 118. Alternatively, accelerometer 168 is coupled to input controller 160 in I/O subsystem 106 as needed. In some embodiments, the information is displayed on the touchscreen display in a portrait or landscape view based on an analysis of the data received from the one or more accelerometers. In addition to accelerometer(s) 168, device 100 also includes magnetometers (not shown) and GPS (or GLONASS) for obtaining information relating to the position and orientation (e.g., longitudinal or lateral) of device 100, as desired. Or other global navigation system) receiver (not shown).

In some embodiments, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact/motion module (or instruction set) 130, a graphics module (or an instruction set). 132. A text input module (or set of instructions) 134, a global positioning system (GPS) module (or set of instructions) 135, and an application (or set of instructions) 136. Moreover, in some embodiments, as shown in FIGS. 1A and 3, memory 102 stores device/global internal state 157. The device/global internal state 157 includes one or more of the following: an active application state indicating which applications (if any) are currently active; a display state indicating which application, view, or other Information occupies various areas of the touch screen display 112; sensor status, including various sensors and input controls of the device Information obtained by device 116; and location information related to the location and/or posture of the device.

Operating system 126 (eg, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as VxWorks) includes controls and management of general system tasks (eg, memory management, storage device control, power management) And other software components and/or drivers, and facilitate communication between various hardware components and software components.

Communication module 128 facilitates communication with other devices via one or more external ports 124, and also includes various software components for handling data received by RF circuitry 108 and/or external port 124. External ports 124 (eg, universal serial bus (USB), FIREWIRE, etc.) are adapted for direct coupling to other devices or indirectly coupled to other via a network (eg, internet, wireless LAN, etc.) Device. In some embodiments, the external port is a multi-pin (eg, 30-pin) connector that is the same as or similar to and/or compatible with the 30-pin connector used on the iPod (trademark of Apple Inc.) device. . In some embodiments, the external port is the same or similar and/or compatible with the 8-pin connector (eg, lightning connector) used on iPhone and iPod (trademark of Apple Inc.) devices. Pin (for example, 8-pin) connector.

The contact/sports module 130 detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (eg, a touchpad or physical point-selection palette) as needed. The contact/motion module 130 includes various software components for performing various operations related to detecting contact, such as determining whether a contact has occurred (eg, detecting a finger down event), determining the strength of the contact (eg, The force or pressure of contact, or a substitute for the force or pressure of contact), determining whether there is movement of the contact and tracking movement across the touch-sensitive surface (eg, detecting one or more finger drag events), and determining whether Stop contact (for example, detect a finger up event or an interruption in contact). The contact/motion module 130 receives contact data from a touch-sensitive surface. Determining the movement of the contact point represented by a series of contact data includes determining the rate (magnitude), velocity (magnitude and direction), and/or acceleration (magnitude and/or) of the contact point. Change of direction). These operations are applied as needed to a single contact (eg, single finger contact) or to multiple simultaneous contacts (eg, "multi-touch" / multi-finger contact). In some embodiments, the contact/motion module 130 and the display controller 156 detect contact on the touchpad.

In some embodiments, the contact/motion module 130 uses a set of one or more intensity thresholds to determine if the operation has been performed by the user (eg, to determine if the user has clicked on the icon) ). In some embodiments, at least a subset of the intensity thresholds are determined based on the software parameters (eg, the intensity threshold is not determined by the activation threshold of the particular physical actuator, and may not change device 100) Make adjustments in the case of physical hardware). For example, the mouse "click" threshold of the track pad or touch screen display can be set to a wide range of predefined thresholds without changing the track pad or touch screen display hardware. Either. Additionally, in some implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in the set of intensity thresholds (eg, by adjusting individual intensity thresholds and/or by Use the system level to select the “strength” parameter to adjust the multiple strength thresholds at once.

The contact/motion module 130 detects the gestures input by the user as needed. Different gestures on the touch-sensitive surface have different contact patterns (eg, different motions, timings, and/or intensities of the detected contacts). Therefore, the gesture is detected by detecting a specific contact pattern as needed. For example, detecting a finger touch gesture includes detecting a finger down event, followed by detecting at the same location (or substantially the same position) as the finger down event (eg, at the location shown) Finger up (off) event. As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event, followed by detecting one or more finger drag events, and then subsequently detecting the finger up (from Off) event.

The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including visual impacts for changing the displayed graphics (eg, brightness, transparency, saturation, contrast) Or other visual characteristics of the component. As used herein, the term "graphic" includes any item that can be displayed to a user, including (but not limited to) text, web pages, icons (such as user interface objects including screen buttons), digital images, video, animation, and the like.

In some embodiments, graphics module 132 stores material representing graphics to be used. Each graphic is assigned a corresponding code as needed. The graphics module 132 receives one or more codes specifying graphics to be displayed, together with coordinate data and other graphics property data (if necessary) from an application or the like, and then generates screen image data for output to the display controller 156.

The haptic feedback module 133 includes various software components for generating instructions for use by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

The text input module 134, which is a component of the graphics module 132, is provided for use in various applications (eg, contacts 137, email 140, IM 141, browser 147, and any other application requiring text input) as needed. Type the text on the screen keypad.

The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location based dialing, to the camera 143 as a picture/video post material, and provides To applications that provide location-based services, such as the weather interface toolset, the local yellow page interface toolset, and the map/navigation interface toolset.

The application 136 includes the following modules (or sets of instructions) or a subset or superset thereof as needed: ̇ contact module 137 (sometimes referred to as an address book or contact list); ̇ phone module 138; ̇ video conference Module 139; ̇ email client module 140; ̇ instant messaging (IM) module 141; ̇ internet radio module 142; 摄影 camera module 143 for static and/or video images; Module 144; The browser module 147; the calendar module 148; the interface tool set module 149 including one or more of the following: the weather interface tool set 149-1, the stock interface tool set 149-2 , calculator interface tool set 149-3, alarm interface tool set 149-4, dictionary interface tool set 149-5, and other interface tools set obtained by the user and user-created interface tool set 149-6; a personal assistant module 150; a vehicle integration module 151; a video and music player module 152, which is composed of a video player module and a music player module as needed; a memo module 153; 154; and/or ̇ record library module 155.

Examples of other applications 136 that are stored in the memory 102 as needed include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA capable applications, encryption, digital rights management, voice. Identification, and voice reproduction.

In conjunction with the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the contact module 137 can be used to manage the address book or contact list (eg, stored in memory). The application internal state 192 of the contact module 137 in the body 102 or the memory 370 includes: adding the name(s) to the address book; deleting the (multiple) name from the address book; The phone number, the email address(s), the physical address(s) or other information associated with a name; associate an image with a name; classify and categorize the name; provide a phone number or email address Initiating and/or facilitating communication, etc., by telephone 138, video conferencing 139, email 140, or IM 141.

Combined with RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch The screen 112, the display controller 156, the contact module 130, the graphic module 132 and the text input module 134, the telephone module 138 is used for inputting a sequence of characters corresponding to the telephone number, and accessing the address book 137 as needed. One or more phone numbers, modify a phone number that has been typed, dial a separate phone number, talk, and disconnect or hang up when the conversation is complete. As noted above, wireless communication uses any of a variety of communication standards, protocols, and techniques as needed.

The RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact module 130, the graphic module 132, a text input module 134, a contact list 137, and a phone module 138, the video conferencing module 139 includes a video conference for initiating, conducting, and terminating a user and one or more other participants based on user instructions. Executable instructions.

In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the email client module 140 includes, in response to a user instruction, Executable instructions for sending, receiving, and managing email. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails with still images or video images captured by the camera module 143.

In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: typing corresponds to an instant Sequence of message characters, modification of previously typed characters, transmission of individual instant messages (eg, use of Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for phone-based instant messaging, or for Internet-based Instant messaging using XMPP, SIMPLE or IMPS), receiving instant messages, and viewing incoming instant messages. In some embodiments, the transmitted and/or received instant messages include graphics, photos, audio files, video files, and/or other additional files as supported in MMS and/or Enhanced Messaging Service (EMS) as needed. As used herein, "ie "Timer" means both phone-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, messages sent using XMPP, SIMPLE, or IMPS).

In conjunction with the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, and the RF circuit 108, the Internet radio module 142 is included to select and play the Internet. A prophecy third party application of executable instructions for a radio station.

In combination with the touch screen 112, the display controller 156, the optical sensor(s) 164, the optical sensor controller 158, the contact module 130, the graphic module 132, and the image management module 144, the camera module 143 The executable instructions include: capturing still images or video (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or video, or deleting still images or video from the memory 102.

In combination with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, the text input module 134, and the camera module 143, the image management module 144 includes executable instructions for performing the following operations: configuration, modification (eg, editing) or otherwise manipulate, mark, delete, render (eg, with digital slides or photo albums) and store still images and/or video images.

In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the browser module 147 includes executable instructions for performing the following operations: Instructing to browse the Internet, including searching, linking to, receiving and displaying web pages or parts thereof, and attaching files and other files linked to web pages.

In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphic module 132, the text input module 134, the email client module 140, and the browser module 147, the calendar module Group 148 includes executable instructions for creating, displaying, modifying, and storing calendars and calendars associated with the calendar based on user instructions. Information (for example, calendar items, to-do lists, etc.).

In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphic module 132, the text input module 134, and the browser module 147, the interface tool set module 149 is used as needed. Micro-apps downloaded and used (for example, Weather Interface Tool Set 149-1, Stock Interface Tool Set 149-2, Calculator Interface Tool Set 149-3, Alarm Interface Tool Set 149-4, and Dictionary Interface Tool Set 149- 5), or a micro-app created by the user (eg, user-created interface tool set 149-6). In some embodiments, the interface toolset includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Forms) files, and JavaScript files. In some embodiments, the interface toolset includes XML (Extensible Markup Language) files and JavaScript files (eg, the Yahoo! Interface Toolkit).

In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the digital personal assistant module 150 records the voice. The command is sent and information representative of the recorded voice command is sent to a server, such as server 510 in FIG. 5, for analysis, and the voice command is responded based on the response from the server.

The vehicle integration module 151 includes executable instructions for controlling one or more intermediary processing programs of a vehicle information display system in a vehicle (eg, a car, truck, van, etc.), the intermediary processing program in the vehicle information A user interface, such as for a mapping application or a music application, is provided on a respective display of the display system (eg, display 547 of external information presentation system 540 in FIG. 5). The vehicle integration application converts information from third-party applications into content that is displayed on the individual displays of the vehicle information display system for the vehicle integration application.

In combination with the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, the RF circuit 108, and the browser module 147, the video and music player module 152 includes the permission. The user downloads and plays recorded music and other sound files (such as MP3 or AAC files) stored in one or more file formats. Executable instructions, and executable instructions for displaying, presenting, or otherwise playing video (eg, on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.) as needed.

The memo module 153 includes a touch screen 112, a display controller 156, a contact module 130, a graphics module 132, and a text input module 134. The memo module 153 includes a memo, a to-do list, and a to-do list. An executable instruction similar to that.

In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphic module 132, the text input module 134, the GPS module 135, and the browser module 147, the map module 154 is required Information for receiving, displaying, modifying and storing maps and associated maps in accordance with user instructions (eg, driving directions; information about stores and other points of interest located at or near a particular location; and other information based on location) ).

In combination with the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, and the RF circuit 108, the record library module 155 is included for selecting and playing from the record library. A prophetic third-party application of a song's executable instructions.

Each of the above identified modules and applications corresponds to one or more of the functions described above and the methods described in this application (eg, computer implemented methods and described herein) An executable instruction set for other information processing methods. Such modules (i.e., instruction sets) need not be implemented as separate software programs, programs, or modules, and thus various subsets of such modules may be combined or otherwise reconfigured as desired in various embodiments. In some embodiments, memory 102 stores a subset of the modules and data structures identified above as needed. In addition, the memory 102 stores additional modules and data structures not described above as needed.

In some embodiments, device 100 is a device that exclusively performs operations of a predefined set of functions on the device via a touchscreen and/or touchpad. By using a touch screen And/or the touchpad as the primary input control device for operation of device 100, reducing the number of physical input control devices (such as push buttons, dials, and the like) on device 100 as needed.

The predefined set of functions that are exclusively performed via the touchscreen and/or touchpad optionally include a navigation between the user interfaces. In some embodiments, the touchpad navigates device 100 from any user interface displayed on device 100 to a main menu, a home menu, or a root menu when touched by a user. In these embodiments, the "function table button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

FIG. 1B is a block diagram illustrating an illustrative component for event handling in accordance with some embodiments. In some embodiments, memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event classifier 170 (eg, in operating system 126) and respective application 136-1 (eg, application 137-13 above). , 155, 380 to 390).

The event classifier 170 receives the event information and determines the application view 191 to which the event information is delivered to the application 136-1 and the application 136-1. The event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates the current application view(s) displayed on the touch-sensitive display 112 while the application is active or in execution. In some embodiments, the device/global internal state 157 is used by the event classifier 170 to determine which application(s) are currently active, and the application internal state 192 is used by the event classifier 170 to determine to deliver event information to Application view 191.

In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: continuation information to be used when the application 136-1 continues execution, indicating that the information is being used by the application 136-1 Displaying or preparing user interface status information displayed by application 136-1, a status queue for enabling a user to return to a previous state or view of application 136-1; and previous actions taken by the user Redo /restore queue.

The event monitor 171 receives event information from the peripheral device interface 118. The event information includes information about sub-events (e.g., user touches on the touch-sensitive display 112 that are part of a multi-touch gesture). The peripheral device interface 118 transmits its information received from the I/O subsystem 106 or sensor (such as the proximity sensor 166), the accelerometer(s) 168, and/or the microphone 113 (via the audio circuit 110). The information that peripheral device interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends a request to peripheral device interface 118 at predetermined time intervals. In response, the peripheral device interface 118 transmits event information. In other embodiments, the peripheral device interface 118 transmits event information only if there is a significant event (eg, receiving an input exceeding a predetermined noise threshold and/or receiving an input for more than a predetermined duration).

In some embodiments, event classifier 170 also includes a hit view decision module 172 and/or an active event recognizer decision module 173.

The hit view decision module 172 provides a software program for determining where a sub-event has occurred within one or more views when the touch-sensitive display 112 displays more than one view. The view is made up of controls and other components that the user can see on the display.

Another aspect of the user interface associated with an application is a collection of views, sometimes referred to herein as an application view or a user interface window, in which information is displayed and a touch-based gesture occurs. The application view that detects the touch (the application view of the respective application) depends on the programming level of the application or the programming level within the view hierarchy as needed. For example, the lowest level view in which the touch is detected is referred to as a hit view as desired, and the set of events identified as being properly input is determined based at least in part on a hit view of the initial touch that begins the touch-based gesture.

The hit view decision module 172 receives information related to sub-events based on gestures of the gesture. Hit view mode when the application has multiple views organized by hierarchy Group 172 identifies the hit view as the lowest view in the hierarchy that should handle the sub-event. In most cases, the hit view is the lowest level view of the occurrence of the initial sub-event (ie, the first sub-event in the sequence of sub-events that form the event or potential event). Once the hit view is identified by the hit view determination module, the hit view typically receives all of the sub-events associated with the same touch or input source (identifying the view as a hit view for the touch or input source).

The active event recognizer decision module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer decision module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognizer decision module 173 determines that all views of the entity assignment including the sub-event are the involved views in use, and thus determines that all of the in-use views in use should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the area associated with a particular view, the higher view in the hierarchy will remain as the view in use.

The event dispatcher module 174 dispatches event information to an event recognizer (e.g., event recognizer 180). In an embodiment including the active event recognizer decision module 173, the event dispatcher module 174 delivers event information determined by the active event recognizer decision module 173 to the event recognizer. In some embodiments, the event dispatcher module 174 stores event information retrieved by the respective event receiver modules 182 in an event queue.

In some embodiments, the operating system 126 includes an event classifier 170. Alternatively, the application 136-1 includes an event classifier 170. In other embodiments, event classifier 170 is a stand-alone module, or a portion of another module (such as contact/motion module 130) stored in memory 102.

In some embodiments, the application 136-1 includes a plurality of event processing routines 190 and one or more application views 191, each of which includes a separate view for handling user interfaces of the application. The instruction of the touch event that occurred. Each application view 191 of application 136-1 includes one or more event recognizers 180. Usually, each should The application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are separate modules (such as a user interface set (not shown) or a higher level object, from which the application 136-1 is Part of the object inheritance method and other features). In some embodiments, the respective event processing routine 190 includes one or more of the following: a data update program 176, an object update program 177, a GUI update program 178, and/or an event received from the event classifier 170. Information 179. The event processing routine 190 utilizes or invokes the data update program 176, the object update program 177, or the GUI update program 178 to update the application internal state 192 as needed. Alternatively, one or more of the application views 191 include one or more respective event processing routines 190. Also, in some embodiments, one or more of the data update program 176, the object update program 177, and the GUI update program 178 are included in the respective application views 191.

The individual event recognizer 180 receives event information (e.g., event material 179) from the event classifier 170 and identifies the event from the event information. Event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of each of: a post material 183, and an event delivery instruction 188 (which optionally includes a sub-event delivery instruction).

Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events (eg, touch or touch movement). Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When a sub-event is related to the motion of the touch, the event information also includes the rate and direction of the sub-event as needed. In some embodiments, the event includes the device rotating from a certain direction to another orientation (eg, from a portrait orientation rotation to a landscape orientation, or from a landscape orientation rotation to a portrait orientation), and the event information includes a current orientation of the device (also known as Corresponding information for the device posture).

The event comparator 184 compares the event information with a predefined event or sub-event definition and determines an event or sub-event based on the comparison, or determines or updates the status of the event or sub-event. In some embodiments, event comparator 184 includes an event definition 186. Event definition 186 contains definitions of events (eg, predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch movement, touch cancellation, and multi-touch. In one example, event 1 (187-1) is defined as a double touch on the displayed object. The double touch (for example) includes a first touch (touch start) on a predetermined period of time on the displayed object, a first departure (a touch end) of a predetermined period, and a second touch (a touch start) on a predetermined period of time on the displayed object And the second departure (end of touch) of the predetermined period. In another example, event 2 (187-2) is defined as a drag on the displayed object. For example, the drag includes a touch (or contact) to a predetermined stage of the displayed object, a movement of the touch across the touch-sensitive display 112, and a touch-off (touch end). In some embodiments, the event also includes information of one or more associated event processing routines 190.

In some embodiments, event definition 187 includes definitions of events for respective user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view in which three user interface objects are displayed on the touch-sensitive display 112, when a touch on the touch-sensitive display 112 is detected, the event comparator 184 performs a hit test to determine three. Which of the user interface objects is associated with the touch (sub-event). If each displayed item is associated with a respective event processing routine 190, the event comparator uses the result of the hit test to determine which event processing routine 190 should be initiated. For example, event comparator 184 selects an event processing routine associated with the sub-event and object that triggered the hit test.

In some embodiments, the definition of the respective event 187 also includes a delay action that delays the delivery of event information until the event sequence of the event identifier is determined to correspond to an event identifier or an event identifier that does not correspond to the event recognizer. Type so far.

When the individual event recognizer 180 determines that the series of sub-events does not match any of the events in the event definition 186, the individual event recognizer 180 types the state of the event impossible, event failure, or event end, where After that, the individual event recognizer 180 ignores the basis Touch the subsequent sub-event of the action. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process the sub-events that are undergoing the touch-based gesture.

In some embodiments, the individual event recognizer 180 includes post-data 183 with configurable characteristics, flags, and/or manifests indicating how the event delivery system should perform sub-event delivery to the involved event recognizer in use. . In some embodiments, the post material 183 includes configurable features, flags, and/or manifests that indicate how the event recognizers interact with each other or are enabled to interact with each other. In some embodiments, the post material 183 includes configurable characteristics, flags, and/or manifests indicating whether sub-events are delivered to different levels in the view or programming hierarchy.

In some embodiments, when one or more particular sub-events are identified, the respective event recognizer 180 initiates an event processing routine 190 associated with the event. In some embodiments, the individual event recognizer 180 delivers event information associated with the event to the event processing routine 190. The start event processing routine 190 is different from sending (and deferring) a sub-event to a respective hit view. In some embodiments, event recognizer 180 throws a flag associated with the identified event, and event processing routine 190 associated with the flag captures the flag and executes a predefined program.

In some embodiments, the event delivery instructions 188 include sub-event delivery instructions that deliver event information about sub-events without launching an event processing routine. Instead, the sub-event delivery instructions deliver event information to an event handler that is associated with the series of sub-events or deliver event information to the involved view in use. The event processing routine associated with the series of sub-events or associated with the involved views in use receives event information and executes a predetermined program.

In some embodiments, the data update program 176 creates and updates the data for use in the application 136-1. For example, the data update program 176 updates the phone number used in the contact module 137 or stores the video file for use in the video player module 145. in In some embodiments, the item update program 177 creates and updates the objects for use in the application 136-1. For example, the object update program 176 creates a new user interface object or updates the location of the user interface object. The GUI update program 178 updates the GUI. For example, the GUI update program 178 is ready to display information and send it to the graphics module 132 for display on a touch sensitive display.

In some embodiments, event processing routine 190 includes or is capable of accessing data update program 176, object update program 177, and GUI update program 178. In some embodiments, the data update program 176, the object update program 177, and the GUI update program 178 are included in a single module of the respective application 136-1 or the application view 191. In other embodiments, the update programs are included in two or more software modules.

It should be understood that the foregoing discussion regarding the handling of user touch events on a touch-sensitive display is also applicable to other forms of user input to operate the multi-function device 100 by an input device, not all of which are on a touch screen. Start on. For example, each of the following may be used as an input corresponding to a sub-event defining an event to be recognized: a mouse movement or a mouse button press that is required to be pressed or maintained in coordination with a single or multiple keyboards; contact on the touchpad Movement, such as touch, drag, scroll, etc.; stylus input; movement of the device; verbal command; detected nitrile movement of the eye; biometric input; and/or any combination thereof.

FIG. 2 illustrates a portable multifunction device 100 having a touch screen 112 in accordance with some embodiments. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and other embodiments described below, the user is enabled, for example, by using one or more fingers 202 (not drawn to scale in the figure) or one or more stylus 203 (in The figure is not drawn to scale) to make a schematic action on the graphic to select one or more of the figures. In some embodiments, the selection of the one or more graphics occurs when the user interrupts contact with one or more graphics. In some embodiments, the gestures optionally include one or more touches, one or more toggles (from left to right, from right to left, up and/or down) and/or have been in contact with device 100. Scrolling of the finger (from right to left, from left To the right, up and / or down). In some implementations or situations, unintentional contact with the graphics does not select graphics. For example, when the gesture corresponding to the selection is a touch, the toggle gesture of sweeping the application icon does not select the corresponding application as needed.

Device 100 also includes one or more physical buttons, such as a "Home" or menu button 204, as desired. As previously described, the menu button 204 is used to navigate to any of the application 136 in the set of applications that are to be executed on the device 100 as needed. Alternatively, in some embodiments, the menu button is implemented as a screen button displayed in a GUI on the touch screen 112.

In one embodiment, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for powering on/off the device and a locking device, a volume adjustment button(s) 208, and a subscriber identification module. (SIM) card slot 210, headphone jack 212, and docking/charging external port 124. Push button 206 is used to: power on/off the device by pressing the button and holding the button in the depressed state for a predefined time interval; by pressing the button and releasing the button before the predefined time interval has elapsed To lock the device; and/or to unlock the device or initiate an unlocking procedure. In an alternative embodiment, device 100 also accepts a verbal input for initiating or deactivating some functions via microphone 113. The device 100 also includes one or more contact intensity sensors 165 for detecting the intensity of the touch on the touch screen 112, and/or one or both for generating a tactile output for the user of the device 100, as desired. A plurality of haptic output generators 167.

3 is a block diagram of an illustrative multi-function device having a display and a touch-sensitive surface, in accordance with some embodiments. The device 300 need not be portable. In some embodiments, device 300 is a laptop, desktop, tablet, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (eg, Home or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication busses 320 for interconnecting such components. Communication bus 320 includes the system as needed A circuit (sometimes referred to as a chipset) that interconnects components and controls communication between system components. Device 300 includes an input/output (I/O) interface 330 that includes display 340, which is typically a touch screen display (eg, touch screen display 112). The I/O interface 330 also includes, as needed, a keyboard and/or mouse (or other indicator device) 350 and a trackpad 355, a haptic output generator 357 for generating a tactile output on the device 300 (eg, similar to Referring to the haptic output generator 167) depicted in FIG. 1A, the sensor 359 (eg, optical, acceleration, proximity, touch sensitive, and/or contact similar to the contact intensity sensor 165 described above with reference to FIG. 1A). Strength sensor). The memory 370 includes a high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory device; and optionally includes non-volatile memory, such as one or more disk storage devices, and optical disk storage. Devices, flash memory devices, or other non-volatile solid state storage devices. The memory 370 optionally includes one or more storage devices located at the far end of the CPU 310. In some embodiments, the memory 370 stores programs, modules, and data structures similar to the programs, modules, and data structures stored in the memory 102 of the portable multifunction device 100 (FIG. 1A), or a subset thereof. . In addition, the memory 370 stores additional programs, modules, and data structures that are not present in the memory 102 of the portable multifunction device 100 as needed. For example, the memory 370 of the device 300 stores the drawing module 380, the presentation module 382, the document processing module 384, the website creation module 386, the disc creation module 388, and/or the spreadsheet module 390 as needed. The memory 102 of the portable multifunction device 100 (Fig. 1A) does not store such modules as needed.

Each of the above identified components in FIG. 3 is optionally stored in one or more of the aforementioned memory devices. Each of the above identified modules corresponds to a set of instructions for performing the functions described above. The modules or programs identified above (ie, the set of instructions) need not be implemented as separate software programs, programs, or modules, and thus, in various embodiments, various subsets of such modules may be combined or otherwise otherwise desired. Reconfigure. In some embodiments, memory 370 stores a subset of the modules and data structures identified above as needed. In addition, memory 370 Additional modules and data structures not described above are stored as needed.

Attention is now directed to an embodiment of a user interface ("UI") implemented on the portable multifunction device 100 as needed.

4A illustrates an exemplary user interface of a function table of an application on the portable multifunction device 100 in accordance with some embodiments. A similar user interface is implemented on device 300 as needed. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof: 信号 signal strength indicator 402 for wireless communication, such as cellular and Wi-Fi signals; ̇ time 404; ̇ Bluetooth indication 405; ̇Battery Status Indicator 406; 匣Tray 408 with an illustration of an application for frequent use, such as: An indicator 414 that includes the number of missed calls or voicemail messages as needed; ○ an icon 418 of the email client module 140 labeled "mail", which includes an indication of the number of unread emails as needed The browser module 147 is labeled as "Browser" icon 420; and ○ is also known as the iPod (trademark of Apple Inc.) module 152. The video and music player module labeled "iPod" Figure 422 of group 152; and 图示 for other applications, such as: ○ icon 424 of the IM module 141 labeled "text"; ○ calendar module 148 marked "calendar" Figure 426; ○ image management module 144 labeled "photo" Show 428; ○ 143 of the camera module labeled as "camera" illustration 430; ○ Record library module 155 is labeled as "record library" icon 432; ○ stock interface tool set 149-2 is labeled as "stock" 434; ○ map module 154 is marked as "map" 436; ○ weather interface tool set 149-1 is labeled as "weather" icon 438; ○ alarm clock interface tool set 149-4 is labeled "clock" icon 440; ○ internet radio module 142 An icon 442 labeled "Internet Radio"; a graphic 444 of the "Memorandum" marked by the Memorandum Module 153; and a graphic 446 of the application or module, which provides settings for the device 100 and Access to its various applications 136.

It should be noted that the pictorial representations illustrated in Figure 4A are merely illustrative. For example, the icon 422 of the video and music player module 152 is labeled "Music" or "Music Player". Other logos are used as needed for various application icons. In some embodiments, the respective application icon icons include the names of the applications corresponding to the respective application icons. In some embodiments, the designation of a particular application icon is different from the name of the application corresponding to a particular application icon.

4B illustrates a device (eg, device 300 of FIG. 3) having a touch-sensitive surface 451 (eg, the tablet or trackpad 355 of FIG. 3) that is separate from display 450 (eg, touch screen display 112). An exemplary user interface. Device 300 also includes one or more contact intensity sensors (eg, one or more of sensors 357) for detecting the intensity of a touch on touch-sensitive surface 451, as desired, and/or for generating One or more haptic output generators 359 for the haptic output of the user of device 300.

Although the following examples will be provided with reference to inputs on touch screen display 112 (which combines a touch sensitive surface with a display), in some embodiments, as shown in Figure 4B, the device detects touches that are separate from the display. Input on a sensitive surface. In some embodiments, the touch-sensitive surface (eg, 451 in FIG. 4B) has a display (eg, 450) The major axis of the upper spindle (e.g., 453 in Figure 4B) (e.g., 452 in Figure 4B). In accordance with such embodiments, the device detects contact with the touch-sensitive surface 451 at locations corresponding to respective locations on the display (e.g., 460 and 462 in Figure 4B) (e.g., in Figure 4B, 460 corresponds to Figure 460). At 468, and 462 corresponds to 470). In this manner, when a touch-sensitive surface (eg, 451 in FIG. 4B) is separated from the display (eg, 450 in FIG. 4B), the user input detected by the device on the touch-sensitive surface ( For example, contacts 460 and 462, and their movement) are used by the device to manipulate the user interface on the display of the multi-function device. It should be understood that similar methods are used as needed for other user interfaces described herein.

Additionally, while the following examples are provided primarily with reference to finger input (eg, finger contact, finger touch gesture, finger toggle gesture), it should be understood that in some embodiments, input from another input device is used (eg, Replace one or more of the finger inputs based on mouse input or stylus input). For example, the mouse can be clicked (eg, not in contact) followed by the movement of the cursor along the toggle path (eg, rather than the movement of the contact) to replace the toggle gesture. As another example, a touch gesture is replaced as needed by the mouse click when the cursor is in the position to touch the gesture (eg, instead of detecting the contact, followed by stopping the detection of the contact). Similarly, when multiple user inputs are detected at the same time, it should be understood that multiple computer mice are used as needed, or that mouse and finger contacts are used as needed.

FIG. 5 illustrates a block diagram of a work environment 500 in accordance with some embodiments. The work environment 500 includes a server 510, one or more communication networks 505, a portable multifunction device 100, and an external information presentation system 540. In some embodiments, the external information presentation system 540 is implemented in a vehicle.

The server 510 typically includes one or more processing units (CPUs) 512 for executing modules, programs, and/or instructions stored in the memory 524 and thereby performing processing operations; one or more Network or other communication interface 520; memory 524; and one or more communication busses 522 for interconnecting such components. Communication bus 522 includes the system as needed A circuit (sometimes referred to as a chipset) that interconnects components and controls communication between system components. Memory 524 includes high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and may include non-volatile memory such as one or more disk storage devices, optical disk storage devices , flash memory devices, or other non-volatile solid-state storage devices. Memory 524 includes one or more storage devices located remotely from CPU 512 as needed. The non-volatile memory device within memory 524 or memory 524 includes a non-transitory computer readable storage medium. In some embodiments, the computer readable storage medium of memory 524 or memory 524 stores the following programs, modules, and data structures or a subset thereof: ̇ operating system 526, including for handling various basic system services and for a program for performing hardware dependent tasks; and a network communication module 528 for communicating via one or more communication network interfaces 520 and one or more communication networks 505 (such as the Internet, other wide area networks, A local area network, a metropolitan area network, etc.) connects the server 510 (wired or wireless) to other computing devices.

The portable multifunction device 100 generally includes the components described with reference to Figures 1A-1B and/or Figure 3.

The external information presentation system 540 generally includes one or more processing units (CPUs) 542 for executing modules, programs, and/or instructions stored in the memory 554 and thereby performing processing operations; A plurality of network or other communication interfaces 550; memory 554; and one or more communication busses 552 for interconnecting such components. The external information presentation system 540 optionally includes a user interface 546 that includes a display device 547 and controls 548 (eg, mechanical visual cues, buttons or knobs, touch-sensitive surfaces such as touch screen displays, and Other input sources). In some embodiments, display 547 is a touch screen display capable of receiving user touch input. Communication bus 552 optionally includes circuitry to interconnect system components and control communication between system components (sometimes Called the chipset). The memory 554 includes a high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory device, and optionally includes non-volatile memory, such as one or more disk storage devices, and optical disk storage. Devices, flash memory devices, or other non-volatile solid state storage devices. The memory 554 includes one or more storage devices located remotely from the CPU 542 as needed. The non-volatile memory device within memory 552 or memory 552 includes a non-transitory computer readable storage medium. In some embodiments, the computer readable storage medium of memory 552 or memory 552 stores the following programs, modules, and data structures or a subset thereof: ̇ operating system 556, including for handling various basic system services and for a program for performing hardware dependent tasks; and a network communication module 558 for communicating via one or more communication network interfaces 550 and one or more communication networks 505 (such as the Internet, other wide area networks, The local area network, the metro network, etc.) connect the server 540 (wired or wireless) to other computing devices.

In some embodiments, device 100 drives display 547 of system 540. For example, device 100 transmits a video signal to system 540, and CPU 542 of system 540 presents the video signal on display 547. In some embodiments, device 100 transmits the video signal directly to display 547, and CPU 542 is not used to visualize the video signal (eg, device 100 uses display 547 as an auxiliary display). In some embodiments, the user interface displayed on the touch screen 112 of the device 100 is synchronized with the user interface displayed on the display 547 of the system 540, and in some other embodiments, is displayed at the touch of the device 100. The user interface on the control screen 112 is not synchronized with the user interface displayed on the display 547 of the system 540. In some embodiments, system 540 transmits information corresponding to user input (eg, user touch input on display 547 or user input via control 548) to device 100, and device 100 updates based on the received information. A user interface displayed on the touch screen 112 of the device 100.

User interface and associated programs

Attention is directed to embodiments of a user interface ("UI") and associated programs that can be implemented on an electronic device having a display and a touch-sensitive surface, such as device 300 or portable multifunction device 100.

6A-6C illustrate an exemplary user interface generated from a template for use by a third party application, in accordance with some embodiments. The user interface in these figures is used to illustrate the procedures described below including the routines of Figures 7A-71.

FIG. 6A illustrates a user interface template 601 displayed on display 547 of external information presentation system 540. In some embodiments, system 540 is included in a vehicle (eg, in a dashboard or steering wheel of a vehicle). In some embodiments, display 547 is a touch screen display configured to receive user touch input. FIG. 6A further illustrates a plurality of user interface objects (sometimes referred to herein as "visual cues" or "selectable user interface objects") including: an application title bar 602, a preset main function table 604, The custom function table 608, the content selection view 606, the active item field 610, and the end button 612. In some embodiments, the user interface template contains fewer visual cues than those shown in the template 601 to further simplify the user interface and prevent distraction of the driver.

FIG. 6B illustrates an example of a user interface generated from user interface template 601 (eg, for use by an internet radio application). 6B illustrates an application title bar 602 displaying the title of the current application (eg, an internet radio application), a main function table 604 having preset controls (eg, up, down, search, Classification, etc., custom menu 608 (eg, favorite item menu) with application-specific controls (eg, filter favorite items, add current items to favorite items corresponding to application-specific functions), and end button 612. 6B further illustrates the following two: a content selection view 606 (eg, station selection) that displays selection options for the current application (eg, relax jazz station 103, Korean hangout 105, etc.); and active item column 610 , which displays information about the currently active item (eg, hard rock station 101).

Figure 6C illustrates a user interface different from that produced by the template 601 shown in Figure 6B. An example of a user interface generated by user interface template 601 (eg, for use by a record library application). 6C illustrates the following: an application title bar 602 displaying the title of the current application (eg, a library application), a main menu having preset controls (eg, up, down, search, category, etc.) 604. A custom function table 608 (eg, a user playlist function table) having application specific controls (eg, playlist 1, playlist 2, etc.), and an end button 612. 6C further illustrates a content selection view 606 (eg, a library of songs) that displays selection options for the current application (eg, stored songs 103, stored songs 105, etc.), and displays information about currently active items (eg, stored songs). 101) The role of the information in the item column 610.

As shown in FIG. 6B to FIG. 6C, a common board (eg, the template 601 shown in FIG. 6A) is used to generate different user interfaces for different third party applications as needed (eg, displayed on The user interface of the Internet radio application in FIG. 6B is different from the user interface displayed in the record library application in FIG. 6C): the information extracted from different third-party applications is inserted into the user interface template 601. . However, the different user interfaces shown in Figures 6B and 6C have similarities: making the user interface easier to use (e.g., visual cues for selecting playable content are displayed in the user interface of Figures 6B and 6C). The content selection view 606 in both, and the information about the currently playing media is displayed in the active item column 610 in both the user interfaces in FIGS. 6B and 6C). In addition, the template needs to enhance the minimum text size as needed to ensure that the text displayed in the user interface of Figures 6B and 6C is easy to read.

7A-7I are flow diagrams illustrating a method 700 of providing a user interface for use by a third party application, in accordance with some embodiments. Method 700 is performed at an electronic device (e.g., device 300 of FIG. 3, or portable multifunction device 100 of FIG. 1A). In some embodiments, an electronic device includes a display and a touch-sensitive surface. In some embodiments, the display is a touch screen display and the touch sensitive surface is attached to the display. In some embodiments, the display is separated from the touch-sensitive surface. Some of the operations of method 700 are combined as needed, and/or Change the order of some operations as needed.

As described below, the method 700 provides a user interface for a third party application that reduces the perceived burden of the user when interacting with a third party application, thereby creating a more efficient presence interface. For battery operated electronic devices, the user is able to interact with the user interface more quickly and efficiently to conserve power and increase the time between battery charges.

The device receives (702) a first display request to display a user interface of the first third party application on a respective display, the respective display being in communication with the electronic device (eg, via a wired or wireless connection). For example, the device receives input from the control associated with the respective display or the control associated with the electronic device to initiate activation of the application launch icon corresponding to the first third party application, for example, for the Internet The touch gesture on the Internet radio icon 442 (FIG. 4A) of the road radio application 142 (FIG. 1A or FIG. 3), or the record library for the record library application 155 (FIG. 1A or 3) The touch gesture of 432 (Fig. 4A) is shown. In some embodiments, the respective display is a display of the device (eg, touch screen 112 of portable multifunction device 100). In some embodiments, the respective displays are displays that are separate from the device (eg, display 547 of external information presentation system 540 in FIG. 5).

In response to receiving the first display request (704), the device obtains (706) a first user interface template configured (or designed) for use by a plurality of third party applications; and the device is from the first third party application The program request (710) is used to populate one or more values of the first user interface template. In some embodiments, the first user interface template is a generic or application-agnostic (or independent) user interface template used by a plurality of different applications. For example, in some embodiments, the device obtains the user interface template 601 shown in Figure 6A.

In some embodiments, the respective display is (708) a display in a vehicle information display system viewable from a driver's seat of the vehicle; and the first user interface template is configured One of a plurality of predetermined user interface templates for creating a user interface for a third party application on a separate display to prevent driver distraction (eg, a plurality of predetermined templates have large fonts, and There are several easy-to-select options that can be operated by the driver without distracting the driver from safe driving. Thus, in some embodiments, the use of such predetermined user interface templates enables the device to provide the user with functional access to one or more third party applications while maintaining use for display on respective displays. Certain controls of the appearance of the interface to ensure that the user interface on each display is simple and intuitive to use.

In some embodiments, the value at the first third party application is configured (711) in a tree structure comprising a plurality of nodes having predetermined characteristics and a tree with one or more other nodes in the tree structure Type relationship (for example, parent or child relationship). In some embodiments, the request to populate one or more values of the first user interface template includes one or more of: returning a list of child nodes corresponding to respective ones of the plurality of nodes Request for value (for example, for a node corresponding to an artist, the child node corresponds to the artist's song or artist's album); a request for returning a value corresponding to a list of characteristics of each of the plurality of nodes; providing a pair a request for content represented by a respective node (eg, providing audio content corresponding to an audio file, audio stream, video file, video stream, and/or playlist or a radio station corresponding to a plurality of distinct content units) Request) a request to provide a list of nodes that match a search query (eg, a request to provide a list of artists, albums, songs, radio stations, audio books, podcasts, or the like, each of the foregoing and one or more searches a set of items and optionally a logical operator match to the search term; and a request to provide a list of actions specific to the first third party application, available upon request Take access to content associated with the first third-party application (for example, "like", "skip", "add to favorite item", "flag" or other application-specific operations) action. In some embodiments, the characteristics of the respective nodes include one or more of the following: a cover of each node, an identifier of each node, and each The duration of the media content associated with the other node, the title of the individual node (the title of the currently playing track or radio station), the subtitle of the individual node (for example, the artist currently playing the song or the track currently playing the radio station) a playable flag indicating whether each node is playable, a play flag indicating whether the respective node has been played, and/or a container flag indicating whether the respective node is a container having a child node.

The device is executed on the electronic device by the first third party application while receiving (712) a first set of values for populating the first user interface template from the first third party application (eg, receiving an internet streaming radio station) List of names). For example, FIG. 6B shows a user interface template 601 populated by data from an article entitled "Internet Radio Application."

In some embodiments, one or more of the first set of values are from the remote source by the first third party application in response to the request to populate one or more values of the first user interface template (For example, a server in communication with the first third party application, such as server 510 in FIG. 5) performs a capture (713). For example, when a streaming Internet radio has selected for playback, the first third party application initiates a request to start streaming the positive streaming radio to the remote server. And requesting the name of the song and the artist and the corresponding album cover of the song to be displayed on the streaming internet radio station. The first third party application receives the value and transmits the value to the mediation handler for incorporation into the "playing" user interface template, and the mediation handler is transmitted to the respective display.

In some embodiments, the apparatus includes (714) a digital personal assistant processing program having speech recognition capabilities (eg, a digital personal assistant 150 that records voice commands and transmits information representative of the recorded voice commands to a servo such as that in FIG. The server of device 510 is for analysis and responds to voice commands based on responses from the server). In some embodiments, the device is based on a plurality of third party applications from the first third party application for values associated with user interface templates that populate the user interface of the respective display. Requesting a response to determine (715) the capabilities of the first third party application (eg, the digital assistant processing program traverses the tree structure of the data after the first third party application and draws the first third party application Various options for program access and post-set data). In some embodiments, the device generates (716) an index of the capabilities of the first third party application (eg, based on various options accessible via the first third party application and a map of the post data). In some embodiments, the device provides (718) access to the capabilities of the first third party application via a digital personal assistant handler (eg, enabling the user to request a display of a list of songs for a particular artist using a voice command, The display of a list of Internet streaming radio stations that can be used for playback, the playback of a particular song of a particular artist, or the search of media that matches the search query of the spoken statement via the media presentation application). For example, for a video streaming internet radio application, the personal assistant will traverse and index the following two: a tree structure that can be used to stream the back-end data of the Internet radio station, and can be Streaming other operations performed by the Internet radio application. The user is then enabled to provide a verbal command to the digital assistant using the voice command for the digital assistant to start playing the radio, stop playing the radio, establish a new radio based on the current song, skip songs or other operations (eg , "Assistant, please play Artist X Radio", "Assistant, please skip this song", "Assistant, please list all radio stations established in the previous week", or "Assistant, please mark this song as your favorite item" ").

In some embodiments, the device detects (720) that the predetermined condition has been met. In response to detecting that the predetermined condition has been met, the device re-determines (721) the capabilities of the first third party application and regenerates an index of the capabilities of the first third party application (eg, via the first third party application) The various options of access are re-indexed: when the device is placed in communication with a respective display, such as display 547 of external information presentation system 540 (FIG. 5), when the device is connected to a vehicle information display that includes the respective display System time, and / or when the first third party application is launched). For example, for streaming internet Radio applications, the availability of streaming internet radio stations is determined based on the availability of the internet connection used to receive the video streaming internet radio; similarly, if the user establishes/subscribes a new radio To delete or unsubscribe old radio stations, then associate with the streaming Internet radio application to change the data (for example, which radio stations are nodes in the tree structure and/or which radio stations are playable) . To address the downstream data streamed at the Internet radio application, the device periodically re-indexes the post-streaming data at the Internet radio application (for example, when streaming Internet radio) When the application is launched, or when the device is connected to the vehicle information display system). In some embodiments, the third party application actively informs the digital assistant to notify that a portion of the post-data has been changed or updated, and prompts the digital assistant to re-edit the portion (or all) of the third-party application. index. The digital assistant responds to the information that the part of the post-set data has been changed or re-indexes the part (or all) of the third-party application's post-data when the third-party application is launched next time.

The device populates (722) the first user interface template with a first set of values received from the first third party application. For example, in accordance with such embodiments, FIG. 6B shows the filling of data (eg, station selections, favorite items, current stations, etc.) from a third party application entitled "Internet Radio Application" for generation The user interface template 601 of the third party application displayed on the display 547 of FIG. 6C of the third party application.

In some embodiments, the first user interface template is used by a plurality of different third party applications including (724) the first third party application and the second third party application. In some embodiments, populating the first user interface template with values received from respective third party applications includes inserting values identified by the respective third party applications as belonging to the first value category to be provided to the respective displays The user interface is in a predetermined area of a predetermined font size. For the first user interface template, the first third party application identifies the first type of information as the first value category, and the second third party application identifies the difference from the first type of application The second type of information is the first value category. For example, Figure 6B shows a user interface template 601 populated with values associated with an internet radio station, and Figure 6C shows a user interface template 601 populated with values associated with stored songs. In accordance with such embodiments, Figures 6B and 6C show user interface templates having the same font size.

In some embodiments, for the first user interface template: the first third party application identifies (726) the name of the remotely generated content station (eg, streaming Internet radio station) as the first value a category; and the second third party application identifies the name of the predetermined playlist as the first value category. In some embodiments, the predetermined playlist is different from the streaming internet radio station, because the predetermined playlist has a set length, and the content of the playlist is set when the user selects the playlist for playing, and the string Streaming internet radio stations are continuous continuous streams of music of indeterminate length (as long as the user is listening) and include remotely selected content (eg, automatically performed by a computer based on a music selection algorithm, or Manually by the music host, the radio host, or other person independent of the actions performed by the user of the device). For example, in Figures 6B-6C, the device identifies the name of the streaming internet radio station as the first value category of the Internet radio application (e.g., 142 in Figure 1A) of Figure 6B (e.g., The description of the media content item can be played, and the device identifies the name of the song as the first value category of the record library application (e.g., 155 in Figure 1A) of Figure 6C.

In some embodiments, for the first user interface template: the first third party application will be streamed (eg, music or other media content via an on-demand distributed data network such as the Internet) Streaming) the name identification (728) of the content to the device is a first value category; and the second third application will name the content stored on the device (eg, language learning program, audio book, music, podcast, or The similarity is identified as the first value category.

The device generates (730) a first user interface for the first third party application using the first user interface template populated with the first set of values. For example, Figure 6B illustrates The user interface template 601 and the user interface generated from the values received from the internet radio application. In accordance with this example, FIG. 6B shows a preset master function table 604 that includes information buttons (eg, to return values corresponding to a list of characteristics of respective nodes), a launch button (eg, to provide representation by respective nodes) Content, a search button (eg, to provide a list of nodes that match the search query), and a custom menu 608 with custom buttons (eg, to provide a list of actions specific to the first third party application, These actions are taken while providing access to the content associated with the first third party application).

In some embodiments, the apparatus includes (732): a plurality of processing programs including a first third party application processing program for executing the first third party application; and A collection of one or more mediation handlers that transfer information between third-party application handlers (eg, for integrating third-party applications with vehicle information display systems such as navigation or entertainment systems or similar to external information in Figure 5) The application of the other external information display system of the presentation system 540). In some embodiments, the request to populate one or more values of the first user interface template is generated by an intermediary handler from a set of one or more mediation handlers; and the first user interface is derived from An intermediary handler for the collection of one or more mediation handlers is generated. For example, a smart phone (eg, portable multifunction device 100 in FIG. 5) has a vehicle integration application 151 that controls a vehicle information display system in a vehicle (eg, a car, truck, van, etc.) (eg, An application comprising one or more mediation handlers, the vehicle information display system providing a user interface on a respective display of the vehicle information display system (eg, display 547 of external information presentation system 540 of FIG. 5), such as Map an app or music app to the map. The smart phone also has one or more third-party applications such as a separate podcast application or a streaming internet radio application, and the vehicle integration application converts information from a third-party application into a vehicle-integrated application. Content displayed on the respective displays of the vehicle information display system.

In some embodiments, the first third party application is a (734) media presentation application (eg, Internet Radio 142 of Figure 1A). In some embodiments, the device includes a first party application that is a media presentation application (eg, video and music player module 152 in FIG. 1A); the first party application includes a request to request media presentation operations. Executing a plurality of selectable visual cues and a first party user interface of one or more content presentation areas (eg, areas for displaying a list of available media) at respective locations of the first party user interface; and generating The first user interface of the first third party application includes a user interface including a selectable visual cues of functions and locations corresponding to the selectable visual cues and content presentation areas of the first party user interface and One or more content presentation areas. Thus, in some embodiments, the first user interface template is a template for adapting information from a third party application to a standardized user interface having similar controls, content areas, and for first party media Presenting the appearance of the application to provide a consistent user interface that is familiar to the user and thus prevents driver distraction when displayed on a vehicle information display system that is visible from the driver's seat of the vehicle .

The device will enable the first user interface for the first third party application to be displayed (736) on the respective display or otherwise provided to the respective display (eg, the portable multifunction device 100 provides information) Up to display 547 of external information presentation system 540 in FIG.

In some embodiments, while the first user interface of the first third party application is displayed on the respective display, the device is input from the respective display device (eg, a touch sensitive surface consistent with the respective display or The buttons or knobs that control the respective displays receive (738) respective inputs corresponding to the selection of options in the first user interface (eg, the selected option is to play, pause, explore, display a new menu, or perform such as " Options for third-party application-specific actions like ", skip" or "add to favorite items"). In response to receiving the respective input, the device provides (740) information to the first third party application executing on the electronic device that enables the first third party application to use the first The choice of options in the interface responds. In some embodiments, the device receives (742) a response to the selection of the option from the first third party application from the first third party application; and will enable a response to the selection of the option to be provided (eg, display) The user's information is sent (744) or otherwise provided to the respective display. For example, the user presents a list of the names of the Internet radio stations by streaming on the respective displays, and the user touches the name of the respective streaming radio network radio station (for example, touching FIG. 5 The touch screen display of the medium and external information presentation system 540, the intermediary processing program at the device (for example, the portable multifunction device 100 in FIG. 5) provides the first third party application with the indication of the respective streaming network. The information that the radio station has selected, the first third party application request server (e.g., server 510 in FIG. 5) provides the cover, title, subtitle, and title for each of the serial streaming internet radio stations. Audio streaming and then providing this information to an intermediary handler (eg, at portable multifunction device 100 in FIG. 5) that generates a user interface based on the information and user interface templates and will generate The user interface is transmitted to the respective display for display (eg, displayed on display 547 of external information presentation system 540 in FIG. 5).

In some embodiments, the first party intermediary processing at the device converts the data from the first third party application into a standardized simplified user interface that prevents distracting driving and is corresponding to and displayed by forwarding The user interaction input of the user interface on the display mediates communication between the respective display and the first third party application, so that the first third party application can respond to the input even in the first third party application Information that indicates the size, resolution, and other characteristics of the respective displays is not provided and does not have direct access to user interaction with the user interface displayed on the respective displays. For example, a third-party application does not have the time and location of touch input, but instead obtains information about what information is requested by the user, such as through a first-party intermediary handler.

In some embodiments, the first user interface of the first third party application is displayed While present on the respective display (746), the device receives (748) a selection indicating individual options in the first user interface (eg, a container or a selection of nodes having child nodes, such as songs including selected artists) Information about the "Artist" folder, or the selection of a particular Internet streaming radio station for a plurality of Internet streaming radio stations. In some embodiments, in response to receiving information indicating a selection of respective options in the first user interface (750): the device obtains (752) a second user interface template that is different from the first user interface template (eg, a template for displaying the contents of the container, such as a playlist template, or a "playing" user interface template; and requesting (754) from the first third party application to populate the second user interface template Multiple values. In some embodiments, the device receives (756) a second set of values for populating the second user interface template from the first third party application while the first third party application is executing on the electronic device (eg, The name and playback time information of the selected artist's song used to play the list template, or the name of the currently playing radio station, the name of the currently playing song, and the cover of the currently playing song for the "Now Playing" template); Filling (758) a second user interface template by receiving a second set of values from the first third party application; by using a second user interface template populated with the second set of values (eg, inserting the selected artist The name of the song and the playing time information are added to the playlist template, or the name of the currently playing radio station, the name of the currently playing song, and the cover of the currently playing song are added to the "Now Playing" template. The "Playing" template is included for Controlling the pause/start/stop/fast forward/rewind control of the currently playing song) to generate (760) the second third party application By the interface; such that the first and the second third-party applications can be displayed on the user interface information transmitted (762) provided on the respective display or otherwise to the respective display.

In some embodiments, after transmitting information that enables the first user interface of the first third party application to be displayed on the respective display to the respective display (764), the device receiving (766) pair will be different from the first Second third party application of a third party application a second display request displayed on the respective display (eg, using an application associated with the respective display or a control associated with the electronic device to receive the application of the second third party application) The startup of the illustration). For example, while the user initially plays audio from the Internet streaming radio application 142 (e.g., using the user interface for the streaming Internet radio application shown in Figure 68), The user switches from the detached podcast application or the audiobook application to play the audio. In some embodiments, in response to receiving the second display request, the device requests (768) from the second third party application to populate one or more values of the first user interface template; in the second third party application Executing on the electronic device while receiving (770) a third set of values for populating the first user interface template from the second third party application; populating with the third set of values received from the second third party application (772) a first user interface template; generating, by the first user interface template filled with the third set of values, (774) a second user interface of the second third party application; and causing the second third party application to be second The user interface can be displayed (776) (or otherwise provided) to the respective displays on a respective display (eg, the user interface of the record library application 155 shown in FIG. 6C). Thus, in some embodiments, when requesting a different third party application using the same first user interface template, the device repeatedly requests a value from the third party application to populate the first user interface template. This procedure is repeated in this case because the first user interface template is designed or configured for use by a plurality of different third party applications.

In some embodiments, after the information is sent to a respective display (764) that enables the first user interface of the first third party application to be displayed on the respective display: the device receives from the first third party application ( 778) An updated first set of values for populating the first user interface template. In some embodiments, in response to receiving the updated first set of values (780) for populating the first user interface template, the device is populated by the updated first set of values received from the first third party application (782) First user interface Generating (784) an updated first user interface of the first third party application using the first user interface template populated by the updated first set of values; and causing the first third party application to pass The updated first user interface can display (786) the information provided on the respective displays to the respective displays. In some embodiments, the updated first set of values is generated based on an event occurring at the first third party application (eg, updating the user interface of the first third party application by the first third party application) Start, not by the intermediary handler). For example, a third-party application is streaming a media application and the device's data connection stops functioning. In response to detecting that the data connection has ceased to function, the third party application transmits the information to the mediation handler at the device, thereby indicating the streamed content previously identified as "playable" (eg, in the presence of work data) When connected, the status has changed, making it "unplayable" (for example, because the data connection is no longer available). In some embodiments, when the content is changed from "playable" to "non-playable", the first user interface of the first third party application is updated to indicate that the content is "unplayable" (eg, by self- Play the content list to remove the content, or change the appearance of the representation of the content by reducing the contrast of the representation or displaying the "unplayable" icon or symbol text to the representation of the content to indicate that the content is currently unplayable). For example, in FIG. 6B, if the device stops having access to the streaming internet radio station displayed in the content selection view 606, the device will regenerate the user interface shown in FIG. 6B to reflect Streaming the unavailability of an internet radio station (for example, by displaying an unplayable icon adjacent to the name of the unavailable station, by reducing the contrast ("ashing the unavailable desk"), or by The content selection view 606 removes the station).

It is to be understood that the specific order in which the operations in the FIGS. 7A through 7I have been described is an example and is not intended to indicate that the described order is the only order in which the operations can be performed. Those of ordinary skill in the art will recognize various ways of reordering the operations described herein.

FIG. 8 shows a functional block diagram of a first electronic device 800 configured in accordance with the principles of various described embodiments, in accordance with some embodiments. Functional blocks of the device are implemented by hardware, software or a combination of hardware and software as desired to carry out the principles of the various described embodiments. Those skilled in the art will appreciate that the functional blocks described in FIG. 8 may be combined or separated into sub-blocks as needed to implement the principles of the various described embodiments. Accordingly, the description herein supports any possible combination or separation or further definition of the functional blocks described herein as needed.

As shown in FIG. 8 , the electronic device 800 includes a memory coupled to the processing unit 802 . In some embodiments, electronic device 800 also includes a communication interface unit 804 coupled to processing unit 802 and configured to communicate with respective display unit 806. In some embodiments, the processing unit 802 includes a receiving unit 807, an obtaining unit 808, a requesting unit 810, a filling unit 812, an interface generating unit 814, a transmitting unit 816, an information providing unit 818, a determining unit 820, an index generating unit 822, and a voice. The providing unit 824, the detecting unit 826, and the digital assistant unit 828. The processing unit 802 is configured to receive (eg, via the communication interface unit 804 via the receiving unit 807) a first display request to display the user interface of the first third party application to the respective display unit in communication with the electronic device 800 (for example, by separate display unit 806). Processing unit 802 is further configured to receive a first display request by obtaining (eg, by obtaining unit 808) a first user interface template configured for use by a plurality of third party applications; A third party application request (eg, by request unit 810) is used to populate one or more values of the first user interface template. The processing unit 802 is further configured to receive from the first third party application (eg, by the receiving unit 807 or the communication interface unit 804) for filling the first while the first third party application is executing on the electronic device a first set of values of the user interface template; a first user interface template populated by the first set of values received from the first third party application (eg, by the fill unit 812); A user interface template is generated (eg, by interface generation unit 814) for the first third party application a first user interface; and information transmission that enables the first user interface of the first third party application to be displayed on the respective display unit (eg, the respective display unit 806) (eg, by the transmitting unit 816) To each display unit.

In some embodiments, the processing unit 802 is further configured to: display the first user interface of the first third party application on a separate display unit (eg, the respective display unit 806) The input device of the unit receives (eg, via the communication interface unit 804 via the receiving unit 807) a respective input corresponding to the selection of the option in the first user interface, and in response to receiving the respective input (eg, by information) The providing unit 818) information to the first third party application executing on the electronic device, the electronic device causing the first third party application to respond to the selection of the option in the first user interface. In some embodiments, processing unit 802 is further configured to receive (eg, via communication interface unit 804 via receiving unit 807) a response from the first third party application to the selection of options from the first third party application. And will enable the response to the selection of options to be provided to the user for information transmission (eg, by sending unit 816) to the respective display unit.

In some embodiments, processing unit 802 is further configured to: while the first user interface of the first third party application is displayed on a respective display unit (eg, individual display unit 806): receiving (eg, Information indicating the selection of the respective options in the first user interface via the communication interface unit 804 via the receiving unit 807); and in response to receiving the information indicating the selection of the respective options in the first user interface: obtaining (eg, And obtaining, by the obtaining unit 808, a second user interface template different from the first user interface template; and requesting (eg, by requesting unit 810) from the first third-party application to fill the second user interface template One or more values. In some embodiments, processing unit 802 is further configured to receive from the first third party application while the first third party application is executing on the electronic device (eg, via communication interface unit 804 via receiving unit 807) a second set of values for populating the second user interface template; received from the first third party application The second set of values is filled (eg, by the filling unit 812) the second user interface template; generated by using the second user interface template populated with the second set of values (eg, by the interface generation unit 814) a second user interface of the first third party application; and information transmission that enables the second user interface of the first third party application to be displayed on the respective display unit (eg, the respective display unit 806) For example, by transmitting unit 816) to each display unit.

In some embodiments, processing unit 802 is further configured to transmit (eg, via transmitting unit 816) information such that the first user interface of the first third party application can be displayed on the respective display unit (eg, After the respective display units on the respective display unit 806), the user of the second third party application that will be different from the first third party application is received (eg, by the receiving unit 807 or via the communication interface unit 804) The interface displays a second display request on a respective display unit (eg, individual display unit 806). In some embodiments, the processing unit 802 is further configured to respond to the receipt of the second display request: from the second third party application request (eg, by the request unit 810) for populating the first user interface template One or more values; received from the second third party application while the second third party application is executing on the electronic device (eg, via the communication interface unit 804 via the receiving unit 807) for populating the first user a third set of values of the interface template; populating (eg, by populating unit 812) the first user interface template with a third set of values received from the second third party application; using the first use populated by the third set of values The interface template generates (eg, via the interface generation unit 814) a second user interface of the second third party application; and enables the second user interface of the second third party application to be displayed on the respective display unit ( For example, the information on the respective display unit 806) is transmitted (e.g., by the transmitting unit 816) to the respective display unit.

In some embodiments, the apparatus includes a plurality of processing programs, the processing program including a first third party application processing program for executing the first third party application, and for forwarding the respective display unit with the first third party Application processing information between programs One or more mediation handler sets; a request to populate one or more values of the first user interface template is generated by an intermediary handler from one or more mediation handler sets; and the first user interface borrows Generated by an intermediary handler from one or more mediation handler collections.

In some embodiments, one or more of the first set of values are from the remote source by the first third party application in response to the request to populate one or more values of the first user interface template Capture.

In some embodiments, the respective display unit (eg, the respective display unit 806) is a display unit in the vehicle information display system viewable from the driver's seat of the vehicle; and the first user interface template is for a plurality of predetermined uses. In one of the interface templates, the predetermined user interface templates are configured to create a user interface for a third party application that prevents distraction of the driver on the respective display unit.

In some embodiments, the first user interface template is used by a plurality of different third party applications including the first third party application and the second third party application; the values are received by the values received from the respective third party applications ( For example, the first user interface template by the filling unit 812 includes inserting values recognized by the respective third party applications as belonging to the first value category into the user interface provided to the respective display unit in a predefined font type. a predefined area of size; and for the first user interface template: the first third party application identifies the first type of information as the first value category; and the second third party application will be different from the first type of information The second type of information is identified as the first value category.

In some embodiments, for the first user interface template: the first third party application identifies the name of the remotely generated content station as the first value category; and the second third party application will reserve the playlist The name is identified as the first value category.

In some embodiments, for the first user interface template: the first third party application identifies the content name that can be streamed to the device as the first value category; and the second third party application is to be stored in The content name on the device is identified as the first value category.

In some embodiments, the value of the first third party application is configured as a tree structure comprising a plurality of nodes having predetermined characteristics and a tree relationship with one or more other nodes in the tree structure; A request to populate one or more values of the first user interface template (eg, by request unit 810) includes one or more of: returning respective nodes corresponding to the plurality of nodes a request for the value of the child node list; a request to return a value corresponding to a list of characteristics of each of the plurality of nodes; a request to provide content represented by the respective node; a request to provide a list of nodes matching the search query; And providing a request for a third party application specific action list that can be taken while providing access to the content associated with the first third party application.

In some embodiments, the first third party application is a media presentation application. In some embodiments, the device 800 includes a first party application that is a media presentation application; the first party application includes a first party user interface having a request for media presentation operations. Executing a plurality of selectable visual cues and one or more content presentation areas at respective locations in the first party user interface; and generating (eg, by interface generation unit 814) the first third party application A user interface includes a user interface, the user interface including selectable visual cues and one or more content presentation areas corresponding to the selectable visual cues and content presentation areas of the first party user interface.

In some embodiments, processing unit 802 is further configured to transmit information to enable a first user interface of the first third party application to be displayed on a respective display unit (eg, respective display unit 806) After the respective display unit: received from the first third party application (eg, via the communication interface unit 804 via the receiving unit 807) for populating the updated first set of values of the first user interface template; and in response to receiving Up to an updated first set of values for populating the first user interface template: populating (eg, by populating unit 812) the first user by the updated first set of values received from the first third party application Interface template; using a first user interface populated with a first set of updated values The template generates (eg, via the interface generation unit 814) the updated first user interface of the first third party application; and causes the updated first user interface of the first third party application to be displayed in each The information on the display unit is provided (e.g., by the information providing unit 818) to the respective display unit.

In some embodiments, device 800 includes a digital personal assistant process (eg, digital assistant unit 828) with speech recognition capabilities. In some embodiments, the processing unit 802 is further configured to: based on a third party application from the first third party application for values associated with the user interface template populating the user interface for the respective display unit Determining (eg, by decision unit 820) the capabilities of the first third party application by a plurality of requests for the program; generating an index of the capabilities of the first third party application (eg, by index generation unit 822); And providing voice access to the capabilities of the first third party application (e.g., by voice providing unit 824) via a digital personal assistant processing program (e.g., via digital assistant unit 828).

In some embodiments, processing unit 802 is further configured to: detect (eg, by detecting unit 826) that the predetermined condition has been met; and in response to detecting that the predetermined condition has been met: re-determining (eg, The capability of the first third party application is determined by the determining unit 820; and an index of the capabilities of the first third party application is regenerated (e.g., by the index generating unit 822).

The information processing method described above is implemented as needed by executing one or more functional modules in an information processing device such as a general purpose processor (eg, as described above with respect to FIGS. 1A and 3) or a special application chip The operation in .

The operations described above with reference to Figures 7A through 7I are implemented as desired by the components depicted in Figures 1A-1B or 8. For example, receive operations 702 and 712, request operation 710, fill operation 722, generate operation 730, and send operation 736 are implemented by event classifier 170, event organizer 180, and event processing routine 190 as needed. The event monitor 171 in the event classifier 170 detects the contact on the touch-sensitive display 112, and the event dispatcher module 174 delivers event information to application 136-1. The respective event recognizer 180 of the application 136-1 compares the event information with the respective event definition 186 and determines whether the first contact at the first location on the touch-sensitive surface (or the orientation of the device) corresponds to a predefined An event or sub-event (such as the selection of an object on the user interface, or the rotation of the device from a certain direction to another orientation). When a respective predefined event or sub-event is detected, event recognizer 180 initiates an event processing routine 190 associated with the detection of an event or sub-event. The event processing routine 190 uses or invokes the data update program 176 or the object update program 177 to update the application internal state 192 as needed. In some embodiments, event processing routine 190 accesses respective GUI update programs 178 to update the content displayed by the application. Similarly, the manner in which other programs may be implemented based on the components depicted in Figures 1A-1B will be apparent to those of ordinary skill in the art.

The foregoing description has been described with reference to specific embodiments for purposes of explanation. However, the illustrative statements above are not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention Various described embodiments.

700‧‧‧ method

Claims (20)

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device having one or more processors and memory, cause the device Executing: receiving a first display request for displaying a user interface of a first third party application on a respective display in communication with the electronic device; in response to receiving the first display request: obtaining a first a user interface template configured to be used by a plurality of third party applications; and requesting, from the first third party application, one or more values for populating the first user interface template; Receiving, by the third-party application, a first set of values for populating the first user interface template from the first third-party application while being executed by the third-party application; receiving from the first third-party application The first set of values is populated with the first user interface template; the first user interface template populated by the first set of values is used to generate a first for the first third party application UI; and for the first user such that the first to third application of the interface can be transmitted to the respective display information on the display in the respective display. The medium of claim 1, wherein the one or more programs further comprise instructions that, when executed by an electronic device having one or more processors and memory, cause the device to execute: for the first The first user interface of the third party application is displayed on the respective display while receiving from an input device for the respective display Corresponding to a respective input of a selection of one of the first user interfaces; and in response to receiving the respective input, providing information to the first third party application executing on the electronic device, The information enables the first third party application to respond to the selection of the option in the first user interface; receiving, from the first third party application, the selection of the option from the first third party application The response; and sending the information to the respective display, the information enabling the response to the selection of the option to be provided to the user. The medium of claim 1, wherein the one or more programs further comprise instructions that, when executed by an electronic device having one or more processors and memory, cause the device to execute: for the first While the first user interface of the third party application is displayed on the respective display: receiving information indicating selection of one of the individual options of the first user interface; and responding to receiving the indication of the first The selected information of the individual options in the user interface: obtaining a second user interface template different from the first user interface template; and requesting to fill the first third party application from the first user interface template One or more values of the user interface template; receiving, from the first third party application, the first third party application to load the second user interface template while the first third party application is executing on the electronic device a second set of values; the second user interface template is populated by the second set of values received from the first third party application; Generating a second user interface for the first third party application by using the second user interface template populated by the second set of values; and causing the first third party application to be used Information that the second user interface can display on the respective display is sent to the respective display. The medium of claim 1, wherein the one or more programs further comprise instructions that, when executed by an electronic device having one or more processors and memory, cause the device to execute: The first user interface of the first third party application can display the information displayed on the respective display to the respective display: the receiving will be used for one of the second third party applications different from the first third party application a second display request displayed by the user interface of the three-party application on the respective display; in response to receiving the second display request, requesting to fill the first user interface from the second third-party application One or more values of the template; receiving, from the second third party application, a third set of values for populating the first user interface template while the second third party application is executing on the electronic device Filling the first user interface template with the third set of values received from the second third party application; using the first user interface template filled with the third value set Generating a second user interface for the second third party application; and transmitting information for the second user interface for the second third party application to be displayed on the respective Individual monitors. The medium of claim 1, wherein: the device comprises a plurality of processing programs, the plurality of processing programs including a first third party application processing program for executing the first third party application and a set of one or more intermediation handlers for forwarding information between the respective display and the first third party application handler; the one or more for populating the first user interface template The request for the value is generated by an intermediary handler from the set of one or more mediation handlers; and the first user interface is processed by an intermediary from the set of one or more mediation handlers The program is generated. The medium of claim 1, wherein one or more of the values in the first set of values are responded to by the first third party application to one of the first user interface templates or The request for multiple values is retrieved from a remote source. The medium of claim 1, wherein: the respective display is a display in a vehicle information display system viewable from a driver's seat of a vehicle; and the first user interface template is a plurality of predetermined user interfaces In one of the templates, the predetermined user interface templates are configured to generate a user interface for the first third party application that prevents driver distraction on the respective displays. The medium of claim 1 wherein the first user interface template is used by a plurality of different third party applications, the third party application comprising the first third party application and a second third party application; Populating the first user interface template from values received from a respective third party application includes inserting, by the respective third party application, a value belonging to a first value category for insertion into one of the respective displays In a predetermined area of a predetermined font size in the interface; and for the first user interface template: The first third party application identifies a first information type as the first value category; and the second third party application identifies the second information type different from the first information type as the first value category . The medium of claim 8, wherein for the first user interface template: the first third party application identifies the name of the remotely generated content station as the first value category; and the second third party application The name of the predetermined playlist is identified as the first value category. The media of claim 8, wherein, for the first user interface template: the first third-party application identifies a name of the content that can be streamed to the device as the first value category; and the second The three-party application identifies the name of the content stored on the device as the first value category. The medium of claim 1, wherein: the value of the first third party application is configured as a tree structure including a plurality of nodes having predetermined characteristics and a tree type with one or more other nodes in the tree structure And the request for populating the one or more values of the first user interface template includes one or more of: returning a child corresponding to each of the plurality of nodes Requesting one of the values of one of the list of nodes; returning a request for one of the values of a list of characteristics corresponding to one of the plurality of nodes; providing a request for the content represented by the respective node; providing Searching for a request to match a list of nodes; and Providing a request for an action list specific to the first third party application may take such action while providing access to the content associated with the first third party application. The medium of claim 1, wherein: the first third party application is a media presentation application; the device comprises a first party application that is a media presentation application; the first party application includes a first a user interface, the first party user interface having a plurality of selectable visual cues for requesting execution of a media presentation operation and one or more content presentations at respective locations in the first party user interface And generating the first user interface for the first third party application includes generating a user interface, the user interface including a selectable visual cue and one or more content presentation areas, the selectable visual The cues and the one or more content presentation areas correspond to the selectable visual cues and content presentation areas of the first party user interface in function and location. The medium of claim 1, after the information for enabling the first user interface for the first third party application to be displayed on the respective display is sent to the respective display: from the first The third party application receives an updated first set of values for populating the first user interface template; and in response to receiving the updated first set of values for populating the first user interface template: by receiving Filling the first user interface template from the updated first value set of the first third party application; generating the first user interface template using the updated first value set An updated first user of a third-party application And an interface that enables the updated first user interface for the first third party application to be displayed on the respective display to be provided to the respective display. The media of claim 1, wherein: the device comprises a digital personal assistant processing program having speech recognition capability; and wherein the one or more programs further comprise instructions having one or more processors and memories Executing, when executed by an electronic device, the device is executed based on the first third party application from the first third party application being associated with a user interface template populated with a user interface for the respective display The plurality of requested responses of the value determine the capabilities of the first third party application; generate an index of the capabilities of the first third party application; and provide the first third party via the digital personal assistant processing program Voice access for these capabilities of the application. The medium of claim 14, wherein the one or more programs further comprise instructions that, when executed by an electronic device having one or more processors and memory, cause the device to perform: detecting a predetermined condition has been And being responsive to detecting that the predetermined condition has been met: re-determining the capabilities of the first third party application; and regenerating the index of the capabilities of the first third party application. A method comprising: at an electronic device having one or more processors and memory: receiving a user interface of a first third party application a first display request on a respective display of the electronic device communication; in response to receiving the first display request: obtaining a first user interface template configured to be used by a plurality of third party applications; Requesting, from the first third-party application, one or more values for populating the first user interface template; the first third-party application is executing on the electronic device from the first third-party application The program receives a first set of values for populating the first user interface template; populating the first user interface template with the first set of values received from the first third-party application; using the first value Collecting the first user interface template to generate a first user interface for the first third party application; and enabling the first user interface for the first third party application to be displayed on Information on the respective displays is sent to the respective display. An electronic device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be by the one or more Executing by the processor, the one or more programs include instructions for: receiving a first user interface of a first third party application to display on a respective display in communication with the electronic device Receiving a first display request in response to receiving the first display request: obtaining a first user interface template configured to be used by a plurality of third party applications; Requesting, from the first third-party application, one or more values for populating the first user interface template; the first third-party application is executing on the electronic device from the first third-party application The program receives a first set of values for populating the first user interface template; populating the first user interface template with the first set of values received from the first third-party application; using the first value Collecting the first user interface template to generate a first user interface for the first third party application; and enabling the first user interface for the first third party application to be displayed on Information on the respective displays is sent to the respective display. The device of claim 17, wherein the one or more programs further comprise instructions for: displaying the first user interface for the first third party application on the respective display while Inputting from one of the respective displays receives a respective input corresponding to a selection of one of the first user interfaces; and in response to receiving the respective input, the first third party application is caused The program can provide information corresponding to the selection of the option in the first user interface to the first third party application executing on the electronic device; receiving the first third party application from the first The third party application's response to the selection of the option; and the response that will enable the response to the selection of the option to be provided to the user to be sent to the respective display. The device of claim 17, wherein the one or more programs further comprise instructions for: displaying the first user interface for the first third party application Simultaneously on a respective display: receiving information indicative of a selection of one of the respective ones of the first user interface; and responding to receiving the selection indicating the respective one of the first user interfaces Information: obtaining a second user interface template different from the first user interface template; and requesting, by the first third party application, to fill one or more values of the second user interface template; Receiving, by the first third party application, a second set of values for populating the second user interface template from the first third party application; receiving from the first third party application The second set of values of the program populates the second user interface template; generating a second user for the first third party application by using the second user interface template populated by the second set of values The interface; and information that enables the second user interface for the first third party application to be displayed on the respective display to be sent to the respective display. The device of claim 17, wherein the one or more programs further comprise instructions for enabling the first user interface for the first third party application to be displayed on the respective display After the information is sent to the respective displays: receiving a second display for displaying a user interface different from the second third party application of the first third party application on the respective display Requesting, in response to receiving the second display request, from the second third party application Searching for one or more values for populating the first user interface template; receiving, from the second third party application, the first third party application while the third third party application is executing on the electronic device a third set of values of the user interface template; populating the first user interface template with the third set of values received from the second third party application; using the first user populated by the third set of values The interface template generates a second user interface for the second third party application; and transmits information for enabling the second user interface for the second third party application to be displayed on the respective display To the respective display.