Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
  • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
  • G06F3/04817—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance using icons
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
  • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
  • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
  • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
  • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
  • G06F3/0486—Drag-and-drop
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
  • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
  • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
  • G06F2203/048—Indexing scheme relating to G06F3/048
  • G06F2203/04808—Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen

Definitions

• This relates generally to electronic devices with touch-sensitive surfaces, including but not limited to electronic devices with touch-sensitive surfaces that use heuristics and other gesture recognition techniques to distinguish between a variety of touch inputs, including drag and drop gestures, long press gestures, swipe gestures, and the like.
• touch-sensitive surfaces As input devices for computers and other electronic computing devices has increased significantly in recent years.
• exemplary touch-sensitive surfaces include touchpads and touch-screen displays. Such surfaces are widely used to manipulate user interface objects on a display.
• Exemplary manipulations include dragging and dropping objects or user interface elements, such as to move information from one application to another, or from one part of an application or data structure to another.
• Other manipulations include selecting user interface objects by touching them in a predefined manner, launching applications by tapping or otherwise interacting with corresponding application launch icons or other user interface elements, accessing application features by performing swipe, drag, tap, pinch, long press, deep press (e.g., with intensity above a threshold) and other touch inputs on respective user interface elements of an application's user interface.
• gesture recognizers with distinct event or gesture definitions have been used to help with such disambiguation.
• Disambiguating between touch-based drag and drop gestures, and spring loading (sometimes written “springloading") gestures e.g., to open an object so as to see elements of the object
• swipe and long press gestures on the other hand
• inefficiency can relate both to the mechanisms for disambiguation, and the delivery of touch input information to various software mechanisms once disambiguation, or at least a portion of the disambiguation, has been achieved.
• the device is a desktop computer.
• the device is portable (e.g., a notebook computer, tablet computer, or handheld device).
• the device is a personal electronic device (e.g., a wearable electronic device, such as a watch).
• the device has a touchpad.
• the device has a touch-sensitive display (also known as a "touch screen” or "touch-screen display").
• the device has a graphical user interface (GUI), one or more processors, memory and one or more modules, programs or sets of instructions stored in the memory for performing multiple functions.
• GUI graphical user interface
• the user interacts with the GUI primarily through stylus and/or finger contacts and gestures on the touch-sensitive surface.
• the functions optionally include image editing, drawing, presenting, word processing, spreadsheet making, game playing, telephoning, video conferencing, e-mailing, instant messaging, workout support, digital photographing, digital videoing, web browsing, digital music playing, note taking, and/or digital video playing. Executable instructions for performing these functions are, optionally, included in a non- transitory computer readable storage medium or other computer program product configured for execution by one or more processors.
• a method is performed at an electronic device with a display, a touch-sensitive surface, and optionally one or more sensors to detect intensities of contacts with the touch-sensitive surface.
• the method includes: displaying, on the display, a user interface for a first application that includes a plurality of user interface objects, and while displaying the user interface for the first application, receiving, at the first application, information (e.g., one or more touch events) that describes a first touch detected at a location on the touch-sensitive surface that corresponds to a respective user interface object of the plurality of user interface objects in the first application.
• information e.g., one or more touch events
• the method includes, in response to receiving the information that describes the first touch, and in accordance with a determination that the first touch meets predefined criteria associated with initiating a respective type of interface operation: initiating a user interface operation of the respective type involving the respective user interface object; while the first touch continues to be detected on the touch-sensitive surface and while the user interface operation involving the respective user interface object continues, receiving, at the first application, instructions to ignore the first touch with respect to user interface operations other than the respective type of interface operation, and ignoring, by the first application, the first touch with respect to user interface operations other than the respective type of interface operation.
• the respective type of interface operation is a drag operation, for dragging an object from the first application or a view of the first application; or a drag and drop operation, in which a dragged object is dropped into a different application or application view than the one from which it was dragged; or a springloading operation, in which a view or container in which content can be stored is opened or a control of a view or object is activated.
• a method is performed at an electronic device with a display and a touch-sensitive surface.
• the method includes displaying, on the display, a user interface for a first application that includes a plurality of user interface objects; and, while displaying the user interface for the first application, receiving, at the first application, information that describes a first touch.
• the first touch is detected at a location on the touch-sensitive surface that corresponds to a respective user interface object of the plurality of user interface objects in the user interface for the first application and moves outside of a region corresponding to the user interface for the first application into a region corresponding to a user interface for a second application.
• the method also includes, in response to receiving the information that describes the first touch: in accordance with a determination that the first touch does not meet predefined criteria associated with initiating a respective type of interface operation, continuing to provide information that describes the first touch to the first application even after the first touch moves outside of the region corresponding to the user interface for the first application and into the region corresponding to the user interface for the second application.
• the method further includes, in accordance with a determination that the first touch meets the predefined criteria associated with initiating the respective type of interface operation: initiating a user interface operation of the respective type involving the respective user interface object; and, while the first touch continues to be detected on the touch-sensitive surface and while the user interface operation involving the respective user interface object continues: receiving, at the first application, instructions to ignore the first touch with respect to user interface operations other than the respective type of interface operation; ignoring, by the first application, the first touch with respect to user interface operations other than the respective type of interface operation.; providing information that describes the first touch to the first application while the first touch is within the region corresponding to the user interface for the first application; and switching to providing information that describes the first touch to the second application when the first touch moves outside of the region corresponding to the user interface for the first application into the region corresponding to the user interface for the second application.
• a method is performed at an electronic device with a display, a touch-sensitive surface, and optionally one or more sensors to detect intensities of contacts with the touch-sensitive surface.
• the method includes: displaying, on the display, a user interface including a plurality of views, including a first view that is associated with a first set of one or more standard gesture recognizers used to process touch inputs that are directed to the first view, and a second view that is associated with a second set of one or more standard gesture recognizers used to process touch inputs that are directed to the second view, wherein one or more of the plurality of views are associated with a first interaction identifier (e.g., a drag, drop, or springloading interaction identifier) that indicates that a first set of one or more supplemental gesture recognizers can be added to the corresponding view.
• a first interaction identifier e.g., a drag, drop, or springloading interaction identifier
• the method further includes detecting, via the touch-sensitive surface, a touch input at a location on the touch-sensitive surface that corresponds to a respective view of the plurality of views; and in response to detecting, via the touch-sensitive surface, the touch input: in accordance with a determination that the touch input meets supplemental- gesture-recognizer addition criteria, wherein the supplemental-gesture-recognizer addition criteria include a criterion that is met when the respective view has the first interaction identifier, adding the first set of one or more supplemental gesture recognizers associated with the first interaction identifier to the respective view; and processing the touch input with the one or more supplemental gesture recognizers and the one or more standard gesture recognizers associated with the respective view.
• the method further includes, in accordance with a determination that the touch input does not meet the supplemental-gesture-recognizer addition criteria, processing the touch input at the respective view without adding the first set of one or more supplemental gesture recognizers associated with the first interaction identifier to the respective view.
• a method is performed at an electronic device with a display, a touch-sensitive surface, and optionally one or more sensors to detect intensities of contacts with the touch-sensitive surface.
• the method includes: displaying, on the display, a user interface of an application; and while displaying the user interface of the application, detecting a user input that corresponds to a portion of the user interface of the application that is associated with a plurality of gesture recognizers, wherein: a first set of one or more standard gesture recognizers of the plurality of gesture recognizers were associated with (e.g., assigned to) the portion of the user interface by the application; a second set of one or more supplemental gesture recognizers in the plurality of gesture recognizers were associated with (e.g., assigned to) the portion of the user interface by a system process (e.g., an application independent process); and a first failure requirement involving a first standard gesture recognizer and a first supplemental gesture recognizer were associated with (e.g., assigned
• the method further includes, in response to detecting the user input, processing the user input in accordance with the first standard gesture recognizer, the first supplemental gesture recognizer, and the first failure requirement.
• the first set of one or more standard gesture recognizers and the second set of one or more gesture recognizers are arranged in a gesture recognition hierarchy, wherein the first failure requirement includes a requirement that a first respective gesture recognizer fail in order for a second respective gesture recognizer to successfully recognize a gesture, wherein the first respective gesture recognizer is at a higher level in the gesture recognition hierarchy than the second respective gesture recognizer.
• an electronic device includes a display, a touch-sensitive surface, optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface, one or more processors, memory, and one or more programs; the one or more programs are stored in the memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of the operations of any of the methods described herein.
• a computer readable storage medium has stored therein instructions which when executed by an electronic device with a display, a touch-sensitive surface, and optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface, cause the device to perform or cause performance of the operations of any of the methods described herein.
• a graphical user interface on an electronic device with a display, a touch-sensitive surface, optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface, a memory, and one or more processors to execute one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, which are updated in response to inputs, as described in any of the methods described herein.
• an electronic device includes: a display, a touch-sensitive surface, and optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface; and means for performing or causing performance of the operations of any of the methods described herein.
• an information processing apparatus for use in an electronic device with a display and a touch-sensitive surface, and optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface, includes means for performing or causing performance of the operations of any of the methods described herein.
• electronic devices with displays, touch-sensitive surfaces and optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface are provided with faster, more efficient methods and interfaces for processing touch-based inputs, including drag and drop gestures, long press gestures, and other gestures that could potentially be validly recognized for use in the same application, or application view, thereby increasing the effectiveness, efficiency, and user satisfaction with such devices.
• Such methods and interfaces may complement or replace conventional methods for processing touch inputs in such electronic devices.
• Figure 1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.
• Figure 2 illustrates a portable multifunction device having a touch screen in accordance with some embodiments.
• Figure 3 A is a block diagram of an example multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.
• Figure 3B is a block diagram illustrating exemplary components for event handling in accordance with some embodiments.
• Figure 3C is a block diagram illustrating exemplary classes and instances of gesture recognizers in accordance with some embodiments.
• Figures 3D is a block diagram illustrating the flow of event information in accordance with some embodiments.
• Figure 4A illustrates an example user interface for a menu of applications on a portable multifunction device in accordance with some embodiments.
• Figure 4B illustrates an example user interface for a multifunction device with a touch-sensitive surface that is separate from the display in accordance with some embodiments.
• Figures 4C-4E illustrate examples of dynamic intensity thresholds in accordance with some embodiments.
• Figures 5 A-5R illustrate example user interfaces for drag and drop gestures and operations, and springloading gestures and operations, in accordance with some embodiments.
• Figure 6A illustrates touch event generation, phases and cancellation, and drag event generation and phases, during performance of a drag gesture that crosses from the user interface region of a first application or application view to the user interface region of a second application or application view, in accordance with some embodiments.
• Figure 6B illustrates touch event generation and phases while a touch is moved across the user interface region of a first application or application view to the user interface region of a second application or application view, without detaching the touch, in accordance with some embodiments.
• Figure 6C depicts mechanisms for detecting a touch, detecting the start of a drag gesture in a source process, detaching the touch, associating a session ID with the touch, and providing the touch to a destination process, in accordance with some embodiments.
• Figures 6D and 6E depicts a gesture recognition hierarchy before and after the addition of one or more supplemental gesture recognizers to an application view, in accordance with some embodiments.
• Figures 7A-7F are flow diagrams illustrating a method of processing touch inputs recognized as drag gestures, in accordance with some embodiments.
• Figures 8A-8E are flow diagrams illustrating a method of dynamically adding supplemental gesture recognizers to an application or application view in response to detecting a touch input when specified criteria are met, in accordance with some
• Figures 9A-9D are flow diagrams illustrating a method of establishing failure dependencies between a first set of standard gesture recognizers associated with a portion of the user interface of an application and a second set of supplemental gesture recognizers associated with the same portion of the user interface of the application, in accordance with some embodiments.
• DESCRIPTION OF EMBODIMENTS are flow diagrams illustrating a method of establishing failure dependencies between a first set of standard gesture recognizers associated with a portion of the user interface of an application and a second set of supplemental gesture recognizers associated with the same portion of the user interface of the application, in accordance with some embodiments.
• Many electronic devices have graphical user interfaces and touch-sensitive surfaces, whether integrated with a display or otherwise, for receiving touch inputs by a user. As the number of touch input-based gestures increases, disambiguation between various possible gestures becomes both more difficult, and more important for successful and efficient use of such electronic devices. In fact, multiple gestures useable in the same user- interface region may begin with the same initial touch characteristics, such as a stationary touch that remains stationary (e.g., moves less than a predefined distance or amount) for at last a predefined initial amount of time.
• gesture recognizers each for recognizing a respective gesture or type of gesture, and to establish failure requirements that certain gesture recognizers fail to recognize a user input before other gesture recognizers are allowed to successfully recognizer the user input as a particular gesture. For example, when a user interface region accepts both a long press input for initiating a first operation, and a drag input for starting a drag and drop operation, the gesture recognizer for recognizing long press gestures may be made dependent on the failure of a drag start gesture recognizer for recognizing the start of a drag and drop operation.
• gesture recognizers may be added by an application to user interface regions for the application, while supplemental gesture recognizers may be added by a system process to the same user interface regions when certain criteria are satisfied.
• the number of gesture recognizers established for a particular user interface region may be kept relatively low until the supplemental gesture recognizers are needed, or potentially needed, as indicated by the aforementioned criteria being satisfied.
• gesture recognizers for handling drag and drop operations may be added to user interface region, as supplemental gesture recognizers, when they are needed or potentially needed.
• interaction identifiers are associated with applications, or application views, to indicate which supplemental gesture recognizers, if any, can be added to those applications or application views. For example, only applications or application views that can participate in drag and drop operations as associated with an interaction identifier for drag operations.
• a second distinct interaction identifier may be used for drop interactions, since some application regions may be the source of a drag and drop operation, but not the destination, or vice versa.
• a third interaction identifier may be used for springloading interactions, since some, but not all application regions, may include containers that can be opened, or have controls that can be activated using a springloading gesture.
• interaction identifiers In this way, only applications and views that allow certain types of gestures or operations, such as drag and drop gestures and operations, or springloading gestures and operations, are associated with corresponding interaction identifiers.
• Figures 1, 2, and 3 A show example devices
• Figure 3B-3D show event and gesture recognition and touch even delivery mechanisms
• Figures 4A-4B and 5A-5R illustrate example user interfaces for drag and drop operations
• Figure 6A-6C depict touch event and draft event generation and management mechanisms.
• Figures 7A-7F illustrate a flow diagram of a method of processing touch inputs recognized as drag gestures.
• Figures 8A-8E illustrate a flow diagram of a method of dynamically adding supplemental gesture
• Figures 9A-9D illustrate a flow diagram of a method of establishing failure dependencies between a first set of standard gesture recognizers associated with a portion of the user interface of an application and a second set of supplemental gesture recognizers associated with the same portion of the user interface of the application.
• the user interfaces in Figures 5A-5R and the mechanisms shown in Figures 3B-3D and 6A-6C are used to illustrate the processes in Figures 7A-7F, 8A-8E, and 9A-9D.
• first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact, unless the context clearly indicates otherwise.
• the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA and/or music player functions.
• portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California.
• Other portable electronic devices such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch-screen displays and/or touchpads), are, optionally, used.
• the device is not a portable communications device, but is a desktop computer with a touch- sensitive surface (e.g., a touch-screen display and/or a touchpad).
• a touch-sensitive surface e.g., a touch-screen display and/or a touchpad.
• an electronic device that includes a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device optionally includes one or more other physical user-interface devices, such as a physical keyboard, a mouse and/or a joystick.
• the device typically supports a variety of applications, such as one or more of the following: a note taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.
• applications such as one or more of the following: a note taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application
• the various applications that are executed on the device optionally use at least one common physical user-interface device, such as the touch-sensitive surface.
• One or more functions of the touch-sensitive surface as well as corresponding information displayed on the device are, optionally, adjusted and/or varied from one application to the next and/or within a respective application.
• a common physical architecture (such as the touch- sensitive surface) of the device optionally supports the variety of applications with user interfaces that are intuitive and transparent to the user.
• FIG. 1 is a block diagram illustrating portable multifunction device 100 with touch-sensitive display system 112 in accordance with some embodiments.
• Touch- sensitive display system 112 is sometimes called a "touch screen" for convenience, and is sometimes simply called a touch-sensitive display.
• Device 100 includes memory 102 (which optionally includes one or more computer readable storage mediums), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem 106, other input or control devices 116, and external port 124.
• Device 100 optionally includes one or more optical sensors 164.
• Device 100 optionally includes one or more intensity sensors 165 for detecting intensity of contacts on device 100 (e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100).
• Device 100 optionally includes one or more tactile output generators 167 for generating tactile outputs on device 100 (e.g., generating tactile outputs on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300).
• the term "tactile output” is physical displacement of a device relative to a previous position of the device, physical displacement of a component (e.g., a touch-sensitive surface) of a device relative to another component (e.g., housing) of the device, or displacement of the component relative to a center of mass of the device that will be detected by a user with the user's sense of touch.
• a component e.g., a touch-sensitive surface
• another component e.g., housing
• the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in physical characteristics of the device or the component of the device.
• a touch-sensitive surface e.g., a touch-sensitive display or trackpad
• the user is, optionally, interpreted by the user as a "down click" or "up click" of a physical actuator button.
• a user will feel a tactile sensation such as an "down click” or “up click” even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movements.
• movement of the touch-sensitive surface is, optionally, interpreted or sensed by the user as "roughness" of the touch-sensitive surface, even when there is no change in smoothness of the touch-sensitive surface. While such interpretations of touch by a user will be subject to the individualized sensory perceptions of the user, there are many sensory perceptions of touch that are common to a large majority of users.
• a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an "up click,” a “down click,” “roughness")
• the generated tactile output corresponds to physical displacement of the device or a component thereof that will generate the described sensory perception for a typical (or average) user.
• device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components.
• the various components shown in Figure 1 are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing and/or application specific integrated circuits.
• Memory 102 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU(s) 120 and the peripherals interface 118, is, optionally, controlled by memory controller 122.
• Peripherals interface 118 can be used to couple input and output peripherals of the device to CPU(s) 120 and memory 102.
• the one or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data.
• peripherals interface 118, CPU(s) 120, and memory controller 122 are, optionally, implemented on a single chip, such as chip 104. In some other embodiments, they are, optionally, implemented on separate chips.
• RF (radio frequency) circuitry 108 receives and sends RF signals, also called electromagnetic signals.
• RF circuitry 108 converts electrical signals to/from electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals.
• RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SFM) card, memory, and so forth.
• SFM subscriber identity module
• RF circuitry 108 optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication.
• networks such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication.
• the wireless communication optionally uses any of a plurality of communications standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM
• EDGE high-speed downlink packet access
• HSDPA high-speed uplink packet access
• HSUPA high-speed uplink packet access
• EV-DO Evolution, Data-Only
• HSPA HSPA+, Dual-Cell HSPA
• DC-HSPDA long term evolution
• LTE long term evolution
• NFC near field communication
• W-CDMA wideband code division multiple access
• CDMA code division multiple access
• TDMA time division multiple access
• Bluetooth Wireless Fidelity
• Wi-Fi e.g., IEEE 802.11a, IEEE 802.1 lac, IEEE 802.1 lax, IEEE 802.1 lb, IEEE 802.1 lg and/or IEEE 802.1 In
• VoIP voice over Internet Protocol
• Wi-MAX a protocol for e-mail
• EVIAP Internet message access protocol
• POP post office protocol
• instant messaging e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Mess
• Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100.
• Audio circuitry 110 receives audio data from peripherals interface 118, converts the audio data to an electrical signal, and transmits the electrical signal to speaker 111.
• Speaker 111 converts the electrical signal to human-audible sound waves.
• Audio circuitry 110 also receives electrical signals converted by microphone 113 from sound waves.
• Audio circuitry 110 converts the electrical signal to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data is, optionally, retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118.
• audio circuitry 110 also includes a headset jack (e.g., 212, Figure 2).
• the headset jack provides an interface between audio circuitry 110 and removable audio input/output peripherals, such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone
• I/O subsystem 106 couples input/output peripherals on device 100, such as touch-sensitive display system 112 and other input or control devices 116, with peripherals interface 118.
• I/O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices.
• the one or more input controllers 160 receive/send electrical signals from/to other input or control devices 116.
• the other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth.
• input controlled s) 160 are, optionally, coupled with any (or none) of the following: a keyboard, infrared port, USB port, stylus, and/or a pointer device such as a mouse.
• the one or more buttons optionally include an up/down button for volume control of speaker 111 and/or microphone 113.
• the one or more buttons optionally include a push button (e.g., 206, Figure 2).
• Touch-sensitive display system 112 provides an input interface and an output interface between the device and a user.
• Display controller 156 receives and/or sends electrical signals from/to touch-sensitive display system 112. Touch-sensitive display system 112 displays visual output to the user.
• the visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively termed “graphics”).
• some or all of the visual output corresponds to user interface objects.
• the term "affordance” is a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to inputs directed toward the graphical user interface object).
• user-interactive graphical user interface objects include, without limitation, a button, slider, icon, selectable menu item, switch, hyperlink, or other user interface control.
• Touch-sensitive display system 112 has a touch-sensitive surface, sensor or set of sensors that accepts input from the user based on haptic and/or tactile contact.
• Touch-sensitive display system 112 and display controller 156 (along with any associated modules and/or sets of instructions in memory 102) detect contact (and any movement or breaking of the contact) on touch-sensitive display system 112 and converts the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed on touch-sensitive display system 112.
• user-interface objects e.g., one or more soft keys, icons, web pages or images
• a point of contact between touch-sensitive display system 1 12 and the user corresponds to a finger of the user or a stylus.
• Touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments.
• LCD liquid crystal display
• LPD light emitting polymer display
• LED light emitting diode
• Touch-sensitive display system 112 and display controller 156 optionally detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 112.
• touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 112.
• projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California.
• Touch-sensitive display system 112 optionally has a video resolution in excess of 100 dpi.
• the touch screen video resolution is in excess of 400 dpi (e.g., 500 dpi, 800 dpi, or greater).
• the user optionally makes contact with touch-sensitive display system 112 using any suitable object or appendage, such as a stylus, a finger, and so forth.
• the user interface is designed to work with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen.
• the device translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user.
• device 100 in addition to the touch screen, device 100 optionally includes a touchpad (not shown) for activating or deactivating particular functions.
• the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output.
• the touchpad is, optionally, a touch-sensitive surface that is separate from touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touch screen.
• Device 100 also includes power system 162 for powering the various components.
• Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light- emitting diode (LED)) and any other components associated with the generation,
• power sources e.g., battery, alternating current (AC)
• AC alternating current
• a recharging system e.g., a recharging system
• a power failure detection circuit e.g., a power failure detection circuit
• a power converter or inverter e.g., a power converter or inverter
• a power status indicator e.g., a light- emitting diode (LED)
• Device 100 optionally also includes one or more optical sensors 164.
• Figure 1 shows an optical sensor coupled with optical sensor controller 158 in I/O subsystem 106.
• Optical sensor(s) 164 optionally include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors.
• CMOS complementary metal-oxide semiconductor
• Optical sensor(s) 164 receive light from the environment, projected through one or more lens, and converts the light to data representing an image.
• imaging module 143 also called a camera module
• optical sensor(s) 164 optionally capture still images and/or video.
• an optical sensor is located on the back of device 100, opposite touch-sensitive display system 112 on the front of the device, so that the touch screen is enabled for use as a viewfinder for still and/or video image acquisition.
• another optical sensor is located on the front of the device so that the user's image is obtained (e.g., for selfies, for videoconferencing while the user views the other video conference participants on the touch screen, etc.).
• Device 100 optionally also includes one or more contact intensity sensors 165.
• Figure 1 shows a contact intensity sensor coupled with intensity sensor controller 159 in I/O subsystem 106.
• Contact intensity sensor(s) 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface).
• Contact intensity sensor(s) 165 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment.
• contact intensity information e.g., pressure information or a proxy for pressure information
• At least one contact intensity sensor is collocated with, or proximate to, a touch-sensitive surface (e.g., touch-sensitive display system 112).
• a touch-sensitive surface e.g., touch-sensitive display system 112
• at least one contact intensity sensor is located on the back of device 100, opposite touch-screen display system 112 which is located on the front of device 100.
• Device 100 optionally also includes one or more proximity sensors 166.
• FIG. 1 shows proximity sensor 166 coupled with peripherals interface 118.
• proximity sensor 166 is coupled with input controller 160 in I/O subsystem 106.
• the proximity sensor turns off and disables touch-sensitive display system 112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).
• Device 100 optionally also includes one or more tactile output generators 167.
• FIG. 1 shows a tactile output generator coupled with haptic feedback controller 161 in I/O subsystem 106.
• Tactile output generator(s) 167 optionally include one or more
• Tactile output generator(s) 167 receive tactile feedback generation instructions from haptic feedback module 133 and generates tactile outputs on device 100 that are capable of being sensed by a user of device 100.
• At least one tactile output generator is collocated with, or proximate to, a touch-sensitive surface (e.g., touch-sensitive display system 112) and, optionally, generates a tactile output by moving the touch-sensitive surface vertically (e.g., in/out of a surface of device 100) or laterally (e.g., back and forth in the same plane as a surface of device 100).
• at least one tactile output generator sensor is located on the back of device 100, opposite touch-sensitive display system 112, which is located on the front of device 100.
• Device 100 optionally also includes one or more accelerometers 168.
• Figure 1 shows accelerometer 168 coupled with peripherals interface 118.
• accelerometer 168 is, optionally, coupled with an input controller 160 in I/O subsystem 106.
• information is displayed on the touch-screen display in a portrait view or a landscape view based on an analysis of data received from the one or more accelerometers.
• Device 100 optionally includes, in addition to accelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information concerning the location and orientation (e.g., portrait or landscape) of device 100.
• GPS or GLONASS or other global navigation system
• the software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact/motion module (or set of instructions) 130, graphics module (or set of instructions) 132, haptic feedback module (or set of instructions) 133, text input module (or set of instructions) 134, Global Positioning System (GPS) module (or set of instructions) 135, and applications (or sets of instructions) 136.
• memory 102 stores instructions for executing instructions.
• Device/global internal state 157 includes one or more of: active application state, indicating which applications, if any, are currently active; display state, indicating what applications, application views or other information occupy various regions of touch-sensitive display system 112; sensor state, including information obtained from the device's various sensors and other input or control devices 116; and location and/or positional information concerning the device's location and/or attitude; and optionally other state information.
• Operating system 126 e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X,
• WINDOWS or an embedded operating system such as VxWorks
• Communication module 128 facilitates communication with other devices over one or more external ports 124 and also includes various software components for handling data received by RF circuitry 108 and/or external port 124.
• External port 124 e.g., Universal Serial Bus (USB), FIREWIRE, etc.
• USB Universal Serial Bus
• FIREWIRE FireWire
• a network e.g., the Internet, wireless LAN, etc.
• the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to and/or compatible with the 30-pin connector used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California.
• the external port is a Lightning connector that is the same as, or similar to and/or compatible with the Lightning connector used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California.
• Contact/motion module 130 optionally detects contact with touch-sensitive display system 112 (in conjunction with display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel).
• Contact/motion module 130 includes various software components for performing various operations related to detection of contact (e.g., by a finger or by a stylus), such as determining if contact has occurred (e.g., detecting a finger-down event), determining an intensity of the contact (e.g., the force or pressure of the contact or a substitute for the force or pressure of the contact), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact).
• determining if contact has occurred e.g., detecting a finger-down event
• an intensity of the contact e.g., the force or pressure of the contact or a
• Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, optionally includes determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations are, optionally, applied to single contacts (e.g., one finger contacts or stylus contacts) or to multiple simultaneous contacts (e.g., "multitouch'Vmultiple finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect contact on a touchpad.
• Contact/motion module 130 optionally detects a gesture input by a user.
• a gesture is, optionally, detected by detecting a particular contact pattern.
• detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (lift off) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon).
• detecting a finger swipe gesture on the touch- sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (lift off) event.
• tap, swipe, drag, and other gestures are optionally detected for a stylus by detecting a particular contact pattern for the stylus.
• detecting a finger tap gesture depends on the length of time between detecting the finger-down event and the finger-up event, but is independent of the intensity of the finger contact between detecting the finger-down event and the finger-up event.
• a tap gesture is detected in accordance with a determination that the length of time between the finger-down event and the finger-up event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4 or 0.5 seconds), independent of whether the intensity of the finger contact during the tap meets a given intensity threshold (greater than a nominal contact-detection intensity threshold), such as a light press or deep press intensity threshold.
• a finger tap gesture can satisfy particular input criteria that do not require that the characteristic intensity of a contact satisfy a given intensity threshold in order for the particular input criteria to be met.
• the finger contact in a tap gesture typically needs to satisfy a nominal contact-detection intensity threshold, below which the contact is not detected, in order for the finger-down event to be detected.
• a similar analysis applies to detecting a tap gesture by a stylus or other contact.
• the nominal contact-detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch sensitive surface.
• a swipe gesture, a pinch gesture, a depinch gesture, and/or a long press gesture are optionally detected based on the satisfaction of criteria that are either independent of intensities of contacts included in the gesture, or do not require that contact(s) that perform the gesture reach intensity thresholds in order to be recognized.
• a swipe gesture is detected based on an amount of movement of one or more contacts; a pinch gesture is detected based on movement of two or more contacts towards each other; a depinch gesture is detected based on movement of two or more contacts away from each other; and a long press gesture is detected based on a duration of the contact on the touch-sensitive surface with less than a threshold amount of movement.
• the statement that particular gesture recognition criteria do not require that the intensity of the contact(s) meet a respective intensity threshold in order for the particular gesture recognition criteria to be met means that the particular gesture recognition criteria are capable of being satisfied if the contact(s) in the gesture do not reach the respective intensity threshold, and are also capable of being satisfied in circumstances where one or more of the contacts in the gesture do reach or exceed the respective intensity threshold.
• a tap gesture is detected based on a determination that the finger-down and finger-up event are detected within a predefined time period, without regard to whether the contact is above or below the respective intensity threshold during the predefined time period, and a swipe gesture is detected based on a determination that the contact movement is greater than a predefined magnitude, even if the contact is above the respective intensity threshold at the end of the contact movement.
• detection of a gesture is influenced by the intensity of contacts performing the gesture (e.g., the device detects a long press more quickly when the intensity of the contact is above an intensity threshold or delays detection of a tap input when the intensity of the contact is higher), the detection of those gestures does not require that the contacts reach a particular intensity threshold so long as the criteria for recognizing the gesture can be met in circumstances where the contact does not reach the particular intensity threshold (e.g., even if the amount of time that it takes to recognize the gesture changes).
• Contact intensity thresholds, duration thresholds, and movement thresholds are, in some circumstances, combined in a variety of different combinations in order to create heuristics for distinguishing two or more different gestures directed to the same input element or region so that multiple different interactions with the same input element are enabled to provide a richer set of user interactions and responses.
• the statement that a particular set of gesture recognition criteria do not require that the intensity of the contact(s) meet a respective intensity threshold in order for the particular gesture recognition criteria to be met does not preclude the concurrent evaluation of other intensity-dependent gesture recognition criteria to identify other gestures that do have a criterion that is met when a gesture includes a contact with an intensity above the respective intensity threshold. For example, in some
• first gesture recognition criteria for a first gesture - which do not require that the intensity of the contact(s) meet a respective intensity threshold in order for the first gesture recognition criteria to be met - are in competition with second gesture recognition criteria for a second gesture - which are dependent on the contact(s) reaching the respective intensity threshold.
• the gesture is, optionally, not recognized as meeting the first gesture recognition criteria for the first gesture if the second gesture recognition criteria for the second gesture are met first. For example, if a contact reaches the respective intensity threshold before the contact moves by a predefined amount of movement, a deep press gesture is detected rather than a swipe gesture.
• a swipe gesture is detected rather than a deep press gesture.
• the first gesture recognition criteria for the first gesture still do not require that the intensity of the contact(s) meet a respective intensity threshold in order for the first gesture recognition criteria to be met because if the contact stayed below the respective intensity threshold until an end of the gesture (e.g., a swipe gesture with a contact that does not increase to an intensity above the respective intensity threshold), the gesture would have been recognized by the first gesture recognition criteria as a swipe gesture.
• particular gesture recognition criteria that do not require that the intensity of the contact(s) meet a respective intensity threshold in order for the particular gesture recognition criteria to be met will (A) in some circumstances ignore the intensity of the contact with respect to the intensity threshold (e.g. for a tap gesture) and/or (B) in some circumstances still be dependent on the intensity of the contact with respect to the intensity threshold in the sense that the particular gesture recognition criteria (e.g., for a long press gesture) will fail if a competing set of intensity- dependent gesture recognition criteria (e.g., for a deep press gesture) recognize an input as corresponding to an intensity-dependent gesture before the particular gesture recognition criteria recognize a gesture corresponding to the input (e.g., for a long press gesture that is competing with a deep press gesture for recognition).
• a competing set of intensity- dependent gesture recognition criteria e.g., for a deep press gesture
• Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast or other visual property) of graphics that are displayed.
• graphics includes any object that can be displayed to a user, including without limitation text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations and the like.
• graphics module 132 stores data representing graphics to be used. Each graphic is, optionally, assigned a corresponding code. Graphics module 132 receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinate data and other graphic property data, and then generates screen image data to output to display controller 156.
• Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator(s) 167 to produce tactile outputs at one or more locations on device 100 in response to user interactions with device 100.
• Text input module 134 which is, optionally, a component of graphics module
• 132 provides soft keyboards for entering text in various applications (e.g., contacts 137, e-mail 140, IM 141, browser 147, and any other application that needs text input).
• applications e.g., contacts 137, e-mail 140, IM 141, browser 147, and any other application that needs text input).
• GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to telephone 138 for use in location-based dialing, to camera 143 as picture/video metadata, and to applications that provide location- based services such as weather widgets, local yellow page widgets, and map/navigation widgets).
• applications e.g., to telephone 138 for use in location-based dialing, to camera 143 as picture/video metadata, and to applications that provide location- based services such as weather widgets, local yellow page widgets, and map/navigation widgets).
• Applications 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof:
• contacts module 137 (sometimes called an address book or contact list);
• camera module 143 for still and/or video images
• calendar module 148 • calendar module 148;
• widget modules 149 which optionally include one or more of: weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; • widget creator module 150 for making user-created widgets 149-6;
• search module 151 • search module 151;
• video and music player module 152 which is, optionally, made up of a video player module and a music player module;
• map module 154 • map module 154;
• Examples of other applications 136 that are, optionally, stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.
• a respective software application typically has, at least during execution, an application state, indicating the state of the respective software application and its
• gesture recognizers see application internal state 321 ( Figure 3B) described below.
• contacts module 137 includes executable instructions to manage an address book or contact list (e.g., stored in application internal state 192 of contacts module 137 in memory 102 or memory 370), including: adding name(s) to the address book; deleting name(s) from the address book;
• an address book or contact list e.g., stored in application internal state 192 of contacts module 137 in memory 102 or memory 370
• telephone module 138 includes executable instructions to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in address book 137, modify a telephone number that has been entered, dial a respective telephone number, conduct a conversation and disconnect or hang up when the conversation is completed.
• the wireless communication optionally uses any of a plurality of communications standards, protocols and technologies.
• videoconferencing module 139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.
• e-mail client module 140 includes executable instructions to create, send, receive, and manage e-mail in response to user instructions.
• e-mail client module 140 makes it very easy to create and send e-mails with still or video images taken with camera module 143.
• the instant messaging module 141 includes executable instructions to enter a sequence of characters corresponding to an instant message, to modify previously entered characters, to transmit a respective instant message (for example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephony-based instant messages or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messages), to receive instant messages and to view received instant messages.
• SMS Short Message Service
• MMS Multimedia Message Service
• XMPP extensible Markup Language
• SIMPLE Apple Push Notification Service
• IMPS Internet Messaging Protocol
• transmitted and/or received instant messages optionally include graphics, photos, audio files, video files and/or other attachments as are supported in a MMS and/or an Enhanced Messaging Service (EMS).
• EMS Enhanced Messaging Service
• instant messaging refers to both telephony-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SFMPLE, APNs, or IMPS).
• workout support module 142 includes executable instructions to create workouts (e.g., with time, distance, and/or calorie burning goals); communicate with workout sensors (in sports devices and smart watches); receive workout sensor data; calibrate sensors used to monitor a workout; select and play music for a workout; and display, store and transmit workout data.
• camera module 143 includes executable
• image management module 144 includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images.
• modify e.g., edit
• present e.g., in a digital slide show or album
• browser module 147 includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
• calendar module 148 includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to do lists, etc.) in accordance with user instructions.
• widget modules 149 are mini-applications that are, optionally, downloaded and used by a user (e.g., weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or created by the user (e.g., user-created widget 149-6).
• a widget includes an HTML
• a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).
• the widget creator module 150 includes executable
• search module 151 includes executable instructions to search for text, music, sound, image, video, and/or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.
• search criteria e.g., one or more user-specified search terms
• video and music player module 152 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present or otherwise play back videos (e.g., on touch- sensitive display system 112, or on an external display connected wirelessly or via external port 124).
• device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).
• notes module 153 includes executable instructions to create and manage notes, to do lists, and the like in accordance with user instructions.
• map module 154 includes executable instructions to receive, display, modify, and store maps and data associated with maps (e.g., driving directions; data on stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.
• maps e.g., driving directions; data on stores and other points of interest at or near a particular location; and other location-based data
• online video module 155 includes executable instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen 112, or on an external display connected wirelessly or via external port 124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264.
• instant messaging module 141 rather than e-mail client module 140, is used to send a link to a particular online video.
• modules and applications correspond to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein).
• modules i.e., sets of instructions
• memory 102 optionally stores a subset of the modules and data structures identified above.
• memory 102 optionally stores additional modules and data structures not described above.
• device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad.
• a touch screen and/or a touchpad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, and the like) on device 100 is, optionally, reduced.
• the predefined set of functions that are performed exclusively through a touch screen and/or a touchpad optionally include navigation between user interfaces.
• the touchpad when touched by the user, navigates device 100 to a main, home, or root menu from any user interface that is displayed on device 100.
• a "menu button" is implemented using a touchpad.
• the menu button is a physical push button or other physical input control device instead of a touchpad.
• Figure 2 illustrates a portable multifunction device 100 having a touch screen
• the touch screen optionally displays one or more graphics within user interface (UI) 200.
• UI user interface
• a user is enabled to select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure).
• selection of one or more graphics occurs when the user breaks contact with the one or more graphics.
• the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and/or downward) and/or a rolling of a finger (from right to left, left to right, upward and/or downward) that has made contact with device 100.
• one or more taps one or more swipes (from left to right, right to left, upward and/or downward) and/or a rolling of a finger (from right to left, left to right, upward and/or downward) that has made contact with device 100.
• Device 100 optionally also includes one or more physical buttons, such as
• menu button 204 is, optionally, used to navigate to any application 136 in a set of applications that are, optionally executed on device 100.
• the menu button is implemented as a soft key in a GUI displayed on the touch-screen display.
• device 100 includes the touch-screen display, menu button 204, push button 206 for powering the device on/off and locking the device, volume adjustment button(s) 208, Subscriber Identity Module (SIM) card slot 210, head set jack 212, and docking/charging external port 124.
• Push button 206 is, optionally, used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process.
• device 100 also accepts verbal input for activation or deactivation of some functions through microphone 113.
• Device 100 also, optionally, includes one or more contact intensity sensors 165 for detecting intensity of contacts on touch-sensitive display system 112 and/or one or more tactile output generators 167 for generating tactile outputs for a user of device 100.
• FIG. 3 A is a block diagram of an example multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.
• Device 300 need not be portable.
• device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or industrial controller).
• Device 300 typically includes one or more processing units (CPU's) 310, one or more network or other communications interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components.
• CPU's processing units
• Communication buses 320 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
• Device 300 includes input/output (I/O) interface 330, including display 340, which is typically a touch-screen display.
• I/O interface 330 also optionally includes a keyboard and/or mouse (or other pointing device) 350 and touch-sensitive touchpad 355, tactile output generator 357 for generating tactile outputs on device 300 (e.g., similar to tactile output generator(s) 167 described above with reference to Figure 1), sensors 359 (e.g., optical sensors 164, accelerometer(s) 168, proximity sensor 166, and/or contact intensity sensors 165 described above with reference to Figure 1).
• sensors 359 e.g., optical sensors 164, accelerometer(s) 168, proximity sensor 166, and/or contact intensity sensors 165 described above with reference to Figure 1).
• Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices remotely located from CPU(s) 310. In some embodiments, memory 370 stores programs, modules, and data structures analogous to the programs, modules, and data structures stored in memory 102 of portable multifunction device 100 ( Figure 1), or a subset thereof. Furthermore, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100.
• memory 370 of device 300 optionally stores drawing module 380, presentation module 382, word processing module 384, website creation module 386, disk authoring module 388, and/or spreadsheet module 390, while memory 102 of portable multifunction device 100 ( Figure 1) optionally does not store these modules.
• Each of the above identified elements in Figure 3 A are, optionally, stored in one or more of the previously mentioned memory devices.
• Each of the above identified modules corresponds to a set of instructions for performing a function described above.
• the above identified modules or programs i.e., sets of instructions
• memory 370 optionally stores a subset of the modules and data structures identified above.
• memory 370 optionally stores additional modules and data structures not described above.
• Figure 3B is a block diagram illustrating exemplary components for event handling (e.g., event handling components 391) in accordance with some embodiments.
• memory 102 Figure 1 includes event recognizer global methods 312 and one or more applications (e.g., 133-1 through 133-3).
• event recognizer global methods 312 include event monitor 311, hit view determination module 314, active event recognizer determination module 316, and event dispatcher module 315.
• event recognizer global methods 312 are located within an event delivery system in operating system 126 ( Figure 1).
• event recognizer global methods 312 are implemented in a respective application 133-1.
• event recognizer global methods 312 are implemented as a stand-alone module, or a part of another module stored in memory 102 (e.g., a contact/motion module (not depicted)).
• Event monitor 311 receives event information from one or more sensors 359, touch-sensitive display 340, and/or one or more input devices 350, 355.
• Event information includes information about an event (e.g., a user touch on touch-sensitive display 156, as part of a multi -touch gesture or a motion of device 102) and/or a sub-event (e.g., a movement of a touch across touch-sensitive display 156).
• event information for a touch event includes one or more of: a location and time stamp of a touch.
• event information for a swipe event includes two or more of: a location, time stamp, direction, and speed of a swipe.
• Sensors 359, touch-sensitive display 156, and input devices 128 transmit information event and sub-event information to event monitor 311 either directly or through a peripherals interface, which retrieves and stores event information.
• sensors 359 include one or more of: proximity sensor, accelerometer(s), gyroscopes, microphone, and video camera.
• sensors 359 also include input devices 128 and/or touch-sensitive display 156.
• event monitor 311 sends requests to sensors 116 and/or the peripherals interface at predetermined intervals.
• sensors 116 and/or the peripherals interface transmit event information.
• sensors 116 and the peripheral interface transmit event information only when there is a significant event (e.g., receiving an input beyond a predetermined noise threshold and/or for more than a
• Event monitor 311 receives event information and relays the event information to event dispatcher module 315. In some embodiments, event monitor 311 determines one or more respective applications (e.g., 133-1) to which to deliver the event information. In some embodiments, event monitor 311 also determines one or more respective views 317 of the one or more respective applications to which to deliver the event information.
• respective applications e.g., 133-1
• event monitor 311 also determines one or more respective views 317 of the one or more respective applications to which to deliver the event information.
• a view is often thought of as being a window or other portion of a user interface, more technically a view is the portion of an application that manages a particular area or region of the application's user interface.
• a view is typically implemented as an instance of a particular class, or one of its subclasses, and manages a rectangular area in an application window.
• a view is an object having an associated display region or user interface portion, and also one or more computer programs, sometimes called "methods," associated with the class of which the view is an instance.
• Views are responsible for drawing content, handling multitouch events, and managing the layout of any subviews. Drawing content involves using various graphics technologies to draw shapes, images, and text inside a view's rectangular area.
• a view responds to touch events in its rectangular area either by using gesture recognizers or by handling touch events directly.
• parent views are responsible for positioning and sizing their child views and can do so dynamically. This ability to modify child views dynamically enables views to adjust to changing conditions, such as interface rotations and animations.
• Views can be viewed as building blocks that a programmer or application developer uses to construct the user interface of an application. Rather than use one view to present all content for the application, several views are typically used to build a view hierarchy. Each view in the hierarchy presents a particular portion of the application's user interface and is often optimized for a specific type of content. For example, an application can have distinct views specifically for presenting images, text and other types of content.
• event recognizer global methods 312 also include a hit view determination module 314 and/or an active event recognizer determination module 316.
• Hit view determination module 3144 provides software procedures for determining where an event or a sub-event has taken place within one or more views, when touch-sensitive display 156 displays more than one view. Views are made up of controls and other elements that a user can see on the display.
• FIG. 3D Another aspect of the user interface associated with a respective application (e.g., 133-1) is a set of views 317, sometimes herein called application views or user interface windows, in which information is displayed and touch-based gestures occur.
• the application views (of a respective application) in which a touch is detected may correspond to a particular view within a view hierarchy of the application. For example, the lowest level view in which a touch is detected may be called the hit view, and the set of events that are recognized as proper inputs may be determined based, at least in part, on the hit view of the initial touch that begins a touch-based gesture. Further discussion of view hierarchies is provided below with reference to Figure 3D.
• Hit view determination module 314 receives information related to events and/or sub-events. When an application has multiple views organized in a hierarchy, hit view determination module 314 identifies a hit view as the lowest view in the hierarchy which should handle the event or sub-event. In most circumstances, the hit view is the lowest level view in which an initiating event or sub-event occurs (i.e., the first event or sub-event in the sequence of events and/or sub-events that form a gesture). Once the hit view is identified by the hit view determination module, the hit view typically receives all events and/or sub- events related to the same touch or input source for which it was identified as the hit view.
• the hit view is not always a sole view that receives all events and/or sub-events related to the same touch or input source for which it was identified as the hit view.
• another application e.g., 133-2
• another view of the same application also receives at least a subset of the events and/or sub-events related to the same touch or input source, even though (or regardless of whether) a hit view has been determined for the touch or input source.
• Active event recognizer determination module 316 determines which view or views within a view hierarchy should receive a particular sequence of events and/or sub- events. In some application contexts, active event recognizer determination module 316 determines that only the hit view should receive a particular sequence of events and/or sub- events. In other application contexts, active event recognizer determination module 316 determines that all views that include the physical location of an event or sub-event are actively involved views, and therefore determines that all actively involved views should receive a particular sequence of events and/or sub-events.
• active event recognizer determination module 316 determines which application(s) in a programmatic hierarchy should receive a particular sequence of events and/or sub-events. Thus, in some embodiments, active event recognizer determination module 316 determines that only a respective application in the programmatic hierarchy should receive a particular sequence of events and/or sub-events. In some embodiments, active event recognizer determination module 316 determines that a plurality of applications in the programmatic hierarchy should receive a particular sequence of events and/or sub -events.
• touches are initially treated as being attached to a particular view, or set of views, but when a drag gesture is detected, the touch or touches are "detached" and events related to the touch or touches may be delivered to other views, or the gesture recognizers of other views, as the touch or touches traverse the user interface regions associated with those other views.
• Event dispatcher module 315 dispatches the event information to an event recognizer (also called herein "gesture recognizer") (e.g., event recognizer 325-1). In embodiments including active event recognizer determination module 316, event dispatcher module 315 delivers the event information to an event recognizer determined by active event recognizer determination module 316. In some embodiments, event dispatcher module 315 stores in an event queue the event information, which is retrieved by a respective event recognizer 325 (or event receiver 331) in a respective event recognizer 325).
• an event recognizer also called herein "gesture recognizer”
• event dispatcher module 315 delivers the event information to an event recognizer determined by active event recognizer determination module 316.
• event dispatcher module 315 stores in an event queue the event information, which is retrieved by a respective event recognizer 325 (or event receiver 331) in a respective event recognizer 325).
• a respective application (e.g., 133-1) includes application internal state 321, which indicates the current application view(s) displayed on touch-sensitive display 156 when the application is active or executing.
• application internal state 321 indicates the current application view(s) displayed on touch-sensitive display 156 when the application is active or executing.
• device/global internal state 134 ( Figure 1C) is used by event recognizer global methods 312 to determine which application(s) is(are) currently active
• application internal state 321 is used by event recognizer global methods 312 to determine application views 317 to which to deliver event information.
• application internal state 321 includes additional information, such as one or more of: resume information to be used when application 133-1 resumes execution, user interface state information that indicates information being displayed or that is ready for display by application 133-1, a state queue for enabling the user to go back to a prior state or view of application 133-1, and a redo/undo queue of previous actions taken by the user.
• application internal state 321 further includes contextual information/text and metadata 323.
• application 133-1 includes one or more application views 317, each of which has corresponding instructions for handling touch events that occur within a respective view of the application's user interface (e.g., a corresponding event handler 319, sometimes called gesture handlers).
• At least one application view 317 of application 133-1 includes one or more event recognizers 325.
• a respective application view 317 includes a plurality of event recognizers 325.
• one or more of event recognizers 325 are part of a separate module, such as a user interface kit (not shown) or a higher level object from which application 133-1 inherits methods and other properties.
• a respective application view 317 also includes one or more of: data updater, object updater, GUI updater, and/or event data received.
• a respective application also includes one or more event handlers 319.
• a respective application e.g., 133-1
• a respective event recognizer 325-1 receives event information from event dispatcher module 315 (directly or indirectly through application 133-1), and identifies an event from the event information.
• Event recognizer 325-1 includes event receiver 331 and event comparator 333.
• the event information includes information about an event (e.g., a touch) or a sub-event (e.g., a touch movement). Depending on the event or sub-event, the event information also includes additional information, such as location of the event or sub-event. When the event or sub-event concerns motion of a touch, the event information may also include speed and direction of the sub-event. In some embodiments, events include rotation of the device from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation (also called device attitude) of the device.
• an event e.g., a touch
• a sub-event e.g., a touch movement
• the event information also includes additional information, such as location of the event or sub-event.
• the event information may also include speed and direction of the sub-event.
• events include rotation of the device from one orientation to another (e.g., from a portrait orientation to
• Event comparator 333 compares the event information to one or more predefined gesture definitions (also called herein “event definitions”) and, based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event.
• event comparator 333 includes one or more gesture definitions 335 (as described above, also called herein “event definitions”).
• Gesture definitions 335 contain definitions of gestures (e.g., predefined sequences of events and/or sub-events), for example, gesture 1 (337-1), gesture 2 (337-2), and others.
• sub-events in gesture definitions 335 include, for example, touch begin, touch end, touch movement, touch cancellation, and multiple touching.
• the definition for gesture 1 (337-1) is a double tap on a displayed object.
• the double tap for example, comprises a first touch (touch begin) on the displayed object for a predetermined phase of the gesture, a first lift-off (touch end) for a next predetermined phase of the gesture, a second touch (touch begin) on the displayed object for a subsequent predetermined phase of the gesture, and a second lift-off (touch end) for a final predetermined phase of the gesture.
• the definition for gesture 2 (337-2) includes a dragging on a displayed object.
• the dragging for example, comprises a touch (or contact) on the displayed object, a movement of the touch across touch-sensitive display 156, and lift-off of the touch (touch end).
• event recognizer 325-1 also includes information for event delivery 339.
• Information for event delivery 339 includes references to corresponding event handlers 319.
• information for event delivery 339 includes action-target pair(s) 354.
• event information e.g., action message(s)
• targets 383 see Figure 3D
• the action-target pair(s) are activated.
• gesture definitions 335 include a definition of a gesture for a respective user-interface object.
• event comparator 333 performs a hit test to determine which user-interface object is associated with a sub-event. For example, in an application view in which three user-interface objects are displayed on touch- sensitive display 156, when a touch is detected on touch-sensitive display 156, event comparator 333 performs a hit test to determine which of the three user-interface objects, if any, is associated with the touch (event). If each displayed object is associated with a respective event handler 319, event comparator 333 uses the result of the hit test to determine which event handler 319 should be activated. For example, event comparator 333 selects an event handler 319 associated with the event and the object triggering the hit test.
• a respective gesture definition 337 for a respective gesture also includes delayed actions that delay delivery of the event information until after it has been determined whether the sequence of events and/or sub-events does or does not correspond to the event recognizer's event type.
• a respective event recognizer 325-1 determines that the series of events and/or sub-events do not match any of the events in gesture definitions 335, the respective event recognizer 325-1 enters an event failed state, after which the respective event recognizer 325-1 disregards subsequent events and/or sub-events of the touch-based gesture. In this situation, other event recognizers, if any, that remain active for the hit view continue to track and process events and/or sub-events of an ongoing touch-based gesture.
• the event information when no event recognizer for the hit view remains, the event information is sent to one or more event recognizers in a higher view in the view hierarchy. Alternatively, when no event recognizer for the hit view remains, the event information is disregarded. In some embodiments, when no event recognizer for views in the view hierarchy remains, the event information is sent to one or more event recognizers in a higher programmatic level in the programmatic hierarchy. Alternatively, when no event recognizer for views in the view hierarchy remains, the event information is disregarded.
• a respective event recognizer 325-1 includes event recognizer state 334.
• Event recognizer state 334 includes a state of the respective event recognizer 325-1.
• event recognizer state 334 includes recognizer metadata and properties 349.
• recognizer metadata and properties 349 include one or more of the following: A) configurable properties, flags, and/or lists that indicate how the event delivery system should perform event and/or sub-event delivery to actively involved event recognizers; B) configurable properties, flags, and/or lists that indicate how event recognizers interact with one another; C) configurable properties, flags, and/or lists that indicate how event recognizers receive event information; D) configurable properties, flags, and/or lists that indicate how event recognizers may recognize a gesture; E) configurable properties, flags, and/or lists that indicate whether events and/or sub-events are delivered to varying levels in the view hierarchy; and F) references to corresponding event handlers 319.
• event recognizer state 334 includes event/touch metadata 351.
• Event/touch metadata 351 includes event/touch information about a respective event/touch that has been detected and corresponds to a respective gesture definition 337 of gesture definitions 335.
• the event/touch information includes one or more of: a location, time stamp, speed, direction, distance, scale (or change in scale), and angle (or change in angle) of the respective event/touch.
• a respective event recognizer 325 activates event handler 319 associated with the respective event recognizer 325 when one or more particular events and/or sub-events of a gesture are recognized. In some embodiments, respective event recognizer 325 delivers event information associated with the event to event handler 319.
• Event handler 319 when activated, performs one or more of: creating and/or updating data, creating and updating objects, and preparing display information and sending it for display on display 126 or touch-sensitive display 156.
• a respective application view 317-2 includes view metadata 341.
• view metadata 341 includes data regarding a view.
• view metadata 341 includes one or more of: stop property 342, skip property 343, NoHit skip property 344, Interaction identifiers 347 (discussed below with respect to dynamically added drag, drop and springloading gesture recognizers), and other view metadata 329.
• a first actively involved view within the view hierarchy may be configured to prevent delivery of a respective sub-event to event recognizers associated with that first actively involved view.
• This behavior can implement the skip property 343.
• delivery of the respective sub-event is still performed for event recognizers associated with other actively involved views in the view hierarchy.
• a first actively involved view within the view hierarchy may be configured to prevent delivery of a respective sub-event to event recognizers associated with that first actively involved view unless the first actively involved view is the hit view. This behavior can implement the conditional NoHit skip property 344.
• a second actively involved view within the view hierarchy is configured to prevent delivery of the respective sub-event to event recognizers associated with the second actively involved view and to event recognizers associated with ancestors of the second actively involved view. This behavior can implement the stop property 342.
• Figure 3C is a block diagram illustrating exemplary classes and instances of gesture recognizers (e.g., event handling components 390) in accordance with some embodiments.
• gesture recognizers e.g., event handling components 390
• a software application (e.g., application 133-1) has one or more event recognizers 345.
• a respective event recognizer e.g., 345-2
• the respective event recognizer includes event recognizer specific code 338 (e.g., a set of instructions defining the operation of event recognizers) and state machine 340.
• application state 321 of a software application includes instances of event recognizers.
• Each instance of an event recognizer is an object having a state (e.g., event recognizer state 334). "Execution" of a respective event recognizer instance is implemented by executing corresponding event recognizer specific code (e.g., 338) and updating or maintaining the state 334 of the event recognizer instance 365.
• the state 334 of event recognizer instance 365 includes the state 351 of the event recognizer instance's state machine 340.
• application state 321 includes a plurality of event recognizer instances 365.
• a respective event recognizer instance 365 typically corresponds to an event recognizer that has been bound (also called “attached") to a view of the application.
• one or more event recognizer instances 365 are bound to a respective application in a programmatic hierarchy without reference to any particular view of the respective application.
• application state 321 includes a plurality of instances (e.g., 365-1 to 365-L) of a respective event recognizer (e.g., 345-2).
• application state 321 includes instances 365 of a plurality of event recognizers (e.g., 345-1 to 345-R).
• a respective instance (e.g., 365-2) of a gesture recognizer 345 includes event recognizer state 334. As discussed above, in some embodiments, in some
• event recognizer state 334 includes recognizer metadata and properties 349 and event/touch metadata 351. In some embodiments, event recognizer state 334 also includes view hierarchy reference(s) 336, indicating to which view the respective instance 365-2 of the gesture recognizer 345-2 is attached.
• recognizer metadata and properties 349 include the following, or a subset or superset thereof:
• this list 327 indicates the set of event recognizers (or gesture recognizers), if any, that must enter the event impossible or event canceled state before the respective event recognizer can recognize a respective event.
• the listed event or gesture recognizers must fail to recognize an input or event before the event recognizer with the wait-for list 327 is allowed to recognize the input or event. In effect, the listed event recognizers have higher priority for recognizing an event than the event recognizer with the wait-for list 327.
• one or more event recognizers may be adapted to delay delivering one or more sub-events of the sequence of sub-events until after the event recognizer recognizes the event.
• This behavior reflects a delayed event. For example, consider a single tap gesture in a view for which multiple tap gestures are possible. In that case, a tap event becomes a "tap + delay" recognizer. In essence, when an event recognizer implements this behavior, the event recognizer will delay event recognition until it is certain that the sequence of sub-events does in fact correspond to its event definition. This behavior may be appropriate when a recipient view is incapable of appropriately responding to cancelled events.
• an event recognizer will delay updating its event recognition status to its respective actively involved view until the event recognizer is certain that the sequence of sub-events does not correspond to its event definition.
• Delay touch began flag 328, delay touch end flag 330, and touch cancellation flag 332 are provided to tailor sub-event delivery techniques, as well as event recognizer and view status information updates to specific needs.
• recognizer metadata and properties 349 include the following, or a subset or superset thereof:
• state machine state/phase 351 which indicates the state of a state machine (e.g., 340) for the respective event recognizer instance (e.g., 365-2); state machine state/phase 351 can have various state values, such as “event possible", “event recognized”, “event failed”, and others, as described below; alternatively or additionally, state machine state/phase 351 can have various phase values, such as "touch phase began” which can indicate that the touch data structure defines a new touch that has not been referenced by previous touch data structures; a "touch phase moved” value can indicate that the touch being defined has moved from a prior position; a “touch phase stationary” value can indicate that the touch has stayed in the same position; a “touch phase ended” value can indicate that the touch has ended (e.g., the user has lifted his/her finger from the surface of a multi touch display); a "touch phase cancelled” value can indicate that the touch has been cancelled by the device; a cancelled touch can be a touch that is not necessarily ended by a user, but which the device
• action-target pair(s) 354 where each pair identifies a target to which the
• respective event recognizer instance sends the identified action message in response to recognizing an event or touch as a gesture or a part of a gesture
• delegate 353 which is a reference to a corresponding delegate when a delegate is assigned to the respective event recognizer instance; when a delegate is not assigned to the respective event recognizer instance, delegate 346 contains a null value;
• gesture recognizers including gesture recognizer states and properties, can be found in U.S. Patent Application 14/290,931, filed May 29, 2014, which is hereby incorporated by reference in its entirety.
• Each touch data structure 361 can comprise various entries.
• touch data structures may include data corresponding to at least the touch- specific entries in event/touch metadata 351 such as the following, or a subset or superset thereof:
• each touch data structure can define what is happening with a respective touch (or other input source) at a particular time (e.g., whether the touch is stationary, being moved, etc.) as well as other information associated with the touch (such as position).
• each touch data structure can define the state of a particular touch at a particular moment in time.
• One or more touch data structures referencing the same time can be added in a touch event data structure that can define the states of all touches a particular view is receiving at a moment in time (as noted above, some touch data structures may also reference touches that have ended and are no longer being received).
• Multiple touch event data structures can be sent to the software implementing a view as time passes, in order to provide the software with continuous information describing the touches that are happening in the view.
• the ability to handle complex touch-based gestures can add complexity to the various software applications.
• additional complexity can be necessary to implement advanced and desirable interface features.
• a game may require the ability to handle multiple simultaneous touches that occur in different views, as games often require the pressing of multiple buttons at the same time, or combining accelerometer data with touches on a touch-sensitive surface.
• some simpler applications and/or views need not require advanced interface features.
• a simple soft button i.e., a button that is displayed on a touch- sensitive display
• the underlying OS may send unnecessary or excessive touch data (e.g., multi-touch data) to a software component associated with a view that is intended to be operable by single touches only (e.g., a single touch or tap on a soft button).
• a software component may need to process this data, it may need to feature all the complexity of a software application that handles multiple touches, even though it is associated with a view for which only single touches are relevant. This can increase the cost of development of software for the device, because software components that have been traditionally easy to program in a mouse interface environment (i.e., various buttons, etc.) may be much more complex in a multi-touch environment.
• delegates can be used to control the behavior of event recognizers in accordance with some embodiments. As described below, delegates can determine, for example, whether a corresponding event recognizer (or gesture recognizer) can receive the event (e.g., touch) information; whether the corresponding event recognizer (or gesture recognizer) can transition from an initial state (e.g., event possible state) of a state machine to another state; and/or whether the corresponding event recognizer (or gesture recognizer) can
• Figure 3D is a block diagram illustrating the flow of event information in accordance with some embodiments.
• event dispatcher module 315 receives event information (e.g., for a touch event in view 378), and sends the event information to one or more applications (e.g., to application 133-1 in this example, but not to application 133-2).
• application 133-1 includes a plurality of views (e.g., 374, 376, and 378 corresponding to views 317, Figure 3B) in view hierarchy 372 and a plurality of gesture recognizers (379-1 through 379-3) in the plurality of views.
• Application 133-1 also includes one or more gesture handlers 319-i, which correspond to the targets 383 in target-action pairs 354 (e.g., targets 383-1 and 383-2).
• event dispatcher module 315 receives hit view information from hit view determination module 314 and sends event information to the hit view (e.g., 378) or event recognizer(s) attached to the hit view (e.g., 379-1 and 379-2). Additionally, or alternatively, event dispatcher module 315 receives hit level information from hit level determination module 352 and sends event information to applications in the hit level (e.g., 133-1 and 133- 2) or one or more event recognizers (e.g., 379-4) in the hit level applications.
• application 133-2 does not have a view that is the hit view for the received event information, and the event information is not sent to any of the gesture recognizers (e.g., gesture recognizer 379-4) of application 133-2.
• one of the applications receiving the event information is a default application (e.g., 133-2 may be a default application).
• only a subset of gesture recognizers in each receiving application is allowed to (or configured to) receive the event information.
• gesture recognizer 379-3 in application 133-1 does not receive the event information in this example.
• the gesture recognizers that receive the event information are called herein receiving gesture recognizers.
• receiving gesture recognizers 379-1 and 379-2 receive the event information, and compare the received event information with a respective gesture definition 337 ( Figure 3B) in the receiving gesture recognizers.
• gesture recognizers 379-1 and 379-4 have respective gesture definitions, and when those definitions match the received event information, the gesture recognizer with the matching gesture definition sends respective action messages
• gesture handlers e.g., 381
• gesture handlers e.g., 319-i
• the gesture handlers are part of an application-independent module that is available for use by a plurality of different applications as an application development framework, and the gesture handlers are included in the code of the application when the application is compiled using the application development framework (e.g., the gesture handlers are not part of an application core that is application-specific but rather are part of the application that is application-independent in the sense that it is available for use by multiple different applications).
• event information would be delivered to one or more gesture recognizers of application 133-2.
• UI user interfaces
• Figure 4A illustrates an example user interface for a menu of applications on portable multifunction device 100 in accordance with some embodiments. Similar user interfaces are, optionally, implemented on device 300.
• user interface 400 includes the following elements, or a subset or superset thereof:
• o Icon 418 for e-mail client module 140, labeled "Mail,” which optionally includes an indicator 410 of the number of unread e-mails
• o Icon 420 for browser module 147, labeled "Browser;”
• o Icon 422 for video and music player module 152, also referred to as iPod (trademark of Apple Inc.) module 152, labeled "iPod;” and Icons for other applications, such as:
• Icon 424 for FM module 141 labeled "Messages;"
• Icon 426 for calendar module 148 labeled "Calendar;"
• Icon 428 for image management module 144 labeled "Photos;"
• Icon 430 for camera module 143 labeled "Camera;"
• Icon 432 for online video module 155 labeled "Online Video;"
• Icon 436 for map module 154 labeled "Map;"
• Icon 442 for workout support module 142 labeled "Workout Support;"
• Icon 444 for notes module 153 labeled "Notes;"
• Icon 446 for a settings application or module, which provides access to settings for device 100 and its various applications 136.
• icon labels illustrated in Figure 4A are merely examples.
• icon 422 for video and music player module 152 is labeled "Music" or "Music Player.”
• Other labels are, optionally, used for various application icons.
• a label for a respective application icon includes a name of an application corresponding to the respective application icon.
• a label for a particular application icon is distinct from a name of an application corresponding to the particular application icon.
• Figure 4B illustrates an example user interface on a device (e.g., device 300, Figure 3) with a touch-sensitive surface 451 (e.g., a tablet or touchpad 355, Figure 3 A) that is separate from the display 450.
• Device 300 also, optionally, includes one or more contact intensity sensors (e.g., one or more of sensors 357) for detecting intensity of contacts on touch-sensitive surface 451 and/or one or more tactile output generators 359 for generating tactile outputs for a user of device 300.
• contact intensity sensors e.g., one or more of sensors 357
• tactile output generators 359 for generating tactile outputs for a user of device 300.
• Figure 4B illustrates an example user interface on a device (e.g., device 300, Figure 3) with a touch-sensitive surface 451 (e.g., a tablet or touchpad 355, Figure 3) that is separate from the display 450.
• a touch-sensitive surface 451 e.g., a tablet or touchpad 355, Figure 3
• the device detects inputs on a touch- sensitive surface that is separate from the display, as shown in FIG. 4B.
• the touch-sensitive surface (e.g., 451 in Figure 4B) has a primary axis (e.g., 452 in Figure 4B) that corresponds to a primary axis (e.g., 453 in Figure 4B) on the display (e.g., 450).
• the device detects contacts (e.g., 460 and 462 in Figure 4B) with the touch-sensitive surface 451 at locations that correspond to respective locations on the display (e.g., in Figure 4B, 460 corresponds to 468 and 462 corresponds to 470).
• user inputs e.g., contacts 460 and 462, and movements thereof
• the device on the touch-sensitive surface e.g., 451 in Figure 4B
• the device on the touch-sensitive surface e.g., 451 in Figure 4B
• the device on the touch-sensitive surface e.g., 451 in Figure 4B
• the device on the touch-sensitive surface e.g., 451 in Figure 4B
• the device on the touch-sensitive surface e.g., 451 in Figure 4B
• Similar methods are, optionally, used for other user interfaces described herein.
• finger inputs e.g., finger contacts, finger tap gestures, finger swipe gestures, etc.
• one or more of the finger inputs are replaced with input from another input device (e.g., a mouse based input or a stylus input).
• a swipe gesture is, optionally, replaced with a mouse click (e.g., instead of a contact) followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact).
• a tap gesture is, optionally, replaced with a mouse click while the cursor is located over the location of the tap gesture (e.g., instead of detection of the contact followed by ceasing to detect the contact).
• a tap gesture is, optionally, replaced with a mouse click while the cursor is located over the location of the tap gesture (e.g., instead of detection of the contact followed by ceasing to detect the contact).
• multiple user inputs it should be understood that multiple computer mice are, optionally, used simultaneously, or a mouse and finger contacts are, optionally, used simultaneously.
• the term "focus selector" is an input element that indicates a current part of a user interface with which a user is interacting.
• the cursor acts as a "focus selector,” so that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in
• a particular user interface element e.g., a button, window, slider or other user interface element
• the particular user interface element is adjusted in accordance with the detected input.
• a detected contact on the touch-screen acts as a "focus selector," so that when an input (e.g., a press input by the contact) is detected on the touch-screen display at a location of a particular user interface element (e.g., a button, window, slider or other user interface element), the particular user interface element is adjusted in accordance with the detected input.
• focus is moved from one region of a user interface to another region of the user interface without
• the focus selector moves in accordance with movement of focus between different regions of the user interface.
• the focus selector is generally the user interface element (or contact on a touch-screen display) that is controlled by the user so as to communicate the user's intended interaction with the user interface (e.g., by indicating, to the device, the element of the user interface with which the user is intending to interact).
• a focus selector e.g., a cursor, a contact, or a selection box
• a press input is detected on the touch-sensitive surface (e.g., a touchpad or touch screen) will indicate that the user is intending to activate the respective button (as opposed to other user interface elements shown on a display of the device).
• the term "intensity" of a contact on a touch-sensitive surface is the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a substitute (proxy) for the force or pressure of a contact on the touch-sensitive surface.
• the intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (e.g., at least 256). Intensity of a contact is, optionally, determined (or measured) using various approaches and various sensors or combinations of sensors.
• one or more force sensors underneath or adjacent to the touch-sensitive surface are, optionally, used to measure force at various points on the touch-sensitive surface.
• force measurements from multiple force sensors are combined (e.g., a weighted average or a sum) to determine an estimated force of a contact.
• a pressure-sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch-sensitive surface.
• the size of the contact area detected on the touch- sensitive surface and/or changes thereto, the capacitance of the touch-sensitive surface proximate to the contact and/or changes thereto, and/or the resistance of the touch-sensitive surface proximate to the contact and/or changes thereto are, optionally, used as a substitute for the force or pressure of the contact on the touch-sensitive surface.
• the substitute measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurements).
• the substitute measurements for contact force or pressure are converted to an estimated force or pressure and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure).
• intensity of a contact as an attribute of a user input allows for user access to additional device functionality that may otherwise not be readily accessible by the user on a reduced-size device with limited real estate for displaying affordances (e.g., on a touch-sensitive display) and/or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical/mechanical control such as a knob or a button).
• contact/motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" on an icon).
• at least a subset of the intensity thresholds is determined in accordance with software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of particular physical actuators and can be adjusted without changing the physical hardware of device 100).
• a mouse "click" threshold of a trackpad or touch-screen display can be set to any of a large range of predefined thresholds values without changing the trackpad or touch-screen display hardware.
• a user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and/or by adjusting a plurality of intensity thresholds at once with a system-level click "intensity" parameter).
• the term "characteristic intensity" of a contact is a characteristic of the contact based on one or more intensities of the contact.
• the characteristic intensity is based on multiple intensity samples.
• the characteristic intensity is, optionally, based on a predefined number of intensity samples, or a set of intensity samples collected during a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) relative to a predefined event (e.g., after detecting the contact, prior to detecting liftoff of the contact, before or after detecting a start of movement of the contact, prior to detecting an end of the contact, before or after detecting an increase in intensity of the contact, and/or before or after detecting a decrease in intensity of the contact).
• a predefined time period e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds
• characteristic intensity of a contact is, optionally based on one or more of: a maximum value of the intensities of the contact, a mean value of the intensities of the contact, an average value of the intensities of the contact, a top 10 percentile value of the intensities of the contact, a value at the half maximum of the intensities of the contact, a value at the 90 percent maximum of the intensities of the contact, or the like.
• the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is an average of the intensity of the contact over time).
• the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an operation has been performed by a user.
• the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold.
• a contact with a characteristic intensity that does not exceed the first threshold results in a first operation
• a contact with a characteristic intensity that exceeds the first intensity threshold and does not exceed the second intensity threshold results in a second operation
• a contact with a characteristic intensity that exceeds the second intensity threshold results in a third operation.
• a comparison between the characteristic intensity and one or more intensity thresholds is used to determine whether or not to perform one or more operations (e.g., whether to perform a respective option or forgo performing the respective operation) rather than being used to determine whether to perform a first operation or a second operation.
• a portion of a gesture is identified for purposes of determining a characteristic intensity.
• a touch-sensitive surface may receive a continuous swipe contact transitioning from a start location and reaching an end location (e.g., a drag gesture), at which point the intensity of the contact increases.
• the characteristic intensity of the contact at the end location may be based on only a portion of the continuous swipe contact, and not the entire swipe contact (e.g., only the portion of the swipe contact at the end location).
• a smoothing algorithm may be applied to the intensities of the swipe contact prior to determining the characteristic intensity of the contact.
• the smoothing algorithm optionally includes one or more of: an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm.
• these smoothing algorithms eliminate narrow spikes or dips in the intensities of the swipe contact for purposes of determining a characteristic intensity.
• the user interface figures described herein optionally include various intensity diagrams that show the current intensity of the contact on the touch-sensitive surface relative to one or more intensity thresholds (e.g., a contact detection intensity threshold ITo, a light press intensity threshold ITL, a deep press intensity threshold ITD (e.g., that is at least initially higher than II), and/or one or more other intensity thresholds (e.g., an intensity threshold IH that is lower than II)).
• intensity thresholds e.g., a contact detection intensity threshold ITo, a light press intensity threshold ITL, a deep press intensity threshold ITD (e.g., that is at least initially higher than II), and/or one or more other intensity thresholds (e.g., an intensity threshold IH that is lower than II)).
• This intensity diagram is typically not part of the displayed user interface, but is provided to aid in the interpretation of the figures.
• the light press intensity threshold corresponds to an intensity at which the device will perform operations typically associated with clicking a button of a physical mouse or a trackpad
• the deep press intensity threshold corresponds to an intensity at which the device will perform operations that are different from operations typically associated with clicking a button of a physical mouse or a trackpad.
• the device when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact-detection intensity threshold ITo below which the contact is no longer detected), the device will move a focus selector in accordance with movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold.
• these intensity thresholds are consistent between different sets of user interface figures.
• the response of the device to inputs detected by the device depends on criteria based on the contact intensity during the input. For example, for some "light press” inputs, the intensity of a contact exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the response of the device to inputs detected by the device depends on criteria that include both the contact intensity during the input and time-based criteria. For example, for some "deep press” inputs, the intensity of a contact exceeding a second intensity threshold during the input, greater than the first intensity threshold for a light press, triggers a second response only if a delay time has elapsed between meeting the first intensity threshold and meeting the second intensity threshold.
• This delay time is typically less than 200 ms in duration (e.g., 40, 100, or 120 ms, depending on the magnitude of the second intensity threshold, with the delay time increasing as the second intensity threshold increases).
• This delay time helps to avoid accidental deep press inputs.
• there is a reduced-sensitivity time period that occurs after the time at which the first intensity threshold is met. During the reduced-sensitivity time period, the second intensity threshold is increased. This temporary increase in the second intensity threshold also helps to avoid accidental deep press inputs. For other deep press inputs, the response to detection of a deep press input does not depend on time-based criteria.
• one or more of the input intensity thresholds and/or the corresponding outputs vary based on one or more factors, such as user settings, contact motion, input timing, application running, rate at which the intensity is applied, number of concurrent inputs, user history, environmental factors (e.g., ambient noise), focus selector position, and the like.
• factors such as user settings, contact motion, input timing, application running, rate at which the intensity is applied, number of concurrent inputs, user history, environmental factors (e.g., ambient noise), focus selector position, and the like.
• Example factors are described in U.S. Patent Application Serial Nos. 14/399,606 and 14/624,296, which are incorporated by reference herein in their entireties.
• FIG. 4C illustrates a dynamic intensity threshold 480 that changes over time based in part on the intensity of touch input 476 over time.
• Dynamic intensity threshold 480 is a sum of two components, first component 474 that decays over time after a predefined delay time pi from when touch input 476 is initially detected, and second component 478 that trails the intensity of touch input 476 over time.
• the initial high intensity threshold of first component 474 reduces accidental triggering of a "deep press” response, while still allowing an immediate “deep press” response if touch input 476 provides sufficient intensity.
• Second component 478 reduces unintentional triggering of a "deep press” response by gradual intensity fluctuations of in a touch input.
• touch input 476 satisfies dynamic intensity threshold 480 (e.g., at point 481 in Figure 4C)
• the "deep press" response is triggered.
• Figure 4D illustrates another dynamic intensity threshold 486 (e.g., intensity threshold ID).
• Figure 4D also illustrates two other intensity thresholds: a first intensity threshold IH and a second intensity threshold II.
• touch input 484 satisfies the first intensity threshold IH and the second intensity threshold II prior to time p2
• no response is provided until delay time p2 has elapsed at time 482.
• dynamic intensity threshold 486 decays over time, with the decay starting at time 488 after a predefined delay time pi has elapsed from time 482 (when the response associated with the second intensity threshold II was triggered).
• This type of dynamic intensity threshold reduces accidental triggering of a response associated with the dynamic intensity threshold ID immediately after, or concurrently with, triggering a response associated with a lower intensity threshold, such as the first intensity threshold IH or the second intensity threshold II.
• Figure 4E illustrate yet another dynamic intensity threshold 492 (e.g., intensity threshold ID).
• intensity threshold ID e.g., intensity threshold ID
• a response associated with the intensity threshold II is triggered after the delay time p2 has elapsed from when touch input 490 is initially detected.
• dynamic intensity threshold 492 decays after the predefined delay time pi has elapsed from when touch input 490 is initially detected. So a decrease in intensity of touch input 490 after triggering the response associated with the intensity threshold II, followed by an increase in the intensity of touch input 490, without releasing touch input 490, can trigger a response associated with the intensity threshold ID (e.g., at time 494) even when the intensity of touch input 490 is below another intensity threshold, for example, the intensity threshold II.
• An increase of characteristic intensity of the contact from an intensity below the light press intensity threshold ITL to an intensity between the light press intensity threshold ITL and the deep press intensity threshold ITD is sometimes referred to as a "light press” input.
• An increase of characteristic intensity of the contact from an intensity below the deep press intensity threshold ITD to an intensity above the deep press intensity threshold ITD is sometimes referred to as a "deep press” input.
• An increase of characteristic intensity of the contact from an intensity below the contact-detection intensity threshold ITo to an intensity between the contact-detection intensity threshold ITo and the light press intensity threshold ITL is sometimes referred to as detecting the contact on the touch-surface.
• a decrease of characteristic intensity of the contact from an intensity above the contact-detection intensity threshold ITo to an intensity below the contact-detection intensity threshold ITo is sometimes referred to as detecting liftoff of the contact from the touch- surface.
• ITo is zero. In some embodiments, ITo is greater than zero.
• a shaded circle or oval is used to represent intensity of a contact on the touch-sensitive surface. In some illustrations, a circle or oval without shading is used represent a respective contact on the touch-sensitive surface without specifying the intensity of the respective contact.
• one or more operations are performed in response to detecting a gesture that includes a respective press input or in response to detecting the respective press input performed with a respective contact (or a plurality of contacts), where the respective press input is detected based at least in part on detecting an increase in intensity of the contact (or plurality of contacts) above a press-input intensity threshold.
• the respective operation is performed in response to detecting the increase in intensity of the respective contact above the press-input intensity threshold (e.g., the respective operation is performed on a "down stroke" of the respective press input).
• the press input includes an increase in intensity of the respective contact above the press-input intensity threshold and a subsequent decrease in intensity of the contact below the press-input intensity threshold, and the respective operation is performed in response to detecting the subsequent decrease in intensity of the respective contact below the press-input threshold (e.g., the respective operation is performed on an "up stroke" of the respective press input).
• the device employs intensity hysteresis to avoid accidental inputs sometimes termed "jitter," where the device defines or selects a hysteresis intensity threshold with a predefined relationship to the press-input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press-input intensity threshold or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press-input intensity threshold).
• the hysteresis intensity threshold is X intensity units lower than the press-input intensity threshold or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press-input intensity threshold.
• the press input includes an increase in intensity of the respective contact above the press-input intensity threshold and a subsequent decrease in intensity of the contact below the hysteresis intensity threshold that corresponds to the press-input intensity threshold, and the respective operation is performed in response to detecting the subsequent decrease in intensity of the respective contact below the hysteresis intensity threshold (e.g., the respective operation is performed on an "up stroke" of the respective press input).
• the press input is detected only when the device detects an increase in intensity of the contact from an intensity at or below the hysteresis intensity threshold to an intensity at or above the press-input intensity threshold and, optionally, a subsequent decrease in intensity of the contact to an intensity at or below the hysteresis intensity, and the respective operation is performed in response to detecting the press input (e.g., the increase in intensity of the contact or the decrease in intensity of the contact, depending on the circumstances).
• the description of operations performed in response to a press input associated with a press-input intensity threshold or in response to a gesture including the press input are, optionally, triggered in response to detecting: an increase in intensity of a contact above the press-input intensity threshold, an increase in intensity of a contact from an intensity below the hysteresis intensity threshold to an intensity above the press-input intensity threshold, a decrease in intensity of the contact below the press-input intensity threshold, or a decrease in intensity of the contact below the hysteresis intensity threshold corresponding to the press-input intensity threshold.
• the operation is, optionally, performed in response to detecting a decrease in intensity of the contact below a hysteresis intensity threshold corresponding to, and lower than, the press-input intensity threshold.
• the triggering of these responses also depends on time-based criteria being met (e.g., a delay time has elapsed between a first intensity threshold being met and a second intensity threshold being met).
• UI user interfaces
• portable multifunction device 100 or device 300 with a display, a touch-sensitive surface, and (optionally) one or more sensors to detect intensities of contacts with the touch-sensitive surface.
• Figures 5A-5R illustrate example user interfaces for drag and drop operations in accordance with some embodiments.
• the user interfaces in these figures are used to illustrate the processes described below, including the processes described below with respect to Figures 7A-7F, 8A-8E, and 9A-9D.
• the focus selector is, optionally: a respective finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., a centroid of a respective contact or a point associated with a respective contact), or a centroid of two or more contacts detected on the touch-sensitive display system 112.
• analogous operations are, optionally, performed on a device with a display 450 and a separate touch- sensitive surface 451 in response to detecting the contacts on the touch-sensitive surface 451 while displaying the user interfaces shown in the figures on the display 450, along with a focus selector.
• touches or touch inputs, and movements of touches or touch inputs are under the control of a respective user, via placement and/or movement on, or in close proximity to, a touch screen (e.g., touch screen 112) or touchpad (e.g., touchpad 355, sometimes called a track pad) or other touch-sensitive surface of A) one or more fingers of a user of an electronic device, or B) a stylus of similar implement.
• a touch screen e.g., touch screen 112
• touchpad e.g., touchpad 355, sometimes called a track pad
• Figure 5 A illustrates the user interface of a portable multifunction device 100, displayed on a touch screen 112, with a user interface region 502 for a first application, Application A, and another user interface region 504 for a second application, Application B.
• the user interface region 502 for application A includes four objects, Object 1, Object 2, Object 3 and Object 4, and the user interface region 504 for application B includes a fifth object, Object 5.
• region 502 of Application A may be a view of Application A.
• the displayed representations of objects 1-4 in region 502 may also be views of Application A.
• region 504 of Application B may be a view of Application B
• the displayed representation of Object 5 in region 504 may also be a view of Application B.
• Application A or one or more of the views of Application A have interaction identifiers for drag operations, drop operations and springloading operations, indicating that supplemental gesture recognizers for drag gestures, drop gestures and springloading gestures can (or will be) added to one or more of the views of Application A when applicable supplemental-gesture-recognizer addition criteria are satisfied.
• interaction identifiers are stored or included in view metadata 341 ( Figure 3B) of each application view having such interaction identifiers.
• no supplemental gesture recognizers have been added to any of the views of Application A, nor have any gesture recognizer failure dependencies been established.
• Figure 5B illustrates a first touch (Touch 1) on Object 4 in the user interface shown in Figure 5 A.
• Touch 1 a first touch
• FIG. 5 A the appearance of Object 4 in user interface region 502 of Application A is updated.
• a focus selector 506 is now overlapping Object4, indicating the position of the first touch input.
• the focus selector associated with first touch moves (e.g., to position 505) without dragging the object, Object 4, on which it was initially placed.
• this outcome may occur when the first touch is moved laterally prior to remaining at an initial touch position for at least the minimum amount of time that a touch input must remain in contact with the touch-sensitive surface of the electronic device, without substantial lateral movement, before beginning lateral movement, in order to satisfy predefined drag start recognition criteria.
• Figure 5C illustrates the user interface generated when the first touch, Touchl, is recognized by a deep press (e.g., intensity based) or long press (e.g., time based) gesture recognizer.
• a deep press e.g., intensity based
• long press e.g., time based
• a set of menu options associated with Object 4 is displayed in response to recognition of the first touch as a long press gesture.
• T3 is at least the minimum amount of time that a touch input must remain in contact with the touch-sensitive surface of the electronic device, without substantial lateral movement, before beginning lateral movement, in order to satisfy predefined drag start recognition criteria.
• Touch 1 would have moved on the display (e.g., to position 505, shown in Figure 5B) without initiating a drag gesture and thus without dragging Object 4.
• an electronic device after initially recognizing a touch input, using a standard gesture recognizer, for example as a long touch, as shown in Figure 5C, an electronic device subsequently recognizes the touch input, using a supplemental gesture recognizer, as a second gesture, such as a drag start gesture, as shown in Figure 5D.
• a supplemental gesture recognizer as a second gesture, such as a drag start gesture, as shown in Figure 5D.
• T3 is a larger or longer time interval than T2 ( Figure 5C).
• a session ID (equal to 1 in this example) has been assigned to the first touch, or to the objects being dragged by the first touch.
• a session ID is assigned to a touch when the touch is recognized as a drag gesture, becomes a detached touch.
• the session ID corresponds to a set of supplemental gesture recognizers, added to Application A or one or more of its views, that govern the processing of the first touch, once it has been recognized as a drag gesture.
• Other drag gestures e.g., concurrent or subsequent drag gestures
• detached touches are depicted using a distinct graphic (an oval with a striped fill pattern) from regular touches (an oval with a cross- hatched fill pattern and a darker perimeter than detached touches), also called attached touches or view-attached touches.
• a detached touch is detached from the standard gesture recognizers of the application views or user interface regions it traverses, and as a result, touch events (sometimes herein called drag events) associated with placement and/or movement of the detached touch are processed by supplemental gesture recognizers associated with those views or user interface regions, but not standard gesture recognizers associated with those views or user interface regions.
• Supplemental gesture recognizers are dynamically added (e.g., by a system process) to an application or view when predefined criteria are satisfied. Touch events associated with placement and/or movement of regular touches (sometimes herein called attached touches or view-attached touches) are processed by the standard gesture recognizers associated with those views or user interface regions. Standard gesture recognizers are gesture recognizers are typically associated with or assigned to the application views or user interface regions of an application by the application.
• Figure 5E shows that Application B or one or more of the views of Application B have interaction identifiers for drop operations and springloading operations, indicating that supplemental gesture recognizers for drop gestures and springloading gestures can (or will be) added to one or more of the views of Application B when applicable supplemental-gesture-recognizer addition criteria are satisfied.
• one or more supplemental gesture recognizers for drop gestures have been added to one or more views of Application B, no gesture recognizer failure dependencies have been established, and a session ID has been assigned (or continues to be assigned) to the drag operation that includes the first touch.
• Other examples of supplemental gesture recognizers added to a view, and other examples of gesture recognizer failure dependencies are discussed below with respect to later figures in this sequence, such as Figure 5M.
• a two-finger touch input (e.g., two concurrently detected touches, both on the same object or view in an application) is detected on Object 4.
• a two-finger touch is used to initiate a drag operation without having to wait for a timeout period before beginning lateral movement (e.g., dragging) of the touch input.
• the interaction identifiers, added gesture recognizers and failure dependencies are the same as in Figure 5D. Recognition of the drag gesture is indicated in Figure 5G by the change in appearance of the dragged object (e.g., the perimeter is drawn with a thicker perimeter) and detachment of the two touches is indicated in Figure 5G by the changed appearance of the two touches.
• user interface region 504 has interaction identifiers for drag, drop and springloading interactions.
• Figure 51 represents a transition that occurs from the user interface shown in
• FIG 5D when a second touch, Touch2, is detected.
• the second touch is on an object, Object 2, in the user interface region 502 for Application A. Since the first touch has already been detached, the second touch is processed independently of the first touch, unless the second input is recognized as a drag add gesture. Thus, if the second touch is not recognized as a drag add gesture, the second touch is processed using the standard gesture recognizers for user interface region 502 of Application A.
• Object 1, Object 2 and Object 3 are elements of a vertically scrollable list in Application A.
• the scrollable list is scrolled downward by an amount corresponding to the amount of vertical movement of the second touch. As shown, the movement of the second touch need not be entirely vertical to cause scrolling.
• the scrolling of the scrollable list results in an additional element, Object Z, being scrolled into view, at the top of the displayed scrollable list.
• the second touch, Touch2 continues to be moved (e.g., by a user of the device 100).
• the movement of the second touch is both downward and horizontally over into the user interface region 504 of Application B. Since the second touch has not been recognized as a drag start gesture, it remains attached to Application A or one or move views of Application A, even as the second touch moves into user interface region 504, and as a result, touch events corresponding to the second touch continue to be delivered to Application A, and in response to those touch events, the scrollable list in user interface region 502 of Application A continues to be scrolled in accordance with the amount of vertical movement of the second touch. The continued scrolling of the scrollable list results in yet another additional element, Object Y, being scrolled into view at the top of the displayed scrollable list.
• a drag add gesture is a tap or other distinctive touch gesture performed on an object that is available for adding to the drag operation (e.g., an object that is within a predefined distance from an object already participating in a drag operation or an object that has been specified as available for adding to a drag gesture (e.g., by the application in which the object is displayed).
• Object 2 is added to the drag operation associated with the first touch, as shown in Figure 5L.
• time t 2 corresponds to the time the second touch, Touch2, is first detected, and T4 is or corresponds to the maximum duration of a drag add gesture.
• Object 2 is stacked behind Object 4, and both objects (Object 2 and Object 4) are dragged as the first touch moves laterally on touch screen 112.
• the drag add gesture is required to be a two-finger tap gesture, instead of the one finger tap gesture shown in Figures 51 and 5L, while in some other embodiments, both one finger tap gestures and two finger tap gestures are can be drag add gestures, so long as they satisfy predefined drag criteria (e.g., a requirement that the user input include a contact at a location that corresponds to a user interface object that can be dragged, and a requirement that a distance between the first touch, for which a drag operation has been recognized, and the second touch be no greater than a predefined distance or proximity threshold).
• predefined drag criteria e.g., a requirement that the user input include a contact at a location that corresponds to a user interface object that can be dragged, and a requirement that a distance between the first touch, for which a drag operation has been recognized, and the second
• FIG. 5M Object 4 and Object 2, shown only as a shadow as it is stacked under Object 4, have both been dragged into user interface region 504 of Application B.
• drop and springloading supplemental gesture recognizers are added to the user interface 504 of Application B in response to a detached touch, Touchl, being dragged into the user interface 504 of Application B, and further more a failure dependency is created or added, making the long press gesture recognizer dependent on failure of the springloading gesture recognizer in order for the long press gesture to be recognized.
• Figure 5N shows an update of the user interface shown in Figure 5K that occurs upon liftoff of the first touch, after Object 4 and Object 2 have been dragged into user interface region 504 of Application B.
• Figure 5N after liftoff of the first touch, Object 4 and Object 2 are unstacked and positioned in user interface region 504 of Application B.
• the supplemental gesture recognizers used to process the touch are automatically removed, and some (or all) gesture recognizer dependencies corresponding to the removed supplemental gesture recognizers are also removed.
• Figure 5N shows a set of interaction identifiers, but the supplemental gesture recognizers listed in Figure 5M are no longer listed in Figure 5N, and similarly, the dependency of the long press (LP) gesture recognizer on the (supplemental) springloading gesture recognizer shown in Figure 5M is no longer present in Figure 5N.
• LP long press
• Figure 50 represents a transition from the user interface shown in Figure 51, after the second touch, Touch2, is recognized as drag gesture.
• This is a second drag gesture that is independent of the first drag gesture by the first touch, and the corresponding touch, Touch2, is assigned a different session ID, for example session ID 2, then the first touch, which was assigned session ID 1 in this example.
• the second touch is detached upon being recognized as a touch gesture, freeing the standard gesture recognizers for user interface region 502 from having to process touch events produced by (or for) the second touch.
• Figure 5P represents a transition from the user interface shown in Figure 50, with the first touch, Touchl, and the second touch, Touch2, being moved in different directions, and thereby dragging Object 4 along with the movement of the first touch and dragging Object 2 along with the movement of the second touch.
• the first touch and second touch are independent detached touches, and each is used to drag a different object or set of objects than the other.
• Figure 5Q represents a transition from the user interface shown in Figure 5E, after the first touch, Touchl, moves downward toward Object 5 in user interface region 504. As a result of this movement, Object 4 is now positioned at least partially over Object 5.
• a supplemental springloading gesture recognizer has been added to the set of gesture recognizers associated with one or more views of Application B.
• supplemental springloading gesture recognizer is added to one or more views of Application B upon detection of a triggering event or condition, such as movement of a dragged object (e.g., Object4) over another object (e.g., Object5) that is capable of responding to a springloading gesture.
• a triggering event or condition such as movement of a dragged object (e.g., Object4) over another object (e.g., Object5) that is capable of responding to a springloading gesture.
• Figure 5R represents a transition from the user interface shown in Figure 5Q, after the placement of the first touch, Touchl, or the placement of the object (Object4) being dragged by the first touch, over Object 5 is recognized as a springloading gesture.
• the recognition of the springloading gesture is accomplished by one of the supplemental gesture recognizers added to user interface region 504 (e.g., a view of Application B) in accordance with a springloading interaction ID and the presence of a touch in interface region 504 that triggers the addition of one or more supplemental gesture recognizers.
• Object 5 which is a folder object (sometimes called a container or container object), is opened, while Object 4 continues to be dragged by the first touch, Touch 1, as shown in Figure 5R.
• the user could continue to move the first touch so as to drag Object 4 into the now-opened folder corresponding to Object 5, and then drop Object 4 into that folder by liftoff of the first touch from touch screen 1 12 after Object 4 has been dragged by the first touch into the interior of the opened folder corresponding to Object 5.
• Touch events for a touch are generated to represent the touch over time.
• Touches have phases selected from a predetermined set of phases: Touch Begin for a respective touch indicates the respective touch that has just been detected on the touch- sensitive surface, Touch Move for a respective touch indicates that the respective touch has moved on the touch-sensitive surface, Touch End for a respective touch indicates that the respective touch that has ceased to be detected on the touch-sensitive surface as part of ending a gesture, and Touch Cancel for a respective touch indicates that the respective touch has been determined to be an accidental touch or has been otherwise identified as a touch that should be ignored.
• touch events are processed by one or more gesture recognizers to identify gestures that are being performed by touches detected on the touch- sensitive surface (e.g., taps, swipes, drags, pinches, etc.).
• a touch cancel event i.e., a touch event for (or that includes) a touch having a phase equal to Touch Cancel
• applications performing an operation based on the respective touch cancel the operation and revert to a state before the touch was detected.
• a touch cancel operation is detected for a respective touch
• one or more gesture recognizers that are evaluating the touch as part of a gesture are cancelled (e.g., provided that cancellation of the touch would make it impossible for the gesture definition used by the gesture recognizer to be met).
• an application responds to a touch cancel event for a respective touch differently than it would respond to a touch end event.
• a tap input that ends with a touch cancel event (e.g., a touch event for a touch, corresponding to the tap input, with a touch cancel phase) is ignored, while a tap input that ends with a touch end event (e.g., a touch event for a touch, corresponding to the tap input, with a touch end phase) is processed as a tap.
• a touch cancel event e.g., a touch event for a touch, corresponding to the tap input, with a touch cancel phase
• a tap input that ends with a touch end event e.g., a touch event for a touch, corresponding to the tap input, with a touch end phase
• a touch event with a touch end phase is detected after the object has moved across the display the object remains where it was moved to or, optionally, continues to advance with some inertia
• a touch event with a touch cancel phase is detected after the object has moved across the display the movement of the object is reversed to the location where it was displayed prior to detecting the moving contact.
• Figure 6A illustrates touch events and drag events generated over a period of time while a touch (sometimes called a touch input) moves from a user interface region for a first application, Application A, to a user interface region for a second application,
• a touch sometimes called a touch input
• the detected touch corresponds to contact of a user's finger, or stylus or the like, on a touch-sensitive surface of an electronic device.
• An example of such a movement of a touch is shown in Figures 5B, 5D and 5E.
• a touch event is generated when a touch is initially detected, prior to recognizing the touch input as a drag gesture.
• the sequence of touch events represents the touch evolving through a sequence of phases, such as begin, moved and cancelled.
• the touch is represented in a touch event with a touch began phase; as movement of the touch is detected in Figure 5D the touch is represented in a touch event with a touch move phase; and then in Figure 5D as the touch is recognized as a drag gesture, the touch is represented in a touch event with a touch cancel phase.
• Instances of the touch event, with touch phase values corresponding to the phase or state of the touch are delivered to Application A for processing, for example to update a user interface of Application A.
• multiple instances of the touch event with a touch phase equal to "moved" may be generated while the touch input moves.
• the touch event is canceled and a drag event is generated.
• Canceling the touch input for Application A enables Application A to reverse any operations that were performed in response to detecting the touch (e.g., the menu in Figure 5C, which was displayed in response to detecting the touch on Object 4 for a threshold amount of time ceases to be displayed when Touch 1 is canceled in Figure 5D due to the recognition of Touch 1 as part of the drag gesture) and ignore the touch for the purpose of identifying gestures performed by attached touches.
• the drag event is a touch event that is marked or processed as a detached touch.
• a touch event delivered to an application includes a list of touches associated with Application A or views of Application A, and the list of touches indicates for each listed touch whether the listed touch is an attached touch (sometimes called a view-attached touch) or detached touch.
• Attached touches delivered to Application A are processed by both the standard gesture recognizers associated with Application A or views of Application A as well as any supplemental gesture recognizers associated with Application A or views of
• Application A e.g., gesture recognizers added to a respective view of Application A as a result of interaction identifiers associated with the respective view of Application A.
• detached touches delivered to Application A are processed by only the supplemental gesture recognizers associated with Application A or views of Application A, and are not processed by, and thus are ignored by the standard gesture recognizers associated with Application A or views of Application A.
• the touch associated with a drag event also has a sequence of phases, such as "enter,” “within” and “exit.” Instances of the drag event are generated over time as the touch moves, with the touch in a first instance of the drag event having a phase (sometimes called a touch phase) equal to "enter,” and the touch in subsequent instances of the drag event having a phase equal to "within” (e.g., corresponding to movement within a view or application view, sometimes called a user interface region), and the touch in a final instance of the drag event having a phase equal to "exit” that is generated either in response to liftoff of the touch, or the touch leaving a respective view.
• a phase sometimes called a touch phase
• within e.g., corresponding to movement within a view or application view, sometimes called a user interface region
• the touch event identifies the touch as being canceled and the touch is included in a corresponding drag event that identifies the touch as having a "enter" phase, all of which occur in response to recognition of the corresponding touch as a drag gesture. It is noted that the initial recognition of a touch as a drag gesture is sometimes called recognition of a drag start gesture.
• Figure 6B is similar to Figure 6A, but corresponds to movement of an attached touch from a starting location in the user interface region of Application A to an ending location in the user interface region of Application B. Since the touch is not detached, for example because it is not recognized as a drag gesture, only touch event instances are generated, and all are delivered to Application A, since attached touches are delivered to the application or application view associated with the initial touch location of the attached touch even if the touch moves outside of the application or application view associated with the initial touch location, in accordance with some embodiments.
• the phases of the touch in touch events generated in this example are "begin,” when the touch is initially detected, “moved” as movement of the touch is detected, and then “ended” when liftoff of the touch is detected.
• Figure 6C schematically shows processing of detected touches by a set of processes: a source process (Process A), corresponding to a first application, such as
• a system drag management process which in some embodiments executes contact/motion module 130 ( Figure 1 and Figure 3A) or a portion of contact/motion module; a system touch management process, which corresponds to hit view determination module 314 ( Figure 3B); and a destination process, corresponding to a second application, such as Application B.
• the system drag management process begins a session for the one or more touches, and sends to the source process a session ID that it assigned to the one or more touches.
• the system drag management process also passes along information about the one or more touches to the system touch management, which detaches the one or more touches, and sends a touch cancel event or instruction to the source process.
• the touch cancel event or instruction causes the standard gesture recognizers of the source process to transition to a terminal state and stop processing the one or more detached touches.
• system touch management passes information about the detached one or more touches to the destination process.
• Figure 6D shows an example of a gesture recognition hierarchy 600-A prior to any supplemental gesture recognizers being added to the set of gesture recognizers for an application, or one or more views of the application
• Figure 6E shows an example of a corresponding gesture recognition hierarchy 600-B after to one or more supplemental gesture recognizers have been added to the set of gesture recognizers for the application, or one or more views of the application.
• the standard gesture recognizers for the application, or application view which are the gesture recognizers for the application or application view prior to any supplemental gesture recognizers being added (e.g., gesture recognizers added to the view by the application itself to support various interactions with the application), include a long press gesture recognizer (LongPress GR), for recognizing a touch gesture based on the duration of touch without substantial lateral movement (e.g., for triggering display of a menu such as the menu displayed in Figure 5C when a long press gesture is detected on Object 4), and an intensity -based gesture recognizer (Intensity -Based GR) for recognizing a touch gesture based on the intensity of the touch gesture satisfying a predefined intensity threshold (e.g., for triggering display of an expanded view of content corresponding to Object 4 when a deep press gesture that satisfies the predefined intensity threshold is detected on Object 4).
• a long press gesture recognizer LongPress GR
• intensity -based gesture recognizer Intensity -Based GR
• the supplemental gesture recognizers added to the set of gesture recognizers for the application include a drag start gesture recognizer (DragStart GR), for recognizing a draft start gesture (e.g., for detecting a drag gesture as shown in Figure 5D where Object 4 is dragged in response to movement of Touch 1).
• DragStart GR drag start gesture recognizer
• gesture recognition hierarchy 600-A indicates that, prior to any supplemental gesture recognizers being added to the set of gesture recognizers for an application or application view, the long press gesture recognizer is failure dependent on the intensity -based gesture recognizer.
• the fact that the long press gesture recognizer is failure dependent on the intensity -based gesture recognizer means that, unless the intensity-based gesture recognizer fails to recognize an intensity -based a touch input (e.g., a deep press), the long press gesture recognizer is unable to recognize that touch input.
• the fact that the long press gesture recognizer is failure dependent on the intensity -based gesture recognizer means that, when the intensity -based gesture recognizer recognizes an intensity-based a touch input (e.g., a deep press), the long press gesture recognizer is unable to recognize that touch input.
• the long press gesture recognizer Upon failure of the intensity- based gesture recognizer to recognize a touch input, the long press gesture recognizer is enabled to allowed to try to recognize the touch input.
• this dependency is implemented by listing the intensity-based gesture recognizer in the wait-for list 127 ( Figure 3C) of the long press gesture recognizer.
• gesture recognition hierarchy 600-B indicates that, after supplemental gesture recognizers have been added to the set of gesture recognizers for the application, the long press gesture recognizer is failure dependent on both the intensity- based gesture recognizer and the drag start gesture recognizer, where the drag start gesture recognizer is a supplemental gesture recognizer added to the set of gesture recognizers for the application or application view. In addition, the drag start gesture recognizer is failure dependent on the intensity-based gesture recognizer.
• the long press gesture recognizer is unable to recognize a touch input as a long press gesture until the intensity- based gesture recognizer has failed to recognize the touch input (e.g., as a deep press) and the drag start gesture recognizer has failed to recognize the touch input (e.g., as a drag gesture).
• the drag start gesture recognizer is unable to recognize a touch input as a drag start gesture until the intensity -based gesture recognizer has failed to recognize the touch input (e.g., as a deep press).
• these failure dependencies are implemented by listing the intensity-based gesture recognizer and the drag start gesture recognizer in the wait-for list 127 ( Figure 3C) of the long press gesture recognizer, and by listing the intensity-based gesture recognizer in the wait-for list 127 ( Figure 3C) of the drag start gesture recognizer.
• FIGS 7A-7F are flow diagrams illustrating a method 700 of processing a touch detected by an electronic device, including determining if the touch corresponds to a drag gesture (sometimes called a drag and drop gesture), and processing the touch in accordance with that determination, in accordance with some embodiments.
• the method 700 is performed at an electronic device (e.g., device 300, Figure 3, or portable multifunction device 100, Figure 1) with a display, a touch-sensitive surface, and optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface.
• an electronic device e.g., device 300, Figure 3, or portable multifunction device 100, Figure 1
• a display e.g., a touch-sensitive surface
• optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface.
• the display is a touch-screen display and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display is separate from the touch- sensitive surface.
• Some operations in method 700 are, optionally, combined and/or the order of some operations is, optionally, changed. [00225] As described below, the method 700 provides an intuitive way to process a touch, detected by a device, that may or may not be a drag gesture. The method reduces the number, extent, and/or nature of the inputs from a user when the user intends to drag one or more objects from a first application view or region to another application view or region, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, enabling a user to perform drag and drop operations faster and more efficiently conserves power and increases the time between battery charges.
• the device displays (702), on its display, a user interface for a first application that includes a plurality of user interface objects. While displaying the user interface for the first application, the device receives (704), at the first application, information (e.g., one or more touch events) that describes a first touch detected at a location on the touch-sensitive surface that corresponds to a respective user interface object of the plurality of user interface objects in the first application.
• information e.g., one or more touch events
• the first touch may be Touch 1 shown in Figure 5B.
• the device In response (706) to receiving the information that describes the first touch, and in accordance with a determination (e.g., by the first application) that the first touch meets predefined criteria associated with initiating a respective type of interface operation (e.g., a drag and drop operation, also sometimes called a drag operation): the device initiates (708) a user interface operation (e.g., a first user interface operation) of the respective type involving the respective user interface object.
• a user interface operation e.g., a first user interface operation
• the predefined criteria associated with initiating a respective type of interface operation are criteria associated with initiating a drag operation, sometimes referred to as drag criteria.
• the drag criteria include a timing criterion (e.g., a first criterion) that is met when the user input remains on the touch- sensitive surface for at least a predefined time period, and a detected intensity of the user input remains below an intensity threshold during the predefined time period (e.g., long press criteria).
• the drag criteria include a lateral displacement criterion (e.g., a second criterion) that is met when a detected lateral displacement of the user input satisfies (e.g., meets, or exceeds) a lateral displacement threshold.
• the drag criteria require that the timing criterion be met before the lateral displacement criterion is met (e.g., for an input to satisfy the drag criteria, the input must first satisfy the timing criterion, and then satisfy the lateral displacement criterion).
• Method 700 further includes, while the first touch continues to be detected on the touch-sensitive surface and while the user interface operation involving the respective user interface object continues (710), receiving (712), at the first application, instructions to ignore the first touch with respect to user interface operations other than the respective type of interface operation (e.g., a drag, drop or springloading operation); and ignoring (714), by the first application, the first touch with respect to user interface operations other than the respective type of interface operation.
• the first touch is implemented by ceasing processing of the first touch, optionally in response to a touch cancel event.
• the first touch is treated as a cancelled touch even though the user interface operation involving the user interface object continues.
• the first application ceases to process (e.g., stops receiving information about) the first touch with respect to user interface operations other than the respective type of interface operation.
• the method while detecting the first touch and prior to determining that the first touch meets the predefined criteria associated with initiating the respective type of interface operation, the method includes processing the touch using one or more gesture recognizers associated with user interface operations other than the respective type of interface operation.
• initiating the user interface operation of the respective type involving the respective user interface object includes processing the touch using a gesture recognizer associated with the respective type of interface operation (e.g., a drag gesture recognizer that is used for processing drag inputs on the touch-sensitive surface).
• a gesture recognizer associated with the respective type of interface operation
• there are multiple instances of a gesture recognizer for example, the first touch is processed using an instance of the drag gesture recognizer.
• method 700 includes, prior to determining (e.g., by the system drag management process) that the first touch meets the predefined criteria associated with initiating the respective type of interface operation, processing (720) the first touch as a view-attached touch event, and in accordance with the determination that the first touch meets the predefined criteria associated with initiating the respective type of interface operation, processing (722) the first touch as a detached touch event (e.g., by initiating the user interface operation of the respective type).
• processing touch inputs as view- attached touch events provides consistency in the processing of touch inputs, and enabling the user to control views to which touch inputs are directed provides a more intuitive interface for controlling the user interface.
• Processing touch inputs as detached touch events after determining that the touch inputs meet predefined criteria associated with particular operations provides flexibility in processing touch inputs for performing the particular operations while enabling the device to continue receiving and processing additional touch inputs as normal touches to perform additional operations in the meantime.
• Providing more consistent and intuitive control options, while providing the flexibility to receive and process multiple touch inputs at a time reduces the number of inputs needed to perform an operation, enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user to provide inputs that more quickly achieve intended results and reducing user mistakes when operating/interacting with the device), which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.
• method 700 further includes, prior to determining (e.g., by the system drag management process) that the first touch meets the predefined criteria associated with initiating the respective type of interface operation, processing (724) the first touch using one or more gesture recognizers associated with the first application (e.g., "standard gesture recognizers," discussed above with reference to Figures 5D, 51, 50 and 6D- 6E) as a view-attached touch event; and, in accordance with the determination that the first user input satisfies the predefined criteria associated with initiating the respective type of interface operation (726): ceasing (728) to process the first touch using the one or more gesture recognizers associated with the first application as a view-attached touch event; and processing (730) the first touch as a detached touch event.
• one or more gesture recognizers associated with the first application e.g., "standard gesture recognizers," discussed above with reference to Figures 5D, 51, 50 and 6D- 6E
• detached touches are processed using a drag gesture recognizer.
• Processing view-attached touch inputs using gesture recognizers associated with an application, and ceasing to process touch inputs using those gesture recognizers after determining that the touch inputs meet predefined criteria associated with particular operations makes the gesture recognizers available to process additional touch inputs.
• Providing additional capability to process touch inputs enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user to provide inputs that more quickly achieve intended results and reducing user mistakes when operating/interacting with the device), which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.
• method 700 includes detecting (732) movement of the first touch outside of a region corresponding to the user interface for the first application. For example, as shown in Figures 5D and 5E, a first touch moves from a user interface region of Application A to a user interface region of Application B.
• method 700 also includes, in response to receiving (734) the information that describes the first touch, in accordance with a determination (e.g., by the first application) that the first touch did not meet the predefined criteria associated with initiating the respective type of interface operation (e.g., the first touch was not identified as part of a drag operation during the initial portion of the gesture), continuing (736) to provide (e.g., by a system process distinct from the first application) information that describes the first touch to the first application even after the first touch moves outside of the region corresponding to the user interface for the first application and into the region corresponding to the user interface for the second application.
• a system process distinct from the first application information that describes the first touch to the first application even after the first touch moves outside of the region corresponding to the user interface for the first application and into the region corresponding to the user interface for the second application.
• information regarding Touch2 would continue to be provided to Application A, even after Touch2 moves outside the region corresponding to the user interface for Application A and into the region corresponding to the user interface for Application
• the first touch is processed using a hit view determined based on an initial contact location of the first touch, independent of movement of the first touch (e.g., even if the first touch moves to a location on the touch-sensitive surface that corresponds to a location in the user interface that is outside of the hit view).
• method 700 further includes, in response to receiving
• the information that describes the first touch in accordance with the determination that the first touch meets (738) the predefined criteria associated with initiating the respective type of interface operation (e.g., the first touch is identified as part of a drag operation during the initial portion of the gesture): providing (740) (e.g., by a system process distinct from the first application) information that describes the first touch to the first application while the first touch is within the region corresponding to the user interface for the first application; and switching (742) to providing information that describes the first touch to the second application (e.g., instead of the first application) when the first touch moves outside of the region corresponding to the user interface for the first application into the region
• operations 740 and 742 correspond to the processing of detached touches.
• method 700 includes, while detecting movement of the first touch, and in accordance with the determination that the first touch meets (738) the predefined criteria associated with initiating the respective type of interface operation: determining (744) one or more respective regions corresponding to respective locations of the first touch, wherein the one or more respective regions are associated with one or more respective applications, and providing (746) information that describes the first touch to the respective application for a respective region while the first touch is within the respective region. For example, as the touch input moves along the touch- sensitive surface, the device performs repeated hit region testing to determine the respective hit views for respective locations of the touch input. In some embodiments, the respective hit views are determined at predefined time intervals (sampling intervals for the touch input).
• the predefined time intervals occur in sequence at a rate of 10 or more time intervals per second.
• the respective hit views correspond to different applications in a plurality of applications, such as Application A and Application B in Figures 5 A-5R.
• a first hit view corresponding to a first location of the first user input is in a first application (e.g., Application A, Figures 5A-5R)
• a second hit view corresponding to a second location of the first user input is in the second application (e.g., Application B, Figures 5A-5R) distinct from the first application.
• method 700 includes, in accordance with the determination that the first touch meets (738) the predefined criteria associated with initiating the respective type of interface operation, providing (748) the information that describes the first touch to the respective application for a respective region in conjunction with providing information that describes one or more additional touches that do not meet the predefined criteria associated with initiating the respective type of interface operation.
• information about detached touches e.g., the first touch
• information about normal, view-attached touches e.g., the one or more additional touches
• a same data structure e.g., a data structure having one or more lists of touches, which is received and processed (e.g., by the first application) in the same processing frame.
• method 700 further includes, while the user interface operation involving the respective user interface object continues (e.g., while a drag operation continues, for example as shown in Figures 51 and 5J), receiving (750) information (e.g., information regarding one or more touch events) that describes a change in a second touch on the touch-sensitive surface at a location that corresponds to the first application (e.g., Touch2 in Figures 51 and 5 J); and in response to detecting the change in the second touch, processing
• information e.g., information regarding one or more touch events
• the second touch may be detected prior to, concurrently with, or subsequent to detecting the first touch.
• the second touch is not affected by first touch so long as it does not satisfy special-case criteria, such as drag-add criteria.
• the first application receives information (e.g., one or more touch events) that describes a second touch (e.g., a second touch detected prior to, concurrently with, or subsequent to detecting the first touch) on the touch-sensitive surface at a location that corresponds to the first application; and in response to detecting the second touch, process the information that describes the second touch without regard to whether the first touch meets the predefined criteria associated with initiating the respective type of interface operation.
• the first application processes current and/or future touches while continuing to detect the first touch as a drag operation.
• method 700 includes receiving (754), at the first application, information that describes a second touch distinct from the first touch (e.g., the second touch is detected at a location on the touch-sensitive surface that corresponds to a second respective user interface object of the plurality of user interface objects in the first application, for instance as shown in Figures 51 and 5 J;
• method 700 in response (756) to receiving the information that describes the second touch, and in accordance with a determination (e.g., by the first application) that the second touch meets predefined add criteria (sometimes herein called drag add criteria) associated with the user interface operation (e.g., criteria for adding a user interface object to a drag operation), method 700 includes receiving (758), at the first application, instructions to ignore (e.g., cease processing of) the second touch with respect to user interface operations other than the respective type of interface operation (e.g., drag, drop, or springloading operations); and ignoring (760), by the first application, the second touch with respect to user interface operations other than the respective type of interface operation (e.g., treating the second touch as a cancelled touch with respect to user interface operations other than drag and drop operations, even though the user interface operation involving the user interface object, such as a drag operation, continues).
• method 700 includes processing the second touch in accordance with
• the predefined add criteria include (762) a criterion that is met when a distance between the first touch (e.g., corresponding to a drag operation) and the second touch (e.g., for adding an object to the drag operation) is less than a predefined proximity threshold.
• the predefined add criteria require, when liftoff of the second touch is detected, that the distance between the first touch and the second touch is less than the predefined proximity threshold.
• method 700 includes, while displaying the user interface for the first application, receiving (764), at the first application, information (e.g., one or more touch events) that describes a third touch (e.g., distinct from the first touch) detected at a location on the touch-sensitive surface that corresponds to a third respective user interface object of the plurality of user interface objects in the first application.
• method 700 further includes, in response to receiving the information that describes the third touch, and in accordance with a
• such embodiments of method 700 include, while the third touch continues to be detected on the touch-sensitive surface and while the second user interface operation involving the third respective user interface object continues (770): receiving (772), at the first application, instructions to ignore (e.g., cease processing of, optionally in response to a touch cancel event) the third touch with respect to user interface operations other than the respective type of interface operation (e.g., drag, drop or springloading); and ignoring (774), by the first application, the third touch with respect to user interface operations other than the respective type of interface operation.
• the third touch is treated as a cancelled touch even though the user interface operation involving the third user interface object continues.
• the third touch is detected while detecting the first touch.
• the second user interface operation (e.g., a second drag operation) is distinct from the first user interface operation (e.g., a first drag operation), as shown in Figures 50 and 5P, in which two separate drag operations are performed at the same time.
• method 700 includes, in response to receiving the information that describes the first touch, determining (776), by the first application, whether the first touch meets the predefined criteria associated with initiating the respective type of interface operation. For example, method 700 optionally includes determining whether gesture recognition criteria associated with a respective gesture recognizer are met. In some embodiments, either a system process or the first application determines whether a touch should be detached. For example, with reference to Figure 5B, 5C and 5D, a touch (e.g., Touchl, initially received in Figure 5B) can be determined to satisfy drag start criteria, in which case the touch is detached, as shown in Figure 5D, or not satisfy the drag start criteria, in which case the touch is processed as an attached touch, as shown in Figure 5C.
• a touch e.g., Touchl, initially received in Figure 5B
• drag start criteria in which case the touch is detached, as shown in Figure 5D
• the touch is processed as an attached touch, as shown in Figure 5C.
• the determination of whether to detach a touch is made by a system process, such as a system touch management process, which corresponds to hit view determination module 314 ( Figure 3B).
• the determination of whether to detach a touch is made by an application having a user interface region corresponding to the initial location of the touch.
• the instructions received at the first application to ignore the first touch with respect to user interface operations other than the respective type of interface operation include a touch cancel event (778).
• a touch cancel event 778
• method 700 includes sending instructions to ignore the first touch with respect to user interface operations other than the respective type of user interface operation to a plurality of applications, including the first application, that are receiving, or have received, information describing the first touch (e.g., any applications that are then-currently monitoring
• the plurality of applications includes a home screen application (sometimes called “SpringBoard”) that monitors touches even while the home screen application is not actively displayed.
• method 700 includes, in response to receiving the touch cancel event, forgoing (780) performance of user interface operations (e.g., use interface operations that involve the respective user interface object) other than the respective type of interface operation, and failing to recognize (e.g., ceasing to attempt to recognize or cancelling attempted recognition of) the first touch as a gesture other than one of a predefined set of gestures associated with the respective type of interface operation.
• user interface operations e.g., use interface operations that involve the respective user interface object
• failing to recognize e.g., ceasing to attempt to recognize or cancelling attempted recognition of
• method 700 optionally cancels, or reverses, user interface behaviors, such as object transformations and/or animations, associated with operations other than drag and drop operations, and optionally sets gesture recognizers for gestures other than drag-and-drop- related gestures to a canceled state (or other terminal state) instead of a "gesture possible" state.
• user interface behaviors such as object transformations and/or animations
• gesture recognizers for gestures other than drag-and-drop- related gestures to a canceled state (or other terminal state) instead of a "gesture possible" state.
• initiating (708) a user interface operation of the respective type involving the respective user interface object includes providing (782), to the first application, a unique session identifier identifying the user interface operation of the respective type and the respective user interface object.
• a session identifier has been assigned to the touch, Touchl, that is initiating a drag operation involving Object4, which is a user interface object in the user interface 502 for Application A.
• the session identifier (sometimes called a "session ID") identifies the drag operation and the content involved (being dragged) in the drag operation.
• each drag operation has a unique session identifier (e.g., concurrent but distinct drag operations can be distinguished based on their respective session identifiers). For example, as shown in Figures 50 and 5P, two separate drag operations are being performed, and each has been assigned a distinct session ID.
• the unique session identifier is associated (784) with additional information about the drag operation, including one or more of: a total number of user interface objects, including the respective user interface object, involved in the user interface operation of the respective type, and one or more content types of user interface objects involved in the user interface operation of the respective type.
• the unique session identifier includes the additional information or the unique session identifier can be used to retrieve the information from the system drag management process or the application from which the drag operation originated.
• the different drag operations have different identifiers, so that the drag operations can be tracked independently.
• the user interface operation of the respective type has (786) an associated operation phase value that describes a current phase of the user interface operation, and the phase value is selected from the group consisting of: exit, enter, and within. Phase values of the touches in drag events are discussed above with reference to Figure 6A.
• Figures 7A-7F have been described is merely an example and is not intended to indicate that the described order is the only order in which the operations could be performed.
• One of ordinary skill in the art would recognize various ways to reorder the operations described herein.
• details of other processes described herein with respect to other methods described herein e.g., methods 800 and 900 are also applicable in an analogous manner to method 700 described above with respect to Figures 7A-7F.
• the association of interaction identifiers with different views to determine which supplemental gesture recognizers to add to a respective view, described below with reference to method 800, and the use of gesture recognition failure dependencies, described below with reference to method 900 are also applicable in an analogous manner to method 700. For brevity, these details are not repeated here.
• Figures 8A-8E are flow diagrams illustrating a method 800 of processing a touch detected by an electronic device, including adding supplemental gesture recognizers to the view of an application so as to provide specialized processing for operations such as drag operations.
• Method 800 is performed at an electronic device (e.g., device 300, Figure 3, or portable multifunction device 100, Figure 1) with a display, a touch-sensitive surface, and optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface.
• the display is a touch-screen display and the touch- sensitive surface is on or integrated with the display.
• the display is separate from the touch-sensitive surface.
• the method 800 provides an intuitive way to process a touch, detected by a device, that may or may not be a drag gesture.
• the method reduces the computational load imposed on an electronic device by touch gestures and other user inputs, improves device responsiveness and thereby reduces latency. Reducing latency in
• responding to touch inputs also reduces mistaken inputs by users, such as duplicative or modified touch inputs made by the user while waiting for the device to respond to the user's earlier input(s). For example, such improvements reduce the number, extent, and/or nature of the inputs from a user when the user intends to drag one or more objects from a first application view or region to another application view or region, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, enabling a user to perform drag and drop operations faster and more efficiently conserves power and increases the time between battery charges.
• the device displays (802), on its display, a user interface including a plurality of views, including a first view that is associated with a first set of one or more standard gesture recognizers used to process touch inputs that are directed to the first view, and a second view that is associated with a second set of one or more standard gesture recognizers used to process touch inputs that are directed to the second view.
• One or more of the plurality of views e.g., the first view and/or second view
• a first interaction identifier e.g., a drag, drop, or springloading interaction identifier
• the plurality of views may include user interface region 502 for Application A and user interface region 504 for Application B.
• Many of Figures 5 A-5R indicate an example of the set of interaction identifiers associated with a view (e.g., interface region 502) of Application A or a view (e.g., interface region 504) of
• Figure 3B shows that the view metadata 341 for a respective application view 317-2 includes interaction identifiers 347. Furthermore, the view metadata 341 for each view of a plurality of views 317 has its own set of interaction identifiers.
• interaction identifiers associated with a particular view are flag values indicating which interaction identifiers of a predefined set of interaction identifiers are associated with (e.g., assigned to) that particular view.
• the interaction identifiers associated with a particular view are identifier values (e.g., alphanumeric or multi-bit values), and the set of interaction identifiers stored for that particular view indicate which interaction identifiers are associated with (e.g., assigned to) that particular view.
• identifier values e.g., alphanumeric or multi-bit values
• the set of interaction identifiers stored for that particular view indicate which interaction identifiers are associated with (e.g., assigned to) that particular view.
• possible interactions include a drag interaction (e.g., an object from a view may be picked up and dragged across the user interface), a drop interaction (e.g., an object being dragged across the user interface may be dropped onto a view), and a springloading interaction (e.g., while dragging an object across the user interface, hovering over the icon for a view may cause that view to open; in another example, hovering over an application icon on the home screen may cause the application itself to open, if the application icon view supports the springloading interaction; in another example, hovering over a folder icon may cause the folder to open; and in yet another example, hovering over an object may cause a control of the object, or a control associated with the object, to be activating or deactivated (e.g., by toggling a state of the control when predefined springloading criteria have been satisfied).
• a drag interaction e.g., an object from a view may be picked up and dragged across the user interface
• a drop interaction e.
• method 800 further includes detecting (804), via the touch-sensitive surface, a touch input at a location on the touch- sensitive surface that corresponds to a respective view of the plurality of views; and in response to detecting, via the touch-sensitive surface, the touch input (806): and (while continuing to detect the touch input) in accordance with a determination (808) that the touch input meets supplemental-gesture-recognizer addition criteria, wherein the supplemental- gesture-recognizer addition criteria include a criterion that is met when the respective view has the first interaction identifier: adding (810) the first set of one or more supplemental gesture recognizers associated with the first interaction identifier to the respective view; and processing (812) the touch input with the one or more supplemental gesture recognizers and the one or more standard gesture recognizers associated with the respective view.
• the one or more supplemental gesture recognizers are attached to or associated with the respective view on the fly by the system, based on a determination of whether an object in the view supports a particular interaction, as determined by a drag, drop, or springloading interaction identifier for the respective view.
• Method 800 further includes, in response to detecting the touch input (806), in accordance with a determination (814) that the touch input does not meet the supplemental- gesture-recognizer addition criteria, processing the touch input at the respective view (e.g., with the one or more standard gesture recognizers associated with the respective view or with one or more supplemental gesture recognizers associated with other interaction identifiers) without adding the first set of one or more supplemental gesture recognizers associated with the first interaction identifier to the respective view.
• processing the touch input at the respective view e.g., with the one or more standard gesture recognizers associated with the respective view or with one or more supplemental gesture recognizers associated with other interaction identifiers
• method 800 includes detecting a sequence of touch inputs, including a first touch input that meets the supplemental-gesture-recognizer addition criteria, and including a second touch input, received after the first touch input, that does not meet the supplemental-gesture-recognizer addition criteria.
• the first touch input is processed in accordance with the aforementioned adding (a first set of one or more supplemental gesture recognizers) (810) and processing (with the first set of one or more supplemental gesture recognizers) (812) operations, while the second input is processed in accordance with the aforementioned processing (814) (without adding the first set of one or more
• a view receives a touch event, and a determination is made as to whether the view supports the interaction associated with the touch event (e.g., a drag interaction, a drop interaction, or a springloading interaction).
• a respective interaction identifier for a view specifies whether the view supports a respective interaction. For example, a view receives a touch event.
• a drag gesture recognizer is initiated and added to the view (e.g., in conjunction with the view receiving the touch event).
• a view receives a drag enter event. If the view has a drop interaction identifier, then a drop gesture recognizer is initiated and added to the view.
• a springloading interaction identifier if the view has a springloading interaction identifier, then a springloading gesture recognizer is initiated and added to the view.
• a single interaction identifier may be used to specify multiple interactions supported by the view (e.g., a single interaction identifier may specify that a view supports drop interactions and springloading interactions).
• the supplemental-gesture-recognizer addition criteria include (820) a criterion that is met when the touch input is a predetermined type of touch input (e.g., a touch begin for a drag interaction, or a detached touch enter for a springloading interaction or a drop interaction).
• a predetermined type of touch input e.g., a touch begin for a drag interaction, or a detached touch enter for a springloading interaction or a drop interaction.
• the predetermined type of touch input is a detached touch entering the respective view (822).
• Figures 5D and 5E show a touch input, Touchl, entering a view, interface region 504, of Application B
• Figure 6A shows the corresponding event is a drag event with the touch (or touch input) in the drag event having a phase value of "enter”
• Figure 6C shows that the touch input entering the view of Application B results in the system touch management process detaching the touch, and thus when the touch input, Touchl, enters the view of Application B, the view receives a detached touch.
• method 800 further includes detecting (824) movement of the touch input outside of the respective view, and in response (826) to detecting the movement of the touch input outside of the respective view, and in accordance with a determination that the touch input is a detached touch exiting the respective view: removing (828) the first set of one or more supplemental gesture recognizers associated with the first interaction identifier from the respective view.
• the removing of the one or more supplemental gesture recognizers is performed when the touch input meets supplemental- gesture-recognizer removal criteria, where the supplemental-gesture recognizer removal criteria include a criterion that is met when the touch input is a predetermined type of touch input, such as a detached touch, exiting the respective view.
• Removing the one or more supplemental gesture recognizers when supplemental-gesture-recognizer removal criteria are satisfied makes the electronic device more efficient, by avoiding processing touch inputs with gesture recognizers no longer needed to process such inputs, and thereby reduces latency in responding to touch inputs.
• reducing latency in responding to touch inputs reduces mistaken inputs by users, such as duplicative or modified touch inputs made by the user while waiting for the device to respond to the user's earlier input(s).
• the touch input corresponds to a drag interaction (830).
• a drag interaction is an interaction in which an object from a view is picked up and dragged across the user interface(s) for one or more applications.
• the first interaction identifier is a drag interaction identifier (832).
• the first set of one or more supplemental gesture recognizers includes (834) a drag start gesture recognizer that includes, in order for a drag start gesture to be recognized: a requirement that a location of the touch input correspond to the respective view for at least a predefined period of time; and a requirement that, subsequent to the location of the touch input corresponding to the respective view for at least the predefined period of time, a change in location of the touch input satisfy a lateral displacement threshold (e.g., while continuing to detect the touch input).
• method 800 further includes, in response to recognition of a drag start gesture, the device (e.g., the system drag management process or the application) performing (836) a drag start action in the user interface that indicates that the drag operation has started.
• the device e.g., the system drag management process or the application
• performing (836) a drag start action in the user interface that indicates that the drag operation has started.
• performance of the drag start action (836) may include moving the user interface object in accordance with movement of the contact, and/or animating a change in appearance of the user interface object, for example to give the appearance that the user interface object had been lifted in a z-direction from the displayed user interface.
• the drag start gesture recognizer includes one or more of the following requirements in order for a touch input to satisfy predefined drag start criteria: [00269] 1) a timing criterion that is met when the touch input is maintained on the touch-sensitive surface for at least a predefined time period;
• displacement criterion (e.g., for an input to satisfy the drag criteria, the input must first satisfy the timing criterion, and then satisfy the displacement criterion);
• the drag gesture recognizer may include other combinations of the aforementioned requirements (e.g., a single contact with a timing requirement and a lateral displacement requirement, or two contacts with a lateral displacement requirement but without a timing requirement, or various other such combinations).
• the first interaction identifier is a drag interaction identifier (832).
• the first set of one or more supplemental gesture recognizers includes (838) a drag add gesture recognizer that includes a requirement that the touch input include one or more contacts (e.g., a single contact, or two distinct contacts detected concurrently) whose locations correspond to the respective view in order for a drag add gesture to be recognized; and in response to the recognition of the drag add gesture, the device performs (840) a drag add start action in the user interface that indicates that the drag add operation has started.
• the drag add gesture recognizer includes a requirement that the touch input include two concurrent contacts and a requirement that the two concurrent contacts cease to be detected within a predefined period of time (e.g., a two-finger tap gesture). In some embodiments, the drag add gesture recognizer includes a requirement that the location of the touch input correspond to the respective view for at least the predefined period of time.
• the drag add gesture recognizer includes a requirement that the touch input include one or two concurrent contacts, cease to be detected within a predefined period of time (e.g., a one or two-finger tap gesture), and a requirement that the one or two concurrent contacts occur within a predefined distance of a prior touch input that has been recognized as a drag gesture, or within a predefined distance of an object being dragged by the prior touch input in a drag operation.
• a predefined period of time e.g., a one or two-finger tap gesture
• the drag start action includes animating a user interface object (e.g., corresponding to the touch input) as though the user interface object had been lifted in a z- direction from the displayed user interface.
• a user interface object e.g., corresponding to the touch input
• the touch input corresponds to a drop interaction (850).
• a drop interaction occurs when an object, which has been dragged across the user interface(s) of one or more applications using a touch input recognized as a drag gesture, is dropped onto a view.
• An example of a drop interaction is discussed above with reference to Figures 5Q and 5R.
• the first interaction identifier is a drop interaction identifier (852)
• the first set of one or more supplemental gesture recognizers includes (854) a drop gesture recognizer that includes a requirement that a location of the touch input when the touch input ceases to be detected corresponds to the respective view.
• method 800 includes, in response to the recognition (856) of the drop gesture, the device performing a drop action in the user interface that indicates that the drop operation has started.
• the drop action includes animating (858) a user interface object (e.g., an object that was being dragged by the touch input) as though the user interface object had been lowered in a z-direction from above the displayed user interface into the displayed user interface.
• a user interface object e.g., an object that was being dragged by the touch input
• the touch input corresponds to a springloading interaction (860).
• a springloading interaction occurs, after dragging an object across the user interface, hovering the dragged object over an icon for a view, which causes that view to open.
• hovering the dragged object over an application icon on the home screen causes the application itself to open, if the application icon view supports the springloading interaction.
• hovering the dragged object over a folder icon causes the folder to open and for a new view of the opened folder to be displayed.
• hovering the dragged object over an object causes a control of the object, or a control associated with the object, to be activating or deactivated (e.g., by toggling a state of the control when predefined springloading criteria have been satisfied).
• the first interaction identifier is a springloading interaction identifier (862)
• the first set of one or more supplemental gesture recognizers includes (864) a springloading gesture recognizer that includes a requirement that a location of the touch input corresponds to the respective view for at least a predefined time period in order for a springloading gesture to be recognized.
• method 800 further includes, in response to the recognition of the springloading gesture (866), the application performing (868) a springloading action that includes one or more of: activating a control (e.g., a control corresponding to the view) as though the control had been selected (e.g., tapped or clicked), and opening a container in which content can be stored (e.g., a folder or album).
• a control e.g., a control corresponding to the view
• FIG. 5R show an example of a springloading action (sometimes called a springloading operation), in which a first object, Object4, is moved over a container object (also called a folder), Object5, by a touch input, Touchl, that is dragging the first object.
• the springloading gesture recognizer further includes a requirement that a velocity of the touch input across the touch-sensitive surface is below a predefined velocity threshold (870).
• a velocity of the touch input across the touch-sensitive surface is below a predefined velocity threshold (870).
• the touch input (Touchl) dragging the object, Object4 has a velocity at or close to zero, essentially hovering over Object5 while the user waits for Object5 to open in response to the springloading gesture.
• method 800 includes, in accordance with a determination that a velocity of the touch input across the touch-sensitive surface satisfies the predefined velocity threshold, changing (872) an appearance of the respective view.
• changing the appearance of the respective view includes displaying an indication (e.g., an animation of the object over which the touch input is hovering, or a change in color or thickness of an outer boundary or perimeter of the object, or a change in background color or blur or other displayed characteristic of the object) that the respective view supports the springloading interaction prior to recognizing the springloading gesture, or prior to performing the springloading operation.
• the respective view is displayed with a first transformation from its immediately prior visual appearance.
• the set of one or more standard gesture recognizers associated with the respective view are associated (872) with the respective view by an application that includes the respective view, and the first set of one or more supplemental gesture recognizers are added to the respective view by a system process (e.g., the system drag management process).
• a system process e.g., the system drag management process
• the application specifies the standard gesture recognizers associated with the respective view
• a system process specifies the supplemental gesture recognizers to be added to the respective view in accordance with the activity identifiers associated with the respective view and the satisfaction of respective triggering conditions associated with those activity identifiers, or alternatively, the satisfaction of triggering conditions associated with the set of the supplemental gesture recognizers corresponding to each of the activity identifiers for the respective view.
• method 800 includes detecting (884), via the touch sensitive surface, a second touch input at a second location on the touch- sensitive surface that corresponds to the respective view; and in response to detecting (886), via the touch-sensitive surface, the second touch input: in accordance with a determination that the second touch input meets (888) the supplemental-gesture-recognizer addition criteria, wherein the supplemental-gesture-recognizer addition criteria include a criterion that is met when the respective view has the first interaction identifier: adding (890) an instance of the first set of one or more supplemental gesture recognizers associated with the first interaction identifier to the respective view; and processing (892) the second touch input with the instance of the one or more supplemental gesture recognizers and the one or more standard gesture recognizers associated with the respective view.
• the same set of operations performed on the first touch input when it meets the supplemental-gesture- recognizer addition criteria are performed with respect to a second touch input when it meets the supplemental-gesture-recognizer addition criteria.
• separate session identifiers are associated with or assigned to each of the two touch inputs, for example as discussed above with respect to Figures 50 and 5P.
• Figures 8A-8E have been described is merely an example and is not intended to indicate that the described order is the only order in which the operations could be performed.
• One of ordinary skill in the art would recognize various ways to reorder the operations described herein.
• details of other processes described herein with respect to other methods described herein e.g., methods 700 and 900 are also applicable in an analogous manner to method 800 described above with respect to Figures 8A-8E.
• the processing of touch inputs that meet predefined criteria as detached touches, using automatically added supplemental gesture recognizers to recognize drag, drop and springloading gestures, described above with reference to method 700, and the use of gesture recognition failure dependencies, described below with reference to method 900 are also applicable in an analogous manner to method 800. For brevity, these details are not repeated here.
• Figures 9A-9D are flow diagrams illustrating a method 900 of processing a touch detected by an electronic device, including establishing failure requirements between one or more supplemental gesture recognizers which provide specialized processing for operations such as drag operations, and one or more standard gesture recognizers, which are established by an application for handling touch inputs in one or more views of the application.
• Method 900 is performed at an electronic device (e.g., device 300, Figure 3, or portable multifunction device 100, Figure 1) with a display, a touch-sensitive surface, and optionally one or more sensors to detect intensity of contacts with the touch-sensitive surface.
• the display is a touch-screen display and the touch- sensitive surface is on or integrated with the display.
• the display is separate from the touch-sensitive surface.
• the method 900 provides an intuitive way to process a touch, detected by a device, that may or may not be a drag gesture.
• the method reduces the computational load imposed on an electronic device by touch gestures and other user inputs, improves device responsiveness and thereby reduces latency and mistaken inputs by users. For example, such improvements reduce the number, extent, and/or nature of the inputs from a user when the user intends to drag one or more objects from a first application view or region to another application view or region, thereby creating a more efficient human- machine interface.
• the device displays (902), on its display, a user interface of an application, and while displaying (904) the user interface of the application, detecting a user input (e.g., a touch input) that corresponds to a portion of the user interface of the application that is associated with a plurality of gesture recognizers, wherein: a first set of one or more standard gesture recognizers of the plurality of gesture recognizers were associated with (e.g., assigned to) the portion of the user interface by the application; a second set of one or more
• supplemental gesture recognizers in the plurality of gesture recognizers were associated with (e.g., assigned to) the portion of the user interface by a system process (e.g., an application independent process); and a first failure requirement involving a first standard gesture recognizer (e.g., in the first set of one or more standard gesture recognizers) and a first supplemental gesture recognizer (e.g., in the second set of one or more supplemental gesture recognizers) were associated with (e.g., assigned to) the portion of the user interface by the system process (e.g., the application independent process).
• a system process e.g., an application independent process
• Method 900 further includes, in response to detecting the user input, processing (906) the user input in accordance with the first standard gesture recognizer, the first supplemental gesture recognizer, and the first failure requirement.
• failure requirements between gesture recognizers can be represented by a gesture recognition hierarchy, such as hierarchy 600-A or 600-B.
• the first set of one or more (e.g., application-specified) standard gesture recognizers and the second set of one or more (e.g., system-process-specified) gesture recognizers are arranged in a gesture recognition hierarchy (908), and the first failure requirement includes a requirement that a first respective gesture recognizer fail in order for a second respective gesture recognizer to successfully recognize a gesture, wherein the first respective gesture recognizer is at a higher level in the gesture recognition hierarchy than the second respective gesture recognizer.
• the first respective gesture recognizer e.g., the intensity- based gesture recognizer
• the second respective gesture recognizer e.g., the long press gesture recognizer
• the second respective gesture recognizer is a child of the first respective gesture recognizer (the intensity -based gesture recognizer) in the gesture recognition hierarchy 600-B.
• one or more gesture recognizers can be inserted into a gesture recognition hierarchy of an existing set of gesture recognizers (e.g., application-specified gesture recognizers). For example, the transition from the gesture recognition hierarchy 600- A of Figure 6D to the gesture recognition hierarchy 60-B of Figure 6E is caused by the insertion of the drag start gesture recognizer into the gesture recognition hierarchy 600-A of Figure 6D.
• a gesture recognizer can be added to the gesture recognition hierarchy at a level higher than an existing gesture recognizer in the hierarchy (e.g., the drag start gesture recognizer is inserted at a level higher than the long press gesture recognizer), such that processing a respective user input using the existing gesture recognizer requires that the added gesture recognizer fail to recognize the respective user input (e.g., in the scenario shown in Figure 6E, the drag start gesture recognizer must fail in order for the long press gesture recognizer to recognize the respective user input).
• a gesture recognizer can be added to (e.g., inserted into) a gesture recognition hierarchy at a level between two existing gesture recognizers. For example, with respect to a gesture recognition hierarchy that does not include a drag start gesture recognizer (e.g., before adding a drag start gesture recognizer to the gesture recognition hierarchy), processing a long press gesture requires determining that an intensity- based gesture recognizer fail to recognize the gesture. In some embodiments, as shown in
• a drag start gesture recognizer is added to the hierarchy between the intensity- based gesture recognizer and the long press gesture recognizer (e.g., the long press gesture recognizer is lower than the drag start gesture recognizer, which in turn is lower than the intensity-based gesture recognizer).
• processing a long press gesture requires determining that the intensity -based gesture recognizer, and then the drag start gesture recognizer, both fail to recognize the user input (e.g., the user input corresponds to neither an intensity -based gesture nor a drag start gesture).
• a gesture recognizer can be added to a gesture recognition hierarchy at a level lower than an existing gesture recognizer in the hierarchy, such that processing a respective user input using the added gesture recognizer requires that the existing gesture recognizer fail to recognize the respective user input (e.g., processing a drag start gesture requires the intensity-based gesture recognizer to fail to recognize the user input).
• the first failure requirement includes
• the first supplemental gesture recognizer fails to recognize a user input (e.g., a touch input) in order for the first standard gesture recognizer to successfully recognize a gesture.
• a user input e.g., a touch input
• the drag start gesture recognizer must fail for the long press gesture recognizer to succeed.
• the drag add gesture recognizer must fail in order for a tap gesture recognizer to succeed.
• the first supplemental gesture recognizer is a drag add gesture recognizer
• the first standard gesture recognizer is a tap gesture recognizer. See, for example, Figure 6E.
• the drag add gesture recognizer includes
• a requirement that the user input include a contact at a location that corresponds to a respective user interface element in the user interface, and liftoff of the contact within a predefined period of time e.g., T4 in Figure 5L.
• a user input recognized as a drag add gesture includes the same elements as a tap gesture.
• the drag add gesture recognizer further includes a requirement that the user input include a second contact (e.g., a contact corresponding to a drag operation).
• the drag add gesture recognizer further includes a requirement that a distance between the contact and the second contact be less than a predefined proximity threshold.
• the respective user interface element is added, using a "tap” (e.g., a contact with liftoff), to an ongoing drag operation (e.g., the second contact).
• the respective user interface element is added to a "nearby" ongoing drag operation (e.g., within the proximity threshold).
• the sequence of Figures 51, 5L shows a drag add gesture being performed by a second touch, Touch2, on Object2, which results in Object2 being added to the drag operation corresponding to a first touch, Touchl .
• time t 2 corresponds to the time the second touch is first detected
• T4 corresponds to the aforementioned predefined time period, which is (or corresponds to) the maximum duration of the drag add gesture.
• processing (916) the user input in accordance with the first standard gesture recognizer, the first supplemental gesture recognizer, and the first failure requirement includes: in accordance with a determination (e.g., by the system process) that content associated with the respective user interface element (e.g., content of, or represented by, the respective user interface element) cannot be added to a drag operation, failing (918) to recognize the user input using the drag add gesture recognizer, and processing the user input using the tap gesture recognizer; and in accordance with a determination that the content associated with the respective user interface element can be added to the drag operation, recognizing (920) a drag add gesture and adding the content to the drag operation.
• a determination e.g., by the system process
• content associated with the respective user interface element e.g., content of, or represented by, the respective user interface element
• the system process queries the application to determine whether the application supports the addition of content (e.g., content associated with the respective user interface element) to drag operations (e.g., whether the application or application view that includes the respective user interface element supports drag interactions).
• content e.g., content associated with the respective user interface element
• drag operations e.g., whether the application or application view that includes the respective user interface element supports drag interactions.
• the first supplemental gesture recognizer is a drag start gesture recognizer
• the first standard gesture recognizer is a long press gesture recognizer (922).
• Such embodiments are represented by, or at least consistent with the gesture recognizer hierarchy discussed above with reference to Figure 6E.
• processing the user input in accordance with the first standard gesture recognizer, the first supplemental gesture recognizer, and the first failure requirement is performed in accordance with a determination that the device satisfies compact display criteria. For example, if the device (e.g., a smart phone) has a display size below a predefined threshold, the drag start gesture recognizer must fail before the long press gesture recognizer can recognize a gesture.
• the device e.g., a smart phone
• the first failure requirement includes (928( a requirement that successful recognition of the user input as a first gesture by the first standard gesture recognizer (e.g., long press) be canceled (e.g., subsequently ignored) in accordance with successful recognition of the user input as a second gesture by the first supplemental gesture recognizer (e.g., drag start).
• processing see 906 the user input in accordance with the first standard gesture recognizer, the first supplemental gesture recognizer, and the first failure requirement includes:
• recognizing (932), using the standard gesture recognizer, the user input as the first gesture includes, as shown in Figure 5C, performing an operation in the user interface associated with the first gesture, such as displaying a menu of options in response to recognizing the long press gesture.
• ceasing (938) to recognize the user input as the first gesture using the standard gesture recognizer optionally, but typically, includes ceasing to perform the operation associated with the first gesture.
• ceasing to perform the operation associated with the first gesture includes ceasing to display the menu of options displayed (see Figure 5C) in response to recognizing the long press gesture (e.g., the ceasing is shown in the transition from Figure 5C to Figure 5D).
• recognizing (936), using the supplemental gesture recognizer, the user input as the second gesture includes performing an operation in the user interface associated with the second gesture, such as displaying an animation of a user interface object being lifted in a z-direction from the user interface to indicate the start of a drag session, in response to recognizing a drag start gesture.
• the first failure requirement includes a requirement (940) that the first standard gesture recognizer fail in order for the first supplemental gesture recognizer to successfully recognize a gesture (e.g., an intensity-based gesture recognizer must fail for drag start gesture recognizer to succeed, as shown by the gesture recognizer hierarchy 600-B of Figure 6E).
• a requirement (940) that the first standard gesture recognizer fail in order for the first supplemental gesture recognizer to successfully recognize a gesture e.g., an intensity-based gesture recognizer must fail for drag start gesture recognizer to succeed, as shown by the gesture recognizer hierarchy 600-B of Figure 6E).
• the first standard gesture recognizer is (942) an intensity-based gesture recognizer (e.g., a gesture recognizer that requires a touch input to meet intensity criteria, such as a requirement that a characteristic intensity of the contact increase above an intensity threshold, in order for the gesture recognizer to successfully recognize the gesture), and the first supplemental gesture recognizer is a drag start gesture recognizer. See discussion of Figure 6E above, which graphically depicts a corresponding gesture recognizer hierarchy 600-B.
• an intensity-based gesture recognizer e.g., a gesture recognizer that requires a touch input to meet intensity criteria, such as a requirement that a characteristic intensity of the contact increase above an intensity threshold, in order for the gesture recognizer to successfully recognize the gesture
• the first supplemental gesture recognizer is a drag start gesture recognizer. See discussion of Figure 6E above, which graphically depicts a corresponding gesture recognizer hierarchy 600-B.
• the first supplemental gesture recognizer (944) is a drag start gesture recognizer
• the drag start gesture recognizer includes, in order for a drag start gesture to be recognized: a requirement that the user input include a contact at a location that corresponds to a respective user interface element in the user interface for at least a predefined period of time; and a requirement that, subsequent to the location of the contact corresponding to the respective user interface element for at least the predefined period of time, a change in location of the contact satisfy a lateral
• the touch input recognized as a drag start gesture initially remains at one location, typically on or over an object (e.g., Touchl on Object4, Figure 5D), for at least a predefined period of time before moving and satisfying a lateral displacement threshold (e.g., by laterally moving at least a certain number of pixels).
• an object e.g., Touchl on Object4, Figure 5D
• the first supplemental gesture recognizer is a drag start gesture recognizer (946), and the drag start gesture recognizer includes, in order for a drag start gesture to be recognized: a requirement that the user input include two concurrent contacts (e.g., two distinct contacts detected concurrently, as shown in Figure 5F), on the touch-sensitive surface, at locations that correspond to a respective user interface element in the user interface (e.g., Object4 in user interface region 502, Figure 5F); and a requirement that changes in locations of the two concurrent contacts satisfy a lateral displacement threshold.
• two concurrent contacts e.g., two distinct contacts detected concurrently, as shown in Figure 5F
• a respective user interface element in the user interface e.g., Object4 in user interface region 502, Figure 5F
• a requirement that changes in locations of the two concurrent contacts satisfy a lateral displacement threshold.
• the drag start gesture recognizer does not include a requirement that the user input be detected for at least the predefined period of time.
• the use of a two-finger input enables the user to start the lateral displacement phase of a drag gesture as soon as the two-finger input makes contact with the touch-sensitive surface.
• the drag start gesture recognizer includes a requirement that the two concurrent contacts be maintained within a second predefined proximity threshold from each other (e.g., the two concurrent contacts move substantially together rather than away from each other).
• the first supplemental gesture recognizer is a drag add gesture recognizer (948), and the drag add gesture recognizer includes, in order for a drag add gesture to be recognized, a requirement that the user input include two concurrent contacts, at locations that correspond to a respective user interface element in the user interface, and liftoff of the two concurrent contacts within a predefined period of time (e.g., the user input is a two-finger tap gesture, instead of the one finger tap gesture discussed above with reference to Figures 51 and 5L).
• the drag add gesture recognizer further includes a requirement that the user input include a third contact (e.g., corresponding to a drag operation, started before the drag add gesture).
• the drag add gesture recognizer further includes a requirement that a distance from the third contact to a closest of the two contacts be less than a predefined proximity threshold.
• the respective user interface element is added, using the two-finger drag add gesture (e.g., the two concurrent contacts), to an ongoing drag operation (e.g., the third contact).
• an ongoing drag operation e.g., the third contact.
• the respective user interface element is added to a "nearby" ongoing drag operation (e.g., within the proximity threshold).
• Figures 9A-9D have been described is merely an example and is not intended to indicate that the described order is the only order in which the operations could be performed.
• One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., methods 800 and 900) are also applicable in an analogous manner to method 900 described above with respect to Figures 9A-9D.
• association of interaction identifiers with different views to determine which supplemental gesture recognizers to add to a respective view, described above with reference to method 800, and the processing of touch inputs that meet predefined criteria as detached touches, using automatically added supplemental gesture recognizers to recognize drag, drop and springloading gestures, described above with reference to method 700, are also applicable in an analogous manner to method 900. For brevity, these details are not repeated here.

Applications Claiming Priority (4)

Publications (1)

Family

ID=64274873

Family Applications (1)

Country Status (3)

Cited By (1)

Families Citing this family (3)

Family Cites Families (22)

• 2018
  • 2018-05-16 EP EP18730536.2A patent/EP3593235A1/de active Pending
  • 2018-05-16 CN CN201880031229.8A patent/CN110651242B/zh active Active
  • 2018-05-16 CN CN201911262546.3A patent/CN111026306B/zh active Active
  • 2018-05-16 WO PCT/US2018/032917 patent/WO2018213414A1/en unknown

Also Published As

Similar Documents

Legal Events