KR102140295B1 - User interface camera effects - Google Patents

Abstract

This disclosure relates generally to user interfaces. In some examples, the electronic device provides a transition between simulated lighting effects. In some examples, the electronic device applies a simulated lighting effect to the image. In some examples, the electronic device provides user interfaces for applying the filter to the image. In some examples, the electronic device provides a reduced filter interface. In some examples, the electronic device provides visual assistance displayed in the viewfinder.

Description

User interface camera effects

This application claims priority to U.S. Patent Application No. 15/728,147, filed October 9, 2017 and entitled "USER INTERFACE CAMERA EFFECTS," and the U.S. Patent Application, September 9, 2017 U.S. Provisional Patent Application No. 62/556,414 filed on the date and entitled "USER INTERFACE CAMERA EFFECTS", and U.S. Provisional Patent Application filed on June 4, 2017 and entitled "USER INTERFACE CAMERA EFFECTS" Priority is claimed on No. 62/514,947. This application is also filed on July 10, 2017 and the invention is entitled "USER INTERFACE CAMERA EFFECTS" in Danish Patent Application No. PA201770563 and filed on September 22, 2017 and the invention is entitled "USER INTERFACE CAMERA EFFECTS" Claims priority to Danish patent application No. PA201770719. The contents of these applications are incorporated herein by reference in their entirety for all purposes.

Technology field

The present disclosure relates generally to a computer user interface of an electronic device, in particular a device having an embedded camera.

The use of electronic devices to record videos and take photos has increased significantly in recent years. Exemplary electronic devices for recording video and taking pictures include smart phones and handheld cameras. Such devices often include a viewfinder that the user can use to preview before taking a picture or recording a video.

However, some techniques for managing camera effects using an electronic device are generally cumbersome and inefficient. For example, modifying the visual effects in the viewfinder such that the captured image and recorded video exhibit visual effects is often inaccurate and requires extensive user input. Existing techniques require more time than necessary, wasting user time and device energy. This latter consideration is particularly important for battery-operated devices.

Accordingly, the present technology provides electronic devices with faster and more efficient methods and interfaces for managing camera effects. These methods and interfaces, optionally, complement or replace other methods for managing camera effects. Such methods and interfaces reduce the cognitive burden on the user and create a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces save power and increase the time between battery charges. In some examples, the techniques provide simulated visual effects and captured images in camera viewfinders without requiring additional hardware components. In some examples, techniques provide the ability to quickly switch between user interfaces with limited user input. In some examples, the techniques efficiently provide enhanced image editing capabilities that result in visually satisfactory results for the displayed digital viewfinder and for captured videos. In some examples, techniques efficiently provide user interfaces for switching between different sets of options with minimal input. In some examples, the techniques efficiently provide user interfaces that provide additional functionality without any direct input. These techniques reduce the number of user inputs required and conserve battery power.

An exemplary method, in an electronic device having one or more cameras, one or more input devices, and a display, simultaneously displaying a camera application user interface on a display, wherein the camera application user interface is configured to A digital viewfinder including live preview; And a representation of a filter picker user interface overlaid on the digital viewfinder -; While simultaneously displaying a representation of the digital viewfinder and filter picker user interface, detecting the first input starting at a location corresponding to a respective portion of the live preview via one or more input devices; And in response to detecting the first input, upon determining that the first criteria are met-the first criteria are such that the filter picker user interface overlays on each portion of the live preview when the first input is detected. Included-applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And performing a respective action in the camera application without applying the preview of the first filter to the live preview according to the determination that the filter picker user interface is not overlaid on each part of the live preview when the first input is detected. Steps.

An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more cameras, one or more input devices, and one or more processors of an electronic device having a display. One or more programs include instructions for simultaneously displaying a camera application user interface on a display, the camera application user interface comprising: a digital viewfinder including a live preview of the field of view of one or more cameras; And a representation of a filter picker user interface overlaid on the digital viewfinder -; Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, upon determining that the first criteria are met-the first criteria are such that the filter picker user interface overlays on each portion of the live preview when the first input is detected. Included-instructions for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And performing a respective action in the camera application without applying the preview of the first filter to the live preview according to the determination that the filter picker user interface is not overlaid on each part of the live preview when the first input is detected. Contains commands for

An exemplary temporary computer-readable storage medium stores one or more programs configured to be executed by one or more cameras, one or more input devices, and one or more processors of an electronic device having a display. One or more programs include instructions for simultaneously displaying a camera application user interface on a display, the camera application user interface comprising: a digital viewfinder including a live preview of the field of view of one or more cameras; And a representation of a filter picker user interface overlaid on the digital viewfinder -; Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, upon determining that the first criteria are met-the first criteria are such that the filter picker user interface overlays on each portion of the live preview when the first input is detected. Included-instructions for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And performing a respective action in the camera application without applying the preview of the first filter to the live preview according to the determination that the filter picker user interface is not overlaid on each part of the live preview when the first input is detected. Contains commands for

Exemplary electronic devices include memory that stores one or more cameras, one or more input devices, a display, one or more processors, and one or more programs configured to be executed by one or more processors. One or more programs include instructions for simultaneously displaying a camera application user interface on a display, the camera application user interface comprising: a digital viewfinder including a live preview of the field of view of one or more cameras; And a representation of a filter picker user interface overlaid on the digital viewfinder -; Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, upon determining that the first criteria are met-the first criteria are such that the filter picker user interface overlays on each portion of the live preview when the first input is detected. Included-instructions for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And performing a respective action in the camera application without applying the preview of the first filter to the live preview according to the determination that the filter picker user interface is not overlaid on each part of the live preview when the first input is detected. Contains commands for

An exemplary electronic device includes one or more cameras; One or more input devices; display; Means for simultaneously displaying a camera application user interface on a display, wherein the camera application user interface comprises: a digital viewfinder including a live preview of the field of view of one or more cameras; And a representation of a filter picker user interface overlaid on the digital viewfinder -; Means for detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, upon determining that the first criteria are met-the first criteria are such that the filter picker user interface overlays on each portion of the live preview when the first input is detected. Including-means for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And performing a respective action in the camera application without applying the preview of the first filter to the live preview according to the determination that the filter picker user interface is not overlaid on each part of the live preview when the first input is detected. Means for.

An example method includes displaying, on an electronic device having one or more input devices and a display, on a display, a representation of image data associated with depth map information; While displaying a representation of image data on a display, detecting the first input through one or more input devices; In response to detecting the first input, applying a first lighting effect to the representation of the image data, the first lighting effect being based on depth map information; Detecting a second input through one or more input devices; And upon detecting the second input, applying a second lighting effect different from the first lighting effect to the representation of the image data, the second lighting effect being based on depth map information.

An exemplary non-transitory computer-readable storage medium stores one or more input devices, and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input through one or more input devices while displaying a representation of image data on a display; Instructions for applying a first lighting effect to the representation of the image data, upon detecting the first input, the first lighting effect being based on depth map information; Instructions for detecting a second input through one or more input devices; And upon detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, the second lighting effect being based on depth map information.

An exemplary temporary computer-readable storage medium stores one or more input devices, and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input through one or more input devices while displaying a representation of image data on a display; Instructions for applying a first lighting effect to the representation of the image data, upon detecting the first input, the first lighting effect being based on depth map information; Instructions for detecting a second input through one or more input devices; And upon detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, the second lighting effect being based on depth map information.

An exemplary electronic device includes one or more input devices; display; One or more processors; And a memory storing one or more programs configured to be executed by one or more processors, the one or more programs comprising: instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input through one or more input devices while displaying a representation of image data on a display; Instructions for applying a first lighting effect to the representation of the image data, upon detecting the first input, the first lighting effect being based on depth map information; Instructions for detecting a second input through one or more input devices; And upon detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, the second lighting effect being based on depth map information.

An exemplary electronic device includes one or more cameras; One or more input devices; display; Means for displaying, on the display, a representation of the image data associated with the depth map information; Means for detecting a first input via one or more input devices while displaying a representation of the image data on the display; Means for applying a first lighting effect to the representation of the image data, upon detecting the first input, the first lighting effect being based on depth map information; Means for detecting a second input through one or more input devices; And means for applying a second lighting effect different from the first lighting effect to the representation of the image data according to detecting the second input, the second lighting effect being based on depth map information.

The exemplary method detects a first input corresponding to a selection of a first image filter of a representation of image data having a first appearance, via one or more input devices, and one or more input devices, in an electronic device having a display. To do; And in response to detecting the first input, applying a first image filter to the representation of the image data, the applying step comprising: upon the determination that the image data has depth information associated therewith-depth information Enables the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the foreground of the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data Applying a first image filter to the region, wherein the first level adjustment represents a first degree by which the first image filter changes the appearance of the representation of the image data; Applying a first image filter to the background area of the representation of the image data using a second level adjustment to change the appearance of the background area of the representation of the image data-the second level adjustment allows the first image filter to Indicates a second degree of changing the appearance of the expression, and the first level adjustment and the second level adjustment are different -; And after applying the first image filter to the representation of the image data, on the display, displaying the representation of each image to which the first filter is applied to the representation of the image data.

An exemplary non-transitory computer-readable storage medium stores one or more input devices, and one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for detecting, via one or more input devices, a first input corresponding to a selection of a first image filter of a representation of image data having a first appearance; And in response to detecting the first input, instructions for applying the first image filter to the representation of the image data, the instructions for applying, upon determining that the image data has depth information associated therewith- Depth information enables the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data Commands for applying the first image filter to the foreground region of the first level adjustment indicates a first degree that the first image filter changes the appearance of the representation of the image data; Instructions for applying the first image filter to the background area of the representation of the image data using the second level adjustment to change the appearance of the background area of the representation of the image data-the second level adjustment is performed by the first image filter Indicates a second degree of changing the appearance of the representation of the data, and the first level adjustment and the second level adjustment are different -; And after applying the first image filter to the representation of the image data, on the display, instructions for displaying the representation of each image to which the first filter is applied to the representation of the image data.

An exemplary temporary computer-readable storage medium stores one or more input devices, and one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for detecting, via one or more input devices, a first input corresponding to a selection of a first image filter of a representation of image data having a first appearance; And in response to detecting the first input, instructions for applying the first image filter to the representation of the image data, the instructions for applying, upon determining that the image data has depth information associated therewith- Depth information enables the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data Commands for applying the first image filter to the foreground region of the first level adjustment indicates a first degree that the first image filter changes the appearance of the representation of the image data; Instructions for applying the first image filter to the background area of the representation of the image data using the second level adjustment to change the appearance of the background area of the representation of the image data-the second level adjustment is performed by the first image filter Indicates a second degree of changing the appearance of the representation of the data, and the first level adjustment and the second level adjustment are different -; And after applying the first image filter to the representation of the image data, on the display, instructions for displaying the representation of each image to which the first filter is applied to the representation of the image data.

An exemplary electronic device includes one or more input devices; display; One or more processors; And a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs correspond to selection of a first image filter of a representation of image data having a first appearance, via one or more input devices. Instructions for detecting a first input; And in response to detecting the first input, instructions for applying the first image filter to the representation of the image data, the instructions for applying, upon determining that the image data has depth information associated therewith- Depth information enables the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data Commands for applying the first image filter to the foreground region of the first level adjustment indicates a first degree that the first image filter changes the appearance of the representation of the image data; Instructions for applying the first image filter to the background area of the representation of the image data using the second level adjustment to change the appearance of the background area of the representation of the image data-the second level adjustment is performed by the first image filter Indicates a second degree of changing the appearance of the representation of the data, and the first level adjustment and the second level adjustment are different -; And after applying the first image filter to the representation of the image data, on the display, instructions for displaying the representation of each image to which the first filter is applied to the representation of the image data.

An exemplary electronic device includes one or more input devices; display; Means for detecting, via one or more input devices, a first input corresponding to a selection of a first image filter of a representation of image data having a first appearance; And in response to detecting the first input, means for applying the first image filter to the representation of the image data, the means for applying, upon determining that the image data has depth information associated therewith- Depth information enables the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the representation of the image data using first level adjustment to change the appearance of the foreground area of the representation of the image data Means for applying a first image filter to the foreground area of-the first level adjustment indicates a first degree by which the first image filter changes the appearance of the representation of the image data; Means for applying a first image filter to the background area of the representation of the image data using a second level adjustment to change the appearance of the background area of the representation of the image data-the second level adjustment means that the first image filter is the image data Represents the second degree of changing the appearance of the expression, and the first level adjustment and the second level adjustment are different -; And means for displaying, on the display, a representation of each image to which the first filter is applied to the representation of the image data, after applying the first image filter to the representation of the image data.

An exemplary method includes displaying, on an electronic device having one or more input devices, and a display, on a display, a filter selection interface that includes representations of a plurality of filters in a set of filters; While simultaneously displaying the filter selection interface and the representation of the image data on the display, the first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria, through one or more input devices Detecting; In response to detecting the first input, stopping displaying the first subset of the representations of the filters in the set of filters-the first subset of the representations of the filters in the set of filters is the first in the filter selection user interface Includes one or more filters in a first direction from the representation of the filter and one or more filters in a second direction from the representation of the first filter -; And maintaining a display of the second subset of representations of the filters in the set of filters, wherein the second subset of representations of the filters includes at least a representation of the first filter.

An exemplary non-transitory computer-readable storage medium stores one or more input devices, and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying, on a display, a filter selection interface that includes representations of a plurality of filters in a set of filters; While simultaneously displaying the filter selection interface and the representation of the image data on the display, the first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria, through one or more input devices Instructions for detecting; In response to detecting the first input, instructions for stopping displaying the first subset of representations of the filters in the set of filters-the first subset of representations of the filters in the set of filters is within the filter selection user interface. Includes one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter -; And instructions for maintaining the display of the second subset of representations of the filters of the set of filters, the second subset of representations of the filters comprising at least the representation of the first filter.

An exemplary temporary computer-readable storage medium stores one or more input devices, and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying, on a display, a filter selection interface that includes representations of a plurality of filters in a set of filters; While simultaneously displaying the filter selection interface and the representation of the image data on the display, the first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria, through one or more input devices Instructions for detecting; In response to detecting the first input, instructions for stopping displaying the first subset of representations of the filters in the set of filters-the first subset of representations of the filters in the set of filters is within the filter selection user interface. Includes one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter -; And instructions for maintaining the display of the second subset of representations of the filters of the set of filters, the second subset of representations of the filters comprising at least the representation of the first filter.

An exemplary electronic device includes one or more input devices; display; A filter selection interface comprising one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising, on a display, representations of a plurality of filters in a set of filters. Instructions for displaying; While simultaneously displaying the filter selection interface and the representation of the image data on the display, the first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria, through one or more input devices Instructions for detecting; In response to detecting the first input, instructions for stopping displaying the first subset of representations of the filters in the set of filters-the first subset of representations of the filters in the set of filters is within the filter selection user interface. Includes one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter -; And instructions for maintaining the display of the second subset of representations of the filters of the set of filters, the second subset of representations of the filters comprising at least the representation of the first filter.

An exemplary electronic device includes one or more input devices; display; Means for displaying, on the display, a filter selection interface comprising representations of a plurality of filters in the set of filters; While simultaneously displaying the filter selection interface and the representation of the image data on the display, the first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria, through one or more input devices Means for detecting; In response to detecting the first input, means for stopping displaying the first subset of representations of the filters in the set of filters-the first subset of representations of the filters in the set of filters is selected in the filter selection user interface. Includes one or more filters in a first direction from the representation of one filter and one or more filters in a second direction from the representation of the first filter -; And means for maintaining a display of a second subset of representations of filters of the set of filters, wherein the second subset of representations of filters includes at least a representation of the first filter.

An exemplary method includes displaying a camera viewfinder for capturing media on a display in a camera, sensor, one or more input devices, and an electronic device having a display; And while displaying the camera viewfinder, based on data from the sensor, upon determining that the device meets the alignment guide display criteria, the alignment guide display criteria are pre-configured with the orientation of the focal plane of the camera so that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-on the display, displaying the alignment guides in the camera viewfinder-the appearance of the alignment guides is such that the orientation of the focal plane of the camera is at a predetermined orientation. As it changes about -; And, based on the data from the sensor, in accordance with the determination that the alignment guide display criteria are not met, holding the display of the alignment guide in the camera viewfinder.

An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a camera, sensor, one or more input devices, and an electronic device having a display. The one or more programs include instructions for displaying a camera viewfinder for capturing media on a display; And while displaying the camera viewfinder, based on data from the sensor, upon determining that the device meets the alignment guide display criteria, the alignment guide display criteria are pre-configured with the orientation of the focal plane of the camera so that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-on the display, instructions for displaying the alignment guides in the camera viewfinder-the appearance of the alignment guides is such that the orientation of the focal plane of the camera is predetermined. As it changes with respect to orientation -; And based on data from the sensor, instructions for suspending displaying the alignment guide in the camera viewfinder, upon determining that the alignment guide display criteria are not met.

An exemplary temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a camera, sensor, one or more input devices, and an electronic device having a display. The one or more programs include instructions for displaying a camera viewfinder for capturing media on a display; And while displaying the camera viewfinder, based on data from the sensor, upon determining that the device meets the alignment guide display criteria, the alignment guide display criteria are pre-configured with the orientation of the focal plane of the camera so that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-on the display, instructions for displaying the alignment guides in the camera viewfinder-the appearance of the alignment guides is such that the orientation of the focal plane of the camera is predetermined. As it changes with respect to orientation -; And based on data from the sensor, instructions for suspending displaying the alignment guide in the camera viewfinder, upon determining that the alignment guide display criteria are not met.

Exemplary electronic devices include a camera; sensor; One or more input devices; display; One or more processors; And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising: instructions for displaying a camera viewfinder for capturing media on a display; And while displaying the camera viewfinder, based on data from the sensor, upon determining that the device meets the alignment guide display criteria, the alignment guide display criteria are pre-configured with the orientation of the focal plane of the camera so that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-on the display, instructions for displaying the alignment guides in the camera viewfinder-the appearance of the alignment guides is such that the orientation of the focal plane of the camera is predetermined. As it changes with respect to orientation -; And based on data from the sensor, instructions for suspending displaying the alignment guide in the camera viewfinder, upon determining that the alignment guide display criteria are not met.

Exemplary electronic devices include a camera; sensor; One or more input devices; display; Means for displaying a camera viewfinder for capturing media on the display; And while displaying the camera viewfinder, based on data from the sensor, upon determining that the device meets the alignment guide display criteria, the alignment guide display criteria are pre-configured with the orientation of the focal plane of the camera so that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-on the display, means for displaying the alignment guides in the camera viewfinder-the appearance of the alignment guides is such that the orientation of the focal plane of the camera is a predetermined orientation Changes as it changes about -; And means for suspending displaying the alignment guide in the camera viewfinder according to the determination that the alignment guide display criteria are not satisfied based on the data from the sensor.

An exemplary method includes displaying, on an electronic device having one or more input devices, one or more cameras, and a display, on a display, a representation of image data associated with depth map information; While displaying a representation of the image data on the display, detecting, via one or more input devices, a first input selecting a respective filter of a plurality of lighting effects based on the depth map information; After detecting the first input, detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras; And in response to detecting the second input, capturing image data corresponding to the field of view of the one or more cameras, wherein the capturing comprises: the respective lighting effects selected based on the first input, the first lighting effects Capturing the image data corresponding to the field of view of one or more cameras and associating the first lighting effect with a representation of the image data, the first lighting effect being based on depth map information; And capturing the image data corresponding to the field of view of one or more cameras according to the determination that the respective lighting effects selected based on the first input are the second lighting effects different from the first lighting effects, and the second lighting effects of the image data. And associating with the representation, the second lighting effect is based on depth map information.

An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of one or more input devices, one or more cameras, and an electronic device having a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input for selecting a respective filter of a plurality of lighting effects based on depth map information, via one or more input devices, while displaying a representation of the image data on the display; Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras after detecting the first input; And in response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, the instructions for capturing, wherein respective lighting effects selected based on the first input are first Instructions for capturing image data corresponding to the field of view of one or more cameras and associating the first lighting effect with a representation of the image data, upon determination of the lighting effect, the first lighting effect being based on depth map information; And capturing the image data corresponding to the field of view of one or more cameras according to the determination that the respective lighting effects selected based on the first input are the second lighting effects different from the first lighting effects, and the second lighting effects of the image data. It includes instructions for associating the representation, and the second lighting effect is based on depth map information.

An exemplary temporary computer-readable storage medium stores one or more programs configured to be executed by one or more input devices, one or more cameras, and one or more processors of an electronic device having a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input for selecting a respective filter of a plurality of lighting effects based on depth map information, via one or more input devices, while displaying a representation of the image data on the display; Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras after detecting the first input; And in response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, the instructions for capturing, wherein respective lighting effects selected based on the first input are first Instructions for capturing image data corresponding to the field of view of one or more cameras and associating the first lighting effect with a representation of the image data, upon determination of the lighting effect, the first lighting effect being based on depth map information; And capturing the image data corresponding to the field of view of one or more cameras according to the determination that the respective lighting effects selected based on the first input are the second lighting effects different from the first lighting effects, and the second lighting effects of the image data. It includes instructions for associating the representation, and the second lighting effect is based on depth map information.

An exemplary electronic device includes one or more input devices; One or more cameras; display; One or more processors; And a memory storing one or more programs configured to be executed by one or more processors, the one or more programs comprising: instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input for selecting a respective filter of a plurality of lighting effects based on depth map information, via one or more input devices, while displaying a representation of the image data on the display; Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras after detecting the first input; And in response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, the instructions for capturing, wherein respective lighting effects selected based on the first input are first Instructions for capturing image data corresponding to the field of view of one or more cameras and associating the first lighting effect with a representation of the image data, upon determination of the lighting effect, the first lighting effect being based on depth map information; And capturing the image data corresponding to the field of view of one or more cameras according to the determination that the respective lighting effects selected based on the first input are the second lighting effects different from the first lighting effects, and the second lighting effects of the image data. It includes instructions for associating the representation, and the second lighting effect is based on depth map information.

An exemplary electronic device includes one or more input devices; One or more cameras; display; Means for displaying, on the display, a representation of the image data associated with the depth map information; Means for displaying, on the display, a representation of the image data associated with the depth map information, while displaying the representation of the image data; Means for detecting a first input for selecting a respective filter of a plurality of lighting effects based on depth map information, via one or more input devices, while displaying a representation of the image data on the display; Means for detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras after detecting the first input; And in response to detecting the second input, means for capturing image data corresponding to the field of view of the one or more cameras, the means for capturing, wherein respective lighting effects selected based on the first input are first Means for capturing image data corresponding to the field of view of one or more cameras and associating the first lighting effect with a representation of the image data, according to the determination of the lighting effect, the first lighting effect being based on depth map information; And capturing the image data corresponding to the field of view of one or more cameras according to the determination that the respective lighting effects selected based on the first input are the second lighting effects different from the first lighting effects, and the second lighting effects of the image data. And means for associating with the representation, the second lighting effect is based on depth map information.

The executable instructions for performing these functions are, optionally, included in a non-transitory computer readable storage medium or other computer program product configured for execution by one or more processors. The executable instructions for performing these functions are optionally included in a temporary computer readable storage medium or other computer program product configured for execution by one or more processors.

Thus, devices are provided with faster and more efficient methods and interfaces for managing camera effects, thereby increasing the effectiveness, efficiency and user satisfaction of these devices. These methods and interfaces can complement or replace other methods for managing camera effects.

For a better understanding of the various described embodiments, reference should be made to the specific details for carrying out the following invention in connection with the following figures, in which like reference numbers indicate corresponding parts throughout the figures.
1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display, in accordance with some embodiments.
1B is a block diagram illustrating example components for event processing, in accordance with some embodiments.
2 illustrates a portable multifunction device with a touch screen, in accordance with some embodiments.
3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, in accordance with some embodiments.
4A illustrates an example user interface for a menu of applications on a portable multifunction device, according to some embodiments.
4B illustrates an example user interface for a multifunction device having a touch-sensitive surface that is separate from the display, according to some embodiments.
5A illustrates a personal electronic device in accordance with some embodiments.
5B is a block diagram illustrating a personal electronic device in accordance with some embodiments.
5C and 5D illustrate example components of a personal electronic device with a touch-sensitive display and intensity sensors, in accordance with some embodiments.
5E-5H illustrate example components and user interfaces of a personal electronic device, according to some embodiments.
6A-6N illustrate example devices and user interfaces for managing camera lighting effects, in accordance with some embodiments.
7A-7F are flow diagrams illustrating a method for managing camera lighting effects, in accordance with some embodiments.
8A-8J illustrate example devices and user interfaces for applying camera lighting effects, in accordance with some embodiments.
9A-9D are flow diagrams illustrating a method for applying camera lighting effects, in accordance with some embodiments.
10A-10N illustrate example devices and user interfaces for managing filter effects, in accordance with some embodiments.
11A-11C are flow diagrams illustrating a method for managing filter effects, in accordance with some embodiments.
12A-12P illustrate example devices and user interfaces for managing a filter user interface, in accordance with some embodiments.
13A-13F are flow diagrams illustrating a method for managing a filter user interface, in accordance with some embodiments.
14A-14K illustrate example devices and user interfaces for capturing an image, in accordance with some embodiments.
15A-15E are flow diagrams illustrating a method for capturing an image, according to some embodiments.
16A-16J illustrate example devices and user interfaces for capturing an image, in accordance with some embodiments.
17A-17G are flow diagrams illustrating a method for capturing an image, according to some embodiments.

The following description describes exemplary methods, parameters, and the like. However, it should be appreciated that this description is not intended as a limitation on the scope of the present disclosure and is instead provided as a description of exemplary embodiments.

Electronic devices are designed and manufactured with more advanced camera features and sensors. However, some electronic devices, due to the nature of their design, cannot capture rich point-of-light photos without additional hardware. Most of the spectral photography techniques require not only a number of expensive light sources to be positioned around the subject, but also a separate backdrop. However, many electronic devices have only a single flash that emits light in one direction. As a result, most of the point light photography technologies cannot be achieved simply through traditional electronic devices.

Embodiments described herein include electronic devices that utilize depth map information to provide improved camera capabilities. In some embodiments, depth map information is used when applying a filter to the image. In some embodiments, visual assistance is provided to help the user capture the perfect image. The described embodiments also include complementary user interfaces that enable these improved camera capabilities.

At the same time, the described embodiments improve the performance of the device's camera capabilities, while allowing efficient packaging and manufacturing of thin and lightweight devices. The use of fixed focal length cameras is beneficial when they are thinner and smaller.

1A and 1B, 2, 3, 4A and 4B and 5A to 5H provide descriptions of example devices for performing techniques for managing event notifications. 6A-6N illustrate example user interfaces for managing a user interface. 7 is a flow diagram illustrating methods of managing a user interface in accordance with some embodiments. The user interfaces of FIGS. 6A-6G are used to illustrate the processes described below, including the processes of FIG. 7.

8A-8J illustrate example user interfaces for applying simulated lighting effects. 9 is a flow diagram illustrating methods of simulating a lighting effect in accordance with some embodiments. The user interfaces of FIGS. 8A-8D are used to illustrate the processes described below, including the processes of FIG. 9.

10A-10N illustrate example user interfaces for applying a filter to an image. 11 is a flow diagram illustrating methods of applying a filter to an image in accordance with some embodiments. The user interfaces of FIGS. 10A-10D are used to illustrate the processes described below, including the processes of FIG. 11.

12A-12P illustrate example user interfaces for displaying a reduced filter user interface. 13 is a flow diagram illustrating methods of displaying a reduced filter user interface in accordance with some embodiments. The user interfaces of FIGS. 12A-12D are used to illustrate the processes described below, including the processes of FIG. 13.

14A-14K illustrate example user interfaces for providing visual assistance. 15 is a flow diagram illustrating methods of providing visual assistance in accordance with some embodiments. The user interfaces of FIGS. 14A-14D are used to illustrate the processes described below, including the processes of FIG. 15.

16A-16J illustrate example user interfaces for providing visual assistance when applying a simulated optical effect. 17 is a flow diagram illustrating methods of providing visual assistance when applying a simulated optical effect according to some embodiments. The user interfaces of FIGS. 16A-16J are used to illustrate the processes described below, including the processes of FIG. 17.

Although the following description uses terms such as "first", "second", etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first touch may be referred to as the second touch, and similarly, the second touch may be referred to as the first touch. The first touch and the second touch are both touches, but they are not the same touch.

The terms used in the description of the various embodiments described herein are for the purpose of describing specific embodiments only, and are not intended to be limiting. As used in the description of the various described embodiments and the appended claims, singular forms (“a”, “an”, and “the”) are intended to include multiple forms as well, unless the context clearly dictates otherwise. Is intended Also, it will be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated items listed. The terms "include", "including", "comprise", and/or "comprising", as used herein, refer to the features, integers Specifies the presence of fields, steps, operations, elements, and/or components, but the presence of one or more other features, integers, steps, operations, elements, components, and/or groups thereof Or it will be further understood that the addition is not excluded.

The term “if” is, optionally, “when” or “upon” or “in response to determining” or “detection, depending on context. It is interpreted to mean "in response to detecting". Similarly, the phrases “when determined to be” or “when [the condition or event mentioned] is detected” are, optionally, depending on the context, “when determined to be” or “in response to what is determined to be” or “[ Upon detection of the stated condition or event] or “in response to detecting the [mentioned condition or event]”.

Embodiments of electronic devices, user interfaces to such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices limit the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, CA. Without. Other portable electronic devices, such as laptop or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touchpads), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer with a touch-sensitive surface (eg, a touch screen display and/or touchpad).

In the following discussion, an electronic device comprising a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and/or joystick.

Devices typically support a variety of applications such as one or more of the following: drawing applications, presentation applications, word processing applications, website authoring applications, disk authoring applications, spreadsheet applications, gaming applications, phone applications, video conferencing applications, email Application, instant messaging application, exercise support application, photo management application, digital camera application, digital video camera application, web browsing application, digital music player application, and/or digital video player application.

Various applications running on the device, optionally, use at least one universal physical user interface device, such as a touch-sensitive surface. The one or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, are optionally adjusted and/or changed from one application to the next and/or within each application. In this way, the device's universal physical architecture (such as a touch-sensitive surface) optionally supports a variety of applications using user interfaces that are intuitive and transparent to the user.

Attention is now directed towards embodiments of portable devices with touch-sensitive displays. 1A is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112, in accordance with some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touch screen” for convenience, and is sometimes known as or referred to as a “touch-sensitive display system”. Device 100 includes memory 102 (optionally including one or more computer readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripheral interface 118, RF Circuitry 108, audio circuitry 110, speaker 111, microphone 113, input/output (I/O) subsystem 106, other input control devices 116, and external port 124 It includes. Device 100 optionally includes one or more optical sensors 164. The device 100 is, optionally, one or more contact intensity sensors for detecting the intensity of contacts on the device 100 (eg, a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100). (165). The device 100 may optionally include a touch-sensitive display system 112 of the device 100 or a touchpad 355 of the device 300 for generating tactile outputs on the device 100. And one or more tactile output generators 167 (for generating tactile outputs on the same touch-sensitive surface). These components optionally communicate over one or more communication buses or signal lines 103.

As used in the specification and claims, the term “strength” of contact on a touch-sensitive surface means the force or pressure (force per unit area) of the contact (eg, finger contact) on the touch-sensitive surface, or touch. It refers to a substitute (proxy) for the force or pressure of contact on a responsive surface. The intensity of the contact has a range of values, including at least four distinct values and more typically hundreds (eg, at least 256) distinct values. The intensity of the contact is selectively determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine the estimated force of contact. Similarly, the pressure sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch sensitive surface. Alternatively, the size and/or changes to the contact area detected on the touch-sensitive surface, the capacitance of the touch-sensitive surface near the contact and/or changes thereto, and/or the touch-sensitive surface near the contact The resistance and/or variations thereof are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, alternative measurements for contact force or pressure are used directly to determine whether the intensity threshold has been exceeded (eg, the intensity threshold is described in units corresponding to the alternative measurements). In some implementations, alternative measurements for contact force or pressure are converted to estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (eg, intensity threshold). Is the pressure threshold measured in units of pressure). Using the intensity of the contact as a property of the user input, otherwise displays the affordances (eg, on a touch-sensitive display) and/or (eg, a touch-sensitive display, a touch-sensitive surface, or Enables user access to additional device functions that may not be accessible by the user on a reduced sized device with a limited real area to receive user input (via a physical or mechanical control, such as a knob or button) To do.

As used in the specification and claims, the term “tactile output” refers to the physical displacement of the device relative to the device's previous location, the device's component (eg, touch-sensitive surface) relative to other components of the device (eg, housing). ), or the displacement of the component relative to the center of mass of the device to be detected by the user using the user's tactile sense. For example, in a situation where a device or component of a device contacts a touch-sensitive user's surface (eg, the user's hand's finger, palm, or other area), the tactile output generated by the physical displacement is generated by the user by the device or It will be interpreted as a tactile sensation corresponding to the perceived change in the physical properties of the component of the device. For example, movement of a touch-sensitive surface (eg, a touch-sensitive display or trackpad) is optionally interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, the user will feel tactile, such as "down click" or "up click" even if there is no movement of the physical actuator button associated with the touch-sensitive surface that is physically pressed (eg, displaced) by the user's movement. . As another example, movement of a touch-sensitive surface is optionally interpreted or sensed as a “roughness” of the touch-sensitive surface by the user, even when there is no change in flatness of the touch-sensitive surface. Although these interpretations of touch by the user will be susceptible to the user's individualized sensory perception, there are many touch sensation perceptions common to most users. Thus, when a tactile output is described as corresponding to a user's specific sensory perception (eg, “up click”, “down click”, “rough”), unless otherwise stated, the tactile output generated is typical (or Average) corresponds to the physical displacement of the device or its components that will generate the described sensory perception of the user.

Device 100 is only one example of a portable multifunction device, and device 100 may, optionally, have more or fewer components than shown, or alternatively, combine two or more components, or optionally components It should be understood that they have different configurations or arrangements of. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and/or application-specific integrated circuits (ASICs).

The memory 102 optionally includes high-speed random access memory, and also optionally, one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Non-volatile memory, such as. The memory controller 122 optionally controls access to the memory 102 by other components of the device 100.

Peripheral interface 118 can be used to couple the input and output peripherals of the device to CPU 120 and memory 102. One or more processors 120 drive or execute various sets of software programs and/or instructions stored in memory 102 to perform various functions for device 100 and to process data. In some embodiments, peripheral interface 118, CPU 120 and memory controller 122 are, optionally, implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

The radio frequency (RF) circuitry 108 receives and transmits RF signals, also called electromagnetic signals. The RF circuitry 108 converts electrical signals to/from electromagnetic signals and communicates with the communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes, but is not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, memory, and the like. Does not include well-known circuitry for performing these functions. RF circuitry 108 may optionally be networks, such as the Internet, intranet, and/or wireless networks, also referred to as the World Wide Web (WWW), such as cellular telephone networks, wireless local area networks (LANs), and/or MANs. (metropolitan area network), and other devices by wireless communication. The RF circuitry 108 optionally includes well-known circuitry for detecting near field communication (NFC) fields, such as by a short-range communication radio. Wireless communication, optionally, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), evolution (EV-DO), Data-Only, HSPA, HSPA+, DC-HSPDA (Dual-Cell HSPA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA) ), time division multiple access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and/or IEEE 802.11ac), voice over Internet Protocol (VoIP), Wi-MAX, protocols for e-mail (e.g. Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging XMPP) and presence protocol, SIMPLE (Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions), IMPS (Instant Messaging and Presence Service), and/or SMS (Short Message Service), or communications that have not been developed as of the filing date of this document. Any other suitable, including protocols Use any of a plurality of communication standards, protocols and technologies, including, but not limited to, communication protocols.

The audio circuit unit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit unit 110 receives audio data from the peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts an electrical signal into sound waves that can be heard by humans. The audio circuitry 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit unit 110 converts an electrical signal into audio data, and transmits the audio data to the peripheral interface 118 for processing. The audio data is, optionally, fetched from the memory 102 and/or RF circuitry 108 by the peripherals interface 118 and/or transmitted to the memory 102 and/or RF circuitry 108. In some embodiments, the audio circuitry 110 also includes a headset jack (eg, 212 in FIG. 2 ). The headset jack is an interface between the audio circuitry 110 and removable audio input/output peripherals, such as a headset having both output-only headphones, or an output (eg, headphone for one or both ears) and an input (eg, microphone). Provides

I/O subsystem 106 couples input/output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripheral interface 118. I/O subsystem 106 may optionally include display controller 156, light sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, depth camera controller 169, and other inputs or controls. And one or more input controllers 160 for devices. One or more input controllers 160 receive/transmit electrical signals to/from other input control devices 116. Other input control devices 116 optionally include physical buttons (eg, push button, rocker button, etc.), dial, slider switch, joystick, click wheel, and the like. In some alternative embodiments, input controller(s) 160 is optionally coupled to (or not coupled to) any of the pointer device, such as a keyboard, infrared port, USB port, and mouse. . The one or more buttons (eg, 208 in FIG. 2) optionally include an up/down button for controlling the volume of the speaker 111 and/or the microphone 113. The one or more buttons optionally include a push button (eg, 206 in FIG. 2 ).

The quick press of a push button initiates the process of selectively unlocking the touch screen 112 or using gestures on the touch screen to unlock the device, which is December 23, 2005. US Patent Application No. 11/322,549, filed as "Unlocking a Device by Performing Gestures on an Unlock Image" (US Pat. No. 7,657,849), which is hereby incorporated by reference in its entirety. . Longer pressing of a push button (eg, 206) optionally turns the device 100 on or off. The functionality of one or more buttons is, optionally, customizable. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. Display controller 156 receives and/or transmits electrical signals to/from touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video and any combination thereof (collectively referred to as “graphics”). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

Touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and/or tactile contact. Touch screen 112 and display controller 156 (and any associated modules and/or sets of instructions in memory 102) touch on touch screen 112 (and any movement or interruption of contact). ) And converts the detected contact into interaction with user interface objects (eg, one or more soft keys, icons, web pages or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 selectively uses liquid crystal display (LCD) technology, light emitting polymer display (LPD) technology, or light emitting diode (LED) technology, but other display technologies are used in other embodiments. The touch screen 112 and the display controller 156 may, optionally, provide one or more contact points with the capacitive, resistive, infrared, and surface acoustic wave technologies as well as other proximity sensor arrays, or the touch screen 112. Contact and any movement or interruption thereof is detected using any of a plurality of touch sensing techniques currently known or developed later, including, but not limited to, other factors for determining. In an exemplary embodiment, projected mutual capacitance sensing technology, such as found on iPhone® and iPod Touch® from Apple Inc. of Cupertino, California, is used.

The touch-sensitive display in some embodiments of the touch screen 112 is, optionally, the following U.S. Patents 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and/or 6,677,932 (Westerman) , And/or similar to the multi-touch sensitive touchpads described in U.S. Patent Publication No. 2002/0015024A1, each of which is incorporated herein by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, while touch-sensitive touchpads do not provide visual output.

Touch-sensitive displays in some embodiments of touch screen 112 are described in the following applications: (1) US Patent Application No. 11/381,313, filed May 2, 2006, "Multipoint Touch Surface Controller"; (2) United States Patent Application Serial No. 10/840,862, filed May 6, 2004, "Multipoint Touchscreen"; (3) United States Patent Application Serial No. 10/903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) United States Patent Application Serial No. 11/048,264, filed January 31, 2005, "Gestures For Touch Sensitive Input Devices"; (5) United States Patent Application Serial No. 11/038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices"; (6) United States Patent Application Serial No. 11/228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface"; (7) United States Patent Application Serial No. 11/228,700, filed September 16, 2005, "Operation Of A Computer With A Touch Screen Interface"; (8) United States Patent Application Serial No. 11/228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard"; And (9) US Patent Application No. 11/367,749, filed March 3, 2006, "Multi-Functional Hand-Held Device". All of these applications are incorporated herein by reference in their entirety.

The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. The user optionally contacts the touch screen 112 using any suitable object or accessory, such as a stylus, finger, or the like. In some embodiments, the user interface is primarily designed to operate using finger-based contacts and gestures, which may be less precise than a stylus-based input due to the larger contact area of the finger on the touch screen. In some embodiments, the device converts the coarse finger-based input to a precise pointer/cursor position or command to perform actions desired by the user.

In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, a touchpad, unlike a touch screen, is a touch-sensitive area of a device that does not display visual output. The touchpad is, optionally, a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

The device 100 also includes a power system 162 for powering various components. The power system 162 is, optionally, a power management system, one or more power sources (eg, batteries, alternating current (AC)), recharging systems, power failure detection circuits, power converters or inverters, power status indicators ( For example, light emitting diodes (LEDs), and any other components associated with the generation, management and distribution of power within portable devices.

Device 100 also optionally includes one or more optical sensors 164. 1A shows an optical sensor coupled to optical sensor controller 158 in I/O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. The optical sensor 164 receives light from the surrounding environment, which is projected through one or more lenses, and converts the light into data representing an image. In conjunction with imaging module 143 (also referred to as a camera module), optical sensor 164 optionally captures still images or video. In some embodiments, the light sensor is located on the back of the device 100 opposite the touch screen display 112 on the front of the device, so that the touch screen display can be used as a viewfinder for still and/or video image acquisition. Have it. In some embodiments, the light sensor is located on the front of the device, so that while the user views other video conference participants on the touch screen display, optionally, the user's image is obtained for the video conference. In some embodiments, the position of the light sensor 164 can be changed by the user (eg, by rotating the lens and sensor in the device housing) such that the single light sensor 164 is imaged with a touch screen display. Used for both meetings and still and/or video image acquisition.

Device 100 also, optionally, includes one or more contact intensity sensors 165. 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 in I/O subsystem 106. The contact strength sensor 165 may optionally be one or more piezo-resistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (eg, And sensors used to measure the force (or pressure) of the contact on the touch-sensitive surface. The contact intensity sensor 165 receives contact intensity information (eg, pressure information or a proxy for pressure information) from the surrounding environment. In some embodiments, the at least one contact intensity sensor is collocated or proximate to a touch-sensitive surface (eg, touch-sensitive display system 112 ). In some embodiments, at least one contact intensity sensor is located on the back side of the device 100 opposite the touch screen display 112 located on the front side of the device 100.

Device 100 also optionally includes one or more proximity sensors 166. 1A shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 is, optionally, coupled to input controller 160 in I/O subsystem 106. Proximity sensor 166 is, optionally, US Patent Applications No. 11/241,839, "Proximity Detector In Handheld Device"; 11/240,788, "Proximity Detector In Handheld Device"; 11/620,702, "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11/586,862, "Automated Response To And Sensing Of User Activity In Portable Devices"; And 11/638,251, “Methods And Systems For Automatic Configuration Of Peripherals,” which are hereby incorporated by reference in their entirety. In some embodiments, the proximity sensor turns off and disables the touch screen 112 when the multifunction device is positioned near the user's ear (eg, when the user is talking on the phone).

Device 100 also optionally includes one or more tactile output generators 167. 1A shows a tactile output generator coupled to a haptic feedback controller 161 in I/O subsystem 106. The tactile output generator 167 optionally generates one or more electroacoustic devices such as speakers or other audio components and/or a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output. Electromechanical devices that convert energy into linear motion, such as a component (eg, a component that converts electrical signals to tactile outputs on the device). The contact intensity sensor 165 receives haptic feedback generation commands from the haptic feedback module 133 to generate tactile outputs on the device 100 that can be sensed by a user of the device 100. In some embodiments, the at least one tactile output generator is positioned or proximate to a touch-sensitive surface (eg, touch-sensitive display system 112 ), and optionally, a touch-sensitive surface is vertically (eg, , By moving in and out of the surface of the device 100 or laterally (eg, back and forth in the same plane as the surface of the device 100) to generate a tactile output. In some embodiments, at least one tactile output generator sensor is located on the back of the device 100 opposite the touch screen display 112 located on the front of the device 100.

Device 100 also optionally includes one or more accelerometers 168. 1A shows accelerometer 168 coupled to peripherals interface 118. Alternatively, accelerometer 168 is optionally coupled to input controller 160 in I/O subsystem 106. The accelerometer 168 is, optionally, US Patent Publication No. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices" and US Patent Publication No. 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or landscape view on a touch screen display based on analysis of data received from one or more accelerometers. The device 100 may optionally include, in addition to the accelerometer(s) 168, a magnetometer (not shown), and GPS for obtaining information about the location and orientation (eg, portrait or landscape) of the device 100 ( Or GLONASS or other global navigation system) receiver (not shown).

Device 100 also optionally includes one or more depth camera sensors 175. 1A shows a depth camera sensor coupled to depth camera controller 169 in I/O subsystem 106. Depth camera sensor 175 receives data from the environment, projected through the sensor. In conjunction with imaging module 143 (also referred to as a camera module), depth camera sensor 175 is optionally used to determine a depth map of different portions of the image captured by imaging module 143. In some embodiments, a depth camera sensor is located on the front of the device 100, for a video conference while the user views other video conference participants on a touch screen display, and to capture selfies with depth map data. , Optionally, allows the user's image with depth information to be obtained. In some embodiments, the location of the depth camera sensors 175 can be changed by the user (eg, by rotating the lens and sensor in the device housing) so that the depth camera sensors 175 can be combined with the touch screen display. It is used for both video conferencing and still and/or video image acquisition.

In some embodiments, software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact/motion module (or set of instructions) 130, graphics module ( Or a set of instructions) 132, a text input module (or set of instructions) 134, a GPS module (or set of instructions) 135, and applications (or sets of instructions) 136. . Moreover, in some embodiments, the memory (102 of FIG. 1A or 370 of FIG. 3) stores the device/global internal state 157 as shown in FIGS. 1A and 3. The device/global internal state 157, if present, includes an active application state indicating which applications are currently active; A display state indicating which applications, views or other information occupy various areas of the touch screen display 112; A sensor state comprising information obtained from various sensors of the device and input control devices 116; And location information regarding the location and/or posture of the device.

Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating system such as VxWorks) includes common system tasks (e.g., memory management, storage device control, power management, etc.) It includes various software components and/or drivers for controlling and managing data, and facilitates communication between various hardware and software components.

The communication module 128 enables communication with other devices through one or more external ports 124, and also for processing data received by the RF circuitry 108 and/or external port 124. Includes software components. The external port 124 (eg, USB, FIREWIRE, etc.) is configured to couple directly to other devices or indirectly through a network (eg, Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (eg, 30-pin) connector that is the same or similar and/or compatible with the 30-pin connector used in iPod® (trademark of Apple Inc.) devices.

The contact/motion module 130 optionally detects contact with the touch screen 112 (along with the display controller 156), and other touch-sensitive devices (eg, a touchpad or physical click wheel). . The contact/motion module 130 determines whether a contact has occurred (eg, detects a finger-down event), the intensity of the contact (eg, the force or pressure of the contact, or Determining the force or pressure of a contact), and determining whether there is a movement of the contact and tracking the movement across the touch-sensitive surface (e.g., one or more finger-dragging events) ), and determining whether the contact is stopped (e.g., detecting a finger-up event or contact break), and various actions related to the detection of the contact. Contains various software components to perform. The contact/motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by the series of contact data, optionally, determines the speed (magnitude), speed (magnitude and direction), and/or acceleration (magnitude and/or direction change) of the contact point. It includes doing. These operations are, optionally, applied to single contacts (eg, one finger contacts) or to multiple simultaneous contacts (eg, “multitouch”/multiple finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect a contact on the touch pad.

In some embodiments, the contact/motion module 130 determines one or more intensity thresholds (eg, to determine whether the user has “clicked” over the icon) to determine whether the action was performed by the user. Use a set. In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (eg, intensity thresholds are not determined by activation thresholds of specific physical actuators, and changing the physical hardware of device 100) Can be adjusted without). For example, the mouse "click" threshold of the trackpad or touch screen display can be set to any of a wide range of predefined thresholds without changing the trackpad or touch screen display hardware. Additionally, in some implementations, the user of the device (eg, by adjusting individual intensity thresholds and/or by adjusting multiple intensity thresholds at once with a system level click “strength” parameter) one of a set of intensity thresholds Software settings are provided to control the anomaly.

The contact/motion module 130 optionally detects a gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different motions, timings, and/or intensities of detected contacts). Therefore, the gesture is selectively detected by detecting a specific contact pattern. For example, detecting a finger tap gesture detects a finger-down event followed by a finger-at the same location (or substantially the same location) as the finger-down event (eg, at the location of the icon). And detecting an up (liftoff) event. As another example, detecting a finger swipe gesture on a touch-sensitive surface detects a finger-down event followed by one or more finger-dragging events, followed by a finger-up (liftoff). ) Detecting an event.

The graphics module 132 renders graphics on the touch screen 112 or other display, and includes components for changing the visual effects (eg, brightness, transparency, saturation, contrast, or other visual properties) of the displayed graphics. It includes various known software components for display. As used herein, the term “graphics” includes without limitation text, web pages, icons (eg, user interface objects including soft keys), digital images, videos, animations, and the like. , Any object that can be displayed to the user.

In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is, optionally, assigned a corresponding code. The graphics module 132 receives one or more codes specifying applications to be displayed, along with coordinate data and other graphic attribute data, if necessary, from applications or the like, and then generates screen image data to display controller 156. Output.

Haptic feedback module 133 is used by tactile output generator(s) 167 to generate tactile outputs at one or more locations on device 100 in response to user interactions with device 100. Includes various software components for generating instructions.

The text input module 134, which is optionally a component of the graphics module 132, requires a variety of applications (eg, contact 137, email 140, IM 141, browser 147, and text input). Soft keyboards for entering text into any other application.

The GPS module 135 determines the location of the device and uses this information on the phone 138 for use in various applications (eg, for use in location-based dialing; camera 143 as photo/video metadata) And to applications that provide location-based services such as weather widgets, regional yellow page widgets and map/navigation widgets).

Applications 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof:

연락처 모듈(137)(때때로 주소록 또는 연락처 목록으로 지칭됨);

Contact module 137 (sometimes referred to as an address book or contact list);

전화 모듈(138);

Telephone module 138;

화상 회의 모듈(139);

Video conference module 139;

이메일 클라이언트 모듈(140);

Email client module 140;

인스턴트 메시징(IM) 모듈(141);

An instant messaging (IM) module 141;

운동 지원 모듈(142);

Exercise support module 142;

정지 및/또는 비디오 이미지들을 위한 카메라 모듈(143);

A camera module 143 for still and/or video images;

이미지 관리 모듈(144);

An image management module 144;

비디오 재생기 모듈;

Video player module;

음악 재생기 모듈;

Music player module;

브라우저 모듈(147);

Browser module 147;

캘린더 모듈(148);

Calendar module 148;

날씨 위젯(149-1), 주식 위젯(149-2), 계산기 위젯(149-3), 알람 시계 위젯(149-4), 사전 위젯(149-5), 및 사용자에 의해 얻어지는 다른 위젯들뿐 아니라 사용자-생성 위젯들(149-6) 중 하나 이상을 선택적으로 포함하는 위젯 모듈들(149);

Weather widget (149-1), stock widget (149-2), calculator widget (149-3), alarm clock widget (149-4), dictionary widget (149-5), and other widgets obtained by the user As well as widget modules 149, optionally including one or more of user-generated widgets 149-6;

사용자-생성 위젯들(149-6)을 만들기 위한 위젯 생성기 모듈(150);

A widget generator module 150 for creating user-created widgets 149-6;

검색 모듈(151);

Search module 151;

비디오 재생기 모듈 및 음악 재생기 모듈을 통합하는 비디오 및 음악 재생기 모듈(152);

A video and music player module 152 integrating the video player module and the music player module;

메모 모듈(153);

A memo module 153;

지도 모듈(154); 및/또는

Map module 154; And/or

온라인 비디오 모듈(155).

Online video module 155.

Examples of other applications 136 that are optionally stored in memory 102 are other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital copyright Management, speech recognition and speech reproduction.

Along with the touch screen 112, the display controller 156, the contact/motion module 130, the graphics module 132, and the text input module 134, the contact module 137 is, optionally, an address book or contact list ( For example, it is used to manage memory 102 or stored in application internal state 192 of contact module 137 in memory 370), which includes: adding name(s) to address book ; Deleting the name(s) from the address book; Associating phone number(s), email address(es), physical address(es) or other information with names; Associating images with names; Sorting and sorting names; Providing telephone numbers or email addresses to initiate and/or facilitate communication by telephone 138, video conferencing module 139, email 140 or IM 141, and the like.

RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text Along with the input module 134, the telephone module 138, optionally, inputs a sequence of characters corresponding to the telephone number, accesses one or more telephone numbers in the contact module 137, and modifies the entered telephone number It is used to dial, dial individual phone numbers, chat, and disconnect or disconnect when the conversation is complete. As described above, wireless communication optionally uses any of a plurality of communication standards, protocols, and technologies.

RF circuit section 108, audio circuit section 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion The video conferencing module 139, along with the module 130, the graphics module 132, the text input module 134, the contact module 137, and the phone module 138, may be configured to use the user with one or more other functions according to user instructions. Includes executable instructions to initiate, enforce and end a video conference between participants.

Email client module 140, along with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132 and text input module 134, is instructed by the user And executable instructions that allow you to compose, send, receive, and manage email in response to these. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails with still or video images taken with the camera module 143.

The instant messaging module 141, along with the RF circuit portion 108, the touch screen 112, the display controller 156, the contact/motion module 130, the graphic module 132, and the text input module 134, is instant Enter a sequence of characters corresponding to the message, modify previously entered characters, (e.g., Short Message Service (SMS) for phone-based instant messages or Multimedia Message Service) , MMS) protocol, or using XMPP, SIMPLE or IMPS for Internet-based instant messages) to include executable instructions to send individual instant messages, receive instant messages, and view received instant messages. . In some embodiments, the instant messages sent and/or received optionally include graphics, photos, audio files, video files and/or other attachments as supported by MMS and/or Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to phone-based messages (eg, messages sent using SMS or MMS) and internet-based messages (eg, messages sent using XMPP, SIMPLE or IMPS). Field).

RF circuit unit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154 , And with the music player module, the exercise support module 142 devises exercises (eg, with time, distance, and/or calorie consumption goals); Communicate with athletic sensors (sports devices); Receive motion sensor data; Calibrate sensors used to monitor exercise; Select and play music for exercise; And executable instructions to display, store and transmit exercise data.

With touch screen 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact/motion module 130, graphics module 132 and image management module 144 , The camera module 143 captures still images or videos (including video streams) and stores them in the memory 102, modifies the characteristics of the still images or videos, or the still images or images from the memory 102. Includes executable instructions to delete the video.

With touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134 and camera module 143, image management module 144 is stationary and And/or executable instructions for arranging, modifying (eg, editing) video images, or otherwise manipulating, labeling, deleting, presenting (eg, in a digital slideshow or album), and saving. .

The browser module 147, along with the RF circuit unit 108, the touch screen 112, the display controller 156, the contact/motion module 130, the graphic module 132, and the text input module 134, is a web page. Executable instructions for browsing the Internet according to user instructions, including retrieving, linking to, receiving, and displaying attachments and other files linked to web pages as well as fields or portions thereof. Includes

RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, email client module 140, and browser module ( Together with 147, the calendar module 148 is executable to create, display, modify, and store calendars and data associated with the calendars (eg, calendar entries, to-do lists, etc.) according to user instructions. Contains instructions.

Widget modules, along with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134 and browser module 147 149 is selectively downloaded and used by the user (eg, weather widget 149-1), stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary Widgets 149-5), or mini-applications created by the user (eg, user-generated widgets 149-6). In some embodiments, the widget includes a Hypertext Markup Language (HTML) file, Cascading Style Sheets (CSS) file, and JavaScript file. In some embodiments, the widget includes an Extensible Markup Language (XML) file and a JavaScript file (eg, Yahoo! widgets).

Widget generator module, along with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134 and browser module 147 150 is used to selectively create widgets by a user (eg, changing a user specific portion of a web page to a widget).

Together with the touch screen 112, the display controller 156, the contact/motion module 130, the graphics module 132, and the text input module 134, the search module 151 may perform one or more search criteria according to user instructions. Executable instructions to retrieve text, music, sound, image, video, and/or other files in memory 102 that match fields (eg, one or more user-specific search terms).

Along with the touch screen 112, the display controller 156, the contact/motion module 130, the graphic module 132, the audio circuit unit 110, the speaker 111, the RF circuit unit 108 and the browser module 147 , Video and music player module 152, executable instructions, and videos that allow a user to download and play recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files ( For example, on the touch screen 112 or on an external connected display via the external port 124, executable instructions to display, screen, or otherwise play. In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

Along with the touch screen 112, the display controller 156, the contact/motion module 130, the graphics module 132, and the text input module 134, the memo module 153 allows memos, according to user instructions. Includes executable instructions to create and manage task lists and the like.

RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147 ), the map module 154 optionally maps and data associated with the maps according to user instructions (eg, driving directions; data regarding shops and other points of interest at or near a particular location; And other location-based data).

Touch screen 112, display controller 156, contact/motion module 130, graphic module 132, audio circuit unit 110, speaker 111, RF circuit unit 108, text input module 134, Together with email client module 140 and browser module 147, online video module 155 allows a user to access, browse, and browse online videos in one or more file formats, such as H.264 (eg, streaming and/or Or by receiving (playing by download), playing (e.g., on a touch screen or via an external port 124 on an external connected display), sending an email with a link to a particular online video, otherwise letting it be managed Contains instructions. In some embodiments, an instant messaging module 141 rather than email client module 140 is used to send a link to a particular online video. For further description of the online video application, US Provisional Patent Application No. 60/936,562, filed June 20, 2007, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos" and December 31, 2007 U.S. Patent Application No. 11/968,067, filed dated, can be found in "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," the contents of which are hereby incorporated by reference in their entirety. .

Each of the modules and applications identified above are executable instructions for performing one or more of the functions described above and methods described herein (eg, computer-implemented methods and other information processing methods described herein). Corresponds to a set of things. These modules (eg, sets of instructions) need not be implemented as separate software programs, procedures or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. . For example, the video player module is optionally combined in a single module (eg, video and music player module 152 of FIG. 1A) with the music player module. In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. In addition, memory 102 optionally stores additional modules and data structures not described above.

In some embodiments, device 100 is a device in which operation of a predefined set of functions on a device is performed exclusively through a touch screen and/or touchpad. By using the touch screen and/or touchpad as the primary input control device for the operation of the device 100, the number of physical input control devices (such as push buttons, dials, etc.) on the device 100 is selectively reduced.

The predefined set of functions performed entirely through the touch screen and/or the touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main, home, or root menu. In these embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.

1B is a block diagram illustrating example components for event processing, in accordance with some embodiments. In some embodiments, the memory (102 of FIG. 1A or 370 of FIG. 3) includes event categorizer 170 (eg, in operating system 126) and individual applications 136-1 (eg, applications described above). Field (any of 137 to 151, 155, 380 to 390).

The event classifier 170 receives event information, and determines an application 136-1 to deliver the event information, and an application view 191 of the application 136-1. The event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates the current application view(s) displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, the device/global internal state 157 is used by the event classifier 170 to determine which application(s) are currently active, and the application internal state 192 is determined by the event classifier 170. It is used to determine application views 191 to deliver event information.

In some embodiments, the application internal state 192 is a user indicating resume information to be used when the application 136-1 resumes execution, information being displayed by the application 136-1 or ready to be displayed. Interface status information, a status queue to enable the user to return to the previous status or view of the application 136-1, and a redo/undo queue of previous actions taken by the user Includes additional information such as one or more.

The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about a sub-event (eg, a user touch on the touch-sensitive display 112 as part of a multi-touch gesture). Peripheral interface 118 may be from I/O subsystem 106 or sensors, such as proximity sensor 166, accelerometer(s) 168, and/or microphone 113 (via audio circuitry 110). Transmit the received information. The information that peripherals interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripherals interface 118 transmits event information only when there is an important event (eg, receiving input that exceeds a predetermined noise threshold and/or input that exceeds a predetermined duration). .

In some embodiments, event classifier 170 also includes a hit view determination module 172 and/or active event recognizer determination module 173.

The hit view determination module 172 provides software procedures for determining where a sub-event occurred within one or more views when the touch-sensitive display 112 displays more than one view. Views consist of controls and other elements that the user can see on the display.

Another aspect of the user interface associated with the application is a set of views, sometimes referred to herein as application views or user interface windows, where information is displayed and a touch-based gesture occurs. The application views (of an individual application) for which a touch is detected, optionally correspond to program levels in the application's program or view hierarchy. For example, the lowest level view where a touch is detected is optionally referred to as a hit view, and the set of events recognized as appropriate inputs is, optionally, based at least in part on the hit view of the initial touch that initiates the touch-based gesture. Is decided.

The hit view determination module 172 receives information related to sub-events of a touch-based gesture. If the application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should handle the subevent. In most situations, a hit view is the lowest level view in which a sub-event that is initiated (eg, a first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events associated with the same touch or input source that caused it to be identified as the hit view.

The active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of subevents. In some embodiments, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognizer determination module 173 determines that all views including the physical location of the sub-events are actively involved views, so that all actively-engaged views are specified in the sub-events. It is determined that the sequence should be received. In other embodiments, even if the touch sub-events are entirely limited to the area associated with one particular view, the upper views in the hierarchy will still be maintained as actively participating views.

The event dispatcher module 174 dispatches event information to an event recognizer (eg, event recognizer 180). In embodiments that include an active event recognizer determination module 173, the event dispatcher module 174 delivers event information to the event recognizer determined by the active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores event information in an event queue, which is fetched by individual event receivers 182.

In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In still other embodiments, the event classifier 170 is a standalone module, or is part of another module stored in the memory 102, such as the contact/motion module 130.

In some embodiments, the application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which is a touch event occurring within each view of the application's user interface. Contains instructions for processing them. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a high-level object in which the user interface kit (not shown) or application 136-1 inherits methods and other attributes. to be. In some embodiments, individual event handler 190 may be one of event data 179 received from data updater 176, object updater 177, GUI updater 178, and/or event classifier 170. Includes above. The event handler 190 optionally updates the application internal state 192 by using or calling the data updater 176, the object updater 177, or the GUI updater 178. Alternatively, one or more of the application views 191 includes one or more respective event handlers 190. Also, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included within individual application views 191.

Each event recognizer 180 receives event information (eg, event data 179) from the event classifier 170 and identifies an event from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (optionally including subevent delivery instructions).

The event receiver 182 receives event information from the event classifier 170. The event information includes sub-events, for example, information about touch or touch movement. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event is related to the motion of the touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, events include rotation of the device from one orientation to another (eg, from portrait orientation to landscape orientation, or vice versa), and the event information is the device's current orientation (also referred to as device orientation). Information).

The event comparator 184 compares the event information with predefined event or sub-event definitions, and based on the comparison, determines the event or sub-event, or determines or updates the status of the event or sub-event. In some embodiments, event comparator 184 includes event definitions 186. Event definitions 186 include definitions of events (eg, predefined sequences of sub-events), such as event 1 187-1, event 2 187-2, and the like. In some embodiments, sub-events within event 187 include, for example, touch start, touch end, touch move, touch cancel, and multiple touch. In one example, the definition for Event 1 187-1 is a double tap on the displayed object. The double tap is, for example, a first touch (start of touch) on a displayed object during a predetermined phase, a first lift-off during a predetermined phase (end of a touch), a displayed object during a predetermined phase And a second touch (start of touch) on the top, and a second lift-off (end of touch) during a predetermined phase. In another example, the definition for event 2 (187-2) is dragging on the displayed object. Dragging includes, for example, a touch (or touch) on a displayed object during a predetermined phase, movement of a touch across the touch-sensitive display 112, and lift-off of a touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

In some embodiments, event definition 187 includes a definition of an event for each user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in the application view in which three user interface objects are displayed on the touch-sensitive display 112, when a touch is detected on the touch-sensitive display 112, the event comparator 184 displays three user interface objects A hit test is performed to determine which of these is associated with the touch (sub-event). When each displayed object is associated with an individual event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggers the hit test.

In some embodiments, the definition for each event 187 also includes delayed actions that delay the delivery of event information until it is determined whether the sequence of sub-events corresponds to or does not correspond to the event type of the event recognizer. .

If the individual event recognizer 180 determines that a series of sub-events does not match any of the events in the event definitions 186, the individual event recognizer 180 is unable to event, event failed, or Enter the event end state, after which the individual event recognizers ignore subsequent sub-events of the touch-based gesture. In this situation, if present, other event recognizers that remain active for the hit view continue to track and process sub-events of the touch-based gesture in progress.

In some embodiments, the individual event recognizer 180 includes configurable attributes, flags, and/or lists indicating how the event delivery system should perform sub-event delivery for actively participating event recognizers. Contains metadata 183. In some embodiments, metadata 183 includes configurable attributes, flags, and/or lists indicating how event recognizers interact with each other or become enabled. In some embodiments, metadata 183 includes configurable attributes, flags, and/or lists indicating whether sub-events are delivered to various levels in the view or program hierarchy.

In some embodiments, the individual event recognizer 180 activates the event handler 190 associated with the event when one or more specific sub-events of the event are recognized. In some embodiments, the individual event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is separate from sending (and delaying) sub-events to individual hit views. In some embodiments, event recognizer 180 sends a flag associated with the recognized event, and event handler 190 associated with the flag catches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include subevent delivery instructions that deliver event information about the subevent without activating the event handler. Instead, the subevent delivery instructions deliver event information to event handlers associated with a series of subevents or to actively participating views. Event handlers associated with a series of sub-events or actively participating views receive event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, the data updater 176 updates the phone number used in the contact module 137, or stores a video file used in the video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, the object updater 177 creates a new user interface object, or updates the location of the user interface object. The GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information for display on a touch-sensitive display and sends it to graphics module 132.

In some embodiments, event handler(s) 190 includes or accesses data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included within a single module of individual applications 136-1 or application views 191. In other embodiments, they are included in two or more software modules.

The discussion above regarding event handling of user touches on a touch-sensitive display also applies to other forms of user inputs for operating multifunction devices 100 with input devices, all of which are disclosed on touch screens. It should be understood that it is not. For example, single or multiple keyboard presses or mouse movements and mouse button presses, optionally coordinated with hold; Contact movements, such as tabs, drags, scrolls, etc., on the touchpad; Pen stylus inputs; Movement of the device; Oral instructions; Detected eye movements; Biometric inputs; And/or any combination thereof are optionally used as inputs corresponding to sub-events defining an event to be recognized.

2 illustrates a portable multifunction device 100 with a touch screen 112, according to some embodiments. The touch screen, optionally, displays one or more graphics within user interface (UI) 200. In these embodiments, as well as other embodiments described below, the user may, for example, include one or more fingers 202 (not shown to scale in the figure) or one or more styli 203 (scale to scale) It is possible to select one or more of the graphics by performing a gesture on the graphic using (not shown). In some embodiments, selection of one or more graphics occurs when the user stops contact with one or more graphics. In some embodiments, the gesture is optionally one or more taps of a finger in contact with the device 100, one or more swipes (left to right, right to left, top and/or bottom) Field, and/or rolling (right to left, left to right, top and/or bottom). In some implementations or situations, the graphic is not selected if it is inadvertently contacted. For example, when the gesture corresponding to the selection is a tap, the swipe gesture sweeping over the application icon selectively does not select the corresponding application.

Device 100 also optionally includes one or more physical buttons, such as “home” or menu button 204. As described above, menu button 204 is optionally used to navigate to any application 136 in a set of applications that are selectively executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in the GUI displayed on the touch screen 112.

In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on and off and locking the device, and a volume control button(s) 208 , Subscriber identification module (SIM) card slot 210, headset jack 212, and docking/charging external port 124. The push button 206 optionally turns the device on/off by pressing the button and keeping the button pressed for a predefined time interval; Lock/lock the device by pressing the button and releasing the button before a predefined time interval has elapsed; Used to unlock the device or initiate the unlocking process. In alternative embodiments, the device 100 also accepts verbal input for activation or deactivation of some functions through the microphone 113. The device 100 may also optionally generate tactile outputs for the user of the device 100 and/or one or more contact intensity sensors 165 to detect the intensity of the contacts on the touch screen 112. And one or more tactile output generators 167.

3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface, in accordance with some embodiments. The device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a child learning toy), gaming system, or control device (eg, for home or industrial use) Controller). Device 300 is typically one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses for interconnecting these components ( 320). Communication buses 320 optionally include circuitry (sometimes referred to as chipsets) that interconnect system components and control communication between them. Device 300 includes an input/output (I/O) interface 330 that includes display 340, which is typically a touch screen display. I/O interface 330 may also optionally generate tactile outputs on keyboard and/or mouse (or other pointing device) 350 and touchpad 355, device 300. Generator 357 (eg, similar to the tactile output generator(s) 167 described above with reference to FIG. 1A ), and sensors 359 (eg, optical sensor, acceleration sensor, proximity sensor, touch sensitive Sensors, and/or contact intensity sensors similar to the contact intensity sensor(s) 165 described above with reference to FIG. 1A. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; Optionally, non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU(s) 310. In some embodiments, memory 370 is programs, modules, and data structures similar to programs, modules, and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1A ). , Or a subset of them. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, the memory 370 of the device 300 is, optionally, a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disc authoring module 388 ), and/or the spreadsheet module 390, while the memory 102 of the portable multifunction device 100 (FIG. 1A ), optionally, does not store these modules.

Each of the previously identified elements in FIG. 3 is, optionally, stored in one or more of the memory devices described above. Each of the modules identified above corresponds to a set of instructions for performing the above-described function. The modules or programs identified above (eg, sets of instructions) need not be implemented as separate software programs, procedures or modules, so that various subsets of these modules in various embodiments are optionally combined Or otherwise rearranged. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. In addition, the memory 370 optionally stores additional modules and data structures not described above.

Attention is now directed to embodiments of user interfaces that are selectively implemented on, for example, portable multifunction device 100.

4A shows an example user interface for a menu of applications on portable multifunction device 100, in accordance with some embodiments. Similar user interfaces are optionally implemented on the device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset of them:

셀룰러 및 Wi-Fi 신호들과 같은 무선 통신(들)을 위한 신호 세기 표시자(들)(402);

Signal strength indicator(s) 402 for wireless communication(s), such as cellular and Wi-Fi signals;

시간(404);

Time 404;

블루투스 표시자(405);

Bluetooth indicator 405;

배터리 상태 표시자(406);

Battery status indicator 406;

다음과 같은, 빈번하게 사용되는 애플리케이션들에 대한 아이콘들을 갖는 트레이(408):

Tray 408 with icons for frequently used applications, such as:

o Icon 416 for the phone module 138 labeled "Phone", optionally including an indicator 414 of the number of missed calls or voicemail messages;

o Icon 418 for email client module 140 labeled "Mail", which optionally includes an indicator 410 of the number of unread emails;

o Icon 420 for browser module 147 labeled "Browser"; and

o icon 422 for the video and music player module 152, labeled “iPod”, also referred to as the iPod (trademark of Apple Inc.) module 152; and

다음과 같은, 다른 애플리케이션들에 대한 아이콘들:

Icons for other applications, such as:

o Icon 424 for the IM module 141 labeled "Message";

o icon 426 for calendar module 148 labeled "calendar";

o Icon 428 for image management module 144 labeled "Photo";

o Icon 430 for the camera module 143 labeled "Camera";

o Icon 432 for the online video module 155 labeled "Online Video";

o Icon 434 for stock widget 149-2 labeled "Stock";

o Icon 436 for map module 154 labeled “Map”;

o icon 438 for weather widget 149-1 labeled “Weather”;

o Icon 440 for the alarm clock widget 149-4 labeled "Clock";

o an icon 442 for the exercise support module 142 labeled “exercise support”;

o Icon 444 for the note module 153 labeled "Notes"; and

o Icon 446 for the configuration application or module, labeled "Settings," which provides access to settings for device 100 and its various applications 136.

It should be noted that the icon labels shown in FIG. 4A are exemplary only. For example, the icon 422 for the video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label for an individual application icon includes the name of the application corresponding to the individual application icon. In some embodiments, the label for a particular application icon is separate from the name of the application corresponding to the particular application icon.

FIG. 4B shows a device having a touch-sensitive surface 451 (eg, the tablet or touchpad 355 of FIG. 3) separate from the display 450 (eg, the touch screen display 112) (eg, of FIG. 3 ). An example user interface on device 300 is shown. Device 300 may also, optionally, include one or more contact intensity sensors (eg, one or more of sensors 359) and/or device 300 to detect the intensity of contacts on touch-sensitive surface 451. ). One or more tactile output generators 357 for generating tactile outputs for the user.

Although some of the following examples will be provided with reference to inputs on the touch screen display 112 (a combination of a touch-sensitive surface and a display), in some embodiments, the device is separate from the display as shown in FIG. 4B. Inputs are detected on a personal touch-sensitive surface. In some embodiments, the touch-sensitive surface (eg, 451 in FIG. 4B) has a major axis (eg, 452 in FIG. 4B) corresponding to the main axis (eg, 453 in FIG. 4B) on the display (eg, 450 ). According to these embodiments, the device contacts the touch-sensitive surface 451 at locations corresponding to individual locations on the display (eg, in FIG. 4B, 460 corresponds to 468, 462 corresponds to 470). Detects contacts (eg, 460 and 462 in FIG. 4B). In this way, user inputs (eg, contacts 460, 462 and their movements) detected by the device on a touch-sensitive surface (eg, 451 in FIG. 4B) are such that the touch-sensitive surface is distinct from the display. Is used by the device to manipulate the user interface on the display of the multifunction device (eg, 450 in FIG. 4B). It should be understood that similar methods are, optionally, used for other user interfaces described herein.

Additionally, while the following examples are primarily referenced with finger inputs (eg, finger contacts, finger tap gestures, finger swipe gestures), in some embodiments, one or more of the finger inputs are different input devices It should be understood that it is replaced with input from (eg, mouse-based input or stylus input). For example, the swipe gesture is optionally replaced by a mouse click (eg, instead of a contact) and the movement of the cursor along the path of the subsequent swipe (eg, instead of a movement of the contact). As another example, the tap gesture is optionally replaced with a mouse click while the cursor is positioned over the location of the tap gesture (e.g., instead of stopping detecting the contact following detection of the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are selectively used simultaneously, or mouse and finger contacts are used simultaneously selectively.

5A shows an exemplary personal electronic device 500. Device 500 includes body 502. In some embodiments, device 500 may include some or all of the features described with respect to devices 100 and 300 (eg, FIGS. 1A-4B ). In some embodiments, device 500 has a touch-sensitive display screen 504 (hereinafter, touch screen 504 ). Alternatively or additionally to the touch screen 504, the device 500 has a display and a touch-sensitive surface. Like devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) is, optionally, one for detecting the intensity of applied contacts (eg, touches). It includes an ideal intensity sensor. One or more intensity sensors of the touch screen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touches. The user interface of the device 500 can respond to touches based on the intensity of the touches, which means that touches of different intensities can invoke different user interface actions on the device 500.

Exemplary techniques for detecting and processing touch intensity are, for example, related applications: filed May 8, 2013 and entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application" international patent application PCT/US2013/040061 (published as WIPO Publication No. WO/2013/169849), and filed November 11, 2013, and the name of the invention is "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships", PCT/US2013/069483 (published as WIPO Publication No. WO/2014/105276), each of which is incorporated herein by reference in its entirety.

In some embodiments, device 500 has one or more input mechanisms 506, 508. The input mechanisms 506, 508 (if included) may be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. These attachment mechanisms (if included) include device 500, for example, hat, glasses, earrings, necklace, shirt, jacket, bracelet, watch strap, chain, pants, belt, shoes, wallet , Backpack, etc. These attachment mechanisms allow device 500 to be worn by a user.

5B shows an exemplary personal electronic device 500. In some embodiments, device 500 may include some or all of the components described with respect to FIGS. 1A, 1B, and 3. Device 500 has a bus 512 that operatively couples I/O section 514 with one or more computer processors 516 and memory 518. I/O section 514 can be connected to display 504, which can have a touch-sensitive component 522, and optionally, an intensity sensor 524 (eg, a contact intensity sensor). In addition, the I/O section 514 communicates with the communication unit 530 to receive application and operating system data using Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular, and/or other wireless communication techniques. And can be connected. Device 500 may include input mechanisms 506 and/or 508. The input mechanism 506 is, optionally, a rotatable input device or, for example, a pressable and rotatable input device. In some examples, the input mechanism 508 is, optionally, a button.

In some examples, input mechanism 508 is, optionally, a microphone. Personal electronic device 500 may optionally include GPS sensor 532, accelerometer 534, direction sensor 540 (eg, compass), gyroscope 536, motion sensor 538, and/or their It includes various sensors, such as a combination, all of which can be operatively connected to I/O section 514.

Memory 518 of personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which may include, for example, one or more computer processors ( 516, causes computer processors to include processes 700, 900, 1100, 1300, 1500, and 1700 (FIGS. 7, 9, 11, 13, 15, and 17). It is possible to perform the techniques described below. A computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a temporary computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic, optical, and/or semiconductor storages. Examples of such storage include persistent disks such as flash, solid state drives, etc., as well as magnetic disks, CD, DVD, or optical disks based on Blu-ray technologies. The personal electronic device 500 is not limited to the components and configuration of FIG. 5B and may include other or additional components in multiple configurations.

As used herein, the term “affordance” is a user interaction that is selectively displayed on the display screen of the devices 100, 300, and/or 500 (FIGS. 1A, 3, and 5A, 5B). Refers to a user-interactive graphical user interface object. For example, each of an image (eg, icon), button, and text (eg, hyperlink) optionally constitutes affordance.

As used herein, the term "focus selector" refers to an input element that represents the current portion of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, a touch-sensitive surface (eg, FIG. 3) while the cursor is over a particular user interface element (eg, button, window, slider or other user interface element). When an input (eg, a press input) is detected on the touchpad 355 of FIG. 4 or the touch-sensitive surface 451 of FIG. 4B, the cursor is “focus selector” so that a particular user interface element is adjusted according to the detected input. Functions as Some implementations including a touch screen display that enables direct interaction with user interface elements on a touch screen display (eg, the touch-sensitive display system 112 of FIG. 1A or the touch screen 112 of FIG. 4A ). In the field, when an input (eg, a touch-by-touch input) is detected on a touch screen display at the location of a particular user interface element (eg, a button, window, slider or other user interface element), a specific user interface element is detected The touch detected on the touch screen functions as a "focus selector" so that it is adjusted according to the input input. In some implementations, the focus is one of the user interfaces without moving the corresponding cursor or moving the touch on the touch screen display (eg, by using the tab key or arrow key to move the focus from one button to another). Is moved from one region to another region of the user interface; In these implementations, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the particular form the focus selector has, the focus selector is typically a user to communicate the user's intended interaction with the user interface (eg, by indicating to the device the elements of the user interface the user is trying to interact with). It is a user interface element (or contact on a touch screen display) controlled by. For example, the position of the focus selector (e.g., cursor, contact, or selection box) on an individual button while a press input is detected on a touch-sensitive surface (e.g., touchpad or touch screen) is located on the display of This will indicate that the user is trying to activate an individual button (unlike other user interface elements shown).

As used in the specification and claims, the term “characteristic strength” of a contact refers to the characteristic of the contact based on one or more strengths of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity may optionally be a predefined number of intensity samples, or (eg, after detecting a contact, before detecting a liftoff of the contact, or before or after detecting the start of movement of the contact, A predetermined period of time (eg, before or after detecting the end of the contact, before or after detecting the increase in the intensity of the contact, and/or before or after detecting the decrease in the intensity of the contact) 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds). The characteristic strength of the contact is, optionally, the maximum value of the strengths of the contact, the mean value of the strengths of the contact, the average value of the strengths of the contact, and the top 10 percentile value of the strengths of the contact. ), a value of half of the maximum value of the contact strengths, a value of 90 percent of the maximum value of the contact strengths, and the like. In some embodiments, the duration of the contact is used to determine the characteristic intensity (eg, when the characteristic intensity is the average of the intensity of the contact over time). In some embodiments, characteristic strength is compared to a set of one or more intensity thresholds to determine whether an operation has been performed by a user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, as a result of the contact having a characteristic strength not exceeding the first threshold, a first operation is performed, as a result of the contact having a characteristic strength exceeding the first intensity threshold but not exceeding the second intensity threshold, the second The operation is performed, and as a result of the contact having a characteristic strength exceeding the second threshold, the third operation is performed. In some embodiments, the comparison between characteristic strength and one or more thresholds is performed rather than being used to determine whether to perform the first or second operation (eg, performing individual operations). Is used to determine whether or not to perform or perform individual actions.

5C shows detecting a plurality of contacts 552A through 552E on the touch-sensitive display screen 504 using a plurality of intensity sensors 524A through 524D. 5C further includes intensity diagrams showing current intensity measurements of intensity sensors 524A- 524D for intensity units. In this example, the intensity measurements of the intensity sensors 524A, 524D are nine intensity units, respectively, and the intensity measurements of the intensity sensors 524B, 524C are seven intensity units, respectively. In some implementations, the total intensity is the sum of the intensity measurements of the plurality of intensity sensors 524A-524D, which is 32 intensity units in this example. In some embodiments, each contact is assigned an individual intensity that is part of the total intensity. 5D shows assigning total intensity to contacts 552A-552E based on their respective distance from the center of force 554. In this example, contacts 552A, 552B, and 552E are each assigned intensity of contact of 8 intensity units of total intensity, and contacts 552C, 552D each are intensity of contact of 4 intensity units of total intensity. Is assigned. More generally, in some implementations, each contact j is assigned an individual intensity Ij that is part of the total intensity A according to a predefined mathematical function Ij = A·(Dj/ΣDi). , Where Dj is the distance of the individual contacts j to the center of the force, and ΣDi is the sum of the distances of all the individual contacts to the center of the force (eg i=1 to last). The operations described with reference to FIGS. 5C and 5D may be performed using an electronic device similar or identical to the device 100, 300, or 500. In some embodiments, the characteristic strength of the contact is based on one or more strengths of the contact. In some embodiments, intensity sensors are used to determine a single property intensity (eg, single property intensity of a single contact). It should be noted that the intensity diagrams are not part of the displayed user interface and are included in FIGS. 5C and 5D to aid the reader.

In some embodiments, a portion of the gesture is identified to determine characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from the start position to the end position (increased contact strength at this point). In this example, the characteristic strength of the contact at the end position is selectively based only on a portion of the continuous swipe contact (eg, only on the part of the swipe contact at the end position), not the entire swipe contact. In some embodiments, a smoothing algorithm is optionally applied to the strengths of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm may optionally include an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. It contains one or more. In some situations, these smoothing algorithms eliminate narrow spikes or dips in the strengths of the swipe contact to determine characteristic strength.

The intensity of the contact on the touch-sensitive surface is optionally characterized for one or more intensity thresholds, such as a contact-detection intensity threshold, a lightly pressed intensity threshold, a deeply pressed intensity threshold, and/or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device will perform actions typically associated with clicking a physical mouse button or trackpad. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device will perform different actions than those typically associated with clicking a physical mouse button or trackpad. In some embodiments, when a contact is detected with a characteristic intensity below the light-press intensity threshold (eg, above a nominal touch-detection intensity threshold (contacts are no longer detected below this)), the device is light-press intensity It will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing an action associated with a threshold or deep press intensity threshold. Generally, unless stated otherwise, these intensity thresholds are consistent between different sets of user interface drawings.

The increase in the characteristic intensity of a contact from an intensity below the light-press intensity threshold to an intensity between the light-press intensity threshold and the deep-press intensity threshold is sometimes referred to as the "light-press" input. The increase in the characteristic intensity of a contact from an intensity below the deep pressing intensity threshold to an intensity above the deep pressing intensity threshold is sometimes referred to as a “deep pressing” input. The increase in the characteristic intensity of a contact from an intensity below the contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting contact on the touch surface. A decrease in the characteristic intensity of a contact from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a liftoff of the contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

In some embodiments described herein, the one or more actions detect individual click inputs performed in response to detecting a gesture that includes individual click inputs or with individual contacts (or multiple contacts). In response, an individual click input is detected based at least in part on detecting an increase in intensity of a contact (or multiple contacts) above a press input intensity threshold. In some embodiments, the individual action is performed in response to detecting an increase in the intensity of the individual contact above the click input intensity threshold (eg, “down stroke” of the individual press input). In some embodiments, the press input includes an increase in the intensity of the individual contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the press input intensity threshold, the individual action being an individual below the press input threshold. Is performed in response to detecting a subsequent decrease in the intensity of the contact of (eg, “up stroke” of the individual press input).

5E-5H show the contact 562 from an intensity below the light press intensity threshold of FIG. 5E (eg, “IT _L ”) to an intensity above the deep press intensity threshold of FIG. 5H (eg, “IT _D ”). It shows the detection of a gesture comprising a click input corresponding to an increase in intensity. On the displayed user interface 570 including application icons 572A to 572D displayed in the predefined area 574, while the cursor 576 is displayed over the application icon 572B corresponding to App 2, Gestures performed using contact 562 are detected on touch-sensitive surface 560. In some embodiments, a gesture is detected on touch-sensitive display 504. The intensity sensors detect the intensity of the contacts on the touch-sensitive surface 560. The device determines that the intensity of the contact 562 has reached a peak above a deep pressing intensity threshold (eg, “IT _D ”). Contact 562 is maintained on touch-sensitive surface 560. Reducing scale representations of recently opened documents for App 2 in response to detection of the gesture, and according to a contact 562 having an intensity that exceeds the deep press intensity threshold during the gesture (eg, "IT _D ") 578A to 578C) (eg, thumbnails) are displayed, as shown in FIGS. 5F to 5H. In some embodiments, the intensity compared to one or more intensity thresholds is the characteristic intensity of the contact. It should be noted that the intensity diagram for contact 562 is not part of the displayed user interface and is included in FIGS. 5E-5H to aid the reader.

In some embodiments, the display of representations 578A-578C includes animation. For example, as shown in FIG. 5F, the representation 578A is initially displayed close to the application icon 572B. As the animation progresses, as shown in FIG. 5G, the representation 578A moves up and the representation 578B is displayed proximate to the application icon 572B. 5H, representation 578A moves upward, representation 578B moves upward toward representation 578A, and representation 578C is displayed proximate to application icon 572B. Representations 578A through 578C form an array over icon 572B. In some embodiments, as shown in FIGS. 5F and 5G, the animation proceeds according to the intensity of the contact 562, where the intensity of the contact 562 pushes a deeply pressed intensity threshold (eg, “IT _D ”). The expressions 578A to 578C appear and move upward as they increase toward the direction. In some embodiments, the intensity upon which the animation's progress is based is the characteristic intensity of the contact. The operations described with reference to FIGS. 5E-5H may be performed using an electronic device similar or identical to the device 100, 300, or 500.

In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, sometimes referred to as "jitter," wherein the device has a hysteresis intensity threshold (with a predefined relationship to a press input intensity threshold) For example, the hysteresis intensity threshold is a unit of X intensity that is lower than the click input intensity threshold, or the hysteresis intensity threshold defines or selects 75%, 90% or any suitable ratio of the click input intensity threshold). As such, in some embodiments, the press input includes an increase in the intensity of the individual contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and the individual The operation of is performed in response to detecting a subsequent decrease in the intensity of the individual contact below the hysteresis intensity threshold (eg, “up stroke” of the individual press input). Similarly, in some embodiments, the press input increases the strength of the contact from the intensity below the hysteresis intensity threshold to the intensity above the press input intensity threshold, and, optionally, the intensity below the hysteresis intensity. It is detected only when detecting a subsequent decrease in the intensity of the contact, and individual actions are performed in response to detecting a press input (eg, an increase in the intensity of the contact or a decrease in the intensity of the contact depending on the surrounding environment).

For convenience of description, a description of the actions performed in response to a press input associated with a press input strength threshold or in response to a gesture including the press input may optionally include an increase in contact strength above a press input strength threshold, a hysteresis strength threshold An increase in the intensity of the contact from an intensity less than the press input intensity threshold to an intensity above the press input intensity threshold, a decrease in the intensity of the contact below the press input intensity threshold, and/or a strength of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold Triggered in response to detecting either of the reductions. Further, in examples where the action is described as being performed in response to detecting a decrease in the intensity of the contact below the press input intensity threshold, the action is, optionally, less than a lower hysteresis intensity threshold corresponding to and lower than the press input intensity threshold Is performed in response to detecting a decrease in the intensity of the contact.

Attention is now directed to embodiments of user interface (“UI”) and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

6A-6N illustrate example user interfaces for managing camera effects, in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including those in FIG. 7.

6A illustrates an electronic device 600 with multiple cameras 602 and 603 (eg, on the back of the electronic device 600). In some embodiments, device 600 includes one or more features of devices 100, 300 and/or 500. In some examples, the electronic device (eg, 600) has multiple cameras with fixed but different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of the electronic device (eg, 600). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures multiple focal lengths, producing the same result as multiple cameras with fixed but different focal lengths. In some examples, the electronic device includes a depth camera, such as an infrared camera, thermographic camera, or combination thereof. In some examples, the device further includes a light emitting device (eg, an optical projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate an object during capture of an image by a visible light camera and a depth camera (eg, IR camera), and information from the visible light camera and depth camera is the depth of different parts of the object captured by the visible light camera It is used to determine the map. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for rear facing images and front facing images (eg, selfie) The depth information from the depth camera combined with the image data from the visible light camera for the images) is displayed. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, it is dramatic to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for users. In some embodiments, while displaying the camera user interface with one of the applied lighting effects, the device detects the selection of the camera switching affordance, maintains the display of the user interface controls for applying the lighting effect, and forwards Two visible lights spaced apart from front facing cameras (e.g., depth camera and visible light camera) from the front facing cameras (e.g., depth camera and visible light camera), replacing the display of the field of view of the facing cameras with the view of the rear facing cameras (or vice versa). Cameras) (or vice versa).

As illustrated in FIG. 6B, the electronic device 600 includes a display 604 that is touch-sensitive (eg, a touch screen), and the display displays image data received from the camera 602. In some embodiments, the display is separate from the touch-sensitive surface.

6B further illustrates that the electronic device 600 displays, on the display 604, a camera application user interface 606 for capturing images with cameras 602 and/or 603. The camera application user interface 606 further includes a digital viewfinder 608 that includes a live preview of the field of view of the camera 602. As shown in FIG. 6B, the field of view of the camera includes an object in the foreground (eg, a person) and an object in the background (eg, a fence).

In addition, in FIG. 6B, the camera application user interface includes a filter container 610 represented as a hexagon overlaid on the digital viewfinder 608. In some examples, the filter container (eg, 610) is represented as a circle, triangle, or any other geometric shape. In some examples, the filter container (eg, 610) is an image, icon, or textual representation that provides the user with an indication of the current filter. In the embodiment of FIG. 6B, the filter container (eg, 610) is a transparent representation of an object (eg, hexagon) that allows various filter representations to appear to be moving through the transparent object. In some embodiments, the filter container may be displayed above, below, left, or right of the digital viewfinder (eg, 608) (eg, not overlapping with the viewfinder).

The user interface of FIG. 6B further illustrates the filter picker user interface 612 in a folded (eg, minimized) state. The folded filter picker user interface 612 is positioned along the edge of the digital viewfinder 608. In some examples, the folded filter picker user interface is optionally displayed above, below, left, or right of the digital viewfinder (eg, 608). In some examples, the folded filter picker user interface (eg, 612) includes one or more icons corresponding to a plurality of filter expressions arranged in one or more rows and columns or icons positioned in a circular orientation. In some examples, the folded filter picker user interface (eg, 612) is displayed at any location corresponding to the digital viewfinder. In some examples, the folded filter picker user interface (eg, 612) is marked with an outline (eg, border) to distinguish the filter picker user interface from the digital viewfinder (eg, 608). In some embodiments, the folded filter picker user interface (eg, 612) is translucent (or partially translucent) and has no visible boundaries. As a result, in some examples, the boundaries of the folded filter picker user interface are blended in (eg, indistinguishable) with a digital viewfinder (eg, 608).

As shown in FIG. 6B, folded filter picker user interface 612 includes one or more filter expressions displayed on display 604 (eg, 614A, 614B, 614C, 614D, 614E). In some examples, the folded filter picker user interface (eg, 612) optionally includes filter expressions that are not displayed on the display (eg, they are off screen). Filter expressions that are not displayed within the folded filter picker user interface (eg, 612), optionally, the device detects an input (eg, swipe gesture) that will cause the filter expressions to scroll through the filter container (eg, 610). Is displayed.

As further illustrated in FIG. 6B, the electronic device 600 displays a filter (eg, 614A) in the filter container (eg, 610) to indicate a filter applied to the live preview displayed in the digital viewfinder 608 Displays. In some embodiments, filter representation 614A corresponds to an “natural light” illumination (eg, filter) option. As a result, objects in digital viewfinder 608 (eg, a person) are displayed without any additional lighting effects applied to the image. In some examples, the filter expression (eg, 614A) is a “Studio Light” light effect filter, a “Contour Light” light effect filter, a “Stage Light” light effect filter, or a “Stage Light Mono”. Corresponds to the light effect filter. Each of the light effect filters described above affects the visual characteristics of the image displayed in the digital viewfinder 608 (eg, different sets of different sets of the subject's face in the digital viewfinder based on the depth map of the subject's face) By simulating the effect of illuminating the lights). In some embodiments, the “natural light” lighting option is based on depth information without applying additional lighting effects, for example by blurring the background of the image without blurring at least a portion of the foreground of the image. And modifying the image.

As further illustrated in FIG. 6B, when the filter picker is in the folded state, filter expressions (eg, 614B, 614C, 614D, 614E) that are not displayed in the filter container are displayed in the filter container (eg, 614A) is displayed using a different visual characteristic (eg, shadow). In some examples, different visual properties include color, shape and size. As further illustrated in FIG. 6B, filter expressions (eg, 614B, 614C, 614D, 614E) that are not displayed in the filter container are placed at their position in the list of filter expressions in the folded filter picker user interface (eg, 612 ). It is displayed at various distances (eg, progressively shorter, progressively longer) to each other based on the basis. Thus, in some examples, the farther the filter representation (eg, 614E) in the list is from the filter container (eg, 610), the closer it will be placed on the display for the adjacent filter representation (eg, 614D).

In some examples, in response to receiving an input (eg, a tap) at a location corresponding to one of the representations of the filters (eg, 614B, 614C, 614D, 614E) within the folded filter picker user interface (eg, 612), The electronic device (eg, 600) applies a filter corresponding to a filter expression corresponding to the position of the input. Thus, a user tapping one of the filters not in the filter container will cause the tapped filter expression to scroll for the filter container, and the electronic device 600 corresponds to the filter expression in the filter container (eg, 610). I will apply a filter. In some examples, a swipe input will cause a scroll input, and filter expressions will scroll the filter container. Consequently, if the filter in the filter container has a change, the electronic device will apply the filter that is currently in the filter container (eg, 610).

6C-6E illustrate a user interface showing activation of the extended filter picker user interface. As illustrated in FIG. 6C, electronic device 600 receives tap input 616 at the location of filter vessel 610. In some examples, the tap input spans any area along the edge of digital viewfinder 608. In some examples, the tap input spans any area corresponding to the folded filter picker user interface 612. In some examples, the input corresponds to a swipe gesture, a tap and hold (eg, tap and hold for a predetermined period of time) gesture, or an input having a characteristic strength that exceeds a respective intensity threshold.

As illustrated in FIG. 6D, in response to receiving the tap input 616, the electronic device 600 is an expanded version of the previously folded (eg, minimized) filter picker user (eg, 612) interface of FIG. 6B. Display conversion to filter picker user interface 613. In some embodiments, the filter picker user interface displayed in the folded state of FIG. 6B begins to be converted into an arc (eg, wheel) shaped object of FIG. 6D. In some examples, as the extended filter picker user interface (eg, 613) expands on the display (eg, overlays the digital viewfinder more), it is converted to a wheel or rectangular shape. In some examples, the extended filter picker user interface (eg, 613) extends from the edge of the digital viewfinder (eg, 608) toward the center of the display 604, such that the filter picker user interface (eg, 612) ) Overlaps the digital viewfinder 608. In some examples, the extended filter picker user interface (eg, 613) is translucent, semi-translucent, transparent, or semi-transparent. In some examples, the extended filter picker user interface (eg, 613) is displayed as opaque or semi-opaque. In FIG. 6D, when the expanded filter picker user interface (eg, 613) begins to expand, the filter vessel 610 is shifted upward to correspond to the expanded filter picker user interface (eg, 613 ).

As illustrated in FIG. 6E, when the extended filter picker user interface (eg, 613) is fully extended, the filter vessel 610 has been shifted upwards to the top of the extended filter picker user interface 613. In some embodiments, additional information (eg, 618) for the filter corresponding to the filter expression (eg, 614A) is displayed within the filter container (eg, 610). In some examples, additional information is displayed within the extended filter picker user interface. In some examples, additional information (eg, 618) is displayed at a location near the filter picker user interface (eg, above, below). In some examples, the additional information is expressed as an icon, text, or graphic.

As illustrated in FIG. 6F, the electronic device 600 receives an input (eg, tab 620) at a location corresponding to a representation of one of the filters (eg, 614B) that is not within the filter container. In response to the tap input, in some embodiments, the electronic device applies a filter corresponding to the filter expression (eg, 614B) corresponding to the location on the display 604 of the tap input. In some examples, the input is, optionally, a swipe, press and hold, or an input with a characteristic strength that is above a respective intensity threshold. In some examples, an input having a characteristic strength that is above each intensity threshold on the representation of one of the filters 614A-614F, optionally, is associated with a location of an input having a characteristic intensity that is above the respective intensity threshold This results in the display of additional functions for the filter expression.

6G-6H illustrate the switching of the filter vessel 610 as a result of the electronic device 600 receiving the input (eg, tap 620) in FIG. 6F. As illustrated in FIG. 6G, filter container 610 is displayed to appear to rotate as the filter representation (eg, 614A-614E) scrolls through it. As shown in FIG. 6G, the filter container 610 appears to be a three-dimensional object (eg, cube) during rotation. In some embodiments, when the filter container (eg, 610) is stationary, it is represented as a two-dimensional image (eg, hexagonal). In some examples, the three-dimensional representation of the filter container is, optionally, a sphere or cone. In some examples, the filter container appears to be two-dimensional when filter expressions traverse through it. As further illustrated in FIG. 6G, the electronic device 600 progressively applies a filter corresponding to the filter expression 614B to the live preview displayed in the digital viewfinder as the filter expression is scrolled through the filter container.

As further illustrated in FIG. 6H, the filter container changes visually when a new representation of the filter (eg, 614B) is displayed in the container. In some embodiments, the visual change of the filter container 610 is based on the type of filter in the filter container (eg, the type of visual/lighting effect applied to the live preview in the digital viewfinder). In some examples, different aspects (eg, faces) of the filter container (eg, 610) will reflect the visual (eg, lighting) effect of the filter corresponding to the filter representation 614B. In some examples, the volume (eg, inner portion) of the filter container (eg, 610) will reflect the visual effect of the filter corresponding to the filter expression 614B.

In some examples, filter expressions (eg, icons 614A-614E) correspond to filters that the electronic device 600 simulates by applying various lighting effects corresponding to point light sources. Thus, the visual properties of the filter vessel (eg, 610) will change to simulate the point light source of the corresponding filter. In some examples, the filter displayed in the filter container (eg, 610) corresponds to a simulated studio light visual effect (eg, 614B) simulating a plurality of point light sources. As a result, the filter container is visually modified to simulate a plurality of point light sources corresponding to the studio lighting filter effect. In some examples, a plurality of point light sources are simulated and displayed on 3D object faces during filter selection switching. In some examples, point light sources are displayed on the faces of a two-dimensional object (eg, hexagon) when the corresponding filter is displayed within the filter container (eg, 610). In some examples, the filter representation 614E will be visually distinct (eg, different representation) while it is within the boundaries of the filter container compared to when it is outside the boundaries of the filter container (eg, 610).

As further illustrated in FIGS. 6F-6H, in some embodiments, the electronic device 600 progressively applies a filter corresponding to the newly selected filter expression 614B during the transition to the digital viewfinder. As illustrated in FIG. 6G, the electronic device 600 initially applies a filter corresponding to the filter expression 614B to the digital viewfinder at 50% intensity. The filter corresponding to the filter expression 614B is fully applied by the electronic device in FIG. 6H (eg, 100% intensity). In some examples, the electronic device applies the filter in intensity increments (eg, 10%, 25%, 50%, 75%) during filter switching until the conversion is complete.

As illustrated in FIGS. 6I-6J, the input received at the electronic device (eg, swipe 622) will cause the representation of filters 614A-614E to scroll through the filter container (eg, 610 ). In some embodiments, in response to a swipe gesture (eg, 622), the representations of the displayed filters will scroll left across the top of the filter picker user interface. In some examples, a single swipe gesture will result in an incremental scroll of filter expressions (eg, filters shifted by one). In some embodiments, the number of scrolled filter expressions will depend on the size of the swipe gesture. Thus, in some examples, a longer swipe will result in a longer scroll than a shorter swipe. 6J illustrates the result of the swipe gesture in FIG. 6I, where the “stage light mono” illumination filter was applied to the live preview in the digital viewfinder (eg, in the field of view of the camera based on the depth map of the subject's face) By simulating illuminating a set of lights corresponding to "stage light mono" on the subject's face).

In some examples, the electronic device 600 must continue to detect contact with the filter picker user interface (eg, 613) to maintain the filter picker user interface (eg, 613) in the extended mode. In some examples, when the electronic device detects a liftoff of the contact, the expanded filter picker user interface (eg, 613) may be folded filter picker user interface state (eg, 612) (eg minimized) Will start folding until you reach ). In some examples, the extended filter picker user interface (eg, 613) will remain displayed and displayed in an extended mode until a predetermined period of time is reached without further contact. Upon detecting that the electronic device 600 has reached a predetermined period of time, the extended filter picker user interface until it reaches the folded filter picker user interface state (eg, 612) (eg minimized). Will be displayed as a contraction.

In FIG. 6K, the filter picker user interface is again folded (eg, 612). As illustrated in FIG. 6K, the electronic device inputs (eg, tab 624) at a location that corresponds to where the extended filter picker user interface (eg, 613) would be if it were overlaid on the digital viewfinder. ). As illustrated in FIG. 6L, since the extended filter picker user interface is not displayed, the electronic device interprets the input as a focus command, and the bounding box (eg, 626) corresponds to the location of the tap input (eg, 624). Is displayed at the location. As shown in 6L, processing of the focus command occurs without any change in filter applications or upon presentation of filter expressions.

As illustrated in FIG. 6M, the electronic device swipes in a position corresponding to where the extended filter picker user interface (eg, 613) will be when the extended filter picker user interface is overlaid on the digital viewfinder. Gesture 628 is received. As illustrated in FIG. 6N, since the extended filter picker user interface is not displayed, the electronic device interprets the swipe gesture 628 as a mode change command and the electronic device (eg, 600) selects a camera selection user interface (eg , 606) to different camera modes (eg, “square” camera mode 630, video camera mode, non-portrait camera mode, slow motion camera mode, time lapse camera mode, or panoramic camera mode).

7A-7F are flow diagrams illustrating a method for changing a simulated lighting effect to a representation of image data using an electronic device in accordance with some embodiments. Method 900 is performed on one or more input devices (eg, a touch-sensitive surface, keyboard, mouse), and a device with a display (eg, 100, 300, 500, 600). In some embodiments, the display is a touch-sensitive display. In some embodiments, the display is not a touch-sensitive display. In some embodiments, the electronic device includes a plurality of cameras. In some embodiments, the electronic device has only one camera. Some operations of method 700 are selectively combined, the order of some operations is selectively changed, and some operations are optionally omitted.

As described below, the method 700 provides an intuitive way to change the simulated lighting effect into a representation of image data. This method reduces the user's cognitive burden on providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. For a battery-operated computing device, allowing the user to initiate various functions faster and more efficiently saves power and increases the time between battery charges.

In blocks 702-706, the electronic device (eg, 600) simultaneously displays the camera application user interface (eg, 606) on the display (eg, 604), where the camera application user interface (eg, 606) is On the digital viewfinder (e.g. 608) including a live preview of the field of view of one or more cameras (e.g., 602 and 603) and the digital viewfinder (e.g. 608) Includes a representation of the filter picker user interface (e.g., 610, 612) (e.g., image, icon, text representation indicating respective filter currently applied to the live preview) overlaid on the. In some embodiments, the representation of the filter picker user interface (eg, 610) is in the first location. In some embodiments, the representation of the filter picker user interface (eg, 610) initially starts at the first location and switches to the second location. In some embodiments, the filter picker user interface (e.g., 610) may have different lighting effects on the image based on depth map information (e.g., as described in more detail below with reference to method 900). It is for peaking between filters that apply them. Displaying the digital viewfinder (e.g. 608) and filter picker user interface (e.g. 613) at the same time visualizes the objects in the fields of view and filters of the camera (e.g. 602, 603) available to be applied to the preview. Provide feedback to the user. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life.

In some embodiments, in blocks 708-710, the representation of the filter picker user interface (eg, 612) is the representation of the first filter (eg, 614A), the representation of the second filter (eg, 614B) , And a representation of the third filter (eg, 614C), wherein the values of the visual properties (eg, size, shading, color) of the representation of the first filter are the values of the visual properties of the representation of the second filter and the third filter It is different from the value of the visual characteristics of the expression. In some embodiments, the filter representations are displayed in a filter picker user interface (eg, 612) and different visual characteristics (eg, they are smaller, different from the first filter displayed in a representation of the filter picker user interface (eg, 612)) Shades, different colors). In some embodiments, the second and third filters have the same value of visual characteristics. Visually distinguishing the various representations of the filters provides the user with feedback as to which filter is currently selected, or additionally or alternatively, provides the user with feedback regarding the type of filter effect provided by each filter. do. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 712, the representation of the filter picker user interface (eg, 610) is a representation of a first three-dimensional object (eg, 610) having multiple faces.

In blocks 714-716, while simultaneously displaying a representation of the digital viewfinder (eg, 608) and filter picker user interface (eg, 612 ), the electronic device (eg, 600) may include one or more inputs. A first input (eg, swipe, tap and hold, tap, button press) corresponding to each part of the live preview (eg, on or near the filter indicator icon) via the devices (eg, 616). In some embodiments, the input can be at any location along the entire edge of the digital viewfinder (eg, 608). In some embodiments, at block 718, the first input (eg, 616) is a tap gesture.

In blocks 720-724, in response to detecting the first input (eg, 616), upon determining that the first criteria (eg, filter application criteria) are met—the first criteria are: When 1 input is detected, the filter picker user interface is overlaid on each part of the live preview (e.g., the filter picker user interface (e.g., 613) is farther into the live preview than when in the folded mode of operation. Includes the extended extended mode of operation) requirement-the electronic device (e.g., 600) first (e.g., to indicate that the first filter will be applied to the captured media while the first filter is the currently selected filter). The preview of the first filter is applied to the live preview of the field of view of the camera (eg, 602, 603) that was not applied before the input was detected.

In blocks 720 and 726 to 728, in response to detecting the first input (eg, 616), a filter picker user interface (eg, 613) is displayed on each portion of the live preview when the first input is detected. Upon determination that it is not overlaid on (e.g., the filter picker user interface (e.g., 613 is in a folded mode of operation that does not extend far into the live preview compared to the expanded state)), the electronic device (e.g., 600) is not provided with live preview. 1 The camera application performs each action without applying the filter preview. Performing an action based on meeting the first criteria (applying the filter to the preview or performing each action without applying the filter) means that initiating the capture of the images and/or recording the video Gives the user visual feedback (in the form of updates in the viewfinder) as to whether or not to include the filter. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life. Folding and expanding the filter user interface when access to fewer or more filters is needed eliminates interferences from the viewfinder, providing the user with an interface to easily switch between filters while the user It enables to perform camera operations (eg focus, brightness, camera mode switching) for more viewfinder elements captured in the field of view. Extending the filter user interface when the user is using the filter makes it easier for the user to manipulate the filters, and folding the filter user interface when the user is not using the filter allows the user to view camera actions (eg, focus , Brightness, camera mode switching). Providing additional controls while limiting the interference of related user interface elements improves the operability of the device (e.g., by helping the user achieve the intended result when operating and/or interacting with the device and reducing user errors) And make the user-device interface more efficient, which further reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently.

In some embodiments, at block 730, performing each action is directed to media capture at an object located in the field of view of the camera (eg, 602, 603) at each part of the live preview where the input is detected. Selecting focus (and, for example, to focus on the respective object's representation of one or more objects in the field of view of one or more cameras (eg, 602, 603) (eg, in box 626) Represented by a digital viewfinder (eg, updating the display of 608). Receiving a tap gesture targeting an object to focus provides a user with a precise targeting mechanism to select an object for focus, multiple or extended inputs to change focus until the desired object is placed within focus This avoids the need to provide them, reducing the number of inputs needed to select focus. Reducing the number of inputs needed to select the focus allows the user to achieve the intended result by providing feedback indicating the input that will cause the device to produce the intended result (eg, when operating the device and/or interacting with it). Helping to achieve and reducing user mistakes), improving the operability of the device and making the user-device interface more efficient, which additionally enables the user to use the device faster and more efficiently, Reduces and improves battery life.

In some embodiments, in blocks 732-734, the filter picker user interface (eg, 613) is displayed and overlaid on each portion of the live preview (eg, filter picker user interface (eg, While 613 is displayed in an extended state) and while the first filter is being applied to a live preview of the field of view of the camera (e.g., 602, 603), the electronic device (e.g., 600) filters the filter picker user interface (e.g., A third input (eg, 620) starting at a position corresponding to 613 (eg, swipe or tap) is detected.

In some embodiments, in blocks 736-740, in response to detecting the third input (eg, 620 ), the electronic device (eg, 600) may include a second filter (eg, in a representation of a plurality of filters). , 614B) to the location on the display (e.g., 604) corresponding to the currently selected filter (e.g., different filter expressions are displayed within the expression of the filter picker UI), the electronic device (e.g., 600) A preview of the 2 filters (eg, 614B) is applied to the live preview of the field of view (eg, to indicate that the second filter will be applied to the captured media while the second filter is the currently selected filter). Moving the representation of the selected filter to a location corresponding to the currently selected filter provides the user with visual feedback about the state of the device, including the filter applied (or applied) to the live preview. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 742, the electronic device (eg, 600) provides a tactile output (eg, a tactile output is provided when switching the currently selected filter from one filter to another). .

In some embodiments, in blocks 744 to 746, the filter picker user interface (eg, 613) is displayed and not overlaid on each portion of the live preview (eg, with the filter picker folded) While displayed), the electronic device (eg, 600) detects a second input (eg, 616) starting at a location corresponding to the filter picker user interface (eg, 612). In some examples, input corresponding to any displayed portion of the folded filter picker user interface causes the interface to expand.

In block 748, in response to detecting the second input, the electronic device (eg, 600) expands the filter picker user interface (eg, 613) to overlay each portion of the live preview. Extending the filter picker user interface (eg, 613) allows the user to more accurately target the desired filter by, for example, spreading individual representations of the filters so that less precise inputs can be used to accurately select the desired target. To give. Providing more accurate targeting controls helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at blocks 750-752, while the representation of the filter picker user interface (eg, 610) is associated with the sixth representation of the representations of the plurality of filters, and the filter picker user interface (eg, 610) While the expression of) presents the first side of the plurality of faces, detecting a sixth input (e.g., 620) (e.g., swipe or tap) starting at a location corresponding to the filter picker user interface (e.g., 613). do.

In some embodiments, in blocks 754-760, in response to detecting the sixth input (eg, 620 ), the electronic device (eg, 600) does not display a plurality of undisplayed before detecting the sixth input. Rotate the container object to present the second side of the faces, switch from the first filter, which is the currently selected filter, to the second filter, which is a currently selected filter that is different from the first filter, and preview the second filter's live preview of the field of view Applies to. In some examples, a 3D object (eg, 610) appears to be a 2D object until it starts to animate (eg, rotate). In some embodiments, if the amount of filter is greater than the amount of faces available on a three-dimensional object (eg, 610), a single face can be used to show two or more different filters. As an example, the first side of the cube (eg, 610) displays the first filter, and after four rotations of the cube, the first side will display the fifth filter from the plurality of filters. Rotating the container object to present the currently selected filter provides the user with visual feedback about the state of the device, including that the authorized filter is being changed. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, the filter picker user interface (eg, 613) includes a representation of the filter picker user interface (eg, 610) and representations of a plurality of filters. In some embodiments, the filter picker user interface (eg, 613) overlaps the digital viewfinder (eg, 608). In some examples, the filter picker user interface (eg, 613) is displayed as a wheel, dial, semi-dial, part of the dial, or slider. In some embodiments, the filter picker user interface (eg, 613) is rectangular.

In some embodiments, a filter picker to overlay each portion of the live preview without applying a preview of the filter to the live preview of the field of view of the camera (eg, 602, 603) that was not applied before the second input was detected. Extends the user interface (eg, 613).

In some embodiments, the representation of the filter picker user interface (eg, 610) includes a representation of the second three-dimensional object when in a first state (eg, in operation, expanded) It is a representation of a container object (eg, 610) having multiple faces surrounding the interior volume. In some embodiments, the representation of the filter picker user interface (eg, 610) looks like a 2D object when not in operation (eg, it is a simulated cube positioned to look like a hexagon). While in the first state, representing the container object as having an interior volume containing a 3D object is used to provide feedback on the positioning of sources such as light sources within a virtual 3D environment corresponding to the previewed environment. To give. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, the representation of the filter picker user interface (eg, 610) is in a second state (eg, stop, minimized) different from the first state, and the representation of the container object is shaded and/or It is altered to remove lighting effects (eg, the representation of the container object appears to be flattened into a 2D object, such as a hexagon). Altering the container object's representation to remove shading and/or lighting effects reduces visual distractions to the user and helps avoid diverting the user's attention from the camera's viewfinder. Also, removing effects when they are not needed reduces the number of computing processing needs. Reducing visual distractions and reducing computing processing improves the device's operability and user-device (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device). It makes the interface more efficient, further reducing the device's power usage and improving battery life.

In some embodiments, the representation of the filter picker user interface (eg, 610) is in a first state (eg, in operation, expanded), and one or more of the faces of the plurality of faces or the internal volume is a plurality of filters. It has a visual appearance based on the currently selected filter being expressed. (For example, show the lighting on the sides of the cube to indicate changes in lighting). In some embodiments, different filter representations are displayed within the representation of the filter picker UI when the cube rotates. While in the first state, based on the currently selected filter, the visual appearance of the plurality of faces or the internal volume of the representation of the container object is positioned of sources such as light sources within the visual three-dimensional environment corresponding to the environment being previewed. Provides feedback to the user about the settings and provides feedback to the user about the state of the light sources (enabled, disabled, brightness level). Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

According to some embodiments, the representation of the filter picker user interface (eg, 610) is in a second state (eg, stopped, minimized) different from the first state, and the filter picker user interface (eg, , 610) is not based on the currently selected filter among the plurality of filters displayed in the interior volume. In some embodiments, the effects of the filter are not displayed when the representation for the filter picker UI is minimized or stationary, but are displayed when the filter picker is expanded and displayed as a 3D representation.

In some embodiments, while the filter picker user interface (eg, 613) is displayed and not overlaid on each portion of the live preview (eg, while the filter picker is displayed folded), the filter picker The representations of the user interface (eg, 610) and the representations of the plurality of filters are located along a line substantially parallel to the edge of the display (eg, 604). Positioning the representations of the filter picker user interface (eg, 610) and the representations of the plurality of filters along a line substantially parallel to the edge of the display (eg, 604) is a reminder to the user of the available lighting effects. It increases the visibility of the object in the live preview while providing, and allows the user to more easily perform camera operations (eg, focus, brightness, camera mode switching) on the viewfinder elements. Providing additional controls while maintaining the visibility of related viewfinder elements improves the device's operability (e.g., by helping the user achieve the intended result when operating and/or interacting with the device and reducing user errors). And make the user-device interface more efficient, which further reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently. In some embodiments, additional filter options are located along the edge of the display (eg, 604) extending in both directions away from the representation of the filter picker user interface (eg, 610), and in some embodiments, the additional filter The options are located along the edge of the display (eg, 604) that extends in one direction from the representation of the filter picker user interface (eg, 610). In some embodiments, a representation of the filter picker user interface (eg, 610) is displayed near the edge of the live preview.

In some embodiments, while the filter picker user interface (eg, 613) is overlaid on each portion of the live preview (eg, while the filter picker user interface (eg, 613) is displayed in an expanded state) , A representation of a filter picker user interface (eg, 610) and a representation of a plurality of filters are located along a curve (eg, a line that is not substantially parallel to the edge of the display (eg, 604)). In some embodiments, the shifted position will appear to be an arc or wheel. In some embodiments, the interaction includes using a gesture (swipe or tap) along any portion of the bottom edge of the display (eg, 604). Using the gestures (swipe or tap) along the bottom edge of the display (e.g., 604) to make the shifted position appear as an arc or wheel, and to interact with the arc or wheel provides continuity in the user interface. Reduces confusion, which allows users to provide fewer inputs to perform desired actions. Reducing the number of inputs needed to perform an operation improves the operability of the device (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device) and It makes the user-device interface more efficient, which additionally enables the user to use the device faster and more efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, the first input is a tap gesture, and the location corresponding to each portion of the live preview is a location corresponding to the expression of the first filter in the expressions of the plurality of filters. In some embodiments, the expanded filter picker UI will be displayed in an expanded state. In some embodiments, the filter picker user interface (eg, 613) will be displayed folded.

In some embodiments, the representation of the second filter is in the first direction from the representation of the first filter, the representation of the third filter is in the first direction from the representation of the second filter, and the value of the visible property is the first filter Gradually changes in a first direction (for example, monotonically increases or monotonically decreases) from the expression of to the expression of the second filter through the expression of the second filter (eg, the expressions of the filters, they are the currently selected filter Progressively smaller and/or smaller in opacity as it becomes farther from the expression of). The value of the visual property gradually changing in the first direction from the expression of the first filter to the expression of the second filter to the expression of the third filter reduces user distraction, improves the visibility of the live preview, and is applied to the filters. It provides a structured system that enables easier identification and access to the system. Providing improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device) Makes it more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently. In some embodiments, filter representations are displayed within a filter picker user interface (eg, 613), and each corresponding filter representation displays a different visual characteristic based on the distance from the representation of the filter picker user interface (eg, 610). Is displayed.

In some embodiments, the filter picker user interface (eg, 613) is overlaid on each portion of the live preview (eg, while the filter picker user interface (eg, 613) is displayed in an expanded state), The filter picker user interface (eg, 613) includes additional information around the displayed first filter in relation to the expression of the first filter (eg, displays the name of the filter).

According to some embodiments, the first input is a swipe gesture (eg, 628), and performing each action includes changing the camera capture mode of the electronic device (eg, 600) (eg, the mode is Video, photo, portrait, square, panorama, slow motion, time lapse).

In some embodiments, applying the preview of the second filter includes gradually switching between applying the preview of the first filter and applying the preview of the second filter. In some embodiments, the gradual transition includes 100% first at first time, 0% second, 90% first at second time, 10% second, and the like.

Note that the details of the processes described above with respect to method 700 (eg, FIGS. 7A-7F) are also applicable in a similar manner to the methods described below. For example, the methods 900, 1100, 1300, 1500, 1700, optionally, include one or more of the characteristics of the various methods described above with respect to method 700. For example, elements of the filter user interface, affordances, and controls can be combined among various methods. As another example, the viewfinder of method 700 is similar to the viewfinder of methods 900, 1100, 1300, 1500, 1700. For brevity, these details are not repeated below.

In some embodiments, electronic device 800 includes some or all of the components of device 600, as illustrated in FIG. 6A. In some embodiments, device 800 includes multiple cameras 602 and 603 (eg, on the back of electronic device 800 ). In some embodiments, device 800 includes one or more features of devices 100, 300 and/or 500. In some examples, the electronic device (eg, 800) is fixed but has multiple cameras with different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of an electronic device (eg, 800). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures multiple focal lengths, producing the same result as multiple cameras with fixed but different focal lengths. In some examples, the electronic device (eg, 800) includes a depth camera, such as an infrared camera, thermographic camera, or combination thereof. In some examples, the device further includes a light emitting device (eg, an optical projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate an object during capture of an image by a visible light camera and a depth camera (eg, IR camera), and information from the visible light camera and depth camera is the depth of different parts of the object captured by the visible light camera It is used to determine the map. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for rear facing images and front facing images (eg, selfie) The depth information from the depth camera combined with the image data from the visible light camera for the images) is displayed. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, it is dramatic to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for users. In some embodiments, while displaying the camera user interface with one of the applied lighting effects, the device detects the selection of the camera switching affordance, maintains the display of the user interface controls for applying the lighting effect, and forwards Two visible lights spaced apart from front facing cameras (e.g., depth camera and visible light camera) from the front facing cameras (e.g., depth camera and visible light camera), replacing the display of the field of view of the facing cameras with the view of the rear facing cameras (or vice versa). Cameras) (or vice versa).

As illustrated in FIG. 8A, the electronic device 800 includes a display 804 that is touch-sensitive (eg, a touch screen), and the display displays image data received from a camera (eg, 602). In some embodiments, the display is separate from the touch-sensitive surface. In some examples, multiple cameras (eg, 602 and 603) are located on the front, back, or both sides of an electronic device (eg, 800).

8A further illustrates that the electronic device 800 displays, on the display 804, a camera application user interface 805 for capturing images with a camera (eg, 602 ). The camera application user interface 805 further includes a digital viewfinder 810 that includes a live preview of the field of view of the camera (eg, 602 or 603). In some embodiments, the camera captures depth information associated with image data in real time. 8A further illustrates a camera capturing distinct depth levels in the field of view, including a subject in the foreground area (eg, 808) (eg, a female), and a fence in the background area (eg, 811). In some examples, the camera (eg, 602) captures 3, 4, 5, 10, 20 or more depth levels in the field of view. The electronic device 800 utilizes various depth levels when applying a filter to the representation of the image data (eg, 806) displayed within the digital viewfinder (eg, 810) as discussed in more detail below.

Also, FIG. 8A illustrates that the electronic device 800 displays the filter picker user interface 813 in an expanded state. The filter picker user interface 813 is located along the edge of the digital viewfinder 810. In some examples, the filter picker user interface (eg, 813) is optionally displayed above, below, left, or right of the digital viewfinder (eg, 810). In some examples, the filter picker user interface (eg, 813) includes one or more representations (eg, representations of visual effects) of filters arranged in a circular orientation or arranged in one or more rows and columns.

In some examples, the extended filter picker user interface (eg, 813) is displayed at any location corresponding to the digital viewfinder. In some examples, the filter picker user interface (eg, 813) is delineated with an outline (eg, border) to distinguish the filter picker user interface from the digital viewfinder. In some embodiments, the filter picker user interface (eg, 813) is translucent (or partially translucent) when folded, and has no visible boundaries. As a result, in some examples, the filter picker user interface (eg, 813) appears to be blended in (eg, indistinguishable) from the digital viewfinder.

As shown in FIG. 8A, the electronic device 800 displays a filter picker user interface 813 that includes one or more filter expressions (eg, 814A, 814B, 814C, 814D, 814E) corresponding to visual effects. do. In some examples, the filter picker user interface (eg, 813) optionally includes filter expressions that are not displayed on the display (eg, they are off screen). In some examples, undisplayed filter expressions are displayed when the electronic device receives an input (eg, swipe gesture), which will cause the filter expressions to scroll through the filter container (eg, 816).

As further illustrated in FIG. 8A, in some embodiments, the electronic device 800 displays a filter representation (eg, 814A) within the expanded filter picker user interface (eg, 813) to display the currently selected visual effect. Display. In some embodiments, filter expression 814A corresponds to a “natural light” light effect filter. Consequently, the foreground area 808 and the background area 809 are displayed with a “natural light” light effect filter (eg, using natural light from the scene). Since the image representation of FIG. 8A is shown without using any synthetic light, natural light from the scene produces varying shadows on the object (eg, face, neck, and clothing). In some examples, possible filter expressions corresponding to lighting effects (eg, 814A to 814E) include “studio light” lighting effect, “contour light” lighting effect, “stage light” lighting effect, and “stage light mono” lighting. Contains effects. Each of the aforementioned lighting effects, when applied to the representation of image data (eg, 806), affects the visual characteristics of the representation of the image data displayed on display 804.

In some embodiments, when the electronic device 800 applies a natural lighting effect, no synthetic lighting is added to the image (eg, the original image is displayed). In contrast, studio lighting effects include modeling of multiple individual point light sources (eg, lights in a photography studio) uniformly positioned around the object (eg, creating a brightly filled lighting effect). The contour lighting effect includes modeling multiple individual point light sources located along the perimeter of the object (eg, creating a slimming effect, and creating shadows on the side of the subject's face and/or on the subject's chin) do). Stage light illumination effects include modeling of a single individual point light source located on an object (eg, creating a spotlight effect). Stage light mono lighting effects include modeling in black and white of a single individual point light source positioned over a subject (eg, creating a spotlight effect in black and white).

In some examples, the electronic device (eg, 800) detects the subject's face in the representation of the image data. As a result, the electronic device uses depth map information of the image data and corresponding facial features when applying the lighting effect. As a result, the lighting effect is applied with greater precision around the subject's face, and certain facial features are based on the selected lighting effect (eg, increasing or decreasing shadows around the subject's jaw and/or cheek bones). Can be differently illuminated. In some examples, the image data includes depth map information including depth contours of objects. As a result, the electronic device uses the contour data to apply the lighting effect around the object more accurately.

As illustrated in FIG. 8B, the electronic device 800 receives input (eg, swipe 818) at a location corresponding to the expanded filter picker user interface (eg, 813 ). 8C, the input (eg, swipe 818) causes the filter expressions 814A-814E to scroll through the filter container (eg, 816). In some embodiments, in response to the input (eg, swipe 818), the filter expressions will scroll left across the top border of the filter picker user interface (eg, 813). In some examples, a single swipe gesture will result in an incremental scroll of filter expressions. In some embodiments, the number of scrolled filter expressions will depend on the size of the swipe gesture. Thus, in some examples, a longer swipe will result in a longer scroll than a shorter swipe.

In response to the input (eg, 818 swipe), in some embodiments, the electronic device 800 applies a lighting effect corresponding to the filter representation (eg, 814B) corresponding to the location on the display 804 of the tap input. do. In some examples, the input is a swipe, press and hold, or an input with a characteristic strength that is above the respective intensity threshold. In some examples, an input having a characteristic strength that is greater than a respective intensity threshold detected on the representation of a filter of one of the filters 814A through 814F is, optionally, a location of an input having a characteristic intensity that is greater than a respective intensity threshold This results in the display of additional functionality for the associated corresponding filter expression.

In some embodiments, additional visual effects may be applied to the overall representation of the image data (eg, 806) before applying a lighting effect corresponding to the filter representation (eg, 814B). For example, some gradient fill may be applied to the representation of the image data (eg, 806) before applying the stage light filter to more fluidly transition from no filter to light effect filter.

8C and 8D illustrate an electronic device 800 that gradually applies a lighting effect as a result of the electronic device receiving an input (eg, swipe 818) in FIG. 8B. In some embodiments, the lighting effect corresponding to the newly selected filter representation 814B is gradually applied to the representation of image data (eg, 806) in live preview. Since the selected lighting effect is "Studio Light", the corresponding visual effect simulates multiple point light sources affecting the object in the foreground area 808. As a result, during the transition stage (FIG. 8C), the lighting effect corresponding to the filter expression 814B is applied to the live preview at 50% intensity. The filter corresponding to the filter expression 814B is fully applied in FIG. 8D (eg, 100%). In some examples, the filter is applied in increments (10%, 25%, 50%, 75%) while the electronic device 800 applies the lighting effect until the transition is complete. In some examples, the background area (eg, 809) is completely darkened when the electronic device 800 applies the “Studio Light” light effect to the representation of the image data. In some embodiments, the background area (eg, 809) is partially darkened when the electronic device 800 applies the “Studio Light” light effect to the representation of the image data.

As illustrated in FIGS. 8C and 8D, since the image data captured by the camera (eg, 602) includes depth map information associated with the image data, the electronic device can use the available depth map information and image data The effect of various point light sources on the expression of 806 can be simulated. In some embodiments, the same lighting effect is applied differently in the background area compared to the foreground area based on depth map information associated with the image data. As a result, objects in the foreground may appear more prominently, and objects in the background may become less prominent through a darkening effect. Additionally, as illustrated in FIG. 8D, since lighting effects simulate point light sources, depth map information is used to cast varying shadows on the subject's face in the foreground area (eg, 808 ). As illustrated in FIG. 8D, since the “studio light” lighting effect simulates a plurality of point light sources, the electronic device 800 uses depth map information to reduce shadows on the face of the object compared to the “natural light” lighting effect. use.

8E and 8F, the electronic device 800 receives an input (eg, tab 820) and causes the representations of filters 814A to 814E to scroll the filter container (eg, 816). something to do. In some embodiments, in response to a tap gesture (eg, 820) corresponding to the filter representation (eg, 814E), the representations of the filters are to the left across the upper perimeter of the extended filter picker user interface (eg, 813). Will scroll. 8F illustrates the result of the tap gesture (eg, 820) of FIG. 8E, where the electronic device 800 applies a “stage light mono” filter to the digital viewfinder. The "stage light mono" lighting effect simulates a single point light source, and as a result is similar to the spotlight effect. Using depth map information, the electronic device 800 applies a “stage light mono” effect from above the object in the foreground area (eg, 808). In some examples, the point light source can be simulated to originate from any direction. In some examples, the “stage light mono” effect is simulated to originate from the front, and as a result, a specific focus (eg, face) is emphasized, but the rest of the representation of the image data is darkened. As illustrated in FIG. 8F, since the simulated point light source originates from above the object, the electronic device can cast deeper shadows on the object (eg, face and neck) using depth map information of the image data. have. In some embodiments, the background of the image is removed and replaced with a solid color, such as black or white, or a user-selected color, to attract more attention to the object in the foreground and allow the user to be photographed against a solid backdrop. Can simulate a studio setup.

As illustrated in FIG. 8G, the electronic device may not be in an appropriate state to capture depth information. FIG. 8G illustrates that when the electronic device is operated in FIG. 8F, the user operates the electronic device 800 after taking a few steps from the location where the user was. As a result of the user retreating, the electronic device may no longer capture depth information (eg, conditions for cameras to capture depth information are no longer detected). In some embodiments, when the conditions for capturing the depth effect are no longer met, the previously applied lighting effect fades out gradually. In some examples, when the conditions for capturing the depth effect are no longer met, the previously applied lighting effect is snapped out (eg, the device returns to apply no lighting filter without switching). In some embodiments, when the filter is snapped out, optionally, a temporary filter (eg, gradient) that is part of the light effect filter is applied to the image representation. Temporary filters help smooth transitions when lighting effects are reapplied (eg, less jarring).

In some embodiments, when the electronic device does not detect the conditions necessary to capture the depth map data, the electronic device 800 determines what actions the electronic device (eg, 800) will take to capture the depth map information. A graphic display (eg, 822) is displayed to instruct the user about the. In some examples, the electronic device detects the object, but the object is too far away (eg, the focus is between 2.5 m and 10 m), and the electronic device moves closer to the camera (eg, within 8 feet). Instruct the user to do so (for example, using a graphical display). In some examples, the electronic device determines that the amount of light is too low (eg, less than 400 lux) and instructs the user (eg, using a graphical display) to provide more light. In some examples, affordances are displayed in the viewfinder to allow the user to disable or enable such instructions. In some examples, when conditions for capturing a depth effect map are met, the electronic device 800 suspends displaying a graphical indication instructing the user. Thus, in some examples, the electronic device 800 does not instruct the user when the user action does not help to apply the lighting effect.

As illustrated in FIG. 8H, when the user steps forward and the electronics again detects the necessary conditions for capturing depth map information, the lighting effect is snapped in (eg, without transition). In some examples, the electronic device 800 gradually applies a lighting effect to the representation of the image data when the conditions for capturing depth map information are met again.

As illustrated in FIG. 8I, the electronic device detects an input (eg, tab 824) at a location corresponding to a photo viewer application (eg, 826 ). In response to receiving the input (eg, 826 ), as illustrated in FIG. 8J, the electronic device 800 is configured to view previously captured images rather than an image viewing mode (eg, rather than a live preview of camera data). Mode).

8J illustrates a user interface for a photo viewer application. The photo viewer application includes a thumb strip of previously captured images (eg, 828A to 828D), where 828D is the last captured image. In some examples, previously captured images were captured using a camera corresponding to an electronic device (eg, 800). In some examples, the electronic device (eg, 800) has received previously captured images (eg, 828A to 828D) from a remote source (eg, server), and, optionally, previously captured images are different from the electronic device ( For example, not 800).

8J illustrates that the last captured image (eg, 828D) was captured with a combination of visual effects (eg, the shadow depth of the field where the foreground is in the focal plane because the background is blurred and a portion of the foreground is not blurred) simulated depth to simulate taking a picture with the effect (bokeh (bokeh)) (830), and the lighting effect 832). In some embodiments, the electronic device displays a “vertical” visual indicator (eg, 834) at the top of the display as an indication to the user that the previously captured image data includes depth map information. In some examples, the electronic device (eg, 800) receives input at a location corresponding to a visual indicator to toggle the simulated depth effect (eg, bokeh effect) on and off. In some embodiments, when the simulated depth effect is toggled off, the lighting effect will be maintained. In some examples, the visual indicator will toggle the simulated depth effect and lighting effect together when activated. In some examples, the electronic device optionally receives input to change the lighting effect in the photo viewer application to a different lighting effect using a filter picker user interface (as described above). In some examples, if previously captured image data does not have depth map information; The electronic device (eg, 800) will not provide an option to apply a simulated depth effect or lighting effect. In some examples, if the previously captured image data does not have depth map information associated with the image data, the electronic device will not display a visual indicator (eg, 834).

In some examples, the electronic device (eg, 800) stores depth map information along with image data in one file. In some examples, the electronic device (eg, 800) stores depth map information separately from the image data. In some embodiments, if the electronic device stores an image with depth map information as a flat image (eg, without depth map information), the electronic device (eg, 800) can no longer apply a lighting effect to the representation of the image data. Will not be able to.

9A-9D are flow diagrams illustrating a method for applying simulated lighting effects to a representation of image data using an electronic device in accordance with some embodiments. Method 900 is performed on one or more input devices (eg, a touch-sensitive surface, keyboard, mouse), and a device with a display (eg, 100, 300, 500, 800). In some embodiments, the display is a touch-sensitive display. In some embodiments, the display is not a touch-sensitive display. In some embodiments, the electronic device includes a plurality of cameras. In some embodiments, the electronic device has only one camera. Some operations of method 900 are selectively combined, the order of some operations is selectively changed, and some operations are optionally omitted.

As described below, the method 900 provides an intuitive way to apply simulated lighting effects to the representation of image data. This method reduces the user's cognitive burden on providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. For a battery-operated computing device, allowing the user to initiate various functions faster and more efficiently saves power and increases the time between battery charges.

In some embodiments, at block 902, before displaying a representation of the image data (eg, 806), the electronic device (eg, 800) may retrieve the image data and depth map information associated with the image data (eg, at the device). , From camera, from memory, from server). In some embodiments, the image data includes RGB and depth map values. In some embodiments, the image data and depth map are received from a source external to the electronic device (eg, 800) (eg, data is received from a server). In some embodiments, the image is stored in a file format that allows separation of depth information (eg, depth map) and RGB data in a single file. In some embodiments, the image data includes depth map information. In some embodiments, depth map information and depth map information are separate. Receiving the image data and depth map information corresponding to the image data before displaying the representation of the image data (eg, 806) means that the device is able to view items in the background of the scene (eg, through the representation of image data (eg, 806)). , 809) or in the foreground (eg, 808 ), to provide the user with visual feedback on the content of the depth map information. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

At block 904, the electronic device (eg, 800), on the display (eg, 804), displays a representation of the image data associated with the depth map information. In some embodiments, an image or photo is displayed on the device's display (eg, 804 ). In some embodiments, a live preview of image data is displayed in a digital viewfinder (eg, 810). Displaying a live preview of the image data in the digital viewfinder (e.g., 810) allows the user to quickly and efficiently frame the picture without having to repeatedly capture the picture, the input required to capture the intended picture It reduces the number of people, reduces the memory requirements for storage of pictures, and makes the user interface more efficient. Reducing the number of inputs required to capture the desired image and reducing memory requirements, such as by helping the user achieve the intended result when operating and/or interacting with the device and reducing user mistakes (eg, by reducing user errors) It improves the operability of the device and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the power consumption of the device and improving the battery life.

In some embodiments, at block 906, the electronic device (eg, 800) further includes one or more cameras (eg, 602 and/or 603), and the representation of the image data (eg, 806) is digital It is a live preview of image data captured within the field of view of one or more cameras (eg, 602 and/or 603) displayed in the viewfinder (eg, 810). In some embodiments, the device includes a plurality of cameras (eg, 602 and/or 603) with various focal lengths.

In some embodiments, at block 908, the depth map information associated with the image data includes information corresponding to at least three different depth levels. For example, the image data includes information corresponding to at least the background depth level, the foreground depth level, and the intermediate depth level. Depth map information comprising three or more different depth levels provides a framework for applying depth-specific filters to the depth positioning of objects within the field of view of the camera(s) (eg, 602 and/or 603). Provides more precise feedback to the user. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 910, the depth map information associated with the image data includes information identifying depth contours of the object in the representation of the image data (eg, 806 ).

At block 912, while displaying a representation of the image data (eg, 806) on the display (eg, 804), the electronic device (eg, 800) optionally performs the techniques of blocks 914-936. . At block 914, the electronic device (eg, 800), via one or more input devices, a first input (eg, 818) (eg, swipe, tap and hold, tap, press a button; the gesture is a light filter or Detection) (which may be at the top of the icon representing another user interface system used to select the filter).

In some embodiments, at block 916, the first input (eg, 818) is the input received while the first criteria (eg, a set of light effect application criteria) are met, and the first criteria are subject to Includes requirements that are detected in the field of view within a predetermined distance from the electronic device (eg, 800) (eg, other conditions in the set include a focal length of the first camera (eg, 602 or 603) exceeding a minimum distance threshold, and 1 The focal length of the camera (e.g., 602 or 603) does not exceed the maximum distance threshold, the object is detected beyond a predetermined minimum distance from the device, the amount of light detected exceeds the minimum light threshold, and the amount of light detected This includes not exceeding the maximum light threshold). In some embodiments, if the first criteria are not met, it is withheld to apply the first lighting effect or the second lighting effect. Applying the lighting effect when it is determined that the object is within a predetermined distance when the first input (eg, 818) is received, the object is properly positioned so that an optimal (or nearly optimal) effect can be achieved with a filter. Provides visual feedback to the user. Similarly, not applying a lighting effect when the object is not within a predetermined distance provides feedback to the user that the object is not properly positioned and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 918, the first input (eg, 818) is an input received while the first criteria (eg, a set of light effect application criteria) are not met, and the first criteria are subject Includes requirements that are met when detected in a field of view within a predetermined distance from this electronic device (e.g., 800) (e.g., other conditions/references in the set have a minimum focal length of the first camera (e.g., 602 or 603). Exceeds the distance threshold, the focal length of the first camera (e.g., 602 or 603) does not exceed the maximum distance threshold, the object is detected beyond a predetermined minimum distance from the device, and the amount of light detected is the minimum light threshold Exceeding, and the amount of light detected does not exceed the maximum light threshold). In some embodiments, if the first criteria are not met, the electronic device (eg, 800) suspends applying the first lighting effect or the second lighting effect.

In some embodiments, in block 920, in response to the first input (eg, 818), the electronic device (eg, 800) places filter on the live preview without applying the first lighting effect to the live preview. (Eg, dimming or desaturation of the background (eg, 809)) is applied. In some embodiments, in response to detecting that the first criteria have been met, the electronic device (eg, 800) applies the first lighting effect to the live preview while continuing to apply the placeholder filter to the live preview (eg, The placeholder filter is part of the first lighting effect that does not take into account depth map information and is therefore displayed regardless of whether the first criteria are met). Applying a placeholder filter regardless of whether the first criteria are met or not, for example, which part of the image corresponds to the parts of the depth map in the background (eg 809) compared to the foreground (eg 808) (Eg, 810) Provides visual feedback to the user for the depth map corresponding to the content. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 922, after displaying the live preview without applying the first lighting effect to the live preview, the electronic device (eg, 800) detects that the first criteria have been met. In response to detecting that the first criteria have been met, the electronic device (eg, 800) applies the first lighting effect to the live preview. Applying the lighting effect when the first criteria are met provides the user with visual feedback that the first criteria have been met (eg, the object is properly positioned) and that the optimal (or nearly optimal) effect can be achieved with the filter. do. Similarly, not applying the lighting effect when the first criteria are not met provides feedback to the user that the first criteria are not met, and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

At block 924, upon detecting the first input (eg, 818), the electronic device (eg, 800) has a first lighting effect (eg, natural light, studio light) on the representation of the image data (eg, 806). , Contour light, stage light, stage light mono), and the first lighting effect is based on depth map information (eg, between two images taken simultaneously from different locations or based on measurements of the depth sensor). Based on disparity mapping). Displaying the lighting effect based on the depth map information provides the user with visual feedback on the depth map information. For example, the placement (or enhancement) and removal (or weakening) of the shadows provides the user with feedback on the particular orientation of objects corresponding to the depth map information. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life.

In some embodiments, in block 926, applying the first lighting effect is to a representation of the image data (eg, 806) displayed in the digital viewfinder (eg, 810), a placeholder filter (eg, a background ( For example, dimming or desaturation of 809), wherein the placeholder filter is based on the first lighting effect (eg, selected) and applied regardless of whether the first criteria are met. Applying a placeholder filter creates a smoother/comfortable transition to the lighting filter. Applying a placeholder filter regardless of whether the first criteria are met or not, for example, which part of the image corresponds to the parts of the depth map in the background (eg 809) compared to the foreground (eg 808) (Eg, 810) Provides visual feedback to the user for the depth map corresponding to the content. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, in block 928, applying the first lighting effect is based on depth map information associated with the image data, in a representation of the image data (eg, 806) displayed in the viewfinder (eg, 810). And applying simulation of one or more point light sources in space. Lighting options include natural light, studio light, contour light, stage light, and stage light mono. Each of the lighting effects model (eg, simulate) the results of one or more of the point light sources in space based on the depth map of the image data. The natural lighting option does not apply any synthetic lighting to the image (e.g., the original image is displayed or a portion of the original image is displayed, and blur is applied to different parts of the original effect to simulate the bokeh effect). Applies). Studio lighting effects include modeling of multiple individual point light sources located around an object (eg, creating a brightly filled light effect). The outline lighting effect includes modeling a number of distinct point light sources located at several points around the object to create shadows on the object's face (eg, creating a slimming effect, and on the side of the object's face and/or Or create shadows on the subject's chin). Stage light illumination effects include modeling of a single individual point light source located on an object (eg, creating a spotlight effect). Stage light mono lighting effects include modeling in black and white of a single individual point light source located around an object (eg, creating a spotlight effect in black and white). In some embodiments, the illumination filter simulates point light sources. In some embodiments, the lighting effect is snapped in when the first (eg, lighting effect application criteria) is met, as described in more detail above. In some embodiments, when the system detects a face, facial features are considered when applying a lighting effect. As a result, the lighting effect changes the appearance of the representation of the image data (eg, 806) based on the specific facial features and face shape of the object. Applying a simulation of a point light source provides the user with a visual representation of the content of depth map information, and the device visualizes the shape and depth positioning of objects within the field of view of the camera(s) (eg, 602 and/or 603). Provide feedback to the user. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve. In addition, applying a simulation of a point light source without requiring real physical studio lights makes electronic devices cheaper and smaller than when real studio lights and backdrops are needed, increasing device portability and reducing manufacturing costs. Order.

At block 930, the electronic device (eg, 800), via one or more input devices, may include a second input (eg, swipe, tap and hold, tap; press a button, a gesture is an illumination filter (eg, 814A) or Detection) (which may be at the top of the icon representing another user interface system used to select the filter).

In block 932, upon detecting the second input, the electronic device (eg, 800) has a second lighting effect (eg, natural light, different from the first lighting effect) in the representation of the image data (eg, 806). Apply studio light, contour light, stage light, stage light mono, and the second lighting effect is based on depth map information (e.g., two images taken simultaneously from different locations or based on measurements of the depth sensor) Based on disparity mapping). Displaying the second lighting effect based on the depth map information provides the user with additional visual feedback on the depth map information. For example, the second lighting effect includes one or more light sources at different locations, intensities, or types (directional, ambient, point) that provide the user with feedback about a particular orientation of objects corresponding to depth map information. can do. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life.

In some embodiments, in block 934, applying the second lighting effect is to a representation of the image data (eg, 806) displayed in the digital viewfinder (eg, 810), and to depth map information associated with the image data. Based on applying simulation of one or more point light sources in space. Lighting options include natural light, studio light, contour light, stage light, and stage light mono. Each of the lighting effects model (eg, simulate) the results of one or more of the point light sources in space based on the depth map of the image data. The natural lighting option does not apply any synthetic lighting to the image (eg, the original image is displayed or a portion of the original image is displayed, and blur is applied to different parts of the original effect to simulate the bokeh effect) . Studio lighting effects include modeling of multiple individual point light sources located around an object (eg, creating a brightly filled light effect). The outline lighting effect includes modeling a number of distinct point light sources located at several points around the object to create shadows on the object's face (eg, creating a slimming effect, and on the side of the object's face and/or Or create shadows on the subject's chin). Stage light illumination effects include modeling of a single individual point light source located on an object (eg, creating a spotlight effect). Stage light mono lighting effects include modeling in black and white of a single individual point light source located around an object (eg, creating a spotlight effect in black and white). In some embodiments, the illumination filter simulates point light sources. In some embodiments, the lighting effect is snapped in when the first criteria are met, as described in more detail above. In some embodiments, when the system detects a face, facial features are considered when applying a lighting effect. As a result, the lighting effect changes the appearance of the representation of the image data (eg, 806) based on the specific facial features and face shape of the object. Applying a simulation of a point light source provides the user with a visual representation of the content of depth map information, and the device visualizes the shape and depth positioning of objects within the field of view of the camera(s) (eg, 602 and/or 603). Provide feedback to the user. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 936, the lighting effects change the appearance of the representation of the image data (eg, 806) based on the curvature and the location of the contours of the object. Including depth contours of objects in the depth map information allows the device to provide the user with more precise visual feedback about the shape and depth positioning of objects in the field of view of the camera(s) (eg, 602 or 603). Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve. Also, including depth contours of objects in the depth map information applies simulations of previews and light sources without requiring real physical studio lights to make the electronic device cheaper and smaller than if real studio lights and backdrops were needed. By making it possible, it increases the portability of the device and reduces the manufacturing cost.

In some embodiments, while the first lighting effect (or second filter) is being applied, the electronic device (eg, 800) determines that the first criteria are not met and the first criteria are not met (eg , Applying a first (or second) lighting effect to the representation of the image data (eg, 806) in response to a determination that the electronic device (eg, 800, no object is detected in the field of view within a predetermined distance from the 800) Stop (in some embodiments, the filter's visual effect is reduced, but the filter remains applied), and on the display (eg, 804), display a representation of the image data without applying the first lighting effect ( For example, an image unchanged with or without a partial filter applied, a graphic display (eg, 822) (eg, 822) of unsatisfied first criteria (eg, lighting conditions application criteria) on a display (eg, 804) Text, icon, image). In some embodiments, when the conditions are met again, the filter is reapplied again.

In some embodiments, the representation of the image data is previously captured image data. (Eg, an image retrieved from memory/storage, not a live preview of image data captured by one or more cameras (eg, 602 and/or 603)). In some embodiments, depth map information is stored for stored images, so that lighting effects applied to the image can be changed and/or removed after the image is captured. Storing depth map information for the stored images provides the ability to modify the lighting after the image is captured, reducing the number of pictures the user must take to achieve the desired effect, thereby capturing the intended picture It reduces the number of inputs required to do, reduces memory requirements for storage of pictures, and makes the user interface more efficient. Reducing the number of inputs required to capture the desired image and reducing memory requirements, such as by helping the user achieve the intended result when operating and/or interacting with the device and reducing user mistakes (eg, by reducing user errors) It improves the operability of the device and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the power consumption of the device and improving the battery life.

In some embodiments, while displaying a representation of the image data (eg, 806 ), the electronic device (eg, 800) displays on the display (eg, 804) a visual indication that the image data includes depth map information (eg , 834) (eg, a “portrait mode” badge is optionally displayed to indicate the availability of depth map information).

In some embodiments, while the first lighting effect is applied, the electronic device (eg, 800) maintains at least one value of the previously applied visual effect (eg, bokeh , lighting). Thus, it is possible to achieve light effects and bokeh effects in one representation of image data.

In some embodiments, the previously applied visual effect is a color filter.

In some embodiments, the second input is an input received while the first lighting effect is applied to the representation of the image data (eg, 806), and applying the second lighting effect is the first lighting effect and the second lighting effect Includes gradually switching between applications of (in some embodiments, gradually switching is 100% first, 0% second, 90% first, 10% second, at first time). Etc.). Gradually switching between lighting effects reduces the user distraction generated by blinking the filters on/off, focusing the user on taking the desired picture, thereby reducing the number of inputs required to capture the desired picture. And memory requirements for storage of photos. Reducing the number of inputs required to capture the desired image and reducing memory requirements, such as by helping the user achieve the intended result when operating and/or interacting with the device and reducing user mistakes (eg, by reducing user errors) It improves the operability of the device and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the power consumption of the device and improving the battery life.

Note that the details of the processes described above with respect to method 900 (eg, FIGS. 9A-9D) are also applicable in a similar manner to the methods described below and above. For example, methods 700, 1100, 1300, 1500, 1700, optionally, include one or more of the characteristics of various methods described above and referring to method 900. For example, elements of the filter user interface, affordances, and controls can be combined among various methods. As another example, the viewfinder of method 900 is similar to the viewfinder of methods 900, 1100, 1300, 1500, and 1700. For brevity, these details are not repeated below.

In some embodiments, the electronic device 1000 includes some or all of the components of the device 600, as illustrated in FIG. 6A. In some embodiments, device 1000 includes multiple cameras 602 and 603 (eg, on the back of electronic device 1000 ). In some embodiments, device 1000 includes one or more features of devices 100, 300 and/or 500. In some examples, the electronic device (eg, 1000) has a fixed number of cameras 602 and 603 with different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of the electronic device (eg, 1000). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures multiple focal lengths, producing the same results as multiple (eg, more than one) cameras with fixed but different focal lengths. In some examples, the electronic device (eg, 1000) includes a depth camera, such as an infrared camera, thermographic camera, or combination thereof. In some examples, the device further includes a light emitting device (eg, an optical projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate an object during capture of an image by a visible light camera and a depth camera (eg, IR camera), and information from the visible light camera and depth camera is the depth of different parts of the object captured by the visible light camera It is used to determine the map. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for rear facing images and front facing images (eg, selfie) The depth information from the depth camera combined with the image data from the visible light camera for the images) is displayed. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, it is dramatic to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for users. In some embodiments, while displaying the camera user interface with one of the filters applied, the device detects the selection of the camera switching affordance, maintains the display of user interface controls for applying the filters and front facing camera Replacing the display of their field of view with the field of view of the rear facing cameras (or vice versa), from the front facing cameras (eg depth camera and visible light camera) to the rear facing cameras (eg two visible light cameras spaced from each other) ) (Or vice versa).

As illustrated in FIG. 10A, the electronic device 1000 includes a display 1004 that is touch-sensitive (eg, a touch screen), and the display displays information received from a camera (eg, 602). In some embodiments, the display is separate from the touch-sensitive surface. In some examples, a camera (eg, 602) is located on the front, back, or both sides of an electronic device (eg, 1000).

As shown in FIG. 10A, the electronic device 1000 displays, on the display 1004, a camera application user interface for capturing images with a camera (eg, 602 ). The camera application user interface further includes a representation of image data (eg, 1006) that includes a live preview of the field of view of the camera (eg, 602). In some embodiments, including the embodiment of FIG. 10A, the camera's field of view captures, in real time, depth information associated with image data. 10A captures depth levels in the field of view, including an object in the foreground area (eg, 1008) (eg, female) and an object (eg, female) with trees surrounding the object in the background area (eg, 1010). The camera to be further illustrated. In some embodiments, the representation of image data 1006 consists of a background area (eg, 1008 and 1006) and a foreground area. As shown in FIG. 10A, since the image data includes depth map information, the electronic device 1000 simulates a depth effect (eg, a simulated depth effect) on the representation of the image data (eg, 1006) before any other filters are applied. For example, bokeh) is applied (as illustrated by the representation of objects and trees).

In addition, in FIG. 10A, the electronic device 1000 displays the filter selection interface 1012 under the representation of the image data 1006. In some embodiments, the filter selection interface overlaps the representation of image data. In some embodiments, filter selection interface 1012 is located along the edge of the representation of image data 1006. In some examples, the filter selection interface is displayed by the electronic device 1000 above, below (as shown in FIG. 10A ), left, or right of the representation of image data (eg, 1006 ). As shown in FIG. 10A, the filter selection interface (eg, 1012) displays one or more representations (eg, 1014A to 1014C) (eg, visual effects) of filters arranged in one or more rows and columns or positioned in a circular orientation. Includes.

In some examples, the filter selection interface is outlined (eg, border) to distinguish the filter selection interface from the representation of image data. In some embodiments, the filter selection interface (eg, 1012) is displayed as translucent, semi-translucent, transparent, or semi-transparent, and has no visible boundaries. As a result, in some examples, the filter selection interface (eg, 1012) appears to be blended in (eg, indistinguishable) from the representation of image data (eg, 1006) and display 1004.

As shown in FIG. 10A, the filter selection interface 1012 includes one or more filter expressions (eg, 1014A, 1014B, 1014C, 1014D) displayed on the display 1004 corresponding to visual effects. In some examples, the filter selection interface (eg, 1012) optionally includes filter expressions that are not displayed (eg, they are off screen). Filter expressions other than the displayed filter expressions may optionally be displayed for input (eg, swipe gesture), causing the filter expressions to scroll through the filter selection location of the filter selection interface (eg, 1012).

As illustrated in FIG. 10B, the electronic device 1000 receives an input (eg, tab 1016) corresponding to the location of one of the filter expressions in the filter selection interface 1012. In Fig. 10B, the filter corresponding to the position of the tap input is "Vivid Warm". In response to receiving the input, as illustrated in FIG. 10, the electronic device 1000 applies a filter (eg, “Vivid Warm”) corresponding to the representation of the image data (eg, 1006).

As illustrated in FIG. 10C, using depth map information, the electronic device 1000 places the selected filter (eg, 1014C) in the background area (eg, 1010) different from the foreground area (eg, 1008). Apply. The foreground area 1008 (including females in the front) is modified using a different filter value than the background area 1010 (eg, females and trees). As shown in FIG. 10C, the electronic device 1000 displays a background area using shades (shown darker) than the foreground area.

As further illustrated in Figure 10C, the selected filter (eg, Vivid Warm") includes a color tone protection algorithm. The electronic device 1000 has a predetermined color (or a predetermined range of color tones) when the filter is applied. Depth map information associated with the image is used to minimize color deviations from, for example, the skin color tone of the object in the foreground area 1008 without a color tone protection algorithm from the original color tone after application of the Vivid Warm filter. Potentially extremely biased.To deal with undesired color tone deviation, the electronic device 1000 will apply a filter to the image representation, but make changes to a predetermined color tone (or a predetermined range of color tones). Thus, in some examples, the color tone corresponding to the skin color tone will remain within a certain predetermined range. Conversely, color tones that are not associated with the skin (eg, blue or green) are further refined by the selected filter. Can be modified to a great extent.

10C further illustrates that the color tone protection algorithm applies the algorithm differently to color tones in both the background area and the foreground area. For example, an object in the background area (eg, 1010) appears to have a skin color tone similar to that in the foreground, and applying a “Vivid Warm” filter without any color correction is potentially extreme from the original skin color tone. Will deflect (eg, make the image look unrealistic). To correct color tone deviation, the electronic device 1000 will apply a filter to the image representation, but will limit changes to the predetermined color tone. Thus, in some examples, the color tone corresponding to the skin color tone will remain within a certain predetermined range. However, the allowable range for color tone protection in the background is different from the allowable range for color protection in the foreground (eg, there is less color tone protection for tones in the background because it is not in focus, or even color tone protection) none). Accordingly, as illustrated in FIG. 10C, the skin tone for an object in the background area 1010 will change to a warmer tone than the skin tone for an object in the foreground area 1008.

As illustrated in FIG. 10D, the electronic device detects the tab 1018 at a location 1020 corresponding to the photo viewer application. In response to receiving the tab 1018, the electronic device switches to the image viewing mode as illustrated in FIG. 10E.

10E illustrates the user interface displayed for the photo viewer application. The photo viewer application includes a thumb strip (eg, 1020A to 1020D) of representations of previously captured images, where 1020D is finally captured. The representation of the image data (eg, 1006) corresponding to the last captured image 1020D is displayed on the display 1004. In some examples, previously captured images were captured using a camera corresponding to an electronic device (eg, 1000). In some examples, the electronic device receives previously captured images (eg, 1020A-1020D) from a remote source (eg, server). In some examples, previously captured images were captured with a different electronic device (eg, not 1000). 10E further illustrates that the last captured image does not have any depth map information associated with it (eg, there are no visual indicators representing the depth map, and no visible bokeh effect). .

As illustrated in FIG. 10F, the electronic device detects a tab 1022 at a location 1024 corresponding to a photo viewer editing mode (eg, 1024 ). In response to receiving the tab 1022, the electronic device switches to the image viewing mode as illustrated in FIG. 10G.

10G illustrates that the filter selection interface 1012 is displayed under the representation of image data 1006, as described above with respect to FIG. 10A. In some examples, the filter selection interface (eg, 1012) includes one or more representations (eg, 1014A to 1014C) (eg, visual effects) of filters arranged in a circular orientation or arranged in one or more rows and columns.

As illustrated in FIG. 10H, the device 1000 detects the tab 1026 at the location of the “Vivid Warm” filter expression 1014C. In response to receiving the tab 1026, the electronic device applies the Vivid Warm filter corresponding to the filter representation 1014C for the image representation, and displays the result of FIG. 10I.

As illustrated in FIG. 10I, since the image data does not have depth information associated with it, the electronic device uniformly applies the “Vivid Warm” filter to the background area 1010 and the foreground area 1008. Also, even if the image data has no depth information association with the image, the color tone protection algorithm still applies the algorithm for the color tones in the image (e.g., it applies the color protection algorithm evenly over the entire image) do). For example, the electronic device 1000 applies the color protection algorithm to both skin color tones (eg, foreground and background) of the object using the same value. Thus, as illustrated in FIG. 10C, after the electronic device applies a filter to the reorientation of the image data, in some embodiments, the object (eg, foreground and background) skin color tone is different from that applied to the rest of the image. It is displayed using the same filter value as the filter value.

As illustrated in FIG. 10J, the electronic device 1000 displays a color wheel indicator affordance 1028 overlaid on the representation of image data 1006.

As illustrated in FIG. 10K, device 1000 detects tab 1030 at the location of color wheel indicator affordance 1028. In response to detecting the tab 1030, as illustrated in FIG. 10L, the electronic device displays an extended color wheel over the representation of the image data. The color wheel allows the user to swipe to rotate the wheel and tap any of the colors represented on the wheel to apply the selected color filter to the representation of the image data.

As illustrated in FIG. 10M, device 1000 detects tab 1034 at a location corresponding to a representation of color in the color wheel. In response to receiving the tab 1032, the electronic device 1000 applies the selected color filter to the representation of the image data 1006 as illustrated in FIG. 10N. As shown in FIG. 10N, a visual indicator 1034 is displayed over the selected color filter to indicate the currently selected filter.

11A-11C are flow diagrams illustrating a method for applying a simulated visual effect to a representation of image data using an electronic device in accordance with some embodiments. Method 1100 is performed on a device (eg, 100, 300, 500, 1000) having one or more input devices (eg, a touch-sensitive surface, mouse, keyboard), and a display (eg, 1004) (some implementations) In examples, the device has one camera). In some embodiments, the device has multiple cameras, each camera having a different focal length. Some operations of method 1100 are selectively combined, the order of some operations is selectively changed, and some operations are optionally omitted.

As described below, the method 1100 provides an intuitive way to apply simulated visual effects to the representation of image data. This method reduces the user's cognitive burden on providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. For a battery-operated computing device, allowing the user to initiate various functions faster and more efficiently saves power and increases the time between battery charges.

In some embodiments, in block 1102, the electronic device (eg, 1000) is on the display (eg, 1104): a representation of the image data (eg, 1006) (eg, an image or photo is displayed on the device (eg, , 1104) and the filter selection interface (eg, 1012) simultaneously.

At block 1104, the electronic device (eg, 1000), via one or more input devices, a first image filter (eg, 1014C) (eg, illumination) of a representation of image data (eg, 1006) having a first appearance. Selection of a first image filter (e.g., 1014C) at a location corresponding to (e.g., above or near) a filter, Vivid, Vivid Warm, vivid cool, dramatic, dramatic warm, dramatic cool, mono, silvertone, noir) For example, a first input (eg, 1016) corresponding to swipe, tap and hold, tap, and button press is detected.

In some embodiments, at block 1106, the electronic device (eg, 1000) further includes one or more cameras (eg, 602 and/or 603), and the representation of image data (eg, 1006) is one This is a live preview of the image data captured within the field of view of the above cameras. In some embodiments, the device includes a plurality of cameras having various focal lengths. In some embodiments, image data and depth information are captured with one or more cameras in an electronic device (eg, 1000). In some embodiments, the device includes a plurality of cameras having various focal lengths.

In some embodiments, at block 1108, a first input is detected while displaying a filter selection user interface (eg, 1012).

In some embodiments, at block 1110, the filter selection user interface (eg, 1012) includes a plurality of filter representations (eg, 1014A to 1014D) including a representation of the first image filter (eg, 1014C). And the first input corresponds to the selection of the representation of the first image filter (eg, 1014C). The first filter is displayed as part of a set of one or more filter expressions located within a row along the edge of the display (eg, 1004). In some embodiments, the set of one or more filter expressions (eg, 1014A to 1014D) directly borders the edge of the display (eg, 1004). In some embodiments, filter expressions (eg, 1014A to 1014D) are adjacent to, but not in contact with, the edge of the display (eg, 1004). In some embodiments, the row is positioned along the short edge of the display (eg, 1004). In some embodiments, the row is positioned along the long edge of the display (eg, 1004). Displaying a plurality of filter representations is available such that different selectable filters are applied to the representation of the image data (e.g., 1006) and optionally, depth map information where different filters correspond to the representation of the image data (e.g., 1006). It provides the user with visual feedback that it provides different techniques for visualizing, thereby providing the user with additional information about the shape and positioning of objects in the representation of image data (eg, 1006). Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In block 1112, in response to detecting the first input, the electronic device (eg, 1000) is first to a representation of image data (eg, 1006), optionally including techniques of blocks 1114 to 1126. An image filter (eg, 1014C) is applied.

In block 1114, upon determining that the image data has depth information associated with it-the depth information is the foreground area (eg, 1008) of the representation of the image data (eg, 1006) and the image data (eg, 1006). To be distinguishable from the background region of the representation (eg, 1010) (eg, depth map)-techniques of blocks 1116 to 1120 are optionally performed. In some embodiments, depth information is stored as a separate track of the image file. In some embodiments, depth information is calculated using image data received from at least two cameras with different focal lengths.

In block 1116, the electronic device (eg, 1000) uses the first level adjustment to change the appearance of the foreground area (eg, 1008) of the representation of the image data (eg, 1006) using image data (eg, 1006). ) Applies a first image filter (eg, 1014C) to the foreground area of the representation, and the first level adjustment is such that the first image filter (eg, 1014C) changes the appearance of the representation of the image data (eg, 1006). 1 degree is displayed. In some embodiments, the filter alters one or more of color warmth, unsaturation, color gradient, light intensity, contrast, color shift, luminance.

In some embodiments, in block 1118, applying a first image filter (eg, 1014C) to the foreground area (eg, 1008) means that image data corresponding to the foreground area (eg, 1008) is the first color. Using a first level color value adjustment according to a determination that a value (eg, a hue value, a tone value, one or more colors associated with skin tones) and the image data corresponding to the background includes a first color value Shift the first color value of the background area (e.g., 1010) (e.g., change, modify, replace), and use the second level color value adjustment different from the first level color value adjustment to adjust the foreground color area (e.g., 1008 ) Shifting (eg, changing, modifying, replacing) the first color value of. In some embodiments, the skin color in the background is colored differently from the skin color in the foreground. In some examples, the skin tone color is modified so as not to dramatically change the appearance of the skin color. In some examples, the skin tone is not modified at all. In some embodiments, the filter is applied uniformly to the background even though the area within the background has a color tone corresponding to the skin tone. Different applications of filters include removing constraints on filters from the foreground or background (eg, removing skin protection requirements against the background). Applying different levels of color adjustment to similarly (or equally) colored objects based on whether objects are in the foreground or background, such as indicating that certain objects are in the foreground and different objects are in the background. , Provides visual feedback for the depth information corresponding to the image data to the user, and visual feedback for distinguishing objects in the foreground from similar objects in the background. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In block 1120, the electronic device (eg, 1000) is image data (eg, 1006) with a second level adjustment to change the appearance of the background area (eg, 1010) of the representation of the image data (eg, 1006). The first image filter (eg, 1014C) is applied to the background area (eg, 1010) of the expression of. The second level adjustment indicates the second degree by which the first image filter (eg, 1014C) changes the appearance of the representation of the image data (eg, 1006), wherein the adjustment of the first level and the adjustment of the second level are different Do.

In block 1122, after applying the first image filter (eg, 1014C) to the representation of the image data (eg, 1006), the electronic device (eg, 1000) is on the display (eg, 1004), and the first filter The representation of each image applied to the representation of the image data (eg, 1006) is displayed. In some embodiments, the result is displayed on a display (eg, 1004). In some embodiments, the electronic device (eg, 1000) displays the result in response to receiving a request to apply a first image filter (eg, 1014C) to each image. Applying image filters to the foreground and background with adjustments of different levels, and displaying an updated representation of the image data (eg, 1006) depth information, in particular which objects are identified as being in the foreground and which objects are in the background It provides the user with visual feedback as to whether it is identified as being. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life. In addition, applying an image filter to the foreground and background with adjustments of different levels, and displaying an updated representation of the image data (e.g., 1006) can cause the skin of objects in the viewfinder to be manually corrected otherwise after the user applies the filter. Allows electronic devices to automatically apply filters to create a more dramatic effect without distorting the tone. Applying a filter to the relevant parts of the image and automatically avoiding distorting the skin tone of objects without additional user input improves the operability of the device (e.g., the user achieves the intended result and the user Helping to reduce the number of inputs) makes the user-device interface more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, in block 1124, applying a first image filter (eg, 1014C) to the representation of the image data (eg, 1006) means that the image data is a third color value (eg, tint value, tone Shift the third color value (e.g., using a third level of color value adjustment (e.g., a given color grade for skin tone protection)) according to a determination to include a value, one or more colors associated with skin tones. Additional changes are included. In some examples, the skin color is colored differently from the rest of the image. In some examples, the skin tone color is modified so as not to dramatically change the appearance of the skin color. In some examples, the skin tone is not modified at all. In some examples, different applications of the filter include removing constraints on the filter from the image (eg, removing skin protection requirements against the background). For example, when depth information is not available, the device selectively applies skin protection algorithms when adjusting the colors of the image to the entire image rather than just the foreground of the image. As such, some features in the background of the image close to skin tones protected by skin protection algorithms will not shift in color. Shifting the colors of parts of the image data using a different (e.g., third level color adjustment) technique allows the color of the skin tones to be distinguished more easily from other objects compared to other objects. Provides the user with visual feedback as to which parts of the image do not (or, alternatively, correspond) to a particular color, such as shifting differently. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve. Also, shifting the colors of parts of the image data using different (e.g., third level color adjustment) techniques allows depth information while still avoiding distorting skin tones of objects regardless of whether depth information is available or not. Enables a more dramatic effect on the filter when is available, avoiding the need for the user to manually correct skin tones later. Applying a filter to the relevant parts of the image and automatically avoiding distorting the skin tone of objects without additional user input improves the operability of the device (e.g., the user achieves the intended result and the user Helping to reduce the number of inputs) makes the user-device interface more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, in block 1126, in response to detecting the first input and applying the first image filter (eg, 1014C) to the representation of the image data (eg, 1006), the image data is at its depth. Upon determination that it is not associated with information, the electronic device (e.g., 1000) may include a foreground area (e.g., 1008) of a representation of image data (e.g., 1006) and a background area of a representation of image data (e.g., 1006). A first image filter (eg, 1014C) is uniformly applied to the representation of the image data (eg, 1006) with a first level adjustment (eg, to uniformly change the appearance of 1010). Uniformly applying an image filter to the representation of the image data (eg, 1006) provides the user with visual feedback about the state of the device, and in particular depth information is not available for the image data. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve. Also, uniformly applying an image filter to the representation of image data (e.g., 1006) allows the depth filter to be used when depth information is available while still avoiding distorting skin tones of objects regardless of whether depth information is available or not. By enabling a more dramatic effect, the user avoids the need to manually correct skin tones later. Applying a filter to the relevant parts of the image and automatically avoiding distorting the skin tone of objects without additional user input improves the operability of the device (e.g., the user achieves the intended result and the user Helping to reduce the number of inputs) makes the user-device interface more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, prior to detecting the first input, the electronic device (eg, 1000) is (eg, from the camera, from memory, from the server), the electronic device (eg, 1000), image data (eg, 1006) ) To receive image data represented by the expression. In some embodiments, the image data includes RGB and depth map values. In some embodiments, image data and depth information is received from a source external to the electronic device (eg, 1000) (eg, data is received from a server).

In some embodiments, while the first image filter (eg, 1014C) is applied, the electronic device (eg, 1000) maintains at least one value of a previously applied visual effect (eg, bokeh , lighting). Thus, it is possible to achieve a light effect and a bokeh effect in one representation of image data (eg, 1006). Maintaining a value of at least one of the previously applied visual effects (eg, bokeh , lighting) while the first image filter (eg, 1014C) is applied is a necessity for bulky lenses and equipment to create the bokeh effect Relieves.

In some embodiments, the electronic device (eg, 1000) displays a camera application user interface on a display (eg, 1004), where the camera application user interface is a digital view that includes a live preview of the field of view of one or more cameras. Representation of the color wheel user interface (e.g., 1028) in a first position overlaid on a finder (including live or near live preview images) and a digital viewfinder (e.g., textual representations of images, icons, filters) (Eg, icon, affordance, button). In some embodiments, the representation of the color wheel interface is displayed in response to user input (eg, a tap on the color wheel affordance or a touch and hold gesture detected on the color wheel affordance).

In some embodiments, in response to detecting a second user input (eg, 1030) corresponding to the representation of the color wheel user interface, the electronic device (eg, 1000) is configured to display a color wheel user interface (eg, 1030). Stop displaying the representation (eg, 1004), display a color wheel user interface (eg, 1031) on the display (eg, 1004), and the color wheel user interface (eg, 1031) displays a plurality of color filters. Displays expressions. In some embodiments, the color wheel is displayed as an arc, wheel, partial/complete oval or circle.

In some embodiments, in response to detecting a third user input corresponding to an area corresponding to at least one of the representations of the color filter displayed within the color wheel user interface, the electronic device (eg, 1000) is configured to display image data ( For example, a corresponding color filter is applied to the image data to correct the color appearance of the representation of 1006) (eg, background, foreground or both background and foreground).

For example, methods 700, 900, 1300, 1500, 1700, optionally, include one or more of the characteristics of the various methods described above with respect to method 1100. For example, elements of the filter user interface, affordances, and controls can be combined among various methods. As another example, the viewfinder of method 1100 is similar to the viewfinder of methods 700, 900, 1300, 1500, and 1700. For brevity, these details are not repeated below.

12A, in some embodiments, the electronic device 1200 includes a display 1204 that is touch-sensitive (eg, a touch screen), and the display is information received from a camera (eg, 1202). Displays. In some embodiments, device 1200 includes one or more features of devices 100, 300 and/or 500. In some embodiments, the display is separate from the touch-sensitive surface. In some examples, a camera (eg, 1202) is located on the front, back, or both sides of an electronic device (eg, 1200). In some embodiments, electronic device 1200 includes some or all of the components of device 600, as illustrated in FIG. 6A. In some embodiments, device 1200 includes multiple cameras 602 and 603 (eg, on the back of electronic device 1200 ).

In some examples, the electronic device (eg, 1200) has multiple cameras with fixed but different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of an electronic device (eg, 1200). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 1202) captures image data using a plurality of focal lengths. In some embodiments, a camera (eg, 1202) captures multiple focal lengths at a single time, thus producing the same result as two or more cameras with fixed but different focal lengths.

12A further illustrates that the electronic device 1200 displays a camera application user interface for capturing images with a camera (eg, 1202). The camera application user interface further includes a representation of image data 1210 that includes a live preview of the camera's field of view. In some embodiments, the camera's field of view captures depth information associated with image data in real time. 12A further illustrates that the electronic device has applied a “Dramatic” filter to the representation of image data.

12A further illustrates that the electronic device 1200 displays the filter selection interface 1206 under the representation of the image (eg, 1210). In some embodiments, the filter selection interface (eg, 1206) overlaps the representation of image data (eg, 1210). In some embodiments, a filter selection interface (eg, 1206) is displayed along the edge of the representation of the image data (eg, 1210). In some examples, the filter selection interface is displayed above, below (as shown in FIG. 12A ), left, or right of the representation of the image data (eg, 1210). In some examples, the filter selection interface (eg, 1206) includes one or more representations (eg, 1208A to 1208G) (eg, visual effects) of filters arranged in a circular orientation or arranged in one or more rows and columns.

As shown in FIG. 12A, a filter selection interface (eg, 1206) is displayed and displayed as an outline (eg, border) to distinguish the filter selection interface (eg, 1206) from the representation of image data (eg, 1210). . In some embodiments, the filter selection interface (eg, 1206) is displayed as translucent, semi-translucent, transparent, or semi-transparent, and in some embodiments, has no visible boundaries. As a result, in some examples, the filter selection interface (eg, 1206) appears to be blended in (eg, indistinguishable) from the digital viewfinder.

12A, the electronic device 1202 displays a filter selection interface 1206 that includes filter expressions 1208A through 1208G corresponding to visual effects. In some examples, the filter selection interface (eg, 1206) optionally includes filter expressions that are not displayed on the display (eg, they are off screen). In some embodiments, filter expressions that are not displayed on the display are displayed when the electronic device detects an input (eg, swipe gesture), where the filter expressions are the filter selection location of the filter selection interface (eg, 1006). (E.g., 1212) will scroll.

As further illustrated in FIG. 12A, the electronic device displays filter representations 1208A through 1208G as icons (eg, thumbnails) showing a representation of the image data 1210 in the form of an icon (eg, smaller). Display. For example, a representation of the image data (eg, 1210) displayed in the main portion of the display is shown within the respective filter representations. Further, in some embodiments, the filter expressions include a filter associated with each individual filter expression applied to each individual thumbnail image by the electronic device 1200. Thus, the user can preview the effect of their respective filters on their thumbnail images before activating their respective filters. In some embodiments, the name of the filter (eg, “Dramatic” in FIG. 10A) associated with the filter expression in the filter selection location (eg, 1212) is displayed over the filter selection interface (eg, 1212). In some embodiments, names of filters associated with the filter representations are displayed in filter representations (eg, 1208A-1208G) that overlay the representation of the image. In some embodiments, icons associated with filter expressions (eg, 1208A through 1208G) include the respective name of the filter associated with each filter expression in the filter expression without the representation of the image.

As further illustrated in FIG. 12A, the filter marker 1214 (eg, visual indicator), by the electronic device 1200, specifies a plurality of filter expressions (eg, 1208B) to designate the most recent filter. ). In some embodiments, the most recent filter is at least one filter preselected by the electronic device 1200 based on the filter last used in the currently selected mode (eg, capture mode or post capture edit mode). In some examples, the most recently used filter is a filter predetermined by the electronic device 1200 based on the last used filter regardless of a particular mode (eg, capture or edit). In some embodiments, the visual indicator can be displayed in relation to the filter used most frequently (eg, the number of times used in a predetermined period) by the user. In some embodiments, the visual indicator is displayed in relation to a customized filter (eg, the most preferred filter). In some examples, multiple filters are designated as preferred filters, and multiple visual indicators are displayed over their respective filter expressions. In some examples, the visual indicator is displayed by the electronic device above, below, around, or inside filter expressions. In some embodiments, for live preview, filter expressions include reduced scale live previews of the camera's field of view with a corresponding filter applied to the live preview. In some embodiments, for a previously captured image, filter expressions include reduced scale copies of the image with a corresponding filter applied to the image. In some embodiments, the visual indicator is an image, text, or affordance. In some examples, the plurality of visual indicators are different in color and/or shape based on predetermined criteria.

12A further illustrates the representation 1216 of the contact (eg, touch input) characteristic strength as detected by the device 1200. The representation 1216 includes indicators corresponding to the initial intensity threshold IT ₀ , the medium intensity threshold IT ₁ , and the high intensity threshold IT ₂ . As described below, in some embodiments, the amount of expressions of filters in the filter selection interface will vary based on the strength of the input.

12B illustrates that the electronic device detects a click input 1218 at a location corresponding to the filter expression 1208C. As illustrated in Figure 12B, the push input 1218 is initially The intensity threshold IT has a characteristic intensity above _zero .

12C-12F illustrate the folding effect (eg, compression effect) of some of the filter expressions when the electronic device detects an input having sufficient characteristic strength on one of the filter expressions (eg, 1208A-1208G). As a result of the folding effect, filter selection interface 1206 is displayed with fewer filter expressions than before the folding effect. Fewer filter presentations within the filter selection interface allow the user to traverse filter expressions more quickly. In some examples, the folding effect occurs in response to the electronic device 1200 receiving a tap and hold input (eg, an input that exceeds a predetermined amount of time).

As illustrated in FIG. 12C, device 1200 continues to detect input 1218 at a location corresponding to filter expression 1208C. In addition, FIG. 12C illustrates that the characteristic strength of the input increases beyond the middle intensity threshold IT ₁ . Some of the representations of the filters (e.g., 1208C, 1208E, 1208F, 1208G) appear to fold towards the filter selection location (e.g., 1212), since the input has been interpreted as having a sufficient characteristic strength greater than IT ₁ It appears to be being removed from the display. In some embodiments, some of the filter expressions (eg, 1208C) are folded to the right. In some embodiments, some of the expressions of the filters (eg, 1208E) are folded to the left. In some examples, filter expressions appear to fade out when they are removed from the display. In some examples, instead of folding towards a filter selection location (eg, 1212), filter expressions start sliding the display. In some examples, instead of folding towards the filter selection location 1212, the filter expressions are folded towards the filter expression corresponding to the location of the input (eg, the location of the filter expression 1208C).

12C further illustrates that the filter applied to the representation of the image data 1210 does not change from the filter applied in FIG. 12B, even though the electronic device has begun to fold. In some examples, the filter expression in filter selection location 1212 changes as the filter expressions begin to collapse. In some examples, the filter expression corresponding to the input location will start moving to the selected location (eg, 1212). As the filter at the selected location changes, the filter applied to the representation of the image data 1210 changes to the filter associated with the new filter expression in the filter selected location 1212.

12C further illustrates that the electronic device 1200 generates haptic feedback (eg, a sequence of one or more tactile outputs) 1220 through a tactile output generator during activation of the folding effect. In some embodiments, the electronic device produces no haptic feedback during activation of the folding effect. In some embodiments, haptic feedback is generated to indicate completion of the folding effect.

12D illustrates that the electronic device detects an increase in the characteristic strength of the input 1218. In some embodiments, as the characteristic strength of the push input increases, the folding effect increases, and some of the filter expressions continue to be critical towards the selected location (eg, 1212). 12D further illustrates that the electronic device continues to generate haptic feedback (eg, a sequence of tactile outputs) during the folding effect.

12E illustrates that the electronic device continues to detect an increase in the characteristic strength of the input 1218. In FIG. 12E, some of the filter expressions appear to be almost completely folded (eg, almost invisible). In FIG. 12E, the electronic device no longer generates haptic feedback (eg, a sequence of one or more tactile outputs) because the intensity of the input has almost reached the high intensity threshold IT ₂ . In some embodiments, the electronic device continues to generate haptic feedback (eg, a sequence of one or more tactile outputs).

In some examples, the electronic device (eg, 1200) stops detecting the input (eg, 1218) during the transitional folding effect (eg, the folding effect as represented in FIGS. 12C-12E) and the characteristic strength of the input is If below a certain threshold (e.g., high intensity threshold IT ₂ ), the electronic device performs an expansion effect (e.g., a reverse folding effect), bringing the filter selection interface to its extended state (e.g., as shown in Figure 12A) Will restore. Thus, in some embodiments, when the electronic device detects a lift-off, the electronic device will stop the folding effect, and the user interface will return to a state (eg, extended) as shown in FIG. 12A.

As illustrated in FIG. 12F, the electronic device 1200 detects that the characteristic intensity of the input exceeds the high intensity threshold IT ₂ . As a result, the electronic device has completed the folding effect, and only three filter expressions are displayed in the filter selection interface 1206. In some embodiments, the filter expression in the filter selection location 1212 at the time the initial input (eg, 1218 in FIG. 12B) was received is not associated with the “no filter” option or the last used filter option, so the filter The three filter expressions displayed in the selection interface are the currently selected filter expression (eg, filter expression in filter selection location 1212), the last used filter (eg, 1208B) or preferred filter, and the “no filter” expression (eg , 1208H). In some embodiments, if the filter interface includes a plurality of recently used or preferred filter expressions, the compressed filter interface will include corresponding last used filter expressions, and the filter expression in the compressed filter interface The total amount of them will exceed 3.

In some examples, when a filter expression in filter selection location 1212 is received at electronic device 1200 and associated with a “no filter” expression at the time the initial input (eg, 1218 of FIG. 12B) is received, two Filter expressions are consequently displayed on a reduced filter selection interface (eg, a folded interface). The two filter expressions include the last used filter (eg, 1208B) expression and the “no filter” expression (eg, 1208H).

In some examples, when the filter expression in the filter selection location 1212 is received at the electronic device 1200 at the time the initial input (eg, 1218 of FIG. 12B) is received and is associated with the last used filter expression, 2 The filter expressions of the dogs are displayed on the reduced filter selection interface (eg, folded interface) as a result. The two filter expressions include the last used filter (eg, 1208B) expression and the “no filter” expression (eg, 1208H). Thus, in some embodiments, the device, in its fully collapsed state, a filter expression that satisfies one or more of the following parameters: the currently selected filter expression, the “no filter” expression, or the last used or preferred filter expression. Keep only the field.

12G, while the filter selection interface is in the folded state, the electronic device 1200 corresponds to the filter 1208B most recently used to change the filter representation displayed at the filter selection location 1212. Swipe 1222 is detected at the location. As shown in FIG. 12G, swipe input 1222 is a continuation of input 1218 (eg, occurs without intervening lift off). In some embodiments, although the swipe input 1222 remains on the display, the characteristic strength of the input 1222 drops below the high threshold IT ₂ , but still above the initial intensity threshold IT ₀ . In some embodiments, even if the characteristic intensity of input 1222 is less than the high intensity threshold IT ₂ , the filter selection interface will drop until the electronic device detects a lift off event (eg, the intensity falls below the initial intensity threshold IT _{0 )} . Until it is in compressed (eg, folded) mode. In some embodiments, swipe input 1222 is a separate input, and the filters remain in a folded mode until the user taps or presses again.

12H and 12I illustrate the result of the electronic device receiving the swipe input 1222 of FIG. 12G. 12H illustrates a switched user interface while the electronic device 1200 detects the swipe input 1222. In some embodiments, as the swipe input 1222 moves along the display, the filter representation displayed within the filter selection location 1212 gradually changes. In some examples, as the filter representation at the selected location changes, the filter corresponding to the filter selected location is gradually applied to the representation of the image data (eg, 1210). In some examples, the electronic device outputs tactile feedback when the filter expressions switch to a selected position (eg, 1212).

12I illustrates the user interface after changing filter expressions at a selected location (eg, 1212). As a result, the last used filter (eg 1208B) is now displayed within the selected location. Additionally, a filter corresponding to the last used filter (eg, “Vivid Warm” 1208B) is applied to the representation of the image data 1210 by the electronic device. 12I further illustrates that the electronic device continues to detect input (eg, 1222).

12J illustrates the user interface when the electronic device 1200 stops detecting an input (eg, 1222) in FIG. 12I. In some embodiments, when the user lifts his or her finger from the touch-sensitive display, the electronic device 1200 detects that the characteristic strength of the input falls below the initial strength threshold IT ₀ . Consequently, as illustrated in FIG. 12J, in some embodiments, filter expressions begin to expand visually within filter selection interface 1206 when the electronic device detects a lift off command.

12K illustrates the user interface after the electronic device has completed the extension process of FIG. 12J. Once the filter expressions are expanded, all of the expressions of the filters are displayed in the filter selection interface. In some embodiments, the filter representation displayed in the filter selection position 1212 when the filter selection interface is compressed (eg, as represented in FIG. 12I ), the filter selection position (when all of the filters are fully displayed in FIG. 12K) 1212). In some embodiments, when all of the filter expressions are expanded, some of the filters are kept away from the visible portion of display 1204. In some embodiments, the user can scroll from the display to the display from outside the display by swiping at a location corresponding to the filter selection interface.

As illustrated in FIG. 12L, the electronic device 1200 detects the swipe input 1226 at a location corresponding to the location of the filter selection interface 1206. As a result of the swipe input 1226, the filter representations are scrolled through the filter selection location 1212, as illustrated in FIG. 12M. As shown in Fig. 12L, the previously displayed filter expression "Vivid Warm" is gradually removed from the selected position and a new filter expression "None" is displayed at the selected position. In some embodiments, the filter expression “None” does not correspond to any particular filter, rather it corresponds to no filter applied to the representation of image data 1210. In some examples, as the filter expression changes from “Vivid Warm” to “None”, the “Vivid Warm” filter is gradually removed from being applied to the representation of image data 1210 by the electronic device 1200.

As illustrated in Figure 12n, the electronic device 1200 detects the input with a characteristic intensity exceeding the high threshold IT _2. As a result of detecting input with characteristic strength, some of the filter expressions are folded (eg, compressed) on the user interface so that they are no longer visible. In some embodiments, two filter expressions are displayed and maintained within filter selection location 1212. As shown in Fig. 12N, the electronic device detects the input when the "None" filter expression is displayed within the filter selection position 1212, and the "None" filter expression position is the expression of the filter in which the filter was most recently used ( For example, a filter with a visual indicator on it (in addition to it) is retained within the selection interface when the filter is folded.

As illustrated in FIG. 12O, the electronic device 1200 detects the swipe input 1228 at a location corresponding to the location of the filter selection interface 1206. In some embodiments, as a result of the swipe input 1228, the filter expressions are scrolled through the filter selection location 1212, as illustrated in FIG. 12O. 12O, the previously displayed filter expression "None" is gradually removed from the filter selection position. As shown in Fig. 12o, a new filter expression "Vivid Warm" is displayed at the filter selection position 1212. In some examples, a filter corresponding to the new filter expression “Vivid Warm” is progressively applied to the representation of image data (eg, 1210) when the filter is scrolled into the filter selection location 1212.

In some examples, the folding effect as described above with reference to FIGS. 12A-12P is optionally implemented by the electronic device 1200, within the image viewer application and/or image editing application, and functions similarly to those described above something to do.

13A-13F are flow diagrams illustrating a method for simplifying a user interface using an electronic device, according to some embodiments. Method 1300 includes one or more input devices (eg, a touch-sensitive surface, keyboard, mouse), and a device having a display (eg, a touch-sensitive display) (eg, 100, 300, 500, 1200) ). In some embodiments, the device includes a touch-sensitive display. Some operations of method 1300 are selectively combined, the order of some operations is selectively changed, and some operations are selectively omitted.

As described below, method 1300 provides an intuitive way to simplify the user interface. This method reduces the user's cognitive burden on providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. For a battery-operated computing device, allowing the user to initiate various functions faster and more efficiently saves power and increases the time between battery charges.

At block 1302, the electronic device (eg, 1200), on a display (eg, 1204), includes a filter selection interface (eg, including representations of a plurality of filters (eg, 1208A to 1208G)) within a set of filters. 1206) (eg, the filter selection mode displays all of the filters).

In some embodiments, in block 1304, the representation of the first filter (eg, 1208A) corresponds to the no filter option, and the first subset of representations of the filters is the representation of the first filter (eg, 1208C) and It includes two or more (eg, all) filters among a plurality of filters except for the expression of the fourth filter corresponding to the most recently applied filter (eg, 1208B). Reducing the number of representations of filters displayed in response to a user input corresponding to the selection of the first filter provides the user with feedback that limits the filters to a curated subset based on the type of the first filter, This declutters the display by removing the display of filters that are less likely to be used by the user and making filters more likely to be used by the user more easily accessible with fewer user inputs. Providing improved visual feedback to the user and reducing the number of inputs required to perform an action (eg, assist the user in providing appropriate inputs when operating/interacting with the device and reducing user errors) By improving the operability of the device and making the user-device interface more efficient, which further enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, in block 1306, the representation of the first filter (eg, 1208B) corresponds to the most recently used filter option, and the first subset of representations of the filters is the first filter (eg, 1208B) ) And two or more (eg, all) filters of the plurality of filters except the expression of the fifth filter (eg, 1208A) corresponding to the no filter option. Reducing the number of representations of filters displayed in response to a user input corresponding to the selection of the first filter provides the user with feedback that limits the filters to a curated subset based on the type of the first filter, This declutters the display by eliminating the display of filters that are less likely to be used by the user and making filters more likely to be used by the user more easily accessible with fewer user inputs. Providing improved visual feedback to the user and reducing the number of inputs required to perform an action (eg, assist the user in providing appropriate inputs when operating/interacting with the device and reducing user errors) By improving the operability of the device and making the user-device interface more efficient, which further enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, in block 1308, the representation of the first filter (eg, 1208C) does not correspond to the most recently applied filter and does not correspond to the no filter option, and the first subset of representations of the filters is Multiple filters except for the expression of one filter (e.g. 1208C), the expression of the sixth filter corresponding to the most recently applied filter (e.g. 1208B), and the expression of the seventh filter corresponding to the no filter option (e.g. 1208A) Two or more (eg all) filters. In some embodiments, the second subset of expressions of filters includes filters from the preference list of filters displayed when the second subset is activated. In some embodiments, the preference list of filters is determined based on the most frequently used filters. In some embodiments, the preference list of filters is a customizable list of filters. Reducing the number of representations of filters displayed in response to a user input corresponding to the selection of the first filter provides the user with feedback that limits the filters to a curated subset based on the type of the first filter, This declutters the display by eliminating the display of filters that are less likely to be used by the user and making filters more likely to be used by the user more easily accessible with fewer user inputs. Providing improved visual feedback to the user and reducing the number of inputs required to perform an action (eg, assist the user in providing appropriate inputs when operating/interacting with the device and reducing user errors) By improving the operability of the device and making the user-device interface more efficient, which further enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, in block 1310, the representation of the first filter in the filter selection interface (eg, 1206) is one of the devices to which the first filter (eg, color shift, lighting effect, sharpening, blurring) has been applied. And a live preview (eg, 1210) of the field of view of the above cameras (eg, 1202). Including live previews as part of the representations of the filters provides the user with visual feedback regarding the effects to be generated when the filters are applied, thereby necessitating the user to provide multiple inputs from which to select various filters to receive feedback. Decreases. Providing improved visual feedback to the user and reducing the number of inputs required to perform an action (eg, assist the user in providing appropriate inputs when operating/interacting with the device and reducing user errors) By improving the operability of the device and making the user-device interface more efficient, which further enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, in block 1312, the representation of the second filter in the filter selection interface (eg, 1206) is a live preview of the field of view of one or more cameras (eg, 1202) of the device to which the second filter was applied (eg , 1210). In some embodiments, if the first filter corresponds to the no filter option, applying a visual effect to the image will not change the appearance of the image. Including live previews as part of the representations of the filters provides the user with visual feedback regarding the effects to be generated when the filters are applied, thereby necessitating the user to provide multiple inputs from which to select various filters to receive feedback. Decreases. Providing improved visual feedback to the user and reducing the number of inputs required to perform an action (eg, assist the user in providing appropriate inputs when operating/interacting with the device and reducing user errors) By improving the operability of the device and making the user-device interface more efficient, which further enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1314, the filter selection interface (eg, 1206) displays a visual indicator (eg, 1214) (eg, dot) identifying the representation of the third filter (eg, over the filter). In addition, the third filter is the most recently used filter. In some embodiments, a visual indicator (eg, 1214) represents the most frequently used filter (eg, based on filters applied to images in the device's photo library or camera roll). In some embodiments, representing the most recently used predetermined number, or preferences (eg, preference filters or filters explicitly indicated by the user as the most frequently used filters by the user) filters Multiple indicators are displayed. In some embodiments, the most recently used filter is determined based on the currently selected camera mode (eg, the most recently used filter is the most recently used or currently selected for the picture captured in the currently selected camera mode). This is the most recently used filter when capturing photos in camera mode). In some embodiments, the most recently used filter is based on the most recently used filter in any camera mode, so the most recently used filter is when the device switches from one camera mode to another camera mode. The same is true for. Including a visual indicator that identifies the most recently used filter provides visual feedback for recently applied filtering techniques so that the user rarely has to switch between multiple filters to identify the most recently used filter. Provides to the user, thereby reducing the number of inputs required to identify and activate the most recently used filter. Providing improved visual feedback to the user and reducing the number of inputs required to perform an action (eg, assist the user in providing appropriate inputs when operating/interacting with the device and reducing user errors) By improving the operability of the device and making the user-device interface more efficient, which further enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In block 1316, on a display (eg, 1204), a representation of image data (eg, 1210) (eg, a previously captured picture, a preview based on data received from the first camera, an image received from the server) And while simultaneously displaying the filter selection interface (eg, 1206), the electronic device (eg, 1200), at block 1318, via one or more input devices, the first filter of the set of filters meets the selection criteria. During the first input (e.g., 1228) at a location corresponding to the filter selection interface (e.g., 1206) (e.g., on or near the filter indicator affordance) (e.g., swipe, tap and hold, tap, press hard) (3-D touch), button press).

In some embodiments, at block 1320, the first filter meets selection criteria when a first input (eg, 1218) is detected at a location corresponding to the first filter. The first filter satisfying the selection criteria when the input is at a position corresponding to the first filter (and not satisfying the selection criteria when the input is not at the location corresponding to the first filter) Allows the device to provide a targeted input to reduce the number of displayed filter expressions, thereby reducing the number of inputs required to select the filter. Reducing the number of inputs needed to perform an action is such that a user provides the intended result by providing feedback indicating the input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to achieve and reducing user mistakes), improving the operability of the device and making the user-device interface more efficient, which additionally enables the user to use the device faster and more efficiently, Reduces and improves battery life.

In some embodiments, at block 1322, when the first input (eg, 1218) is detected (eg, when an increase in the intensity of the contact is detected), the first filter is selected from the representation of the first filter ( For example, the selection criteria are met when in 1212 (eg, a position corresponding to a filter currently applied to the representation of image data, a focal position (eg, 1210)). A first filter that meets the selection criteria when the expression of the first filter is selected at the time the input is detected (and does not meet the selection criteria when the expression of the first filter is not selected at the time the input is received) Can allow the user to provide targeting information to cause the device to reduce the number of displayed filter expressions, thereby reducing the number of inputs needed to select the filter. Reducing the number of inputs needed to perform an action is such that a user provides the intended result by providing feedback indicating the input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to achieve and reducing user mistakes), improving the operability of the device and making the user-device interface more efficient, which additionally enables the user to use the device faster and more efficiently, Reduces and improves battery life.

In some embodiments, at block 1324, the first input (eg, 1218) corresponds to a contact on a touch-sensitive display, and at block 1326, the contact exceeds a first intensity threshold (eg, a strong press). It has the characteristic strength of. A device that determines that the characteristic intensity of the first input (eg, 1218) exceeds the first intensity threshold, and in response, stops displaying the first subset of filters while continuing to display the second subset of filters Reduces the number of inputs required by clarifying various inputs received at the same location on the touch-sensitive display and providing the device with the ability to perform appropriate operations. Reducing the number of inputs needed to perform an action is such that a user provides the intended result by providing feedback indicating the input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to achieve and reducing user mistakes), improving the operability of the device and making the user-device interface more efficient, which additionally enables the user to use the device faster and more efficiently, Reduces and improves battery life.

In block 1328, in response to detecting the first input (eg, 1218 ), the electronic device (eg, 1200) optionally performs the techniques of blocks 1330-1338.

At block 1330, the electronic device (eg, 1200) is configured to display (eg, completely/partially remove, shrink from the entire set) of the first subset (eg, at least one) of the representations of the filters among the set of filters. Stop. In block 1332, a first subset of the representations of the filters in the set of filters, in block 1334, the representation of the first filter in the filter selection user interface (eg, the filters to the right of the representation of the first filter). One or more filters in a first direction from one or more representations and in block 1336, from a representation of the first filter (eg, one or more representations of filters to the left of the representation of the first filter) in a second direction It includes one or more filters. Responsive to detecting the first input, reducing the number of displayed filter representations by stopping displaying the first subset of filters and maintaining the display of the second subset of filter representations is a visual of the relevant filter options. It requires fewer user inputs to provide feedback to the user, reduce clutter on the user interface, and navigate between filters to find the desired filter. Providing visual feedback and reducing the number of inputs needed to perform an action (e.g., by providing feedback indicative of inputs that will cause the device to produce the intended result when operating and/or interacting with the device) Improves the device's operability and makes the user-device interface more efficient (by helping to achieve the intended result and reducing user errors), in addition, by enabling the user to use the device more quickly and efficiently. It reduces the device's power consumption and improves battery life.

At block 1338, the electronic device (eg, 1200) maintains a display of the second subset of representations of the filters in the set of filters, and the second subset of representations of the filters includes at least the representation of the first filter. . In some embodiments, a subset of the initial set of filters is maintained on the screen. In some embodiments, maintaining filters on the screen involves shifting the positions of the filters while maintaining the filter on the screen.

In some embodiments, at block 1340, the first subset of representations of the filters includes two or more (eg, all) filters of the plurality of filters except filters at blocks 1342 to 1346. . In block 1342, the first subset of expressions of the filters is two or more (eg, all) filters of the plurality of filters except the expression of the first filter and the expression of the fourth filter corresponding to the most recently applied filter. Includes

In some embodiments, at block 1344, the first subset of representations of the filters is two or more of the plurality of filters (eg, the representation of the first filter and the representation of the fifth filter corresponding to the no filter option). , All) filters.

In some embodiments, at block 1346, the first subset of representations of the filters is the representation of the first filter, the representation of the sixth filter corresponding to the most recently applied filter, and the seventh filter corresponding to the no filter option It includes two or more (eg, all) filters among a plurality of filters except for the expression of.

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and the first input (eg, 1218) corresponds to a first contact on the touch-sensitive display, and at block 1338, a first While continuing to detect the first contact on the display (eg, 1204) after detecting one input (eg, 1218), the techniques of blocks 1350 and 1352 are performed. In block 1350, the electronic device (eg, 1200) detects the movement of the first contact. At block 1352, the electronic device (eg, 1200) displays on the display (eg, 1204) a representation of image data with applied visual effects corresponding to the second filter (eg, color shift, lighting effect, sharpening, Blurring) (and, for example, on the display (eg, 1204), stop displaying a representation of the image data with the applied visual effect corresponding to the first filter for the image data). Updating the representation of the image data to reflect the second filter in response to detecting the movement of the contact provides feedback to the user about the state of the device, in particular visual feedback corresponding to the activated filter. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life.

In some embodiments, in response to detecting the movement of the first contact, the electronic device (eg, 1200) stops displaying the representation of the first filter at the second selected location (eg, 1212), and the second The representation of the second filter is displayed at the selected location (eg, 1212). (Eg, moving the representation of the second filter to the “currently selected” filter position on the display (eg, 1204), and removing the representation of the first filter from the “currently selected” filter position on the display (eg, 1204) that)

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and at block 1354, the first input (eg, 1218) is a characteristic strength of the contact on the touch-sensitive display (eg, 1216) ), in response to detecting an increase in the characteristic strength of the contact, the electronic device (eg, 1200) dynamically shifts the position on the display (eg, 1204) of the first filter representation toward focus. do. In some embodiments, as the position of the first filter representations moves towards the focus, the representations of other filters that are not within the second subset of filters' expressions fade out and/or decrease as the intensity of the contact increases. . In some embodiments, representations of other filters that are not within the second subset of representations of filters are horizontal in one or more dimensions (eg, as representations of filters in the second subset of representations of filters move toward focus) Shrinking (by stretching or compressing in the direction). Dynamically shifting the position of the first filter representation towards the focus as the user increases the characteristic strength of the contact provides the user with feedback regarding the level of intensity being detected by the device based on the user's input, Pressing harder provides the user with visual feedback indicating that the device will perform an action associated with the user interface element. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first input (eg, 1218) corresponds to a contact on a touch-sensitive display having a first contact portion and a second contact portion; The first contact portion has a first characteristic strength; The second contact portion has a second characteristic strength; The method determines whether the first characteristic intensity is above the second intensity threshold (eg, the folding threshold at which compression of the filter selection interface (eg, 1206) begins) and is below the third intensity threshold, on the display (eg, 1204). To, gradually shifting the position of the first filter expression (eg, the filter expressions move toward the focus); And upon determining that the second characteristic intensity is above a third intensity threshold (eg, a folding threshold at which compression of the filter selection interface (eg, 1206) ends (eg, the point at which the first subset is stopped being displayed), And stopping shifting the position of the first filter expression.

In some embodiments, at block 1356, the electronic device (eg, 1200) detects a decrease in the characteristic strength of the contact while continuing to detect the contact on the touch-sensitive display. In some embodiments, in block 1358, in response to detecting a decrease in the characteristic strength of the contact, blocks 1360-1362 are selectively performed.

In some embodiments, at block 1360, the respective strength of the contact before detecting a decrease in the characteristic strength of the contact (eg, a threshold strength to stop displaying the first subset of filters). Upon determining that a threshold (eg, 1216) has been reached, the electronic device (eg, 1200) maintains the display of the second subset of representations of the filters without displaying the first subset of representations of the filters.

In some embodiments, at block 1362, the respective strength of the contact before detecting a decrease in the characteristic strength of the contact (eg, a threshold strength to stop displaying the first subset of filters). Upon determining that the threshold (eg, 1216) has not been reached, the electronic device (eg, 1200) dynamically shifts the position on the display (eg, 1204) of the first filter representation away from the focus. In some embodiments, their size increases as the position of the first filter representations moves away from the focus.

Maintaining the display of the second subset of filters or dynamically shifting the position of the first filter as the user decreases the characteristic strength of the contact is such that the user's input is at a respective intensity threshold (eg, 1216). Visual feedback that provides the user with feedback about the level of intensity detected by the device based on what has been reached, and indicates whether the user needs to press more strongly to cause the device to perform an action associated with the user interface element. To the user. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, before displaying a representation of the image data, the electronic device (eg, 1200) receives image data (eg, from the camera, from memory, from the server), where the image data is image data (eg, , 1210). In some embodiments, the image data includes RGB and depth map values. In some embodiments, image data and depth information is received from a source external to the electronic device (eg, 1200) (eg, data is received from a server).

In some embodiments, in addition to detecting the movement of the contact, the electronic device (eg, 1200) provides a tactile output indicating that the filter applied to the image data has changed. In some examples, the electronic device (eg, 1200) provides a tactile output when switching between different filters. Performing a tactile output when switching between different filters provides the user with additional feedback regarding changes to the state of the selected filter and helps the user perform operations more efficiently. Providing improved feedback improves the device's operability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device). And additionally, reduce device power usage and improve battery life.

In some embodiments, before detecting the first input (eg, 1218), while the third filter is in the second selected position (eg, 1212), and while displaying the first subset of expressions, the electronic device (Eg, 1200) detects the movement of the second contact on the touch-sensitive display. In response to detecting the movement of the second contact, the electronic device (eg, 1200) stops displaying the representation of the third filter at the second selected position (eg, 1212), and the second selected position (eg, 1212) displays a representation of the fourth filter, applies a visual effect (eg, color shift, lighting effect, sharpening, blurring) corresponding to the fourth filter to the image data, and displays it on the display (eg, 1204). , Displays a representation of image data with the applied visual effect, and the first contact is the second contact. Switching between filters in response to detecting the movement of the second contact allows the user to easily switch between filters (rather than preferred or recent filters) without requiring a strong push, thereby selecting the desired filter. Improve your ability to target accurately. Providing improved targeting capabilities improves the device's operability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device) This, in addition, reduces the device's power usage and improves battery life.

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and the first input (eg, 1218) corresponds to a contact on the touch-sensitive display. While maintaining the display of the second subset of representations of the filters without displaying the first subset of representations of the filters, the electronic device (eg, 1200) detects a liftoff of the contact. In response to detecting the liftoff of the contact, the electronic device (eg, 1200) restores displaying the first subset of the representations of the filters. Restoring the display of the first subset of representations of the filters in response to detecting the liftoff of the contact reduces the number of user inputs required to access the first subset of representations of the filters. Reducing the number of inputs required to perform an operation improves the device's operability and improves the user's operability (e.g., by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device) -Make the device interface more efficient, additionally, reduce the device's power usage and improve battery life.

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display. While not displaying the first subset of representations of the filters and maintaining the display of the second subset of representations of the filters, the electronic device (eg, 1200) detects an input that includes an increase in the intensity of the contact. In response to detecting the input, a determination is made that the input includes an increase in the characteristic strength of the contact from a characteristic strength below the fourth intensity threshold to a characteristic intensity above the fourth intensity threshold contact having a property strength above the fourth intensity threshold. Accordingly, the electronic device (eg, 1200) restores the display of the first subset of the representations of the filters. In response to detecting the input, it is determined that the input does not include an increase in the characteristic strength of the contact from a characteristic strength below the fourth intensity threshold to a characteristic intensity above the fourth intensity threshold contact having a property strength above the fourth intensity threshold Accordingly, the electronic device (eg, 1200) maintains the display of the second subset of representations of the filters without displaying the first subset of representations of the filters. Restore the display of the first subset of representations of the filters when the characteristic intensity of the input increases above the threshold, and display the second subset of the representations of the filters when the characteristic intensity of the input does not increase beyond the threshold Retaining gives the user a preview of the actions (when the user presses a little) and locks the action when the users press more strongly, so that users do not need to maintain a greater input characteristic strength while switching between filters, This improves the operability of the device and makes the user-device interface more efficient.

In some embodiments, stopping displaying the first subset of representations of the filters (eg, completely/partially removing, reducing from the entire set) means that the representation of the first filter is the device to which the first filter was applied. Progressively reducing the size of the representation of the first filter while continuously displaying at least a portion of the live preview of the field of view of one or more cameras (eg, 1210); And gradually reducing the size of the representation of the second filter while the representation of the second filter continues to display at least a portion of a live preview (eg, 1210) of the field of view of one or more cameras of the device to which the second filter is applied. Includes.

In some embodiments, in response to detecting the movement of the second contact, the fourth filter is determined to be the first type of filter (eg, a predetermined preference, most frequently used, most used). Accordingly, the electronic device (eg, 1200) generates its own tactile output, and the fourth filter is a second type of filter (eg, a predetermined preference, most frequently used, other than the most used) The electronic device (e.g., 1200) withholds generating its own tactile output (e.g., generating any tactile output or generating tactile output with a different tactile output pattern). Withhold). In some embodiments, a tactile output is provided whenever a new filter is displayed within the filter selection interface (eg, 1206). Performing a tactile output when switching between different filters (and performing different tactile outputs based on filters) provides the user with additional feedback regarding changes to the state of the selected filter, Helps the user perform actions more efficiently. Providing improved feedback improves the device's operability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device). And additionally, reduce device power usage and improve battery life.

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and the first input (eg, 1218) corresponds to a contact on the touch-sensitive display maintained for a predetermined period of time. In some embodiments, the retention time is greater than a predetermined period of time. Performing an action when the input is maintained for a duration exceeding a predetermined period of time is an additional (eg, all) user selectable without requiring the display of additional user interface elements that obscure the viewfinder or clutter the display. Make filters accessible to users. Providing additional control options without cluttering the user interface with additional displayed controls (e.g., by helping the user achieve the intended result when operating and/or interacting with the device and reducing user errors) Improves the operability of the device, makes the user-device interface more efficient, and further reduces the device's power usage and improves battery life.

In some embodiments, in response to detecting the first input (eg, 1218), the electronic device (eg, 1200) provides a tactile output (eg, 1220) at the electronic device (eg, 1200). In some embodiments, the tactile output (eg, 1220) is responsive to the characteristic intensity of the contact reaching its respective intensity threshold (eg, 1216) (eg, representation of filters other than the second subset of representations of the filters) Are created) (as described above in relation to the process where the display is stopped). In some embodiments, a tactile output is provided when a subset of filters (eg, as described above) is displayed again (eg, in response to a liftoff of a contact or in response to a subsequent hard press). . Performing a tactile output when a subset of filters is displayed again provides feedback to the user about the status of the selected filter and helps the user perform actions more efficiently. Providing improved feedback improves the device's operability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device). And additionally, reduce device power usage and improve battery life.

In some embodiments, representations of a plurality of filters in a set of filters include a third subset of representations of filters that are not displayed on a display (eg, 1204) while detecting a first input (eg, 1218). . In some embodiments, input comprising movement of a contact along a filter selection interface (eg, 1206) causes the filter selection interface (eg, 1206) to scroll and expose one or more representations of filters in the third subset of filters. do.

Note that the details of the processes described above with respect to method 1300 (eg, FIGS. 13A-13F) are also applicable in a manner similar to the methods described below/above. For example, methods 700, 900, 1100, 1500, 1700, optionally, include one or more of the characteristics of the various methods described above with respect to method 1300. For example, elements of the filter user interface, affordances, and controls can be combined among various methods. As another example, the viewfinder of method 1300 is similar to the viewfinder of methods 700, 900, 1100, 1500, and 1700. For brevity, these details are not repeated below.

As illustrated in FIG. 14A, the electronic device 1400 includes a display 1404 that is touch-sensitive (eg, a touch screen), and the display displays information received from a camera (eg, 1402). In some embodiments, electronic device 1400 includes one or more features of devices 100, 300 and/or 500. In some embodiments, the display is separate from the touch-sensitive surface. In some examples, a camera (eg, 1402) is located on the front, back, or both sides of an electronic device (eg, 1400).

14A further illustrates that a camera application user interface is displayed on display 1404 to capture images with a camera (eg, 1402). The camera application user interface further includes a representation of image data 1410 that includes a live preview of the camera's field of view. 14A further illustrates that grid mode is enabled within the camera application and corresponding grids are displayed on the display to assist the user framing the image.

14A further illustrates the focal plane 1406, the horizontal plane 1408, and the relative angle between the focal plane 1406 and the horizontal plane 1412 for the camera. 14A, the focal plane 1406 is a virtual two-dimensional plane in front of the camera that intersects the focal point. 14A also includes a representation of the table and a representation 1411 of the displayed pie as lying on the representation of the table. In some embodiments, instead of a horizontal plane, different planes of orientation (eg, vertical) are used to determine a relative angle (eg, 1412). In some examples, the predetermined orientation is horizontal to the table, and the horizontal orientation provides an optimal orientation for capturing images from a top-down perspective.

14B illustrates that the electronic device is tilted forward (eg, towards) the representation of the table. As the electronic device tilts towards the table, the relative angle between the focal plane and the horizontal plane (parallel to the table) decreases from 90 degrees in Figure 14A to 70 degrees in Figure 14B.

As the user continues to tilt forward toward the table as illustrated in FIG. 14C, the relative angle between the horizontal plane and the focal plane decreases by 45 degrees. As further illustrated in FIG. 14C, at certain angles (eg, 45 degrees or less), the relative angle (eg, relative difference) between the horizontal plane and the focal plane is a predetermined alignment threshold (eg, a predetermined respective alignment). Threshold) (eg, 45 degrees) (eg, below). As a result, visual indicators (eg, 1414A and 1414B) are displayed. In some examples, the predetermined alignment threshold is 35, 30, 25, 20, 15, 10, or 5 degrees.

In some examples, the visual indicator is represented as two separate graphic objects 1414A and 1414B. In some embodiments, one of the two graphic objects (eg, 1414A) remains stationary in the middle of the display. In some embodiments, the second graphical object (eg, 1414B) changes position based on the orientation of the electronic device. The visual indicator assists the user in guiding the user when positioning the device for a predetermined orientation. As described below, as the relative angle between focal plane 1406 and horizontal plane 1408 continues to decrease (eg, as they become closer to being parallel), graphic objects 1414B are graphic objects. Gradually great toward (1414B). In some embodiments, if the relative angle between focal plane 1406 and horizontal plane 1408 is 0 (eg, when they are parallel), graphic objects (eg, 1414A and 1414B) appear to overlap completely. (For example, it appears as a graphic object).

In some embodiments, the visual indicator appears to fade in gradually when the electronic device determines that the relative angle between the focal plane and the horizontal plane is within a predetermined alignment threshold (eg, 45 degrees). In some embodiments, the visual indicator is displayed completely opaque when the electronic device determines that the relative angle between the focal plane and the horizontal is within a predetermined alignment threshold (eg, 45 degrees).

In some examples, the visual indicator is displayed only after the electronic device determines that the relative angle between the horizontal plane and the focal plane is maintained within an alignment threshold (eg, 45 degrees) for a predetermined period of time (eg, 2, 5, 10 seconds). do. Thus, in some examples, the electronic device must be oriented to remain within a predetermined alignment threshold for a predetermined period of time before the visual indicator is displayed. In some examples, the visual indicator is not displayed when the electronic device determines that the electronic device is not maintained within an alignment threshold (eg, 45 degrees) for a predetermined period of time (eg, 2, 5, 10 seconds). In some examples, time is not a factor in determining whether a visual indicator is displayed.

14D and 14E illustrate the user interface when the electronic device continues to tilt forward. As illustrated in FIG. 14D, when the electronic device 1400 determines that the relative angle between the horizontal plane and the focal plane continues to decrease, the graphic object 1414B progressively continues to become critical toward the graphic object 1414A. . Additionally, as the electronic device 1400 determines that the relative angle between the horizontal plane and the focal plane (eg, 20 degrees in FIG. 14D) continues to decrease, the graphic objects 1414A and 1414B continue to fade in. Additionally, in some embodiments, once the electronic device determines that the relative angle between the horizontal plane and the focal plane (eg, 11 degrees in FIG. 14E) reaches a predetermined fade threshold (eg, 15 degrees), graphic objects (1414A and 1414B) are displayed with maximum opacity. In some examples, the fade threshold is 30, 25, 20, 15, 10, 5, or 2 degrees.

As illustrated in FIG. 14F, the electronic device 1400 is now located within 5 degrees of the horizontal plane. As a result, graphic objects 1414A and 1414B are snapped to the central display position because the relative angle between the horizontal plane and the focal plane is within the snap threshold (eg, 10 degrees). In some embodiments, the snap threshold is 25, 20, 15, 10, 5, 4, 3, 2, or 1 degree. Thus, in some embodiments, even if the focal plane is not perfectly parallel to the horizontal plane, if it is determined that the relative angle between the horizontal plane and the focal plane (eg, 5 degrees) is within the snap threshold (eg, 10 degrees), the graphic Objects 1414A and 1414B are displayed (eg, snapped) in the middle of the display in a predetermined alignment. In some embodiments, graphic objects represent animation when snapped to the center location of the display. In some embodiments, the animation includes one or more of color differentiation, shape differentiation, or positional differentiation (eg, some bouncing effect).

As illustrated in FIG. 14G, although the electronic device 1400 continues to tilt forward, as determined by the electronic device, the relative angle between the horizontal plane and the focal plane (eg, 9 degrees) is the snap threshold ( For example, 10 degrees). As a result, in some embodiments, graphic objects (eg, 1414A and 1414B) are displayed and maintained at the center of the display.

As illustrated in FIG. 14H, once the electronic device 1400 determines that the relative angle between the horizontal plane and the focal plane (eg, 11 degrees) exceeds the snap threshold (eg, 10 degrees), graphic objects (eg, , 1414A and 1414B) are shown to be separated from each other. In some embodiments, the initial snap threshold (eg, 10 degrees) is increased to account for some unintended hand movements (eg, 15 degrees) once the initial snap occurs. Thus, in some embodiments, the graphic objects (eg, 1414A and 1414B) are displayed as long as the electronic device determines that the relative angle between the horizontal plane and the focal plane is within an increased snap threshold (eg, 15 degrees). It will remain snapped to the center position of the image.

As illustrated in FIGS. 14I-14K, when the electronic device continues to detect that the relative angle between the horizontal plane and the focal plane is increasing, the graphic object 1414B moves further away from the graphic object 1414A. Is continuously displayed. Also, as illustrated in FIGS. 14I and 14J, when the electronic device continues to detect that the relative angle between the horizontal plane and the focal plane is increasing, the visual indicator fades out gradually. As illustrated in FIG. 14K, when the electronic device determines that the relative angle between the horizontal plane and the focal plane is no longer within a predetermined alignment threshold, the visual indicator stops being displayed.

15A-15E are flow diagrams illustrating a method for constructing an image using an electronic device (eg, 1400) in accordance with some embodiments. Method 700 includes a camera, sensor (eg, gyroscope, accelerometer, compass), one or more input devices (eg, a touch-sensitive surface, keyboard, mouse), and a device with a display (eg, 100) , 300, 500, 1400). In some embodiments, the device has a plurality of cameras, each camera optionally having a different focal length. Some operations of method 1500 are selectively combined, the order of some operations is selectively changed, and some operations are optionally omitted.

As described below, method 1500 provides an intuitive method for constructing an image. This method reduces the user's cognitive burden on providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. For a battery-operated computing device, allowing the user to initiate various functions faster and more efficiently saves power and increases the time between battery charges.

At block 1502, the electronic device (e.g., 1400) is on the display (e.g., 1404), a camera viewfinder (e.g., 1410) for capturing media (e.g., live or near-live of content within the camera's field of view). Display (including preview images).

At block 1504, while displaying the camera viewfinder (eg, 1410), the techniques of blocks 1506-1550 are selectively performed.

In block 1506, based on the data from the sensor, according to the determination that the device meets the alignment guide display criteria, the techniques of blocks 1516 to 1524 are performed, and in block 1508, the alignment guide display criteria The relative difference between the orientation of the focal plane (e.g., 1406) of the camera (e.g., 1402) and the predetermined orientation (e.g., 1408) (e.g., orientation relative to a predetermined plane) such that alignment guide display criteria are met (e.g. , 1412) are within their respective alignment thresholds. In some embodiments, each alignment threshold is a range of predefined angles (eg, less than 5 degrees, less than 10 degrees) where the focal plane (eg, 1406) of the camera (eg, 1402) is parallel or perpendicular to the predetermined plane. , Or less than 15 degrees). Displaying the alignment guide in the camera viewfinder (eg, 1410) when the alignment threshold is met provides user feedback in the form of alignment guide when the user is trying to position the device in a particular orientation, which allows the user to Allows positioning more accurately in a specific orientation. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1510, the alignment guide display criteria are between an orientation of a focal plane (eg 1406) of the camera (eg 1402) and a predetermined orientation (eg 1408) such that the alignment guide display criteria are met. Includes the requirement that the relative difference (e.g., 1412) remains within their alignment threshold for at least a threshold amount of time (e.g., alignment guidance display criteria, at least a predetermined amount such that alignment guidance (e.g., 1414A and 1414B) is displayed. Requesting that the device remain within a range of orientations corresponding to displaying the alignment guides (eg, 1414A and 1414B) for a period of time, the alignment guides (eg, 1414A and 1414B) require the device to For example, if it is briefly moved over a range of orientations corresponding to displaying 1414A and 1414B) it is not displayed). Displaying the alignment guide in the camera viewfinder (e.g., 1410) when the alignment threshold is met for a threshold amount of time is more specific than when the device is only being redirected by the user (not keeping the device in a specific orientation). Orientation provides user feedback in the form of alignment guides when the user is positioning the device. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, in block 1512, the alignment guide display criteria are between an orientation of a focal plane (eg, 1406) of the camera (eg, 1402) and a predetermined orientation (eg, 1408) such that the alignment guide display criteria are met. Includes the requirement that the orientation of the device does not change much more than the threshold in time of the threshold while the relative difference (eg, 1412) of is maintained within the respective alignment threshold (eg, the alignment guidance display criteria, such as alignment guidance (eg, 1414A and 1414B) are displayed so that the device remains substantially stationary relative to a predetermined orientation (eg, 1408) while within a range of orientations corresponding to displaying the alignment guide (eg, 1414A and 1414B) to be displayed, The alignment guide (eg, 1414A and 1414B) is not displayed if the device is moving too much while in range of orientations corresponding to displaying the alignment guide (eg, 1414A and 1414B).

In block 1514, based on data from the sensor, alignment guide display criteria are not met (eg, the angle between a predetermined plane and a focal plane (eg, 1406) of the camera (eg, 1402) is a threshold value. Exceeding is determined (e.g., the camera lens is not sufficiently parallel to the horizontal/vertical plane), the electronic device (e.g., 1400) aligns the alignment guides (e.g., 1414A and 1414B) with the camera viewfinder (e.g., 1410). Holding display of alignment guides (e.g., 1414A and 1414B) in the camera viewfinder reduces visual distraction to the user, helps to avoid diverting the user's attention from the camera's viewfinder, and helps the camera viewfinder Avoids interfering with objects. Reducing visual distractions and avoiding obstructing the viewfinder improves the operability of the device (e.g., by helping the user achieve the intended result when operating and/or interacting with the device and reducing user errors) It makes the user-device interface more efficient, further reducing the device's power usage and improving battery life.

At block 1516, the electronic device (eg, 1400), on the display (eg, 1404), alignment guides (eg, 1414A and 1414B) in the camera viewfinder (eg, 1410) (eg, crosses, rectangles, ovals) Fields, triangles) (eg, overlaid on at least a portion of a live preview of the camera's (eg, 1402) field of view). At block 1518, the appearance of the alignment guides (e.g., 1414A and 1414B) changes as the orientation of the focal plane (e.g., 1406) of the camera (e.g., 1402) changes relative to a predetermined orientation (e.g., 1408). (For example, alignment guides (eg, 1414A and 1414B) indicate discrepancies from alignment). Displaying the alignment guide in the camera viewfinder (and holding the display) based on whether the alignment guide display criteria are met provides feedback to the user about the orientation of the device. For example, a user attempting to take a picture with the device's camera (eg, 1402) facing directly below is provided with visual feedback in the form of an alignment guide when the device is substantially downward, whereby the device Allows the user to better align the device (using alignment guides) to point downwards more accurately. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the sensor is part of a camera (eg, 1402). In some embodiments, the sensor is independent of the camera (eg, 1402). In some embodiments, the focal plane (eg, 1406) of the camera (eg, 1402) should be sufficiently parallel to a horizontal plane, such as a plane perpendicular to the Earth's surface or Earth's gravity. Parallel alignment allows perfect or near perfect alignment to capture photos from above. In some embodiments, the focal plane (eg, 1406) of the camera (eg, 1402) should be sufficiently perpendicular to a horizontal plane, such as a plane parallel to the Earth's surface or Earth's gravity. Vertical alignment allows perfect or almost perfect alignment to capture straight-line photos.

In some embodiments, the focal plane (eg, 1406) is a virtual two-dimensional plane in front of the camera (eg, 1402) passing through the focal point.

In some embodiments, alignment guidance (eg, 1414A and 1414B) is displayed when the guidance display mode is active (and when other criteria are met) and when the guidance display mode is not active (even if other criteria are met). It is not displayed. In some embodiments, the guidance display mode is a mode in which permanent guidances (eg, a grid) are displayed. Permanently displaying the guidance while the guidance display mode is enabled avoids user distraction that may result from the guidance being repeatedly displayed and removed from the display. Reducing visual distraction improves the device's operability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device). And additionally, reduce device power usage and improve battery life.

In some embodiments, at block 1520, the alignment guide (eg, 1411A and 1411B) includes at least two visual indicators (eg, one indicator displays independently of the other indicators (eg, 1404). ).

In some embodiments, at block 1522, at least one of the at least two visual indicators (eg, one indicator changes the position on the display (eg, 1404) independently of the other indicators) of the camera. It remains stationary as the orientation of the focal plane (eg, 1402) (eg, 1402) changes with respect to a predetermined orientation (eg, 1408). Displaying a visual indicator that remains stationary on the display of the device (eg, 1404) is a predetermined orientation of the device (eg, 1408) when the visual indicator of the stationary state is compared to the position of the visual indicator of the non-stationary state. It provides users with visual feedback on how close they are to. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, in block 1524, the distance between the two visual indicators is a relative difference between the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408). (E.g., 1412). In some embodiments, the farther the indicators are, the farther away the device is from being positioned to take a top-down picture of the field of view of one or more cameras. Dynamically updating the distance between two visual indicators provides the user with visual feedback on the degree of change in the orientation of the device required to achieve a predetermined orientation (eg, 1408), and changes the orientation of the device Doing gives the user visual feedback indicating that the distance between the two visual indicators will change. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1526, while displaying alignment guidance (eg, 1414A and 1414B) and while the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is the first orientation, the former The device (eg, 1400) detects a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation, based on data from the sensor.

In some embodiments, at block 1528, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation, block 1530. ), the electronic device (eg, 1400) is based on the degree of change in orientation (eg, orientation) of the first visual indicator (eg, a moving or non-static indicator) of at least two visual indicators. The calculated position, a position not based on the position of the stop indicator), and the displayed position of the first visual indicator is changed based on the relative difference between the first orientation and the second orientation (eg, 1412). (For example, one indicator changes position on the display (eg, 1404) independently of the other indicators). In some embodiments, at least one of the two distinct visual indicators is frozen when the devices change position beyond a predetermined threshold.

In some embodiments, at block 1532, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation, at block 1532, the camera The relative difference between the second orientation of the focal plane (eg, 1402) (eg, 1406) and the predetermined orientation (eg, 1408) (eg, 1412) is the first visual indicator alignment threshold (eg, indicator snapping) Upon determination that the indicators are not within a range of relative differences (eg, 1412) values to be displayed (eg, 0 degrees to 5 degrees) in a predetermined ordered arrangement, the electronic device (eg, 1400) is updated. The first visual indicator is displayed at the display position. In some embodiments, at block 1534, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation, at block 1534, the camera Upon determining that the relative difference (eg, 1412) between the second orientation of the focal plane (eg, 1402) and the predetermined orientation (eg, 1408) (eg, 1402) is within the first visual indicator alignment threshold, the former The device (eg, 1400) displays the first visual indicator at a predetermined display position (eg, a snapped position, eg, full alignment with a stop indicator). Snapping indicators into alignment when the device's current orientation is within a threshold amount of a predetermined orientation (eg, 1408) provides the user with a visual feedback of the orientation of the device's orientation, particularly the device. 1408). Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1536, while the first visual indicator is displayed at a predetermined display position (eg, a snapped position, eg, complete alignment with a stop indicator), the electronic device (eg, , 1400) detects a change in the orientation of the focal plane (eg 1406) of the camera (eg 1402) from the third orientation to the fourth orientation based on data from the sensor, the third orientation being the first It has a relative difference (eg, 1412) to a predetermined orientation (eg, 1408) that is within the visual indicator alignment threshold. In some embodiments, the change in orientation of the focal plane (eg, 1406) of the camera (eg, 1402) begins while the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is the third orientation. In some embodiments, the fourth orientation has a relative difference (eg, 1412) to a predetermined orientation (eg, 1408) that is outside the first visual indicator alignment threshold.

In some embodiments, in block 1538, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the third orientation to the fourth orientation, the blocks ( 1540 to 1542) technique is performed.

In some embodiments, at block 1540, the relative difference (eg, 1412) between the fourth orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408) is eliminated. 2 To determine that the visual indicator alignment threshold (e.g., indicator snapping threshold, a range of relative difference (e.g., 1412) values to which indicators will be displayed (e.g., 0 degrees to 5 degrees) in a predetermined alignment arrangement) is outside Accordingly, the electronic device (eg, 1400) maintains the display of the first visual indicator at a predetermined display location. In some embodiments, the second visual indicator alignment threshold is the same as the first visual indicator alignment threshold. In some embodiments, the second visual indicator alignment threshold (eg, to prevent the visual indicator from accidentally snapping out of alignment when the user holds the device near the first visual indicator alignment threshold) It is different from the indicator alignment threshold.

In some embodiments, at block 1542, the relative difference (eg, 1412) between the fourth orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408) is eliminated. 2 Upon determining that the visual indicator alignment is outside the threshold, a first visual indicator is displayed at the second updated display location, and the location of the second updated display location is the focal plane (eg 1406) of the camera (eg 1402) Is based on the relative difference (eg, 1412) between the orientation of and a predetermined orientation (eg, 1408). In some embodiments, while the moving visual indicator is snapped to the stationary visual indicator, the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the visual indicator alignment threshold are outside (eg, exceeded) ) A change in orientation that results in a relative difference in a predetermined orientation (e.g., 1408) (e.g., 1412), the moving visual indicator is based on the degree of change in orientation and not the position of the stop indicator. It will be displayed in a new location. In contrast, a change in orientation resulting in a relative difference (eg, 1412) below the visual indicator alignment threshold does not cause the position of the moving visual indicator to be updated (eg, it remains snapped to the stationary visual indicator). . Snapping indicators out of alignment when the device's current orientation exceeds a threshold amount relative to a predetermined orientation (eg, 1408) provides the user with a visual feedback of the device's orientation status, particularly when the device has a predetermined orientation (eg , 1408). Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1544, the relative difference (eg, 1) to a predetermined orientation (eg, 1408) in which the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is within a respective alignment threshold. 1412), and while the third visual indicator of the at least two visual indicators is displayed, the electronic device (eg, 1400) based on the data from the sensor, the eighth orientation from the seventh orientation Changes in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) with are detected.

In some embodiments, in block 1546, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the seventh orientation to the eighth orientation, the blocks ( 1548-1550).

In some embodiments, at block 1548, the relative difference (eg, 1412) between the eighth orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408) is visual. Upon determining that it is within the indicator alignment threshold, the electronic device (e.g., 1400) maintains the display of the third visual indicator (e.g., maintains the display at the same or different location, and similar visual characteristics (e.g., display intensity). Or maintain displays with different visual properties).

In some embodiments, at block 1550, the relative difference (eg, 1412) between the eighth orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408) is visual. Upon determining that it is not within the indicator alignment threshold, the electronic device (eg, 1400) stops displaying the third visual indicator. Stopping the display of alignment guides (e.g., 1414A and 1414B) in the camera viewfinder when the difference between the device's current orientation and a predetermined orientation exceeds a threshold amount allows the user to display the alignment guides (e.g., 1414A and 1414B). Wow, it is possible to switch between not displaying alignment guides (e.g., 1414A and 1414B) to display additional controls when they are helpful and not to display additional controls when they are not. Providing them when additional control options are helpful and not cluttering the UI with them when additional displayed control options are not helpful (e.g., when the user activates the device and/or interacts with him) Helping to provide and reducing user mistakes), improving the operability of the device and making the user-device interface more efficient, which additionally enables the user to use the device faster and more efficiently, Reduces and improves battery life.

In some embodiments, the predetermined orientation (eg, 1408) corresponds to a horizontal orientation (eg, a surface of the Earth or a plane perpendicular to the Earth's gravity).

In some embodiments, the predetermined orientation (eg, 1408) is a vertical orientation (eg, parallel to the Earth's gravity).

In some embodiments, at least one of the at least two visual indicators (eg, one indicator changes the position on the display (eg, 1404) independently of the other indicators) of the camera viewfinder (eg, 1410 ) Is displayed close to the center (eg, at or near the center). Displaying one of the visual indicators close to the center of the camera viewfinder provides additional space in all directions of the central visual indicator for displaying the other visual indicator, thereby between the predetermined orientation and the current device orientation. The difference in maximizing the particle size provided to the user as visual feedback. Providing the user with improved visual feedback improves the device's operability and user-device interface (e.g., by helping the user provide appropriate inputs when operating/interacting with the device and reducing user errors) Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, displaying the first visual indicator at a predetermined display location displays each (eg, predefined) animation at one or more locations of the visual indicators (eg, displaying a highlight). , Flashing, switching the flash and/or moving visual indicators from the first position to a predetermined display position at a fixed visual indicator position). In some embodiments, animation allows the user to understand that the image is aligned. Animation draws the user's attention to that fact.

In some embodiments, a fifth having a relative difference (eg, 1412) relative to a predetermined orientation (eg, 1408) in which the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is within their alignment threshold. While in orientation, and while the second visual indicator of the at least two visual indicators is displayed with a first value of the visual property (eg, first visual intensity, first color, first size), the electronic device (eg, 1400) ) Detects a change in the orientation of the focal plane (e.g., 1406) of the camera (e.g., 1402) from the fifth orientation to the sixth orientation, based on data from the sensor, and focuses on the camera (e.g., 1402). The relative difference (eg, 1412) between the sixth orientation of the plane (eg, 1406) and the predetermined orientation (eg, 1408) is within the visual indicator alignment threshold. In response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the fifth orientation to the sixth orientation, having a second value of visual property different from the first value (eg , Display a larger (or smaller) visual or display intensity, a second visual indicator (with different color, different size). Displaying visual indicators with different visual characteristics provides the user with additional visual feedback on the alignment of the device. Providing improved visual feedback improves the device's operability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device) This, in addition, reduces the device's power usage and improves battery life.

In some embodiments, the fifth orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408). And a first relative difference (eg, 1412). In some embodiments, the sixth orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408). Have a second relative difference between (e.g., 1412), and the second relative distance is greater than the first relative difference (e.g., 1412) (e.g., the sixth orientation is in a predetermined orientation (e.g., 1408) (e.g., horizontal). More out of alignment). In some embodiments, the visual characteristic is display intensity (eg, a degree of fade to maximum display intensity). In some embodiments, the first value is greater than the second value (eg, the first visual indicator is displayed more prominently (eg, less faded) when the camera orientation is closer to a predetermined orientation (eg, 1408). , As the camera orientation moves away from the predetermined orientation (eg, 1408), the visual indicator fades further until it fades out completely). Enhancing the orientation guide as the camera orientation moves toward a predetermined orientation, and fading the orientation guide as the camera orientation moves further away from the predetermined orientation, visualizes that the device is moving in and out of the range where the orientation guide will be displayed. Provide feedback to the user. Providing improved visual feedback improves the device's operability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result and reducing user mistakes when operating/interacting with the device) This, in addition, reduces the device's power usage and improves battery life.

Note that the details of the processes described above with respect to method 1500 (eg, FIGS. 15A-15E) are also applicable in a similar manner to the methods described above and below. For example, the methods 700, 900, 1100, 1300, 1700, optionally, include one or more of the characteristics of the various methods described above with respect to method 1500. For example, elements of the alignment interface, affordances, and controls can be combined among various methods. As another example, the viewfinder of method 1500 is similar to the viewfinder of methods 700, 900, 1100, 1300, 1700. For brevity, these details are not repeated below.

16A illustrates device 1600. In some embodiments, the electronic device 1600 includes some or all of the components of the device 600. In some embodiments, device 1600 includes multiple cameras 602 and 603 (eg, on the back of electronic device 1600 ). In some embodiments, device 1600 includes one or more features of devices 100, 300 and/or 500. In some examples, the electronic device (eg, 1600) has multiple cameras with fixed but different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of the electronic device (eg, 1600). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures multiple focal lengths, producing the same result as multiple cameras with fixed but different focal lengths. In some examples, the electronic device (eg, 1600) includes a depth camera (eg, 1690), such as an infrared camera, thermographic camera, or combination thereof. In some examples, the device further includes a light emitting device (eg, an optical projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate an object during capture of an image by a visible light camera and a depth camera (eg, IR camera), and information from the visible light camera and depth camera (eg, 1690) is captured by the visible light camera It is used to determine the depth map of different parts of. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for rear facing images and front facing images (eg, selfie) The depth information from the depth camera combined with the image data from the visible light camera (eg, 1692) for the images) is displayed. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, it is dramatic to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for users. In some embodiments, while displaying the camera user interface with one of the applied lighting effects, the device detects the selection of the camera switching affordance, maintains the display of the user interface controls for applying the lighting effect, and forwards Rear facing cameras (e.g., from each other) from front facing cameras (e.g., depth camera 1690 and visible light camera 1692), replacing the display of the field of view of the facing cameras with the field of view of the rear facing cameras (or vice versa). (Or vice versa) spaced apart two visible light cameras.

As illustrated in FIG. 16A, the electronic device 1600 includes a touch-sensitive (eg, touch screen) display 1604, and the display displays image data received from the camera (eg, 602). In some embodiments, the display is separate from the touch-sensitive surface. In some examples, multiple cameras (eg, 602 and 603) are located on the front, back, or both sides of an electronic device (eg, 1600).

16A further illustrates that the electronic device 1600 displays, on the display 1604, a camera application user interface 1605 for capturing images with a camera (eg, 602 ). The camera application user interface 1605 further includes a digital viewfinder 1610 that includes a live preview of the field of view of the camera (eg, 602 or 603). In some embodiments, the camera captures depth information associated with image data in real time. 16A further illustrates a camera capturing distinct depth levels in the field of view, including a subject in the foreground area (eg, 1608) (eg, female), and a fence in the background area (eg, 1609). In some examples, the camera (eg, 602) captures 3, 4, 5, 10, 20 or more depth levels in the field of view. The electronic device 1600 utilizes various depth levels when applying a filter to the representation of image data (eg, 1606) displayed in the digital viewfinder (eg, 1610), as discussed in more detail below.

Also, FIG. 16A illustrates that the electronic device 1600 displays the filter picker user interface 1613 in an expanded state. The filter picker user interface 1613 is located along the edge of the digital viewfinder 1610. In some examples, the filter picker user interface (eg, 1613) is optionally displayed above, below, left, or right of the digital viewfinder (eg, 1610). In some examples, the filter picker user interface (eg, 1613) includes one or more representations of filters (eg, representations of visual effects) arranged in a circular orientation or arranged in one or more rows and columns.

In some examples, the extended filter picker user interface (eg, 1613) is displayed at any location corresponding to the digital viewfinder. In some examples, the filter picker user interface (eg, 1613) is delineated (eg, bordered) to distinguish the filter picker user interface from the digital viewfinder. In some embodiments, the filter picker user interface (eg, 1613) is translucent (or partially translucent) when folded, and has no visible boundaries. As a result, in some examples, the filter picker user interface (eg, 1613) appears to be blended in with the digital viewfinder (eg, indistinguishable).

As shown in FIG. 16A, the electronic device 1600 displays a filter picker user interface 1613 that includes one or more filter expressions (eg, 1614A, 1614B, 1614C, 1614D, 1614E) corresponding to visual effects. do. In some examples, the filter picker user interface (eg, 1613) optionally includes additional filter expressions that are not displayed on the display (eg, they are off screen). In some examples, undisplayed filter expressions are displayed when the electronic device receives an input (eg, swipe gesture), which will cause the filter expressions to scroll through the filter container (eg, 1616).

As further illustrated in FIG. 16A, in some embodiments, electronic device 1600 displays filter representation 1614A in an extended filter picker user interface (eg, 1613) to display the currently selected visual effect. . In some embodiments, filter expression 1614A corresponds to a “natural light” light effect filter. Consequently, the foreground area 1608 and the background area 1609 are displayed with a “natural light” light effect filter (eg, using natural light from the scene). Since the image representation of FIG. 16A is shown without using any synthetic light (eg, simulated), natural light from the scene produces varying shadows on the object (eg, face, neck, and clothing). do. In some examples, possible filter expressions corresponding to lighting effects (eg, 1614A to 1614E) include “studio light” lighting effect, “contour light” lighting effect, “stage light” lighting effect, and “stage light mono” lighting. Contains effects. The lighting effects described above, when applied to the representation of image data (eg, 1606), affect the visual properties of the representation of the image data displayed on display 1604.

In some embodiments, when the electronic device 1600 applies the natural lighting effect, no synthetic lighting is added to the image (eg, the original image is displayed). In contrast, studio lighting effects include modeling of multiple individual point light sources (eg, lights in a photography studio) uniformly positioned around the object (eg, creating a brightly filled lighting effect). The contour lighting effect includes modeling multiple individual point light sources located along the perimeter of the object (eg, creating a slimming effect, and creating shadows on the side of the subject's face and/or on the subject's chin) do). Stage light illumination effects include modeling of a single individual point light source located on an object (eg, creating a spotlight effect). Stage light mono lighting effects include modeling in black and white of a single individual point light source positioned over a subject (eg, creating a spotlight effect in black and white).

In some examples, the electronic device (eg, 1600) detects the subject's face in the representation of the image data. As a result, the electronic device uses depth map information of the image data and corresponding facial features when applying the lighting effect. As a result, the lighting effect is applied with greater precision around the subject's face, and certain facial features are based on the selected lighting effect (eg, increasing or decreasing shadows around the subject's jaw and/or cheek bones). Can be differently illuminated. In some examples, the image data includes depth map information including depth contours of objects. As a result, the electronic device uses the contour data to apply the lighting effect around the object more accurately.

As illustrated in FIG. 16B, the electronic device 1600 receives an input (eg, tab 1618) at a location corresponding to the expanded filter picker user interface (eg, 1613 ). As shown in FIG. 16C, input (eg, swipe 1618) causes filter expressions 1614A-1614E to scroll through filter vessel 1616. In some embodiments, in response to the input (eg, swipe 1618), the filter expressions will scroll left across the top border of the filter picker user interface (eg, 1613). In some examples, a single swipe gesture will result in an incremental scroll of filter expressions. In some embodiments, the number of scrolled filter expressions will depend on the size of the swipe gesture. Thus, in some examples, a longer swipe will result in a longer scroll than a shorter swipe.

In response to the input (eg, 1618 swipe), in some embodiments, the electronic device 1600 applies a lighting effect corresponding to the filter representation (eg, 1614B) corresponding to the location on the display 1604 of the tap input. do. In some examples, the input is a swipe, press and hold, or an input with a characteristic strength that is above the respective intensity threshold. In some examples, an input having a characteristic strength that is greater than a respective intensity threshold detected on the representation of a filter of one of the filters 1614A to 1614F is, optionally, a location of an input having a characteristic intensity that is greater than a respective intensity threshold This results in the display of additional functionality for the associated corresponding filter expression.

In some embodiments, additional visual effects may be applied to the overall representation of the image data (eg, 1606) before applying a lighting effect corresponding to the filter representation (eg, 1614B). For example, some gradient fill may be applied to the representation of the image data (eg, 1606) before applying the stage light filter to more fluidly transition from no filter to light effect filter.

16C and 16D illustrate an electronic device 1600 that gradually applies a lighting effect as a result of the electronic device receiving an input (eg, swipe 1618) in FIG. 16B. In some embodiments, the lighting effect corresponding to the newly selected filter representation 1614B is gradually applied to the representation of image data (eg, 1606) in live preview. Since the selected lighting effect is "Studio Light", the corresponding visual effect simulates multiple point light sources affecting the object in the foreground area 1608. As a result, during the transition stage (FIG. 16C), the lighting effect corresponding to the filter expression 1614B is applied to the live preview at 50% intensity. The filter corresponding to the filter expression 1614B is fully applied (eg 100%) in FIG. 16D. In some examples, the filter is applied in increments (10%, 25%, 50%, 75%) while the electronic device 1600 applies the lighting effect until the transition is complete. In some examples, the background area (eg, 1609) is completely darkened when the electronic device 1600 applies the “Studio Light” light effect to the representation of the image data. In some embodiments, the background area (eg, 1609) is partially darkened when the electronic device 1600 applies the “Studio Light” light effect to the representation of the image data.

As illustrated in FIGS. 16C and 16D, since the image data captured by the camera (eg, 602) includes depth map information associated with the image data, the electronic device can use the available depth map information and image data It is possible to simulate the effect of various point light sources on the expression of (1606). In some embodiments, the same lighting effect is applied differently in the background area compared to the foreground area based on depth map information associated with the image data. As a result, objects in the foreground may appear more prominently, and objects in the background may become less prominent through a darkening effect. Additionally, as illustrated in FIG. 16D, since lighting effects simulate point light sources, depth map information is used to cast varying shadows on the subject's face in the foreground area (eg, 1608). As illustrated in FIG. 16D, since the “studio light” lighting effect simulates a plurality of point light sources, the electronic device 1600 uses depth map information to reduce shadows on the face of the object compared to the “natural light” lighting effect. use.

In some embodiments, when capturing an image, lighting effects are applied to the digital viewfinder for preview. In some embodiments, the lighting effect is associated with the image after capture.

As illustrated in FIG. 16E, the electronic device receives a gesture (eg, tap 1632) at a location corresponding to shutter button affordance 1630. In some embodiments, in response to receiving a gesture (eg, tab 1632 ), electronic device 1600 will capture image data corresponding to the field of view of one or more cameras (eg, 602 and 603 ). , The electronic device will associate the currently selected (eg, activated) lighting effect with the representation of the image data. The resulting image with the applied lighting effect will be stored in the camera roll of the electronic device and will be available for review in the camera roll application (eg, as represented by affordance 1634).

As illustrated in FIG. 16F, the electronic device 1600 will receive input (eg, tab 1620 ), and the representations of filters 1614A- 1614E will cause the filter container (eg, 1616) to scroll. In some embodiments, in response to a tap gesture (eg, 1620) corresponding to the filter representation (eg, 1614E), the representations of the filters are to the left across the upper perimeter of the extended filter picker user interface (eg, 1613). Will scroll. 16F further illustrates that the affordance for the camera roll application (eg, 1634) changes the appearance to correspond to the last captured image (eg, the studio light image captured in FIG. 16E). In some embodiments, the affordance for a camera roll application (eg, 1634) is displayed as a thumbnail representation of the captured image corresponding to the representation of the image. In some embodiments, the affordance for a camera roll application (eg, 1634) is displayed as a thumbnail icon of dissimilar to the captured image corresponding to the representation of the image.

16G illustrates the result of the tap gesture (eg, 1620) of FIG. 16F. In response to the tap gesture, the electronic device 1600 switches to the “stage light mono” light effect mode. As illustrated in FIG. 16G, while the electronic device 1600 is in the “stage light mono” mode, the alignment guide 1640 is displayed as a circle and positioned slightly off center in the vertical direction on the display 1604. . In some embodiments, the alignment guide (eg, 1640) is represented by a hexagonal, elliptical, or triangular shape. In some embodiments, the alignment guide is located in the middle of the display (eg, 1604) in both vertical and horizontal directions. In some embodiments, the alignment guide (eg, 1640) is positioned off-center on the display (eg, 1604) in one or both of the vertical and horizontal directions.

In some embodiments, the alignment guide is used to assist the user in aligning the body parts of the object (eg, their faces) within a specific part of the digital viewfinder. As illustrated in FIG. 16F, since the electronic device has determined that certain criteria are not met (eg, the subject's face is not properly aligned within the alignment guide 1640 ), the electronic device can take the appropriate action for the user. Display a message 1622 to prompt for action. In some embodiments, the lighting effect is image data (unless the electronic device determines that appropriate criteria have been met (eg, the subject's face is properly aligned with the alignment guide, and the object is at a suitable distance from the device). For example, 1606).

As further illustrated in FIG. 16G, the area outside the alignment guide 1640 is slightly shaded, while the area inside the alignment guide 1640 remains unchanged. In some embodiments, the area outside the alignment guide (eg, 1640) is not shaded, and the area inside the alignment guide (eg, 1640) is also not shaded.

16H illustrates that the user takes some steps towards the device such that the subject's face is properly aligned within the alignment guide 1640. As illustrated in FIG. 16H, when the subject's face is properly aligned with the alignment guide 1640, the area outside the alignment guide (eg, 1640) is shaded with a darker color and the area inside the alignment guide 1640 is It is not shaded (for example, no artificial lighting is applied). In some embodiments, the dark shade acts as a “lock-on” indicator and provides the user with an indication that criteria for the selected lighting effect (eg, stage light mono) have been met. In some embodiments, the currently selected lighting effect (eg, stage light mono) is applied to an area within the alignment guide (eg, 1640). In some embodiments, additional criteria (eg, the object is at a required distance from the electronic device, lighting conditions) must be met for dark shading outside the alignment guide to occur.

Additionally, as further illustrated in FIG. 16H, the blurring (eg, bokeh; shown as dashed line) optical effect is a full representation of the image data 1606 (eg, both inside the alignment guide and in the area without the alignment guide) ) Is applied to the background of the representation of the image data 1606 (eg, fence). In some embodiments, the blurring optical effect is disproportionately applied to different portions of the representation of image data (eg, 1606) displayed on the display (eg, 1604) when alignment criteria are met.

In some embodiments, the alignment guide (eg, 1640) is displayed in other (eg, not stage light mono) lighting effects that require removal of the background (eg, blurring).

In some examples, when the conditions for the alignment guide are no longer met (eg, the subject's face is no longer aligned with the alignment guide, the object has moved back too far from the device, and is no longer at an appropriate distance from the device Without, the lighting conditions are not correct), and the previously applied lighting effect snaps out (e.g., as shown in FIG. 16G, the electronic device returns slightly darker/not darker outside of the alignment guide). In some embodiments, when the dark shade snaps out, optionally, a temporary filter (eg, gradient) that is part of the light effect filter is applied to the image representation. Temporary filters help smooth transitions when shadows are reapplied (eg, less jarring).

As further illustrated in FIG. 16H, the electronic device 1600 receives a gesture (eg, tab 1636) at a location corresponding to the shutter button affordance 1630. In some embodiments, in response to receiving a gesture (eg, tap 1636 ), electronic device 1600 captures image data corresponding to the field of view of one or more cameras (eg, 602 and 603 ), The electronic device associates the currently selected (eg, activated) lighting effect with the representation of the image data. Subsequently, in these embodiments, the generated image with the applied lighting effect is stored in the camera roll of the electronic device and is available for review in the camera roll application (eg, as represented by affordance 1634). . In some embodiments, the lighting effect will be applied to the image data after capture, even if it is not applied to the digital viewfinder during capture (eg, the captured image will appear different from that seen by the user in the digital viewfinder). .

As illustrated in FIG. 16I, the electronic device detects an input (eg, tab 1645) at a location corresponding to a photo viewer application (eg, 1634). In response to receiving the input (eg, 1645), as illustrated in FIG. 16J, the electronic device 1600 is configured to view previously captured images rather than an image viewing mode (eg, rather than a live preview of camera data). Mode).

16J illustrates a user interface for a photo viewer application. The photo viewer application includes a thumb strip of previously captured images (eg, 1628A to 1628C), where 1628C is the last captured image. In some examples, previously captured images were captured using a camera corresponding to an electronic device (eg, 1600). In some examples, the electronic device (eg, 1600) has received previously captured images (eg, 1628A to 1628C) from a remote source (eg, server), and, optionally, previously captured images are different from the electronic device ( For example, not 1600).

16J further shows that the last captured image (eg, 1628C) was captured using stage light mono illumination. The "stage light mono" lighting effect simulates a single point light source, and as a result is similar to the spotlight effect. Using depth map information, the electronic device 1600 applies the “stage light mono” effect from above the object in the foreground area (eg, 1608). In some examples, the point light source can be simulated to originate from any direction. In some examples, the “stage light mono” effect is simulated to originate from the front, and as a result, a specific focus (eg, face) is emphasized, but the rest of the representation of the image data is darkened. As illustrated in FIG. 16J, since the simulated point light source originates from above the object, the electronic device can cast deeper shadows on the object (eg, face and neck) using depth map information of the image data. have. In some embodiments, the background of the image is removed and replaced with a solid color, such as black or white, or a user-selected color, to attract more attention to the object in the foreground and allow the user to be photographed against a solid backdrop. Can simulate a studio setup.

In some embodiments, the electronic device displays a “vertical” visual indicator (eg, 1638) at the top of the display as an indication to the user that previously captured image data includes depth map information. In some examples, the electronic device (eg, 1600) receives input at a location corresponding to a visual indicator to toggle the simulated depth effect (eg, bokeh effect) on and off. In some embodiments, when the simulated depth effect is toggled off, the lighting effect will be maintained. In some examples, the visual indicator will toggle the simulated depth effect and lighting effect together when activated. In some examples, the electronic device optionally receives input to change the lighting effect in the photo viewer application to a different lighting effect using a filter picker user interface (as described above). In some examples, if previously captured image data does not have depth map information; The electronic device (eg, 1600) will not provide an option to apply a simulated depth effect or lighting effect. In some examples, if the previously captured image data does not have depth map information associated with the image data, the electronic device will not display a visual indicator (eg, 1638).

In some examples, the electronic device (eg, 1600) stores depth map information along with image data in one file. In some examples, the electronic device (eg, 1600) stores depth map information separately from the image data. In some embodiments, if the electronic device stores an image with depth map information as a flat image (eg, without depth map information), the electronic device (eg, 1600) no longer applies a lighting effect to the representation of the image data. Will not be able to.

17A-17G for applying simulated lighting effects to a representation of image data (eg, 1606) and displaying alignment guidance (eg, 1640) using an electronic device (eg, 1600) according to some embodiments. It is a flow chart illustrating a method. Method 1700 includes one or more input devices (eg, a touch-sensitive surface, keyboard, mouse), one or more cameras (eg, cameras with variable focal lengths and/or depth-sensing cameras and cameras that capture color images) ), and a device with a display (eg 1604) (eg 100, 300, 500, 1700). In some embodiments, the display (eg, 1604) is a touch-sensitive display. In some embodiments, the display (eg, 1604) is not a touch-sensitive display. In some embodiments, the electronic device (eg, 1600) includes a plurality of cameras. In some embodiments, the electronic device (eg, 1600) has only one camera. Some operations of method 1700 are selectively combined, the order of some operations is selectively changed, and some operations are optionally omitted.

As described below, the method 1700 provides an intuitive way to display alignment guides when certain criteria are met and apply simulated lighting effects to the representation of image data. This method reduces the user's cognitive burden on providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. For a battery-operated computing device, allowing the user to initiate various functions faster and more efficiently saves power and increases the time between battery charges.

At block 1702, the electronic device (eg, 1600) displays, on the display (eg, 1604 ), a representation of the image data (eg, 1606) associated with the depth map information (eg, the image or photo is of the device. Display (eg, displayed on 1604). In some embodiments, a live preview of image data is displayed in a digital viewfinder (eg, 1610).

In some embodiments, at block 1704, the representation of the image data is a live preview of the image data captured within the field of view of one or more cameras displayed in a digital viewfinder (eg, 1610).

In some embodiments, at block 1706, the depth map information associated with the image data includes information corresponding to at least three different depth levels (eg, at least background, depth level, foreground depth level, and intermediate depth level). Includes. Depth map information comprising three or more different depth levels provides the user with a framework for applying depth-specific filters and provides the user with more precise feedback on depth positioning of objects within the field of view of the camera(s). . Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 1708, the depth map information associated with the image data includes information identifying depth contours of the object in the representation of the image data; The lighting effects change the appearance of the representation of the image data based on the position and curvature of the contours of the object. Including depth contours of objects in the depth map information allows the device to provide the user with more precise visual feedback on the shape and depth positioning of objects in the field of view of the camera(s). Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve. Changing the appearance of the representation of image data based on the position and curvature of the contours of the object creates a lighting effect that simulates the use of the external lighting equipment without requiring external lighting equipment. Generating these lighting effects without requiring external lighting equipment reduces the size and cost of the equipment needed to create an image with specific lighting effects.

At block 1710, while displaying a representation of the image data on a display (eg, 1604), the electronic device (eg, 1600), at block 1712, via one or more input devices, a first input (eg, , Swipe, tap and hold, tap (e.g., 1620), detect button presses; gestures are two images based on depth map information (e.g., based on measurements of the depth sensor or taken simultaneously from different locations) Multiple lighting effects (based on disparity mapping between them) (eg, one or more of natural light, studio light, contour light, stage light, stage light mono, as described in more detail in paragraph [0499] below) It can be at the top of the icon representing the light filter (eg, filter expressions 1614A, 1614B, 1614C, 1614D, 1614E) (or other user interface system used to select the filter) to select the respective filter of Simultaneously displaying a representation of the image data (eg, 1606) and providing a filter selection interface (eg, filter picker user interface 1613) provides visuals for filters available to be applied to objects and previews of the camera field of view. Provide feedback to the user Providing improved visual feedback to the user (eg, by providing the feedback indicative of input that will cause the device to produce the intended result when the device is activated and/or interacts with the device) It improves the operability of the device (by helping to achieve the intended result and reduces user errors) and makes the user-device interface more efficient, in addition, by enabling the user to use the device more quickly and efficiently. Reduces power usage and improves battery life.

In some embodiments, at block 1714, the first input is an input received while the first criteria (eg, a set of lighting effect application criteria) are met, and the first criteria are electronic to ensure that the first criteria are met. And the requirement that the subject is detected in the field of view within a predetermined distance from the device (eg, 1600). Other criteria include: the focal length of the first camera exceeds the minimum distance threshold, the focal length of the first camera does not exceed the maximum distance threshold, the object is detected beyond a predetermined minimum distance from the device, One or more of the amount of light detected that exceeds the minimum light threshold, and the amount of light detected that does not exceed the maximum light threshold. In some embodiments, if the first criteria are not met, it is withheld to apply the first lighting effect or the second lighting effect. Applying a lighting effect when it is determined that the object is within a predetermined distance when the first input is received allows the user to visually feedback that the object is properly positioned so that an optimal (or nearly optimal) effect can be achieved with a filter. To give. Similarly, not applying a lighting effect when the object is not within a predetermined distance provides feedback to the user that the object is not properly positioned and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, at block 1716, the first input is an input received while the first criteria are not met, and the first criteria are a predetermined distance from the electronic device (eg, 1600) such that the first criteria are met Includes the requirement that the subject is detected in the field of view; The method includes displaying a live preview without applying a first change in appearance to the live preview, and then detecting that the first criteria are met; And in response to detecting that the first criteria have been met, applying a first change in appearance to the live preview. In some embodiments, other conditions in the set include: the focal length of the first camera exceeds the minimum distance threshold, the focal length of the first camera does not exceed the maximum distance threshold, the subject has a predetermined minimum from the device It includes one or more of being detected over a distance, the amount of light detected exceeds the minimum light threshold, and the amount of light detected not exceeding the maximum light threshold. In some embodiments, if the first criteria are not met, it is withheld to apply the first lighting effect or the second lighting effect. Applying the lighting effect when the first criteria are met gives the user visual feedback that the lighting effect application criteria have been met (eg, the object is properly positioned) and that the optimal (or almost optimal) effect can be achieved with the filter. to provide. Similarly, not applying the lighting effect when the first criteria are not met provides feedback to the user that the first criteria are not met, and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

At block 1710, while displaying a representation of image data on a display (eg, 1604), the electronic device (eg, 1600) optionally performs the techniques of blocks 1718 to 1742.

In some embodiments, in block 1718, in response to detecting the first input, the electronic device (eg, 1600) prepares to capture image data using a respective filter.

In some embodiments, at block 1720, depending on the determination that each light effect is a first light effect (eg, stage light or other light effect including background removal), the electronic device (eg, 1600) is one Capture user interface for a first lighting effect resulting in a first change in the appearance of a portion of the representation of the image data when an object in the field of view of the above cameras meets the first criteria (e.g., lighting effect application criteria) For example, an alignment guide (eg, 1640) is displayed (eg, an alignment user interface that changes appearance when the first criteria are met).

In some embodiments, at block 1722, according to the determination that each light effect is a second light effect (eg, natural light, studio light, or other light effect that does not include background light or background removal), the electronic device (Eg, 1600) is a second illumination that results in a second change in the appearance of a portion of the representation of the image data when an object in the field of view of one or more cameras meets the first criteria (eg, lighting effect application criteria). Displays the capture user interface for effects. (Eg preview of 3D lighting effect).

In some embodiments, in block 1724, displaying the capture user interface for the first lighting effect includes: representation of image data; And simultaneously displaying an alignment guide (eg, 1640) (eg, circle, ellipse, hexagon) displayed in the first area within the representation of the image data, wherein the first criteria are representations of the image data such that the first criteria are met. Includes the requirement that the representation of the face of the object displayed in is within the alignment guide (eg, 1640). In some embodiments, alignment guides (eg, 1640) are overlaid on the representation of the image data. In some embodiments, the user will align a portion of their body (eg, face) within an alignment guide (eg, 1640). In some embodiments, the subject's face is substantially aligned with the perimeter of the alignment guide (eg, 1640). In some embodiments, the alignment guide (eg, 1640) is not displayed during some modes (eg, portrait mode, studio light). In some embodiments, an alignment guide (eg, 1640) is located at the center of the live preview. In some embodiments, the alignment guide (eg, 1640) is positioned off-center. In some embodiments, when multiple objects are detected, multiple alignment guides are displayed in a live preview. Displaying the alignment guide (eg, 1640) in the representation of the image data provides the user with visual feedback as to where to position the subject's face to capture the image with the desired optical effect. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life.

In some embodiments, at block 1726, the first criteria are that of the subject's face such that the first criteria are met (eg, when the user's face is detected within an alignment guide (eg, 1640) in the live preview). The representation includes the requirement to be detected in the first area within the representation of the image data.

In some embodiments, in block 1728, applying the first change in appearance to the image data is within the representation of the image data compared to the appearance of the representation of the image data displayed in the first area within the representation of the image data. And changing the appearance of the representation of the image data displayed in the second area (eg, outside the alignment guide (eg, 1640)), the second area being separate (eg, non-overlapping, separate) from the first area . In some embodiments, changing the appearance of the representation of the image data displayed in the second area compared to the representation of the image data displayed in the first area includes applying a filter (eg, an alignment filter) to the second area. do. In some embodiments, the alignment filter is not applied to the first region. In some embodiments, the alignment filter is uniformly applied to the outer area of the alignment guide (eg, 1640). In some embodiments, the alignment filter gradually changes in intensity (eg, darker/lighter closer to the edge of the live preview compared to how the alignment filter is applied near the edge of the alignment guide (eg, 1640)). do). In some embodiments, the alignment filter is a single optical effect (eg, bokeh, saturation, color). In some embodiments, the alignment filter includes multiple optical effects (eg, bokeh, saturation, color) (eg, combinations thereof). In some embodiments, each of the multiple optical effects is applied differently to different portions of the live preview.

In some embodiments, at blocks 1732-1742, the electronic devices (eg, 1600) determine that the first criteria are not met; In response to determining that the first criteria are not met (eg, no object is detected in the field of view within a predetermined distance from the electronic device (eg, 1600); the face is not aligned within the alignment guide (eg, 1640)): Stop applying the first change in appearance to the representation of the image data; Display on the display (eg, 1604) a representation of the image data (eg, 1606) without a first change in applied appearance (eg, an unchanged image with no filters applied or with partial filters applied); On the display (eg, 1604), a graphical representation of unsatisfied lighting condition application criteria is displayed. In some embodiments, the device displays instructions for placing the face (eg, moving closer, moving further) within the alignment guide (eg, 1640). In some embodiments, when the conditions are met again, the filter is reapplied again.

At block 1710, while displaying the representation of the image data on the display (eg, 1604), the electronic device (eg, 1600) further performs the techniques of blocks 1744-1750.

In block 1744, after detecting the first input, the electronic device (eg, 1600) has a second input (eg, 1636) corresponding to a request to capture image data corresponding to the field of view of one or more cameras (eg, 1636) (eg, Swipe, tap and hold, tap, press a button; the gesture detects that it can be at the top of the icon representing the shutter button).

In block 1746, the respective lighting effects selected based on the first input are the first lighting effects (eg, natural light, studio light, contour light, stage light, as described in more detail in paragraph [0499] below). , In accordance with the determination that one or more of the stage light mono), at block 1748, the electronic device (eg, 1600) corresponds to the field of view of the one or more cameras and the first lighting effect and the representation of the image data (eg, 1606). Associating (e.g., applying the first lighting effect to the image data, preparing to apply the first lighting effect to the image data, or capturing the captured image data, the first lighting effect by default when viewing of the image data is requested) Captures image data (marking as image data to be applied), and the first illumination effect is based on depth map information (and, for example, between two images based on measurements of a depth sensor or taken simultaneously from different locations). Based on mismatch mapping). Associating the first lighting effect to the representation of the image data (eg, 1606) based on the depth map information provides the user with additional visual feedback on the depth map information. For example, the first lighting effect includes one or more light sources at different locations, intensities, or types (directional, ambient, point) that provide the user with feedback about a particular orientation of objects corresponding to depth map information. can do. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life.

In block 1750, the natural light, the studio light, as described in more detail in the second light effect (e.g., paragraph below, where the respective light effects selected based on the first input is different from the first light effect) , According to the determination of contour light, stage light, or one or more of stage light mono), the electronic device (eg, 1600) corresponds to the field of view of one or more cameras, and the second lighting effect is associated with the representation of the image data (eg, 1606). Associating (e.g., applying a second lighting effect to the image data, preparing to apply the second lighting effect to the image data, or capturing the captured image data, the second lighting effect by default when viewing of the image data is requested) Captures image data (marking as image data to be applied), and the second lighting effect is based on depth map information. (Eg, based on measurements of depth sensor or disparity mapping between two images taken simultaneously from different locations). Applying a second lighting effect to the representation of the image data (eg, 1606) based on the depth map information provides the user with additional visual feedback on the depth map information. For example, the second lighting effect includes one or more light sources at different locations, intensities, or types (directional, ambient, point) that provide the user with feedback about a particular orientation of objects corresponding to depth map information. can do. Providing improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result when operating and/or interacting with the device. Helping to reduce user errors) and improve the device's operability and make the user-device interface more efficient, which further reduces the device's power usage and battery by enabling the user to use the device faster and more efficiently. Improves life.

In block 1572, after capturing image data corresponding to the field of view of one or more cameras, the electronic device (eg, 1600) may select a second input (eg, camera roll affordance (eg, 1634) or select a photo viewer application). Response) to receive a request to display captured image data.

In blocks 1754 to 1758, in response to receiving a request to display captured image data in response to the second input (eg, in response to shutter shutter being activated), the image while the first lighting effect is selected Upon determining that the data has been captured, the electronic device (eg, 1600) displays a representation of the image data (eg, 1606) with the first lighting effect applied (eg, displays a representation of the image data with the stage light filter applied) ); Upon determining that the image data was captured while the second lighting effect (eg, not stage light) was selected, the electronic device (eg, 1600) displays a representation of the image data (eg, 1606) to which the second lighting effect was applied. .

In some embodiments, displaying the capture user interface for the first lighting effect applies a placeholder filter to the representation of the image data displayed by the electronic device (eg, 1600) in the digital viewfinder (eg, 1610). And (eg, dimming or desaturating the background), the placeholder filter is based on the first lighting effect and applied regardless of whether the first criteria are met. Applying a placeholder filter creates a smoother and more intuitive (less confusing) transition to the light filter. Applying a placeholder filter regardless of whether the first criteria are met or not is viewfinder content such as which parts of the image correspond to parts of the depth map in the background (eg 1609) compared to the foreground (eg 1608). Provides visual feedback to the depth map corresponding to the user. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, in response to the first input, the electronic device (eg, 1600) places the filter filter (eg, dimming or desaturation in the background) into the live preview without applying a first change in appearance to the live preview. ) Is applied; In response to detecting that the first criteria have been met, the electronic device (eg, 1600) applies a first change in appearance to the live preview while continuing to apply the placeholder filter to the live preview (eg, the placeholder filter is It is part of the first lighting effect that does not take into account depth map information and is therefore displayed regardless of whether the first criteria have been met). Applying a placeholder filter regardless of whether the first criteria are met or not, for example, which part of the image corresponds to the parts of the depth map in the background (eg 1609) compared to the foreground (eg 1608) It provides the user with visual feedback on the depth map corresponding to the content. Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve.

In some embodiments, while displaying a representation of the image data, the electronic device (eg, 1600) displays on the display (eg, 1604) a visual indication (eg, 1638) that the image data includes depth map information. do. (Eg, a “portrait mode” badge is optionally displayed to indicate the availability of depth map information.

In some embodiments, the electronic device (eg, 1600) that applies the first or second lighting effect is a depth map associated with the image data, in a representation of the image data displayed in the digital viewfinder (eg, 1610). And applying a simulation of one or more point light sources in space based on the information. Lighting options include natural light, studio light, contour light, stage light, and stage light mono. Different lighting effects model (eg, simulate) the results of one or more of the point sources in space based on the depth map of the image data. The natural lighting option does not apply any synthetic lighting to the image (eg, the original image is displayed or a portion of the original image is displayed, and blur is applied to different parts of the original effect to simulate the bokeh effect) . Studio lighting effects include modeling of multiple individual point light sources located around an object (eg, creating a brightly filled light effect). The outline lighting effect includes modeling a number of distinct point light sources located at several points around the object to create shadows on the object's face (eg, creating a slimming effect, and on the side of the object's face and/or Or create shadows on the subject's chin). Stage light illumination effects include modeling of a single individual point light source located on an object (eg, creating a spotlight effect). Stage light mono lighting effects include modeling in black and white of a single individual point light source located around an object (eg, creating a spotlight effect in black and white). In some embodiments, the illumination filter simulates point light sources. In some embodiments, the lighting effect is snapped in when the first criteria are met, as described in more detail above. In some embodiments, when the system detects a face, facial features are considered when applying a lighting effect. As a result, the lighting effect changes the appearance of the representation of the image data based on the specific facial features and face shape of the object. Applying a simulation of a point light source provides the user with a visual representation of the content of depth map information, and allows the device to provide the user with visual feedback on the shape and depth positioning of objects within the field of view of the camera(s). . Providing improved visual feedback helps the user achieve the intended result and provides the user with feedback indicating, for example, input that will cause the device to produce the intended result when operating and/or interacting with the device. It improves the operability of the device (by reducing mistakes) and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the device's power usage and saving battery life. Improve. Applying a simulation of one or more point light sources to the representation of the image data displayed in the viewfinder (eg, 1610) creates a lighting effect that simulates the use of the external lighting equipment without requiring external lighting equipment. Generating these lighting effects without requiring external lighting equipment reduces the size and cost of the equipment needed to create an image with specific lighting effects.

In some embodiments, while the first lighting effect is applied, the electronic device (eg, 1600) maintains at least one value of the previously applied visual effect. (Eg, bokeh, lighting) Thus, it is possible to achieve a light effect and a bokeh effect in one representation of image data.

In some embodiments, the previously applied visual effect is a color filter.

In some embodiments, the second input is an input received while the first lighting effect is applied to the representation of the image data (eg, 1606), and applying the second lighting effect is the first lighting effect and the second lighting effect It involves a gradual transition between applications. In some embodiments, the gradual transition includes 100% first at first time, 0% second, 90% first at second time, 10% second, and the like. Gradually switching between lighting effects reduces the user distraction generated by blinking the filters on/off, focusing the user on taking the desired picture, thereby reducing the number of inputs required to capture the desired picture. And memory requirements for storage of photos. Reducing the number of inputs required to capture the desired image and reducing memory requirements, such as by helping the user achieve the intended result when operating and/or interacting with the device and reducing user mistakes (eg, by reducing user errors) It improves the operability of the device and makes the user-device interface more efficient, which additionally enables the user to use the device more quickly and efficiently, thereby reducing the power consumption of the device and improving the battery life.

Note that the details of the processes described above with respect to method 1700 (eg, FIGS. 17A-17G) are also applicable in a similar manner to the methods described above. For example, methods 700, 900, 1100, 1300, and 1500, optionally, include one or more of the characteristics of the various methods described above and referring to method 1700. For example, elements of the filter user interface, affordances, and controls can be combined among various methods. As another example, the viewfinder of method 1700 is similar to the viewfinder of methods 700, 900, 1100, 1300, 1500. For another example, a preview of the first filter described in method 700 can be displayed on a display (eg, 1604). As another example, as described in method 900, the result of applying the first lighting effect to the representation of the image data can be displayed on a display (eg, 1604). As another example, as described in method 1100, the result of applying the first image filter to the foreground area of the image data can be displayed on a display (eg, 1604). As another example, as described in method 1300, displaying a filter selection interface that includes representations of a plurality of filters in a set of filters may also be displayed on a display (eg, 1604). As another example, as described in method 1500, displaying an alignment guide may also be displayed on a display (eg, 1604). For brevity, these details are not repeated.

The operations in the information processing methods described above may optionally include one or more functional modules in an information processing apparatus, such as general purpose processors or application specific chips (eg, as described in connection with FIGS. 1A, 3, and 5A). It is implemented by driving. Additionally, the operations described above with reference to FIGS. 17A-17G are, optionally, implemented by the components shown in FIGS. 1A and 1B. For example, display operation 1702, detection operation 1712, display operation 1720, display operation 1722, display operation 1724, and application operation 1728, optionally, event classifier 170 , Event recognizer 180, and event handler 190. The event monitor 171 in the event classifier 170 detects a contact on the touch-sensitive display 604, and the event dispatcher module 174 delivers the event information to the application 136-1. Each event recognizer 180 of application 136-1 compares the event information to respective event definitions 186, and the first contact at the first location on the touch-sensitive surface selects the object on the user interface. It is determined whether it corresponds to a predefined event or sub-event. When an individual predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally updates the application internal state 192 by using or calling the data updater 176 or the object updater 177. In some embodiments, event handler 190 accesses individual GUI updater 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A and 1B.

The foregoing description, for purposes of explanation, has been described with reference to specific embodiments. However, the above exemplary discussions are not intended to identify or limit the invention in the precise forms disclosed. Many modifications and variations are possible in terms of the above teachings. The embodiments have been selected and described in order to best describe the principles of technology and their practical application. Thus, it will be possible for those skilled in the art to best utilize the techniques and various embodiments using various modifications as appropriate for the particular use contemplated.

It should be noted that although the present invention and examples have been sufficiently described with reference to the accompanying drawings, various changes and modifications will be apparent to those skilled in the art. It should be understood that such changes and modifications are included within the scope of the disclosure and examples as defined by the claims.

As described above, one aspect of the present technology is the collection and use of data available from various sources, to enhance the delivery of invited content or any other content that users may be interested in to users. The present disclosure contemplates that in some cases, such collected data may include personal information data that can be used to uniquely identify or contact a particular individual or to locate him. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, home addresses, or any other identifying information.

This disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data can be used to deliver targeted content of greater interest to the user. Thus, the use of such personal information data enables calculated control of the delivered content. In addition, other uses of personal information data that benefit the user are also contemplated by the present disclosure.

The present disclosure further contemplates that entities responsible for the collection, analysis, disclosure, delivery, storage, or other use of such personal information data will comply with well-established privacy policies and/or privacy practices. In particular, such entities should implement and continue to use privacy policies and practices that are generally recognized to meet or exceed industrial or administrative requirements to keep personal information data private and secure. For example, personal information from users should be collected for the legitimate and appropriate uses of entities, and should not be shared or sold beyond these legitimate uses. In addition, such collection should only occur after receiving informed consent from users. Additionally, such entities will take any necessary steps to protect and secure access to such personal information data and to ensure that others with access to personal information data adhere to their privacy policies and procedures. . In addition, such entities can themselves be evaluated by third parties to demonstrate their adherence to widely recognized privacy policies and practices.

Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of advertisement delivery services, the present technology allows users to select "opt in" or "opt out" of participation in the collection of personal information data during registration for the service. It can be configured to allow. In another example, users can choose not to provide location information for targeted content delivery services. In another example, users may choose not to provide accurate location information, but to allow delivery of location zone information.

Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without the need to access such personal information data. do. That is, various embodiments of the present technology are not rendered inoperable due to lack of all or part of such personal information data. For example, the content may be directed to content requested by the device associated with the user, other non-personal information available for content delivery services, or non-personal information data such as publicly available information or a minimum amount of personal information. It can be selected and inferred to users by inferring preferences based thereon.

Claims (141)

As a method,
In an electronic device having one or more input devices and a display,
Displaying on the display a representation of image data obtained by one or more cameras from a physical environment, the image data corresponding to when the image data was obtained by the one or more cameras and filter picker user interface Associated with depth map information determined based on one or more sensor measurements of the physical environment in time, the filter picker user interface comprising a first representation of a first lighting effect and a second representation of a second lighting effect, wherein A first lighting effect simulates at least one light source located at a first position relative to an object in the representation of the image data and is based on the depth map information, and the second lighting effect is the image data different from the first position Simulating at least one light source located at a second position with respect to the object in the representation of and based on the depth map information;
While displaying a representation of the image data on the display,
Detecting, via the one or more input devices, a first input in a first area on the display corresponding to the first representation of the first lighting effect in the filter picker user interface;
Applying the first lighting effect to the representation of the image data by detecting the first input in the first area on the display corresponding to the first representation of the first lighting effect in the filter picker user interface To do;
Detecting, via the one or more input devices, a second input in a second area on the display corresponding to the second representation of the second lighting effect in the filter picker user interface; And
Different from the first lighting effect in the representation of the image data, as detecting the second input in the second area on the display corresponding to the second representation of the second lighting effect in the filter picker user interface Applying the second lighting effect, and stopping applying the first lighting effect to the representation of the image data; The method of claim 1, wherein the electronic device further comprises the one or more cameras, and the representation of the image data is a live preview of image data captured within the field of view of the one or more cameras displayed in a digital viewfinder. . 3. The method of claim 2, wherein the first input is an input received while first criteria are met, and the first criteria include a requirement that an object is detected in the field of view within a predetermined distance from the electronic device. According to claim 3, Applying the first lighting effect,
And applying a placeholder filter to the representation of the image data displayed in the viewfinder, wherein the placeholder filter is based on the first lighting effect and applied regardless of whether the first criteria are met. Become, how. According to claim 4,
While the first lighting effect is being applied,
Determining that the first criteria are not met; And
In response to the determination that the first criteria are not met,
Stopping applying the first lighting effect to the representation of the image data;
Displaying a representation of the image data to which the first lighting effect is not applied, on the display; And
And displaying, on the display, a graphical representation of the first criteria that are not met. According to claim 3,
In response to the first input, applying a placeholder filter to the live preview without applying the first lighting effect to the live preview; And
And in response to detecting that the first criteria have been met, further comprising applying the first lighting effect to the live preview while continuing to apply the placeholder filter to the live preview. According to claim 2,
The first input is an input received while the first criteria are not met, and the first criteria include a requirement that an object is detected in the field of view within a predetermined distance from the electronic device;
The method is:
Displaying the live preview without applying the first lighting effect to the live preview and detecting that the first criteria are satisfied; And
And in response to detecting that the first criteria have been met, further comprising applying the first lighting effect to the live preview. The method of claim 1, wherein the representation of the image data is previously captured image data. According to claim 1,
And before displaying the representation of the image data, at the device, receiving the image data and the depth map information associated with the representation of the image data. The method of claim 9,
While displaying the representation of the image data, displaying on the display, a visual indication that the image data includes depth map information. The method of claim 1, wherein the depth map information associated with the image data includes information corresponding to at least three different depth levels. The method of claim 1, wherein the depth map information associated with the image data includes information identifying depth contours of an object in the representation of the image data;
And the lighting effects change the appearance of the representation of the image data based on the position and curvature of the contours of the object. According to claim 1, Applying the first lighting effect or the second lighting effect,
And applying a simulation of one or more point light sources in space based on the depth map information associated with the image data to a representation of the image data displayed in a digital viewfinder. According to claim 1,
While the first lighting effect is applied, further comprising maintaining at least one value of the previously applied visual effect. 15. The method of claim 14, wherein the previously applied visual effect is a color filter. The method of claim 1, wherein the second input is an input received while the first lighting effect is being applied to the representation of the image data, and applying the second lighting effect comprises:
And gradually transitioning between application of the first lighting effect and the second lighting effect. According to claim 1,
Identifying a first portion of the representation of the image data and a second portion of the representation of the image data, wherein the first portion of the representation of the image data includes an object,
Applying the first lighting effect includes applying the first lighting effect to the first portion using a first lighting characteristic, and using the second lighting characteristic different from the first lighting characteristic to a second portion Applying the first lighting effect to the,
Applying the second lighting effect includes applying the second lighting effect to the first portion using a third lighting characteristic, and using the fourth lighting characteristic different from the third lighting characteristic to a second portion Including the Dean system to apply the second lighting effect to,
Wherein the first and third illumination characteristics are different. As an electronic device,
One or more input devices;
display;
One or more processors; And
An electronic device comprising memory for storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of claim 1. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display, wherein the one or more programs are any one of claims 1 to 17. A computer readable storage medium comprising instructions for performing a method. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images