KR20190142397A - User Interface Camera Effects - Google Patents

Abstract

The present disclosure relates generally to a user interface. In some examples, the electronic device provides a transition between simulated lighting effects. In some examples, the electronic device applies the simulated lighting effect to the image. In some examples, the electronic device provides user interfaces for applying a filter to an image. In some examples, the electronic device provides a reduced filter interface. In some examples, the electronic device provides visual assistance displayed in the viewfinder.

Description

User Interface Camera Effects

This application claims the benefit of US patent application Ser. No. 15 / 728,147, filed Oct. 9, 2017, entitled “USER INTERFACE CAMERA EFFECTS,” which is the September 9, 2017 application. US Provisional Patent Application No. 62 / 556,414, filed dated and entitled "USER INTERFACE CAMERA EFFECTS," and US Provisional Patent Application, filed June 4, 2017, entitled "USER INTERFACE CAMERA EFFECTS." Prioritizes to 62 / 514,947. This application is also filed on July 10, 2017 and filed with the patent application `` PATTERNATIONAL CAMERA EFFECTS '', Danish Patent Application PA201770563 and filed on September 22, 2017, and named USER INTERFACE CAMERA EFFECTS. Claims priority to Danish patent application PA201770719. The contents of these applications are incorporated herein in their entirety by reference for all purposes.

Technical Field

The present disclosure relates generally to a computer user interface of an electronic device, in particular a device having a built-in camera.

The use of electronic devices to record video and take pictures has increased significantly in recent years. Exemplary electronic devices for recording video and taking pictures include smartphones and handheld cameras. Such devices often include a viewfinder that a user can use to preview before taking a picture or recording a video.

However, some techniques for managing camera effects using electronic devices are generally cumbersome and inefficient. For example, modifying visual effects in the viewfinder such that the captured image and the recorded video exhibit visual effects often require extensive user input and are inaccurate. Existing techniques require more time than is needed, wasting user time and device energy. This latter consideration is particularly important in battery-operated devices.

Thus, the present technology provides electronic devices with faster and more efficient methods and interfaces for managing camera effects. These methods and interfaces optionally complement or replace other methods for managing camera effects. Such methods and interfaces reduce the cognitive burden on the user and create a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces save power and increase the time between battery charges. In some examples, the techniques provide simulated visual effects and captured images in camera viewfinders without requiring additional hardware components. In some examples, the techniques provide the ability to quickly switch between user interfaces with limited user input. In some examples, the techniques efficiently provide enhanced image editing capabilities that result in visually satisfactory results for the displayed digital viewfinder and for the captured videos. In some examples, the techniques efficiently provide user interfaces for switching between different sets of options with minimal input. In some examples, the techniques efficiently provide user interfaces that provide additional functionality without any direct input. These techniques reduce the number of user inputs required and conserve battery power.

An example method includes simultaneously displaying, on a display, a camera application user interface on an display in an electronic device having one or more cameras, one or more input devices, and a display, wherein the camera application user interface is adapted to the field of view of the one or more cameras. A digital viewfinder including a live preview; And a representation of the filter picker user interface overlaid on the digital viewfinder; While simultaneously displaying a representation of the digital viewfinder and filter picker user interface, detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices; And in response to detecting the first input, in accordance with the determination that the first criteria are met—the first criteria establish a requirement that the filter picker user interface is overlaid on a respective portion of the live preview when the first input is detected. Including, applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And performing the respective action in the camera application without applying the preview of the first filter to the live preview in accordance with the determination that the filter picker user interface is not overlaid on respective portions of the live preview when the first input is detected. Includes steps

An exemplary non-transitory computer readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras, one or more input devices, and a display. The one or more programs can include instructions for simultaneously displaying on the display a camera application user interface, the camera application user interface comprising: a digital viewfinder comprising a live preview of the field of view of one or more cameras; And a representation of the filter picker user interface overlaid on the digital viewfinder; Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, in accordance with the determination that the first criteria are met—the first criteria establish a requirement that the filter picker user interface is overlaid on a respective portion of the live preview when the first input is detected. Includes: instructions for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And perform the respective action in the camera application without applying the preview of the first filter to the live preview in accordance with the determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. Contains instructions for

An exemplary temporary computer readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras, one or more input devices, and a display. The one or more programs can include instructions for simultaneously displaying on the display a camera application user interface, the camera application user interface comprising: a digital viewfinder comprising a live preview of the field of view of one or more cameras; And a representation of the filter picker user interface overlaid on the digital viewfinder; Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, in accordance with the determination that the first criteria are met—the first criteria establish a requirement that the filter picker user interface is overlaid on a respective portion of the live preview when the first input is detected. Includes: instructions for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And perform the respective action in the camera application without applying the preview of the first filter to the live preview in accordance with the determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. Contains instructions for

An example electronic device includes a memory that stores one or more cameras, one or more input devices, a display, one or more processors, and one or more programs configured to be executed by one or more processors. The one or more programs can include instructions for simultaneously displaying on the display a camera application user interface, the camera application user interface comprising: a digital viewfinder comprising a live preview of the field of view of one or more cameras; And a representation of the filter picker user interface overlaid on the digital viewfinder; Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, in accordance with the determination that the first criteria are met—the first criteria establish a requirement that the filter picker user interface is overlaid on a respective portion of the live preview when the first input is detected. Includes: instructions for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And perform the respective action in the camera application without applying the preview of the first filter to the live preview in accordance with the determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. Contains instructions for

An example electronic device includes one or more cameras; One or more input devices; display; Means for simultaneously displaying a camera application user interface on the display, the camera application user interface comprising: a digital viewfinder comprising a live preview of the field of view of one or more cameras; And a representation of the filter picker user interface overlaid on the digital viewfinder; Means for detecting a first input starting at a position corresponding to a respective portion of the live preview via one or more input devices while simultaneously displaying a representation of the digital viewfinder and filter picker user interface; And in response to detecting the first input, in accordance with the determination that the first criteria are met—the first criteria establish a requirement that the filter picker user interface is overlaid on a respective portion of the live preview when the first input is detected. Means for applying a preview of the first filter to a live preview of the field of view of the camera that was not applied before the first input was detected; And perform the respective action in the camera application without applying the preview of the first filter to the live preview in accordance with the determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. Means for;

An example method includes displaying, on a display, a representation of image data associated with depth map information in an electronic device having one or more input devices and a display; While displaying a representation of the image data on the display, detecting the first input via the one or more input devices; In accordance with detecting the first input, applying a first lighting effect to the representation of the image data, the first lighting effect based on depth map information; Detecting a second input via one or more input devices; And according to detecting the second input, applying a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on the depth map information.

An exemplary non-transitory computer readable storage medium stores one or more input devices and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input via the one or more input devices while displaying a representation of the image data on the display; Instructions for applying the first lighting effect to the representation of the image data in accordance with detecting the first input, the first lighting effect based on depth map information; Instructions for detecting a second input via one or more input devices; And in accordance with detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on the depth map information.

An exemplary temporary computer readable storage medium stores one or more input devices and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; Instructions for detecting a first input via the one or more input devices while displaying a representation of the image data on the display; Instructions for applying the first lighting effect to the representation of the image data in accordance with detecting the first input, the first lighting effect based on depth map information; Instructions for detecting a second input via one or more input devices; And in accordance with detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on the depth map information.

An example electronic device includes one or more input devices; display; One or more processors; And a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs comprising: instructions on the display to display a representation of the image data associated with the depth map information; Instructions for detecting a first input via the one or more input devices while displaying a representation of the image data on the display; Instructions for applying the first lighting effect to the representation of the image data in accordance with detecting the first input, the first lighting effect based on depth map information; Instructions for detecting a second input via one or more input devices; And in accordance with detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on the depth map information.

An example electronic device includes one or more cameras; One or more input devices; display; Means for displaying on the display a representation of the image data associated with the depth map information; Means for detecting a first input via the one or more input devices while displaying a representation of the image data on the display; Means for applying a first lighting effect to the representation of the image data in accordance with detecting the first input, the first lighting effect based on depth map information; Means for detecting a second input via one or more input devices; And means for applying, in detecting the second input, a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on depth map information.

An example method detects, via one or more input devices, an electronic device having a display, a first input corresponding to a selection of a first image filter of a representation of image data having a first appearance, through one or more input devices. Making; And in response to detecting the first input, applying the first image filter to the representation of the image data, the applying step in accordance with determining that the image data has depth information associated therewith-depth information. Allows the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the foreground of the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data. Applying a first image filter to the region, wherein the first level adjustment indicates a first degree at which the first image filter changes the appearance of the representation of the image data; Applying a first image filter to the background region of the representation of the image data using the second level adjustment to change the appearance of the background region of the representation of the image data-the second level adjustment is performed by the first image filter A second degree of changing the appearance of the expression, wherein the first level adjustment and the second level adjustment are different; And after applying the first image filter to the representation of the image data, displaying on the display a representation of the respective image to which the first filter is applied to the representation of the image data.

An exemplary non-transitory computer readable storage medium stores one or more input devices and one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for detecting, via one or more input devices, a first input corresponding to the selection of the first image filter of the representation of the image data having the first appearance; And in response to detecting the first input, instructions for applying the first image filter to the representation of the image data, the instructions for applying in accordance with a determination that the image data has depth information associated therewith; Depth information allows the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data. Instructions for applying a first image filter to a foreground area of the first level adjustment indicates a first degree by which the first image filter changes the appearance of the representation of the image data; Instructions for applying a first image filter to a background area of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data. A second degree of changing the appearance of the representation of the data, wherein the first level adjustment and the second level adjustment are different; And after applying the first image filter to the representation of the image data, displaying on the display a representation of the respective image to which the first filter is applied to the representation of the image data.

An exemplary temporary computer readable storage medium stores one or more input devices and one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for detecting, via one or more input devices, a first input corresponding to the selection of the first image filter of the representation of the image data having the first appearance; And in response to detecting the first input, instructions for applying the first image filter to the representation of the image data, the instructions for applying in accordance with a determination that the image data has depth information associated therewith; Depth information allows the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data. Instructions for applying a first image filter to a foreground area of the first level adjustment indicates a first degree by which the first image filter changes the appearance of the representation of the image data; Instructions for applying a first image filter to a background area of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data. A second degree of changing the appearance of the representation of the data, wherein the first level adjustment and the second level adjustment are different; And after applying the first image filter to the representation of the image data, displaying on the display a representation of the respective image to which the first filter is applied to the representation of the image data.

An example electronic device includes one or more input devices; display; One or more processors; And a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs corresponding to selection of the first image filter of the representation of the image data having the first appearance via the one or more input devices. Instructions for detecting a first input to make; And in response to detecting the first input, instructions for applying the first image filter to the representation of the image data, the instructions for applying in accordance with a determination that the image data has depth information associated therewith; Depth information allows the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data-the representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data. Instructions for applying a first image filter to a foreground area of the first level adjustment indicates a first degree by which the first image filter changes the appearance of the representation of the image data; Instructions for applying a first image filter to a background area of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data. A second degree of changing the appearance of the representation of the data, wherein the first level adjustment and the second level adjustment are different; And after applying the first image filter to the representation of the image data, displaying on the display a representation of the respective image to which the first filter is applied to the representation of the image data.

An example electronic device includes one or more input devices; display; Means for detecting, via one or more input devices, a first input corresponding to the selection of the first image filter in the representation of the image data having the first appearance; And in response to detecting the first input, means for applying the first image filter to the representation of the image data, the means for applying in accordance with determining that the image data has depth information associated therewith- Depth information allows the foreground area of the representation of the image data to be distinguished from the background area of the representation of the image data—representation of the image data using a first level adjustment to change the appearance of the foreground area of the representation of the image data. Means for applying a first image filter to a foreground area of the first level adjustment indicates a first degree by which the first image filter changes the appearance of the representation of the image data; Means for applying a first image filter to a background area of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data-the second level adjustment is performed by the first image filter. Indicate a second degree of changing the appearance of the expression of the first level adjustment and the second level adjustment being different; And means for applying a first image filter to the representation of the image data and then displaying on the display a representation of the respective image to which the first filter is applied to the representation of the image data.

An example method includes, in an electronic device having a display, with one or more input devices, displaying, on a display, a filter selection interface that includes representations of a plurality of filters in a set of filters; While simultaneously displaying on the display a representation of the image data and the filter selection interface, via one or more input devices, a first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria Detecting; In response to detecting the first input, stopping displaying the first subset of representations of the filters in the set of filters, wherein the first subset of representations of the filters in the set of filters is a first in the filter selection user interface. One or more filters in a first direction from the representation of the filter and one or more filters in a second direction from the representation of the first filter; And maintaining a display of the second subset of representations of the filters in the set of filters, wherein the second subset of representations of the filters includes at least a representation of the first filter.

An exemplary non-transitory computer readable storage medium stores one or more input devices and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying a filter selection interface on a display, the filter selection interface including representations of a plurality of filters in a set of filters; While simultaneously displaying on the display a representation of the image data and the filter selection interface, via one or more input devices, a first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria Instructions for detecting a problem; In response to detecting the first input, instructions to stop displaying the first subset of representations of the filters in the set of filters—the first subset of representations of the filters in the set of filters is in the filter selection user interface. One or more filters in a first direction from the representation of the first filter and one or more filters in the second direction from the representation of the first filter; And instructions for maintaining a display of the second subset of representations of the filters in the set of filters, wherein the second subset of representations of the filters includes at least a representation of the first filter.

An exemplary temporary computer readable storage medium stores one or more input devices and one or more programs configured to be executed by one or more processors of an electronic device having a display. One or more programs include instructions for displaying a filter selection interface on a display, the filter selection interface including representations of a plurality of filters in a set of filters; While simultaneously displaying on the display a representation of the image data and the filter selection interface, via one or more input devices, a first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria Instructions for detecting a problem; In response to detecting the first input, instructions to stop displaying the first subset of representations of the filters in the set of filters—the first subset of representations of the filters in the set of filters is in the filter selection user interface. One or more filters in a first direction from the representation of the first filter and one or more filters in the second direction from the representation of the first filter; And instructions for maintaining a display of the second subset of representations of the filters in the set of filters, wherein the second subset of representations of the filters includes at least a representation of the first filter.

An example electronic device includes one or more input devices; display; One or more processors, and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including on the display, representations of the plurality of filters in the set of filters. Instructions for displaying a message; While simultaneously displaying on the display a representation of the image data and the filter selection interface, via one or more input devices, a first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria Instructions for detecting a problem; In response to detecting the first input, instructions to stop displaying the first subset of representations of the filters in the set of filters—the first subset of representations of the filters in the set of filters is in the filter selection user interface. One or more filters in a first direction from the representation of the first filter and one or more filters in the second direction from the representation of the first filter; And instructions for maintaining a display of the second subset of representations of the filters in the set of filters, wherein the second subset of representations of the filters includes at least a representation of the first filter.

An example electronic device includes one or more input devices; display; Means for displaying on the display a filter selection interface comprising representations of a plurality of filters in a set of filters; While simultaneously displaying on the display a representation of the image data and the filter selection interface, via one or more input devices, a first input at a position corresponding to the filter selection interface while the first filter of the set of filters meets the selection criteria Means for detecting; In response to detecting the first input, means for stopping displaying the first subset of representations of the filters in the set of filters, wherein the first subset of representations of the filters in the set of filters is defined in the filter selection user interface. One or more filters in a first direction from the representation of the first filter and one or more filters in the second direction from the representation of the first filter; And means for maintaining a display of the second subset of representations of the filters in the set of filters, wherein the second subset of representations of the filters includes at least a representation of the first filter.

An example method includes, in an electronic device having a camera, a sensor, one or more input devices, and a display, displaying a camera viewfinder for capturing media on the display; And while displaying the camera viewfinder, based on the data from the sensor, in accordance with the determination that the device meets the alignment guide display criteria—the alignment guide display criteria are pre-aligned with the orientation of the camera's focal plane such that the alignment guide display criteria are met. Including the requirement that the relative difference between the determined orientations is within their alignment thresholds,-displaying, on the display, the alignment guides in the camera viewfinder-the appearance of the alignment guides in which the orientation of the camera's focal plane is dependent on the predetermined orientation. Change with respect to; And suspending displaying the alignment guide in the camera viewfinder, in accordance with the determination that the alignment guide display criteria are not met, based on the data from the sensor.

An exemplary non-transitory computer readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, a sensor, one or more input devices, and a display. One or more programs include instructions for displaying a camera viewfinder for capturing media on the display; And while displaying the camera viewfinder, based on the data from the sensor, in accordance with the determination that the device meets the alignment guide display criteria—the alignment guide display criteria are pre-aligned with the orientation of the camera's focal plane such that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-instructions for displaying, on the display, the alignment guide in the camera viewfinder-the appearance of the alignment guide is such that the orientation of the focal plane of the camera is predetermined; Changing as the orientation changes; And instructions for withholding displaying the alignment guide in the camera viewfinder in accordance with the determination that the alignment guide display criteria are not met based on the data from the sensor.

An exemplary temporary computer readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, a sensor, one or more input devices, and a display. One or more programs include instructions for displaying a camera viewfinder for capturing media on the display; And while displaying the camera viewfinder, based on the data from the sensor, in accordance with the determination that the device meets the alignment guide display criteria—the alignment guide display criteria are pre-aligned with the orientation of the camera's focal plane such that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-instructions for displaying, on the display, the alignment guide in the camera viewfinder-the appearance of the alignment guide is such that the orientation of the focal plane of the camera is predetermined; Changing as the orientation changes; And instructions for withholding displaying the alignment guide in the camera viewfinder in accordance with the determination that the alignment guide display criteria are not met based on the data from the sensor.

Exemplary electronic devices include a camera; sensor; One or more input devices; display; One or more processors; And a memory storing one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for displaying a camera viewfinder for capturing media on the display; And while displaying the camera viewfinder, based on the data from the sensor, in accordance with the determination that the device meets the alignment guide display criteria—the alignment guide display criteria are pre-aligned with the orientation of the camera's focal plane such that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds-instructions for displaying, on the display, the alignment guide in the camera viewfinder-the appearance of the alignment guide is such that the orientation of the focal plane of the camera is predetermined; Changing as the orientation changes; And instructions for withholding displaying the alignment guide in the camera viewfinder in accordance with the determination that the alignment guide display criteria are not met based on the data from the sensor.

Exemplary electronic devices include a camera; sensor; One or more input devices; display; Means for displaying a camera viewfinder for capturing media on the display; And while displaying the camera viewfinder, based on the data from the sensor, in accordance with the determination that the device meets the alignment guide display criteria—the alignment guide display criteria are pre-aligned with the orientation of the camera's focal plane such that the alignment guide display criteria are met. Includes the requirement that the relative difference between the determined orientations is within their respective alignment thresholds,-means for displaying the alignment guides on the display, in the camera viewfinder-the appearance of the alignment guides in which the orientation of the camera's focal plane is predetermined Change as it changes with respect to-; And means for withholding displaying the alignment guide in the camera viewfinder in accordance with the determination that the alignment guide display criteria are not met based on the data from the sensor.

An example method includes, in an electronic device having one or more input devices, one or more cameras, and a display, displaying on the display a representation of image data associated with depth map information; While displaying a representation of the image data on the display, detecting, via one or more input devices, a first input for selecting a respective filter of the plurality of lighting effects based on the depth map information; After detecting the first input, detecting a second input corresponding to the request to capture image data corresponding to the field of view of the one or more cameras; And in response to detecting the second input, capturing image data corresponding to the field of view of the one or more cameras, wherein the capturing comprises: the respective lighting effects selected based on the first input being the first lighting effect. Capturing image data corresponding to the field of view of the one or more cameras and associating the first lighting effect with a representation of the image data, wherein the first lighting effect is based on depth map information; And capturing image data corresponding to the field of view of the one or more cameras and determining the second lighting effect according to the determination that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect. Associating with the representation, wherein the second lighting effect is based on depth map information.

An exemplary non-transitory computer readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices, one or more cameras, and a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; While displaying a representation of the image data on the display, instructions for detecting, via the one or more input devices, a first input for selecting a respective filter of the plurality of lighting effects based on the depth map information; Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras after detecting the first input; And in response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, wherein the instructions for capturing include: firstly the respective lighting effects selected based on the first input; Instructions for capturing image data corresponding to a field of view of one or more cameras and associating the first lighting effect with a representation of the image data in accordance with the determination of the lighting effect, wherein the first lighting effect is based on depth map information; And capturing image data corresponding to the field of view of the one or more cameras and determining the second lighting effect according to the determination that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect. Instructions for associating with the representation, wherein the second lighting effect is based on depth map information.

An exemplary temporary computer readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices, one or more cameras, and a display. One or more programs include instructions for displaying, on a display, a representation of image data associated with depth map information; While displaying a representation of the image data on the display, instructions for detecting, via the one or more input devices, a first input for selecting a respective filter of the plurality of lighting effects based on the depth map information; Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras after detecting the first input; And in response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, wherein the instructions for capturing include: firstly the respective lighting effects selected based on the first input; Instructions for capturing image data corresponding to a field of view of one or more cameras and associating the first lighting effect with a representation of the image data in accordance with the determination of the lighting effect, wherein the first lighting effect is based on depth map information; And capturing image data corresponding to the field of view of the one or more cameras and determining the second lighting effect according to the determination that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect. Instructions for associating with the representation, wherein the second lighting effect is based on depth map information.

An example electronic device includes one or more input devices; One or more cameras; display; One or more processors; And a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs comprising: instructions on the display to display a representation of the image data associated with the depth map information; While displaying a representation of the image data on the display, instructions for detecting, via the one or more input devices, a first input for selecting a respective filter of the plurality of lighting effects based on the depth map information; Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of one or more cameras after detecting the first input; And in response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, wherein the instructions for capturing include: firstly the respective lighting effects selected based on the first input; Instructions for capturing image data corresponding to a field of view of one or more cameras and associating the first lighting effect with a representation of the image data in accordance with the determination of the lighting effect, wherein the first lighting effect is based on depth map information; And capturing image data corresponding to the field of view of the one or more cameras and determining the second lighting effect according to the determination that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect. Instructions for associating with the representation, wherein the second lighting effect is based on depth map information.

An example electronic device includes one or more input devices; One or more cameras; display; Means for displaying on the display a representation of the image data associated with the depth map information; Means for displaying a representation of the image data associated with the depth map information on the display while displaying the representation of the image data on the display; Means for detecting, via the one or more input devices, a first input for selecting a respective filter of the plurality of lighting effects based on the depth map information while displaying a representation of the image data on the display; Means for detecting a second input after detecting the first input, the second input corresponding to the request to capture image data corresponding to the field of view of the one or more cameras; And in response to detecting the second input, means for capturing image data corresponding to the field of view of the one or more cameras, wherein the means for capturing comprises: firstly the respective lighting effects selected based on the first input; Means for capturing image data corresponding to a field of view of one or more cameras and associating the first lighting effect with a representation of the image data in accordance with the determination of the lighting effect, the first lighting effect based on depth map information; And capturing image data corresponding to the field of view of the one or more cameras and determining the second lighting effect according to the determination that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect. Means for associating with the representation, wherein the second lighting effect is based on depth map information.

Executable instructions for performing these functions are optionally included in a non-transitory computer readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a temporary computer readable storage medium or other computer program product configured for execution by one or more processors.

Thus, devices are provided with faster and more efficient methods and interfaces for managing camera effects, thereby increasing the effectiveness, efficiency and user satisfaction of such devices. These methods and interfaces may complement or replace other methods for managing camera effects.

For a better understanding of the various described embodiments, reference should be made to the following detailed description of the invention in connection with the following drawings in which like reference numerals indicate corresponding parts throughout the figures thereof.
1A is a block diagram illustrating a portable multifunction device having a touch-sensitive display in accordance with some embodiments.
1B is a block diagram illustrating example components for event processing in accordance with some embodiments.
2 illustrates a portable multifunction device having a touch screen in accordance with some embodiments.
3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, in accordance with some embodiments.
4A illustrates an example user interface for a menu of applications on a portable multifunction device, in accordance with some embodiments.
4B illustrates an example user interface for a multifunction device having a touch-sensitive surface separate from the display, in accordance with some embodiments.
5A illustrates a personal electronic device in accordance with some embodiments.
5B is a block diagram illustrating a personal electronic device in accordance with some embodiments.
5C and 5D illustrate example components of a personal electronic device having a touch-sensitive display and intensity sensors, in accordance with some embodiments.
5E-5H illustrate example components and user interfaces of a personal electronic device, in accordance with some embodiments.
6A-6N illustrate example devices and user interfaces for managing camera lighting effects, in accordance with some embodiments.
7A-7F are flow diagrams illustrating a method for managing camera lighting effects, in accordance with some embodiments.
8A-8J illustrate example devices and user interfaces for applying camera lighting effects, in accordance with some embodiments.
9A-9D are flow diagrams illustrating a method for applying camera lighting effects, in accordance with some embodiments.
10A-10N illustrate example devices and user interfaces for managing filter effects, in accordance with some embodiments.
11A-11C are flow diagrams illustrating a method for managing filter effects, in accordance with some embodiments.
12A-12P illustrate example devices and user interfaces for managing a filter user interface, in accordance with some embodiments.
13A-13F are flow diagrams illustrating a method for managing a filter user interface, in accordance with some embodiments.
14A-14K illustrate example devices and user interfaces for capturing an image, in accordance with some embodiments.
15A-15E are flow diagrams illustrating a method for capturing an image, in accordance with some embodiments.
16A-16J illustrate example devices and user interfaces for capturing an image, in accordance with some embodiments.
17A-17G are flow diagrams illustrating a method for capturing an image, in accordance with some embodiments.

The following description describes example methods, parameters, and the like. However, it should be appreciated that this description is not intended as a limitation on the scope of the present disclosure but instead is provided as a description of exemplary embodiments.

Electronic devices are being designed and manufactured with more advanced camera features and sensors. However, some electronic devices, due to the nature of their designs, cannot capture rich point-of-light photos without additional hardware. Most of the point light photography techniques require a large number of expensive light sources to be located around the object as well as a separate backdrop. However, many electronic devices have only a single flash that emits light in one direction. As a result, most of the point light photography techniques simply cannot be achieved through traditional electronic devices.

Embodiments described herein include electronic devices that utilize depth map information to provide improved camera capabilities. In some embodiments, depth map information is used when applying a filter to an image. In some embodiments, visual assistance is provided to help the user capture a complete image. The described embodiments also include complementary user interfaces that enable these improved camera capabilities.

At the same time, the described embodiments improve the performance of the device's camera capabilities while allowing efficient packaging and manufacturing of thin and light weight devices. The use of fixed focal length cameras is beneficial when they are thinner and smaller.

In the following, FIGS. 1A and 1B, 2, 3, 4A and 4B and 5A-5H provide a description of an example device for performing techniques for managing event notifications. 6A-6N illustrate example user interfaces for managing a user interface. 7 is a flow diagram illustrating methods of managing a user interface in accordance with some embodiments. The user interfaces of FIGS. 6A-6G are used to illustrate the processes described below, including the processes of FIG. 7.

8A-8J illustrate example user interfaces for applying a simulated lighting effect. 9 is a flow diagram illustrating methods of simulating a lighting effect in accordance with some embodiments. The user interfaces of FIGS. 8A-8D are used to illustrate the processes described below, including the processes of FIG. 9.

10A-10N illustrate example user interfaces for applying a filter to an image. 11 is a flowchart illustrating methods of applying a filter to an image in accordance with some embodiments. The user interfaces of FIGS. 10A-10D are used to illustrate the processes described below, including the processes of FIG. 11.

12A-12P illustrate example user interfaces for displaying a reduced filter user interface. 13 is a flowchart illustrating methods of displaying a reduced filter user interface in accordance with some embodiments. The user interfaces of FIGS. 12A-12D are used to illustrate the processes described below, including the processes of FIG. 13.

14A-14K illustrate example user interfaces for providing visual assistance. 15 is a flow diagram illustrating methods of providing visual assistance in accordance with some embodiments. The user interfaces of FIGS. 14A-14D are used to illustrate the processes described below, including the processes of FIG. 15.

16A-16J illustrate example user interfaces for providing visual assistance when applying a simulated optical effect. 17 is a flowchart illustrating methods of providing visual assistance when applying a simulated optical effect in accordance with some embodiments. The user interfaces of FIGS. 16A-16J are used to illustrate the processes described below, including the processes of FIG. 17.

Although the following description uses terms such as "first", "second", and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first touch may be referred to as the second touch, and similarly, the second touch may be referred to as the first touch. The first touch and the second touch are both touches, but they are not the same touch.

The terminology used in the description of the variously described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is intended. It is also to be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated items listed. The terms “include”, “including”, “comprise”, and / or “comprising”, as used herein, refer to features, integers, and the like. Specify presence of steps, steps, actions, elements, and / or components, but exist one or more other features, integers, steps, actions, elements, components, and / or groups thereof Or it will be further understood that no addition is excluded.

The term "if", optionally, "when" or "upon" or "in response to determining" or "detection," depending on the context. It is interpreted to mean "in response to detecting." Similarly, the phrase "if determined to" or "if a condition or event mentioned] is detected" may optionally, depending on the context, "in response to determining" or "in response to" or "[ When detecting the mentioned condition or event] or "in response to detecting the [referred condition or event]".

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functions. Exemplary embodiments of portable multifunction devices restrict iPhone®, iPod Touch®, and iPad® devices from Apple Inc., Cupertino, CA, USA. Includes without. Other portable electronic devices such as laptop or tablet computers with touch sensitive surfaces (eg, touch screen displays and / or touchpads) are optionally used. In some embodiments, it should also be understood that the device is not a portable communication device but a desktop computer having a touch-sensitive surface (eg, a touch screen display and / or a touchpad).

In the discussion that follows, an electronic device is described that includes a display and a touch-sensitive surface. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, mouse and / or joystick.

The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website authoring applications, disk authoring applications, spreadsheet applications, game applications, telephony applications, video conferencing applications, email Applications, instant messaging applications, athletic assistance applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

Various applications running on the device optionally use at least one universal physical user interface device, such as a touch-sensitive surface. Corresponding information displayed on the device, as well as one or more functions of the touch-sensitive surface, are optionally coordinated and / or changed from one application to the next and / or within an individual application. In this way, the universal physical architecture (such as touch-sensitive surface) of the device optionally supports various applications using user interfaces that are intuitive and transparent to the user.

Attention is now directed to embodiments of portable devices with touch-sensitive displays. 1A is a block diagram illustrating a portable multifunction device 100 having a touch-sensitive display system 112, in accordance with some embodiments. The touch-sensitive display 112 is sometimes referred to as "touch screen" for convenience, sometimes referred to as or so called "touch-sensitive display system." Device 100 may include memory 102 (optionally including one or more computer readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripheral interface 118, RF Circuitry 108, audio circuitry 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input control devices 116, and external port 124 It includes. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact intensity sensors for detecting the intensity of contacts on device 100 (eg, a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). (165). Device 100 may optionally include a touch pad 355 of device 100 or touch sensitive display system 112 of device 100 (eg, for generating tactile outputs on device 100). One or more tactile output generators 167) for generating tactile outputs on the same touch-sensitive surface. These components optionally communicate over one or more communication buses or signal lines 103.

As used in the specification and claims, the term "strength" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area), or touch, of a contact (eg, finger contact) on the touch-sensitive surface. It refers to a substitute (proxy) for the force or pressure of contact on the sensitive surface. The intensity of the contact has a range of values that includes at least four distinct values and more typically includes hundreds (eg, at least 256) distinct values. The intensity of the contact is selectively determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine an estimated force of the contact. Similarly, a pressure sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch sensitive surface. Alternatively, the size and / or changes in the detected contact area on the touch-sensitive surface, the capacitance and / or the changes in the touch-sensitive surface near the contact, and / or the touch-sensitive surface near the contact The resistance and / or variations thereon are optionally used as a substitute for the force or pressure of contact on the touch-sensitive surface. In some implementations, alternative measurements for contact force or pressure are directly used to determine whether the intensity threshold has been exceeded (eg, the intensity threshold is described in units corresponding to the alternative measurements). In some implementations, alternative measurements for contact force or pressure are converted to estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (eg, intensity threshold). Is the pressure threshold measured in units of pressure). Using the intensity of contact as an attribute of user input may otherwise display affordances (eg, on a touch-sensitive display) and / or (eg, touch-sensitive display, touch-sensitive surface, or Enable user access to additional device functions that may not be accessible by the user on a reduced size device with limited real estate to receive user input (via a physical or mechanical control such as a knob or button) Let's do it.

As used in the specification and claims, the term “tactile output” refers to the physical displacement of the device relative to its previous position, the component of the device relative to other components of the device (eg, a housing) (eg, a touch-sensitive surface). ), Or the displacement of the component with respect to the center of mass of the device to be detected by the user using the user's tactile sense. For example, in situations where a device or component of the device is in contact with a touch sensitive user's surface (eg, a finger, palm, or other area of the user's hand), the tactile output generated by the physical displacement may be caused by the device or It will be interpreted as tactile sensation corresponding to a perceived change in the physical characteristics of the component of the device. For example, movement of a touch-sensitive surface (eg, a touch-sensitive display or trackpad) is optionally interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, the user will feel tactile, such as “down click” or “up click” even when there is no movement of the physical actuator button associated with the touch-sensitive surface that is physically pressed (eg, displaced) by the user's movement. . As another example, the movement of the touch-sensitive surface is optionally interpreted or sensed by the user as a "roughness" of the touch-sensitive surface even when there is no change in the flatness of the touch-sensitive surface. While these interpretations of touch by the user will be susceptible to the user's sensory perception, there are many touch sensory perceptions that are common to the majority of users. Thus, when tactile output is described as corresponding to a user's specific sensory perception (eg, "up click", "down click", "roughness"), unless otherwise stated, the generated tactile output is typical (or On average) corresponds to the physical displacement of the device or component thereof that will produce the described sensory perception of the user.

Device 100 is only one example of a portable multifunction device, and device 100 optionally has more or fewer components than shown, optionally, combines two or more components, or optionally a component. It should be understood that these have different configurations or arrangements. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits (ASICs).

Memory 102 optionally includes high speed random access memory, and optionally also one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Non-volatile memory such as; Memory controller 122 optionally controls access to memory 102 by other components of device 100.

Peripheral interface 118 may be used to couple input and output peripherals of the device to CPU 120 and memory 102. One or more processors 120 run or execute various sets of software programs and / or instructions stored in memory 102 to perform various functions for device 100 and to process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

Radio frequency (RF) circuitry 108 receives and transmits RF signals, also referred to as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes, but is not limited to, an antenna system, an RF transceiver, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, memory, and the like. And, well known circuitry for performing these functions. RF circuitry 108 may optionally include networks, such as the Internet, intranet, and / or wireless networks, also referred to as the World Wide Web (WWW), such as cellular telephone networks, wireless local area networks, and / or MANs. (metropolitan area network), and other devices by wireless communication. RF circuitry 108 optionally includes well-known circuitry for detecting near field communication (NFC) fields, such as by short-range communication radio. Wireless communications may optionally include Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSPAA), Evolution (EV-DO) Data-Only, HSPA, HSPA +, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA) ), Time division multiple access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and / or IEEE 802.11ac, voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging) and presence protocol), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (IMPLE), Instant Messaging and Presence Service (IMPS) , And / or a plurality of communication standards, protocols, including but not limited to any other suitable communication protocol, including, but not limited to, Short Message Service (SMS), or communication protocols not yet developed at the time of filing of this document. And any of the techniques.

The audio circuitry 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit unit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts an electrical signal into sound waves that can be heard by a person. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. The audio circuitry 110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 118 for processing. Audio data is optionally withdrawn from memory 102 and / or RF circuitry 108 and / or transmitted to memory 102 and / or RF circuitry 108 by peripherals interface 118. In some embodiments, audio circuitry 110 also includes a headset jack (eg, 212 of FIG. 2). The headset jack is an interface between the audio circuitry 110 and removable audio input / output peripherals, such as a headset having output-only headphones or an output (eg, one or both ear headphones) and an input (eg, a microphone). To provide.

I / O subsystem 106 couples input / output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, depth camera controller 169, and other input or control. One or more input controllers 160 for the devices. One or more input controllers 160 receive / transmit electrical signals from / to other input control devices 116. Other input control devices 116 optionally include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, input controller (s) 160 is optionally coupled to (or not coupled to) any of a pointer device such as a keyboard, an infrared port, a USB port, and a mouse. . One or more buttons (eg, 208 of FIG. 2) optionally include an up / down button for volume control of the speaker 111 and / or the microphone 113. One or more buttons optionally include a push button (eg, 206 of FIG. 2).

A quick press of a push button initiates the process of selectively using the gestures on the touch screen to selectively unlock the touch screen 112 or unlock the device, which is dated December 23, 2005. As described in US Patent Application No. 11 / 322,549, "Unlocking a Device by Performing Gestures on an Unlock Image," US Patent No. 7,657,849, which is hereby incorporated by reference in its entirety. . Longer pressing of a push button (eg, 206) optionally turns on or off the device 100. The functionality of one or more buttons is, optionally, user customizable. Touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

Touch-sensitive display 112 provides an input interface and an output interface between the device and the user. Display controller 156 receives and / or transmits electrical signals from / to touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

Touch screen 112 has a touch-sensitive surface, sensor or set of sensors that accepts input from a user based on haptic and / or tactile contact. The touch screen 112 and the display controller 156 are in contact with the touch screen 112 (and any movement or interruption of the contact) (along with any associated modules and / or sets of instructions in the memory 102). ) And convert the detected contact into an interaction with user interface objects (eg, one or more soft keys, icons, web pages or images) displayed on touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 optionally uses liquid crystal display (LCD) technology, light emitting polymer display (LPD) technology, or light emitting diode (LED) technology, but other display technologies are used in other embodiments. The touch screen 112 and the display controller 156 optionally select one or more contact points with capacitive, resistive, infrared, and surface acoustic wave technologies as well as other proximity sensor arrays, or the touch screen 112. Detect a contact and any movement or interruption thereof using any of a number of touch sensing techniques now known or later developed, including but not limited to other elements for determining. In an exemplary embodiment, projected mutual capacitance sensing technology such as found on iPhone® and iPod Touch® from Apple Inc., Cupertino, Calif., Is used.

Touch-sensitive displays in some embodiments of touch screen 112 may optionally include the following US Pat. Nos. 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), And / or 6,677,932 (Westerman) And / or similar to the multi-touch sensitive touchpads described in US Patent Publication No. 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, touch screen 112 displays visual output from device 100, while touch-sensitive touchpads do not provide visual output.

Touch-sensitive displays in some embodiments of touch screen 112 are described in the following applications: (1) US Patent Application No. 11 / 381,313, filed May 2, 2006, “Multipoint Touch Surface Controller "; (2) US Patent Application No. 10 / 840,862, “Multipoint Touchscreen,” filed May 6, 2004; (3) US Patent Application No. 10 / 903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) US Patent Application No. 11 / 048,264, entitled "Gestures For Touch Sensitive Input Devices," filed Jan. 31, 2005; (5) US Patent Application No. 11 / 038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices"; (6) US patent application Ser. No. 11 / 228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface"; (7) US Patent Application No. 11 / 228,700, filed September 16, 2005, "Operation Of A Computer With A Touch Screen Interface"; (8) US Patent Application No. 11 / 228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard"; And (9) US Patent Application No. 11 / 367,749, "Multi-Functional Hand-Held Device", filed March 3, 2006. All of these applications are incorporated herein by reference in their entirety.

The touch screen 112 optionally has a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. The user optionally contacts the touch screen 112 using any suitable object or appendage, such as a stylus, a finger, or the like. In some embodiments, the user interface is designed to operate primarily using finger based contacts and gestures, which may be less precise than stylus based input due to the wider contact area of a finger on the touch screen. In some embodiments, the device converts the rough finger-based input into a precise pointer / cursor position or command for performing the actions desired by the user.

In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad, unlike the touch screen, is a touch-sensitive area of the device that does not display visual output. The touchpad is optionally an extension of the touch-sensitive surface formed by the touch screen or a touch-sensitive surface separate from the touch screen 112.

Device 100 also includes a power system 162 for powering various components. The power system 162 may optionally include a power management system, one or more power sources (eg, batteries, alternating current (AC)), recharging systems, power failure detection circuits, power converters or inverters, power status indicators ( For example, light emitting diodes (LEDs), and any other components associated with the generation, management, and distribution of power in portable devices.

Device 100 also optionally includes one or more optical sensors 164. 1A shows an optical sensor coupled to an optical sensor controller 158 in I / O subsystem 106. Optical sensor 164 optionally includes charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Optical sensor 164 receives light from the environment, which is projected through one or more lenses, and converts the light into data representing an image. In conjunction with imaging module 143 (also referred to as camera module), optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the back side of the device 100 opposite the touch screen display 112 on the front of the device so that the touch screen display can be used as a viewfinder for still and / or video image acquisition. To be. In some embodiments, the optical sensor is located on the front of the device such that, while the user views other video conference participants on the touch screen display, optionally, an image of the user is obtained for the video conference. In some embodiments, the position of the light sensor 164 can be changed by the user (eg, by rotating the lens and sensor in the device housing) such that the single light sensor 164 is imaged with the touch screen display. It is used for both conference and still and / or video image acquisition.

Device 100 also optionally includes one or more contact intensity sensors 165. 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 in I / O subsystem 106. The contact intensity sensor 165 may optionally include one or more piezo resistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (eg, Sensors used to measure the force (or pressure) of the contact on the touch-sensitive surface. The contact intensity sensor 165 receives contact intensity information (eg, pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is collocated with or in proximity to a touch-sensitive surface (eg, touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back side of the device 100, opposite the touch screen display 112 located on the front side of the device 100.

Device 100 also optionally includes one or more proximity sensors 166. 1A shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 is optionally coupled to input controller 160 in I / O subsystem 106. Proximity sensor 166 may optionally include US Patent Applications 11 / 241,839, "Proximity Detector In Handheld Device"; 11 / 240,788, "Proximity Detector In Handheld Device"; 11 / 620,702, "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, "Automated Response To And Sensing Of User Activity In Portable Devices"; And 11 / 638,251, "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, the proximity sensor turns off and disables the touch screen 112 when the multifunction device is located near the user's ear (eg, when the user is making a phone call).

Device 100 also optionally includes one or more tactile output generators 167. 1A shows a tactile output generator coupled to haptic feedback controller 161 in I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components and / or a motor, solenoid, electroactive polymer, piezo actuator, electrostatic actuator, or other tactile output generation. Electromechanical devices that convert energy into linear motion, such as a component (eg, a component that converts electrical signals into tactile outputs on the device). The contact intensity sensor 165 receives tactile feedback generation instructions from the haptic feedback module 133 and generates tactile outputs on the device 100 that can be sensed by the user of the device 100. In some embodiments, the at least one tactile output generator is located with or close to the touch-sensitive surface (eg, touch-sensitive display system 112), and optionally, perpendicularly (eg, to) the touch-sensitive surface. To produce tactile output by moving in or out of the surface of device 100 or laterally (eg, back and forth in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back side of the device 100 opposite the touch screen display 112 located on the front side of the device 100.

Device 100 also optionally includes one or more accelerometers 168. 1A shows accelerometer 168 coupled to peripherals interface 118. Alternatively, accelerometer 168 is optionally coupled to input controller 160 in I / O subsystem 106. Accelerometer 168 may optionally include US Patent Publication No. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices" and US Patent Publication No. 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An". Accelerometer, "both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on a touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer (not shown), in addition to the accelerometer (s) 168, and a GPS for obtaining information regarding the position and orientation (eg, vertical or horizontal) of the device 100. Or GLONASS or other global navigation system) receiver (not shown).

Device 100 also optionally includes one or more depth camera sensors 175. 1A shows a depth camera sensor coupled to depth camera controller 169 in I / O subsystem 106. Depth camera sensor 175 receives data from the environment, which is projected through the sensor. In conjunction with imaging module 143 (also referred to as camera module), depth camera sensor 175 is optionally used to determine depth maps of different portions of the image captured by imaging module 143. In some embodiments, the depth camera sensor is located on the front of the device 100 to capture video for the video conference and to capture selfies with depth map data while the user is viewing other video conference participants on the touch screen display. , Optionally, allows an image of a user with depth information to be obtained. In some embodiments, the position of the depth camera sensors 175 can be changed by the user (eg, by rotating the lens and sensor in the device housing) such that the depth camera sensors 175 can be combined with the touch screen display. It is used for both video conferencing and still and / or video image acquisition.

In some embodiments, software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact / motion module (or set of instructions) 130, graphics module ( Or a set of instructions) 132, a text input module (or set of instructions) 134, a GPS module (or set of instructions) 135, and applications (or sets of instructions) 136. . In addition, in some embodiments, memory 102 of FIG. 1A or 370 of FIG. 3 stores device / global internal state 157 as shown in FIGS. 1A and 3. The device / global internal state 157, if present, includes an active application state indicating which applications are currently active; Display status indicating which applications, views, or other information occupy various areas of touch screen display 112; Sensor status including information obtained from various sensors of the device and input control devices 116; And location information regarding the location and / or attitude of the device.

Operating system 126 (e.g., an embedded operating system such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or VxWorks) provides general system tasks (e.g., memory management, storage device control, power management, etc.). Various software components and / or drivers for controlling and managing the network, and facilitating communication between various hardware and software components.

The communication module 128 enables communication with other devices via one or more external ports 124, and also for processing data received by the RF circuitry 108 and / or the external port 124. Contains software components. The external port 124 (eg, USB, FIREWIRE, etc.) is configured to couple directly to other devices or indirectly through a network (eg, the Internet, a wireless LAN, etc.). In some embodiments, the external port is a multi-pin (eg, 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used in iPod® (trademark of Apple Inc.) devices.

The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156), and other touch-sensitive devices (eg, a touchpad or physical click wheel). . The contact / motion module 130 determines whether a contact has occurred (eg, detecting a finger-down event), the strength of the contact (eg, the force or pressure of the contact, or Determining a substitute for the force or pressure of the contact, determining whether there is a movement of the contact, and tracking the movement across the touch-sensitive surface (eg, one or more finger-dragging events). ), And determining whether the contact has been stopped (eg, detecting a finger-up event or contact interruption), It includes various software components for performing. The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by the series of contact data optionally determines the speed (magnitude), speed (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. It involves doing. These actions optionally apply to single contacts (eg, one finger contacts) or to multiple simultaneous contacts (eg, “multi-touch” / multiple finger contacts). In some embodiments, contact / motion module 130 and display controller 156 detect a contact on the touch pad.

In some embodiments, contact / motion module 130 may determine one or more of the intensity thresholds to determine whether the action was performed by the user (eg, to determine whether the user “clicked” over the icon). Use a set. In some embodiments, at least a subset of the intensity thresholds is determined in accordance with the software parameters (eg, the intensity thresholds are not determined by the activation thresholds of specific physical actuators, and changing the physical hardware of the device 100 is Can be adjusted without). For example, the mouse “click” threshold of a trackpad or touch screen display can be set to any of a wide range of predefined thresholds without changing the trackpad or touch screen display hardware. In addition, in some implementations, a user of the device can be one of a set of intensity thresholds (eg, by adjusting individual intensity thresholds and / or by adjusting a plurality of intensity thresholds all at once with a system level click “intensity” parameter). Software settings for adjusting the above are provided.

The contact / motion module 130 optionally detects a gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different motions, timings, and / or intensities of detected contacts). Thus, the gesture is optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture may include detecting a finger-down event followed by a finger-at the same location (or substantially the same location) as the finger-down event (eg, at the icon's location). Detecting an up (liftoff) event. As another example, detecting a finger swipe gesture on a touch-sensitive surface detects a finger-down event followed by one or more finger-dragging events, followed by finger-up (liftoff). ) Detecting the event.

Graphics module 132 renders graphics on touch screen 112 or other displays, including components for modifying the visual effects (eg, brightness, transparency, saturation, contrast, or other visual properties) of the displayed graphics. It includes various known software components for display. As used herein, the term “graphics” includes without limitation text, web pages, icons (eg, user interface objects including soft keys), digital images, videos, animations, and the like. , Any object that can be displayed to the user.

In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from applications or the like that specify graphics to be displayed, along with coordinate data and other graphics attribute data, if necessary, and then generates screen image data to display controller 156. Output

The haptic feedback module 133 is used by the haptic output generator (s) 167 to generate tactile outputs at one or more locations on the device 100 in response to user interactions with the device 100. It includes various software components for generating instructions.

Optionally, the text input module 134, which is a component of the graphics module 132, may require various applications (eg, contact 137, email 140, IM 141, browser 147, and text input). Soft keyboards for inputting text into any other application).

The GPS module 135 determines the location of the device and uses this information to call 138 for use in various applications (eg, for use in location based dialing; camera 143 as photo / video metadata). And to applications that provide location based services such as weather widgets, local yellow page widgets and map / navigation widgets.

The applications 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof:

연락처 모듈(137)(때때로 주소록 또는 연락처 목록으로 지칭됨);

A contact module 137 (sometimes referred to as an address book or contact list);

전화 모듈(138);

Telephone module 138;

화상 회의 모듈(139);

Video conferencing module 139;

이메일 클라이언트 모듈(140);

Email client module 140;

인스턴트 메시징(IM) 모듈(141);

Instant messaging (IM) module 141;

운동 지원 모듈(142);

Exercise support module 142;

정지 및/또는 비디오 이미지들을 위한 카메라 모듈(143);

Camera module 143 for still and / or video images;

이미지 관리 모듈(144);

Image management module 144;

비디오 재생기 모듈;

Video player module;

음악 재생기 모듈;

A music player module;

브라우저 모듈(147);

Browser module 147;

캘린더 모듈(148);

Calendar module 148;

날씨 위젯(149-1), 주식 위젯(149-2), 계산기 위젯(149-3), 알람 시계 위젯(149-4), 사전 위젯(149-5), 및 사용자에 의해 얻어지는 다른 위젯들뿐 아니라 사용자-생성 위젯들(149-6) 중 하나 이상을 선택적으로 포함하는 위젯 모듈들(149);

Only the weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user Widget modules 149, optionally including one or more of user-created widgets 149-6;

사용자-생성 위젯들(149-6)을 만들기 위한 위젯 생성기 모듈(150);

Widget generator module 150 for creating user-created widgets 149-6;

검색 모듈(151);

Search module 151;

비디오 재생기 모듈 및 음악 재생기 모듈을 통합하는 비디오 및 음악 재생기 모듈(152);

A video and music player module 152 that integrates a video player module and a music player module;

메모 모듈(153);

A memo module 153;

지도 모듈(154); 및/또는

Map module 154; And / or

온라인 비디오 모듈(155).

Online video module 155.

Examples of other applications 136, optionally stored in memory 102, include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital copyrights. Management, speech recognition and speech replication.

Together with the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the contact module 137 may optionally include an address book or a contact list ( For example, stored in the application internal state 192 of the contact module 137 in the memory 102 or the memory 370. This includes: adding name (s) to the address book; ; Deleting name (s) from the address book; Associating telephone number (s), email address (es), physical address (es) or other information with a name; Associating an image with a name; Categorizing and sorting names; Providing telephone numbers or email addresses to initiate and / or facilitate communication by telephone 138, video conferencing module 139, email 140, or IM 141.

RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text In conjunction with the input module 134, the telephone module 138 optionally enters a sequence of characters corresponding to the telephone number, accesses one or more telephone numbers in the contact module 137, and modifies the entered telephone number. It is used to dial individual phone numbers, to have a conversation, and to disconnect or disconnect when the conversation is complete. As mentioned above, wireless communication optionally uses any of a plurality of communication standards, protocols and technologies.

RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, light sensor 164, light sensor controller 158, contact / motion In conjunction with module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 may be associated with the user and one or more other users. Executable instructions to initiate, initiate, and end a video conference between the participants.

Together with the RF circuitry 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the e-mail client module 140 provides user instructions. Executable instructions to respond to, create, send, receive, and manage email. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails with still or video images taken with the camera module 143.

The instant messaging module 141, along with the RF circuitry 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, is instantaneous. Enter a sequence of characters corresponding to the message, modify previously entered characters (eg, Short Message Service (SMS) or Multimedia Message Service for phone-based instant messages) , MMS) protocol, or XMPP, SIMPLE or IMPS for Internet-based instant messages), executable instructions for sending individual instant messages, receiving instant messages, and viewing received instant messages. . In some embodiments, sent and / or received instant messages optionally include graphics, pictures, audio files, video files and / or other attachments as supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to telephone-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, messages sent using XMPP, SIMPLE or IMPS). S) both.

RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154 And, together with the music player module, exercise support module 142 devises exercises (eg, with time, distance, and / or calorie consumption goals); Communicate with athletic sensors (sports devices); Receive exercise sensor data; Calibrate sensors used to monitor exercise; Select and play music for a workout; Executable instructions for displaying, storing, and transmitting athletic data.

With touch screen 112, display controller 156, optical sensor (s) 164, optical sensor controller 158, contact / motion module 130, graphics module 132 and image management module 144. Camera module 143 captures still images or videos (including video streams) and stores them in memory 102, modifies the characteristics of still images or videos, or captures still images or videos from memory 102. Executable instructions for deleting the video.

Together with the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the camera module 143, the image management module 144 stops and And / or executable instructions for arranging, modifying (eg, editing), or otherwise manipulating, labeling, deleting, presenting (eg, in a digital slide show or album) and / or storing video images. .

The browser module 147, along with the RF circuitry 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132 and the text input module 134, is a web page. Executable instructions to browse the Internet in accordance with user instructions, including searching for, linking to, receiving, and displaying attachments and other files linked to web pages, as well as portions thereof or portions thereof Include them.

RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module ( In conjunction with 147, the calendar module 148 is executable to generate, display, modify, and store calendars and data associated with calendars (eg, calendar entries, to-do lists, etc.) in accordance with user instructions. Contains instructions.

Widget modules, along with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134 and browser module 147. 149 may be selectively downloaded and used by the user (eg, weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary). Widget 149-5), or mini-applications generated by a user (eg, user-generated widget 149-6). In some embodiments, the widget includes a Hypertext Markup Language (HTML) file, a Cascading Style Sheets (CSS) file, and a JavaScript (JavaScript) file. In some embodiments, the widget includes an Extensible Markup Language (XML) file and a JavaScript file (eg, Yahoo! widgets).

Widget generator module, with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134 and browser module 147 150 is optionally used by the user to create widgets (eg, changing a user specific portion of the web page to a widget).

In conjunction with the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the search module 151 may generate one or more search criteria according to user instructions. Executable instructions for retrieving text, music, sound, image, video, and / or other files in memory 102 that match ones (eg, one or more user-specific search terms).

With touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108 and browser module 147 The video and music player module 152 executes instructions that allow a user to download and play recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and videos. Executable instructions to display, display, or otherwise play back on the touch screen 112 or on an externally connected display via an external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

Together with the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the memo module 153 is capable of taking notes, to do according to user instructions. Executable instructions for creating and managing to-do lists and the like.

RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147 Map module 154 optionally includes, in accordance with user instructions, data associated with the maps and maps (eg, driving directions; data relating to shops and other points of interest at or near a particular location); And other location-based data).

Touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, In conjunction with email client module 140 and browser module 147, online video module 155 allows a user to access, browse, and (eg, stream and / or) online videos in one or more file formats, such as H.264. Or by downloading), playing (e.g., on a touch screen or on an externally connected display via external port 124), sending an email with a link to a particular online video, and otherwise managing it. Contains instructions. In some embodiments, instant messaging module 141 rather than email client module 140 is used to send a link to a particular online video. Additional descriptions of online video applications are described in U.S. Provisional Patent Application 60 / 936,562, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," filed June 20, 2007 and 31 December 2007. See, for example, US Patent Application No. 11 / 968,067, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," which is hereby incorporated by reference in its entirety, which is hereby incorporated by reference in its entirety. .

Each of the modules and applications identified above are executable instructions for performing one or more of the functions described above and the methods described herein (eg, computer-implemented methods and other information processing methods described herein). Corresponds to the set. These modules (eg, sets of instructions) need not be implemented as separate software programs, procedures or modules, so that various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. . For example, the video player module is optionally combined with the music player module into a single module (eg, the video and music player module 152 of FIG. 1A). In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. In addition, the memory 102 optionally stores additional modules and data structures not described above.

In some embodiments, device 100 is a device in which operation of a predefined set of functions on the device is performed exclusively through the touch screen and / or touchpad. By using the touch screen and / or touchpad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, etc.) on device 100 is selectively reduced.

The predefined set of functions performed solely through the touch screen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to the main, home, or root menu. In these embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of the touchpad.

1B is a block diagram illustrating example components for event processing in accordance with some embodiments. In some embodiments, memory (102 of FIG. 1A or 370 of FIG. 3) may include event classifier 170 (eg, in operating system 126) and individual applications 136-1 (eg, the applications described above). (137-151, 155, 380-390).

The event classifier 170 receives the event information, and determines an application 136-1 and an application view 191 of the application 136-1 to deliver the event information. The event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that represents the current application view (s) displayed on touch-sensitive display 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application (s) are currently active, and application internal state 192 is used by event classifier 170. Used to determine application views 191 to convey event information.

In some embodiments, application internal state 192 is a user indicating resume information to be used when application 136-1 resumes execution, information being displayed by or ready to be displayed by application 136-1. Interface state information, a state queue to enable the user to return to the previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user. Include additional information such as one or more.

Event monitor 171 receives event information from peripherals interface 118. The event information includes information about the subevent (eg, user touch on the touch-sensitive display 112 as part of a multi-touch gesture). Peripheral interface 118 may be from I / O subsystem 106 or a sensor such as proximity sensor 166, accelerometer (s) 168, and / or microphone 113 (via audio circuitry 110). Send the information you receive. The information that peripheral interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, peripheral interface 118 transmits event information only when there is a significant event (eg, receiving input exceeding a predetermined noise threshold and / or input during a predetermined duration exceeding). .

In some embodiments, event classifier 170 also includes a hit view determination module 172 and / or an active event recognizer determination module 173.

The hit view determination module 172 provides software procedures for determining where a subevent occurred within one or more views when the touch-sensitive display 112 displays more than one view. Views consist of controls and other elements that a user can see on the display.

Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. Application views (of individual applications) in which a touch is detected optionally correspond to program levels in the application's program or view hierarchy. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as appropriate inputs is, optionally, based at least in part on the hit view of the initial touch initiating a touch based gesture. Is determined.

The hit view determination module 172 receives information related to subevents of a touch based gesture. If the application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should handle the subevent. In most situations, the hit view is the lowest level view in which an initiating subevent (eg, a first subevent in a sequence of subevents forming an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all subevents associated with the same touch or input source that caused it to be identified as a hit view.

The active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of subevents. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of subevents. In other embodiments, the active event recognizer determination module 173 determines that all views including the physical location of the subevent are actively involved views, so that all actively involved views are specific to the subevents. Determine that you need to receive the sequence. In other embodiments, even if touch subevents are entirely confined to an area associated with one particular view, the parent views in the hierarchy will still remain as actively participating views.

Event dispatcher module 174 dispatches event information to an event recognizer (eg, event recognizer 180). In embodiments that include an active event recognizer determination module 173, the event dispatcher module 174 delivers event information to an event recognizer determined by the active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores event information in an event queue, where the event information is retrieved by individual event receiver 182.

In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In still other embodiments, the event classifier 170 is a standalone module or part of another module stored in the memory 102, such as the contact / motion module 130.

In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which is a touch event occurring within each view of the user interface of the application. Instructions for processing them. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a high level object from which application 136-1 inherits methods and other attributes. to be. In some embodiments, the individual event handler 190 is one of the data updater 176, the object updater 177, the GUI updater 178, and / or the event data 179 received from the event classifier 170. It includes the above. Event handler 190 optionally uses or invokes data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views 191 include one or more respective event handlers 190. Further, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included within individual application view 191.

Individual event recognizer 180 receives event information (eg, event data 179) from event classifier 170 and identifies the event from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (optionally including subevent delivery instructions).

The event receiver 182 receives event information from the event classifier 170. The event information includes information regarding sub-events, for example touch or touch movement. Depending on the subevent, the event information also includes additional information such as the location of the subevent. If the subevent is related to the motion of the touch, the event information also optionally includes the speed and direction of the subevent. In some embodiments, the events include rotation of the device from one orientation to another (eg, from vertical orientation to landscape orientation, or vice versa), wherein the event information is referred to as the device's current orientation (also referred to as the device attitude). Corresponding information).

The event comparator 184 compares the event information with predefined event or subevent definitions, and based on the comparison, determines the event or subevent, or determines or updates the status of the event or subevent. In some embodiments, event comparator 184 includes event definitions 186. Event definitions 186 include definitions of events (eg, predefined sequences of subevents), for example event 1 187-1, event 2 187-2, and the like. In some embodiments, subevents within event 187 include, for example, touch start, touch end, touch move, touch cancel, and multiple touch. In one example, the definition for event 1 187-1 is a double tap on the displayed object. The double tap is, for example, a first touch on the displayed object during a predetermined phase (start of touch), a first liftoff during the predetermined phase (end of touch), a displayed object during the predetermined phase. A second touch on the touch (start of touch), and a second liftoff (touch end) during a predetermined phase. In another example, the definition for event 2 187-2 is dragging on the displayed object. Dragging includes, for example, a touch (or touch) on the displayed object during a predetermined phase, the movement of the touch across the touch-sensitive display 112, and the liftoff (touch end) of the touch. In some embodiments, the event also includes information about one or more associated event handlers 190.

In some embodiments, event definition 187 includes a definition of an event for a respective user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on the touch-sensitive display 112, when a touch is detected on the touch-sensitive display 112, the event comparator 184 may display three user interface objects. A heat test is performed to determine which of these are associated with the touch (subevent). When each displayed object is associated with an individual event handler 190, the event comparator uses the results of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the object and subevent that triggers the hit test.

In some embodiments, the definition for each event 187 also includes delayed actions that delay delivery of event information until after determining whether the sequence of subevents corresponds to or does not correspond to the event type of the event recognizer. .

If the individual event recognizer 180 determines that the series of subevents does not match any of the events in the event definitions 186, the individual event recognizer 180 is unable to event, event fails, or The event exit state is entered, and then the individual event recognizer ignores subsequent subevents of the touch based gesture. In such a situation, other event recognizers, if any, that remain active for the hit view continue to track and process subevents of an ongoing touch-based gesture.

In some embodiments, the individual event recognizer 180 may configure configurable attributes, flags, and / or lists that indicate how the event delivery system should perform subevent delivery for actively participating event recognizers. It has metadata 183 having. In some embodiments, metadata 183 includes configurable attributes, flags, and / or lists that indicate how event recognizers interact or become interactable with each other. In some embodiments, metadata 183 includes configurable attributes, flags, and / or lists that indicate whether subevents are delivered at various levels in the view or program hierarchy.

In some embodiments, the individual event recognizer 180 activates an event handler 190 associated with the event when one or more specific subevents of the event are recognized. In some embodiments, individual event recognizer 180 passes event information associated with the event to event handler 190. Activating the event handler 190 is separate from sending (and delaying sending) subevents to individual hit views. In some embodiments, event recognizer 180 sends a flag associated with the recognized event, and event handler 190 associated with the flag catches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include subevent delivery instructions that convey event information about a subevent without activating the event handler. Instead, subevent delivery instructions deliver event information to event handlers associated with a series of subevents or to actively participating views. Event handlers associated with the series of subevents or actively participating views receive the event information and perform a predetermined process.

In some embodiments, data updater 176 generates and updates data used in application 136-1. For example, the data updater 176 updates the phone number used in the contact module 137 or stores the video file used in the video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, the object updater 177 creates a new user interface object or updates the location of the user interface object. GUI updater 178 updates the GUI. For example, the GUI updater 178 prepares display information for display on the touch-sensitive display and sends it to the graphics module 132.

In some embodiments, event handler (s) 190 includes or accesses data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included within a single module of individual application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

The foregoing discussion regarding event handling of user touches on touch-sensitive displays also applies to other forms of user inputs for operating multifunction devices 100 having input devices, all of which are disclosed on touch screens. It should be understood that it is not. For example, mouse movement and mouse button presses, optionally coordinated with single or multiple keyboard presses or hold; Contact movements such as tap, drag, scroll, and the like on the touchpad; Pen stylus inputs; Movement of the device; Verbal instructions; Detected eye movements; Biometric inputs; And / or any combination thereof are optionally used as inputs corresponding to subevents defining an event to be recognized.

2 illustrates a portable multifunction device 100 having a touch screen 112 in accordance with some embodiments. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In these embodiments as well as in other embodiments described below, the user may, for example, have one or more fingers 202 (not shown to scale in the figure) or one or more stylus 203 (scale in the drawing). It is possible to select one or more of the graphics by making a gesture on the graphics (not shown). In some embodiments, the selection of one or more graphics occurs when the user stops contacting the one or more graphics. In some embodiments, the gesture may optionally include one or more taps of a finger in contact with device 100, one or more swipes (from left to right, from right to left, up and / or down) And / or rolling (right to left, left to right, up and / or down). In some implementations or situations, the graphic is not selected if it is inadvertently contacted with the graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over the application icon optionally does not select the corresponding application.

Device 100 also optionally includes one or more physical buttons, such as “home” or menu button 204. As described above, the menu button 204 is optionally used to navigate to any application 136 in the set of applications that are selectively executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in the GUI displayed on the touch screen 112.

In some embodiments, device 100 includes touch screen 112, menu button 204, push button 206 to turn the device on and off and lock the device, volume control button (s) 208. Subscriber identity module (SIM) card slot 210, headset jack 212, and docking / charging external port 124. Push button 206 may optionally turn on / off the device by pressing the button and holding the button pressed for a predefined time interval; Lock the device by pressing the button and releasing the button before the predefined time interval elapses; Used to unlock the device or initiate the unlock process. In alternative embodiments, device 100 also accepts verbal input for activation or deactivation of some functions via microphone 113. The device 100 may also optionally generate tactile outputs for the user of the device 100 and / or one or more contact intensity sensors 165 to detect the intensity of the contacts on the touch screen 112. One or more tactile output generators 167 for.

3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, in accordance with some embodiments. Device 300 need not be portable. In some embodiments, device 300 may be a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (eg, home or industrial). Controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses for interconnecting these components ( 320). Communication buses 320 optionally include circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between them. Device 300 includes an input / output (I / O) interface 330 that includes a display 340, which is typically a touch screen display. I / O interface 330 is also optionally a tactile output for generating tactile outputs on keyboard and / or mouse (or other pointing device) 350 and touchpad 355, device 300. Generator 357 (eg, similar to tactile output generator (s) 167 described above with reference to FIG. 1A), and sensors 359 (eg, light sensor, acceleration sensor, proximity sensor, touch-sensitive) Sensor, and / or contact intensity sensors similar to the contact intensity sensor (s) 165 described above with reference to FIG. 1A). Memory 370 includes fast random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; Optionally include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU (s) 310. In some embodiments, memory 370 is programs, modules, and data structures similar to programs, modules, and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1A). , Or a subset thereof. Also, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of portable multifunction device 100. For example, the memory 370 of the device 300 may optionally include a drawing module 380, a presentation module 382, a word processing module 384, a website authoring module 386, a disk authoring module 388. ) And / or spreadsheet module 390, while memory 102 of portable multifunction device 100 (FIG. 1A) optionally does not store such modules.

Each of the above identified elements in FIG. 3 is optionally stored in one or more of the aforementioned memory devices. Each of the above identified modules corresponds to a set of instructions for performing the above-described function. The modules or programs (eg, sets of instructions) identified above do not need to be implemented as separate software programs, procedures or modules, so that various subsets of these modules may optionally be combined in various embodiments. Or otherwise rearranged. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Also, memory 370 optionally stores additional modules and data structures not described above.

Attention is now directed to embodiments of user interfaces that are optionally implemented, for example, on the portable multifunction device 100.

4A illustrates an example user interface for a menu of applications on portable multifunction device 100, in accordance with some embodiments. Similar user interfaces are optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

셀룰러 및 Wi-Fi 신호들과 같은 무선 통신(들)을 위한 신호 세기 표시자(들)(402);

Signal strength indicator (s) 402 for wireless communication (s), such as cellular and Wi-Fi signals;

시간(404);

Time 404;

블루투스 표시자(405);

A Bluetooth indicator 405;

배터리 상태 표시자(406);

Battery status indicator 406;

다음과 같은, 빈번하게 사용되는 애플리케이션들에 대한 아이콘들을 갖는 트레이(408):

Tray 408 with icons for frequently used applications, such as:

o An icon 416 for telephony module 138 labeled “telephone” optionally including an indicator 414 of the number of missed calls or voicemail messages;

o An icon 418 for the email client module 140 labeled "mail" that optionally includes an indicator 410 of the number of unread emails;

o An icon 420 for the browser module 147 labeled “Browser”; and

o Icon 422 for video and music player module 152 labeled “iPod”, also referred to as iPod (trademark of Apple Inc.) module 152; and

다음과 같은, 다른 애플리케이션들에 대한 아이콘들:

Icons for other applications, such as:

o An icon 424 for the IM module 141 labeled "Message";

o An icon 426 for the calendar module 148 labeled "Calendar";

o An icon 428 for the image management module 144 labeled "Photo";

o An icon 430 for camera module 143 labeled “camera”;

o An icon 432 for the online video module 155 labeled "online video";

o Icon 434 for stock widget 149-2, labeled “Stock”;

o An icon 436 for the map module 154 labeled "Map";

o An icon 438 for the weather widget 149-1 labeled "Weather";

o An icon 440 for the alarm clock widget 149-4 labeled "Clock";

o Icon 442 for workout support module 142 labeled “Workout Support”;

o An icon 444 for the note module 153 labeled “Notes”; and

o Icon 446 for the settings application or module, labeled “Settings”, which provides access to the settings for device 100 and its various applications 136.

It should be noted that the icon labels shown in FIG. 4A are merely illustrative. For example, icon 422 for video and music player module 152 is labeled "Music" or "Music Player." Other labels are optionally used for various application icons. In some embodiments, the label for the individual application icon includes the name of the application corresponding to the individual application icon. In some embodiments, the label for a particular application icon is separate from the name of the application corresponding to the particular application icon.

FIG. 4B illustrates a device (eg, FIG. 3) having a touch-sensitive surface 451 (eg, tablet or touchpad 355 of FIG. 3) separate from display 450 (eg, touch screen display 112). An example user interface on device 300 is shown. Device 300 may also optionally include one or more contact intensity sensors (eg, one or more of sensors 359) and / or device 300 for detecting the intensity of contacts on touch-sensitive surface 451. One or more tactile output generators 357 for generating tactile outputs for the user.

Some of the following examples will be provided with reference to inputs on touch screen display 112 (combined with touch-sensitive surface and display), but in some embodiments, the device is separate from the display as shown in FIG. 4B. Detect inputs on a personal touch-sensitive surface. In some embodiments, the touch-sensitive surface (eg, 451 of FIG. 4B) has a major axis (eg, 452 of FIG. 4B) corresponding to the major axis (eg, 453 of FIG. 4B) on the display (eg, 450). According to these embodiments, the device may interact with the touch-sensitive surface 451 at locations corresponding to individual locations on the display (eg, in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470). Detect contacts (eg, 460 and 462 of FIG. 4B). In this way, user inputs (eg, contacts 460, 462 and their movements) detected by the device on the touch-sensitive surface (eg, 451 of FIG. 4B) are such that the touch-sensitive surface is separate from the display. When used by the device to manipulate the user interface on the display of the multifunction device (eg, 450 of FIG. 4B). It should be understood that similar methods are optionally used for the other user interfaces described herein.

Additionally, while the following examples are given with reference to finger inputs (eg, finger contacts, finger tap gestures, finger swipe gestures), in some embodiments, one or more of the finger inputs may be connected to another input device. It should be understood that it is replaced by input from (eg, mouse based input or stylus input). For example, the swipe gesture is optionally replaced by mouse click (eg, instead of contact) and movement of the cursor along the path of the subsequent swipe (eg, instead of movement of the contact). As another example, the tap gesture is optionally replaced with a mouse click while the cursor is positioned over the location of the tap gesture (eg, instead of stopping detecting the contact following detection of the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are selectively used simultaneously, or that mouse and finger contacts are selectively used simultaneously.

5A illustrates an example personal electronic device 500. Device 500 includes a body 502. In some embodiments, device 500 may include some or all of the features described with respect to devices 100, 300 (eg, FIGS. 1A-4B). In some embodiments, device 500 has a touch-sensitive display screen 504 (hereinafter touch screen 504). Alternatively or in addition to touch screen 504, device 500 has a display and a touch-sensitive surface. Like the devices 100, 300, in some embodiments, the touch screen 504 (or touch-sensitive surface) is, optionally, one for detecting the intensity of applied contacts (eg, touches). It includes the above intensity sensor. One or more intensity sensors of touch screen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touches. The user interface of device 500 may respond to touches based on the strength of the touches, meaning that touches of different intensities may invoke different user interface operations on device 500.

Exemplary techniques for detecting and processing touch intensity are described, for example, in related applications: May 8, 2013 and entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application, an international patent application PCT / US2013 / 040061, published as WIPO Publication No. WO / 2013/169849, and filed Nov. 11, 2013 and entitled "Device, Method, and Graphical User Interface" for Transitioning Between Touch Input to Display Output Relationships, "International Patent Application PCT / US2013 / 069483, published as WIPO Publication No. WO / 2014/105276, each of which is incorporated herein by reference in its entirety.

In some embodiments, device 500 has one or more input mechanisms 506, 508. The input mechanisms 506, 508 (if included) may be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) are such that the device 500 may be, for example, a hat, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, chains, pants, belts, shoes, wallets. It can be attached to a backpack, etc. These attachment mechanisms allow the device 500 to be worn by a user.

5B shows an example personal electronic device 500. In some embodiments, device 500 may include some or all of the components described with respect to FIGS. 1A, 1B, and 3. Device 500 has a bus 512 that operatively couples I / O section 514 with one or more computer processors 516 and memory 518. I / O section 514 may be connected to display 504, which may have a touch-sensitive component 522, and, optionally, an intensity sensor 524 (eg, a contact intensity sensor). In addition, I / O section 514 may include communication unit 530 to receive application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication techniques. It can be connected with. Device 500 may include input mechanisms 506 and / or 508. The input mechanism 506 is optionally a rotatable input device or for example a pushable and rotatable input device. In some examples, input mechanism 508 is, optionally, a button.

In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 may optionally include GPS sensor 532, accelerometer 534, direction sensor 540 (eg, compass), gyroscope 536, motion sensor 538, and / or their Various sensors, such as combinations, may all be operatively connected to the I / O section 514.

The memory 518 of the personal electronic device 500 may include one or more non-transitory computer readable storage media for storing computer executable instructions, and the computer executable instructions may include, for example, one or more computer processors ( Computer processor includes processes 700, 900, 1100, 1300, 1500, and 1700 (FIGS. 7, 9, 11, 13, 15, and 17), when executed by 516). It may be possible to perform the techniques described below. A computer readable storage medium can be any medium that can tangibly contain or store computer executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is temporary computer readable storage medium. In some examples, the storage medium is a non-transitory computer readable storage medium. Non-transitory computer readable storage media may include, but are not limited to, magnetic, optical, and / or semiconductor storage. Examples of such storage include optical disks based on magnetic disks, CD, DVD, or Blu-ray technologies, as well as persistent solid state memory such as flash, solid state drives, and the like. The personal electronic device 500 is not limited to the components and configurations of FIG. 5B, and may include other or additional components in a number of configurations.

As used herein, the term " affordance " refers to user interaction that is selectively displayed on the display screen of devices 100, 300, and / or 500 (FIGS. 1A, 3, and 5A and 5B). Refers to a user-interactive graphical user interface object. For example, images (eg icons), buttons, and text (eg hyperlinks) each optionally constitute an affordance.

As used herein, the term "focus selector" refers to an input element that represents the current portion of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the touch-sensitive surface (eg, FIG. 3) while the cursor is over a particular user interface element (eg, a button, window, slider or other user interface element). When an input (eg, a press input) is detected on the touchpad 355 of FIG. 4 or the touch-sensitive surface 451 of FIG. 4B, the cursor is “focus selector” such that a particular user interface element is adjusted according to the detected input. Function as. Some implementations include a touch screen display (eg, touch sensitive display system 112 of FIG. 1A or touch screen 112 of FIG. 4A) that enables direct interaction with user interface elements on the touch screen display. In particular, when an input (eg, a press input by contact) is detected on a touch screen display at a location of a particular user interface element (eg, a button, window, slider or other user interface element), the particular user interface element is detected. The contact detected on the touch screen functions as a "focus selector" so that it is adjusted according to the input. In some implementations, the focus is one of the user interface without moving the corresponding cursor or moving the contact on the touch screen display (eg, by using a tab key or an arrow key to move the focus from one button to another). Is moved from the area of to another area of the user interface; In such implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the particular form of the focus selector, the focus selector is generally used to convey the user's intended interaction with the user interface (eg, by presenting to the device an element of the user interface the user wishes to interact with). Is a user interface element (or contact on a touch screen display) controlled by the. For example, while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of the focus selector (e.g., cursor, contact, or selection box) on an individual button is placed on the display of the device. Unlike the other user interface elements shown, this will indicate that the user is trying to activate an individual button.

As used in the specification and claims, the term "characteristic strength" of a contact refers to the characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity may optionally include a predefined number of intensity samples, or (eg, after detecting a contact, prior to detecting a liftoff of the contact, before or after detecting the start of the movement of the contact, A predetermined period of time (eg, before detecting an end of contact, before or after detecting an increase in the intensity of the contact, and / or before or after detecting a decrease in the intensity of the contact) (eg, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) based on the set of intensity samples collected. The characteristic intensity of a contact may optionally include a maximum value of the intensities of the contact, a mean value of the intensities of the contact, an average value of the intensities of the contact, and a top 10 percentile value of the intensities of the contact. ), Half of the maximum value of the intensities of the contact, 90 percent of the maximum value of the intensities of the contact, and the like. In some embodiments, the duration of the contact is used to determine the characteristic strength (eg, when the characteristic strength is an average of the strength of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether the operation was performed by the user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, as a result of the contact having a characteristic strength that does not exceed the first threshold, the first operation is performed and as a result of the contact having a characteristic strength that exceeds the first intensity threshold but does not exceed the second intensity threshold, the second An operation is performed, and a third operation is performed as a result of the contact having a characteristic strength above the second threshold. In some embodiments, the comparison between the characteristic strength and the one or more thresholds is used to determine whether to perform one or more actions (eg, perform individual actions) rather than being used to determine whether to perform a first action or a second action. Or to suspend performing an individual action).

FIG. 5C illustrates detecting a plurality of contacts 552A through 552E on a touch-sensitive display screen 504 using a plurality of intensity sensors 524A through 524D. 5C further includes intensity diagrams showing current intensity measurements of intensity sensors 524A-524D for intensity units. In this example, the intensity measurements of the intensity sensors 524A and 524D are each nine intensity units, and the intensity measurements of the intensity sensors 524B and 524C are each seven intensity units. In some implementations, the total intensity is the sum of the intensity measurements of the plurality of intensity sensors 524A-524D, which in this example is 32 intensity units. In some embodiments, each contact is assigned an individual intensity that is part of the total intensity. 5D shows assigning total intensity to contacts 552A-552E based on their distance from the center of force 554. In this example, the contacts 552A, 552B, 552E are each assigned an intensity of contact in eight intensity units of the total intensity, and the contacts 552C, 552D each have an intensity of contact in four intensity units of the total intensity. Is assigned. More generally, in some embodiments, each contact j is assigned an individual intensity Ij that is part of the total intensity A according to a predefined mathematical function I j = A · (Dj / ΣDi). Where Dj is the distance of the individual contacts j to the center of force and ΣDi is the sum of the distances of all the individual contacts to the center of force (eg i = 1 to last). Operations described with reference to FIGS. 5C and 5D may be performed using an electronic device that is similar or identical to device 100, 300, or 500. In some embodiments, the characteristic strength of a contact is based on one or more strengths of the contact. In some embodiments, intensity sensors are used to determine a single characteristic intensity (eg, a single characteristic intensity of a single contact). Note that the intensity diagrams are not part of the displayed user interface, but are included in FIGS. 5C and 5D to assist the reader.

In some embodiments, a portion of the gesture is identified to determine the characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from the start position to the end position, at which point the strength of the contact increases. In this example, the characteristic strength of the contact at the end position is optionally based only on the portion of the continuous swipe contact (eg, only on the portion of the swipe contact at the end position), rather than the entire swipe contact. In some embodiments, a smoothing algorithm is optionally applied to the strengths of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm may optionally include one of an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. It includes one or more. In some situations, these smoothing algorithms eliminate narrow spikes or dips in the strengths of the swipe contact to determine the characteristic strength.

The intensity of the contact on the touch-sensitive surface is optionally characterized for one or more intensity thresholds, such as a contact-detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device will perform operations typically associated with clicking a button or trackpad of a physical mouse. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device will perform operations different from those typically associated with clicking a button or trackpad of a physical mouse. In some embodiments, when a contact is detected with a characteristic strength below the light click intensity threshold (eg, and above the nominal touch detection intensity threshold (contact no longer detected)), the device is lightly pressed. The focus selector will move in response to the movement of the contact on the touch-sensitive surface without performing an action associated with the threshold or the deep press intensity threshold. In general, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface figures.

The increase in the characteristic intensity of the contact from the intensity below the light press intensity threshold to the intensity between the light click intensity threshold and the deep click intensity threshold is sometimes referred to as a "light press" input. An increase in the characteristic intensity of a contact from an intensity below the deep click intensity threshold to an intensity above the deep click intensity threshold is sometimes referred to as a "deep click" input. An increase in the characteristic intensity of a contact from an intensity below the touch detection intensity threshold to an intensity between the touch detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting contact on the touch surface. The reduction in the characteristic intensity of a contact from an intensity above the touch detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a liftoff of the contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

In some embodiments described herein, one or more operations detect individual press input performed in response to detecting a gesture including an individual press input or performed with an individual contact (or a plurality of contacts). In response to, wherein the individual press input is detected based at least in part on detecting an increase in the intensity of the contact (or the plurality of contacts) above the press input intensity threshold. In some embodiments, the individual actions are performed in response to detecting an increase in the intensity of an individual contact above the click input intensity threshold (eg, a “down stroke” of the individual click input). In some embodiments, the press input includes an increase in the strength of an individual contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the press input intensity threshold, wherein the individual actions are individual below the press input threshold. Is performed in response to detecting a subsequent decrease in the intensity of the contact of (e.g., an " up stroke " of the individual press input).

5E-5H illustrate the contact 562 from an intensity below the light press intensity threshold (eg, “IT _L ”) of FIG. 5E to an intensity above the deep press intensity threshold (eg, “IT _D ”) of FIG. 5H. Illustrates the detection of a gesture comprising a press input corresponding to an increase in intensity. On the displayed user interface 570 that includes application icons 572A through 572D displayed in the predefined area 574, while the cursor 576 is displayed over the application icon 572B corresponding to App 2, Gestures performed using contact 562 are detected on touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch sensitive display 504. Intensity sensors detect the intensity of the contacts on the touch-sensitive surface 560. The device determines that the intensity of contact 562 has reached a peak above the deep push intensity threshold (eg, “IT _D ”). Contact 562 is maintained on touch-sensitive surface 560. Responsive to the detection of the gesture, and in response to a contact 562 having an intensity that exceeds the deep press intensity threshold (eg, “IT _D ”) during the gesture, reduced scale representations of recently opened documents for App 2 ( 578A-578C) (eg, thumbnails) are displayed, as shown in FIGS. 5F-5H. In some embodiments, the intensity compared to one or more intensity thresholds is the characteristic intensity of the contact. It should be noted that the intensity diagram for contact 562 is not part of the displayed user interface, but is included in FIGS. 5E-5H to assist the reader.

In some embodiments, the display of representations 578A-578C includes animation. For example, as shown in FIG. 5F, the representation 578A is initially displayed in proximity to the application icon 572B. As the animation proceeds, as shown in FIG. 5G, the representation 578A moves up and the representation 578B is displayed close to the application icon 572B. Then, as shown in FIG. 5H, the representation 578A moves up, the representation 578B moves up towards the representation 578A, and the representation 578C is displayed close to the application icon 572B. Representations 578A-578C form an array over icon 572B. In some embodiments, as shown in FIGS. 5F and 5G, the animation proceeds in accordance with the intensity of the contact 562, where the intensity of the contact 562 is pressed deeply to the intensity threshold (eg, “IT _D ”). As they increase, representations 578A-578C appear and move up. In some embodiments, the intensity on which the progress of the animation is based is the characteristic intensity of the contact. Operations described with reference to FIGS. 5E-5H may be performed using an electronic device similar or identical to device 100, 300, or 500.

In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, sometimes referred to as " jitter, " wherein the device has a hysteresis intensity threshold having a predefined relationship to the click input intensity threshold ( For example, the hysteresis intensity threshold is a unit of X intensity lower than the click input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or any suitable ratio of the click input intensity threshold). As such, in some embodiments, the press input includes an increase in the intensity of an individual contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold. The operation of is performed in response to detecting a subsequent decrease in the intensity of the individual contact below the hysteresis intensity threshold (eg, an "up stroke" of the individual press input). Similarly, in some embodiments, the press input causes the device to increase in intensity of contact from an intensity below the hysteresis intensity threshold to an intensity above the press input intensity threshold, and optionally, to an intensity below the hysteresis intensity. Only when detecting a subsequent decrease in the intensity of the contact is detected, the individual action is performed in response to detecting a press input (eg, an increase in the intensity of the contact or a decrease in the intensity of the contact).

For convenience of explanation, the description of the operations performed in response to a press input associated with the press input intensity threshold or in response to a gesture comprising a press input may optionally include an increase in the strength of the contact above the press input intensity threshold, a hysteresis intensity threshold. An increase in the intensity of the contact from an intensity below the click input intensity threshold, a decrease in the intensity of the contact below the click input intensity threshold, and / or an intensity of the contact below the hysteresis intensity threshold corresponding to the click input intensity threshold. Triggered in response to detecting either of the decreases. Also, in the examples in which the operation is described as being performed in response to detecting a decrease in the intensity of the contact below the click input intensity threshold, the action is optionally below the hysteresis intensity threshold that corresponds to and is lower than the press input intensity threshold. Is performed in response to detecting a decrease in the intensity of the contact.

Attention is now directed to embodiments of user interfaces (“UIs”) and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

6A-6N illustrate example user interfaces for managing camera effects, in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIG. 7.

6A illustrates an electronic device 600 having multiple cameras 602 and 603 (eg, on the backside of the electronic device 600). In some embodiments, device 600 includes one or more features of devices 100, 300 and / or 500. In some examples, the electronic device (eg, 600) has multiple cameras that are fixed but have different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of an electronic device (eg, 600). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures a plurality of focal lengths, producing the same result as a plurality of cameras with fixed but different focal lengths. In some examples, the electronic device includes a depth camera, such as an infrared camera, thermographic camera, or a combination thereof. In some examples, the device further includes a light emitting device (eg, a light projector), such as an IR flood light, structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the subject during the capture of an image by a visible light camera and a depth camera (eg, an IR camera), wherein information from the visible light camera and the depth camera is used to determine the depth of the different portions of the object captured by the visible light camera. Used to determine the map. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for the back face images and front face images (eg, selfies). Images are displayed using depth information from a depth camera combined with image data from a visible light camera. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, so as to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for the user. In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects the selection of the camera switching affordance, maintains and displays the front of the user interface controls for applying the lighting effect. Two visible lights spaced apart from the rear facing cameras (eg, from each other) from the front facing cameras (eg, depth camera and visible light camera), replacing the display of the field of view of the facing cameras with the field of view of the rear facing cameras (or vice versa). Cameras) or vice versa.

As illustrated in FIG. 6B, the electronic device 600 includes a display 604 that is touch sensitive (eg, a touch screen), the display displaying image data received from the camera 602. In some embodiments, the display is separate from the touch-sensitive surface.

6B further illustrates that the electronic device 600 displays a camera application user interface 606 on the display 604 to capture images with the camera 602 and / or 603. The camera application user interface 606 further includes a digital viewfinder 608 that includes a live preview of the field of view of the camera 602. As shown in FIG. 6B, the field of view of the camera includes an object (eg, a person) in the foreground and an object (eg, a fence) in the background.

In addition, in FIG. 6B, the camera application user interface includes a filter container 610 that is represented as a hexagon overlaid on the digital viewfinder 608. In some examples, filter vessel (eg, 610) is represented as a circle, triangle, or any other geometric shape. In some examples, filter container (eg, 610) is an image, icon, or textual representation that provides a user with an indication about the current filter. In the embodiment of FIG. 6B, the filter container (eg, 610) is a transparent representation of an object (eg, a hexagon) that allows various filter representations to appear to be moving through the transparent object. In some embodiments, the filter container may be displayed above, below, left, or right of the digital viewfinder (eg, 608) (eg, not overlapping with the viewfinder).

The user interface of FIG. 6B further illustrates filter picker user interface 612 in a folded (eg, minimized) state. The folded filter picker user interface 612 is located along the edge of the digital viewfinder 608. In some examples, the folded filter picker user interface is optionally displayed above, below, left, or right of the digital viewfinder (eg, 608). In some examples, the folded filter picker user interface (eg, 612) includes one or more icons corresponding to a plurality of filter representations arranged in one or more rows and columns or icons located in a circular orientation. In some examples, the folded filter picker user interface (eg, 612) is displayed at any location corresponding to the digital viewfinder. In some examples, the folded filter picker user interface (eg, 612) is indicated by a contour (eg, a border) to distinguish the filter picker user interface from the digital viewfinder (eg, 608). In some embodiments, the folded filter picker user interface (eg, 612) is translucent (or partially translucent) and has no visible boundaries. As a result, in some examples, the boundaries of the folded filter picker user interface are blended in (eg, indistinguishable) from the digital viewfinder (eg, 608).

As shown in FIG. 6B, the folded filter picker user interface 612 includes one or more filter representations (eg, 614A, 614B, 614C, 614D, 614E) displayed on the display 604. In some examples, the folded filter picker user interface (eg, 612) optionally includes filter representations that are not displayed on the display (eg, they are off screen). Filter representations that are not displayed within the collapsed filter picker user interface (eg, 612) optionally detect the input (eg, a swipe gesture) that will cause the filter representations to scroll through the filter container (eg, 610). Is displayed.

As further illustrated in FIG. 6B, the electronic device 600 may display a filter representation (eg, 614A) in a filter vessel (eg, 610) to display a filter applied to the live preview displayed in the digital viewfinder 608. Is displayed. In some embodiments, filter representation 614A corresponds to a "natural light" illumination (eg, filter) option. As a result, the object in the digital viewfinder 608 (eg, person) is displayed without any additional lighting effects applied to the image. In some examples, the filter expression (eg, 614A) may be a "Studio Light" light effect filter, a "Contour Light" light effect filter, a "stage light" light effect filter, or a "stage light mono". Corresponds to the lighting effect filter. Each of the aforementioned light effect filters affects the visual characteristics of the image displayed in the digital viewfinder 608 (eg, based on the depth map of the subject's face, the different sets of objects on the subject's face in the digital viewfinder). By simulating the effect of shining lights). In some embodiments, the “natural light” illumination option is based on depth information without applying additional lighting effects, eg, by blurring the background of the image without blurring at least a portion of the foreground of the image. This involves modifying the image.

As further illustrated in FIG. 6B, when the filter picker is in the folded state, filter representations that are not displayed in the filter vessel (eg, 614B, 614C, 614D, 614E) are displayed in the filter representation (eg, in the filter vessel). Display using a different visual characteristic (eg, shading) than 614A. In some examples, different visual characteristics include color, shape, and size. As further illustrated in FIG. 6B, filter representations (eg, 614B, 614C, 614D, 614E) that are not displayed in the filter vessel are located at their position in the list of filter representations in the folded filter picker user interface (eg, 612). Are displayed at various (eg, progressively shorter, progressively longer) distances from each other on the basis. Thus, in some examples, the further the filter representation (eg, 614E) is from the filter vessel (eg, 610) in the list, the closer it will be located on the display for the adjacent filter representation (eg, 614D).

In some examples, in response to receiving an input (eg, a tap) at a location corresponding to one of the representations of filters (eg, 614B, 614C, 614D, 614E) in the folded filter picker user interface (eg, 612), The electronic device (eg, 600) applies a filter corresponding to the filter representation corresponding to the location of the input. Thus, a user tapping on one of the filters that is not in the filter container will cause the tapped filter expression to scroll relative to the filter container, and the electronic device 600 corresponds to the filter expression in the filter container (eg, 610). We will apply a filter. In some examples, the swipe input will cause a scroll input and the filter expressions will scroll the filter container. As a result, if the filter in the filter vessel has a change, the electronic device will apply the filter currently present in the filter vessel (eg, 610).

6C-6E illustrate a user interface showing activation of the extended filter picker user interface. As illustrated in FIG. 6C, the electronic device 600 receives a tap input 616 at the location of the filter vessel 610. In some examples, the tap input spans any area along the edge of the digital viewfinder 608. In some examples, the tap input spans any area corresponding to the folded filter picker user interface 612. In some examples, the input corresponds to a swipe gesture, a tap and hold (eg, tap and hold for a predetermined period of time) gesture, or an input having a characteristic intensity above a respective intensity threshold.

As illustrated in FIG. 6D, in response to receiving the tap input 616, the electronic device 600 expands the previously folded (eg, minimized) filter picker user (eg, 612) interface of FIG. 6B. Display conversion to filter picker user interface 613. In some embodiments, the filter picker user interface displayed in the folded state of FIG. 6B begins to be converted to an arc (eg, wheel) shaped object of FIG. 6D. In some examples, as the extended filter picker user interface (eg, 613) expands on the display (eg, overlaying more of the digital viewfinder), it is converted to the shape of a wheel or rectangle. In some examples, the extended filter picker user interface (eg, 613) extends from the edge of the digital viewfinder (eg, 608) toward the center of the display 604, such that the filter picker user interface (eg, 612). ) Overlaps with the digital viewfinder 608. In some examples, the extended filter picker user interface (eg, 613) is translucent, translucent, transparent, or translucent. In some examples, the extended filter picker user interface (eg, 613) is displayed as opaque or semi-opaque. In FIG. 6D, when the expanded filter picker user interface (eg, 613) begins to expand, the filter vessel 610 is shifted upward to correspond to the expanded filter picker user interface (eg, 613).

As illustrated in FIG. 6E, when the expanded filter picker user interface (eg, 613) is fully extended, the filter vessel 610 has been shifted upward to the top of the expanded filter picker user interface 613. In some embodiments, additional information (eg, 618) for the filter corresponding to the filter representation (eg, 614A) is displayed in the filter vessel (eg, 610). In some examples, the additional information is displayed within the extended filter picker user interface. In some examples, additional information (eg, 618) is displayed at a location near (eg, above, below) the filter picker user interface. In some examples, the additional information is represented as an icon, text, or graphic.

As illustrated in FIG. 6F, the electronic device 600 receives an input (eg, a tap 620) at a location corresponding to the representation of one of the filters (eg, 614B) that are not in the filter vessel. In response to the tap input, in some embodiments, the electronic device applies a filter corresponding to a filter representation (eg, 614B) corresponding to the location on the display 604 of the tap input. In some examples, the input is, optionally, a swipe, press and hold, or an input having a characteristic intensity that is above a respective intensity threshold. In some examples, an input having a characteristic strength above a respective intensity threshold on the representation of one of the filters 614A-614F, optionally, corresponds to a location associated with the location of the input having a characteristic strength above a respective intensity threshold. Results in the display of additional functionality for the filter representation.

6G-6H illustrate the switching of the filter vessel 610 as a result of the electronic device 600 receiving an input (eg, a tap 620) in FIG. 6F. As illustrated in FIG. 6G, filter vessel 610 is displayed to appear to rotate as the filter representations (eg, 614A-614E) scroll through it. As shown in FIG. 6G, filter vessel 610 appears to be a three-dimensional object (eg, a cube) during rotation. In some embodiments, when the filter vessel (eg, 610) is stationary, it is represented as a two-dimensional image (eg, hexagon). In some examples, the three dimensional representation of the filter vessel is optionally a sphere or a cone. In some examples, the filter vessel appears to be two dimensional when the filter representations traverse through it. As further illustrated in FIG. 6G, the electronic device 600 gradually applies a filter corresponding to the filter representation 614B to the live preview displayed in the digital viewfinder as the filter representation scrolls through the filter vessel.

As further illustrated in FIG. 6H, the filter vessel visually changes when a new representation of a filter (eg, 614B) is displayed in the vessel. In some embodiments, the visual change of the filter vessel 610 is based on the type of filter in the filter vessel (eg, the type of visual / lighting effect applied to the live preview in the digital viewfinder). In some examples, different aspects (eg, faces) of the filter container (eg, 610) will reflect the visual (eg, lighting) effect of the filter corresponding to the filter representation 614B. In some examples, the volume (eg, interior portion) of the filter vessel (eg, 610) will reflect the visual effect of the filter corresponding to the filter representation 614B.

In some examples, filter representations (eg, icons 614A through 614E) correspond to filters that the electronic device 600 simulates by applying various lighting effects corresponding to the point light sources. Thus, the visual characteristics of the filter vessel (eg, 610) will change to simulate the point light source of the corresponding filter. In some examples, a filter displayed in a filter vessel (eg, 610) corresponds to a simulated studio light visual effect (eg, 614B) that simulates a plurality of point light sources. As a result, the filter container is visually modified to simulate a plurality of point light sources corresponding to the studio lighting filter effect. In some examples, the plurality of point light sources are simulated and displayed on three-dimensional object faces during filter selection transition. In some examples, the point light sources are displayed on the faces of the two-dimensional object (eg hexagon) when the corresponding filter is displayed in the filter vessel (eg 610). In some examples, filter representation 614E will be visually distinguished (eg, different representation) while it is within the boundaries of the filter vessel compared to when it is outside the boundaries of the filter vessel (eg, 610).

As further illustrated in FIGS. 6F-6H, in some embodiments, the electronic device 600 gradually applies a filter to the digital viewfinder that corresponds to the newly selected filter representation 614B during the transition. As illustrated in FIG. 6G, the electronic device 600 initially applies a filter corresponding to the filter representation 614B to the digital viewfinder at 50% intensity. The filter corresponding to filter representation 614B is fully applied by the electronic device in FIG. 6H (eg, 100% intensity). In some examples, the electronic device applies the filter in intensity increments (eg, 10%, 25%, 50%, 75%) during the filter transition until the transition is complete.

As illustrated in FIGS. 6I-6J, the input received at the electronic device (eg, swipe 622) will cause the representation of filters 614A-614E to scroll through the filter vessel (eg, 610). In some embodiments, in response to the swipe gesture (eg, 622), representations of the displayed filters will scroll to the left across the top of the filter picker user interface. In some examples, a single swipe gesture will result in an incremental scrolling of filter expressions (eg, filters move by one). In some embodiments, the number of scrolled filter expressions will depend on the size of the swipe gesture. Thus, in some examples, a longer swipe will result in a longer scroll than a shorter swipe. 6J illustrates the results of the swipe gesture in FIG. 6I, where a “stage light mono” illumination filter was applied to the live preview in the digital viewfinder (eg, in the field of view of the camera based on the depth map of the subject's face). By simulating shining a set of lights corresponding to “stage light mono” on the subject's face).

In some examples, the electronic device 600 must continue to detect contact with the filter picker user interface (eg, 613) to maintain the filter picker user interface (eg, 613) in the extended mode. In some examples, when the electronic device detects a liftoff of the contact, the extended filter picker user interface (eg, 613) may cause the folded filter picker user interface state (eg, 612) to be folded (eg minimized). It will start folding until you reach). In some examples, the extended filter picker user interface (eg, 613) will be displayed and remain in extended mode until a predetermined period of time is reached without further contact. When the electronic device 600 detects that a predetermined period of time has been reached, the extended filter picker user interface until it reaches a collapsed filter picker user interface state (eg, 612) (eg, minimized). It will be displayed as a contraction.

In FIG. 6K, the filter picker user interface is in the folded state (eg, 612). As illustrated in FIG. 6K, the electronic device may input (eg, tap 624) at a location corresponding to where the extended filter picker user interface (eg, 613) would be overlaid on the digital viewfinder. ). As illustrated in FIG. 6L, since the extended filter picker user interface is not displayed, the electronic device interprets the input as a focus command, and the bounding box (eg, 626) corresponds to the position of the tap input (eg, 624). Is displayed at the position. As shown at 6l, processing of the focus command occurs without any change in the filter applications or upon presentation of the filter representations.

As illustrated in FIG. 6M, the electronic device swipes in a position corresponding to where the extended filter picker user interface (eg, 613) will be if the extended filter picker user interface is overlaid on the digital viewfinder. Receive gesture 628. As illustrated in FIG. 6N, since the extended filter picker user interface is not displayed, the electronic device interprets the swipe gesture 628 as a mode change command and the electronic device (eg, 600) may display a camera selection user interface (eg, , 606 changes to a different camera mode (eg, “square” camera mode 630, video camera mode, non-vertical camera mode, slow motion camera mode, time lapse camera mode, or panoramic camera mode).

7A-7F are flow diagrams illustrating a method for changing a simulated lighting effect into a representation of image data using an electronic device in accordance with some embodiments. The method 900 is performed on one or more input devices (eg, touch-sensitive surface, keyboard, mouse), and on a device having a display (eg, 100, 300, 500, 600). In some embodiments, the display is a touch sensitive display. In some embodiments, the display is not a touch sensitive display. In some embodiments, the electronic device includes a plurality of cameras. In some embodiments, the electronic device has only one camera. Some operations of the method 700 are optionally combined, some orders are optionally changed, and some operations are optionally omitted.

As described below, the method 700 provides an intuitive way to change the simulated lighting effect into a representation of the image data. This method reduces the user's cognitive burden of providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, allowing the user to launch various functions faster and more efficiently saves power and increases the time between battery charges.

In blocks 702-706, the electronic device (eg, 600) simultaneously displays a camera application user interface (eg, 606) on the display (eg, 604), where the camera application user interface (eg, 606) is On a digital viewfinder (eg, 608) (eg, including live or near live preview images) and a digital viewfinder (eg, 608), including a live preview of the field of view of one or more cameras (eg, 602 and 603). And a representation of a filter picker user interface (eg, 610, 612) overlaid on (eg, a textual representation indicating respective filter currently applied to the image, icon, live preview). In some embodiments, the representation of the filter picker user interface (eg, 610) is in a first location. In some embodiments, the representation of the filter picker user interface (eg, 610) initially switches from the first position to the second position. In some embodiments, the filter picker user interface (eg, 610) may have different lighting effects on the image based on depth map information (eg, as described in more detail below with reference to method 900). To pick between filters that apply them. Simultaneously displaying a digital viewfinder (eg 608) and a filter picker user interface (eg 613) provides a visual field of view of the camera (eg, 602, 603) and objects within the filters available to be applied to the preview. Provide feedback to the user. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life.

In some embodiments, at blocks 708-710, the representation of the filter picker user interface (eg, 612) is a representation of a first filter (eg, 614A), a representation of a second filter (eg, 614B). And a representation of a third filter (eg, 614C), wherein the value of the visual characteristic (eg, size, shade, color) of the representation of the first filter is a value of the visual characteristic of the representation of the second filter and the third filter. Is different from the value of the visual characteristic of the expression of. In some embodiments, filter representations are displayed in a filter picker user interface (eg, 612), and different visual characteristics (eg, they are smaller, different from the first filter displayed in the representation of the filter picker user interface (eg, 612). Shading, different colors). In some embodiments, the second and third filters have the same value of visual characteristics. Visually distinguishing the various representations of the filters may provide the user with feedback as to which filter is currently selected, or additionally or alternatively, provide the user with feedback as to the type of filter effect provided by the respective filters. do. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 712, the representation of the filter picker user interface (eg, 610) is a representation of a first three-dimensional object (eg, 610) having a plurality of faces.

In blocks 714-716, while simultaneously displaying a representation of a digital viewfinder (eg, 608) and a filter picker user interface (eg, 612), the electronic device (eg, 600) receives one or more inputs. A first input (eg, swipe, tap and hold, tap, button press) starting at a location corresponding to a respective portion of the live preview (eg, on or near the filter indicator icon) through the devices 616). In some embodiments, the input can be at any location along the entire edge of the digital viewfinder (eg, 608). In some embodiments, at block 718, the first input (eg, 616) is a tap gesture.

In blocks 720-724, in response to detecting the first input (eg, 616), in accordance with a determination that the first criteria (eg, filter application criteria) are met—the first criteria may be determined by: When the 1 input is detected, the filter picker user interface is overlaid on each part of the live preview (e.g., the filter picker user interface (e.g., 613) is farther into the live preview than when in the collapsed mode of operation. Extended operation mode) requirement, wherein the electronic device (eg, 600) is configured to include a first filter (eg, to indicate that the first filter will be applied to captured media while the first filter is the currently selected filter). The preview of the first filter is applied to a live preview of the field of view of the cameras (eg, 602, 603) that were not applied before the input was detected.

In blocks 720 and 726-728, in response to detecting the first input (eg, 616), the filter picker user interface (eg, 613) is on a respective portion of the live preview when the first input is detected. In accordance with the determination that it is not overlaid (e.g., the filter picker user interface (e.g., 613) is in a collapsed mode of operation that does not extend far into the live preview relative to the expanded state), the electronic device (e.g., 600) may be in live preview. 1 Perform your own actions in the camera application without applying a preview of the filter. Performing an action based on meeting the first criteria (applying the filter to the preview or performing the respective action without applying the filter) may result in initiating the capture of images and / or recording of the video. Provide visual feedback (in the form of updates in the viewfinder) to the user on whether to include a filter. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life. Folding and extending the filter user interface when access to fewer or more filters is needed eliminates disturbances from the viewfinder, which provides an interface for the user to easily switch between the filters while allowing the user to Enables camera operations (eg focus, brightness, camera mode switching) on more viewfinder elements captured in the field of view. Extending the filter user interface when the user is using the filter makes it easier for the user to manipulate the filters, and folding the filter user interface when the user is not using the filter means that the user can perform camera actions (eg, focus). , Brightness, camera mode switching). Providing additional controls while limiting the disturbance of the associated user interface elements improves the operability of the device (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device). And make the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 730, performing the respective action is directed to media capture at an object located in the field of view of the camera (eg, 602, 603) at each portion of the live preview from which the input is detected. Selecting a focus (and, for example, optionally, to focus on a representation of a respective object of one or more objects within the field of view of one or more cameras (eg, 602, 603) (eg, in box 626). Represented by a digital viewfinder (eg, updating the display of 608). Receiving a tap gesture that targets an object for focusing provides the user with a precise targeting mechanism for selecting an object for focus, and multiple or extended inputs to change the focus until the desired object is placed in focus. Avoiding the need to provide them, reducing the number of inputs needed to select a focus. Reducing the number of inputs needed to select a focal point means that the user may receive the intended result by providing feedback indicative of input that will cause the device to produce the intended result (e.g. when operating and interacting with the device). Helping to achieve and reducing user mistakes) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently Reduce and improve battery life.

In some embodiments, at blocks 732-734, while the filter picker user interface (eg, 613) is displayed and overlayed on respective portions of the live preview (eg, the filter picker user interface (eg, While 613 is displayed in the expanded state and while the first filter is being applied to a live preview of the field of view of the cameras (eg, 602, 603), the electronic device (eg, 600) is connected to a filter picker user interface (eg, Detects a third input (eg, 620) (eg, swipe or tap) starting at a location corresponding to 613.

In some embodiments, in blocks 736-740, in response to detecting the third input (eg, 620), the electronic device (eg, 600) may determine a second filter (eg, in the representations of the plurality of filters). Move the representation of 614B to a location on the display (eg, 604) corresponding to the currently selected filter (eg, a different filter representation is displayed in the representation of the filter picker UI), and the electronic device (eg, 600) A preview of the second filter (eg, 614B) is applied to the live preview of the field of view (eg, to indicate that the second filter will be applied to the captured media while the second filter is the currently selected filter). Moving the representation of the selected filter to a location corresponding to the currently selected filter provides the user with visual feedback on the state of the device, including the filter applied to (or to be applied to) the live preview. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 742, the electronic device (eg, 600) provides a tactile output (eg, a tactile output is provided when switching a currently selected filter from one filter to another). .

In some embodiments, at blocks 744-746, while the filter picker user interface (eg, 613) is displayed and not overlaid on respective portions of the live preview (eg, with the filter picker folded) While being displayed, the electronic device (eg, 600) detects a second input (eg, 616) starting at a location corresponding to the filter picker user interface (eg, 612). In some examples, input corresponding to any displayed portion of the folded filter picker user interface causes the interface to be expanded.

At block 748, in response to detecting the second input, the electronic device (eg, 600) extends the filter picker user interface (eg, 613) to overlay the respective portion of the live preview. Extending the filter picker user interface (eg, 613) allows the user to more accurately target the desired filter by, for example, spreading individual representations of the filters so that less precise inputs can be used to accurately select the desired target. To provide. Providing more accurate targeting controls may help the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at blocks 750-752, while the representation of the filter picker user interface (eg, 610) is associated with a sixth representation of representations of the plurality of filters, and the filter picker user interface (eg, 610) Detects a sixth input (e.g., 620) (e.g., swipe or tap) starting at a location corresponding to the filter picker user interface (e.g., 613) while the expression do.

In some embodiments, in blocks 754-760, in response to detecting the sixth input (eg, 620), the electronic device (eg, 600) is configured to display a plurality of non-displayed prior to detecting the sixth input. Rotate the container object to present a second face of the faces, switch from the first filter, the currently selected filter, to a second filter, the currently selected filter, different from the first filter, and preview the second filter's live preview of the field of view. Applies to In some examples, the three dimensional object (eg, 610) appears to be a two dimensional object until it begins to animate (eg, rotate). In some embodiments, where the amount of filter is greater than the amount of faces available on the three-dimensional object (eg, 610), a single face may be used to show two or more different filters. As one example, the first side of the cube (eg, 610) displays the first filter, and after four rotations of the cube, the first side will display the fifth filter from the plurality of filters. Rotating the container object to present the currently selected filter provides the user with visual feedback on the state of the device, including the change in the applied filter. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, the filter picker user interface (eg, 613) includes a representation of the filter picker user interface (eg, 610) and representations of the plurality of filters. In some embodiments, the filter picker user interface (eg, 613) overlaps with the digital viewfinder (eg, 608). In some examples, the filter picker user interface (eg, 613) is displayed as a wheel, dial, half-dial, part of the dial, or slider. In some embodiments, the filter picker user interface (eg, 613) is rectangular.

In some embodiments, the filter picker to overlay respective portions of the live preview without applying the preview of the filter to a live preview of the field of view of the camera (eg, 602, 603) that was not applied before the second input was detected. Extend the user interface (eg, 613).

In some embodiments, the representation of the filter picker user interface (eg, 610) includes a representation of a second three-dimensional object when in a first state (eg, in operation, expanded). A representation of a container object (eg, 610) having a plurality of faces surrounding an interior volume. In some embodiments, the representation of the filter picker user interface (eg, 610) looks like a 2D object when not in operation (eg, it is a simulated cube positioned to look like a hexagon). While in the first state, representing the container object as having an interior volume containing the three-dimensional object provides a user with feedback on the positioning of sources such as light sources within the virtual three-dimensional environment corresponding to the environment being previewed. To provide. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, the representation of the filter picker user interface (eg, 610) is in a second state (eg, stationary, minimized) different from the first state, and the representation of the container object is shaded and / or It is changed to remove lighting effects (eg, the representation of the container object appears to be flattened into a 2D object, such as a hexagon). Changing the representation of the container object to remove shading and / or lighting effects reduces visual distractions for the user and helps to avoid directing the user's attention from the camera's viewfinder. Also, eliminating such effects when they are not needed reduces the number of computing processing needs. Reducing visual distractions and reducing computing processing improves device operability and user-device (eg, by helping the user achieve the intended result and reducing user mistakes when operating and / or interacting with the device). Makes the interface more efficient, and additionally reduces the device's power usage and improves battery life.

In some embodiments, the representation of the filter picker user interface (eg, 610) is in a first state (eg, in operation, expanded), and one or more of the faces or the interior volume of the plurality of faces are with the plurality of filters. It has a visual appearance based on the currently selected filter being represented. (Eg, show illumination on the sides of the cube to indicate changes in illumination). In some embodiments, different filter representations are displayed within the representation of the filter picker UI as the cube rotates. While in the first state, based on the currently selected filter, the visual appearance of the plurality of faces or the internal volume of the representation of the container object, the location of sources such as light sources within the visual three dimensional environment corresponding to the environment being previewed Provide feedback to the user about the setting and feedback to the state of the light sources (enabled, disabled, brightness level). Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

According to some embodiments, the representation of the filter picker user interface (eg, 610) is in a second state (eg, stopped, minimized) different from the first state, and the filter picker user interface (eg, The visual appearance of the representation of 610 is not based on the currently selected one of the plurality of filters displayed in the interior volume. In some embodiments, the effects of the filter are not displayed when the representation for the filter picker UI is minimized or stationary but is displayed when the filter picker is expanded and displayed as a 3D representation.

In some embodiments, while the filter picker user interface (eg, 613) is displayed and not overlaid on respective portions of the live preview (eg, while the filter picker is displayed folded), the filter picker Representations of the user interface (eg, 610) and representations of the plurality of filters are located along a line substantially parallel to the edge of the display (eg, 604). Positioning the representation of the filter picker user interface (eg 610) and the representations of the plurality of filters along a line substantially parallel to the edge of the display (eg 604) is a reminder to the user of available lighting effects. Increasing the visibility of the object in the live preview while allowing the user to more easily perform camera operations (eg, focus, brightness, camera mode switching) on the viewfinder elements. Providing additional controls while maintaining the visibility of the associated viewfinder elements improves the operability of the device (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device). And make the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life. In some embodiments, additional filter options are located along the edge of the display (eg, 604) extending in both directions away from the representation of the filter picker user interface (eg, 610), and in some embodiments, the additional filter. The options are located along the edge of the display (eg, 604) extending in one direction from the representation of the filter picker user interface (eg, 610). In some embodiments, a representation of the filter picker user interface (eg, 610) is displayed near the edge of the live preview.

In some embodiments, while the filter picker user interface (eg, 613) is overlaid on respective portions of the live preview (eg, while the filter picker user interface (eg, 613) is displayed in an expanded state) The representation of the filter picker user interface (eg, 610) and the representations of the plurality of filters are located along a curve (eg, a line that is not substantially parallel to the edge of the display (eg, 604)). In some embodiments, the shifted position will appear to be an arc or wheel. In some embodiments, the interaction includes using a gesture (swipe or tap) along any portion of the bottom edge of the display (eg, 604). Making the shifted position appear as an arc or wheel, and using gestures (swipe or tap) along the bottom edge of the display (eg 604) to interact with the arc or wheel provides continuity in the user interface Reduces confusion, which allows users to provide fewer inputs to perform the desired actions. Reducing the number of inputs needed to perform an action improves the operability of the device (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device) and Makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, the first input is a tap gesture and the position corresponding to each portion of the live preview is a position corresponding to the representation of the first filter within the representations of the plurality of filters. In some embodiments, the extended filter picker UI will be displayed in an expanded state. In some embodiments, the filter picker user interface (eg, 613) will be displayed folded.

In some embodiments, the representation of the second filter is in a first direction from the representation of the first filter, the representation of the third filter is in a first direction from the representation of the second filter, and the value of the visible characteristic is the first filter. Progressively change (eg monotonically increase or monotonically decrease) in the first direction from the representation of the second filter to the representation of the third filter (eg, the representations of the filters are Progressively smaller and / or reduced in opacity as it is further away from the representation of. The value of the visual characteristic that gradually changes in the first direction from the representation of the first filter through the representation of the second filter to the representation of the third filter reduces user distraction, improves visibility of the live preview, It provides a structured system that allows for easier identification and access to the system. Providing improved visual feedback improves the operability of the device and reduces user mistakes (eg, by helping the user achieve the intended result when operating / interacting with the device) and reducing user mistakes. Makes the device more efficient, which further allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life. In some embodiments, filter representations are displayed in a filter picker user interface (eg, 613), and each corresponding filter representation may display a different visual characteristic based on a distance from the representation of the filter picker user interface (eg, 610). Is displayed using.

In some embodiments, the filter picker user interface (eg, 613) is overlaid on each portion of the live preview (eg, while the filter picker user interface (eg, 613) is displayed in an expanded state), The filter picker user interface (eg, 613) includes additional information around the first filter that is displayed in relation to the representation of the first filter (eg, displaying the name of the filter).

According to some embodiments, the first input is a swipe gesture (eg, 628), and performing the respective action includes changing a camera capture mode of the electronic device (eg, 600) (eg, the mode is Video, photo, portrait, square, panorama, slow motion, time lapse).

In some embodiments, applying the preview of the second filter includes gradually switching between applying the preview of the first filter and applying the preview of the second filter. In some embodiments, the gradual transition includes first 100% at first time, second 0% at first time, 90% first at second time, second 10%, and the like.

Note that the details of the processes described above in connection with the method 700 (eg, FIGS. 7A-7F) are also applicable in a similar manner to the methods described below. For example, the method 900, 1100, 1300, 1500, 1700 optionally includes one or more of the features of the various methods described above with respect to the method 700. For example, elements of the filter user interface, affordances and controls can be combined among the various methods. As another example, the viewfinder of method 700 is similar to the viewfinder of methods 900, 1100, 1300, 1500, 1700. For brevity, these details are not repeated below.

In some embodiments, electronic device 800 includes some or all of the components of device 600, as illustrated in FIG. 6A. In some embodiments, device 800 includes a number of cameras 602 and 603 (eg, on the backside of electronic device 800). In some embodiments, device 800 includes one or more features of devices 100, 300 and / or 500. In some examples, the electronic device (eg, 800) has multiple cameras with fixed but different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of an electronic device (eg, 800). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures a plurality of focal lengths, producing the same result as a plurality of cameras with fixed but different focal lengths. In some examples, the electronic device (eg, 800) includes a depth camera, such as an infrared camera, thermographic camera, or a combination thereof. In some examples, the device further includes a light emitting device (eg, a light projector), such as an IR flood light, structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the subject during the capture of an image by a visible light camera and a depth camera (eg, an IR camera), wherein information from the visible light camera and the depth camera is used to determine the depth of the different portions of the object captured by the visible light camera. Used to determine the map. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for the back face images and front face images (eg, selfies). Images are displayed using depth information from a depth camera combined with image data from a visible light camera. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, so as to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for the user. In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects the selection of the camera switching affordance, maintains and displays the front of the user interface controls for applying the lighting effect. Two visible lights spaced apart from the rear facing cameras (eg, from each other) from the front facing cameras (eg, depth camera and visible light camera), replacing the display of the field of view of the facing cameras with the field of view of the rear facing cameras (or vice versa). Cameras) or vice versa.

As illustrated in FIG. 8A, the electronic device 800 includes a display 804 that is touch sensitive (eg, a touch screen), the display displaying image data received from a camera (eg, 602). In some embodiments, the display is separate from the touch-sensitive surface. In some examples, multiple cameras (eg, 602 and 603) are located on the front, back, or both sides of the electronic device (eg, 800).

8A further illustrates that the electronic device 800 displays on the display 804 a camera application user interface 805 for capturing images with a camera (eg, 602). The camera application user interface 805 further includes a digital viewfinder 810 that includes a live preview of the field of view of the camera (eg, 602 or 603). In some embodiments, the camera captures depth information associated with image data in real time. 8A further illustrates a camera that captures distinct depth levels in the field of view, including an object in the foreground area (eg, 808) and a fence in the background area (eg, 811). In some examples, the camera (eg, 602) captures three, four, five, ten, twenty or more depth levels in the field of view. The electronic device 800 utilizes various depth levels when applying a filter to the representation of image data (eg, 806) displayed in the digital viewfinder (eg, 810) as discussed in more detail below.

8A also illustrates displaying the filter picker user interface 813 with the electronic device 800 expanded. The filter picker user interface 813 is located along the edge of the digital viewfinder 810. In some examples, filter picker user interface (eg, 813) is optionally displayed above, below, left, or right of the digital viewfinder (eg, 810). In some examples, filter picker user interface (eg, 813) includes one or more representations (eg, representations of visual effects) of filters arranged in one or more rows and columns or positioned in a circular orientation.

In some examples, the extended filter picker user interface (eg, 813) is displayed at any location corresponding to the digital viewfinder. In some examples, the filter picker user interface (eg, 813) is outlined (eg, a border) to distinguish the filter picker user interface from the digital viewfinder. In some embodiments, the filter picker user interface (eg, 813) is translucent (or partially translucent) when in the folded state and has no visible boundaries. As a result, in some examples, the filter picker user interface (eg, 813) appears to be blended in (eg, indistinguishable) from the digital viewfinder.

As shown in FIG. 8A, the electronic device 800 displays a filter picker user interface 813 that includes one or more filter representations (eg, 814A, 814B, 814C, 814D, 814E) corresponding to visual effects. do. In some examples, filter picker user interface (eg, 813) optionally includes filter representations that are not displayed on the display (eg, they are off screen). In some examples, non-displayed filter representations are displayed when the electronic device receives an input (eg, a swipe gesture), which will cause the filter representations to scroll through the filter container (eg, 816).

As further illustrated in FIG. 8A, in some embodiments, the electronic device 800 may include a filter representation (eg, 814A) in an extended filter picker user interface (eg, 813) to display a currently selected visual effect. Display. In some embodiments, filter representation 814A corresponds to a "natural light" light effect filter. As a result, foreground area 808 and background area 809 are displayed with a “natural light” light effect filter (eg, using natural light from the scene). Since the image representation of FIG. 8A is shown without using any synthetic light, natural light from the scene produces varying shadows on the object (eg, face, neck, and clothing). In some examples, possible filter representations (eg, 814A through 814E) corresponding to the lighting effects are “studio light” lighting effect, “contour light” lighting effect, “stage light” lighting effect, and “stage light mono” lighting. Includes effects. Each of the foregoing lighting effects, when applied to the representation of the image data (eg, 806), affects the visual characteristics of the representation of the image data displayed on the display 804.

In some embodiments, when the electronic device 800 applies the natural lighting effect, no synthetic lighting is added to the image (eg, the original image is displayed). In contrast, studio lighting effects include modeling of a number of individual point light sources (eg, lights in a photography studio) that are uniformly positioned around a subject (eg, creating a brightly filled lighting effect). The contour illumination effect includes the modeling of a number of individual point light sources located along the perimeter of the subject (eg, creating a slimming effect, and generating shadows on the side of the subject's face and / or over the subject's chin). do). Stage light illumination effects include modeling (eg, generating spotlight effects) of a single individual point light source positioned over an object. Stage light mono illumination effects include modeling in black and white (eg, generating spotlight effects in black and white) of a single individual point light source positioned over an object.

In some examples, the electronic device (eg, 800) detects the subject's face in the representation of the image data. As a result, the electronic device uses depth map information of the image data and corresponding facial features when applying the lighting effect. As a result, the lighting effect is applied with greater precision around the subject's face, and certain facial features are based on the selected lighting effect (eg, increasing or decreasing shadows around the subject's jaw and / or cheek bones). Can be illuminated differently. In some examples, the image data includes depth map information that includes depth contours of the objects. As a result, the electronic device uses the contour data to more accurately apply the lighting effect around the object.

As illustrated in FIG. 8B, the electronic device 800 receives an input (eg, swipe 818) at a location corresponding to the extended filter picker user interface (eg, 813). As shown in FIG. 8C, input (eg, swipe 818) causes filter representations 814A through 814E to scroll through the filter vessel (eg, 816). In some embodiments, in response to input (eg, swipe 818), the filter representations will scroll left across the top boundary of the filter picker user interface (eg, 813). In some examples, a single swipe gesture will result in an incremental scrolling of filter expressions. In some embodiments, the number of scrolled filter expressions will depend on the size of the swipe gesture. Thus, in some examples, a longer swipe will result in a longer scroll than a shorter swipe.

In response to an input (eg, 818 swipe), in some embodiments, electronic device 800 applies an illumination effect corresponding to a filter representation (eg, 814B) corresponding to a location on display 804 of the tap input. do. In some examples, the input is a swipe, press and hold, or an input having a characteristic intensity that is above a respective intensity threshold. In some examples, an input having a characteristic intensity above a respective intensity threshold detected on the representation of one of the filters 814A-814F may optionally be located with a location of the input having a characteristic intensity above a respective intensity threshold. This results in the display of additional functionality for the associated corresponding filter representation.

In some embodiments, additional visual effects may be applied to the overall representation of the image data (eg, 806) before applying the lighting effect corresponding to the filter representation (eg, 814B). For example, some gradient filling may be applied to the representation of the image data (eg, 806) before applying the stage lighting filter to more fluidly switch from no filter to the lighting effect filter.

8C and 8D illustrate an electronic device 800 that progressively applies lighting effects as a result of the electronic device receiving an input (eg, swipe 818) in FIG. 8B. In some embodiments, the lighting effect corresponding to the newly selected filter representation 814B is gradually applied to the representation of the image data (eg, 806) in the live preview. Because the selected lighting effect is "studio light", the corresponding visual effect simulates a number of point sources that affect the object in the foreground area 808. As a result, during the transition stage (FIG. 8C), the lighting effect corresponding to the filter representation 814B is applied to the live preview at 50% intensity. The filter corresponding to filter representation 814B is fully applied in FIG. 8D (eg, 100%). In some examples, the filter is applied in increments (10%, 25%, 50%, 75%) while the electronic device 800 applies the lighting effect until the transition is complete. In some examples, the background area (eg, 809) becomes completely dark when the electronic device 800 applies the "studio light" light effect to the representation of the image data. In some embodiments, the background area (eg, 809) is partially darkened when the electronic device 800 applies the “studio light” light effect to the representation of the image data.

As illustrated in FIGS. 8C and 8D, since the image data captured by the camera (eg, 602) includes depth map information associated with the image data, the electronic device may use the available depth map information and the image data. The effect of various point sources on the representation of 806 can be simulated. In some embodiments, the same lighting effect is applied differently in the background area compared to the foreground area based on depth map information associated with the image data. As a result, objects in the foreground may appear more prominent, and objects in the background may become less prominent through the darkening effect. Additionally, as illustrated in FIG. 8D, since the lighting effects simulate point sources, depth map information is used to cast varying shadows on the subject's face in the foreground area (eg, 808). As illustrated in FIG. 8D, since the “studio light” lighting effect simulates a plurality of point light sources, the electronic device 800 uses depth map information to reduce shadow on the subject's face compared to the “natural light” lighting effect. use.

As illustrated in FIGS. 8E and 8F, the electronic device 800 receives an input (eg, tap 820) and causes the representations of the filters 814A-814E to scroll through the filter vessel (eg, 816). something to do. In some embodiments, in response to a tap gesture (eg, 820) corresponding to the filter expression (eg, 814E), the representations of the filters are leftward across the upper periphery of the extended filter picker user interface (eg, 813). Will scroll. FIG. 8F illustrates the result of the tap gesture (eg, 820) of FIG. 8E, where the electronic device 800 has applied a “stage light mono” filter to the digital viewfinder. The "stage light mono" lighting effect simulates a single point light source, with a result similar to the spotlight effect. Using the depth map information, the electronic device 800 applies the "stage light mono" effect from above the object in the foreground area (eg, 808). In some examples, the point light source can be simulated to originate from any direction. In some examples, the “stage light mono” effect is simulated to come from the front, as a result of which a particular focus (eg, face) is emphasized, but the remainder of the representation of the image data is darkened. As illustrated in FIG. 8F, since the simulated point light source is from above the object, the electronic device can cast deeper shadows on the object (eg, face and neck) using the depth map information of the image data. have. In some embodiments, the background of the image is removed and replaced with a solid color, such as black or white, or a user-selected color, to attract more attention to the subject in the foreground and allow the user to be photographed for a solid backdrop. Simulate studio settings.

As illustrated in FIG. 8G, the electronic device may not be in a proper state to capture depth information. FIG. 8G illustrates the user operating the electronic device 800 after a few steps back from where the user was when operating the electronic device in FIG. 8F. As a result of the user's withdrawal, the electronic device may no longer capture depth information (eg, conditions for cameras to capture depth information are no longer detected). In some embodiments, when the conditions for capturing the depth effect are no longer met, the previously applied lighting effect fades out gradually. In some examples, when the conditions for capturing the depth effect are no longer met, the previously applied lighting effect snaps out (eg, the device returns so that no lighting filter is applied without switching). In some embodiments, when the filter is snapped out, optionally, a temporary filter (eg, a gradient) that is part of the lighting effect filter is applied to the image representation. Temporary filters help to smooth the transition when the lighting effect is reapplied (eg less jarring).

In some embodiments, when the electronic device does not detect the conditions needed to capture depth map data, the electronic device 800 determines what actions the electronic device (eg, 800) will take to capture depth map information. Display a graphical representation (eg, 822) to instruct the user about. In some examples, the electronic device detects an object, but the object is too far away (eg, between 2.5 m and 10 m in focus), and the electronic device moves closer to the camera (eg, within 8 feet). Instruct the user to use the graphical representation (eg, using a graphical representation). In some examples, the electronic device determines that the amount of light is too low (eg, 400 lux or less), and instructs the user (eg, using a graphical representation) to provide more light. In some examples, the affordance is displayed in the viewfinder to allow the user to disable or enable such instructions. In some examples, when the conditions for capturing the depth effect map are met, the electronic device 800 suspends displaying the graphical indication instructing the user. Thus, in some examples, the electronic device 800 does not direct the user when the user behavior does not help in applying the lighting effect.

As illustrated in FIG. 8H, the lighting effect snaps in (eg, without switching) when the user steps forward and the electronics again detects the necessary conditions for capturing the depth map information. In some examples, the electronic device 800 gradually applies the lighting effect to the representation of the image data when the conditions for capturing depth map information are again met.

As illustrated in FIG. 8I, the electronic device detects an input (eg, tap 824) at a location corresponding to the photo viewer application (eg, 826). In response to receiving the input (eg, 826), as illustrated in FIG. 8J, the electronic device 800 may use an image viewing mode (eg, to view previously captured images, rather than a live preview of camera data). Mode).

8J illustrates a user interface for a photo viewer application. The photo viewer application includes a thumb strip of previously captured images (eg, 828A through 828D), where 828D is the last captured image. In some examples, previously captured images were captured using a camera corresponding to an electronic device (eg, 800). In some examples, the electronic device (eg, 800) received previously captured images (eg, 828A through 828D) from a remote source (eg, a server), and optionally, previously captured images may be different from the electronic device ( For example, not 800).

8J illustrates that the last captured image (eg, 828D) was captured in a combination of visual effects (eg, the background is blurred and a portion of the foreground is not blurred to determine the shadow depth of the field where the foreground is in the focal plane). simulated depth to simulate taking a picture with the effect (bokeh (bokeh)) (830), and the lighting effect 832). In some embodiments, the electronic device displays a “vertical” visual indicator (eg, 834) at the top of the display as an indication to the user that previously captured image data includes depth map information. In some examples, the electronic device (eg, 800) receives input at a location corresponding to the visual indicator to toggle the simulated depth effect (eg, bokeh effect) on and off. In some embodiments, if the simulated depth effect is toggled off, the lighting effect will be maintained. In some examples, the visual indicator will toggle the simulated depth and lighting effects together when activated. In some examples, the electronic device optionally receives input for changing the lighting effect in the photo viewer application to a different lighting effect using the filter picker user interface (as described above). In some examples, previously captured image data does not have depth map information; The electronic device (eg, 800) will not provide the option to apply the simulated depth effect or lighting effect. In some examples, if the previously captured image data does not have depth map information associated with the image data, the electronic device will not display a visual indicator (eg, 834).

In some examples, the electronic device (eg, 800) stores depth map information along with the image data in one file. In some examples, the electronic device (eg, 800) stores depth map information separately from image data. In some embodiments, if the electronic device stores an image with depth map information as a flat image (eg, without depth map information), the electronic device (eg, 800) may no longer apply lighting effects to the representation of the image data. I will not be able to.

9A-9D are flow diagrams illustrating a method for applying a simulated lighting effect to a representation of image data using an electronic device in accordance with some embodiments. The method 900 is performed on one or more input devices (eg, touch-sensitive surface, keyboard, mouse), and on a device having a display (eg, 100, 300, 500, 800). In some embodiments, the display is a touch sensitive display. In some embodiments, the display is not a touch sensitive display. In some embodiments, the electronic device includes a plurality of cameras. In some embodiments, the electronic device has only one camera. Some operations of the method 900 are optionally combined, some orders are optionally changed, and some operations are optionally omitted.

As described below, the method 900 provides an intuitive way to apply the simulated lighting effect to the representation of the image data. This method reduces the user's cognitive burden of providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, allowing the user to launch various functions faster and more efficiently saves power and increases the time between battery charges.

In some embodiments, at block 902, prior to displaying a representation of the image data (eg, 806), the electronic device (eg, 800), at the device, obtains depth map information associated with the image data and the image data (eg, , From the camera, from the memory, from the server). In some embodiments, the image data includes RGB and depth map values. In some embodiments, image data and depth map are received from a source external to the electronic device (eg, 800) (eg, data is received from a server). In some embodiments, the image is stored in a file format that allows separation of depth information (eg, depth map) and RGB data in a single file. In some embodiments, the image data includes depth map information. In some embodiments the depth map information and the depth map information are separate. Receiving the image data and depth map information corresponding to the image data prior to displaying a representation of the image data (eg, 806) allows the device to display the background (eg, a scene) of the scene through the representation of the image data (eg, 806). 809) or visual feedback on the content of the depth map information, such as whether it is in the foreground (eg, 808). Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

At block 904, the electronic device (eg, 800) displays a representation of the image data associated with the depth map information on the display (eg, 804). In some embodiments, an image or photo is displayed on the display (eg, 804) of the device. In some embodiments, a live preview of image data is displayed in a digital viewfinder (eg, 810). Displaying a live preview of image data in a digital viewfinder (eg, 810) enables the user to quickly and efficiently frame the picture without having to repeatedly capture the picture, thereby providing the input required to capture the intended picture. Reduce the number of photos, reduce memory requirements for storage of photos, and make the user interface more efficient. Reducing the number of inputs and reducing memory requirements needed to capture the desired image can be achieved by, for example, helping the user achieve the intended result and reducing user mistakes when operating and / or interacting with the device. Improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 906, the electronic device (eg, 800) further includes one or more cameras (eg, 602 and / or 603), and the representation of the image data (eg, 806) is digital. A live preview of image data captured within the field of view of one or more cameras (eg, 602 and / or 603) displayed in the viewfinder (eg, 810). In some embodiments, the device includes a plurality of cameras (eg, 602 and / or 603) with various focal lengths.

In some embodiments, at block 908, the depth map information associated with the image data includes information corresponding to at least three different depth levels. For example, the image data includes information corresponding to at least a background depth level, a foreground depth level, and an intermediate depth level. Depth map information, including three or more different depth levels, provides a user with a framework for applying depth-specific filters and provides for depth positioning of objects within the field of view of the camera (s) (eg, 602 and / or 603). Provide users with more precise feedback. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 910, the depth map information associated with the image data includes information identifying depth contours of the object in the representation of the image data (eg, 806).

At block 912, while displaying a representation of the image data (eg, 806) on the display (eg, 804), the electronic device (eg, 800) optionally performs the techniques of blocks 914-936. . In block 914, the electronic device (eg, 800) may, via one or more input devices, perform a first input (eg, 818) (eg, swipe, tap and hold, tap, button press; gesture is an illumination filter or May be on top of an icon representing another user interface system used to select a filter.

In some embodiments, at block 916, the first input (eg, 818) is an input received while the first criteria (eg, a set of lighting effect application criteria) are met, the first criteria being the target. A requirement that is detected in a field of view within a predetermined distance from the electronic device (eg, 800) (eg, other conditions in the set include that the focal length of the first camera (eg, 602 or 603) exceeds a minimum distance threshold, and The focal length of one camera (eg, 602 or 603) does not exceed the maximum distance threshold, the subject is detected beyond a predetermined minimum distance from the device, the amount of light detected exceeds the minimum light threshold, and the amount of light detected This does not exceed the maximum light threshold). In some embodiments, if the first criteria are not met, the application withholds the first or second lighting effect. Applying the lighting effect when it is determined that the object is within a predetermined distance when a first input (eg, 818) is received may cause the object to be properly positioned so that an optimal (or nearly optimal) effect can be achieved with the filter. The user with visual feedback. Similarly, not applying the lighting effect when the object is not within a predetermined distance provides the user with feedback that the object is not properly positioned and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 918, the first input (eg, 818) is an input received while the first criteria (eg, a set of lighting effect application criteria) are not met, and the first criteria are the subject. Includes requirements that are met when detected in a field of view within a predetermined distance from the electronic device (eg, 800) (eg, other conditions / criteria in the set include a minimum focal length of the first camera (eg, 602 or 603). A distance threshold is exceeded, the focal length of the first camera (eg, 602 or 603) does not exceed the maximum distance threshold, the subject is detected beyond a predetermined minimum distance from the device, and the amount of detected light And the amount of light detected does not exceed the maximum light threshold). In some embodiments, if the first criteria are not met, the electronic device (eg, 800) withholds applying the first lighting effect or the second lighting effect.

In some embodiments, at block 920, in response to the first input (eg, 818), the electronic device (eg, 800) places the placeholder filter in the live preview without applying the first lighting effect to the live preview. (Eg, dimming or desaturation of the background (eg, 809)). In some embodiments, in response to detecting that the first criteria have been met, the electronic device (eg, 800) applies the first lighting effect to the live preview while continuing to apply the placeholder filter to the live preview (eg, The placeholder filter is part of the first lighting effect without taking into account the depth map information and is therefore displayed regardless of whether the first criteria have been met). Applying the placeholder filter irrespective of whether the first criteria are met is a viewfinder such as, for example, which portions of the image correspond to portions of the depth map in the background (eg, 809) relative to the foreground (eg, 808). (Eg, 810) provide visual feedback to the user for a depth map corresponding to the content. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 922, after displaying the live preview without applying the first lighting effect to the live preview, the electronic device (eg, 800) detects that the first criteria have been met. In response to detecting that the first criteria have been met, the electronic device (eg, 800) applies the first lighting effect to the live preview. Applying the lighting effect when the first criteria are met provides the user with visual feedback that the first criteria have been met (eg, the object is properly positioned) and that an optimal (or nearly optimal) effect can be achieved with the filter. do. Similarly, not applying the lighting effect when the first criteria are not met provides feedback to the user that the first criteria are not met and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In block 924, upon detecting the first input (eg, 818), the electronic device (eg, 800) may cause the first lighting effect (eg, natural light, studio light) to appear in the image data (eg, 806). , Contour light, stage light, stage light mono), and the first lighting effect is based on depth map information (e.g., between two images taken simultaneously or from different locations based on measurements of the depth sensor). Based on disparity mapping). Displaying the lighting effect based on the depth map information provides the user with visual feedback on the depth map information. For example, placement (or enhancement) and removal (or weakening) of the shadows provide the user with feedback about the particular orientation of the objects corresponding to the depth map information. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life.

In some embodiments, at block 926, applying the first lighting effect includes a placeholder filter (eg, a background) on the representation of the image data (eg, 806) displayed in the digital viewfinder (eg, 810). Eg, dimming or desaturation of 809), wherein the placeholder filter is based on (eg, selected) the first lighting effect and is applied regardless of whether the first criteria are met. Applying a placeholder filter makes a smoother / comfortable transition to the light filter. Applying the placeholder filter irrespective of whether the first criteria are met is a viewfinder such as, for example, which portions of the image correspond to portions of the depth map in the background (eg, 809) relative to the foreground (eg, 808). (Eg, 810) provide visual feedback to the user for a depth map corresponding to the content. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 928, applying the first lighting effect is based on the depth map information associated with the image data in the representation of the image data (eg, 806) displayed in the viewfinder (eg, 810). To apply a simulation of one or more point sources in space. Lighting options include natural light, studio light, contour light, stage light, and stage light mono. Each of the lighting effects models (eg, simulates) the result of one or more of the point light sources in space based on a depth map of the image data. The natural lighting option does not apply any composite lighting to the image (e.g., the original image is displayed or a portion of the original image is displayed, and blur is applied to different parts of the original effect to simulate the bokeh effect). Applies). Studio lighting effects include the modeling of a number of individual point light sources located around an object (eg, produce brightly filled light effects). The contour lighting effect includes the modeling of a number of distinct point light sources located at several points around the subject to create shadows on the subject's face (eg, create a slimming effect, on the side of the subject's face and / or Or create shadows on the subject's chin). Stage light illumination effects include modeling (eg, generating spotlight effects) of a single individual point light source positioned over an object. Stage light mono illumination effects include modeling in black and white (eg, generating spotlight effects in black and white) of a single individual point light source positioned around the subject. In some embodiments, the illumination filter simulates point sources. In some embodiments, the lighting effect snaps in when the first (eg, lighting effect application criteria) is met, as described in more detail above. In some embodiments, when the system detects a face, facial features are taken into account when applying the lighting effect. As a result, the lighting effect changes the appearance of the representation of the image data (eg, 806) based on the specific facial features and face shape of the object. Applying a simulation of the point light source provides the user with a visual representation of the content of the depth map information and the device provides a visual representation of the shape and depth positioning of the objects within the field of view of the camera (s) (eg, 602 and / or 603). Allows you to provide feedback to the user. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve. In addition, applying a simulation of point sources without requiring real physical studio lights makes electronic devices cheaper and smaller than when real studio lights and backdrops are needed, increasing device portability and reducing manufacturing costs. Let's do it.

In block 930, the electronic device (eg, 800) may, via one or more input devices, display a second input (eg, swipe, tap and hold, tap; press a button, gesture a lighting filter (eg, 814A) or May be on top of an icon representing another user interface system used to select a filter.

In block 932, upon detecting the second input, the electronic device (eg, 800) may display a second lighting effect (eg, natural light, different from the first lighting effect) in the representation of the image data (eg, 806). Studio light, contour light, stage light, stage light mono), and the second lighting effect is based on depth map information (eg, two images taken simultaneously or based on measurements of the depth sensor or from different locations). Based on disparity mapping between). Displaying the second lighting effect based on the depth map information provides the user with additional visual feedback on the depth map information. For example, the second lighting effect includes one or more light sources at different locations, intensities, or types (directivity, perimeter, point) that provide the user with feedback about a particular orientation of the objects corresponding to the depth map information. can do. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life.

In some embodiments, at block 934, applying the second lighting effect is to a representation of the image data (eg, 806) displayed in the digital viewfinder (eg, 810), to a depth map information associated with the image data. Applying a simulation of one or more point light sources in space based on that. Lighting options include natural light, studio light, contour light, stage light, and stage light mono. Each of the lighting effects models (eg, simulates) the result of one or more of the point light sources in space based on a depth map of the image data. The natural lighting option does not apply any composite lighting to the image (eg, the original image is displayed or a portion of the original image is displayed, and blur is applied to different parts of the original effect to simulate the bokeh effect). . Studio lighting effects include the modeling of a number of individual point light sources located around an object (eg, produce brightly filled light effects). The contour lighting effect includes the modeling of a number of distinct point light sources located at several points around the subject to create shadows on the subject's face (eg, create a slimming effect, on the side of the subject's face and / or Or create shadows on the subject's chin). Stage light illumination effects include modeling (eg, generating spotlight effects) of a single individual point light source positioned over an object. Stage light mono illumination effects include modeling in black and white (eg, generating spotlight effects in black and white) of a single individual point light source positioned around the subject. In some embodiments, the illumination filter simulates point sources. In some embodiments, the lighting effect is snapped in when the first criteria are met, as described in more detail above. In some embodiments, when the system detects a face, facial features are taken into account when applying the lighting effect. As a result, the lighting effect changes the appearance of the representation of the image data (eg, 806) based on the specific facial features and face shape of the object. Applying a simulation of the point light source provides the user with a visual representation of the content of the depth map information and the device provides a visual representation of the shape and depth positioning of the objects within the field of view of the camera (s) (eg, 602 and / or 603). Allows you to provide feedback to the user. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 936, the lighting effects change the appearance of the representation of the image data (eg, 806) based on the position and curvature of the contours of the object. Including depth contours of the objects in the depth map information allows the device to provide the user with more precise visual feedback on the shape and depth positioning of the objects within the field of view of the camera (s) (eg, 602 or 603). Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve. Also, including the depth contours of the objects in the depth map information applies simulations of the preview and light sources without requiring real physical studio lights and a real physical studio lights to make the electronic device cheaper and smaller than if a backdrop is needed. Makes it possible to increase the portability of the device and reduce the manufacturing cost.

In some embodiments, while the first lighting effect (or second filter) is being applied, the electronic device (eg, 800) determines that the first criteria are not met and the first criteria are not met (eg In response to the determination, no object is detected in the field of view within a predetermined distance from the electronic device (eg, 800), the first (or second) lighting effect is applied to the representation of the image data (eg, 806). Stop (in some embodiments, the visual effect of the filter is reduced but the filter remains applied), and on the display (eg, 804), display a representation of the image data without the first lighting effect being applied ( For example, an image unaltered with no filter or with partial filter applied), graphical representation of unsatisfied first criteria (eg, illumination condition application criteria) on the display (eg, 804) (eg Displays the units, 822) (e. G., Text, icons, images). In some embodiments, when the conditions are met again, the filter is reapplied.

In some embodiments, the representation of image data is previously captured image data. (Eg, an image retrieved from memory / storage, not a live preview of image data captured by one or more cameras (eg, 602 and / or 603)). In some embodiments, depth map information is stored for the stored images, allowing the lighting effects applied to the image to be changed and / or removed after the image is captured. Storing depth map information for the stored images provides the ability to modify the lighting after the image is captured, thereby reducing the number of pictures the user must take to achieve the desired effect, thereby capturing the intended picture. It reduces the number of inputs required to make, reduces memory requirements for the storage of photos, and makes the user interface more efficient. Reducing the number of inputs and reducing memory requirements needed to capture the desired image can be achieved by, for example, helping the user achieve the intended result and reducing user mistakes when operating and / or interacting with the device. Improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, while displaying a representation of image data (eg, 806), the electronic device (eg, 800) may display on the display (eg, 804) a visual indication (eg, that the image data includes depth map information). 834 (eg, a “portrait mode” badge is optionally displayed to indicate the availability of depth map information).

In some embodiments, while the first lighting effect is applied, the electronic device (eg, 800) maintains at least one value of a previously applied visual effect (eg, bokeh , lighting). Thus, it is possible to achieve light effects and bokeh effects in one representation of the image data.

In some embodiments, the previously applied visual effect is a color filter.

In some embodiments, the second input is an input received while the first lighting effect is applied to the representation of the image data (eg, 806), and applying the second lighting effect to the first lighting effect and the second lighting effect. Progressively switching between applications of (in some embodiments, progressively switching is first 100% at first time, second 0%, first 90% at second time, second 10% at second time). And the like). Gradual switching between lighting effects reduces user distraction generated by blinking filters on / off, concentrating the user on taking the desired picture, thereby reducing the number of inputs required to capture the desired picture. Reduce and reduce memory requirements for storage of photos. Reducing the number of inputs and reducing memory requirements needed to capture the desired image can be achieved by, for example, helping the user achieve the intended result and reducing user mistakes when operating and / or interacting with the device. Improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

It is noted that the details of the processes described above with respect to method 900 (eg, FIGS. 9A-9D) are also applicable in a similar manner to the methods described below and above. For example, the method 700, 1100, 1300, 1500, 1700 optionally includes one or more of the features of the various methods described above and referring to the method 900. For example, elements of the filter user interface, affordances and controls can be combined among the various methods. As another example, the viewfinder of method 900 is similar to the viewfinder of methods 900, 1100, 1300, 1500, 1700. For brevity, these details are not repeated below.

In some embodiments, the electronic device 1000 includes some or all of the components of the device 600, as illustrated in FIG. 6A. In some embodiments, device 1000 includes a number of cameras 602 and 603 (eg, on the backside of electronic device 1000). In some embodiments, device 1000 includes one or more features of devices 100, 300 and / or 500. In some examples, the electronic device (eg, 1000) has multiple cameras 602 and 603 with fixed but different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of an electronic device (eg, 1000). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures a plurality of focal lengths, producing the same result as a plurality of (eg more than one) cameras with fixed but different focal lengths. In some examples, the electronic device (eg, 1000) includes a depth camera, such as an infrared camera, thermographic camera, or a combination thereof. In some examples, the device further includes a light emitting device (eg, a light projector), such as an IR flood light, structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the subject during the capture of an image by a visible light camera and a depth camera (eg, an IR camera), wherein information from the visible light camera and the depth camera is used to determine the depth of the different portions of the object captured by the visible light camera. Used to determine the map. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for the back face images and front face images (eg, selfies). Images are displayed using depth information from a depth camera combined with image data from a visible light camera. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, so as to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for the user. In some embodiments, while displaying a camera user interface with one of the filters applied, the device detects the selection of the camera switching affordance, maintains a display of the user interface controls for applying the filters and front-facing camera. Two visible light cameras spaced apart from each other from the front facing cameras (eg, depth camera and visible light camera), replacing the display of the field of view with the field of view of the back facing cameras (or vice versa). ) Or vice versa.

As illustrated in FIG. 10A, the electronic device 1000 includes a display 1004 that is touch sensitive (eg, a touch screen), and the display displays information received from a camera (eg, 602). In some embodiments, the display is separate from the touch-sensitive surface. In some examples, the camera (eg, 602) is located on the front, back, or both sides of the electronic device (eg, 1000).

As shown in FIG. 10A, the electronic device 1000 displays a camera application user interface for capturing images with a camera (eg, 602) on the display 1004. The camera application user interface further includes a representation of image data (eg, 1006) that includes a live preview of the field of view of the camera (eg, 602). In some embodiments, including the embodiment of FIG. 10A, the field of view of the camera captures, in real time, depth information associated with the image data. FIG. 10A captures depth levels in a field of view, including an object (eg, female) in a foreground area (eg, 1008), and an object (eg, female) with trees surrounding the object in the background area (eg, 1010). The camera is further illustrated. In some embodiments, the representation of image data 1006 consists of a background area (eg, 1008 and 1006) and a foreground area. As shown in FIG. 10A, since the image data includes depth map information, the electronic device 1000 may not simulate the simulated depth effect (eg, 1006) on the representation of the image data (eg, 1006) before any other filter is applied. For example, Bokeh ) applies (as illustrated by the representation of objects and trees).

In addition, in FIG. 10A, the electronic device 1000 displays a filter selection interface 1012 under the representation of the image data 1006. In some embodiments, the filter selection interface overlaps with the representation of the image data. In some embodiments, filter selection interface 1012 is located along an edge of the representation of image data 1006. In some examples, the filter selection interface is displayed by the electronic device 1000 above, below (as shown in FIG. 10A), left, or right of the representation of the image data (eg, 1006). As shown in FIG. 10A, a filter selection interface (eg, 1012) provides one or more representations (eg, 1014A-1014C) (eg, visual effects) of filters arranged in one or more rows and columns or positioned in a circular orientation. Include.

In some examples, the filter selection interface is represented by a contour (eg, a border) to distinguish the filter selection interface from the representation of the image data. In some embodiments, the filter selection interface (eg, 1012) is displayed as translucent, semi-translucent, transparent, or translucent, and with no visible boundaries. As a result, in some examples, the filter selection interface (eg, 1012) appears to be blended in (eg, indistinguishable) on the display 1004 with the representation of the image data (eg, 1006).

As shown in FIG. 10A, filter selection interface 1012 includes one or more filter representations (eg, 1014A, 1014B, 1014C, 1014D) displayed on display 1004 corresponding to visual effects. In some examples, the filter selection interface (eg, 1012) optionally includes filter representations that are not displayed (eg, they are off screen). Filter representations that are not displayed filter representations may optionally be displayed for input (eg, swipe gesture) to cause the filter representations to scroll through the filter selection position of the filter selection interface (eg, 1012).

As illustrated in FIG. 10B, the electronic device 1000 receives an input (eg, tap 1016) corresponding to the location of one of the filter representations in the filter selection interface 1012. In FIG. 10B, the filter corresponding to the position of the tap input is "Vivid Warm". In response to receiving the input, as illustrated in FIG. 10, the electronic device 1000 applies a filter (eg, “Vivid Warm”) corresponding to the representation of the image data (eg, 1006).

As illustrated in FIG. 10C, using the depth map information, the electronic device 1000 may assign the selected filter (eg, 1014C) to the background area (eg, 1010) differently from the foreground area (eg, 1008). Apply. The foreground area 1008 (including the woman in front) is modified using a different filter value than the background area 1010 (eg, woman and trees). As shown in FIG. 10C, the electronic device 1000 displays the background area using shades warmer (shown darker) than the foreground area.

As further illustrated in FIG. 10C, the selected filter (eg, Vivid Warm ") includes a color tone protection algorithm. The electronic device 1000 may determine a predetermined color (or a predetermined range of color tones) when the filter is applied. Use depth map information associated with the image to minimize color deviation from, for example, the skin color tone of the subject in the foreground area 1008 without the color tone protection algorithm from the original color tone after application of the Vivid Warm filter. Potentially extremely biased.To address undesirable color tone variations, the electronic device 1000 will apply a filter to the image representation, but will vary the change to a predetermined color tone (or a predetermined range of color tones). Thus, in some examples, the color tone corresponding to the skin color tone remains within a certain predetermined range. Conversely, color tones that are not associated with the skin (eg, blue or green) can be corrected to a greater degree by the selected filter.

10C further illustrates that the color tone protection algorithm applies the algorithm differently for color tones that are in both the background area and the foreground area. For example, an object in the background area (eg, 1010) appears to have a skin color tone similar to that in the foreground, and applying a “Vivid Warm” filter without any color correction is potentially extreme from the original skin color tone. Will make the image look unrealistic, for example. To correct the color tone deviation, the electronic device 1000 will apply a filter to the image representation, but will limit the change to the predetermined color tone. Thus, in some examples, the color tone corresponding to the skin color tone will remain within a certain predetermined range. However, the allowable range for color tone protection in the background is different from the allowable range for color protection in the foreground (e.g. there is less color tone protection for tones in the background or not even color tone protection because it is not in focus). none). Thus, as illustrated in FIG. 10C, the skin tone for the subject in the background area 1010 will change to a warmer tone than the skin tone for the subject in the foreground area 1008.

As illustrated in FIG. 10D, the electronic device detects a tab 1018 at a location 1020 corresponding to the photo viewer application. In response to receiving the tap 1018, the electronic device switches to image viewing mode as illustrated in FIG. 10E.

10E illustrates the user interface displayed for the photo viewer application. The photo viewer application includes a thumb strip of representations of previously captured images (eg, 1020A to 1020D), where 1020D is finally captured. Finally, a representation of image data (eg, 1006) corresponding to the captured image 1020D is displayed on display 1004. In some examples, previously captured images were captured using a camera corresponding to an electronic device (eg, 1000). In some examples, the electronic device receives previously captured images (eg, 1020A-1020D) from a remote source (eg, a server). In some examples, previously captured images were captured with different electronic devices (eg, not 1000). 10E further illustrates that the last captured image does not have any depth map information associated with it (eg, no visual indicator exists representing the depth map, and no visual bokeh effect). .

As illustrated in FIG. 10F, the electronic device detects a tab 1022 at a location 1024 corresponding to a photo viewer editing mode (eg, 1024). In response to receiving the tab 1022, the electronic device switches to image viewing mode as illustrated in FIG. 10G.

FIG. 10G illustrates that filter selection interface 1012 is displayed below the representation of image data 1006, as described above with respect to FIG. 10A. In some examples, filter selection interface (eg, 1012) includes one or more representations (eg, 1014A-1014C) (eg, visual effects) of filters arranged in one or more rows and columns or positioned in a circular orientation.

As illustrated in FIG. 10H, device 1000 detects tap 1026 at the location of “Vivid Warm” filter representation 1014C. In response to receiving the tap 1026, the electronic device applies a Vivid Warm filter corresponding to the filter representation 1014C for the image representation and displays the result of FIG. 10I.

As illustrated in FIG. 10I, since the image data does not have depth information associated therewith, the electronic device applies the “Vivid Warm” filter uniformly to the background area 1010 and the foreground area 1008. Also, even if the image data does not have a depth information association with the image, the color tone protection algorithm still applies the algorithm for the color tones in the image (e.g. it applies the color protection algorithm uniformly over the entire image). do). For example, the electronic device 1000 uses the same value to apply a color protection algorithm to both the skin color tones (eg, foreground and background) of the subject. Thus, as illustrated in FIG. 10C, after the electronic device applies a filter to the reorientation of the image data, in some embodiments, the object (eg, foreground and background) skin color tone is different from that applied to the rest of the image. It is displayed using the same filter value as the filter value.

As illustrated in FIG. 10J, the electronic device 1000 displays a color wheel indicator affordance 1028 overlaid on the representation of the image data 1006.

As illustrated in FIG. 10K, device 1000 detects tab 1030 at the location of color wheel indicator affordance 1028. In response to detecting the tap 1030, as illustrated in FIG. 10L, the electronic device displays an extended color wheel over the representation of the image data. The color wheel allows the user to swipe to rotate the wheel and tap any of the colors represented on the wheel to apply the selected color filter to the representation of the image data.

As illustrated in FIG. 10M, device 1000 detects tab 1034 at a location corresponding to a representation of color in the color wheel. In response to receiving the tab 1032, the electronic device 1000 applies the selected color filter to the representation of the image data 1006 as illustrated in FIG. 10N. As shown in FIG. 10N, a visual indicator 1034 is displayed over the selected color filter to indicate the currently selected filter.

11A-11C are flow diagrams illustrating a method for applying a simulated visual effect to a representation of image data using an electronic device in accordance with some embodiments. Method 1100 is performed on a device having one or more input devices (eg, a touch-sensitive surface, a mouse, a keyboard), and a display (eg, 1004) (eg, 100, 300, 500, 1000) (some implementations) In examples, the device has one camera). In some embodiments, the device has a plurality of cameras, each camera having a different focal length. Some operations of the method 1100 are optionally combined, some orders are optionally changed, and some operations are optionally omitted.

As described below, the method 1100 provides an intuitive way to apply the simulated visual effect to the representation of the image data. This method reduces the user's cognitive burden of providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, allowing the user to launch various functions faster and more efficiently saves power and increases the time between battery charges.

In some embodiments, at block 1102, the electronic device (eg, 1000) is on a display (eg, 1104): a representation of the image data (eg, 1006) (eg, an image or picture is displayed on the device's display (eg, Display on 1104) and a filter selection interface (eg, 1012) simultaneously.

In block 1104, the electronic device (eg, 1000), via one or more input devices, receives a first image filter (eg, 1014C) (eg, illumination) of the representation of the image data (eg, 1006) having the first appearance. Selection of a first image filter (eg, 1014C) at a location corresponding to (eg, above or near) a filter, Vivid, Vivid Warm, vivid cool, dramatic, dramatic warm, dramatic cool, mono, silvertone, noir) For example, a first input (eg, 1016) corresponding to a swipe, tap and hold, tap, and button press is detected.

In some embodiments, at block 1106, the electronic device (eg, 1000) further includes one or more cameras (eg, 602 and / or 603), and the representation of the image data (eg, 1006) is one. It is a live preview of the image data captured within the field of view of the above cameras. In some embodiments, the device includes a plurality of cameras having various focal lengths. In some embodiments, image data and depth information is captured with one or more cameras in an electronic device (eg, 1000). In some embodiments, the device includes a plurality of cameras having various focal lengths.

In some embodiments, at block 1108, a first input is detected while displaying a filter selection user interface (eg, 1012).

In some embodiments, at block 1110, the filter selection user interface (eg, 1012) includes a plurality of filter representations (eg, 1014A-1014D) that include a representation of a first image filter (eg, 1014C). And the first input corresponds to the selection of the representation of the first image filter (eg, 1014C). The first filter is displayed as part of a set of one or more filter representations located in a row along an edge of the display (eg, 1004). In some embodiments, one or more sets of filter representations (eg, 1014A-1014D) are directly bounded by the edge of the display (eg, 1004). In some embodiments, filter representations (eg, 1014A-1014D) are adjacent to but not in contact with the edge of the display (eg, 1004). In some embodiments, the row is located along the short edge of the display (eg, 1004). In some embodiments, the row is located along the long edge of the display (eg, 1004). Displaying the plurality of filter representations is available such that different selectable filters are applied to the representation of the image data (eg, 1006) and optionally, different filters are depth map information corresponding to the representation of the image data (eg, 1006). Provides the user with visual feedback that they provide different techniques for visualizing the, providing additional information about the shape and positioning of the objects in the representation of the image data (eg, 1006). Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

At block 1112, in response to detecting the first input, the electronic device (eg, 1000) optionally displays a first in a representation of image data (eg, 1006) that includes the techniques of blocks 1114-1126. Apply an image filter (eg, 1014C).

In block 1114, in accordance with the determination that the image data has depth information associated therewith, the depth information is determined by the foreground region (eg, 1008) of the representation of the image data (eg, 1006) being determined by the image data (eg, 1006); To be distinguishable from the background area of the representation (eg, 1010) (eg, depth map) —the techniques of blocks 1116-1120 are optionally performed. In some embodiments, depth information is stored as a separate track of the image file. In some embodiments, depth information is calculated using image data received from at least two cameras with different focal lengths.

In block 1116, the electronic device (eg, 1000) uses the first level adjustment to change the appearance of the foreground area (eg, 1008) of the representation of the image data (eg, 1006). Apply a first image filter (eg, 1014C) to the foreground region of the representation of the second image; 1 degree is displayed. In some embodiments, the filter changes one or more of color warmth, unsaturation, color gradient, light intensity, contrast, hue shift, and luminance.

In some embodiments, in block 1118, applying the first image filter (eg, 1014C) to the foreground area (eg, 1008) is such that the image data corresponding to the foreground area (eg, 1008) is the first color. Using a color level adjustment of the first level, in accordance with a determination that the image data includes a value (e.g., a hue value, a tone value, one or more colors associated with skin tones) and that the image data corresponding to the background includes the first color value. Shift (eg, change, modify, replace) a first color value of a background area (eg, 1010) and use a second level of color value adjustment that is different from the first level of color value adjustment to use the foreground area (eg, 1008). Shifting (eg, changing, modifying, replacing) the first color value of c). In some embodiments, the skin color in the background is colored differently than the skin color in the foreground. In some examples, the skin tone color is modified so as not to dramatically change the appearance of the skin color. In some instances, the skin tone is not modified at all. In some embodiments, the filter is applied uniformly to the background even if the area in the background has a color tone corresponding to the skin tone. Different applications of filters include removing constraints on the filter from the foreground or background (eg, removing skin protection requirements for the background). Applying different levels of color adjustment to similarly (or identically) colored objects based on whether the objects are in the foreground or background, such as indicating that certain objects are in the foreground and different objects are in the background Provide the user with visual feedback for depth information corresponding to the image data, and provide the user with visual feedback for distinguishing objects in the foreground from similar objects in the background. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In block 1120, the electronic device (eg, 1000) is configured with image data (eg, 1006) with a second level of adjustment to change the appearance of the background area (eg, 1010) of the representation of the image data (eg, 1006). Apply a first image filter (eg, 1014C) to a background region (eg, 1010) of the representation of. The second level adjustment indicates a second degree at which the first image filter (eg, 1014C) changes the appearance of the representation of the image data (eg, 1006), where the adjustment of the first level and the adjustment of the second level are different. Do.

At block 1122, after applying the first image filter (eg, 1014C) to the representation of the image data (eg, 1006), the electronic device (eg, 1000) is placed on the display (eg, 1004), so that the first filter is removed. Displays a representation of each image applied to the representation of image data (eg, 1006). In some embodiments, the result is displayed on a display (eg, 1004). In some embodiments, the electronic device (eg, 1000) displays the result in response to receiving a request to apply the first image filter (eg, 1014C) to each image. Applying image filters to the foreground and background at different levels of adjustment, and displaying an updated representation of the image data (e.g., 1006) can provide depth information, in particular which objects are identified as being in the foreground and which objects are in the background. Provide the user with visual feedback as to whether they are identified as one. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life. In addition, applying an image filter to the foreground and background with different levels of adjustment, and displaying an updated representation of the image data (e.g., 1006) is a skin of the objects in the viewfinder that the user must otherwise manually correct after applying the filter. It allows electronic devices to automatically apply filters to create more dramatic effects without distorting the tone. Applying a filter to relevant portions of the image and automatically avoiding distorting the skin tone of the subjects without additional user input improves the operability of the device (eg, the user achieves the intended result and the user Helping to reduce the number of inputs) makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1124, applying the first image filter (eg, 1014C) to the representation of the image data (eg, 1006) may cause the image data to have a third color value (eg, hue value, tone). In accordance with the determination that the value includes one or more colors associated with the skin tones, the third color value is shifted (e.g., a predetermined color grade for skin tone protection). Change, modify, replace). In some examples, the skin color is colored differently than the rest of the image. In some examples, the skin tone color is modified so as not to dramatically change the appearance of the skin color. In some instances, the skin tone is not modified at all. In some examples, different applications of the filter include removing constraints on the filter from the image (eg, removing skin protection requirements for the background). For example, when depth information is not available, the device optionally applies skin protection algorithms when adjusting the colors of the image to the entire image rather than just the foreground of the image. As such, some features in the background of the image close to the skin tones protected by the skin protection algorithms will not shift in color. Shifting the colors of portions of the image data using different (eg, third-level color adjustment) techniques can alter the color of the skin tones from the background so that people with specific skin tones are more easily distinguishable than other objects. As with shifting differently, it provides the user with visual feedback as to which portions of the image do not correspond to (or alternatively correspond to) a particular color. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve. In addition, shifting the colors of portions of the image data using a different (eg, third level of color adjustment) technique allows depth information while still avoiding distorting the skin tones of the objects regardless of whether depth information is available or not. Enables a more dramatic effect on the filter when is available, thereby avoiding the need for the user to manually correct skin tones later. Applying a filter to relevant portions of the image and automatically avoiding distorting the skin tone of the subjects without additional user input improves the operability of the device (eg, the user achieves the intended result and the user Helping to reduce the number of inputs) makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1126, in response to detecting the first input and applying the first image filter (eg, 1014C) to the representation of the image data (eg, 1006), the image data is at that depth. In accordance with the determination that the information is not associated with the electronic device (eg, 1000), the electronic device (eg, 1000) may determine the foreground area (eg, 1008) of the representation of the image data (eg, 1006) and the background area of the representation of the image data (eg, 1006). The first image filter (eg, 1014C) is uniformly applied to the representation of the image data (eg, 1006) with a first level of adjustment (eg, to uniformly change the appearance of 1010). Applying the image filter uniformly to the representation of the image data (eg, 1006) provides the user that visual feedback on the state of the device, and especially depth information, is not available for the image data. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve. In addition, applying the image filter uniformly to the representation of the image data (eg, 1006) can be used to filter the depth information when the depth information is available while still avoiding distorting the skin tones of the objects regardless of whether the depth information is available. By enabling a more dramatic effect, the user avoids the need to manually correct skin tones later. Applying a filter to relevant portions of the image and automatically avoiding distorting the skin tone of the subjects without additional user input improves the operability of the device (eg, the user achieves the intended result and the user Helping to reduce the number of inputs) makes the user-device interface more efficient, which additionally allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, prior to detecting the first input, the electronic device (eg, 1000) is configured (eg, from a camera, from memory, from a server), at the electronic device (eg, 1000), from image data (eg, 1006). Receive image data represented by the expression In some embodiments, the image data includes RGB and depth map values. In some embodiments, image data and depth information are received from a source external to the electronic device (eg, 1000) (eg, data is received from a server).

In some embodiments, while the first image filter (eg, 1014C) is applied, the electronic device (eg, 1000) maintains at least one value of a previously applied visual effect (eg, bokeh , lighting). Thus, it is possible to achieve light effects and bokeh effects in one representation of image data (eg, 1006). Maintaining at least one value of a previously applied visual effect (eg, bokeh , lighting) while the first image filter (eg, 1014C) is applied is a need for bulky lenses and equipment to produce the bokeh effect. To alleviate

In some embodiments, the electronic device (eg, 1000) displays a camera application user interface on a display (eg, 1004), where the camera application user interface includes a digital view that includes a live preview of the field of view of one or more cameras. Representation of a color wheel user interface (eg, 1028) in a first position overlaid on a finder (including live or nearly live preview images) and a digital viewfinder (eg, textual representation representing an image, icon, filter). (Eg, icons, affordances, buttons). In some embodiments, the representation of the color wheel interface is displayed in response to user input (eg, a tap on the color wheel afford or a touch and hold gesture detected on the color wheel affordance).

In some embodiments, in response to detecting a second user input (eg, 1030) corresponding to the representation of the color wheel user interface, the electronic device (eg, 1000) is configured to display the color wheel user interface (eg, 1030). Stop displaying the representation (eg, 1004), display a color wheel user interface (eg, 1031) on the display (eg, 1004), and the color wheel user interface (eg, 1031) display a plurality of color filters. Display the representations. In some embodiments, the color wheel is displayed as an arc, wheel, part / complete oval or circle.

In some embodiments, in response to detecting a third user input corresponding to an area corresponding to at least one of the representations of the color filter displayed in the color wheel user interface, the electronic device (eg, 1000) may display image data ( For example, a corresponding color filter is applied to the image data to modify the color appearance of the representation of 1006 (eg, background, foreground or both background and foreground).

For example, the methods 700, 900, 1300, 1500, 1700 optionally include one or more of the features of the various methods described above with respect to the method 1100. For example, elements of the filter user interface, affordances and controls can be combined among the various methods. As another example, the viewfinder of the method 1100 is similar to the viewfinder of the methods 700, 900, 1300, 1500, 1700. For brevity, these details are not repeated below.

As illustrated in FIG. 12A, in some embodiments, electronic device 1200 includes a display 1204 that is touch sensitive (eg, a touch screen), and the display includes information received from a camera (eg, 1202). Is displayed. In some embodiments, device 1200 includes one or more features of devices 100, 300 and / or 500. In some embodiments, the display is separate from the touch-sensitive surface. In some examples, the camera (eg, 1202) is located on the front, back, or both sides of the electronic device (eg, 1200). In some embodiments, electronic device 1200 includes some or all of the components of device 600, as illustrated in FIG. 6A. In some embodiments, device 1200 includes a number of cameras 602 and 603 (eg, on the backside of electronic device 1200).

In some examples, the electronic device (eg, 1200) has multiple cameras with fixed but different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of an electronic device (eg, 1200). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, a camera (eg, 1202) captures image data using a plurality of focal lengths. In some embodiments, the camera (eg, 1202) captures a plurality of focal lengths in a single time, thus producing the same result as two or more cameras with fixed but different focal lengths.

12A further illustrates that the electronic device 1200 displays a camera application user interface for capturing images with a camera (eg, 1202). The camera application user interface further includes a representation of image data 1210 that includes a live preview of the camera's field of view. In some embodiments, the camera's field of view captures depth information associated with the image data in real time. 12A further illustrates that the electronic device has applied a “Dramatic” filter to the representation of the image data.

12A further illustrates that the electronic device 1200 displays the filter selection interface 1206 under the representation of an image (eg, 1210). In some embodiments, the filter selection interface (eg, 1206) overlaps with the representation of the image data (eg, 1210). In some embodiments, the filter selection interface (eg, 1206) is displayed along the edge of the representation of the image data (eg, 1210). In some examples, the filter selection interface is displayed above, below (as shown in FIG. 12A), left, or right of the representation of the image data (eg, 1210). In some examples, the filter selection interface (eg, 1206) includes one or more representations (eg, 1208A through 1208G) (eg, visual effects) of filters arranged in one or more rows and columns or positioned in a circular orientation.

As shown in FIG. 12A, a filter selection interface (eg, 1206) is displayed and displayed as an outline (eg, a border) to distinguish the representation of the image data (eg, 1210) from the filter selection interface (eg, 1206). . In some embodiments, the filter selection interface (eg, 1206) is displayed as translucent, translucent, transparent, or translucent, and in some embodiments, does not have a visible boundary. As a result, in some examples, the filter selection interface (eg, 1206) appears to be blended in (eg, indistinguishable) from the digital viewfinder.

As shown in FIG. 12A, the electronic device 1202 displays a filter selection interface 1206 that includes filter representations 1208A- 1208G corresponding to visual effects. In some examples, the filter selection interface (eg, 1206) optionally includes filter representations that are not displayed on the display (eg, they are off screen). In some embodiments, filter representations that are not displayed on the display are displayed when the electronic device detects an input (eg, a swipe gesture), where the filter representations are filter selection locations of the filter selection interface (eg, 1006). (Eg, 1212).

As further illustrated in FIG. 12A, the electronic device displays filter representations 1208A- 1208G as icons (eg, thumbnails) showing a representation of image data 1210 in the form of an icon (eg, a smaller one). Display. For example, the representation of the image data (eg, 1210) displayed in the main portion of the display is shown in the respective filter representations. In addition, in some embodiments, the filter representations include a filter associated with each individual filter representation applied to each individual thumbnail image by the electronic device 1200. Thus, the user can preview the effect of the respective filter on the respective thumbnail image before activating the respective filter. In some embodiments, the name of the filter (eg, “Dramatic” in FIG. 10A) associated with the filter representation in the filter selection location (eg, 1212) is displayed above the filter selection interface (eg, 1212). In some embodiments, the names of the filters associated with the filter representations are displayed in filter representations (eg, 1208A through 1208G) that overlay the representation of the image. In some embodiments, icons associated with filter representations (eg, 1208A through 1208G) include respective names of filters associated with respective filter representations within the filter representation without a representation of the image.

As further illustrated in FIG. 12A, filter marker 1214 (eg, a visual indicator) is used by the electronic device 1200 to specify a plurality of filter representations (eg, 1208B) to designate the most recent filter. ) Is displayed on at least one of In some embodiments, the most recent filter is at least one filter preselected by the electronic device 1200 based on the filter last used in the currently selected mode (eg, capture mode or post capture edit mode). In some examples, the most recently used filter is a filter predetermined by the electronic device 1200 based on the last used filter regardless of the particular mode (eg, capture or edit). In some embodiments, the visual indicator may be displayed in relation to the filter used most frequently (eg, the number of times used in a predetermined period) by the user. In some embodiments, the visual indicator is displayed in association with a user specified filter (eg, most preferred filter). In some examples, a plurality of filters are designated as preference filters and a plurality of visual indicators are displayed over respective filter representations. In some examples, the visual indicator is displayed by the electronic device above, below, around, or inside the filter representations. In some embodiments, for live preview, the filter representations include reduced scale live previews of the field of view of the camera with the corresponding filter applied to the live preview. In some embodiments, for a previously captured image, the filter representations include reduced scale copies of the image with the corresponding filter applied to the image. In some embodiments, the visual indicator is an image, text, or affordance. In some examples, the plurality of visual indicators differ in color and / or shape based on predetermined criteria.

12A further illustrates a representation 1216 of the contact (eg, touch input) characteristic strength as detected by the device 1200. Representation 1216 includes indicators corresponding to initial intensity threshold IT ₀ , middle intensity threshold IT ₁ , and high intensity threshold IT ₂ . As described below, in some embodiments, the amount of representations of the filters in the filter selection interface will vary based on the strength of the input.

12B illustrates that the electronic device detects a press input 1218 at a location corresponding to the filter representation 1208C. As illustrated in FIG. 12B, the press input 1218 is initially initialized. Intensity threshold IT has a characteristic intensity above _zero .

12C-12F illustrate the folding effect (eg, the compressive effect) of some of the filter representations when the electronic device detects an input having sufficient characteristic strength on one of the filter representations (eg, 1208A through 1208G). As a result of the folding effect, filter selection interface 1206 is displayed with fewer filter representations than before the folding effect. Less filter presentations in the filter selection interface can allow a user to traverse filter expressions more quickly. In some examples, the folding effect occurs in response to the electronic device 1200 receiving a tap and hold input (eg, an input that exceeds a predetermined amount of time).

As illustrated in FIG. 12C, device 1200 continues to detect input 1218 at a location corresponding to filter representation 1208C. 12C also illustrates that the characteristic intensity of the input has increased beyond the intermediate intensity threshold IT ₁ . Since the input has been interpreted to have sufficient characteristic strength above IT ₁ , some of the representations of the filters (eg, 1208C, 1208E, 1208F, 1208G) appear to fold towards the filter selection position (eg, 1212), and they It appears to be being removed from the display. In some embodiments, some of the filter representations (eg, 1208C) are folded to the right. In some embodiments, some of the representations of the filters (eg, 1208E) are folded to the left. In some examples, filter expressions appear to fade out when they are removed from the display. In some examples, instead of folding toward the filter selection location (eg, 1212), the filter representations begin sliding the display. In some examples, instead of folding toward filter selection position 1212, the filter representations are folded toward the filter representation corresponding to the position of the input (eg, the position of filter representation 1208C).

12C further illustrates that the filter applied to the representation of the image data 1210 does not change from the filter applied to FIG. 12B even though the electronic device has begun the folding effect. In some examples, the filter representation in filter selection position 1212 changes when the filter representations begin to collapse. In some examples, the filter representation corresponding to the input location will begin to move to the selection location (eg, 1212). As the filter at the selection location changes, the filter applied to the representation of image data 1210 changes to a filter associated with the new filter representation in filter selection location 1212.

12C further illustrates that the electronic device 1200 generates haptic feedback (eg, a sequence of one or more tactile outputs) 1220 through the tactile output generator during activation of the folding effect. In some embodiments, the electronic device does not generate any haptic feedback during activation of the folding effect. In some embodiments, haptic feedback is generated to indicate completion of the folding effect.

12D illustrates that the electronic device detects an increase in the characteristic strength of the input 1218. In some embodiments, as the characteristic strength of the press input increases, the folding effect increases, and some of the filter representations continue to be critical towards the selection location (eg, 1212). 12D further illustrates that the electronic device continues to generate haptic feedback (eg, a sequence of tactile outputs) during the folding effect.

12E illustrates that the electronic device continues to detect an increase in the characteristic strength of the input 1218. In FIG. 12E, some of the filter representations appear to be almost completely folded (eg, little visible). In FIG. 12E, the electronic device no longer generates haptic feedback (eg, a sequence of one or more tactile outputs) because the intensity of the input has almost reached the high intensity threshold IT ₂ . In some embodiments, the electronic device continues to generate haptic feedback (eg, a sequence of one or more tactile outputs).

In some examples, the electronic device (eg, 1200) stops detecting an input (eg, 1218) during the transitional folding effect (eg, the folding effect as represented in FIGS. 12C-12E) and the characteristic strength of the input is increased. If below a certain threshold (eg, high intensity threshold IT ₂ ), the electronic device performs an expansion effect (eg, a reverse folding effect) to bring the filter selection interface to its extended state (eg, as shown in FIG. 12A). Will restore. Thus, in some embodiments, when the electronic device detects a lift off, the electronic device will stop the folding effect and the user interface will return to the state (eg, expansion) as shown in FIG. 12A.

As illustrated in FIG. 12F, the electronic device 1200 detects that the characteristic strength of the input exceeds the high intensity threshold IT ₂ . As a result, the electronic device has completed the folding effect, and only three filter representations are displayed in the filter selection interface 1206. In some embodiments, the filter representation in filter selection position 1212 at the time the initial input (eg, 1218 of FIG. 12B) was received, because the filter representation is not associated with the “no filter” option or the last used filter option, The three filter representations displayed in the selection interface are the currently selected filter representation (eg, filter representation within filter selection position 1212), the last used filter (eg, 1208B) or preferred filter, and the "no filter" representation (eg, 1208H). In some embodiments, if the filter interface includes a plurality of recently used or preferred filter representations, the compressed filter interface will include corresponding last used filter representations, and the filter representation in the compressed filter interface. The total amount of them will exceed three.

In some examples, at the time an initial input (eg, 1218 of FIG. 12B) is received, two filter representations within filter selection location 1212 are received at electronic device 1200 and associated with a “no filter” representation. The filter representations are subsequently displayed in the reduced filter selection interface (eg, the folded interface). The two filter representations include the last used filter (eg 1208B) representation and the "no filter" representation (eg 1208H).

In some examples, when a filter representation within filter selection location 1212 is received at electronic device 1200 at the time an initial input (eg, 1218 of FIG. 12B) is received, and is associated with the last used filter representation, 2 Filter representations are consequently displayed in a reduced filter selection interface (eg, a folded interface). The two filter representations include the last used filter (eg 1208B) representation and the "no filter" representation (eg 1208H). Thus, in some embodiments, the device is in a fully folded state that meets one or more of the following parameters: the currently selected filter expression, the "no filter" expression, or the last used or preferred filter expression. Keep only them.

As illustrated in FIG. 12G, while the filter selection interface is in the collapsed state, the electronic device 1200 corresponds to the filter 1208B most recently used to change the filter representation displayed at the filter selection position 1212. The swipe 1222 is detected at the position to be. As shown in FIG. 12G, swipe input 1222 is a continuation of input 1218 (eg, occurs without intervening lift off). In some embodiments, even if the swipe input 1222 is maintained on the display, the characteristic strength of the input 1222 drops below the high threshold IT ₂ , but is still above the initial intensity threshold IT ₀ . In some embodiments, even though the characteristic intensity of input 1222 is below high intensity threshold IT ₂ , the filter selection interface may not drop until the electronic device detects a lift off event (eg, the intensity falls below the initial intensity threshold IT ₀₎ . Until it is in a compressed (eg, folded) mode. In some embodiments, swipe input 1222 is a separate input and the filters remain in folded mode until the user taps or presses again.

12H and 12I illustrate the result of the electronic device receiving the swipe input 1222 of FIG. 12G. 12H illustrates the transition user interface while the electronic device 1200 detects a swipe input 1222. In some embodiments, as the swipe input 1222 moves along the display, the filter representation displayed within the filter selection position 1212 changes gradually. In some examples, as the filter representation at the selection location changes, the filter corresponding to the filter selection location is gradually applied to the representation of the image data (eg, 1210). In some examples, the electronic device outputs tactile feedback when the filter representations switch to the selection location (eg, 1212).

12I illustrates a user interface after changing filter representations at a selection location (eg, 1212). As a result, the last used filter (eg, 1208B) is now displayed within the selection position. Additionally, a filter (eg, “Vivid Warm” 1208B) corresponding to the filter used last is applied by the electronic device to the representation of the image data 1210. 12I further illustrates that the electronic device continues to detect an input (eg, 1222).

FIG. 12J illustrates a user interface when the electronic device 1200 stops detecting an input (eg, 1222) in FIG. 12I. In some embodiments, when the user raises his finger from the touch-sensitive display, the electronic device 1200 detects that the characteristic strength of the input falls below the initial intensity threshold IT ₀ . As a result, as illustrated in FIG. 12J, in some embodiments, filter representations begin to expand visually within filter selection interface 1206 when the electronic device detects a lift off command.

12K illustrates the user interface after the electronic device completes the extension process of FIG. 12J. Once the filter representations are expanded, all of the representations of the filters are displayed in the filter selection interface. In some embodiments, the filter representation displayed within filter selection position 1212 when the filter selection interface is compressed (eg, as represented in FIG. 12I) may cause the filter selection position ( 1212) to be displayed and maintained. In some embodiments, when all of the filter representations are expanded, some of the filters remain away from the visible portion of display 1204. In some embodiments, the user can swipe at a location corresponding to the filter selection interface to scroll the list of filter expressions from outside the display onto the display.

As illustrated in FIG. 12L, the electronic device 1200 detects a swipe input 1226 at a location corresponding to the location of the filter selection interface 1206. As a result of the swipe input 1226, the filter representations are scrolled through the filter selection position 1212, as illustrated in FIG. 12M. As shown in Fig. 12L, the previously displayed filter expression "Vivid Warm" is gradually removed from the selection position and a new filter expression "None" is displayed at the selection position. In some embodiments, the filter expression "None" does not correspond to any particular filter, but rather it corresponds to no filter applied to the representation of image data 1210. In some examples, as the filter representation is changed from "Vivid Warm" to "None", the "Vivid Warm" filter is gradually removed from being applied to the representation of image data 1210 by the electronic device 1200.

As illustrated in FIG. 12N, the electronic device 1200 detects an input having a characteristic strength that exceeds a high threshold IT ₂ . As a result of detecting the input with characteristic strength, some of the filter expressions are folded (eg, compressed) on the user interface so that it is no longer visible. In some embodiments, two filter representations are displayed and maintained within the filter selection position 1212. As shown in FIG. 12N, the electronic device detects an input when a "None" filter representation is displayed within the filter selection position 1212, and the "None" filter representation position is a representation of the filter that the filter was most recently used ( For example, a filter with a visual indicator thereon) is maintained within the selection interface when the filter is collapsed.

As illustrated in FIG. 12O, the electronic device 1200 detects a swipe input 1228 at a location corresponding to the location of the filter selection interface 1206. In some embodiments, as a result of swipe input 1228, the filter representations are scrolled through filter selection position 1212, as illustrated in FIG. 12O. As shown in Fig. 12O, the previously displayed filter expression "None" is gradually removed from the filter selection position. As shown in FIG. 12O, a new filter representation "Vivid Warm" is displayed at the filter selection position 1212. In some examples, the filter corresponding to the new filter expression “Vivid Warm” is gradually applied to the representation of the image data (eg, 1210) as the filter scrolls into the filter selection position 1212.

In some examples, the folding effect as described above with reference to FIGS. 12A-12P is optionally implemented by the electronic device 1200 in an image viewer application and / or image editing application, and functions similar to that described above. something to do.

13A-13F are flow diagrams illustrating a method for simplifying a user interface using an electronic device, in accordance with some embodiments. The method 1300 may include one or more input devices (eg, touch-sensitive surface, keyboard, mouse), and a device having a display (eg, touch-sensitive display) (eg, 100, 300, 500, 1200). Is performed in In some embodiments, the device includes a touch sensitive display. Some operations of the method 1300 are optionally combined, some orders are optionally changed, and some operations are optionally omitted.

As described below, the method 1300 provides an intuitive way to simplify the user interface. This method reduces the user's cognitive burden of providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, allowing the user to launch various functions faster and more efficiently saves power and increases the time between battery charges.

In block 1302, the electronic device (eg, 1200) includes a filter selection interface (eg, on a display (eg, 1204) that includes representations of a plurality of filters (eg, 1208A through 1208G) in the set of filters. 1206 (eg, the filter selection mode displays all of the filters).

In some embodiments, at block 1304, the representation of the first filter (eg, 1208A) corresponds to the no filter option, and wherein the first subset of representations of the filters is the representation of the first filter (eg, 1208C) and Two or more (eg, all) filters of the plurality of filters except the representation of the fourth filter corresponding to the most recently applied filter (eg, 1208B). Reducing the number of representations of filters displayed in response to user input corresponding to the selection of the first filter provides the user with feedback that limits the filters to a subset that is curated based on the type of first filter, This declutters the display by eliminating the display of filters that the user is less likely to use and making the filters more likely to be used by the user more easily accessible with less user inputs. Reducing the number of inputs required to provide an improved visual feedback to a user and perform an action (eg, helps the user provide appropriate inputs and reduces user mistakes when operating and / or interacting with a device). By improving the device's operability and making the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1306, the representation of the first filter (eg, 1208B) corresponds to the most recently used filter option, and the first subset of representations of the filters is a first filter (eg, 1208B). And filters of two or more (eg, all) of the plurality of filters except for the representation of the fifth filter (eg, 1208A) corresponding to the expression of the filter and the no filter option. Reducing the number of representations of filters displayed in response to user input corresponding to the selection of the first filter provides the user with feedback that limits the filters to a subset that is curated based on the type of first filter, This declutters the display by removing the display of filters that are less likely to be used by the user and making the filters more likely to be used by the user more easily accessible with less user inputs. Reducing the number of inputs required to provide an improved visual feedback to a user and perform an action (eg, helps the user provide appropriate inputs and reduces user mistakes when operating and / or interacting with a device). By improving the device's operability and making the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1308, the representation of the first filter (eg, 1208C) does not correspond to the most recently applied filter and does not correspond to the no filter option, and the first subset of representations of the filters is removed. A plurality of filters excluding a representation of one filter (eg 1208C), a representation of a sixth filter corresponding to the most recently applied filter (eg 1208B) and a representation of a seventh filter corresponding to no filter option (eg 1208A) Two or more (eg, all) of the filters. In some embodiments, the second subset of representations of the filters includes filters from a preference list of filters displayed when the second subset is activated. In some embodiments, a preference list of filters is determined based on the filters most frequently used. In some embodiments, the preference list of filters is a customizable list of filters. Reducing the number of representations of filters displayed in response to user input corresponding to the selection of the first filter provides the user with feedback that limits the filters to a subset that is curated based on the type of first filter, This declutters the display by removing the display of filters that are less likely to be used by the user and making the filters more likely to be used by the user more easily accessible with less user inputs. Reducing the number of inputs required to provide an improved visual feedback to a user and perform an action (eg, helps the user provide appropriate inputs and reduces user mistakes when operating and / or interacting with a device). By improving the device's operability and making the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1310, the representation of the first filter in the filter selection interface (eg, 1206) is one of the devices to which the first filter (eg, color shift, lighting effect, sharpening, blurring) is applied. A live preview (eg, 1210) of the field of view of the cameras (eg, 1202). Including live previews as part of the representations of the filters provides the user with visual feedback regarding the effects to be generated when the filters are applied, thereby providing the user with a number of inputs from which to select various filters to receive the feedback. Decrease. Reducing the number of inputs required to provide an improved visual feedback to a user and perform an action (eg, helps the user provide appropriate inputs and reduces user mistakes when operating and / or interacting with a device). By improving the device's operability and making the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1312, the representation of the second filter in the filter selection interface (eg, 1206) is a live preview (eg, of the field of view of one or more cameras (eg, 1202) of the device to which the second filter has been applied. , 1210). In some embodiments, if the first filter corresponds to the no filter option, applying a visual effect to the image will not change the appearance of the image. Including live previews as part of the representations of the filters provides the user with visual feedback regarding the effects to be generated when the filters are applied, thereby providing the user with a number of inputs from which to select various filters to receive the feedback. Decrease. Reducing the number of inputs required to provide an improved visual feedback to a user and perform an action (eg, helps the user provide appropriate inputs and reduces user mistakes when operating and / or interacting with a device). By improving the device's operability and making the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In some embodiments, at block 1314, the filter selection interface (eg, 1206) includes a visual indicator (eg, 1214) (eg, a dot) that identifies a representation of the third filter (eg, above the filter). Further included, the third filter is the most recently used filter. In some embodiments, the visual indicator (eg, 1214) represents the most frequently used filter (eg, based on filters applied to images in the device's photo library or camera roll). In some embodiments, expresses a most recently used predetermined number or preference (eg, filters that are explicitly indicated by the user as preference filters or filters most frequently used by the user). A plurality of indicators are displayed. In some embodiments, the most recently used filter is determined based on the currently selected camera mode (eg, the most recently used filter is the most recently used or currently selected for the photos captured in the currently selected camera mode). The most recently used filter when capturing photos in camera mode). In some embodiments, the most recently used filter is based on the most recently used filter in any camera mode, so the most recently used filter is when the device switches from one camera mode to another camera mode. The same is true for. Including a visual indicator identifying the most recently used filter provides visual feedback on recently applied filtering techniques so that the user rarely needs to switch between multiple filters to identify the most recently used filter. To the user, thereby reducing the number of inputs required to identify and activate the most recently used filter. Reducing the number of inputs required to provide an improved visual feedback to a user and perform an action (eg, helps the user provide appropriate inputs and reduces user mistakes when operating and / or interacting with a device). By improving the device's operability and making the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

In block 1316, on the display (eg, 1204), a representation of the image data (eg, 1210) (eg, a previously captured picture, a preview based on data received from the first camera, an image received from the server) And while simultaneously displaying a filter selection interface (eg, 1206), the electronic device (eg, 1200), at block 1318, allows the first filter of the set of filters to meet the selection criteria, via one or more input devices. First input (eg, 1228) (eg, swipe, tap and hold, tap, press hard) at a location corresponding to the filter selection interface (eg, 1206) (eg, on or near the filter indicator affordance). (3-D touch), button press).

In some embodiments, at block 1320, the first filter satisfies selection criteria when a first input (eg, 1218) is detected at a location corresponding to the first filter. The first filter satisfies the selection criteria when the input is in a position corresponding to the first filter (and does not meet the selection criteria when the input is not in a position corresponding to the first filter). Allows the device to provide targeted input to reduce the number of displayed filter representations to reduce the number of inputs required to select a filter. Reducing the number of inputs needed to perform an action may result in the user receiving the intended result by providing feedback indicative of the input that will cause the device to produce the intended result (eg, when operating and interacting with the device). Helping to achieve and reducing user errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently Reduce and improve battery life.

In some embodiments, at block 1322, when a first input (eg, 1218) is detected (eg, when an increase in intensity of contact is detected), the first filter is configured to display a selection position ( For example, the selection criteria are met when at 1212 (eg, a position corresponding to a filter currently being applied to the representation of image data, a focal position (eg, 1210)). A first filter that satisfies the selection criteria when the representation of the first filter is selected at the time that the input is detected (and does not meet the selection criteria when the representation of the first filter is not selected at the time the input is received) May allow the user to provide targeting information to allow the device to reduce the number of displayed filter representations, thereby reducing the number of inputs needed to select a filter. Reducing the number of inputs needed to perform an action may result in the user receiving the intended result by providing feedback indicative of the input that will cause the device to produce the intended result (eg, when operating and interacting with the device). Helping to achieve and reducing user errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently Reduce and improve battery life.

In some embodiments, at block 1324, the first input (eg, 1218) corresponds to a contact on the touch-sensitive display, and at block 1326, the contact exceeds a first intensity threshold (eg, a hard press). Has the characteristic strength of. A device that determines that a characteristic intensity of a first input (eg, 1218) exceeds a first intensity threshold and, in response, stops displaying the first subset of filters while continuing to display the second subset of filters. Reduces the number of inputs required by clarifying the various inputs received at the same location on the touch-sensitive display and providing the device with the ability to perform the appropriate action. Reducing the number of inputs needed to perform an action may result in the user receiving the intended result by providing feedback indicative of the input that will cause the device to produce the intended result (eg, when operating and interacting with the device). Helping to achieve and reducing user errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently Reduce and improve battery life.

At block 1328, in response to detecting the first input (eg, 1218), the electronic device (eg, 1200) optionally performs the techniques of blocks 1330-1338.

In block 1330, the electronic device (eg, 1200) is configured to display (eg, completely / partly remove, reduce) the first subset (eg, at least one) of the representations of the filters in the set of filters. Stop it. At block 1332, the first subset of representations of the filters in the set of filters is determined, at block 1334, of the filters that are to the right of the representation of the first filter in the filter selection user interface (eg, to the right of the representation of the first filter). One or more filters in the first direction from the one or more representations, and in block 1336, from the representation of the first filter (eg, one or more representations of the filters to the left of the representation of the first filter) in the second direction. It includes one or more filters. In response to detecting the first input, reducing the number of filter representations displayed by stopping displaying the first subset of filters and maintaining display of the second subset of filter representations results in a visual display of relevant filter options. It requires less user input to provide feedback to the user, reduce clutter on the user interface, and navigate between the filters to find the desired filter. Reducing the number of inputs needed to provide visual feedback and perform an action may be achieved by providing feedback indicative of input that will cause the device to produce an intended result (eg, when operating and / or interacting with the device). Helps to achieve the intended result and reduces user mistakes) and improves the device's operability and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently Reduce device power usage and improve battery life.

At block 1338, the electronic device (eg, 1200) maintains a display of the second subset of representations of the filters in the set of filters, the second subset of representations of the filters including at least a representation of the first filter. . In some embodiments, a subset of the initial set of filters is maintained on the screen. In some embodiments, maintaining the filters on the screen involves shifting the position of the filters while keeping the filters on the screen.

In some embodiments, at block 1340, the first subset of representations of the filters includes two or more (eg, all) filters of the plurality of filters except the filters at blocks 1342-1346. . At block 1342, the first subset of representations of the filters is two or more (eg, all) filters of the plurality of filters except the representation of the first filter and the representation of the fourth filter corresponding to the most recently applied filter. Include them.

In some embodiments, at block 1344, the first subset of representations of the filters is two or more of the plurality of filters (eg, the representation of the fifth filter corresponding to the representation of the first filter and the no filter option). , All) filters.

In some embodiments, at block 1346, the first subset of representations of the filters is the representation of the first filter, the representation of the sixth filter corresponding to the most recently applied filter, and the seventh filter corresponding to the no filter option. It includes two or more (eg, all) filters of the plurality of filters except the expression of.

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and the first input (eg, 1218) corresponds to a first contact on the touch-sensitive display, and at block 1348, the first The techniques of blocks 1350 and 1352 are performed while continuing to detect the first contact on the display (eg, 1204) after detecting one input (eg, 1218). In block 1350, the electronic device (eg, 1200) detects movement of the first contact. In block 1352, the electronic device (eg, 1200) may display on the display (eg, 1204) a representation of image data (eg, color shift, lighting effect, sharpening, Blurring) (and, for example, stop displaying the representation of the image data with the applied visual effect corresponding to the first filter on the image data, on the display (eg, 1204)). Updating the representation of the image data to reflect the second filter in response to detecting the movement of the contact provides the user with feedback on the state of the device, and in particular, visual feedback corresponding to the activated filter. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life.

In some embodiments, in response to detecting the movement of the first contact, the electronic device (eg, 1200) stops displaying the representation of the first filter at the second selected location (eg, 1212), and the second Display a representation of the second filter at a selection location (eg, 1212). (Eg, moving the representation of the second filter to a “currently selected” filter position on the display (eg 1204), and removing the representation of the first filter from the “currently selected” filter position on the display (eg 1204). that)

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and at block 1354, the first input (eg, 1218) is a characteristic intensity (eg, 1216) of the contact on the touch-sensitive display. In response to detecting an increase in the characteristic intensity of the contact, the electronic device (eg, 1200) dynamically shifts the position on the display (eg, 1204) of the first filter representation toward focus. do. In some embodiments, as the position of the first filter representations moves toward focus, the representations of other filters that are not within the second subset of representations of the filters fade out and / or decrease as the intensity of the contact increases. . In some embodiments, representations of other filters that are not in the second subset of representations of the filters are horizontal in one or more dimensions (eg, as representations of the filters in the second subset of representations of filters move toward focus). By stretching or compressing in the direction). Dynamically shifting the position of the first filter expression toward focus as the user increases the characteristic strength of the contact provides feedback to the user regarding the level of intensity being detected by the device based on the user's input, Press harder to provide the user with visual feedback indicating that the device will perform the action associated with the user interface element. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the first input (eg, 1218) corresponds to a contact on a touch-sensitive display having a first contact portion and a second contact portion; The first contact portion has a first characteristic strength; The second contact portion has a second characteristic strength; The method may be performed on a display (eg, 1204) in accordance with a determination that the first characteristic intensity is above a second intensity threshold (eg, the folding threshold at which compression of the filter selection interface (eg, 1206 begins)) and below the third intensity threshold. Progressively shifting the position of the first filter representation (eg, the filter representations move towards focus); And in accordance with the determination that the second characteristic intensity is above a third intensity threshold (eg, a folding threshold at which compression of the filter selection interface (eg, 1206) ends (eg, the point at which the first subset stops being displayed). Ceasing to shift the position of the first filter expression.

In some embodiments, at block 1356, the electronic device (eg, 1200) detects a decrease in the characteristic strength of the contact while continuing to detect the contact on the touch-sensitive display. In some embodiments, at block 1358, blocks 1360-1362 are optionally performed in response to detecting a decrease in the characteristic strength of the contact.

In some embodiments, at block 1360, the respective intensity of the contact prior to detecting a decrease in the characteristic intensity of the contact (eg, a threshold intensity to stop displaying the first subset of filters). In response to determining that a threshold (eg, 1216) has been reached, the electronic device (eg, 1200) maintains the display of the second subset of representations of the filters without displaying the first subset of representations of the filters.

In some embodiments, at block 1362, the respective intensity of the contact prior to detecting a decrease in the characteristic intensity of the contact (eg, a threshold intensity to stop displaying the first subset of filters). In response to determining that a threshold (eg, 1216) has not been reached, the electronic device (eg, 1200) dynamically shifts the position on the display (eg, 1204) of the first filter representation away from the focus. In some embodiments, their size increases as the position of the first filter representations moves away from the focal point.

Maintaining the display of the second subset of filters or dynamically shifting the position of the first filter as the user reduces the characteristic intensity of the contact may cause the user's input to be at a respective intensity threshold (eg, 1216). Visual feedback that provides the user with feedback about the level of intensity detected by the device based on the arrival and indicates whether the user needs to press harder to cause the device to perform an action associated with the user interface element. To the user. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, prior to displaying a representation of the image data, the electronic device (eg, 1200) receives image data (eg, from a camera, from a memory, from a server), where the image data is associated with the image data (eg, , 1210). In some embodiments, the image data includes RGB and depth map values. In some embodiments, image data and depth information are received from a source external to the electronic device (eg, 1200) (eg, data is received from a server).

In some embodiments, in addition to detecting the movement of the contact, the electronic device (eg, 1200) provides a tactile output indicating that the filter applied to the image data has changed. In some examples, the electronic device (eg, 1200) provides a tactile output when switching between different filters. Performing tactile output when switching between different filters provides the user with additional feedback regarding changes to the state of the selected filter and helps the user perform the operations more efficiently. Providing improved feedback improves the operability of the device and reduces the user-device interface more efficiently (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device). In addition, it reduces the power usage of the device and improves battery life.

In some embodiments, before detecting the first input (eg, 1218), while the third filter is at the second selected position (eg, 1212), and while displaying the first subset of representations, the electronic device. 1200 (eg, 1200) detects movement of the second contact on the touch-sensitive display. In response to detecting the movement of the second contact, the electronic device (eg, 1200) stops displaying the representation of the third filter at the second selection location (eg, 1212) and stops displaying the second selection location (eg, Display a representation of the fourth filter, apply a visual effect (eg, color shift, lighting effect, sharpening, blurring) corresponding to the fourth filter to the image data, and display on the display (eg, 1204). , Display the representation of the image data with the applied visual effect, the first contact being the second contact. Switching between filters in response to detecting the movement of the second contact allows the user to easily switch between filters (rather than just preferred or recent filters) without requiring a strong press to select the desired filter. Improve your ability to target accurately. Providing improved targeting capabilities (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device) improves device operability and makes the user-device interface more efficient This, in addition, reduces the power usage of the device and improves battery life.

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and the first input (eg, 1218) corresponds to a contact on the touch-sensitive display. While maintaining the display of the second subset of representations of the filters without displaying the first subset of representations of the filters, the electronic device (eg, 1200) detects a liftoff of the contact. In response to detecting the liftoff of the contact, the electronic device (eg, 1200) restores displaying the first subset of representations of the filters. Restoring the display of the first subset of representations of the filters in response to detecting the liftoff of the contact reduces the number of user inputs required to access the first subset of representations of the filters. Reducing the number of inputs required to perform an action improves the operability of the device (eg, by helping the user achieve the intended result and reduces user mistakes when operating and interacting with the device) and the user Makes the device interface more efficient and, in addition, reduces the device's power usage and improves battery life.

In some embodiments, the electronic device (eg, 1200) includes a touch sensitive display. While maintaining the display of the second subset of representations of the filters without displaying the first subset of representations of the filters, the electronic device (eg, 1200) detects an input that includes an increase in the intensity of the contact. In response to detecting the input, determining that the input includes an increase from a characteristic strength below the fourth intensity threshold to a characteristic strength above the fourth intensity threshold contact with a characteristic intensity above the fourth intensity threshold in the characteristic intensity of the contact. Accordingly, the electronic device (eg, 1200) restores the display of the first subset of representations of the filters. In response to detecting the input, determining that the input does not include an increase from the characteristic strength below the fourth intensity threshold to the characteristic strength above the fourth intensity threshold contact with the characteristic strength above the fourth intensity threshold in the characteristic intensity of the contact. Accordingly, the electronic device (eg, 1200) maintains the display of the second subset of representations of the filters without displaying the first subset of representations of the filters. Restore the display of the first subset of representations of the filters when the characteristic strength of the input increases above the threshold and restore the display of the second subset of representations of the filters when the characteristic strength of the input does not increase above the threshold. Maintaining gives the user a preview of the actions (when the user presses a little) and locks the actions when the user presses more strongly, so that users do not have to maintain greater input characteristic strength while switching between filters, This improves the operability of the device and makes the user-device interface more efficient.

In some embodiments, stopping displaying the first subset of representations of the filters (eg, removing / reducing completely / partly from the entire set) may result in that the representation of the first filter is changed to that of the device to which the first filter is applied. Progressively reducing the size of the representation of the first filter while continuing to display at least a portion of the live preview (eg, 1210) of the field of view of the one or more cameras; And progressively reducing the size of the representation of the second filter while the representation of the second filter continues to display at least a portion of the live preview (eg, 1210) of the field of view of the one or more cameras of the device to which the second filter has been applied. Include.

In some embodiments, in response to detecting the movement of the second contact, determining that the fourth filter is a first type of filter (eg, a predetermined preference, most frequently used, most used). Accordingly, the electronic device (eg, 1200) generates its own tactile output, and the fourth filter is a second type of filter (eg, a filter other than a predetermined preference, most frequently used, most used). In accordance with the determination, the electronic device (eg, 1200) suspends generating its respective tactile output (eg, generating any tactile output or generating tactile output with different tactile output patterns). Withhold). In some embodiments, a tactile output is provided whenever a new filter is displayed in the filter selection interface (eg, 1206). Performing tactile outputs (and performing different tactile outputs based on filters) when switching between different filters provides the user with additional feedback regarding changes to the state of the selected filter, Help the user perform the operations more efficiently. Providing improved feedback improves the operability of the device and reduces the user-device interface more efficiently (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device). In addition, it reduces the power usage of the device and improves battery life.

In some embodiments, the electronic device (eg, 1200) includes a touch-sensitive display, and the first input (eg, 1218) corresponds to a contact on the touch-sensitive display that is maintained for a predetermined period of time. In some embodiments, the retention time is above a predetermined period of time. Performing an action when the input is maintained for a duration above a predetermined period of time is additional (eg, all) that users can select without requiring display of additional user interface elements that hide the viewfinder or clutter the display. Make the filters accessible to users. Providing additional control options without cluttering the user interface with additional displayed controls (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device) Improves the operability of the device, makes the user-device interface more efficient, and further reduces the device's power usage and improves battery life.

In some embodiments, in response to detecting the first input (eg, 1218), the electronic device (eg, 1200) provides a tactile output (eg, 1220) at the electronic device (eg, 1200). In some embodiments, the tactile output (eg, 1220) is in response to the characteristic intensity of the contact reaching a respective intensity threshold (eg, 1216) (eg, representation of filters other than the second subset of representations of the filters). As described above in connection with the process in which they are stopped from being displayed. In some embodiments, a tactile output is provided when a subset of filters are displayed again (eg, as described above) (eg, in response to a liftoff of a contact or subsequent hard pressing). . Performing tactile output when the subset of filters is displayed again provides feedback to the user regarding the status of the selected filter and helps the user perform the operations more efficiently. Providing improved feedback improves the operability of the device and reduces the user-device interface more efficiently (eg, by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device). In addition, it reduces the power usage of the device and improves battery life.

In some embodiments, the representations of the plurality of filters in the set of filters include a third subset of representations of the filters that are not displayed on the display (eg, 1204) while detecting the first input (eg, 1218). . In some embodiments, the input comprising movement of a contact along the filter selection interface (eg, 1206) causes the filter selection interface (eg, 1206) to scroll and expose one or more representations of the filters in the third subset of filters. do.

Note that the details of the processes described above in connection with the method 1300 (eg, FIGS. 13A-13F) are also applicable in a similar manner to the methods described below / above. For example, the method 700, 900, 1100, 1500, 1700 optionally includes one or more of the characteristics of the various methods described above with respect to the method 1300. For example, elements of the filter user interface, affordances and controls can be combined among the various methods. As another example, the viewfinder of the method 1300 is similar to the viewfinder of the methods 700, 900, 1100, 1500, 1700. For brevity, these details are not repeated below.

As illustrated in FIG. 14A, the electronic device 1400 includes a display 1404 that is touch sensitive (eg, a touch screen), and the display displays information received from a camera (eg, 1402). In some embodiments, electronic device 1400 includes one or more features of devices 100, 300, and / or 500. In some embodiments, the display is separate from the touch-sensitive surface. In some examples, the camera (eg, 1402) is located on the front, back, or both sides of the electronic device (eg, 1400).

14A further illustrates that a camera application user interface is displayed on the display 1404 to capture images with a camera (eg, 1402). The camera application user interface further includes a representation of image data 1410 that includes a live preview of the camera's field of view. 14A further illustrates that grid mode is enabled within the camera application and corresponding grids are displayed on the display to assist the user in framing the image.

14A further illustrates the focal plane 1406, the horizontal plane 1408, and the relative angle between the focal plane 1406 and the horizontal plane 1412 for the camera. In FIG. 14A, the focal plane 1406 is a virtual two dimensional plane in front of the camera that intersects the focal point. 14A also includes a representation of the table and a representation of the pie 1411 as displayed on the representation of the table. In some embodiments, instead of the horizontal plane, different planes of orientation (eg, vertical) are used to determine the relative angle (eg, 1412). In some examples, the predetermined orientation is horizontal to the table, and the horizontal orientation provides an optimal orientation for capturing images from a top-down point of view.

14B illustrates that the electronic device is tilted forward (eg toward) the representation of the table. As the electronic device is tilted toward the table, the relative angle between the focal plane and the horizontal plane (parallel to the table) decreases from 90 degrees in FIG. 14A to 70 degrees in FIG. 14B.

As the user continues to tilt forward toward the table as illustrated in FIG. 14C, the relative angle between the horizontal plane and the focal plane decreases by 45 degrees. As further illustrated in FIG. 14C, at certain angles (eg, 45 degrees or less), the relative angle (eg, relative difference) between the horizontal plane and the focal plane is determined by a predetermined alignment threshold (eg, a predetermined respective alignment). Threshold (eg, 45 degrees) (eg, below). As a result, visual indicators (eg, 1414A and 1414B) are displayed. In some examples, the predetermined alignment threshold is 35, 30, 25, 20, 15, 10, or 5 degrees.

In some examples, the visual indicator is represented as two separate graphical objects 1414A and 1414B. In some embodiments, one of the two graphical objects (eg, 1414A) remains stationary in the middle of the display. In some embodiments, the second graphical object (eg, 1414B) changes position based on the orientation of the electronic device. The visual indicator assists the user to guide the user when positioning the device for a predetermined orientation. As described below, as the relative angle between the focal plane 1406 and the horizontal plane 1408 continues to decrease (eg, as they get closer to being parallel), the graphical object 1414B becomes a graphical object. Progressively greater toward 1414B. In some embodiments, when the relative angle between focal plane 1406 and horizontal plane 1408 is zero (eg, when they are parallel), the graphical objects (eg, 1414A and 1414B) appear to overlap completely. (Eg, seen as one graphical object).

In some embodiments, the visual indicator appears to fade in gradually when the electronic device determines that the relative angle between the focal plane and the horizontal plane is within a predetermined alignment threshold (eg, 45 degrees). In some embodiments, the visual indicator is displayed completely opaque when the electronic device determines that the relative angle between the focal plane and the horizontal is within a predetermined alignment threshold (eg, 45 degrees).

In some examples, the visual indicator displays only after the electronic device determines that the relative angle between the horizontal plane and the focal plane remains within the alignment threshold (eg, 45 degrees) for a predetermined period of time (eg, 2, 5, 10 seconds). do. Thus, in some examples, the electronic device must be oriented to remain within the predetermined alignment threshold for a predetermined period of time before the visual indicator is displayed. In some examples, the visual indicator is not displayed when the electronic device determines that the electronic device does not remain within the alignment threshold (eg, 45 degrees) for a predetermined period of time (eg, 2, 5, 10 seconds). In some examples, time is not a factor in determining whether a visual indicator is displayed.

14D and 14E illustrate the user interface when the electronic device continues to tilt forward. As illustrated in FIG. 14D, when the electronic device 1400 determines that the relative angle between the horizontal plane and the focal plane continues to decrease, the graphical object 1414B continues to be progressively greater towards the graphical object 1414A. . Additionally, as the electronic device 1400 determines that the relative angle between the horizontal plane and the focal plane (eg, 20 degrees in FIG. 14D) continues to decrease, the graphical objects 1414A and 1414B continue to fade in. Additionally, in some embodiments, once the electronic device determines that the relative angle between the horizontal plane and the focal plane (eg, 11 degrees in FIG. 14E) reaches a predetermined fade threshold (eg, 15 degrees), the graphical objects 1414A and 1414B are displayed with maximum opacity. In some examples, the fade threshold is 30, 25, 20, 15, 10, 5, or 2 degrees.

As illustrated in FIG. 14F, the electronic device 1400 is now located within 5 degrees of the horizontal plane. As a result, the graphical objects 1414A and 1414B are snapped to the central display position because the relative angle between the horizontal plane and the focal plane is within the snap threshold (eg, 10 degrees). In some embodiments, the snap threshold is 25, 20, 15, 10, 5, 4, 3, 2, or 1 degree. Thus, in some embodiments, if the relative angle between the horizontal plane and the focal plane (eg, 5 degrees) is determined to be within the snap threshold (eg, 10 degrees), even if the focal plane is not perfectly parallel to the horizontal plane, the graphic Objects 1414A and 1414B are displayed (eg, snapped) in the middle of the display in a predetermined alignment. In some embodiments, graphical objects represent animation when snapped to a center position of the display. In some embodiments, the animation includes one or more of color differentiation, shape differentiation, or positional differentiation (eg, some bouncing effect).

As illustrated in FIG. 14G, even if the electronic device 1400 continues to tilt forward, the relative angle between the horizontal plane and the focal plane (eg, 9 degrees) is determined by the snap threshold (as determined by the electronic device). For example, 10 degrees). As a result, in some embodiments, graphical objects (eg, 1414A and 1414B) remain displayed and centered in the display.

As illustrated in FIG. 14H, once the electronic device 1400 determines that the relative angle (eg, 11 degrees) between the horizontal plane and the focal plane exceeds the snap threshold (eg, 10 degrees), the graphical objects (eg, , 1414A and 1414B are shown to be separated from each other. In some embodiments, the initial snap threshold (eg, 10 degrees) is increased to account for some unintended hand movements (eg, 15 degrees) once the initial snap occurs. Thus, in some embodiments, as long as the electronic device determines that the graphical objects (eg, 1414A and 1414B) are within an increased snap threshold (eg, 15 degrees), the relative angle between the horizontal plane and the focal plane is increased. It will remain snapped to the center position of the image.

As illustrated in FIGS. 14I-14K, when the electronic device continues to detect that the relative angle between the horizontal plane and the focal plane is increasing, the graphical object 1414B moves further away from the graphical object 1414A. Is displayed continuously. In addition, as illustrated in FIGS. 14I and 14J, when the electronic device continues to detect that the relative angle between the horizontal plane and the focal plane is increasing, the visual indicator fades out gradually. As illustrated in FIG. 14K, when the electronic device determines that the relative angle between the horizontal plane and the focal plane is no longer within the predetermined alignment threshold, the visual indicator stops being displayed.

15A-15E are flow diagrams illustrating a method for composing an image using an electronic device (eg, 1400) in accordance with some embodiments. The method 700 may comprise a camera, a sensor (eg, a gyroscope, an accelerometer, a compass), one or more input devices (eg, a touch-sensitive surface, a keyboard, a mouse), and a device having a display (eg, 100). , 300, 500, 1400). In some embodiments, the device has a plurality of cameras, each camera optionally having a different focal length. Some operations of the method 1500 are optionally combined, some orders are optionally changed, and some operations are optionally omitted.

As described below, the method 1500 provides an intuitive method for constructing an image. This method reduces the user's cognitive burden of providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, allowing the user to launch various functions faster and more efficiently saves power and increases the time between battery charges.

In block 1502, the electronic device (eg, 1400) is on a display (eg, 1404) a camera viewfinder (eg, 1410) (eg, live or near live of content within the camera's field of view for capturing media). Display preview images).

At block 1504, the techniques of blocks 1506-1550 are optionally performed while displaying a camera viewfinder (eg, 1410).

At block 1506, according to the determination that the device meets the alignment guide display criteria, based on the data from the sensor, the techniques of blocks 1516 to 1524 are performed, and at block 1508, the alignment guide display criteria The relative difference between the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the predetermined orientation (eg, 1408) (eg, orientation with respect to the predetermined plane) such that the alignment guide display criteria are met (eg, , 1412) is within its alignment threshold. In some embodiments, each alignment threshold is a range of predefined angles (eg, less than 5 degrees, less than 10 degrees) in which the focal plane (eg, 1406) of the camera (eg, 1402) is parallel or perpendicular to the predetermined plane. Or less than 15 degrees). Displaying the alignment guide in the camera viewfinder (eg, 1410) when the alignment threshold is met provides user feedback in the form of alignment guide when the user is trying to position the device in a particular orientation, which allows the user to Allows for more precise positioning of in a specific orientation. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1510, the alignment guide display criteria are between the orientation of the focal plane (eg, 1406) and the predetermined orientation (eg, 1408) of the camera (eg, 1402) such that the alignment guide display criteria are met. Includes a requirement that a relative difference of (e.g., 1412) remain within its respective alignment threshold for at least a threshold amount of time (e.g., the alignment guide display criteria is at least a predetermined amount such that the alignment guide (e.g., 1414A and 1414B) is displayed). Requires that the device remain within the range of orientations corresponding to displaying the alignment guides (eg, 1414A and 1414B), so that the alignment guides (eg, 1414A and 1414B) allow the device to display the alignment guides (eg, 1414A and 1414B). For example, if briefly moved across a range of orientations corresponding to displaying 1414A and 1414B, it is not displayed. C). Displaying the alignment guide in the camera viewfinder (eg, 1410) when the alignment threshold is met for a threshold amount of time is more specific than when the device is only being redirected by the user (which does not keep the device in a particular orientation). Provide user feedback in the form of alignment guidance when the user is positioning the device in orientation. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1512, the alignment guide display criteria are between the orientation of the focal plane (eg, 1406) and the predetermined orientation (eg, 1408) of the camera (eg, 1402) such that the alignment guide display criteria are met. Includes the requirement that the orientation of the device does not change more than the threshold amount at the time of the threshold amount while the relative difference of (e.g., 1412) is maintained within the respective alignment threshold (e.g., the alignment guide display criteria include By requiring the device to remain in a substantially stationary orientation relative to a predetermined orientation (eg 1408) while within the range of orientations corresponding to displaying the alignment guide (eg, 1414A and 1414B) such that 1414A and 1414B are displayed, Alignment guides (eg, 1414A and 1414B) allow the device to display alignment guides (eg, 1414A and 1414B). If the following is too far, while a range of orientation corresponding to that is not displayed).

At block 1514, based on the data from the sensor, the alignment guide display criteria are not met (eg, the angle between the predetermined plane and the focal plane (eg, 1406) of the camera (eg, 1402) is a threshold value. Excess), depending on the determination (e.g., the camera lens is not sufficiently parallel to the horizontal / vertical plane), the electronic device (e.g., 1400) provides an alignment guide (e.g., 1414A and 1414B) to the camera viewfinder (e.g., Hold the display within 1410). Pending display of alignment instructions (eg, 1414A and 1414B) within the camera viewfinder reduces visual distractions to the user, helps to avoid turning the user's attention from the camera's viewfinder, and Avoid obstructing objects in the Reducing visual distractions and avoiding disturbing the viewfinder improves the operability of the device (e.g., by helping the user achieve the intended result and reducing user mistakes when operating and interacting with the device) and Makes the user-device interface more efficient, in addition, reduces the device's power usage and improves battery life.

In block 1516, the electronic device (eg, 1400), on the display (eg, 1404), an alignment guide (eg, 1414A and 1414B) in the camera viewfinder (eg, 1410) (eg, cross-shaped, rectangular, elliptical). , Triangles) (eg, overlaid on at least a portion of a live preview of the camera's (eg, 1402) field of view). At block 1518, the appearance of the alignment guides (eg, 1414A and 1414B) changes as the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) changes with respect to the predetermined orientation (eg, 1408). (Eg, alignment guides (eg, 1414A and 1414B) indicate inconsistencies from alignment). Displaying the alignment guide in the camera viewfinder (and withholding the display) based on whether the alignment guide display criteria are met provides feedback to the user about the orientation state of the device. For example, a user attempting to take a picture with the device's camera (eg, 1402) facing directly below is provided with visual feedback in the form of an alignment guide when the device is substantially facing downwards, whereby the device is It allows the user to better align the device (using the alignment guide) to face downward more accurately. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, the sensor is part of a camera (eg, 1402). In some embodiments, the sensor is independent of a camera (eg, 1402). In some embodiments, the focal plane (eg, 1406) of the camera (eg, 1402) should be sufficiently parallel to a horizontal plane, such as a plane perpendicular to the earth's surface or the earth's gravity. Parallel alignment allows for a perfect or near perfect alignment to capture the photos from above. In some embodiments, the focal plane (eg, 1406) of the camera (eg, 1402) should be sufficiently perpendicular to a horizontal plane, such as a plane parallel to the earth's surface or the earth's gravity. Vertical alignment allows for a perfect or near perfect alignment to capture straight line photos.

In some embodiments, the focal plane (eg, 1406) is a virtual two dimensional plane in front of the camera (eg, 1402) passing the focal point.

In some embodiments, the alignment guide (eg, 1414A and 1414B) is displayed when the guided display mode is active (and other criteria are met), and when the guided display mode is not active (even if other criteria are met). It is not displayed. In some embodiments, the guide display mode is the mode in which permanent guides (eg, grid) are displayed. Permanently displaying the guides while the guided display mode is enabled avoids user distractions that may result from the guides being repeatedly displayed and removed from the display. Reducing visual distractions improves the operability of the device (e.g., by helping the user achieve the intended result when operating and / or interacting with the device and reducing user mistakes) and makes the user-device interface more efficient. In addition, it reduces the power usage of the device and improves battery life.

In some embodiments, at block 1520, the alignment guide (eg, 1411A and 1411B) includes at least two visual indicators (eg, one indicator is displayed independently of the other indicator (eg, 1404). Change the position of the image).

In some embodiments, at block 1522, at least one of the at least two visual indicators (eg, one indicator changes position on the display (eg, 1404) independently of the other indicator) is a camera Stay stationary as the orientation of the focal plane (eg, 1406) of (eg, 1402) changes with respect to the predetermined orientation (eg, 1408). Displaying a visual indicator that remains stationary on a device's display (eg, 1404) may cause the device to determine a predetermined orientation (eg, 1408) when the visual indicator of the stationary state is compared to the position of the visual indicator of the non-still state. Provide the user with visual feedback as to how close they are to. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1524, the distance between two visual indicators is a relative difference between the orientation of the focal plane (eg, 1406) and the predetermined orientation (eg, 1408) of the camera (eg, 1402). (Eg, 1412) dynamically. In some embodiments, the further the indicators are, the further away the device is from being positioned to take a top down picture of the field of view of the one or more cameras. Dynamically updating the distance between two visual indicators provides the user with visual feedback as to the degree of change in the orientation of the device required to achieve a predetermined orientation (eg, 1408), and alters the orientation of the device. Doing so provides the user with visual feedback indicating that the distance between the two visual indicators will change. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1526, while displaying an alignment guide (eg, 1414A and 1414B) and while the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is the first orientation, the electronic The device (eg, 1400) detects a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation based on the data from the sensor.

In some embodiments, at block 1528, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation, the block 1530. ), The electronic device (eg, 1400) is based on the displayed position (eg, the degree of change in orientation) of the first visual indicator (eg, moving or non-static indicator) of the at least two visual indicators. The calculated position, the position not based on the position of the stop indicator, and the displayed position of the first visual indicator is changed based on a relative difference (eg, 1412) between the first and second orientations. (Eg, one indicator changes position on the display (eg, 1404) independently of the other indicator). In some embodiments, at least one of the two separate visual indicators is unfrozen when the devices reposition beyond a predetermined threshold.

In some embodiments, at block 1532, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation, at camera 1532. The relative difference (eg, 1412) between the second orientation of the focal plane (eg, 1406) (eg, 1406) and the predetermined orientation (eg, 1408) is determined by the first visual indicator alignment threshold (eg, indicator snapping). In accordance with a determination that the indicators in the threshold, predetermined sorted arrangement, are not within the range of relative differences (eg, 1412) values to be displayed (eg, 0 degrees to 5 degrees), the electronic device (eg, 1400) is updated. Display the first visual indicator at the display position. In some embodiments, at block 1534, the camera is further responsive to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the first orientation to the second orientation. In accordance with the determination that the relative difference (eg, 1412) between the second orientation of the focal plane (eg, 1406) (eg, 1406) and the predetermined orientation (eg, 1408) is within the first visual indicator alignment threshold, the former The device (eg, 1400) displays the first visual indicator at a predetermined display position (eg, a complete alignment with the snapped position, eg, a stop indicator). Snapping indicators into alignment when the current orientation of the device is within a threshold amount of a predetermined orientation (eg, 1408) provides the user with visual feedback of the orientation state of the device, in particular the device may Substantially aligned with 1408. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1536, while the first visual indicator is displayed at a predetermined display location (eg, a fully aligned with a snapped position, eg, a stop indicator), the electronic device (eg, , 1400 detects a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the third orientation to the fourth orientation, based on the data from the sensor, the third orientation being the first Have a relative difference (eg, 1412) relative to a predetermined orientation (eg, 1408) that is within the visual indicator alignment threshold. In some embodiments, the change in orientation of the focal plane (eg 1406) of the camera (eg 1402) begins while the orientation of the focal plane (eg 1406) of the camera (eg 1402) is the third orientation. In some embodiments, the fourth orientation has a relative difference (eg, 1412) relative to a predetermined orientation (eg, 1408) that is outside the first visual indicator alignment threshold.

In some embodiments, at block 1538, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the third orientation to the fourth orientation, the blocks ( 1540-1542) is performed.

In some embodiments, at block 1540, the relative difference (eg, 1412) between the fourth orientation of the focal plane (eg, 1406) and the predetermined orientation (eg, 1408) of the camera (eg, 1402) is determined. Two visual indicator alignment thresholds (e.g., indicator snapping threshold, determination that the indicators are outside the range of relative differences (e.g., 1412) values (e.g., 0 degrees to 5 degrees) that are to be displayed in a predetermined alignment arrangement. Accordingly, the electronic device (eg, 1400) maintains the display of the first visual indicator at the predetermined display position. In some embodiments, the second visual indicator alignment threshold is equal to the first visual indicator alignment threshold. In some embodiments, the second visual indicator alignment threshold is equal to the first visual indicator (eg, to prevent the visual indicator from accidentally snapping out of alignment if the user keeps the device near the first visual indicator alignment threshold). It is different from the indicator alignment threshold.

In some embodiments, at block 1542, the relative difference (eg, 1412) between the fourth orientation of the focal plane (eg, 1406) and the predetermined orientation (eg, 1408) of the camera (eg, 1402) is determined. In accordance with the determination that it is outside the 2 visual indicator alignment threshold, display the first visual indicator at the second updated display position, wherein the position of the second updated display position is determined by the focal plane (eg, 1406) of the camera (eg, 1402). Is based on a relative difference between the orientation of and a predetermined orientation (eg, 1408) (eg, 1412). In some embodiments, while the moving visual indicator is snapped to the stationary visual indicator, the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) and the visual indicator alignment threshold are outside (eg, exceeded). The change in orientation resulting in a relative difference (eg, 1412) in a predetermined orientation (eg, 1408) is such that the moving visual indicator is based on the degree of change in orientation and not based on the position of the stop indicator. Will be displayed in the new position. In contrast, a change in orientation that results in a relative difference (eg, 1412) that is below the visual indicator alignment threshold does not cause the position of the moving visual indicator to be updated (eg, it remains snapped to the still visual indicator). . Snapping indicators out of alignment when the current orientation of the device exceeds a threshold amount relative to a predetermined orientation (eg, 1408) provides the user with visual feedback of the orientation state of the device, in particular the device having a predetermined orientation (eg, 1408, which is not substantially aligned. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, at block 1544, the relative difference (eg, 1408) with respect to the predetermined orientation (eg, 1408) where the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is within its respective alignment threshold. While in the seventh orientation with 1412, and while the third visual indicator of the at least two visual indicators is displayed, the electronic device (eg, 1400) is based on the data from the sensor, from the seventh orientation to the eighth orientation. Detect a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) to.

In some embodiments, at block 1546, in response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the seventh orientation to the eighth orientation, the blocks ( 1548-1550) are performed.

In some embodiments, at block 1548, the relative difference (eg, 1412) between the eighth orientation of the focal plane (eg, 1406) and the predetermined orientation (eg, 1408) of the camera (eg, 1402) is visual. In accordance with the determination that it is within an indicator alignment threshold, the electronic device (eg, 1400) maintains the display of the third visual indicator (eg, maintains the display in the same or different positions, and similar visual characteristics (eg, display intensity)). Or maintain displays with different visual characteristics).

In some embodiments, at block 1550, the relative difference (eg, 1412) between the eighth and predetermined orientations (eg, 1408) of the focal plane (eg, 1406) of the camera (eg, 1402) is visual. In accordance with a determination that it is not within an indicator alignment threshold, the electronic device (eg, 1400) stops displaying the third visual indicator. Stopping displaying alignment guides (eg, 1414A and 1414B) in the camera viewfinder when the difference between the current orientation and the predetermined orientation of the device exceeds a threshold amount is such that the user displays the display of the alignment guides (eg, 1414A and 1414B). To switch between not displaying alignment guidance (eg, 1414A and 1414B) so that they display additional controls when they are helpful and do not display additional controls when they are not helpful. Providing them when the additional control options are helpful and not cluttering the UI with them when the additional displayed control options are not helpful (eg, when the user operates and / or interacts with the device, To improve the device's operability and make the user-device interface more efficient, by helping to provide and reducing user mistakes, in addition to enabling the user to use the device more quickly and efficiently Reduce and improve battery life.

In some embodiments, the predetermined orientation (eg, 1408) corresponds to a horizontal orientation (eg, a plane perpendicular to the earth's surface or the earth's gravity).

In some embodiments, the predetermined orientation (eg, 1408) is a vertical orientation (eg, parallel to the gravity of the earth).

In some embodiments, at least one of the at least two visual indicators (eg, one indicator changes position on the display (eg, 1404) independently of the other indicators) is a camera viewfinder (eg, 1410). Are displayed close to (eg, at or near) the center. Displaying one of the visual indicators close to the center of the camera viewfinder provides additional space in all directions of the central visual indicator for displaying the other visual indicator, whereby between the predetermined orientation and the current device orientation. The difference in maximizes the granularity provided to the user as visual feedback. Providing the user with improved visual feedback improves the operability of the device (eg, by helping the user provide appropriate inputs and reduces user mistakes when operating and interacting with the device) and the user-device interface Makes the device more efficient, which further reduces the power usage of the device and improves battery life by allowing the user to use the device more quickly and efficiently.

In some embodiments, displaying the first visual indicator at a predetermined display position includes displaying a respective (eg, predefined) animation at one or more of the visual indicators (eg, displaying a highlight). , Flashing, switching the flashing and / or moving visual indicator at the position of the fixed visual indicator from the first position to the predetermined display position. In some embodiments, the animation allows the user to understand that the image is aligned. Animation draws the user's attention to that fact.

In some embodiments, a fifth with a relative difference (eg, 1412) relative to a predetermined orientation (eg, 1408) where the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) is within its alignment threshold. While in orientation, and while the second visual indicator of the at least two visual indicators is displayed at the first value of the visual characteristic (eg, first visual intensity, first color, first size), the electronic device (eg, 1400). ) Detects a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the fifth orientation to the sixth orientation based on the data from the sensor, and detects the focus of the camera (eg, 1402). The relative difference (eg, 1412) between the sixth orientation of the plane (eg, 1406) and the predetermined orientation (eg, 1408) is within the visual indicator alignment threshold. In response to detecting a change in the orientation of the focal plane (eg, 1406) of the camera (eg, 1402) from the fifth orientation to the sixth orientation, having a second value of a visual characteristic different from the first value (eg, Display a second visual indicator having a larger (or smaller) visual or display intensity, different color, different size). Displaying visual indicators with different visual characteristics provides the user with additional visual feedback on the alignment of the device. Providing improved visual feedback improves the operability of the device (e.g., by helping the user achieve the intended result and reduces user mistakes when operating and interacting with the device) and makes the user-device interface more efficient. This, in addition, reduces the power usage of the device and improves battery life.

In some embodiments, the fifth orientation of the focal plane (eg 1406) of the camera (eg 1402) is the orientation of the focal plane (eg 1406) of the camera (eg 1402) and the predetermined orientation (eg 1408). Have a first relative difference (eg, 1412) between. In some embodiments, the sixth orientation of the focal plane (eg 1406) of the camera (eg 1402) is the orientation of the focal plane (eg 1406) of the camera (eg 1402) and the predetermined orientation (eg 1408). Have a second relative difference (e.g., 1412), and the second relative distance is greater than the first relative difference (e.g., 1412) (e.g., the sixth orientation is in a predetermined orientation (e.g., 1408) (e.g., horizontal). More out of alignment). In some embodiments, the visual characteristic is display intensity (eg, degree of fade relative to maximum display intensity). In some embodiments, the first value is greater than the second value (eg, the first visual indicator is more prominently displayed (eg, less faded) when the camera orientation is closer to the predetermined orientation (eg, 1408)). As the camera orientation moves away from the predetermined orientation (eg, 1408), the visual indicator fades further until it completely fades out. Enhancing the orientation guide as the camera orientation moves toward a predetermined orientation, and fading the orientation guide as the camera orientation moves further away from the predetermined orientation results in a visual indication that the device is moving in and out of the range in which the orientation guide will be displayed. Provide feedback to the user. Providing improved visual feedback improves the operability of the device (e.g., by helping the user achieve the intended result and reduces user mistakes when operating and interacting with the device) and makes the user-device interface more efficient. This, in addition, reduces the power usage of the device and improves battery life.

Note that the details of the processes described above with respect to the method 1500 (eg, FIGS. 15A-15E) are also applicable in a similar manner to the methods described above and below. For example, the method 700, 900, 1100, 1300, 1700 optionally includes one or more of the features of the various methods described above with respect to the method 1500. For example, elements of the alignment interface, affordances and controls can be combined among the various methods. As another example, the viewfinder of method 1500 is similar to the viewfinder of methods 700, 900, 1100, 1300, 1700. For brevity, these details are not repeated below.

16A illustrates the device 1600. In some embodiments, electronic device 1600 includes some or all of the components of device 600. In some embodiments, device 1600 includes a number of cameras 602 and 603 (eg, on the backside of electronic device 1600). In some embodiments, device 1600 includes one or more features of devices 100, 300 and / or 500. In some examples, the electronic device (eg, 1600) has multiple cameras with fixed but different focal lengths. In some examples, multiple cameras are on the front, back, or both sides of an electronic device (eg, 1600). In some embodiments, in addition to having different fixed focal lengths, multiple cameras have different fixed fields of view and different fixed optical magnification properties. In some embodiments, the camera (eg, 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (eg, 602) captures a plurality of focal lengths, producing the same result as a plurality of cameras with fixed but different focal lengths. In some examples, the electronic device (eg, 1600) includes a depth camera (eg, 1690), such as an infrared camera, thermographic camera, or a combination thereof. In some examples, the device further includes a light emitting device (eg, a light projector), such as an IR flood light, structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the subject during the capture of the image by the visible light camera and the depth camera (eg IR camera), and information from the visible light camera and the depth camera (eg 1690) is captured by the visible light camera. It is used to determine the depth map of the different parts of. In some embodiments, the lighting effects described herein use parallax information from two cameras (eg, two visible light cameras) for the back face images and front face images (eg, selfies). Display using depth information from a depth camera combined with image data from a visible light camera (eg, 1692) for the images. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when the depth camera is used to determine depth information, so as to determine the information used when generating lighting effects. Even when using different technologies, it provides a consistent experience for the user. In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects the selection of the camera switching affordance, maintains and displays the front of the user interface controls for applying the lighting effect. Rear facing cameras (eg, from each other) from front facing cameras (eg, depth camera 1690 and visible light camera 1662), replacing the display of the field of view of facing cameras with the field of view of rear facing cameras (or vice versa). Two visible light cameras spaced apart (or vice versa).

As illustrated in FIG. 16A, the electronic device 1600 includes a display 1604 that is touch sensitive (eg, a touch screen), the display displaying image data received from a camera (eg, 602). In some embodiments, the display is separate from the touch-sensitive surface. In some examples, multiple cameras (eg, 602 and 603) are located on the front, back, or both sides of the electronic device (eg, 1600).

16A further illustrates that the electronic device 1600 displays a camera application user interface 1605 on the display 1604 for capturing images with a camera (eg, 602). Camera application user interface 1605 further includes a digital viewfinder 1610 that includes a live preview of the field of view of the camera (eg, 602 or 603). In some embodiments, the camera captures depth information associated with image data in real time. 16A further illustrates a camera that captures distinct depth levels in the field of view, including a subject (eg, female) in the foreground area (eg, 1608), and a fence in the background area (eg, 1609). In some examples, the camera (eg, 602) captures three, four, five, ten, twenty or more depth levels in the field of view. The electronic device 1600 utilizes various depth levels when applying a filter to the representation of image data (eg, 1606) displayed in the digital viewfinder (eg, 1610) as discussed in more detail below.

16A also illustrates displaying the filter picker user interface 1613 with the electronic device 1600 expanded. The filter picker user interface 1613 is located along the edge of the digital viewfinder 1610. In some examples, the filter picker user interface (eg, 1613) is optionally displayed above, below, left, or right of the digital viewfinder (eg, 1610). In some examples, filter picker user interface (eg, 1613) includes one or more representations (eg, representations of visual effects) of filters arranged in one or more rows and columns or positioned in a circular orientation.

In some examples, the extended filter picker user interface (eg, 1613) is displayed at any location corresponding to the digital viewfinder. In some examples, the filter picker user interface (eg, 1613) is outlined (eg, a border) to distinguish the filter picker user interface from the digital viewfinder. In some embodiments, the filter picker user interface (eg, 1613), when in the folded state, is translucent (or partially translucent) and has no visible boundaries. As a result, in some examples, the filter picker user interface (eg, 1613) appears to blend in with the digital viewfinder (eg, indistinguishable).

As shown in FIG. 16A, the electronic device 1600 displays a filter picker user interface 1613 that includes one or more filter representations (eg, 1614A, 1614B, 1614C, 1614D, 1614E) corresponding to visual effects. do. In some examples, the filter picker user interface (eg, 1613) optionally includes additional filter representations that are not displayed on the display (eg, they are off screen). In some examples, non-displayed filter representations are displayed when the electronic device receives an input (eg, a swipe gesture), which will cause the filter representations to scroll through the filter container (eg, 1616).

As further illustrated in FIG. 16A, in some embodiments, electronic device 1600 displays filter representation 1614A in an extended filter picker user interface (eg, 1613) to display a currently selected visual effect. . In some embodiments, filter representation 1614A corresponds to a "natural light" light effect filter. As a result, foreground area 1608 and background area 1609 are displayed with a “natural light” light effect filter (eg, using natural lighting from the scene). Since the image representation of FIG. 16A is shown without using any synthetic light (eg, simulated), natural light from the scene produces varying shadows on an object (eg, face, neck, and clothing). do. In some examples, possible filter representations (eg, 1614A through 1614E) corresponding to lighting effects include a “studio light” lighting effect, a “contour light” lighting effect, a “stage light” lighting effect, and a “stage light mono” lighting. Includes effects. The aforementioned lighting effects, when applied to the representation of the image data (eg, 1606), affect the visual characteristics of the representation of the image data displayed on the display 1604.

In some embodiments, when the electronic device 1600 applies a natural lighting effect, no synthetic lighting is added to the image (eg, the original image is displayed). In contrast, studio lighting effects include modeling of a number of individual point light sources (eg, lights in a photography studio) that are uniformly positioned around a subject (eg, creating a brightly filled lighting effect). The contour illumination effect includes the modeling of a number of individual point light sources located along the perimeter of the subject (eg, creating a slimming effect, and generating shadows on the side of the subject's face and / or over the subject's chin). do). Stage light illumination effects include modeling (eg, generating spotlight effects) of a single individual point light source positioned over an object. Stage light mono illumination effects include modeling in black and white (eg, generating spotlight effects in black and white) of a single individual point light source positioned over an object.

In some examples, the electronic device (eg, 1600) detects the subject's face in the representation of the image data. As a result, the electronic device uses depth map information of the image data and corresponding facial features when applying the lighting effect. As a result, the lighting effect is applied with greater precision around the subject's face, and certain facial features are based on the selected lighting effect (eg, increasing or decreasing shadows around the subject's jaw and / or cheek bones). Can be illuminated differently. In some examples, the image data includes depth map information that includes depth contours of the objects. As a result, the electronic device uses the contour data to more accurately apply the lighting effect around the object.

As illustrated in FIG. 16B, the electronic device 1600 receives an input (eg, tap 1618) at a location corresponding to the extended filter picker user interface (eg, 1613). As shown in FIG. 16C, input (eg, swipe 1618) causes filter representations 1614A- 1614E to scroll through filter vessel 1616. In some embodiments, in response to input (eg, swipe 1618), the filter representations will scroll left across the top boundary of the filter picker user interface (eg, 1613). In some examples, a single swipe gesture will result in an incremental scrolling of filter expressions. In some embodiments, the number of scrolled filter expressions will depend on the size of the swipe gesture. Thus, in some examples, a longer swipe will result in a longer scroll than a shorter swipe.

In response to an input (eg, 1618 swipe), in some embodiments, the electronic device 1600 applies a lighting effect corresponding to a filter representation (eg, 1614B) corresponding to a location on the display 1604 of the tap input. do. In some examples, the input is a swipe, press and hold, or an input having a characteristic intensity that is above a respective intensity threshold. In some examples, an input having a characteristic intensity above a respective intensity threshold detected on the representation of one of the filters 1614A-1614F may optionally be located with a location of the input having a characteristic intensity above a respective intensity threshold. This results in the display of additional functionality for the associated corresponding filter representation.

In some embodiments, additional visual effects may be applied to the overall representation of the image data (eg, 1606) before applying the lighting effect corresponding to the filter representation (eg, 1614B). For example, some gradient filling may be applied to the representation of the image data (eg, 1606) before applying the stage lighting filter to more fluidly switch from no filter to the lighting effect filter.

16C and 16D illustrate the electronic device 1600 gradually applying the lighting effect as a result of the electronic device receiving an input (eg, swipe 1618) in FIG. 16B. In some embodiments, the lighting effect corresponding to the newly selected filter representation 1614B is gradually applied to the representation of image data (eg, 1606) in the live preview. Since the selected lighting effect is "studio light", the corresponding visual effect simulates a number of point light sources affecting the object in the foreground area 1608. As a result, during the transition stage (FIG. 16C), the lighting effect corresponding to the filter representation 1614B is applied to the live preview at 50% intensity. The filter corresponding to filter representation 1614B is fully applied in FIG. 16D (eg, 100%). In some examples, the filter is applied in increments (10%, 25%, 50%, 75%) while the electronic device 1600 applies the lighting effect until the transition is complete. In some examples, the background area (eg, 1609) becomes completely dark when the electronic device 1600 applies a "studio light" light effect to the representation of the image data. In some embodiments, the background area (eg, 1609) is partially darkened when the electronic device 1600 applies a “studio light” light effect to the representation of the image data.

As illustrated in FIGS. 16C and 16D, since the image data captured by the camera (eg, 602) includes depth map information associated with the image data, the electronic device may use the available depth map information and the image data. The effect of various point sources on the representation of 1606 can be simulated. In some embodiments, the same lighting effect is applied differently in the background area compared to the foreground area based on depth map information associated with the image data. As a result, objects in the foreground may appear more prominent, and objects in the background may become less prominent through the darkening effect. Additionally, as illustrated in FIG. 16D, since the lighting effects simulate point sources, depth map information is used to cast varying shadows on the subject's face in the foreground area (eg, 1608). As illustrated in FIG. 16D, since the “studio light” lighting effect simulates a plurality of point light sources, the electronic device 1600 may extract depth map information to reduce shadow on the subject's face compared to the “natural light” lighting effect. use.

In some embodiments, when capturing an image, lighting effects are applied to the digital viewfinder for preview. In some embodiments, the lighting effect is associated with the image after capture.

As illustrated in FIG. 16E, the electronic device receives a gesture (eg, tap 1632) at a location corresponding to shutter button affordance 1630. In some embodiments, in response to receiving a gesture (eg, tap 1632), the electronic device 1600 will capture image data corresponding to the field of view of one or more cameras (eg, 602 and 603). The electronic device will associate the currently selected (eg, activated) lighting effect with the representation of the image data. The generated image with the applied lighting effect will be stored in the camera roll of the electronic device and will be available for review in the camera roll application (eg, as represented by affordance 1634).

As illustrated in FIG. 16F, the electronic device 1600 receives an input (eg, tap 1620) and causes the representations of the filters 1614A- 1614E to scroll through the filter container (eg, 1616). In some embodiments, in response to a tap gesture (eg, 1620) corresponding to the filter representation (eg, 1614E), the representations of the filters are leftward across the upper periphery of the extended filter picker user interface (eg, 1613). Will scroll. FIG. 16F further illustrates that the affordance for the camera roll application (eg, 1634) changes appearance to correspond to the last captured image (eg, the studio light image captured in FIG. 16E). In some embodiments, the affordance for the camera roll application (eg, 1634) is displayed as a thumbnail representation of the captured image corresponding to the representation of the image. In some embodiments, the affordance for the camera roll application (eg, 1634) is displayed as a thumbnail icon of a that is not similar to the captured image corresponding to the representation of the image.

FIG. 16G illustrates the result of the tap gesture (eg, 1620) of FIG. 16F. In response to the tap gesture, the electronic device 1600 switches to a "stage light mono" light effect mode. As illustrated in FIG. 16G, while the electronic device 1600 is in “stage light mono” mode, the alignment guide 1640 is displayed as a circle and positioned slightly off-center in the vertical direction on the display 1604. . In some embodiments, the alignment guide (eg 1640) is represented in a hexagonal, elliptical, or triangular shape. In some embodiments, the alignment guide is located in the middle of the display (eg, 1604) in both the vertical and horizontal directions. In some embodiments, the alignment guide (eg, 1640) is located on the display (eg, 1604) off center in one or both of the vertical and horizontal directions.

In some embodiments, the alignment guide is used to assist the user in aligning the body part of the subject (eg, their face) within a particular portion of the digital viewfinder. As illustrated in FIG. 16F, because the electronic device has determined that certain criteria are not met (eg, the subject's face is not properly aligned within the alignment guide 1640), the electronic device may take appropriate action on the user. Display message 1622 to prompt for action. In some embodiments, the lighting effect is based on image data (unless the electronic device determines that appropriate criteria have been met (eg, the subject's face is properly aligned with the alignment guide and the subject is at the proper distance from the device). For example, it will not apply to the representation of 1606.

As further illustrated in FIG. 16G, the area outside the alignment guide 1640 is slightly shaded while the area inside the alignment guide 1640 remains unchanged. In some embodiments, the area outside the alignment guide (eg, 1640) is not shaded, and the area inside the alignment guide (eg, 1640) is also not shaded.

16H illustrates the user taking some steps towards the device such that the subject's face is properly aligned within the alignment guide 1640. As illustrated in FIG. 16H, when the subject's face is properly aligned with the alignment guide 1640, the area outside the alignment guide (eg, 1640) is shaded in a darker color and the area inside the alignment guide 1640 It is not shaded (eg no artificial lighting is applied). In some embodiments, the dark shade serves as a “lock-on” indicator and provides the user with an indication that the criteria for the selected lighting effect (eg, stage light mono) have been met. In some embodiments, the currently selected lighting effect (eg, stage light mono) is applied to an area within the alignment guide (eg, 1640). In some embodiments, additional criteria (eg, object is at the required distance from the electronic device, lighting conditions) must be met for dark shades outside the alignment guide that will occur.

In addition, as further illustrated in FIG. 16H, the blurring (eg, bokeh; depicted in dashed lines) optical effects may result in both the full representation of the image data 1606 (eg, both the interior of the alignment guide and the area without the alignment guide). ) Is applied to the background (eg, fence) of the representation of image data 1606. In some embodiments, the blurring optical effect is applied disproportionately to different portions of the representation of the image data (eg, 1606) displayed on the display (eg, 1604) when the alignment criteria are met.

In some embodiments, the alignment guide (eg, 1640) is displayed in other (eg, not stage light mono) lighting effects that require removal of the background (eg, blurring).

In some examples, when the conditions for the alignment guide are no longer met (eg, the subject's face is no longer aligned with the alignment guide, the subject has moved too far back from the device, and is no longer at a suitable distance from the device). And the lighting conditions are not accurate), the previously applied lighting effect is snapped out (eg, the electronic device returns slightly darker / not dark at the outside of the alignment guide as illustrated in FIG. 16G). In some embodiments, when the dark shade snaps out, optionally, a temporary filter (eg, a gradient) that is part of the lighting effect filter is applied to the image representation. Temporary filters help smooth transitions when shading is reapplied (eg less jarring).

As further illustrated in FIG. 16H, the electronic device 1600 receives a gesture (eg, tap 1636) at a location corresponding to the shutter button affordance 1630. In some embodiments, in response to receiving a gesture (eg, tap 1636), the electronic device 1600 captures image data corresponding to the field of view of one or more cameras (eg, 602 and 603), The electronic device associates the currently selected (eg, activated) lighting effect with the representation of the image data. In these embodiments, the generated image with the applied lighting effect is then stored in the camera roll of the electronic device and available for review in the camera roll application (eg, as represented by affordance 1634). . In some embodiments, the lighting effect will be applied to the image data after capture even if it is not applied to the digital viewfinder during capture (eg, the captured image will appear different from what is seen by the user in the digital viewfinder). .

As illustrated in FIG. 16I, the electronic device detects an input (eg, tap 1645) at a location corresponding to the photo viewer application (eg, 1634). In response to receiving an input (eg, 1645), as illustrated in FIG. 16J, the electronic device 1600 may perform an image viewing mode (eg, for viewing previously captured images, rather than a live preview of camera data). Mode).

16J illustrates a user interface for a photo viewer application. The photo viewer application includes a thumb strip of previously captured images (eg, 1628A-1628C), where 1628C is the last captured image. In some examples, previously captured images were captured using a camera corresponding to an electronic device (eg, 1600). In some examples, the electronic device (eg, 1600) received previously captured images (eg, 1628A through 1628C) from a remote source (eg, a server), and optionally, previously captured images may be different from the electronic device ( For example, not 1600).

16J further shows that the last captured image (eg, 1628C) was captured using stage light mono illumination. The "stage light mono" lighting effect simulates a single point light source, with a result similar to the spotlight effect. Using the depth map information, the electronic device 1600 applies a “stage light mono” effect from above the object in the foreground area (eg, 1608). In some examples, the point light source can be simulated to originate from any direction. In some examples, the “stage light mono” effect is simulated to come from the front, as a result of which a particular focus (eg, face) is emphasized, but the remainder of the representation of the image data is darkened. As illustrated in FIG. 16J, since the simulated point light source is from above the object, the electronic device can cast deeper shadows on the object (eg, face and neck) using the depth map information of the image data. have. In some embodiments, the background of the image is removed and replaced with a solid color, such as black or white, or a user-selected color, to attract more attention to the subject in the foreground and allow the user to be photographed for a solid backdrop. Simulate studio settings.

In some embodiments, the electronic device displays a “vertical” visual indicator (eg, 1638) at the top of the display as an indication to the user that previously captured image data includes depth map information. In some examples, the electronic device (eg, 1600) receives input at a location corresponding to the visual indicator to toggle the simulated depth effect (eg, bokeh effect) on and off. In some embodiments, if the simulated depth effect is toggled off, the lighting effect will be maintained. In some examples, the visual indicator will toggle the simulated depth and lighting effects together when activated. In some examples, the electronic device optionally receives input for changing the lighting effect in the photo viewer application to a different lighting effect using the filter picker user interface (as described above). In some examples, previously captured image data does not have depth map information; The electronic device (eg, 1600) will not provide the option to apply the simulated depth effect or lighting effect. In some examples, if the previously captured image data does not have depth map information associated with the image data, the electronic device will not display a visual indicator (eg, 1638).

In some examples, the electronic device (eg, 1600) stores depth map information along with the image data in one file. In some examples, the electronic device (eg, 1600) stores depth map information separately from image data. In some embodiments, if the electronic device stores an image with depth map information as a flat image (eg, without depth map information), the electronic device (eg, 1600) may no longer apply the lighting effect to the representation of the image data. I will not be able to.

17A-17G illustrate a method for applying a simulated lighting effect to a representation of image data (eg, 1606) and displaying an alignment guide (eg, 1640) using an electronic device (eg, 1600) in accordance with some embodiments. A flowchart illustrating the method. The method 1700 may include one or more input devices (eg, touch-sensitive surface, keyboard, mouse), one or more cameras (eg, cameras with variable focal lengths and / or cameras that capture depth sensing cameras and color images). ) And a device (eg, 100, 300, 500, 1700) having a display (eg, 1604). In some embodiments, the display (eg, 1604) is a touch sensitive display. In some embodiments, the display (eg, 1604) is not a touch sensitive display. In some embodiments, the electronic device (eg, 1600) includes a plurality of cameras. In some embodiments, the electronic device (eg, 1600) has only one camera. Some operations of the method 1700 are optionally combined, some orders are optionally changed, and some operations are optionally omitted.

As described below, the method 1700 provides an intuitive way to display alignment guidance and apply simulated lighting effects to the representation of image data when certain criteria are met. This method reduces the user's cognitive burden of providing inputs corresponding to the functions, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, allowing the user to launch various functions faster and more efficiently saves power and increases the time between battery charges.

At block 1702, the electronic device (eg, 1600) displays, on the display (eg, 1604), a representation of the image data (eg, 1606) associated with the depth map information (eg, the image or photo is displayed on the device). Display on a display (eg, 1604). In some embodiments, a live preview of image data is displayed in a digital viewfinder (eg, 1610).

In some embodiments, at block 1704, the representation of image data is a live preview of image data captured within the field of view of one or more cameras displayed in a digital viewfinder (eg, 1610).

In some embodiments, at block 1706, the depth map information associated with the image data includes information corresponding to at least three different depth levels (eg, at least background, depth level, foreground depth level, and intermediate depth level). Include. Depth map information comprising three or more different depth levels provides the user with a framework for applying depth-specific filters and provides the user with more precise feedback on depth positioning of objects within the field of view of the camera (s). . Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 1708, the depth map information associated with the image data includes information identifying depth contours of an object in the representation of the image data; The lighting effects change the appearance of the representation of the image data based on the position and curvature of the contours of the object. Including the depth contours of the objects in the depth map information allows the device to provide the user with more precise visual feedback on the shape and depth positioning of the objects within the field of view of the camera (s). Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve. Changing the appearance of the representation of the image data based on the position and curvature of the contours of the object creates a lighting effect that simulates the use of external lighting equipment without requiring external lighting equipment. Creating such lighting effects without requiring external lighting equipment reduces the size and cost of the equipment needed to produce an image with specific lighting effects.

At block 1710, while displaying a representation of the image data on a display (eg, 1604), the electronic device (eg, 1600), at block 1712, receives a first input (eg, via one or more input devices). Detect swipes, taps and hold, taps (eg, 1620), button presses; gestures based on depth map information (eg, based on measurements of depth sensor or simultaneously taken from different locations A plurality of lighting effects (eg, one or more of natural light, studio light, contour light, stage light, stage light mono, as described in more detail in paragraph [0499] below) based on the disparity mapping between them) An icon representing an illumination filter (eg, filter representations 1614A, 1614B, 1614C, 1614D, 1614E) (or other user interface used to select a filter) Face system) Simultaneously displaying a representation of the image data (eg, 1606) and providing a filter selection interface (eg, filter picker user interface 1613) can be applied to objects and previews in the camera's field of view. Provide the user with visual feedback on the filters available to be applied Providing improved visual feedback to the user may cause the device to generate the intended result (eg, when operating and interacting with the device). By providing feedback indicative of input to help the user achieve the intended result and reduce user mistakes), it improves the device's operability and makes the user-device interface more efficient, which in addition, allows the user to make the device faster and I can use it efficiently It by reducing the power consumption of the device and improve the battery life.

In some embodiments, at block 1714, the first input is an input received while the first criteria (eg, a set of lighting effect application criteria) are met, the first criteria being electronic so that the first criteria are met. A requirement that an object is detected in a field of view within a predetermined distance from the device (eg, 1600). Other criteria include that the focal length of the first camera exceeds the minimum distance threshold, that the focal length of the first camera does not exceed the maximum distance threshold, that the object is detected above the predetermined minimum distance from the device, The amount of detected light exceeds the minimum light threshold, and the amount of detected light does not exceed the maximum light threshold. In some embodiments, if the first criteria are not met, the application withholds the first or second lighting effect. Applying the lighting effect when it is determined that the object is within a predetermined distance when the first input is received may result in a user receiving visual feedback that the object is properly positioned so that an optimal (or nearly optimal) effect can be achieved with the filter. To provide. Similarly, not applying the lighting effect when the object is not within a predetermined distance provides the user with feedback that the object is not properly positioned and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, at block 1716, the first input is an input received while the first criteria are not met, and the first criteria are a predetermined distance from the electronic device (eg, 1600) such that the first criteria are met. Includes a requirement that an object is detected in the field of view within; The method comprises: after displaying the live preview without applying the first change in appearance to the live preview, detecting that the first criteria have been met; And in response to detecting that the first criteria have been met, applying the first change in appearance to the live preview. In some embodiments, other conditions in the set include: the focal length of the first camera exceeds a minimum distance threshold, the focal length of the first camera does not exceed a maximum distance threshold, and the subject has a predetermined minimum from the device. One or more of being detected over a distance, an amount of detected light exceeds a minimum light threshold, and an amount of detected light does not exceed a maximum light threshold. In some embodiments, if the first criteria are not met, the application withholds the first or second lighting effect. Applying the lighting effect when the first criteria are met gives the user visual feedback that the lighting effect application criteria have been met (eg, the object is properly positioned) and that an optimal (or near optimal) effect can be achieved with the filter. to provide. Similarly, not applying the lighting effect when the first criteria are not met provides feedback to the user that the first criteria are not met and indicates to the user that a corrective action is required. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

At block 1710, while displaying a representation of the image data on a display (eg, 1604), the electronic device (eg, 1600) optionally performs the techniques of blocks 1718-1742.

In some embodiments, at block 1718, in response to detecting the first input, the electronic device (eg, 1600) prepares to capture image data using a respective filter.

In some embodiments, at block 1720, the electronic device (eg, 1600) is one in accordance with determining that the respective lighting effect is a first lighting effect (eg, stage light or other lighting effect including background removal). Capture user interface for a first lighting effect that results in a first change in appearance of a portion of the representation of the image data when an object in the field of view of the cameras meets the first criteria (eg, lighting effect application criteria). For example, an alignment guide (eg, 1640) (eg, an alignment user interface that changes appearance when the first criteria are met).

In some embodiments, at block 1722, the electronic device according to determining that the respective lighting effect is a second lighting effect (eg, natural light, studio light, or contour light or other lighting effect that does not include background removal). (Eg, 1600) indicates a second illumination that results in a second change in appearance of the portion of the representation of the image data when an object in the field of view of the one or more cameras meets the first criteria (eg, lighting effect application criteria). Display the capture user interface for the effect. (Eg, preview of 3D lighting effects).

In some embodiments, at block 1724, displaying the capture user interface for the first lighting effect includes: representation of the image data; And simultaneously displaying an alignment guide (eg, 1640) (eg, a circle, an ellipse, a hexagon) displayed in the first area within the representation of the image data, wherein the first criteria represent the representation of the image data such that the first criteria are met. And a requirement that the representation of the face of the subject displayed at is within the alignment guide (eg, 1640). In some embodiments, the alignment guide (eg, 1640) is overlaid on the representation of the image data. In some embodiments, users will align portions of their bodies (eg, faces) within the alignment guide (eg, 1640). In some embodiments, the subject's face is substantially aligned with the perimeter of the alignment guide (eg, 1640). In some embodiments, the alignment guide (eg 1640) is not displayed during some modes (eg portrait mode, studio light). In some embodiments, the alignment guide (eg, 1640) is located at the center of the live preview. In some embodiments, the alignment guide (eg, 1640) is located off center. In some embodiments, when multiple objects are detected, multiple alignment guides are displayed in the live preview. Displaying an alignment guide (eg, 1640) in the representation of the image data provides the user with visual feedback as to where to position the subject's face to capture an image with the desired optical effect. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life.

In some embodiments, at block 1726, the first criteria are applied to the subject's face such that the first criteria are met (eg, when the user's face is detected within the alignment guide (eg, 1640) in the live preview). The representation includes a requirement to be detected in a first area within the representation of the image data.

In some embodiments, at block 1728, applying the first change in appearance to the image data is less within the representation of the image data compared to the appearance of the representation of the image data displayed in the first area within the representation of the image data. Altering the appearance of the representation of the image data displayed in the second area (eg, outside of the alignment guide (eg, 1640)), the second area being separate (eg, non-overlapping, separate) from the first area. . In some embodiments, changing the appearance of the representation of the image data displayed in the second region relative to the representation of the image data displayed in the first region includes applying a filter (eg, an alignment filter) to the second region. do. In some embodiments, the sort filter is not applied to the first region. In some embodiments, the alignment filter is uniformly applied to the outer region of the alignment guide (eg, 1640). In some embodiments, the alignment filter changes gradually in intensity (eg, darker / lighter closer to the edge of the live preview compared to how the alignment filter is applied near the edge of the alignment guide (eg, 1640)). do). In some embodiments, the alignment filter is a single optical effect (eg bokeh, saturation, color). In some embodiments, the alignment filter includes a number of optical effects (eg, bokeh, saturation, color) (eg, a combination thereof). In some embodiments, each of the plurality of optical effects is applied differently to different portions of the live preview.

In some embodiments, at blocks 1732 to 1742, the electronic devices (eg, 1600) determine that the first criteria are not met; In response to determining that the first criteria are not met (eg, no object is detected in the field of view within a predetermined distance from the electronic device (eg, 1600); the face is not aligned in the alignment guide (eg, 1640)): Stop applying the first change in appearance to the representation of the image data; Display on the display (eg, 1604) a representation of the image data (eg, 1606) (eg, an unchanged image with no filter applied or a partial filter applied) without a first change in appearance applied; On a display (eg, 1604), display a graphical representation of unmet lighting condition application criteria. In some embodiments, the device displays instructions for placing (eg, moving closer, moving further) the face within the alignment guide (eg, 1640). In some embodiments, when the conditions are met again, the filter is reapplied.

At block 1710, while displaying a representation of the image data on a display (eg, 1604), the electronic device (eg, 1600) further performs the techniques of blocks 1744-1750.

At block 1744, after detecting the first input, the electronic device (eg, 1600) receives a second input (eg, 1636) (eg, corresponding to the request to capture image data corresponding to the field of view of the one or more cameras). Swipe, tap and hold, tap, press a button; a gesture may be on top of an icon representing a shutter button).

In block 1746, the respective lighting effects selected based on the first input may include the natural light, studio light, contour light, stage light, as described in more detail in the first lighting effect (eg, in paragraph [0499] below). In accordance with a determination that at least one of the stage light mono), at block 1748, the electronic device (eg, 1600) corresponds to the field of view of the one or more cameras and associates the first lighting effect with the representation of the image data (eg, 1606). Associating (e.g., applying the first lighting effect to the image data, preparing to apply the first lighting effect to the image data, or capturing the captured image data, by default when viewing of the image data is requested Captures image data, which is marked as image data to be applied, and the first lighting effect is based on depth map information (and, for example, based on measurements of the depth sensor or Based on a mapping inconsistencies between the two images taken from different positions at the same time). Associating the first lighting effect with the representation of the image data (eg, 1606) based on the depth map information provides the user with additional visual feedback on the depth map information. For example, the first lighting effect includes one or more light sources at different locations, intensities, or types (directivity, perimeter, point) that provide the user with feedback about a particular orientation of objects corresponding to depth map information. can do. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life.

In block 1750, a second lighting effect, wherein the respective lighting effects selected based on the first input are different from the first lighting effect (eg, natural light, studio light, as described in more detail in paragraph [0499] below). In accordance with the determination of at least one of contour light, stage light, and stage light mono), the electronic device (eg, 1600) corresponds to the field of view of one or more cameras and provides a second lighting effect with a representation of the image data (eg, 1606). Associating (eg, applying the second lighting effect to the image data, preparing to apply the second lighting effect to the image data, or capturing the captured image data, by default, when the viewing of the image data is requested, the second lighting effect. Captures the image data (marked as image data to be applied), and the second lighting effect is based on the depth map information. (Eg, based on measurements of the depth sensor or based on disparity mapping between two images taken simultaneously from different locations). Applying the second lighting effect to the representation of the image data (eg, 1606) based on the depth map information provides the user with additional visual feedback on the depth map information. For example, the second lighting effect includes one or more light sources at different locations, intensities, or types (directivity, perimeter, point) that provide the user with feedback about a particular orientation of the objects corresponding to the depth map information. can do. Providing an improved visual feedback to the user is such that the user achieves the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By improving the operability of the device and making the user-device interface more efficient, by helping to reduce user errors), which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and battery Improve life.

At block 1552, after capturing image data corresponding to the field of view of the one or more cameras, the electronic device (eg, 1600) selects a second input (eg, selection of a camera roll affordance (eg, 1634) or a picture viewer application. Activation) to receive a request to display captured image data.

At blocks 1754-1758, in response to receiving a request to display the captured image data in response to the second input (eg, in response to the shutter affordance being activated), the image while the first lighting effect is selected. In accordance with the determination that the data has been captured, the electronic device (eg, 1600) displays a representation of the image data (eg, 1606) having the first lighting effect applied (eg, displays a representation of the image data to which the stage light filter is applied and ); In accordance with a determination that image data was captured while a second lighting effect (eg, not stage light) was selected, the electronic device (eg, 1600) displays a representation of the image data (eg, 1606) to which the second lighting effect has been applied. .

In some embodiments, displaying the capture user interface for the first lighting effect is such that the electronic device (eg, 1600) applies a placeholder filter to the representation of the image data displayed in the digital viewfinder (eg, 1610). (Eg, dimming or desaturating the background), the placeholder filter is based on the first lighting effect and applied regardless of whether the first criteria are met. Applying a placeholder filter makes a smoother and more intuitive (less confusing) transition to the light filter. Applying the placeholder filter irrespective of whether the first criteria are met results in viewfinder content such as which portions of the image correspond to portions of the depth map in the background (eg, 1609) relative to the foreground (eg, 1608). Provide visual feedback to the depth map corresponding to the user. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, in response to the first input, the electronic device (eg, 1600) may place a placeholder filter (eg, dimming or desaturating the background) of the live preview without applying the first change in appearance to the live preview. ); In response to detecting that the first criteria have been met, the electronic device (eg, 1600) applies the first change in appearance to the live preview while continuing to apply the placeholder filter to the live preview (eg, the placeholder filter Part of the first lighting effect without taking into account the depth map information and thus displayed whether or not the first criteria have been met). Applying the placeholder filter irrespective of whether the first criteria are met is a viewfinder such as, for example, which portions of the image correspond to portions of the depth map in the background (eg, 1609) relative to the foreground (eg, 1608). Provide the user with visual feedback on the depth map corresponding to the content. Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve.

In some embodiments, while displaying a representation of the image data, the electronic device (eg, 1600) displays a visual indication (eg, 1638) on the display (eg, 1604) that the image data includes depth map information. do. (Eg, a “portrait mode” badge is optionally displayed to indicate the availability of depth map information.

In some embodiments, the electronic device (eg, 1600) applying the first lighting effect or the second lighting effect is a depth map associated with the image data in the representation of the image data displayed in the digital viewfinder (eg, 1610). Applying a simulation of one or more point light sources in space based on the information. Lighting options include natural light, studio light, contour light, stage light, and stage light mono. Different lighting effects model (eg, simulate) the result of one or more of the point light sources in space based on the depth map of the image data. The natural lighting option does not apply any composite lighting to the image (eg, the original image is displayed or a portion of the original image is displayed, and blur is applied to different parts of the original effect to simulate the bokeh effect). . Studio lighting effects include the modeling of a number of individual point light sources located around an object (eg, produce brightly filled light effects). The contour lighting effect includes the modeling of a number of distinct point light sources located at several points around the subject to create shadows on the subject's face (eg, create a slimming effect, on the side of the subject's face and / or Or create shadows on the subject's chin). Stage light illumination effects include modeling (eg, generating spotlight effects) of a single individual point light source positioned over an object. Stage light mono illumination effects include modeling in black and white (eg, generating spotlight effects in black and white) of a single individual point light source positioned around the subject. In some embodiments, the illumination filter simulates point sources. In some embodiments, the lighting effect is snapped in when the first criteria are met, as described in more detail above. In some embodiments, when the system detects a face, facial features are taken into account when applying the lighting effect. As a result, the lighting effect changes the appearance of the representation of the image data based on the specific facial features and face shape of the object. Applying a simulation of the point light source provides the user with a visual representation of the content of the depth map information and allows the device to provide the user with visual feedback on the shape and depth positioning of the objects within the field of view of the camera (s). . Providing improved visual feedback helps the user achieve the intended result by providing feedback indicative of input that will cause the device to produce the intended result (eg, when operating the device and / or interacting with the device). By reducing errors) to improve the device's operability and make the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and reducing battery life. Improve. In the representation of the image data displayed in the viewfinder (eg, 1610), applying a simulation of one or more point light sources creates a lighting effect that simulates the use of external lighting equipment without requiring external lighting equipment. Creating such lighting effects without requiring external lighting equipment reduces the size and cost of the equipment needed to produce an image with specific lighting effects.

In some embodiments, while the first lighting effect is applied, the electronic device (eg, 1600) maintains at least one value of the previously applied visual effect. (E.g., bokeh, illumination) Thus, it is possible to achieve a light effect and a bokeh effect in one representation of the image data.

In some embodiments, the previously applied visual effect is a color filter.

In some embodiments, the second input is an input received while the first lighting effect is applied to the representation of the image data (eg, 1606), and applying the second lighting effect is the first lighting effect and the second lighting effect. Gradual transition between the application of. In some embodiments, the gradual transition includes first 100% at first time, second 0% at first time, 90% first at second time, second 10%, and the like. Gradual switching between lighting effects reduces user distraction generated by blinking filters on / off, concentrating the user on taking the desired picture, thereby reducing the number of inputs required to capture the desired picture. Reduce and reduce memory requirements for storage of photos. Reducing the number of inputs and reducing memory requirements needed to capture the desired image can be achieved by, for example, helping the user achieve the intended result and reducing user mistakes when operating and / or interacting with the device. Improves the operability of the device and makes the user-device interface more efficient, which in addition allows the user to use the device more quickly and efficiently, thereby reducing the device's power usage and improving battery life.

Note that the details of the processes described above with respect to the method 1700 (eg, FIGS. 17A-17G) are also applicable in a similar manner to the methods described above. For example, the methods 700, 900, 1100, 1300, and 1500 optionally include one or more of the features of the various methods described above and referring to the method 1700. For example, elements of the filter user interface, affordances and controls can be combined among the various methods. As another example, the viewfinder of the method 1700 is similar to the viewfinder of the methods 700, 900, 1100, 1300, 1500. For another example, a preview of the first filter described in method 700 may be displayed on a display (eg, 1604). For another example, as described in the method 900, the result of applying the first lighting effect to the representation of the image data may be displayed on a display (eg, 1604). For another example, as described in the method 1100, the result of applying the first image filter to the foreground area of the image data may be displayed on a display (eg, 1604). For another example, as described in the method 1300, displaying a filter selection interface that includes representations of a plurality of filters in a set of filters may also be displayed on a display (eg, 1604). For another example, as described in method 1500, displaying the alignment guide may also be displayed on a display (eg, 1604). For brevity, these details are not repeated.

Operations in the information processing methods described above may optionally include one or more functional modules within an information processing apparatus, such as general purpose processors or application specific chips (eg, as described in connection with FIGS. 1A, 3, and 5A). Implemented by driving. In addition, the operations described above with reference to FIGS. 17A-17G are optionally implemented by the components shown in FIGS. 1A-1B. For example, display operation 1702, detection operation 1712, display operation 1720, display operation 1722, display operation 1724, and apply operation 1728, optionally, event classifier 170. , Event recognizer 180, and event handler 190. Event monitor 171 in event classifier 170 detects a contact on touch-sensitive display 604, and event dispatcher module 174 delivers the event information to application 136-1. Each event recognizer 180 of the application 136-1 compares the event information with the respective event definitions 186, and the first contact at a first location on the touch-sensitive surface is selected by the object on the user interface. Determine whether to correspond to a predefined event or a sub-event, such as. When an individual predefined event or subevent is detected, event recognizer 180 activates an event handler 190 associated with the detection of the event or subevent. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update application internal state 192. In some embodiments, event handler 190 accesses individual GUI updater 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes can be implemented based on the components shown in FIGS. 1A and 1B.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. Embodiments have been selected and described in order to best explain the principles of the techniques and their practical application. Thus, those skilled in the art will be able to make the best use of the techniques and various embodiments using various modifications as appropriate for the particular use contemplated.

While the invention and examples have been described fully with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosure and examples as defined by the claims.

As mentioned above, one aspect of the present technology is the collection and use of data available from various sources to enhance the delivery of invited content or any other content that users may be interested in. The present disclosure contemplates that in some cases, such collected data may include personal information data that can be used to uniquely identify, contact, or locate a particular individual. Such personal information data may include demographic data, location based data, telephone numbers, email addresses, home addresses, or any other identifying information.

The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data may be used to deliver targeted content of greater interest to a user. Thus, the use of such personal information data enables the calculated control of the delivered content. In addition, other uses for personal information data that benefit the user are also contemplated by this disclosure.

The present disclosure further contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with well-established privacy policies and / or privacy practices. In particular, such entities must implement and continue to use privacy policies and practices that are generally perceived to meet or exceed industrial or administrative requirements for keeping personal information data private and secure. For example, personal information from users should be collected for legitimate and proper use of the entity and should not be shared or sold outside these legitimate uses. In addition, such collection should only occur after receipt of informed consent of the users. In addition, such entities will take any necessary steps to protect and secure access to such personal data and to ensure that others having access to the personal data are adhering to their privacy policies and procedures. . In addition, such entities may themselves be evaluated by third parties to prove their adherence to widely recognized privacy policies and practices.

Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of advertising delivery services, the technology allows users to select "opt in" or "opt out" of their participation in the collection of personal information data during registration for the service. It can be configured to allow. In another example, users can choose not to provide location information for target content delivery services. In another example, users may choose not to provide accurate location information but allow delivery of location zone information.

Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without the need to access such personal information data. do. In other words, various embodiments of the present technology are not rendered inoperable due to the lack of all or part of such personal information data. For example, the content may include a minimum amount of personal information or non-personal information data, such as content requested by a device associated with a user, other non-personal information available for content delivery services, or publicly available information. Can be selected and communicated to users by inferring a preference based on that.

Claims (141)

As a method,
In an electronic device having one or more cameras, one or more input devices, and a display,
Simultaneously displaying on the display a camera application user interface, wherein the camera application user interface comprises:
A digital viewfinder including a live preview of the field of view of one or more cameras; And
A representation of a filter picker user interface overlaid on the digital viewfinder;
While simultaneously displaying a representation of the digital viewfinder and the filter picker user interface, detecting, via the one or more input devices, a first input starting at a location corresponding to a respective portion of the live preview; And
In response to detecting the first input starting at a position corresponding to the respective portion of the live preview,
In accordance with a determination that one or more sets of conditions are met—a first condition of the set of one or more conditions is met when the first input is detected—a filter picker user interface is placed on the respective portion of the live preview. While overlaying, applying a preview of a first filter to the live preview of the field of view of the camera that was not applied before the first input was detected; And
In the camera application without applying the preview of the first filter to the live preview in accordance with a determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. Performing the respective operation. The method of claim 1, wherein
Detecting a second input starting at a location corresponding to the filter picker user interface while the filter picker user interface is not displayed and overlayed on the respective portion of the live preview; And
In response to detecting the second input, expanding the filter picker user interface to overlay the respective portion of the live preview. The method of claim 1 or 2, wherein the filter picker user interface comprises a representation of the filter picker user interface and representations of a plurality of filters. 3. The method of claim 2, wherein extending the filter picker user interface to overlay respective portions of the live preview comprises previewing a filter on the live preview of the field of view of the camera that was not applied before the second input was detected. A method that occurs without applying. The method of claim 2, wherein the representation of the filter picker user interface is a representation of a container object having a plurality of faces surrounding an interior volume that includes a representation of the first three-dimensional object when in the first state. . The method of claim 5, wherein while the representation of the filter picker user interface is in a second state different from the first state, the representation of the container object is changed to remove shading and / or lighting effects. 6. The method of claim 5, wherein while the representation of the filter picker user interface is in the first state, one or more of the faces or interior volume of the plurality of faces exhibits a visual appearance based on a currently selected filter represented by the plurality of filters. Having, method.
6. The method of claim 5, wherein while the representation of the filter picker user interface is in a second state different from the first state, the visual appearance of the representation of the filter picker user interface is one of the plurality of filters displayed in the interior volume. Not based on the currently selected filter. The method according to any one of claims 1 to 8,
While the filter picker user interface is displayed and not overlaid on respective portions of the live preview, the representation of the filter picker user interface and the representations of the plurality of filters are located along a line substantially parallel to the edge of the display. How. The method of claim 3,
While the filter picker user interface is overlaid on respective portions of the live preview, the representation of the filter picker user interface and the representations of the plurality of filters are positioned along a curve. 4. The method of claim 3, wherein the first input is a tap gesture, and a position corresponding to the respective portion of the live preview corresponds to a representation of the first filter in representations of the plurality of filters. That's how. The method of claim 3,
The representation of the filter picker user interface includes a representation of the first filter, a representation of a second filter, and a representation of a third filter;
Wherein the value of the visual characteristic of the representation of the first filter is different from the value of the visual characteristic of the representation of the second filter and the value of the visual characteristic of the representation of the third filter. The method of claim 12,
The representation of the second filter is in a first direction from the representation of the first filter;
The representation of the third filter is in the first direction from the representation of the second filter;
And the value of the visual characteristic varies progressively in a first direction from the representation of the first filter to the representation of the third filter via the representation of the second filter. 14. The method of claim 13, wherein when the filter picker user interface is overlaid on the respective portion of the live preview, the filter picker user interface is further associated with the first filter displayed in relation to the representation of the first filter. Including information. 15. The method of any one of claims 1 to 14, wherein the first input is a tap gesture, and wherein performing the respective action comprises: the camera entering the respective portion of the live preview where the input is detected. Selecting a focus for media capture at an object located in the field of view. The method of claim 1, wherein the first input is a swipe gesture, and wherein performing the respective action comprises changing a camera capture mode of the electronic device. , Way. The method according to any one of claims 1 to 16,
While the filter picker user interface is displayed and overlaid on the respective portion of the live preview, and while the first filter is being applied to the live preview of the field of view of the camera, corresponding to the filter picker user interface. Detecting a third input starting at a location; And
In response to detecting the third input,
Moving a representation of a second filter in the representations of the plurality of filters to a location on the display corresponding to the currently selected filter; And
Applying a preview of the second filter to the live preview of the field of view. The method of claim 17, wherein the representation of the filter picker user interface is a representation of a second three-dimensional object having a plurality of faces, wherein the method comprises
While the representation of the filter picker user interface is associated with a sixth representation of representations of the plurality of filters, and while the representation of the filter picker user interface presents a first side of the plurality of faces, the filter picker user interface Detecting a sixth input starting at a position corresponding to; And
In response to detecting the sixth input,
Rotating the container object to present a second face of the plurality of faces that is not displayed prior to detecting the sixth input;
Switching from the first filter that is the currently selected filter to a second filter that is different from the first filter that is the currently selected filter; And
Applying a preview of the second filter to the live preview of the field of view. The method of claim 17, wherein applying a preview of the second filter,
Gradually switching between applying a preview of the first filter and applying a preview of the second filter. The method of claim 17, further providing a tactile output in response to detecting the third input. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more cameras, one or more input devices, and one or more processors of an electronic device having a display, wherein the one or more programs are subject to claim 1. A non-transitory computer readable storage medium comprising instructions for performing the method of claim 20. As an electronic device,
One or more cameras;
One or more input devices;
display;
One or more processors; And
21. An electronic device comprising a memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 1-20. As an electronic device,
One or more cameras;
One or more input devices;
display; And
An electronic device comprising means for performing the method of any one of claims 1 to 20. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more cameras, one or more input devices, and one or more processors of an electronic device having a display, wherein the one or more programs are:
Instructions for simultaneously displaying a camera application user interface on the display, wherein the camera application user interface comprises:
A digital viewfinder comprising a live preview of the field of view of one or more cameras; And
A representation of a filter picker user interface overlaid on the digital viewfinder;
Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via the one or more input devices while simultaneously displaying a representation of the digital viewfinder and the filter picker user interface; And
In response to detecting the first input starting at a position corresponding to the respective portion of the live preview,
In accordance with a determination that one or more sets of conditions are met—a first condition of the set of one or more conditions is met when the first input is detected—a filter picker user interface is placed on the respective portion of the live preview. While overlaying instructions for applying a preview of a first filter to the live preview of the field of view of the camera that was not applied before the first input was detected; And
In the camera application without applying the preview of the first filter to the live preview in accordance with a determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. A non-transitory computer readable storage medium comprising instructions for performing a respective operation. As an electronic device,
One or more cameras;
One or more input devices;
display;
One or more processors; And
A memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs are:
Instructions for simultaneously displaying a camera application user interface on the display, wherein the camera application user interface comprises:
A digital viewfinder comprising a live preview of the field of view of one or more cameras; And
A representation of a filter picker user interface overlaid on the digital viewfinder;
Instructions for detecting a first input starting at a location corresponding to a respective portion of the live preview via the one or more input devices while simultaneously displaying a representation of the digital viewfinder and the filter picker user interface; And
In response to detecting the first input starting at a position corresponding to the respective portion of the live preview,
In accordance with a determination that one or more sets of conditions are met—a first condition of the set of one or more conditions is met when the first input is detected—a filter picker user interface is placed on the respective portion of the live preview. While overlaying instructions for applying a preview of a first filter to the live preview of the field of view of the camera that was not applied before the first input was detected; And
In the camera application without applying the preview of the first filter to the live preview in accordance with a determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. And instructions for performing a respective operation. As an electronic device,
One or more cameras;
One or more input devices;
display;
Means for simultaneously displaying a camera application user interface on the display, wherein the camera application user interface comprises:
A digital viewfinder comprising a live preview of the field of view of one or more cameras; And
A representation of a filter picker user interface overlaid on the digital viewfinder;
Means for detecting a first input starting at a location corresponding to a respective portion of the live preview via the one or more input devices while simultaneously displaying a representation of the digital viewfinder and the filter picker user interface; And
In response to detecting the first input starting at a position corresponding to the respective portion of the live preview,
In accordance with a determination that one or more sets of conditions are met—a first condition of the set of one or more conditions is met when the first input is detected—a filter picker user interface is placed on the respective portion of the live preview. Means for applying a preview of a first filter to the live preview of the field of view of the camera that was not applied before the first input was detected while being overlaid; And
In the camera application without applying the preview of the first filter to the live preview in accordance with a determination that the filter picker user interface is not overlaid on the respective portion of the live preview when the first input is detected. Means for performing a respective operation. As a method,
In an electronic device having one or more input devices and a display,
Displaying, on the display, a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Detecting a first input via the one or more input devices;
In response to detecting the first input, applying a first lighting effect to the representation of the image data, the first lighting effect based on the depth map information;
Detecting a second input via the one or more input devices; And
In response to detecting the second input, applying a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on the depth map information. . The method of claim 27, wherein the electronic device further comprises one or more cameras, wherein the representation of the image data is a live preview of image data captured within the field of view of the one or more cameras displayed in a digital viewfinder. . The method of claim 28, wherein the first input is an input received while first criteria are met, the first criteria comprising a requirement that an object is detected in the field of view within a predetermined distance from the electronic device. The method of claim 29, wherein applying the first lighting effect,
And applying a placeholder filter to the representation of the image data displayed in the viewfinder, wherein the placeholder filter is based on the first lighting effect and applied regardless of whether the first criteria are met. How. The method of claim 30, wherein while the first lighting effect is being applied,
Determining that the first criteria are not met; And
In response to determining that the first criteria are not met,
Stopping applying the first lighting effect to the representation of the image data;
Displaying on the display a representation of the image data to which no first lighting effect has been applied; And
And displaying on the display a graphical representation of the unmet first criteria. The method of claim 29,
In response to the first input, applying a placeholder filter to the live preview without applying the first lighting effect to the live preview; And
In response to detecting that the first criteria have been met, applying the first lighting effect to the live preview while continuing to apply the placeholder filter to the live preview. The method of claim 28,
The first input is an input received while the first criteria are not met, the first criteria including a requirement that an object is detected in the field of view within a predetermined distance from the electronic device;
The method,
After displaying the live preview without applying the first lighting effect to the live preview, detecting that the first criteria have been met; And
In response to detecting that the first criteria have been met, applying the first lighting effect to the live preview. The method of claim 27, wherein the representation of the image data is previously captured image data. The method according to any one of claims 27 to 34,
Prior to displaying the representation of the image data, the device further comprising receiving the image data and the depth map information associated with the representation of the image data. 36. The method of claim 35 wherein
While displaying the representation of the image data, further comprising displaying on the display a visual indication that the image data includes depth map information. 37. The method of any one of claims 27-36, wherein the depth map information associated with the image data includes information corresponding to at least three different depth levels. 38. The apparatus of any one of claims 27 to 37, wherein the depth map information associated with the image data includes information identifying depth contours of an object in the representation of the image data;
The lighting effects change the appearance of the representation of the image data based on the position and curvature of the contours of the object. 39. The method of any of claims 27-38, wherein applying the first lighting effect or the second lighting effect comprises:
Applying to a representation of the image data displayed in the digital viewfinder, a simulation of one or more point light sources in space based on the depth map information associated with the image data. The method according to any one of claims 27 to 39,
While the first lighting effect is applied, further comprising maintaining at least one value of a previously applied visual effect. 41. The method of claim 40, wherein the previously applied visual effect is a color filter. 41. The method of any of claims 27-40, wherein the second input is an input received while the first lighting effect is being applied to the representation of the image data, and wherein applying the second lighting effect is ,
Gradually switching between application of the first lighting effect and the second lighting effect. 43. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display, wherein the one or more programs are any one of claims 27-42. A non-transitory computer readable storage medium comprising instructions for performing the method of claim 1. As an electronic device,
One or more input devices;
display;
One or more processors; And
43. An electronic device comprising memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 27-42. As an electronic device,
One or more input devices;
display; And
43. An electronic device comprising means for performing the method of any one of claims 27-42. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display, wherein the one or more programs are:
Instructions for displaying, on the display, a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Instructions for detecting a first input via the one or more input devices;
Instructions for applying a first lighting effect to the representation of the image data in accordance with detecting the first input, the first lighting effect based on the depth map information;
Instructions for detecting a second input via the one or more input devices; And
In response to detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on the depth map information; Non-transitory computer readable storage medium. As an electronic device,
One or more input devices;
display;
One or more processors; And
A memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs are:
Instructions for displaying, on the display, a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Instructions for detecting a first input via the one or more input devices;
Instructions for applying a first lighting effect to the representation of the image data in accordance with detecting the first input, the first lighting effect based on the depth map information;
Instructions for detecting a second input via the one or more input devices; And
In response to detecting the second input, instructions for applying a second lighting effect different from the first lighting effect to the representation of the image data, wherein the second lighting effect is based on the depth map information; Electronic device. As an electronic device,
One or more cameras;
One or more input devices;
display;
Means for displaying on the display a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Means for detecting a first input via the one or more input devices;
Means for applying a first lighting effect to the representation of the image data in accordance with detecting the first input, the first lighting effect based on the depth map information;
Means for detecting a second input via the one or more input devices; And
Means for applying a second lighting effect, different from the first lighting effect, to the representation of the image data in accordance with detecting the second input, wherein the second lighting effect is based on the depth map information; Electronic device. As a method,
In an electronic device having one or more input devices and a display,
Detecting, via the one or more input devices, a first input corresponding to the selection of a first image filter of the representation of image data having a first appearance; And
In response to detecting the first input, applying the first image filter to a representation of the image data, wherein applying:
In accordance with a determination that the image data has depth information associated therewith, wherein the depth information allows a foreground region of the representation of the image data to be distinguished from a background region of the representation of the image data;
Applying the first image filter to the foreground region of the representation of the image data using a first level adjustment to change the appearance of the foreground region of the representation of the image data, wherein the first level adjustment comprises One image filter indicates a first degree of changing the appearance of the representation of the image data;
Applying the first image filter to the background region of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data, wherein the second level adjustment is performed by the first image filter. One image filter indicates a second degree of changing an appearance of the representation of the image data, wherein the first level adjustment and the second level adjustment are different; And
After applying the first image filter to the representation of the image data, displaying on the display a representation of the respective image to which the first filter is applied to the representation of the image data. The method of claim 49, wherein the electronic device further comprises one or more cameras, wherein the representation of the image data is a live preview of image data captured within the field of view of the one or more cameras. 51. The method of claim 49 or 50, wherein the first input is detected while displaying a filter selection user interface. The method of claim 51,
On the display,
Representation of the image data; And
And simultaneously displaying the filter selection interface. The method of claim 51, wherein the filter selection user interface comprises a plurality of filter representations comprising a representation of the first image filter, and wherein the first input corresponds to the selection of the representation of the first image filter. The method of any one of claims 49-53,
Prior to detecting the first input, receiving, at the electronic device, image data represented by the representation of the image data. 55. The method of any one of claims 49-54, wherein in response to detecting the first input, applying the first image filter to the representation of the image data:
In accordance with the determination that the image data does not have depth information associated therewith, further comprising applying the first image filter uniformly to the representation of the image data using the first level adjustment. 55. The method of any of claims 49-54, wherein applying the first image filter to the representation of the image data:
And in accordance with the determination that the image data includes a third color value, shifting the third color value using a third level of color value adjustment. 57. The method of any of claims 49-56, wherein applying the first image filter to the foreground area comprises:
The background using a color value adjustment of a first level according to a determination that the image data corresponding to the foreground region includes a first color value and the image data corresponding to the background includes the first color value. Shifting the first color value of the region and shifting the first color value of the foreground region using a color value adjustment of a second level that is different from the color value adjustment of the first level, Way. 58. The method of any of claims 49-57, wherein while maintaining the first image filter, at least one value of a previously applied visual effect is maintained. The method of claim 49,
And displaying a camera application user interface on the display, wherein the camera application user interface comprises:
A digital viewfinder comprising a live preview of the field of view of one or more cameras; And
A representation of a color wheel user interface at a first location overlaid on the digital viewfinder. The method of claim 59,
In response to detecting a second user input corresponding to the representation of the color wheel user interface,
Stopping displaying the representation of the color wheel user interface; And
Displaying a color wheel user interface on the display, the color wheel user interface displaying a plurality of representations of color filters. The method of claim 60,
In response to detecting a third user input corresponding to an area corresponding to at least one of the representations of a color filter displayed in the color wheel user interface,
Applying the corresponding color filter to the image data to modify the color appearance of the representation of the image data. 62. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display, wherein the one or more programs are any one of claims 49-61. A non-transitory computer readable storage medium comprising instructions for performing the method of claim 1. As an electronic device,
One or more input devices;
display;
One or more processors; And
62. A device comprising memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 49-61. As an electronic device,
One or more input devices;
display; And
62. An electronic device comprising means for performing the method of any one of claims 49-61. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display, wherein the one or more programs are:
Instructions for detecting, via the one or more input devices, a first input corresponding to selection of a first image filter in the representation of image data having a first appearance; And
In response to detecting the first input, instructions for applying the first image filter to a representation of the image data, wherein the instructions for applying are:
In accordance with a determination that the image data has depth information associated therewith, wherein the depth information allows a foreground region of the representation of the image data to be distinguished from a background region of the representation of the image data;
Instructions for applying the first image filter to the foreground region of the representation of the image data using a first level adjustment to change the appearance of the foreground region of the representation of the image data; Indicate a first degree by which the first image filter changes the appearance of the representation of the image data;
Instructions for applying the first image filter to the background region of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data—the second level adjustment is Indicate a second degree at which the first image filter changes the appearance of the representation of the image data, wherein the first level adjustment and the second level adjustment are different; And
And after applying the first image filter to the representation of the image data, instructions on the display to display a representation of the respective image to which the first filter is applied to the representation of the image data. Readable storage medium. As an electronic device,
One or more input devices;
display;
One or more processors; And
A memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs are:
Instructions for detecting, via the one or more input devices, a first input corresponding to selection of a first image filter in the representation of image data having a first appearance; And
In response to detecting the first input, instructions for applying the first image filter to a representation of the image data, wherein the instructions for applying are:
In accordance with a determination that the image data has depth information associated therewith, wherein the depth information allows a foreground region of the representation of the image data to be distinguished from a background region of the representation of the image data;
Instructions for applying the first image filter to the foreground region of the representation of the image data using a first level adjustment to change the appearance of the foreground region of the representation of the image data; Indicate a first degree by which the first image filter changes the appearance of the representation of the image data;
Instructions for applying the first image filter to the background region of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data—the second level adjustment is Indicate a second degree at which the first image filter changes the appearance of the representation of the image data, wherein the first level adjustment and the second level adjustment are different; And
And after applying the first image filter to the representation of the image data, displaying, on the display, a representation of the respective image to which the first filter is applied to the representation of the image data. As an electronic device,
One or more input devices;
display;
Means for detecting, via the one or more input devices, a first input corresponding to the selection of a first image filter in the representation of the image data having a first appearance; And
In response to detecting the first input, means for applying the first image filter to a representation of the image data, wherein the means for applying comprises:
In accordance with a determination that the image data has depth information associated therewith, wherein the depth information allows a foreground region of the representation of the image data to be distinguished from a background region of the representation of the image data;
Means for applying the first image filter to the foreground region of the representation of the image data using a first level adjustment to change the appearance of the foreground region of the representation of the image data, the first level adjustment being the Indicate a first degree by which a first image filter alters the appearance of the representation of the image data;
Means for applying the first image filter to the background area of the representation of the image data using a second level adjustment to change the appearance of the background region of the representation of the image data, the second level adjustment being the A second degree at which a first image filter changes the appearance of the representation of the image data, wherein the first level adjustment and the second level adjustment are different; And
Means for displaying on the display a representation of the respective image to which the first filter is applied to the representation of the image data after applying the first image filter to the representation of the image data. As a method,
In an electronic device having one or more input devices and a display,
Displaying, on the display, a filter selection interface comprising representations of a plurality of filters in a set of filters;
While simultaneously displaying a representation of image data and the filter selection interface on the display, a location corresponding to the filter selection interface while the first filter of the set of filters meets selection criteria, through the one or more input devices Detecting a first input at;
In response to detecting the first input,
Stopping displaying the first subset of representations of filters in the set of filters, wherein the first subset of representations of the filters in the set of filters is derived from a representation of the first filter in the filter selection user interface. One or more filters in one direction and one or more filters in a second direction from the representation of the first filter; And
Maintaining a display of a second subset of representations of filters in the set of filters, wherein the second subset of representations of filters includes at least a representation of the first filter. 69. The method of claim 68, wherein the first filter satisfies the selection criteria when the first input is detected at a location corresponding to the first filter. 69. The method of claim 68, wherein the first filter satisfies the selection criteria when the representation of the first filter is at a selection position when the first input is detected. The method of any one of claims 68 to 70,
Prior to displaying the representation of the image data, further comprising receiving image data at the electronic device, the image data corresponding to the representation of the image data. 71. The apparatus of any of claims 68-70, wherein the electronic device comprises a touch-sensitive display, the first input corresponding to a contact on the touch-sensitive display, and wherein the contact is greater than a first intensity threshold. Having a large characteristic strength, the method. 73. The method of claim 72, wherein the electronic device comprises a touch-sensitive display, the first input corresponding to a first contact on the touch-sensitive display, and the method further comprising:
While continuing to detect the first contact on the display after detecting the first input,
Detecting a movement of the first contact; And
Displaying, on the display, a representation of the image data having an applied visual effect corresponding to the second filter. 74. The method of claim 73, further in response to detecting the movement of the contact,
Providing a tactile output indicating that the filter applied to the image data has changed. The method of claim 73, wherein the method is
Detecting a movement of a second contact on the touch-sensitive display while a third filter is in the second selected position and while displaying the first subset of representations prior to detecting the first input. ; And
In response to detecting the movement of the second contact,
Stopping displaying the representation of a third filter at the second selection position;
Displaying a representation of a fourth filter at the second selection position;
Applying a visual effect corresponding to the fourth filter to the image data;
Displaying on the display a representation of the image data having the applied visual effect;
And the first contact is the second contact. The method of claim 68,
The electronic device includes a touch sensitive display,
The first input corresponds to an increase in the characteristic intensity of a contact on the touch-sensitive display;
The method further comprises, in response to detecting the increase in the characteristic intensity of the contact, dynamically shifting a position of the first filter representation toward a focal point on a display. 77. The method of claim 76,
Detecting a decrease in the characteristic intensity of the contact while continuing to detect the contact on the touch-sensitive display; And
In response to detecting the decrease in the characteristic strength of the contact,
Prior to detecting a decrease in the characteristic strength of the contact, determining that the characteristic strength of the contact has reached a respective intensity threshold, without displaying the first subset of representations of the filters. Maintaining a display of the second subset; And
According to the determination that the characteristic intensity of the contact has not reached the respective intensity threshold prior to detecting the decrease in the characteristic intensity of the contact, on the display, the position of the first filter representation is dynamically moved away from the focus. Shifting to. 78. The method of any of claims 68-77, wherein the electronic device comprises a touch-sensitive display, the first input corresponding to a contact on the touch-sensitive display, and the method further comprising:
Detecting a liftoff of the contact while maintaining display of the second subset of representations of the filters without displaying the first subset of representations of the filters; And
In response to detecting the liftoff of the contact, restoring a display of the first subset of representations of the filters. 79. The method of any of claims 68 to 78, wherein the electronic device comprises a touch sensitive display,
Detecting an input comprising an increase in the intensity of the contact while maintaining display of the second subset of representations of the filters without displaying the first subset of representations of the filters; And
In response to detecting the input,
In accordance with a determination that the input included an increase from a characteristic intensity of less than a fourth intensity threshold to a characteristic intensity of more than the fourth intensity threshold contact with a characteristic intensity above the fourth intensity threshold in the characteristic intensity of a contact. Restoring the display of the first subset of representations; And
In accordance with a determination that the input did not include an increase from a characteristic intensity below the fourth intensity threshold to a characteristic strength above the fourth intensity threshold contact with a characteristic intensity above the fourth intensity threshold in the characteristic intensity of the contact. Maintaining the display of the second subset of representations of the filters without displaying the first subset of representations of the filters. 80. The method of any of claims 68-79,
The representation of the first filter in the filter selection interface includes a live preview of the field of view of the one or more cameras of the device to which the first filter is applied;
The representation of the second filter in the filter selection interface includes a live preview of the field of view of the one or more cameras of the device to which the second filter is applied. 81. The method of any of claims 68-80, wherein stopping displaying the first subset of representations of the filters comprises:
Progressively reducing the size of the representation of the first filter while the representation of the first filter continues to display at least a portion of the live preview of the field of view of the one or more cameras of the device to which the first filter is applied; And
Gradually reducing the size of the representation of the second filter while the representation of the second filter continues to display at least a portion of the live preview of the field of view of the one or more cameras of the device to which the second filter has been applied. Further comprising. 82. The method of claim 81, wherein the method is
In response to detecting the movement of the second contact,
Generating a respective tactile output in accordance with determining that the fourth filter is a first type of filter; And
In accordance with determining that the fourth filter is a second type of filter, suspending generating the respective tactile output. 69. The method of claim 68, wherein the electronic device comprises a touch-sensitive display, and wherein the first input corresponds to a contact on the touch-sensitive display maintained for a predetermined period of time. 84. The method of any of claims 68 to 83, wherein the method further comprises providing a tactile output at the electronic device in response to detecting the first input. 69. The method of claim 68, wherein the representations of the plurality of filters in the set of filters include a third subset of representations of filters that are not displayed on the display while detecting the first input. 86. The method of any of claims 68 to 85, wherein the filter selection interface further comprises a visual indicator identifying a representation of a third filter, wherein the third filter is the most recently used filter. 87. The method of any one of claims 68 to 86, wherein the representation of the first filter corresponds to the no filter option, and wherein the first subset of representations of the filters is the representation of the first filter and the most recently applied filter. And at least two of said plurality of filters except a representation of a fourth filter corresponding to. 88. The representation of any of claims 68-87, wherein the representation of the first filter corresponds to a most recently used filter option, and wherein the first subset of representations of the filters is a representation and filter of the first filter. And at least two of said plurality of filters except a representation of a fifth filter corresponding to the none option. 89. The method of any of claims 68 to 88, wherein the representation of the first filter does not correspond to the most recently applied filter and does not correspond to the no filter option, and wherein the first subset of representations of the filters is And at least two of said plurality of filters except a representation of a first filter, a representation of a sixth filter corresponding to the most recently applied filter, and a representation of a seventh filter corresponding to the no filter option. 90. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display, wherein the one or more programs are any one of claims 68-89. A non-transitory computer readable storage medium comprising instructions for performing the method of claim 1. As an electronic device,
One or more input devices;
display;
One or more processors; And
89. A device comprising memory for storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any one of claims 68-89. As an electronic device,
One or more input devices;
display; And
89. An electronic device comprising means for performing the method of any of claims 68-89. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display, wherein the one or more programs are:
Instructions for displaying, on the display, a filter selection interface that includes representations of a plurality of filters in a set of filters;
While simultaneously displaying a representation of image data and the filter selection interface on the display, a location corresponding to the filter selection interface while the first filter of the set of filters meets selection criteria, through the one or more input devices Instructions for detecting a first input at;
In response to detecting the first input,
Instructions to stop displaying a first subset of representations of filters in the set of filters, wherein the first subset of representations of the filters in the set of filters is a representation of the first filter in the filter selection user interface One or more filters in a first direction from and one or more filters in a second direction from a representation of the first filter; And
Instructions for maintaining a display of a second subset of representations of filters in the set of filters, the second subset of representations of filters including at least a representation of the first filter. Storage media. As an electronic device,
One or more input devices;
display;
One or more processors; And
A memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs are:
Instructions for displaying, on the display, a filter selection interface that includes representations of a plurality of filters in a set of filters;
While simultaneously displaying a representation of image data and the filter selection interface on the display, a location corresponding to the filter selection interface while the first filter of the set of filters meets selection criteria, through the one or more input devices Instructions for detecting a first input at;
In response to detecting the first input,
Instructions to stop displaying a first subset of representations of filters in the set of filters, wherein the first subset of representations of the filters in the set of filters is a representation of the first filter in the filter selection user interface One or more filters in a first direction from and one or more filters in a second direction from a representation of the first filter; And
And instructions for maintaining a display of a second subset of representations of filters in the set of filters, wherein the second subset of representations of filters includes at least a representation of the first filter. As an electronic device,
One or more input devices;
display;
Means for displaying on the display a filter selection interface that includes representations of a plurality of filters in a set of filters;
While simultaneously displaying a representation of image data and the filter selection interface on the display, a location corresponding to the filter selection interface while the first filter of the set of filters meets selection criteria, through the one or more input devices Means for detecting a first input at;
In response to detecting the first input,
Means for stopping displaying the first subset of representations of filters in the set of filters, wherein the first subset of representations of the filters in the set of filters is derived from the representation of the first filter in the filter selection user interface. One or more filters in a first direction and one or more filters in a second direction from the representation of the first filter; And
Means for maintaining a display of a second subset of representations of filters in the set of filters, wherein the second subset of representations of filters includes at least a representation of the first filter. As a method,
In an electronic device having a camera, a sensor, one or more input devices and a display,
Displaying a camera viewfinder for capturing media on the display; And
While displaying the camera viewfinder,
Based on the data from the sensor, in accordance with a determination that the device meets alignment guide display criteria—the alignment guide display criteria are between an orientation of the focal plane of the camera and a predetermined orientation such that the alignment guide display criteria are met. Including the requirement that the relative difference of s is within a respective alignment threshold,-displaying, on the display, an alignment guide in the camera viewfinder-wherein the appearance of the alignment guide is such that the orientation of the focal plane of the camera is in advance; Change as it changes with respect to the determined orientation; And
Based on the data from the sensor, suspending displaying the alignment guide in the camera viewfinder in accordance with a determination that the alignment guide display criteria are not met. 97. The apparatus of claim 96, wherein the alignment guide display criteria are within the respective alignment threshold for the relative difference between the orientation of the focal plane of the camera and the predetermined orientation such that the alignment guide display criteria are met. Comprising the requirements maintained in the. 97. The apparatus of claim 96, wherein the alignment guide display criteria are adapted while the relative difference between the orientation of the focal plane of the camera and the predetermined orientation is maintained within the respective alignment threshold such that the alignment guide display criteria are met. And a requirement that the orientation of the device does not change more than the threshold amount at the threshold amount of time. 97. The method of claim 96, wherein the predetermined orientation corresponds to a horizontal orientation. 97. The method of claim 96, wherein the predetermined orientation is a vertical orientation. 101. The method of any one of claims 96-100, wherein the alignment guide comprises at least two visual indicators. 102. The method of claim 101, wherein at least one of the at least two visual indicators remains stationary as the orientation of the focal plane of the camera changes with respect to the predetermined orientation. 102. The method of claim 101 or 102, wherein at least one of the at least two visual indicators is displayed proximate to the center of the camera viewfinder. 103. The method of any one of claims 101-103, wherein the distance between the two visual indicators is dynamically based on a relative difference between the orientation of the focal plane of the camera and the predetermined orientation. 105. The method of any one of claims 101-104, wherein
While displaying the alignment guide and while the orientation of the focal plane of the camera is a first orientation, the orientation of the focal plane of the camera from the first orientation to the second orientation based on data from the sensor Detecting a change in; And
Changing the displayed position of the first visual indicator of the at least two visual indicators in response to detecting a change in the orientation of the focal plane of the camera from the first orientation to the second orientation. And wherein the displayed position of the first visual indicator is changed based on the relative difference between the first orientation and the second orientation. 107. The method of claim 105, wherein the method is
In response to detecting a change in the orientation of the focal plane of the camera from the first orientation to the second orientation,
Displaying the first visual indicator at an updated display position in accordance with a determination that the relative difference between the second orientation and the predetermined orientation of the focal plane of the camera is not within a first visual indicator alignment threshold. ; And
Displaying the first visual indicator at a predetermined display position according to a determination that the relative difference between the second orientation and the predetermined orientation of the focal plane of the camera is within the first visual indicator alignment threshold. Further comprising. 107. The method of claim 106, wherein displaying the first visual indicator at the predetermined display position comprises displaying a respective animation at the location of one or more of the visual indicators. 107. The method of claim 106 or 107, wherein the method is
While the first visual indicator is displayed at the predetermined display position, detecting a change in orientation of the focal plane of the camera from a third orientation to a fourth orientation based on data from the sensor; The third orientation has a relative difference to the predetermined orientation that is within the first visual indicator alignment threshold;
In response to detecting a change in the orientation of the focal plane of the camera from the third orientation to the fourth orientation,
The display of the first visual indicator at the predetermined display position in accordance with a determination that the relative difference between the fourth orientation and the predetermined orientation of the focal plane of the camera is outside the second visual indicator alignment threshold. Maintaining; And
Displaying the first visual indicator at a second updated display position according to a determination that the relative difference between the fourth and the predetermined orientation of the focal plane of the camera is outside the second visual indicator alignment threshold. And the location of the second updated display position is based on the relative difference between the orientation of the focal plane of the camera and the predetermined orientation. 109. The method of any one of claims 101-108, wherein the method comprises
While the orientation of the focal plane of the camera is in a fifth orientation with a relative difference with respect to the predetermined orientation that is within the respective alignment threshold, and wherein the second visual indicator of the at least two visual indicators is defined as the first of the visual characteristics. While displaying a value of 1, detecting a change in the orientation of the focal plane of the camera from the fifth orientation to a sixth orientation based on the data from the sensor—the above of the focal plane of the camera The relative difference between the sixth orientation and the predetermined orientation is within the visual indicator alignment threshold; And
In response to detecting a change in the orientation of the focal plane of the camera from the fifth orientation to the sixth orientation, the second visual indicator having a second value of the visual characteristic different from the first value; And further comprising displaying. 109. The method of claim 109,
The fifth orientation of the focal plane of the camera has a first relative difference between an orientation of the focal plane of the camera and a predetermined orientation,
The sixth orientation of the focal plane of the camera has a second relative difference between the orientation of the focal plane of the camera and a predetermined orientation, the second relative distance being greater than the first relative difference,
The visual characteristic is display intensity,
The first value is greater than the second value. 109. The method of any one of claims 101-108, wherein the method comprises
While the orientation of the focal plane of the camera is in a seventh orientation with a relative difference to the predetermined orientation that is within the respective alignment threshold, and while a third visual indicator of the at least two visual indicators is displayed, Based on data from the sensor, detecting a change in orientation of the focal plane of the camera from the seventh orientation to an eighth orientation,
In response to detecting a change in the orientation of the focal plane of the camera from the seventh orientation to the eighth orientation,
Maintaining a display of the third visual indicator in accordance with a determination that the relative difference between the eighth orientation and the predetermined orientation of the focal plane of the camera is within the visual indicator alignment threshold; And
Stopping the display of the third visual indicator in accordance with a determination that the relative difference between the eighth orientation and the predetermined orientation of the focal plane of the camera is not within the visual indicator alignment threshold. How to. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having a camera, a sensor, one or more input devices, and a display, the one or more programs comprising: A non-transitory computer readable storage medium comprising instructions for performing the method of any one of claims 111. As an electronic device,
camera;
sensor;
One or more input devices;
display;
One or more processors; And
112. A device comprising memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 96-111. As an electronic device,
camera;
sensor;
One or more input devices;
display; And
111. An electronic device comprising means for performing the method of any of claims 96-111. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having a camera, a sensor, one or more input devices, and a display, wherein the one or more programs include:
Instructions for displaying a camera viewfinder for capturing media on the display; And
While displaying the camera viewfinder,
Based on the data from the sensor, in accordance with a determination that the device meets alignment guide display criteria—the alignment guide display criteria are between an orientation of the focal plane of the camera and a predetermined orientation such that the alignment guide display criteria are met. Includes the requirement that the relative difference of s is within a respective alignment threshold-instructions for displaying, on the display, an alignment guide in the camera viewfinder-the appearance of the alignment guide is such that the orientation of the focal plane of the camera is Vary as it changes with respect to the predetermined orientation; And
And instructions for withholding displaying the alignment guide in the camera viewfinder, based on the data from the sensor, in accordance with a determination that the alignment guide display criteria are not met. As an electronic device,
camera;
sensor;
One or more input devices;
display;
One or more processors; And
A memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs are:
Instructions for displaying a camera viewfinder for capturing media on the display; And
While displaying the camera viewfinder,
Based on the data from the sensor, in accordance with a determination that the device meets alignment guide display criteria—the alignment guide display criteria are between an orientation of the focal plane of the camera and a predetermined orientation such that the alignment guide display criteria are met. Includes the requirement that the relative difference of s is within a respective alignment threshold-instructions for displaying, on the display, an alignment guide in the camera viewfinder-the appearance of the alignment guide is such that the orientation of the focal plane of the camera is Vary as it changes with respect to the predetermined orientation; And
Based on data from the sensor, instructions for withholding displaying the alignment guide in the camera viewfinder in accordance with a determination that the alignment guide display criteria are not met. As an electronic device,
camera;
sensor;
One or more input devices;
display;
Means for displaying a camera viewfinder for capturing media on the display; And
While displaying the camera viewfinder,
Based on the data from the sensor, in accordance with a determination that the device meets alignment guide display criteria—the alignment guide display criteria are between an orientation of the focal plane of the camera and a predetermined orientation such that the alignment guide display criteria are met. Means for displaying an alignment guide in the camera viewfinder on the display, the appearance of the alignment guide being such that the orientation of the focal plane of the camera is Change as it changes for a predetermined orientation; And
And means for withholding displaying the alignment guide in the camera viewfinder in accordance with the determination that the alignment guide display criteria are not met based on data from the sensor. As a method,
In an electronic device having one or more input devices, one or more cameras and a display,
Displaying, on the display, a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Detecting, via the one or more input devices, a first input for selecting a respective filter of a plurality of lighting effects based on the depth map information;
After detecting the first input, detecting a second input corresponding to a request to capture image data corresponding to a field of view of the one or more cameras; And
In response to detecting the second input, capturing image data corresponding to the field of view of the one or more cameras, wherein the capturing comprises:
In accordance with a determination that the respective lighting effects selected based on the first input are first lighting effects, capture image data corresponding to the field of view of the one or more cameras and compare the first lighting effect with the representation of the image data. Associating, wherein the first lighting effect is based on the depth map information; And
Capturing image data corresponding to the field of view of the one or more cameras and determining that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect Associating an effect with a representation of the image data, wherein the second lighting effect is based on the depth map information. 118. The method of claim 118, wherein the representation of the image data is a live preview of image data captured within the field of view of the one or more cameras displayed in a digital viewfinder. 119. The apparatus of claim 119, wherein the first input is an input received while first criteria are met, the first criteria being detected by an object in the field of view within a predetermined distance from the electronic device such that the first criteria are met. A method comprising the requirements. 121. The method of any one of claims 118-120, comprising, in response to detecting the first input, preparing to capture image data using the respective filter, wherein the preparing comprises:
In accordance with determining that the respective lighting effect is the first lighting effect, causing a first change in appearance of a portion of the representation of the image data when an object in the field of view of the one or more cameras meets the first criteria. Displaying a capture user interface for the first lighting effect; And
In accordance with determining that the respective lighting effect is the second lighting effect, causing a second change in appearance of a portion of the representation of the image data when an object in the field of view of the one or more cameras meets the first criteria. Displaying a capture user interface for the second lighting effect. 121. The method of claim 121, wherein displaying the capture user interface for the first lighting effect comprises:
Representation of the image data; And
Simultaneously displaying an alignment guide displayed in a first area within the representation of the image data, wherein the first criteria include a representation of the face of the subject displayed in the representation of the image data such that the first criteria are met. A requirement to be within the alignment guide. 123. The method of claim 121 or 122,
The first criteria include a requirement that a representation of the face of the subject was detected in a first region within the representation of the image data such that the first criteria are met;
Applying the first change in the appearance to the image data is compared with the appearance of the representation of the image data displayed in the first region in the representation of the image data, wherein the display is displayed in the second region in the representation of the image data. Changing an appearance of the representation of image data, wherein the second region is separate from the first region. 123. The method of any of claims 121-123, wherein displaying the capture user interface for the first lighting effect comprises:
Applying a placeholder filter to the representation of the image data displayed in the digital viewfinder, wherein the placeholder filter is based on the first lighting effect and irrespective of whether the first criteria are met Applied, method. 126. The method of claim 124, wherein displaying the capture user interface for the first lighting effect comprises: while a first change in appearance of the image data is being applied,
Determining that the first criteria are not met; And
In response to determining that the first criteria are not met,
Stopping applying the first change in appearance to the representation of the image data;
Displaying on the display a representation of the image data to which the first change in appearance has not been applied; And
And displaying on the display a graphical representation of the unmet first criteria. 121. The method of claim 121, wherein
In response to the first input, applying a placeholder filter to the live preview without applying the first change in appearance to the live preview; And
In response to detecting that the first criteria have been met, applying the first change in appearance to the live preview while continuing to apply the placeholder filter to the live preview. 121. The method of claim 121, wherein
The first input is an input received while the first criteria are not met, the first criteria including a requirement that an object is detected in the field of view within a predetermined distance from the electronic device such that the first criteria are met; ;
The method,
After displaying the live preview without applying the first change in appearance to the live preview, detecting that the first criteria have been met; And
In response to detecting that the first criteria have been met, applying the first change in appearance to the live preview. 118. The method of claim 118 wherein
After capturing image data corresponding to the field of view of the one or more cameras, receiving a request to display captured image data in response to the second input; And
In response to receiving a request to display captured image data in response to the second input,
Displaying a representation of the image data to which the first lighting effect has been applied in accordance with a determination that the image data was captured while the first lighting effect was selected; And
In accordance with a determination that the image data was captured while the second lighting effect was selected, displaying a representation of the image data to which the second lighting effect has been applied. 129. The method of claim 128,
While displaying the representation of the image data, further comprising displaying on the display a visual indication that the image data includes depth map information. 129. The method of any of claims 118-129, wherein the depth map information associated with the image data includes information corresponding to at least three different depth levels. 133. The apparatus of any of claims 118-130, wherein the depth map information associated with the image data includes information identifying depth contours of an object in the representation of the image data;
The lighting effects change the appearance of the representation of the image data based on the position and curvature of the contours of the object. The method of any one of claims 118 to 131, wherein applying the first lighting effect or the second lighting effect,
Applying a simulation of one or more point light sources in space based on the depth map information associated with the image data to the representation of the image data displayed in the viewfinder. 129. The method of any of claims 118-132,
While the first lighting effect is applied, further comprising maintaining at least one value of a previously applied visual effect. 138. The method of any one of claims 118-133, wherein the previously applied visual effect is a color filter. 138. The method of any of claims 118-134, wherein the second input is an input received while the first lighting effect is being applied to the representation of the image data, and wherein applying the second lighting effect ,
Gradually switching between application of the first lighting effect and the second lighting effect. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices, one or more cameras, and a display, the one or more programs of claim 118. A non-transitory computer readable storage medium comprising instructions for performing the method of any one of claims 1-35. As an electronic device,
One or more input devices;
One or more cameras;
display;
One or more processors; And
135. A device comprising memory for storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions for performing the method of any of claims 118-135. As an electronic device,
One or more input devices;
One or more cameras;
display; And
135. An electronic device comprising means for performing the method of any of claims 118-135. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices, one or more cameras, and a display, wherein the one or more programs include:
Instructions for displaying, on the display, a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Instructions for detecting, via the one or more input devices, a first input for selecting a respective filter of a plurality of lighting effects based on the depth map information;
Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of the one or more cameras after detecting the first input; And
In response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, wherein the instructions for capturing,
In accordance with a determination that the respective lighting effects selected based on the first input are first lighting effects, capture image data corresponding to the field of view of the one or more cameras and compare the first lighting effect with the representation of the image data. Instructions for associating, wherein the first lighting effect is based on the depth map information; And
Capturing image data corresponding to the field of view of the one or more cameras and determining that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect Instructions for associating an effect with a representation of the image data, wherein the second lighting effect is based on the depth map information. As an electronic device,
One or more input devices;
One or more cameras;
display;
One or more processors; And
A memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs are:
Instructions for displaying, on the display, a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Instructions for detecting, via the one or more input devices, a first input for selecting a respective filter of a plurality of lighting effects based on the depth map information;
Instructions for detecting a second input corresponding to a request to capture image data corresponding to a field of view of the one or more cameras after detecting the first input; And
In response to detecting the second input, instructions for capturing image data corresponding to the field of view of the one or more cameras, wherein the instructions for capturing,
In accordance with a determination that the respective lighting effects selected based on the first input are first lighting effects, capture image data corresponding to the field of view of the one or more cameras and compare the first lighting effect with the representation of the image data. Instructions for associating, wherein the first lighting effect is based on the depth map information; And
Capturing image data corresponding to the field of view of the one or more cameras and determining that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect And instructions for associating an effect with a representation of the image data, wherein the second lighting effect is based on the depth map information. As an electronic device,
One or more input devices;
One or more cameras;
display;
Means for displaying on the display a representation of image data associated with depth map information;
While displaying a representation of the image data on the display,
Means for detecting, via the one or more input devices, a first input for selecting a respective filter of a plurality of lighting effects based on the depth map information;
Means for detecting a second input corresponding to a request to capture image data corresponding to a field of view of the one or more cameras after detecting the first input; And
In response to detecting the second input, means for capturing image data corresponding to the field of view of the one or more cameras, wherein the means for capturing comprises:
In accordance with a determination that the respective lighting effects selected based on the first input are first lighting effects, capture image data corresponding to the field of view of the one or more cameras and compare the first lighting effect with the representation of the image data. Means for associating, wherein the first lighting effect is based on the depth map information; And
Capturing image data corresponding to the field of view of the one or more cameras and determining that the respective lighting effects selected based on the first input are second lighting effects different from the first lighting effect Means for associating an effect with a representation of the image data, wherein the second lighting effect is based on the depth map information.

Images