JP2015518295A - Color adjuster for color classification - Google Patents

Abstract

A non-transitory machine readable medium having a computer program for adjusting color values of an image represented in a color space is described. This image contains a set of pixels. Each pixel has a set of color values. The computer program receives user input for a user interface (UI) item for adjusting the color value of the image associated with the type of content. The computer program identifies a subset of pixels having color values that fall within the range of color values associated with that content type. The computer program performs a color adjustment operation on the identified subset of pixels. [Selection] Figure 4

Description

  Digital graphic design and image editing applications (hereinafter collectively referred to as image editing applications) provide graphic designers, media artists, and other users with the tools necessary to view and edit images. Examples of such applications include iPhoto®, Aperture®, and Final Cut Pro®, all of which are available from Apple, Inc. It is sold by. These applications give the user the ability to edit images in various ways. For example, some applications provide various range sliders for adjusting various color values of an image or video.

  However, many image editing applications do not provide intuitive color adjustment controls. For example, a user is required to have extensive knowledge about color editing in order to effectively use most of the existing color adjustment tools. Furthermore, controls for adjusting various features of the color values of the image are distributed at various locations on the user interface. These deficiencies cause unnecessary inconvenience when editing images.

  Several novel user interface (UI) tool embodiments for editing images in an image editing application are described. In some embodiments, the image editing application is for adjusting color values of only a portion of the image associated with the type of content (eg, sky, foliage, etc.) associated with a particular color range. Provides a set of UI controls. When input is received through a UI control to adjust the color value of a particular content type on the image, the application automatically sets the set of pixels in the image associated with that content type. Identify. The application then adjusts only the color values of the identified set of pixels based on user input.

  In some embodiments, each UI control is for adjusting color values associated with different types of content in the image. The application of some embodiments defines a range of color values within the color space for each type of content. When input is received through a particular UI control, the application searches for a pixel in the image and has a color value that falls within the color value range defined for the type of content corresponding to that particular UI control. Identify a set of pixels.

  The image includes pixels that have color values defined in a color space. In some embodiments, the color values of the image are defined in a color space that is different from the specific color space in which the range of color values associated with the various content types is defined. In these embodiments, the application converts the color values of the image from their original color space to a specific color space in which the range of color values is defined before specifying the set of pixels.

  In some embodiments, each color adjustment includes a range of adjustment values for specifying various degrees of adjustment applied to the image. In these embodiments, the UI control also provides a means for the user to specify an adjustment value for controlling the degree of adjustment applied to the image. Various embodiments implement a set of UI controls using various techniques. For example, some embodiment applications implement a set of UI controls as a set of range sliders. In these embodiments, the user can specify various adjustment values by selecting various positions on the range slider.

  In some embodiments, the application performs only one type of adjustment (eg, saturation adjustment) to the color values of the image in response to user input for a particular UI control. However, some other embodiments of the application perform more than one type of adjustment to the color values of the image in response to a single user input for a particular UI control. For example, in response to user input, an application of some embodiments can perform saturation adjustment, contrast adjustment, and brightness adjustment on image color values. In these embodiments, the application uses a single user input to determine adjustment values for each adjustment operation and applies these adjustment operations to the image.

  Some embodiments provide another novel UI for editing images in an image editing application. In these embodiments, the application allows the user to select a location on the image and applies to the image when various content types are detected at the selected location on the image. An image editing tool for displaying various UI controls associated with various image editing operations is provided. In these embodiments, the user selects a location on the image, and the image editing tool of the application obtains the color value of the pixel corresponding to the selected location on the image from the image. The color editing tool then performs a series of analyzes on those pixel values to detect whether the selected location represents a particular type of content. Based on the type of content detected, the color editing tool determines a set of image editing operations and displays a set of UI controls associated with the determined image editing operations. In some embodiments, UI controls are superimposed on the image.

  Various embodiments perform various analyzes on the color values of the acquired image. In some embodiments, the application performs a series of analyzes to detect whether the selected location of the image represents a predefined content type. In these embodiments, the application defines different color value ranges for different content types. The application then determines whether the acquired color value falls within any one of those color value ranges. If the acquired color value falls within a specific color value range, the application associates the acquired color value, as well as the selected location, with the corresponding content type. The application then provides a set of overlay UI controls based on the type of content detected.

  In some embodiments, the set of UI controls are superimposed on the image at or near the selected location. In some embodiments, this overlay UI control is either opaque or transparent. A user can manipulate these UI controls by providing input to the image. In some of these embodiments, the UI control is a direction-dependent UI control that allows the user to select a specific image editing operation by providing a specific direction input. When the user provides a directional input to the image, the application associates the input with one of the UI controls based on the direction of the input and applies the corresponding adjustment to the image.

  In some embodiments, each image editing operation that can be performed on an image through a superposition of UI controls allows adjustment values to be specified for varying degrees of editing performed on the image. Includes range. In these embodiments, the application also determines the size based on the directional input provided by the user. The application then calculates an adjustment value for the image editing operation based on the identified input magnitude, and performs the image editing operation on the image using the calculated adjustment value.

  In addition to content-dependent UI control, some embodiment applications also provide a new UI for adjusting the color balance of an image. In some embodiments, the application provides a color balance UI control that allows the user to select a location on the image. The application then obtains the color value of the pixel corresponding to the selected location and changes the color value of the pixel to a set reference color (eg, gray, ideal skin color, etc.) To decide. Based on this color adjustment, the application generates a color space transformation that maps each color in the image to a different color. The application then applies the color space transformation to every pixel in the image.

  Various embodiments provide various reference colors for this color balance adjustment operation. In some embodiments where gray is set as the reference color, the user can adjust the color balance of the image through the color balance UI by selecting a location in the image that displays objects that appear as gray. . In other embodiments, the application can set the ideal skin color as the reference color. In these embodiments, the user adjusts the color balance of the image through the color balance UI by selecting a location in the image that displays a human face.

  In some embodiments, the application provides a color balance UI control that is overlaid on the image to adjust the color balance of the image. In some of these embodiments, the color balance UI control is movable on the image, allowing the user to specify a location on the image by moving the color balance UI control to the desired location. To.

  In some embodiments, the color balance UI control provides a zooming capability to allow the user to accurately select a location for white balance operations. In these embodiments, the color balance UI control includes a closed border. The application zooms in (ie, enlarges) the portion of the image within the closed border of this color balance UI control. Since the image inside the closed border of the color balance UI control appears to be larger, the user can select a location on the image more accurately.

  In addition to zooming capabilities, the color balance UI control can also provide the user with a preview of the edited image. In some embodiments, the application displays an edited version of the portion of the image that is outside the closed border of the color balance UI control and the portion of the image that is inside the closed border. Display the unedited version.

  The above "Summary of the Invention" is intended to serve as a concise introduction to some embodiments described herein. It is not meant to be an introduction or summary of all inventive subject matter disclosed in this document. The drawings referred to in the following “DETAILED DESCRIPTION OF THE INVENTION” and “MODE FOR CARRYING OUT THE INVENTION” further illustrate the embodiment described in this “Summary of the Invention” and other embodiments. Explain. Accordingly, in order to understand all of the embodiments described by this document, a thorough examination of the "Summary of Invention", "Mode for Carrying Out the Invention", and the drawings is required. Furthermore, the claimed subject matter is not limited by the exemplary details in the "Summary of Invention", "Mode for Carrying Out the Invention", and drawings, but rather is claimed subject matter. Can be embodied in other specific forms without departing from the spirit of the subject matter, and thus is defined by the appended claims.

The novel features described herein are set forth in the appended claims. However, for illustrative purposes, some embodiments are described in the following figures.
FIG. 6 illustrates an example of providing different overlay UI controls based on different locations selected on an image. FIG. 10 is a diagram illustrating an example in which the color balance of an image is adjusted using a custom color balance UI control. It is a figure which shows the Example which adjusts the saturation of an image through GUI of the image editing application of some embodiment. It is a figure which shows another Example which adjusts the saturation of an image through GUI of the image editing application of some embodiment. FIG. 5 is a diagram illustrating an example of identifying a set of pixels having color values that fall within a range of color values defined for skin tone. It is a figure which shows the Example which adjusts the color temperature of an image based on the color tone of the skin on an image. It is a figure which shows the Example which adjusts only the part of the image expressing the sky through GUI of the image editing application of some embodiment. FIG. 6 illustrates an example of identifying a set of pixels having color values that fall within a range of color values defined for a sky color. It is a figure which shows the Example which adjusts only a part of image which expresses a foliage through GUI of the image editing application of some embodiment. It is a figure which shows another Example which adjusts only the one part of the image expressing a foliage through GUI of the image editing application of some embodiment. FIG. 6 is a diagram illustrating an example of identifying a set of pixels having color values that fall within a range of color values defined for foliage colors. FIG. 6 is a diagram illustrating an example of calling a set of color adjustment tools through different GUIs of an image editing application of some embodiments. FIG. 6 is a diagram illustrating an example of deselecting a set of color adjustment tools through a different GUI of an image editing application of some embodiments. FIG. 6 is a diagram illustrating another example of adjusting only a portion of an image representing the sky through different GUIs of an image editing application of some embodiments. It is a figure which shows another Example which adjusts only the part of the image expressing a foliage through different GUI of the image editing application of some embodiment. FIG. 3 conceptually illustrates a process of some embodiments for adjusting only a portion of an image based on a user selection of a location on the image. FIG. 5 conceptually illustrates a process of some embodiments for performing multiple color adjustments on an image based on a single user input. It is a figure which shows notionally the software architecture of the image editing application of some embodiment. FIG. 3 conceptually illustrates a process of some embodiments for providing different on-image UI controls based on different locations selected on an image. FIG. 6 illustrates an example of providing different on-image UI controls based on different locations selected on an image. It is a figure which shows the Example which adjusts the saturation of an image through GUI of the image editing application of some embodiment. It is a figure which shows the Example which adjusts the color temperature of an image based on the color tone of the skin on an image through GUI of the image editing application of some embodiment. It is a figure which shows the Example which adjusts only the part of the image expressing the sky through GUI of the image editing application of some embodiment. It is a figure which shows the Example which adjusts only a part of image which expresses a foliage through GUI of the image editing application of some embodiment. FIG. 2 illustrates two example implementations of on-image UI control. FIG. 6 conceptually illustrates a process of some embodiments for providing different color adjustment UI controls based on different locations selected on an image. It is a figure which shows notionally the Example which specifies the direction and magnitude | size of a direction input. FIG. 6 conceptually illustrates a process of some embodiments for determining a particular color adjustment based on a direction input. It is a figure which shows notionally the software architecture of the image editing application of some embodiment. FIG. 6 illustrates the benefits of using UI controls on images. FIG. 6 illustrates an example of providing different on-image UI controls based on different locations selected on an image. It is a figure which shows the Example which adjusts the contrast of an image through GUI of the image editing application of some embodiment. It is a figure which shows the Example which adjusts the shadow of an image through GUI of the image editing application of some embodiment. It is a figure which shows the Example which adjusts the highlight of an image through GUI of the image editing application of some embodiment. It is a figure which shows the Example which adjusts the brightness of an image through GUI of the image editing application of some embodiment. FIG. 6 conceptually illustrates a process of some embodiments for providing different color adjustment UI controls based on different locations selected on an image. It is a figure which shows the Example which performs custom white balance operation with respect to an image through GUI of the image editing application of some embodiment. It is a figure which shows the Example which produces | generates the color space conversion regarding custom white balance operation. It is a figure which shows the Example which performs custom face balance operation with respect to an image through GUI of the image editing application of some embodiment. It is a figure which shows the Example which produces | generates the color space conversion regarding custom face balance operation. It is a figure which shows the Example which adjusts the color temperature of an image based on the color tone of an average skin through GUI of the image editing application of some embodiment. It is a figure which shows the Example which adjusts the color temperature of an image based on the ideal tone of the skin through GUI of the image editing application of some embodiment. FIG. 6 is a diagram illustrating an example of calling a set of color balance tools through different GUIs of an image editing application of some embodiments. FIG. 3 conceptually illustrates a process of some embodiments for performing a custom color balance operation on an image. FIG. 5 conceptually illustrates a process of some embodiments for generating a color space transformation for a custom color balance operation. It is a figure which shows notionally the software architecture of the image editing application of some embodiment. FIG. 3 illustrates an example GUI of an image editing application of some embodiments. It is a figure which shows notionally the image data structure of some embodiment. FIG. 2 conceptually illustrates a hardware architecture of an apparatus of some embodiments in which an image editing application is executed. FIG. 2 conceptually illustrates an electronic system in which some embodiments of the present invention are implemented.

  In the following Detailed Description, numerous details, examples, and embodiments of image editing tools and image editing applications are described and described. However, it will be apparent to those skilled in the art that the present invention is not limited to the described embodiments, and that the present invention may be practiced without some of the specific details and examples discussed. It will be clear and obvious.

  Several novel user interface (UI) tool embodiments for editing images in an image editing application are described. In some embodiments, the image editing application is for adjusting color values of only a portion of the image associated with the type of content (eg, sky, foliage, etc.) associated with a particular color range. Provides a set of UI controls. When input is received through a UI control to adjust the color value of a particular content type on the image, the application automatically sets the set of pixels in the image associated with that content type. Identify. The application then adjusts only the color values of the identified set of pixels based on user input.

  In some embodiments, the set of UI controls includes UI controls for adjusting color values of only the sky colors in the image. In these embodiments, the application first identifies a set of pixels in the image that have color values that fall within a predefined range of sky color values. The application then applies color adjustments only to the identified set of pixels. In addition to adjusting the color value of the sky color, the set of UI controls in some embodiments also includes a UI control for adjusting the color values of only the foliage colors in the image. Similarly, this application identifies a set of pixels in the image that have color values that fall within a predefined foliage color value range, and applies color adjustments only to that identified set of pixels. To do. The color adjustment operation for adjusting the blue sky color and foliage color is described in more detail in Section I below.

  Further, the image editing application of some embodiments also allows a user to select a location on the image, and when various content types are detected from the selected location on the image. An image editing tool is also provided that displays various UI controls associated with various image editing operations for application to the image. In these embodiments, the user selects a location on the image, and the image editing tool of the application obtains the color value of the pixel corresponding to the selected location on the image from the image. The application then performs a series of analyzes on those pixel values to detect whether the selected location represents a particular content type. Based on the type of content detected, the color editing tool determines a set of image editing operations and displays a set of UI controls associated with the determined image editing operations. Various embodiments for providing content-dependent on-image UI control are described in more detail in Section II below.

  FIG. 1 illustrates an exemplary GUI 100 of an image editing application of some embodiments that provides a set of UI controls for adjusting color values of only a portion of the image. Specifically, FIG. 1 illustrates the operation of activating and manipulating various sets of UI controls at four different stages 105, 110, 115, and stage 120. Each of these stages is described in more detail below.

  As shown in FIG. 1, the GUI 100 includes an image 195 and a set of color adjustment UI controls 125-135. Each of the color adjustment UI controls is for initiating different color adjustments for different parts of the image. For example, the UI control 125 is for applying “Adjustment A” to the entire image, the UI control 130 is for applying “Adjustment B” to the empty area of the image, and the UI control 135 is This is for applying “Adjustment C” to the foliage region of the image.

  Various embodiments implement a set of UI controls using various techniques. In this example, the application implements a set of UI controls as a set of range sliders. In these embodiments, the user can initiate different adjustment operations on the image by selecting and manipulating different UI controls 125-135. In addition to the range slider, the GUI 100 also initiates an adjustment operation by the user activating a set of superimposed UI controls (also referred to as on-image UI controls) corresponding to the set of UI controls 125-135. It is also possible to do.

  The second stage 110 shows the GUI 100 after the user activates a set of superposition UI controls. In some embodiments, the user can activate a set of superimposed UI controls by selecting a location on the image. In these embodiments, the application obtains the color value of the pixel corresponding to the selected location and associates that color value with one of the predefined content. The application then provides a set of overlay UI controls that are associated with that associated content type. As shown, the user has selected a location on the image 195 that displays the sky. Selection of a location on the image 195 can be performed by performing a gesture at a location on the device that has the touch or near-touch sensitive screen that displays the image, or by placing the cursor at the location of the image and providing input ( For example, by clicking on the cursor control device, pressing a hot key, etc.). In one embodiment, the user can select a location by tapping the finger at the location of the image or placing the finger at that location. In other embodiments, the location can be selected by performing other gestures. As described above, after the user selects a location on the image, the image editing application of some embodiments performs at least one analysis to determine the selected location as a predefined content type. Associate with. Various embodiments use various techniques to determine the type of content associated with the selected location. The application of some embodiments initially defines a range of different color values associated with different types of content. The application then determines whether the color value of the set of pixels corresponding to the selected location in the image falls within the color value range associated with the particular content type, and the identification Displays a set of on-image UI controls associated with the content type. In this example, the application determines that the selected location is associated with a blue sky. Therefore, this application corresponds to the UI control 140 for performing “Adjust B” on the empty area of the image, and the UI control 155 on the image and the UI control 160 on the image (arrows in two directions along the horizontal axis). As a set).

  This application is also displayed as an overlay UI control 145 and an overlay UI control 150 (two direction arrows along the vertical axis) corresponding to the UI control 125 for performing “Adjust A” on the entire image. ) Set. The set of the overlay UI control 145 and the overlay UI control 150 does not depend on the content. That is, the application provides a set of this overlay UI control 145 and overlay UI control 150 regardless of the location selected on the image. In some embodiments, the application also highlights UI control 125 and UI control 130 to indicate that UI control 125 and UI control 130 are activated. (The UI control border is bolded to show that the UI control is highlighted.) Although not shown in this figure, the user at this second stage 110 can By providing a directional input, a particular color adjustment (ie, “Adjust A” or “Adjust B”) can be selected. Conversely, it should be understood that adjustment A can be performed in the vertical direction and adjustment B can be performed in the horizontal direction.

  The third stage 115 shows the GUI 100 after the user has selected a different location on the image 195. Selection of a location on the image 195 can be performed by performing a gesture at a location on the device that has the touch or near-touch sensitive screen that displays the image, or by placing the cursor at the location of the image and providing input ( For example, by clicking on the cursor control device, pressing a hot key, etc.). In one embodiment, the user can select a location by tapping the finger at the location of the image or placing the finger at that location. In other embodiments, the location can be selected by performing other gestures. As shown, the user has selected a location on the image showing the mountain. Based on the selection of this new location, the application can use the overlay UI control 175 and the overlay UI control 180 (on the horizontal axis) corresponding to the UI control 135 for performing “Adjustment C” on the foliage region of the image. Display different sets (displayed as arrows in two directions along). The application also highlights UI control 135 to indicate that UI control 130 is activated. In some embodiments, the UI control 135 is highlighted by having a thicker border on the shape surrounding the UI control. In other embodiments, the UI control 135 is highlighted by being displayed in a lighter color than other UI controls displayed in the GUI. Similar to the second stage 110, this application also provides a set of overlay UI control 145 and overlay UI control 150 corresponding to the UI control 125 for performing “Adjust A” on the image, This is because the superposition UI control set does not depend on the content.

  The fourth stage 120 shows the GUI 100 after the user has started color adjustment on the image by providing direction input. This directional input can be provided by dragging the user's finger or dragging the cursor on a device having a touch or near-touch sensitive screen along a substantially horizontal direction. In this example, the user provides direction input by dragging the user's finger to the right of the image, as indicated by arrow 140. The application associates this direction input with an on-image UI control 180 for performing adjustment C on the image 195. As shown, the foliage region of image 195 (e.g., a region displaying a mountain) has been adjusted as indicated by the diagonal lines across that portion of image 195. The fourth stage 120 also removes the deactivated on-image UI control 145 and on-image UI control 150 from the display after the application associates the direction input with one set of on-image UI controls. The corresponding UI control 125 is also no longer highlighted. In some embodiments, UI control 145 and UI control 150 gradually disappear. The fourth stage 120 also shows that when the user moves the finger to a different location, the activated overlay UI control follows the finger. In other words, the displayed overlay UI control is animated to move across the displayed image, while the image remains stationary, as if it is attached to the finger. And substantially under the user's finger. Therefore, on-image UI controls are always around the location of the finger.

  In addition to content-dependent UI control, some embodiment applications also provide a new UI for adjusting the color balance of an image. The color balance adjustment is a color adjustment operation for making the colors in the image coincide with the colors of the scenery and the object at the moment when they are captured in the image. In some embodiments, the application provides a color balance UI control that allows the user to select a location on the image. The application then obtains the color value of the pixel corresponding to the selected location and changes the color value of the pixel to a set reference color (eg, gray, ideal skin color, etc.) To decide. Based on this color adjustment, the application generates a color space transformation that maps each color in the image to a different color. The application then applies the color space transformation to every pixel in the image.

  FIG. 2 shows an image editing application GUI 200 that provides color balance UI control for adjusting the color balance of the image. Specifically, FIG. 2 illustrates the color balance adjustment operation with four different stages 205, 210, 215, and stage 220. Each of these stages is described in more detail below.

  As shown by the first stage 205, the GUI 200 includes an image 195 and a selectable UI item 225 for activating the color balance adjustment UI control. The second stage 210 shows the GUI 200 after the user has selected the selectable UI item 225. Selection of the selectable UI item 225 may be performed by performing a gesture at a location on the device having a touch or near touch sensitive screen that displays the selectable UI item 225, or by placing the cursor on the selectable UI item 225, It can be performed by providing an input (eg, clicking on a cursor control device, pressing a hot key, etc.). In one embodiment, the user can select the selectable UI item 225 by tapping or placing a finger on the selectable UI item 225. In other embodiments, the selectable UI item 225 can be selected by performing other gestures. As shown, the user has selected the selectable UI item 225 by tapping or placing a finger on the selectable UI item 225.

  In some embodiments, the application provides a color balance UI control that is displayed (eg, overlaid) on the image to adjust the color balance of the image. In some of these embodiments, the color balance UI control is movable on the image, and the user moves the color balance UI control to the desired location (or drags) to change the location on the image. Allows you to specify.

  As shown in the second stage 210, as a result of selection of the selectable UI item 225, the color balance UI control 230 appears on the image. In this embodiment, the color balance UI control 230 has a square shape and has a crosshair in the center of the control. In other embodiments, the color balance UI control can be any other shape. A non-comprehensive list of these other shapes includes triangles, ellipses, rectangles, and the like. This shape can also be an irregular shape such as a hand or a flower. Furthermore, the cross hair may or may not exist in the center. Alternatively, objects other than crosshairs can be displayed in shapes such as circles, squares, single dots, or some other image used to show a small area within the image 195.

  The crosshairs of the color balance UI control 230 allow the user to specify a location on the image for adjusting the color balance of the image 195. As described above, after the user specifies a location for the color balance operation, the application obtains the color value of at least one pixel corresponding to the selected location and sets the color value of that pixel. The color adjustment to be changed to the selected reference color (for example, gray, ideal skin color, etc.) is determined. Based on this color adjustment, the application generates a color space transformation that maps each color in the image to a different color. The application then applies the color space transformation to every pixel in the image. As shown in the second stage 210, a color balance operation based on the selected location is performed on the image 195, as shown by the 45 degree diagonal lines across the image 195.

  As described above, the color balance UI control 230 of some embodiments is movable within the image 195 to allow the user to select different locations for color balance operations. In these embodiments, the user performs a gesture on the color balance UI control 230 (eg, by placing or tapping the user's finger) and dragging the finger to a different location, thereby causing the color balance. The UI control 230 can be relocated. The third stage 215 is a transient stage showing the GUI 200 after the user selects the color balance UI control 230 but before moving the color balance UI control 230 to a different location.

  The fourth stage 220 shows the GUI 200 after the user has moved the color balance UI control 230 to a different location. As shown, the user moves the color balance UI control 230 by dragging the user's finger in that direction toward the upper right corner of the image, as indicated by arrow 235. As a result of moving the color balance UI control 230 to a new location, the application readjusts the color balance of the image. Specifically, the application obtains the color value of at least one pixel corresponding to the new location selected by the user through the color balance UI control 230. The application then determines a color adjustment that changes the color of the selected location to the reference color. Based on this color adjustment, the application generates a color space transformation that maps each color in image 195 to a different color. The application then applies the color space transformation to all pixels in the image 195. As shown in this fourth stage 220, a color balance operation based on the newly selected location has been performed on the image 195, as indicated by a 135 degree diagonal line across the image 195.

  Some more detailed embodiments of the present invention are provided below. Many of these embodiments refer to sliders that are part of an image editing application. This application in some embodiments is a stand-alone application that runs on the device's operating system, while in other embodiments it is part of the operating system. Many of the following examples (FIGS. 3, 4, 6, 7, 9, 14, 15, 20, 21, 22, 23, 24, 31, 32, 33, 34, 35, 37, 39, 41, And the device on which the application is executed has a touch screen through which the user can interact with the image editing application. However, those skilled in the art will recognize the sliders and applications shown in these examples with respect to other embodiments implemented on a device using a cursor and cursor controller, or other input mechanism (eg, voice control). It will be appreciated that a cursor controller or other input device can be used to interact.

  Some more detailed embodiments of the invention are described in the following sections. Specifically, Section I describes various UI controls for adjusting the color value of only a portion of the image associated with the detected content type. Section II describes details of various embodiments that provide on-image UI control based on the type of content detected from a selected location on the image. Section III describes providing a set of color balance UI controls to adjust the color balance of the image. Finally, Section VI describes an electronic system that implements some embodiments of the present invention.

I. Color Control The image editing application of some embodiments provides a set of UI controls for adjusting the color value of only a portion of the image associated with a particular content type. In some embodiments, each UI control is for adjusting color values associated with different types of content in the image. When input is received through a particular UI control, the application automatically identifies the set of pixels in the image that are associated with the type of content controlled by that particular UI control. The application then adjusts only the color values of the identified set of pixels based on user input.

  In some embodiments, the set of UI controls includes different UI controls for adjusting color values associated with different content types. For example, one UI control is for adjusting the color value associated with the skin tone, and another UI control is for adjusting the color value associated with the sky color. There is yet another UI control for adjusting the color value associated with the foliage color.

  FIG. 3 illustrates an example GUI 300 of an image editing application of some embodiments that provides a set of UI controls for adjusting the color values of only a portion of the image associated with a particular content type. Shown with five different stages 305, 310, 315, 320 and stage 325. Each of these stages is described in more detail below.

  As shown in FIG. 3, the GUI 300 includes a thumbnail display area 330, a toolbar 340, an image display area 345, and a UI control display area 350. The thumbnail display area 330 displays a set of thumbnails of images in the album for the user to browse and select. An image thumbnail is a reduced size version of an image with reduced resolution. The user can select one or more images to be displayed in the image display area 345 by selecting a thumbnail corresponding to the image in the thumbnail display area 330.

  The toolbar 340 displays a set of selectable UI items that are associated with various types of image adjustment controls. The set of selectable UI items includes a selectable UI item associated with the set of image crop tools, a selectable UI item associated with the set of exposure adjustment tools, and a selection associated with the set of color adjustment tools. A possible UI item 342 is included. When the user selects one of these selectable UI items, a set of UI controls associated with the selected UI item is displayed in the UI control display area 350.

  Here, the image editing operation will be described with reference to the state of the GUI 300 between the five stages 305, 310, 315, 320, and 325. The first stage 305 shows the GUI 300 of the image editing application after the user has selected an image 355 from the album “Album 1” for display in the image display area 345. Selecting an image to display in the image display area 345 includes performing a gesture (eg, placing a finger, pointing at a location) on a device with a touch or near-touch sensitive screen that displays a thumbnail of that image. Or by tapping a finger) or by placing the cursor on the thumbnail of the image and providing input (eg, clicking on a cursor control device, pressing a hotkey, etc.) . As illustrated, the user selects an image 355 to be displayed in the image display area 345 by tapping a finger on the thumbnail 335. This selection is also indicated by a highlighting of the thumbnail 335 in the thumbnail display area 330. As a result of this selection, the image 355 is displayed in the image display area 345. The image 355 is a photograph of a person standing beside the sea with a mountain in the background.

  The second stage 310 shows the GUI 300 after the user has selected one of the selectable UI items in the toolbar 340. Selection of the selectable UI item 342 includes performing a gesture (eg, placing one or more fingers, pointing, or By tapping a finger) or by placing the cursor on a selectable UI item 342 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.) . As shown, the user has selected the selectable UI item 342 by tapping a finger on the selectable UI item 342 as indicated by the highlighting of the selectable UI item 342.

  As a result of selecting the selectable UI item 342, a set of color adjustment UI controls 352-358 is displayed in the UI control display area 350. Various embodiments provide various types of range-related UI controls (eg, dials, buttons, numeric fields, etc.) for adjusting the color of the image. In this embodiment, the color adjustment UI control is a range slider. The user can initiate various color adjustments to the image by sliding the knob of any one of those UI controls to different positions along the range slider. These sliders can provide a visual indication because the knob slides along the slider.

  Each of the color adjustment UI controls 352-358 is for initiating a different type of color adjustment for the image. For example, the color adjustment UI control 352 is for starting saturation adjustment for an image. The color adjustment UI control 354 is for starting the sky adjustment for the image, and adjusts only the color value that falls within the range of color values defined for the sky color. The color adjustment UI control 356 is for starting the foliage adjustment for the image, and adjusts only the color values that fall within the color value range defined for the foliage color. The color adjustment UI control 358 is for starting skin tone adjustment for an image, and adjusts only the color values that fall within the range of color values defined for the skin tone. In addition to the color adjustment UI controls 352-358, the GUI 300 also displays a selectable UI item 360 for invoking a settings menu.

  The third stage 315 displays the GUI 300 after the user has selected the color adjustment UI control 352 for the user to initiate saturation adjustment for the image 355, as indicated by the highlighting of the color adjustment UI control 352. Show. Selection of the color adjustment UI control 352 includes performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on the device having a touch or near-touch sensitive screen that displays the color adjustment UI control 352 Or by placing the cursor on the color adjustment UI control 352 and providing input (eg, clicking the cursor control device, pressing a hot key, etc.).

  The fourth stage 320 shows the GUI 300 after the user has started adjusting the saturation of the image 355 by moving the knob of the UI control 352 to the right as indicated by the arrow 362. In some embodiments, the user can move the thumb of UI control 352 by performing a gesture (eg, dragging the user's finger to a different location). As a result of this finger movement, the saturation of the colors in the image 355 is increased, as indicated by the diagonal lines across the image 355. In some embodiments, the application adjusts the color saturation of the image by adjusting the intensity of the colors in the image. In these embodiments, the application determines an algorithm or mathematical equation for adjusting the color values of the image based on user input to the color adjustment UI control 352. The application then applies the algorithm or mathematical equation to all pixel values in the image. In some embodiments, the application may use a color space (eg, a YCrCb color space, a YIQ color space, an IPT color space, etc.) that includes a luminance channel from a color space that defines the color values of the image (eg, RGB color space). ) To convert the color value of the image. A chrominance color channel (ie, a channel other than a luminance channel) can be represented in a polar coordinate system (eg, radial value and angle), in which the radial value represents color saturation, The angle represents various shades. When the color value of the image is converted into this new color space, the application can adjust the saturation of the image by adjusting the radius value of each color in the image.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 342 in the fourth stage 320 after the user performs saturation adjustment on the image 355.

  The fifth stage 325 shows the GUI 300 after the user has further adjusted the saturation of the image 355 by moving the knob of the UI control 352 further to the right as indicated by the arrow 364. As shown, the colors in the image 355 at this stage 325 are more saturated than the colors in the image at the fourth stage 320, as indicated by the denser diagonal lines throughout the image 355. Is shown increased.

  FIG. 3 shows an embodiment in which the saturation of the entire image is adjusted through the saturation UI control 352. The image editing application of some embodiments also provides a color adjustment UI control that adjusts the saturation of only a portion of the image. For example, this application adjusts the saturation of an image while maintaining the skin tone, known as natural saturation adjustment (ie, adjusting the saturation of all colors in the image except the skin tone) Yes) Saturation adjustment UI control can be provided.

  Various embodiments provide various techniques in providing UI controls for adjusting the natural saturation of colors in an image. In one approach, the application allows the user to switch between saturation adjustment and natural saturation adjustment by modifying application settings. FIG. 4 shows an example of such an approach. Specifically, FIG. 4 shows an embodiment for invoking and manipulating natural saturation UI controls, with six different stages 405, 410, 415, 420, 425, and stage 430.

  The first stage 405 is the same as the second stage 310 of FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. The second stage 410 shows the GUI 300 after the user has selected the selectable UI item 360 to bring up the application settings pop-up menu, as indicated by the highlighting of the selectable UI item 360. Selection of the selectable UI item 360 includes performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on the device that has a touch or near touch sensitive screen that displays the selectable UI item 360. Or by placing the cursor on the selectable UI item 360 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As illustrated, selection of the selectable UI item 360 causes a pop-up menu 435 to be displayed at the top of the UI control display area 350. Pop-up menu 435 provides a number of selectable items for modifying the settings of this application, such as selectable item 440 labeled “no skin tone” to toggle the setting of saturation UI control 352. Including.

  The third stage 415 shows the GUI 300 after the user has selected the selectable item 440 that is labeled “no skin tone”, as indicated by the highlighting of the selectable item 440. Selection of selectable item 440 includes performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device having a touch or near touch sensitive screen that displays selectable item 440. Or by placing the cursor on the selectable item 440 and providing input (eg, clicking a cursor control device, pressing a hot key, etc.). As a result of this selection, the saturation UI control 352 is for controlling the natural saturation adjustment for the image at this point. As described above, natural saturation adjustment adjusts only the saturation of the colors in the image 355 while maintaining the skin tone.

  The fourth stage 420 shows the GUI 300 after the user has selected the saturation UI control 352, as indicated by the highlighting of the saturation UI control 352. Selection of the saturation UI control 352 includes performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on the device having a touch or near-touch sensitive screen that displays the saturation UI control 352 Or by placing the cursor on the saturation UI control 352 and providing input (eg, clicking the cursor control device, pressing a hot key, etc.).

  The fifth stage 425 shows the GUI 300 after the user begins to increase the natural saturation of the image by moving the knob of the saturation UI control 352 to the right, as indicated by the arrow 470. In some embodiments, the user can move the knob of the saturation UI control 352 by dragging the user's finger (or dragging the cursor) to different locations along the range slider. . As a result of this finger movement, the saturation of the colors in the image 355 (other than the skin tone) is increased as shown by the diagonal lines across the image 355, while in the image 355 The human face area is maintained unaffected. In some embodiments, the application adjusts the color saturation of the image by adjusting the chrominance values of the image pixels. In these embodiments, the application determines an algorithm or mathematical equation for adjusting the pixel values of the image based on user input to the saturation UI control 352. The application then applies the algorithm or mathematical equation to all color values of the image, specified as skin tone.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 342 in the fourth stage 320 after the user performs saturation adjustment on the image 355.

  The sixth stage 430 shows the GUI 300 after the user has further increased the natural saturation of the image by moving the knob of the saturation UI control 352 further to the right as indicated by arrow 475. . As shown, the colors in the image 355 at this stage 430 are more saturated than the colors in the image at the fifth stage 425, as indicated by the denser diagonal lines across the image 355. Is shown increased, while the human face region in the image 355 remains unaffected.

  In some embodiments, the application selects default settings for the saturation UI control 352 based on the content of the image. For example, the application of some embodiments performs a face detection algorithm on the image and then determines settings for the saturation UI control 352 based on whether a face is detected on the image.

  Various embodiments use various techniques to identify skin tone. For example, the image editing application of some embodiments defines the skin tone as a range of color values in the opposite color space. The opposite color space is a color space that depends on three opposite color channels (eg, white / black channel, red / green channel, and yellow / blue channel), commonly known as the YCC color space. Some examples of opposite color spaces include the YIQ color space, the YcbCr color space, and the IPT color space. In the following description, the color space used to define the range of color values for various content types is described as a YCC color space. However, those skilled in the art should know that the YCC color space can be any one of the color spaces described above. In some of these embodiments, the color space used to define the skin tone range is different from the color space (eg, RGB color space) where the color values of the image are defined. In these embodiments, some embodiment applications are initially used to define a range for skin tone from the original color space (eg, RGB color space) in which the image color values are defined. The image is converted to the YCC color space. The application then identifies the pixels of the image that have color values that fall within the range of color values in the transformed color space defined for the skin tone. FIG. 5 illustrates an exemplary conversion from the RGB color space 505 to the YCC color space 500.

  As shown in FIG. 5, the RGB color space 505 has three axes: an axis 525 representing the range of red component values, an axis 515 representing the range of green component values, and an axis 535 representing the range of blue component values. Defined along. Therefore, any color in the image 355 can be represented by a location within the RGB color space 505.

  FIG. 5 also presents a YCC color space 500 that is used with some embodiments to define skin tone. As shown, this YCC color space has three axes: an axis 520 representing a range of red and green color values, an axis 530 representing a range of blue and yellow color values, and various shades of gray. Defined along a vertical axis 510 that represents a range (ie, a range of luminance values), the bottom location 540 represents black and the top location 550 represents white. The brighter the color, the farther the color appears from the bottom surface of the column 500 in this color space.

  With respect to color saturation, the greater the color saturation, the farther the color appears from the center of the cylinder. The center represents grayscale, its lowest point 540 does not contain color (ie, black), and highest point 550 represents that all colors are present at its maximum intensity (ie, white) All other points along the vertical line connecting this highest point to the lowest point represent various shades of gray. Each shade of gray represents that all colors are present equally, but the darker the gray, the lower the intensity of those colors. Also, the brighter the gray, the greater the intensity of those colors.

  In some embodiments, the skin tone is a cylinder 500 in the YCC color space between a red hue and a yellow hue having a specific saturation range and a specific intensity range, such as region 560. Defined within the region. Some embodiments also broadly define skin tone to cover a range of skin tones associated with various races and / or ethnicities. Such a broad definition eliminates the need for the user to manually define the skin tone. It should be apparent to those skilled in the art that some embodiments provide a threshold parameter to increase or decrease the range of skin tones perceived by the application.

  The application of some embodiments converts the image from the RGB color space 505 to the YCC color space 500 by mapping each color in the RGB color space 505 to another color in the YCC color space 500. For example, the color 545 in the RGB color space 505 is mapped to the color 570 in the YCC color space 500, and the color 555 in the RGB color space 505 is mapped to the color 580 in the YCC color space 500. By converting the image from the RGB color space to the YCC color space, the application allows colors that fall within the defined skin tone area 560 (eg, color 555) and the defined skin tone area 560 to be defined. It is possible to identify colors that do not fall within the range (eg, color 545).

  FIG. 4 illustrates an exemplary operation for adjusting the saturation of an image. Instead of adjusting the saturation of the image, some embodiments provide a skin tone UI control that allows the user to adjust the color temperature of the image. Color temperature is a characteristic of visible light reflected from an object in an image. The warmer light that strikes an object in the image creates a warmer tone (ie, more red and yellow) in the color of the object in the image, while the object in the image The cooler light that hits creates a cooler tone (ie, more blue and cyan) in the color of the object in the image. Therefore, adjusting the color temperature of the image means adding more red / yellow or more blue / cyan to the image.

  FIG. 6 shows such an embodiment. Specifically, FIG. 6 illustrates four different stages 605, 610, 615, and an operation for improving the skin tone by adjusting the color temperature of the image by operating the skin tone UI control. 620.

  The first stage 605 is the same as the second stage 310 of FIG. As shown, the user has selected a selectable UI item 342 from the toolbar 340 to adjust the color of the image 355. The second stage 610 shows the GUI 300 after the user has selected the skin tone UI control 356, as indicated by the highlighting of the skin tone UI control 356. Selection of the skin tone UI control 356 includes performing a gesture (eg, placing a finger, pointing, or fingering) at a location on the device that has a touch or near-touch sensitive screen that displays the skin tone UI control 356. By tapping) or by placing the cursor on the skin tone UI control 356 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.).

  The third stage 615 shows the GUI 300 after the user has started adjusting the skin tone of the image by moving the knob of the skin tone UI control 356 to the right as indicated by the arrow 625. In some embodiments, the user can move the knob of the skin tone UI control 356 by dragging the user's finger (or dragging the cursor) to a different location. As a result of this finger movement, the color values in the image 355 have been adjusted as indicated by the diagonal lines across the image 355. In some embodiments, the application adjusts the color temperature of the image to make the skin tone more pleasant. In some of these embodiments, the application fits within the defined skin tone range using the same defined skin tone range as described above with reference to FIG. Adjust the color value of the image to make the colors in the image more comfortable.

  In some embodiments, the application determines an algorithm or mathematical equation for adjusting the color values of the image based on user input to the skin tone UI control 356. The application then applies the algorithm or mathematical equation to every pixel of the image. In some embodiments, the application first converts the color value of the image to a different color space (eg, YCC color space) before applying the adjustment to the color value. As described above, the YCC color space is defined along the black / white color component, the red / green color component, and the yellow / blue color component. The conversion from the RGB color space to the YIQ color space first applies about 1/4 gamma to the color values in the RGB color space, and then applies a 3 × 3 matrix to convert the color values. This can be done by converting to the YIQ color space. In these embodiments, the application adjusts only the values along the red / green color component and the yellow / blue color component without changing the values along the white / black component. Adjust the color temperature of the image in the color space. In some embodiments, the application adjusts the color temperature of the image in the YIQ color space, which replaces the original color space of the image's color values (eg, RGB color space) instead of the YIQ. This is because applying the adjustment within the color space produces a more visually pleasing result. Furthermore, the applications in these embodiments convert the color values of the image back to the original color space after the adjustment is complete.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As shown in the drawing, a bar appears above the selectable UI item 342 in the third stage 615 after the user performs skin tone adjustment on the image 355.

  The fourth stage 620 shows the GUI 300 after the user has further adjusted the skin tone of the image by moving the knob of the skin tone UI control 356 further to the right as indicated by the arrow 630. As shown, the color of the image 355 at this stage 620 is further adjusted than the color in the image at the third stage 615, as indicated by the denser diagonal lines throughout the image 355. Indicated.

  In the above example illustrated by FIG. 6, some embodiment applications adjust the color temperature of the entire image in response to user input to the skin tone UI control 356. Or the application of other embodiment adjusts the color temperature of only the color tone of the skin in an image. In these embodiments, the application uses the same technique as described above to identify pixels in the image that have color values that fall within a predefined range of color values associated with the skin tone. use. The application then adjusts only the color temperature of the identified pixel within the image. The effect is the opposite of what natural saturation adjustment performs on the image as shown in FIG.

  As described above, the image editing application of some embodiments also provides UI controls for adjusting only a portion of the image (eg, the sky color or foliage color of the image). FIG. 7 illustrates an exemplary operation for adjusting or enhancing the sky color of an image at four different stages 705, 710, 715, and stage 720 without affecting the residual colors of the image.

  The first stage 705 is the same as the second stage 310 of FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. The second stage 710 shows the GUI 300 after the user selects the empty UI control 354, as indicated by the highlighting of the empty UI control 354. Selection of the empty UI control 354 includes performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on the device that has a touch or near touch sensitive screen that displays the empty UI control 354. Or by placing the cursor on an empty UI control 354 and providing input (eg, clicking a cursor control device, pressing a hot key, etc.).

  The third stage 715 shows the GUI 300 after the user begins to enhance the sky color of the image by moving the knob of the sky UI control 354 to the right as indicated by the arrow 725. In some embodiments, the user can move the thumb of the empty UI control 354 by dragging the user's finger to a different location (or dragging the cursor). As a result of this finger movement, the sky color in image 355 is highlighted, as shown by the diagonal lines across the sky region in image 355, while the rest in image 355 is Maintained unaffected. In some embodiments, the application enhances the sky color by adjusting the saturation of the sky color. In these embodiments, the application determines an algorithm or mathematical equation for adjusting the color values of the image based on user input to the sky UI control 354. The application then applies the algorithm or mathematical equation to the pixels of the image that have been identified as having a sky color. Instead of or in addition to adjusting the saturation, some other embodiment applications may also adjust the overall brightness for the sky colors in the image by adjusting the brightness and contrast of the sky colors in the image. Emphasis. In these embodiments, the application is identified as having a sky color using a single adjustment (specifying a single adjustment value) provided by the thumb movement of the user's sky UI control 354. Determine the adjustments for saturation, contrast, and lightness for the pixels of the current image.

For example, if the sky UI control allows the user to specify an adjustment value between -0.5 and 0.5, the application of some embodiments may have a contrast adjustment value (ie, gain value) The value is determined to be the larger of the value of 1 or the value obtained by adding 1 to the adjustment value. The application also determines the lightness adjustment value (ie, gamma) to be the absolute value of the adjustment value plus one. The application then applies these two adjustments to the image in the RGB color space. The formula for calculating the color value of each pixel using contrast and gamma is R = gain * (colorvalue) ^gamma , where the parameter “gain” represents the contrast adjustment value and the parameter “colorvalue” “Represents the color value of each pixel in the image (ie, the red channel value, the green channel value, or the blue channel value), and the parameter gamma represents the brightness adjustment value.

  In order to adjust the saturation of the image, the application of some embodiments uses the following formula: R = colorvalue + (colorvalue-ave) * (gamma-1) * 0.5 Where the parameter color value represents the color value of each pixel in the image (ie, the red channel value, the green channel value, or the blue channel value), and the parameter “ave” The average of the color component values (for example, the sum of (red channel value, green channel value, blue channel value) is divided by 3), and the parameter “gamma” represents the brightness adjustment value.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As shown in the figure, a bar appears above the selectable UI item 342 in the third stage 715 after the user performs the sky adjustment on the image 355.

  The fourth stage 720 shows the GUI 300 after the user has further enhanced the sky color of the image by moving the knob of the sky UI control 354 further to the right as indicated by the arrow 730. As shown, the sky color of the image 355 at this stage 720 is the sky color in the image at the third stage 715, as shown by the denser diagonal lines across the sky area in the image 355. Is shown with more emphasis than the color of, while the remainder of image 355 remains unaffected.

  Various embodiments use various techniques to identify the sky color. For example, the image editing application of some embodiments defines a sky color as a range of color values in the opposite color space. The opposite color space is a color space that depends on three opposite color channels (eg, white / black channel, red / green channel, and yellow / blue channel), commonly known as the YCC color space. Some examples of opposite color spaces include the YIQ color space, the YcbCr color space, and the IPT color space. In some embodiments, the opposite color space is preferred to define the sky color because of its uniform perceptual behavior, especially in the blue region. In the following description, the color space used to define the range of color values for various content types is described as a YCC color space. However, those skilled in the art should know that the YCC color space can be any one of the color spaces described above. In some of these embodiments, the color space used to define the sky color range is different from the color space (eg, RGB color space) in which the color values of the image are defined. In these embodiments, some embodiment applications are initially used to define a sky color range from the original color space (eg, RGB color space) in which the color values of the image are defined. The image is converted to the YCC color space. The application then identifies pixels of the image that have color values that fall within the range defined for the sky color within the new color space. FIG. 8 shows an exemplary conversion from the RGB color space 805 to the YCC color space 800.

  As shown in FIG. 8, the RGB color space 805 is along three axes: an axis 825 representing the range of red component values, 815 representing the range of green component values, and an axis 835 representing the range of blue component values. Defined. Therefore, any color in the image 355 can be represented by a location within the RGB color space 805.

  FIG. 8 also presents a YCC color space 800 that is used with some embodiments to define sky colors. As shown, this YCC color space has three axes: an axis 820 representing a range of red and green color values, an axis 830 representing a range of blue and yellow color values, and various shades of gray. Defined along the vertical axis 810 representing the range (ie, the range of luminance values), the bottom location 840 represents black and the top location 850 represents white. The brighter the color, the farther it appears from the bottom surface of the column 800 in this color space.

  In some embodiments, the sky color is defined within a region of a cylinder 800 in the YCC color space around a blue, red, and yellow region that has a specific black / white range, such as region 860. Is done. Therefore, this range of sky color values includes blue sky, clouds, sunrise, and sunset colors. It should be apparent to those skilled in the art that some embodiments provide a threshold parameter to increase or decrease the sky color range recognized by the application.

  The application of some embodiments converts the image from the RGB color space 805 to the YCC color space 800 by mapping each color in the RGB color space 805 to another color in the YCC color space 800. For example, the color 845 in the RGB color space 805 is mapped to the color 870 in the YCC color space 800, and the color 855 in the RGB color space 805 is mapped to the color 880 in the YCC color space 800. By converting the image from the RGB color space to the YCC color space, this application allows the color that falls within the defined sky color region 860 (eg, color 855) and the defined sky color region. A color that does not fall within the range of 860 (eg, color 845) can be identified.

  In addition to adjusting only the skin tone or blue sky color of the image, the image editing application of some embodiments also provides UI controls for adjusting only the foliage (ie green trees) in the image. To do. FIG. 9 illustrates an exemplary operation at four different stages 905, 910, 915, and stage 920 for adjusting or enhancing the foliage color of the image without affecting the residual color of the image.

  The first stage 905 is the same as the second stage 310 of FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. The second stage 910 shows the GUI 300 after the user selects the foliage UI control 358, as indicated by the highlight of the foliage UI control 358. Selection of the foliage UI control 358 can be done by tapping a finger at a location on the touch screen device that displays the foliage UI control 358 or by placing the cursor on the foliage UI control 358 and providing input. (E.g., clicking on a cursor control device, pressing a hot key, etc.).

  The third stage 915 shows the GUI 300 after the user begins to enhance the foliage color of the image by moving the knob of the foliage UI control 358 to the right as indicated by the arrow 925. In some embodiments, the user can move the knob of the foliage UI control 358 by dragging the user's finger (or dragging the cursor) to a different location. As a result of this finger movement, the color of the foliage in the image 355 (eg, the color of the mountain in the background of the image 355) is emphasized, as indicated by the diagonal lines across the mountain area in the image 355. On the other hand, the rest of the image 355 remains unaffected. In some embodiments, the application enhances the foliage color by adjusting the saturation, contrast, and brightness of the foliage color. In these embodiments, the application uses the single adjustment provided by the user's movement of the foliage UI control 358 knob to saturation the pixels of the image identified as having foliage color. Determine adjustments for contrast, brightness, and brightness. Similar to sky adjustment, this application determines an algorithm or mathematical equation for adjusting the color values of the image based on user input to the foliage UI control 358. The application then applies the algorithm or mathematical equation to the pixels of the image that have been identified as having foliage color. In addition to adjusting the color saturation, contrast, and brightness of foliage in the image, the application of some embodiments also moves the brownish color of dead grass towards a purer green color. In the same way, the color of the foliage also shifts.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 342 in the third stage 915 after the user performs the foliage adjustment on the image 355.

  The fourth stage 920 shows the GUI 300 after the user further enhances the foliage color of the image by moving the knob of the foliage UI control 358 further to the right as indicated by the arrow 930. As shown, the foliage color of image 355 at this stage 920 is shown in the image at third stage 915 as indicated by the denser diagonal lines across the mountain region in image 355. Although shown more emphasized than the foliage color, the rest of the image 355 remains unaffected.

  In the embodiments described above with reference to FIGS. 3, 4, 6, 7, and 9, the UI control is implemented as a range slider, and the user slides the knob along the linear slider, Provide input. As described above, various embodiments use various types of UI controls for adjusting color values on an image. In some embodiments, the range slider can be a different shape or geometry that allows the user to move the thumb anywhere in the area of the slider. In other embodiments, the application implements UI control as a dial. FIG. 10 shows an example of an operation that uses the dial UI control to adjust or enhance the foliage color of the image at four different stages 1005, 1010, 1015, and stage 1020.

  The first stage 1005 is the same as the first stage 905 of FIG. 9, except that the various UI controls for initiating various color adjustments on the image have different appearances. As described above, various embodiments use various types of UI controls for adjusting color values on an image. In this example, the application implements a set of UI controls 1052-1058 as a dial. Similar to the range slider, the different dials are for initiating different color adjustment operations on the image. For example, the UI control 1052 is for starting the saturation adjustment for the image, the UI control 1054 is for starting the color adjustment for the sky color of the image, and the UI control 1058 is for the image control. The UI control 1056 is for starting the color adjustment for the color of the foliage, and the UI control 1056 is for starting the color adjustment for the skin tone of the image. The user can initiate a specific color adjustment by selecting one of these dials and rotating the dial in a clockwise or counterclockwise direction.

  The second stage 1010 shows the GUI 300 after the user selects the foliage UI control 1058, as indicated by the highlight of the foliage UI control 1058. The selection of the foliage UI control 1058 is to perform a gesture (place a finger, point at it, or tap a finger) at a location on the device with a touch or near touch sensitive screen that displays the foliage UI controller 1058. Or by placing the cursor on the foliage UI control 1058 and providing input (eg, clicking a cursor control device, pressing a hot key, etc.).

  The third stage 1015 shows the GUI 300 after the user begins to enhance the foliage color of the image by providing a circular input to the foliage UI control 1058, as indicated by the arrow 1025. In some embodiments, the user provides circular input to the foliage UI control 358 by dragging the user's finger (or dragging the cursor) in a clockwise or counterclockwise direction. Can do. As a result of this finger movement, the color of the foliage in the image 355 (eg, the color of the mountain in the background of the image 355) is emphasized, as indicated by the diagonal lines across the mountain area in the image 355. On the other hand, the rest of the image 355 remains unaffected. In some embodiments, the application enhances the foliage color by adjusting the saturation, contrast, and brightness of the foliage color. In these embodiments, the application uses a single adjustment value specified by the user's movement on the foliage UI control 1058, for pixels of the image identified as having foliage color. Determine adjustments for saturation, contrast, and brightness. Similar to sky adjustment, this application determines an algorithm or mathematical equation for adjusting the color value of the image based on user input to the foliage UI control 1058. The application then applies the algorithm or mathematical equation to the pixels of the image that have been identified as having foliage color.

  The fourth stage 1020 shows the GUI 300 after the user has further enhanced the foliage color of the image by providing more circular input for the foliage UI control 1058, as indicated by the arrow 1030. As shown, the foliage color of the image 355 at this stage 1020 is shown in the image at the third stage 1015 as indicated by the denser diagonal lines across the mountain area in the image 355. Although shown more emphasized than the foliage color, the rest of the image 355 remains unaffected.

  FIG. 10 above shows one alternative implementation for UI control. Some other example implementations for UI controls include buttons (ie, the user can specify the degree to which adjustments are applied depending on how long the button is pressed), numeric fields (ie, high numeric values) This indicates that more adjustments to the image are applied).

  Various embodiments identify the foliage color using various techniques. For example, the image editing application of some embodiments defines the foliage color as a range of color values in the opposite color space. The opposite color space is a color space that depends on three opposite color channels (eg, white / black channel, red / green channel, and yellow / blue channel), commonly known as the YCC color space. Some examples of opposite color spaces include the YIQ color space, the YcbCr color space, and the IPT color space. In some embodiments, the opposite color space is preferred for defining sky colors because of its uniform perceptual behavior. In the following description, the color space used to define the range of color values for various content types is described as a YCC color space. However, those skilled in the art should know that the YCC color space can be any one of the color spaces described above. In some of these embodiments, the color space used to define the skin tone range is different from the color space (eg, RGB color space) where the color values of the image are defined. In these embodiments, the application of some embodiments first defines the foliage color range from the native color space (eg, RGB color space) in which the image color values are defined. Convert the image to the YCC color space used. The application then identifies pixels of the image that have color values that fall within the range defined for the foliage color within the new color space. FIG. 11 illustrates an exemplary conversion from the RGB color space 1105 to the YCC color space 1100.

  As shown in FIG. 11, the RGB color space 1105 is along three axes: an axis 1125 representing a range of red component values, 1115 representing a range of green component values, and an axis 1135 representing a range of blue component values. Defined. Therefore, any color in the image 355 can be represented by a location within the RGB color space 1105.

  FIG. 11 also presents a YCC color space 1100 that is used with some embodiments to define foliage colors. As shown, this YCC color space has three axes: an axis 1120 representing a range of red and green color values, an axis 1130 representing a range of blue and yellow color values, and various shades of gray. Defined along a vertical axis 1110 representing a range (ie, a range of luminance values), the bottom location 1140 represents black and the top location 1150 represents white. The brighter the color, the farther the color appears from the bottom surface of the column 1100 in this color space.

  In some embodiments, the foliage color is defined within a region of a cylinder 1100 in the YCC color space around a green and yellow region having a specific black / white range, such as region 1160. . It should be apparent to those skilled in the art that some embodiments provide a threshold parameter to increase or decrease the foliage color range recognized by the application.

  The application of some embodiments converts the image from the RGB color space 1105 to the YCC color space 1100 by mapping each color in the RGB color space 1105 to another color in the YCC color space 1100. For example, color 1145 in RGB color space 1105 is mapped to color 1170 in YCC color space 1100, and color 1155 in RGB color space 1105 is mapped to color 1180 in YCC color space 1100. By converting the image from the RGB color space to the YCC color space, this application allows the color that falls within the defined foliage color region 1160 (eg, color 1155) and the defined foliage color. It is possible to specify a color that does not fall within the range of the area 1160 (for example, the color 1145).

  The GUI of the image editing application described in the above figure is shown under the assumption that the device on which the image editing application is executed has a sufficiently large screen to display the GUI. However, some of the devices on which image editing applications are executed have limited screen sizes to display UI items in a way that they are displayed in a larger screen on devices with larger items. There may be. Also, a larger screen of a larger device may be considered limited if the device is held in a different orientation (eg, portrait). In some embodiments, the image editing application adapts to this limited screen space by displaying different sets of different UI items at different time instances.

  FIG. 12 conceptually illustrates an example of invoking a set of color adjustment tools through the GUI 1200 of some embodiments, with six different stages 1201-1206. As illustrated, the GUI 1200 includes a control pane 1215, an image display area 1216, a thumbnail display area 1220, and a tool navigation pane 1225.

  The image display area 1216 is similar to the image display area 345 described above with reference to FIG. 3 in that the image display area 1216 displays an image that the user desires to edit using the image editing application. It is. The thumbnail display area 1215 is similar to the thumbnail display area 330 in that the thumbnail display area 1215 displays a collection of thumbnails of images that can be used for editing using an image editing application. In some embodiments, the image editing application allows the user to scroll up and down or left and right to display different thumbnails. The image editing application also provides a visual display (eg, highlighting) on the thumbnail so that the thumbnail is selected and the corresponding image is displayed in the display area. Show. Furthermore, the image editing application of some embodiments provides a visual display (eg, a toolbox icon) on the thumbnail to indicate that the corresponding image is being edited.

  Tool navigation pane 1225 displays different sets of selectable or controllable UI items at different time instances. In some embodiments, the set of user interface items currently displayed in the tool navigation pane 1225 is determined in response to user interaction with the set of UI items previously displayed in the tool navigation pane 1225. Is done. For example, when a user touches a particular UI item that represents a set of editing tools, the set of UI items for that set of editing tools associated with that particular UI item is displayed in that particular UI in the tool navigation pane 1225. Replaces a set of UI items, including items. As shown, the tool navigation pane 1225 displays UI items 1230 and a set 1292 of other UI items.

  The image editing application can replace the first set of UI items displayed in the tool navigation pane 1225 with a second set of UI items in a number of different ways. For example, the image editing application slides out the first set and slides in the second set. The image editing application can superimpose the first set and the second set when the second set is slid in. Furthermore, the direction of sliding by the first set and the second set can be any direction (up, down, left, right, diagonal, etc.).

  An exemplary operation for invoking a color adjustment tool through the GUI 1200 will now be described. In the first stage 1201, the image editing application displays an image 1217 in the image display area 1216. The image 1217 corresponds to the thumbnail 1235 displayed in the thumbnail display area 1220. That is, this image editing application displays the image 1217 in the image display area 1216 in response to the user's selection of the thumbnail 1235.

  The second stage 1202 shows the GUI 1200 after the user begins to change the UI items displayed in the navigation pane 1225. In some embodiments, selection of a UI item 1230, which is an icon representing a toolbox, represents a UI item displayed in the navigation pane 1225 representing a set of image editing tools including a color adjustment tool. Change to a set. As shown, the user has selected UI item 1230 as indicated by the highlighting of UI item 1230.

  The third stage 1203 is a transitional stage showing the animation provided by the GUI 1200 when the application changes the set of UI items displayed in the tool navigation pane 1225. As described above, the application of some embodiments provides an animation that replaces one set of UI items in the tool navigation pane 1225 with another set of UI items. In this example, the GUI displays an animation where a new set of UI items 1226 replaces the set of UI items 1292 that were displayed in the tool navigation pane 1225. As shown, the UI item set 1292 slides in from the bottom of the UI item 1230 to the right and covers the UI item set 1292 in the process.

  The fourth stage 1204 shows the GUI 1200 after the UI item set 1226 has been completely replaced by the UI item set 1292. As shown, the tool navigation pane 1225 displays a new set of UI items 1226. The set of UI items 1226 in this example includes six UI items 1240, 1245, 1246, 1247, 1250, and UI items 1255. UI item 1245 represents an on-image crop and angle correction tool. UI item 1246 represents a set of exposure adjustment tools. UI item 1247 represents a set of color adjustment tools. UI item 1250 represents a brush on image tool. The UI item 1255 represents an image effect tool. At this stage, the user selects UI item 1247 to invoke the color adjustment tool.

  Furthermore, the UI item 1230 is also replaced with the UI item 1240 by replacing the set of UI items. The UI item 1240 is for replacing a new set of UI items 1226 with another set of UI items in the tool navigation pane 1225. UI item 1240 is also for the user to touch to indicate that the user does not want to use the editing tool represented by the set of UI items.

  The fourth stage 1204 also indicates that the user has begun calling the color adjustment tool by selecting the UI item 1247. As shown, the user has tapped a finger on the UI item 1247 as indicated by the highlighting of the UI item 1247.

  The fifth stage 1205 responds to the user's selection of the UI item 1247 with the animation provided by the GUI 1200 when the application replaces the set of UI items 1226 in the tool navigation pane 1225 with the set of color adjustment tools 1260. This is a transitional stage. As shown, the UI item set 1226 begins to slide left into the UI item 1240, exposing a new set of color adjustment tools 1260 during the process.

  The sixth stage 1206 shows the GUI 1200 after the set of color adjustment tools 1260 has completely replaced the set of UI items 1226 in the tool navigation pane 1225. As shown in the figure, this set of color adjustment tools includes a UI item 1271 for calling the saturation UI control, a UI item 1272 for calling the empty UI control, a UI item 1273 for calling the foliage UI control, and a skin color UI. It includes a UI item 1274 for invoking controls, and a UI item 1275 for invoking a set of color balance UI controls. Furthermore, the replacement of the UI item set 1226 also causes the UI item 1240 to change to a UI item 1262 that looks like a color adjustment icon. The user can select UI item 1262 to exit (or deselect a set of color adjustment tools).

  FIG. 12 shows an example of calling a set of color adjustment tools through the GUI 1200. FIG. 13 illustrates an example of deselecting a set of color adjustment tools through the GUI 1200 in six different stages.

  The first stage 1301 is the same as the sixth stage 1206 of FIG. 12 except that the user has selected the UI item 1262 in the tool navigation pane 1225 to deselect the color adjustment tool set 1260. is there.

  The second stage 1302 is a transitional stage showing the animation provided by the GUI 1200 when the application replaces the set of color adjustment tools 1260 with a set of UI items 1226. As shown, the UI item set 1226 begins to slide into the tool navigation pane 1225 from below the UI item 1262 and covers the color adjustment tool set 1260 in the process.

  The third stage 1303 shows that the UI item set 1226 has completely replaced the color adjustment tool set 1260. Furthermore, UI item 1262 has also been changed to UI item 1240. As described above, the UI item 1240 is for the user to select to indicate that the user does not want to use the editing tool represented by the set of UI items.

  The fourth stage 1304 shows the GUI 1200 when the user selects the UI item 1240. The fifth stage 1305 is a transitional stage that shows an animation in which the application replaces the UI item set 1226 with the UI item set 1292 in response to the user selecting the UI item 1240. As shown, UI item set 1226 begins to slide left into UI item 1240, exposing UI item set 1292 during the process.

  The sixth stage 1306 shows the GUI 1200 after the set of UI items 1292 has completely replaced the set of UI items 1226 in the tool navigation pane 1225. Furthermore, the UI item 1240 is changed to a UI item 1230.

  FIG. 14 shows an example of performing an empty adjustment operation on an image through a GUI 1200 with six stages 1405, 1410, 1415, 1420, 1422, and stage 1424. As shown in FIG. 14, the GUI 1200 has a thumbnail display area 1220 that includes a set of thumbnail versions of an image for the user to view and select. By scrolling left or right on the thumbnail display area 1220 (for example, by placing a finger on the thumbnail display area 1220 and dragging the finger to the left or right), the user can view more thumbnail images. You can browse the street. The GUI 1200 also includes an image display area 1216 for displaying a set of images, a tool navigation pane 1225, and a UI control display area 1450.

  The first stage 1405 shows the GUI 1200 after the user has selected the image 1470 displayed in the image display area 1216. Selecting an image to be displayed in the image display area 1216 is performed by performing a gesture (place a finger, pointing, Or by tapping a finger) or by placing the cursor on the thumbnail 1475 of the image and providing input (eg, clicking on a cursor control device, pressing a hotkey, etc.) it can. This selection is also indicated by a highlighting of the thumbnail 1475 in the thumbnail display area 1220.

  The second stage 1410 shows the GUI 1200 after the user has started scrolling thumbnail images within the thumbnail display area 1220. As described above, the user can browse a larger number of thumbnail images by scrolling left or right on the thumbnail display area 1220. This scrolling is performed on a device with a touch or near-touch sensitive screen that displays a thumbnail display area 1220 (eg, by placing, pointing, or tapping the finger) and moving the finger. This can be done by dragging left or right. As shown, the user scrolls the thumbnail image by dragging his finger to the left.

  The third stage 1415 shows the GUI 1200 after the user has finished scrolling the thumbnail images in the thumbnail display area 1220 and selected a new image to be displayed in the image display area 1216. As illustrated, the thumbnail display area 1220 includes a thumbnail 1435 that has not been displayed in the thumbnail display area 1220 in the first stage 1405 at this time. As shown in the third stage 1415, the user has also selected an image 1455 displayed in the image display area 1216. Selecting an image to be displayed in the image display area 1216 includes performing a gesture (place a finger, pointing, Or by tapping a finger) or by placing the cursor on the thumbnail 1435 of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.) . This selection is also indicated by the highlighting of the thumbnail 1435 in the thumbnail display area 1220.

  As shown in the third stage 1415, the user has also chosen to perform color adjustment on the image 1455, as indicated by the highlighting of the color adjustment icon 1442. As a result, a set of selectable UI items 1452-1458 are displayed in the tool navigation pane 1225. Each of the selectable UI items 1452-1458 is for initiating a different color adjustment for the image. For example, the selectable UI item 1452 is for starting saturation adjustment for an image, the selectable UI item 1454 is for starting empty adjustment for an image, and the selectable UI item 1458 is The selection UI item 1456 is used to start skin tone adjustment related to an image. These color adjustments are the same as the various color adjustments described above with reference to FIGS. 3, 4, 6, 7 and 9.

  The fourth stage 1420 shows the GUI 1200 after the user selects a selectable UI item 1454 in the tool navigation pane 1225. Selection of selectable UI item 1454 involves performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device having a touch or near-touch sensitive screen that displays selectable UI item 1454 Or by placing the cursor on selectable UI item 1454 and providing input (eg, clicking on a cursor control device, pressing a hotkey, etc.). As shown, the user has selected the selectable UI item 1454 by tapping a finger on the selectable UI item 1454, as indicated by the highlighting of the selectable UI item 1454. As a result of this selection, an empty adjustment UI control 1464 appears in the UI control display area 1220.

  Various embodiments provide various types of UI controls for adjusting the color of the image. In this embodiment, the empty adjustment UI control is a range slider. The user can adjust the sky color in the image by sliding the knob of the sky UI control 1464 to a different position along the range slider.

  The fifth stage 1422 is a transitional stage indicating that the user has selected the knob of the sky UI control 1464 but has not started adjusting the sky color in the image. The sixth stage 1424 shows the GUI 1200 after the user has started adjusting the sky color of the image by moving the knob of the sky UI control 1464 to the right as indicated by the arrow 1460. In some embodiments, the user can move the knob of the empty UI control 1464 by dragging the user's finger (or dragging the cursor) to a different location. As a result of this finger movement, the sky color in image 1455 is enhanced, as shown by the diagonal lines across the sky region in image 355, while the rest of image 1455 is affected. It is maintained without receiving. In some embodiments, the application adjusts the sky color in the image in the same manner as described above with reference to FIG.

  FIG. 14 shows an embodiment in which an empty adjustment operation is executed through the GUI 1200. The user selects other selectable items 1452 to 1458 in the same manner and selects other types of color adjustment operations (for example, saturation adjustment, natural saturation adjustment, foliage adjustment, skin tone adjustment, etc.). Can be executed. FIG. 15 provides an example of performing foliage adjustment operations through a GUI 1200 with four stages 1505, 1510, 1515, and stage 1520.

  The first stage 1505 is the same as the first stage 1405 in FIG. As illustrated, the user has selected an image 1455 displayed in the image display area 1216. Second stage 1510 shows GUI 1200 after the user selects selectable UI item 1458 in tool navigation pane 1225. Selection of selectable UI item 1458 is performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on the device having a touch or near-touch sensitive screen that displays selectable UI item 1458. Or by placing the cursor on selectable UI item 1458 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown, the user has selected the selectable UI item 1458 by tapping a finger on the selectable UI item 1458 as indicated by the highlighting of the selectable UI item 1458. As a result of this selection, a foliage adjustment UI control 1564 appears in the UI control display area 1450.

  The third stage 1515 is a transitional stage indicating that the user has selected the knob of the foliage UI control 1564 but has not started adjusting the foliage color in the image. The fourth stage 1520 shows the GUI 1200 after the user has started adjusting the foliage color of the image by moving the foliage UI control 1564 knob to the right as indicated by the arrow 1560. In some embodiments, the user can move the knob of the foliage UI control 1564 by dragging the user's finger (or dragging the cursor) to a different location. As a result of this finger movement, the color of the foliage in image 1455 is enhanced, as shown by the diagonal lines across the mountain region in image 1455, while the rest of image 1455 is Maintained unaffected. In some embodiments, the application adjusts the foliage color in the image in the same manner as described above with reference to FIG.

  FIG. 16 conceptually illustrates a process 1600 for adjusting a color value of a portion of an image based on a user input to a color adjustment UI control. In some embodiments, this process is performed by an image editing application. The process begins by performing content analysis on the image (at 1605). Various embodiments perform various content analysis on images. For example, an application of some embodiments performs face detection analysis on an image to determine whether the image includes any person's face.

  The process then defines (at 1610) a set of parameters for the set of image adjustment UI controls, as needed, based on the results from the content analysis. In some embodiments, at least one of the adjustment UI controls can be configured to perform various adjustment operations on the image based on application settings. In some of these embodiments, the application defines default settings for the image based on results from content analysis. For example, if the application detects that the image contains a human face, the application may adjust the natural saturation (i.e., maintain the skin tone) rather than adjusting the saturation of the entire image. The saturation UI control is configured to perform the following:

  The process then receives user input (at 1615) for a particular UI control. As described above, part of the UI control is for adjusting the color values of various parts of the image. In some embodiments, the application includes a set of predefined color value ranges for those UI controls. For example, the application may include a range of color values associated with a sky color for a sky UI control, a range of color values associated with a foliage color for a foliage UI control, and Defines a range of color values associated with skin tone for saturation UI control and skin tone UI control. In some of these embodiments, the color values of the image are defined in a color space that is different from the color space used to define the range of these color values. Thus, for a particular UI control, after user input is received, this process converts (at 1620) the image color values into the color space used to define the range of color values. .

  The process then identifies (at 1625) the set of pixels in the image that have color values that fall within the range of color values associated with that particular UI control. In some embodiments, color value ranges associated with various content types are stored in the media storage device. In these embodiments, the process obtains a range of color values associated with a particular UI control from the media storage device and identifies pixels having color values that fall within that range of color values. In other embodiments, the range of color values is defined in executable code for performing color adjustment. In these other embodiments, this particular operation is performed simultaneously as the process performs color adjustments on the image.

  Based on the user input, the process then adjusts (at 1630) the color value of the set of pixels identified in operation 1625. Process 1600, in some embodiments, the process first identifies a set of pixels in the image that fall within the range of color values, and then performs an adjustment on the identified set of pixels. Indicates. However, in some other embodiments, the process identifies pixels for adjustment at the same time as performing the adjustment operation. Specifically, for each pixel in the image, this process iteratively determines whether the color value of that pixel falls within the color value range, and only if the color value is within the range. Execute the adjustment.

  In some embodiments, prior to adjusting the color values of the image, the application generates an algorithm or mathematical equation that can be applied to the color values of the image. The application then applies the algorithm or mathematical equation to the color value of each pixel in the image identified in operation 1625.

  The process then determines (at 1635) whether more user input is received for the set of UI controls. If more inputs are received, the process returns to 1625 to identify another set of pixels. This process repeats operations 1625-1635 until no more input is received for the set of UI controls. The process then ends.

  Various embodiments use various techniques to identify the set of pixels at operation 1625 of FIG. In some embodiments, the application identifies a set of pixels using the techniques described above with reference to FIGS.

  In some embodiments, the application performs only one type of adjustment (eg, saturation adjustment) to the color values of the image in response to user input for a particular UI control. As described above, some other embodiment applications perform more than one type of adjustment to the color values of an image in response to a single user input for a particular UI control. For example, if the user provides input to the sky UI control 354, some embodiments of the application perform saturation adjustment, contrast adjustment, and brightness adjustment for the color values of a portion of the image. In these embodiments, the application uses a single user input to adjust an adjustment value for adjusting contrast, an adjustment value for adjusting saturation, and an adjustment for adjusting brightness for the image. The values are determined and these individual adjustment values are applied to the color values of the image.

  FIG. 17 conceptually illustrates a process 1700 for performing several types of adjustments on a limited range of color values in an image based on a single user input to a color adjustment UI control. Show. In some embodiments, this process is performed during operations 1620-1630 of FIG. Process 1700 begins by using (at 1705) user input to a UI control to determine a contrast adjustment value for adjusting the contrast of the image. In some embodiments, the contrast of an image represents a range of color values within the image. In other words, an image with higher contrast has a wider range of color values than an image with lower contrast. Therefore, increasing the contrast of an image involves extending the range of color values in the image (ie, increasing the difference between the bright and dark color values of the image) and increasing the contrast of the image. Reducing the range of color values in the image (ie, reducing the difference between the bright and dark color values of the image). Hence, the contrast adjustment value determines the extent to which the application adjusts the range of color values in the image.

  The process then uses the same user input to the UI control (at 1710) to determine a saturation adjustment value for adjusting the saturation of the image. In some embodiments, the saturation of an image represents the intensity of color values in the image. Therefore, increasing the saturation of an image involves increasing the intensity of the color value of that image. Therefore, the saturation adjustment value determines the degree to which the application adjusts the intensity of the color values in the image.

  The process then uses the same user input for a particular UI control (at 1715) to determine a brightness adjustment value for adjusting the brightness of the image. In some embodiments, the brightness of an image represents the brightness or brightness within the image. Therefore, increasing the brightness of an image involves increasing the luminance value of the color values in the image. Therefore, the brightness adjustment value determines the degree to which the application adjusts the brightness or brightness value of the color values in the image.

  After determining three different adjustment values for adjusting the contrast, saturation, and lightness of the image, the process converts the contrast adjustment value, saturation, to the color value of the set of pixels identified in operation 1625. Apply the adjustment value and the brightness adjustment value (at 1720). In some embodiments, the application generates a set of algorithms or a set of mathematical equations based on various adjustment values to apply to the color values of the image. The application then applies the set of algorithms, or set of mathematical equations, to the color values of each pixel in the image identified in operation 1625.

  The process then converts (at 1725) the color value of the image back to its original color space, if necessary. In some embodiments, after identifying a set of pixels for adjustment in a transformed color space (eg, YCC color space), the application may use the image color in the transformed color space. Perform adjustments to values. For example, some embodiments of the application perform foliage adjustments within the YCC color space. In some embodiments, it is preferable to perform foliar adjustments within a particular YCC color space (eg, IPT color space) because of its uniform perceptual behavior. For example, the hue of a color can be shifted toward green without making the color yellow. In these embodiments, once the adjustment operation is complete, the process needs to convert the color value of the image back to its original color space. In other embodiments, after identifying a set of pixels for adjustment, the application performs adjustments to the color values of the image within its native color space. For example, the application of some embodiments performs a sky adjustment within the original color space (eg, RGB color space) for that color value. In these embodiments, the process need not perform this conversion step 1720. The process then ends.

  Some embodiments perform all adjustments in the same color space, while other embodiments perform color adjustments in different color spaces.

  FIG. 18 illustrates an image editing application 1800 of some embodiments that provides a set of UI controls for adjusting color values for only a portion of the image. In some embodiments, image editing application 1800 performs process 1600 and process 1700. As shown in FIG. 18, the image editing application 1800 includes an image processor 1840, a set of content analysis modules 1845, a color space conversion module 1815, a pixel separation module 1810, and a color adjustment engine 1820.

  When UI module 1805 receives user input to invoke a set of color adjustment UI controls, UI module 1805 passes the request to image processor 1840. Image processor 1840 obtains an image from media storage device 1850 and requests a set of content analysis modules 1845 to perform a series of content analysis on the image. In some embodiments, media storage device 1850 is a set of file folders configured by image editing application 1800 and stored on a particular set of storage devices. The storage device may include a boot drive of the electronic device on which the application runs, a different partition of the disk, a separate internal or external hard drive, a flash drive, an SD card, etc.

  The set of content analysis modules 1845 performs the set of content analysis modules on the image. Various embodiments perform various content analysis on images. For example, the application of some embodiments performs a face detection analysis to determine whether a human face appears on the image. The set of content analysis modules 1845 then sends the results of the analysis back to the image processor 1840. Based on the results of the analysis, the image processor 1840 of some embodiments defines a set of parameters for a set of UI controls. For example, if a human face is detected on the image, the image processor 1840 configures the saturation adjustment UI control to be associated with a natural saturation adjustment operation.

  When UI module 1805 receives user input for one of the UI controls to initiate a color adjustment operation, UI module 1805 passes the user input to image processor 1840. The image processor 1840 then requests the pixel separation module 1810 to identify a set of pixels in the image that have color values that fall within the range of color values associated with the UI control that receives the user input. . In some embodiments, the color values of the image are in a color space (eg, RGB color space) that is different from the color space (eg, YCC color space) used to define the range of those color values. Defined. In these embodiments, before requesting the pixel separation module 1810 to identify a set of pixels, the image processor 1840 determines the color values of the image and the colors used to define the range of those color values. Requests the color space conversion module 1815 to convert to space.

  After the set of pixels is identified, the image processor 1840 sends the image and information about the identified pixels to a color adjustment engine 1820 for performing a series of color adjustments on the image. The color adjustment engine 1820 applies various color adjustments to the identified pixels in the image. In some embodiments, the color adjustment engine 1820 performs a process 1700 to apply adjustments to identified pixels of the image. As shown, the color adjustment engine 1820 includes a contrast adjustment engine 1825 for applying contrast adjustment to the image, a brightness adjustment engine 1830 for applying brightness adjustment to the image, and a saturation adjustment for applying the saturation adjustment to the image. A saturation adjustment engine 1835 is included. These are merely exemplary color adjustments that can be performed on an image, and the color adjustment engine 1820 provides additional colors for performing additional types of color adjustments on the image. It should be apparent to those skilled in the art that a conditioning engine can be included. For example, the brightness adjustment engine 1830 uses a user input to determine a brightness adjustment value for adjusting the brightness of the image, and the contrast adjustment engine 1825 uses the user input to adjust the contrast of the image. The saturation adjustment engine 1835 uses a user input to determine a saturation adjustment value for adjusting the saturation of the image. Various adjustment engines 1825-1835 then apply this lightness adjustment, contrast adjustment, and saturation adjustment to the image to produce an edited image.

  In some embodiments, the color adjustment engine 1820 adjusts the color values of the image within the converted color space. In these embodiments, the image processor 1840 sends the adjusted image to the color space conversion module 1815 and converts the color value of the image back to its original color space. The image processor 1840 then sends the adjusted image to the media storage device and to the UI module 1805 for display.

II. On-Image Controls As described above with reference to FIG. 1, the image editing application of some embodiments provides a set of UI controls that are superimposed on the image for editing the image. Each set of on-image UI controls is associated with a different adjustment. In some embodiments, the user can activate a set of superimposed UI controls (also referred to as on-image UI controls) by selecting a location on the image. In these embodiments, when the user selects a particular location on the image, the application performs a series of analyzes to determine the specific content type associated with the selected location on the image. To do. The application then displays a different set of UI controls associated with different color adjustment functions for application to the image, as the selected location is associated with a different content type. In some embodiments, the set of superimposed UI controls is opaque or transparent. Some embodiments of these on-image UI controls are described in Section II. A and section II. B will be described in more detail.

  FIG. 19 conceptually illustrates a process 1900 for providing a set of on-image UI controls for adjusting color values of an image based on a user selection of a location on the image. In some embodiments, this process is performed by an image editing application. The process begins by receiving (at 1905) a selection of a location on the image. The process then obtains (at 1910) the color value of the set of pixels corresponding to the selected location on the image. In some embodiments, this set of pixels is a pixel located near the selected location.

  The process then performs (at 1915) a series of analyzes on the acquired color values. In some embodiments, each analysis determines whether the selected location is associated with a particular content type based on the obtained color values. For example, one analysis is for determining whether a selected location is associated with skin tone, and another analysis is for determining whether a selected location is associated with the sky. It is for judging.

  Based on the results from the series of analyses, this process provides (at 1920) a set of UI controls to control the set of image editing operations. In some embodiments, this set of UI controls are on-image controls that are displayed on the image. In some of these embodiments, the set of on-image UI controls is a direction-dependent UI control. The user can initiate various image editing operations on the image by providing direction input to the image.

  The process then determines (at 1930) whether another location is selected on the image. If another location is selected, the process returns to 1910 to obtain a color value for another set of pixels corresponding to the newly selected location. This process repeats operations 1910-1925 until no more locations are selected. The process then ends.

A. On-Image Color Control In some embodiments, the set of overlay UI controls provided by the image editing application is for applying various color adjustments to the image. FIG. 20 is an example of an image editing application of some embodiments that provides a set of on-image UI controls for applying color adjustments to an image based on a location on the image selected by the user. The GUI 300 is shown. Specifically, FIG. 20 illustrates the operation of activating various sets of on-image UI controls at four different stages 2005, 2010, 2015, and stage 2020. Each of these stages is described in more detail below.

  The first stage 2005 is the same as the second stage 310 of FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. The UI control display area 350 includes a UI control 352 for adjusting the saturation of the image, a UI control 354 for adjusting the sky color in the image, and a UI control 358 for adjusting the color of the foliage in the image. , And a UI control 356 for adjusting the skin tone in the image. The second stage 2010 shows the GUI 300 after the user has selected a location on the image 355. As illustrated, the user has selected a location on the image where a human face is displayed. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As described above, after the user selects a location on the image, some embodiments of the image editing application may use a series of steps to determine the type of specific content associated with the selected location. Perform analysis. Various embodiments use various techniques to determine the type of content associated with the selected location. The application of some embodiments initially defines a range of different color values associated with different types of content. For example, the application of some embodiments may include color value ranges associated with skin tones, color value ranges associated with sky colors, and color values associated with foliage colors. Define the range. In some embodiments, the application defines a range of color values in the same manner as described above with reference to FIGS. The application then determines whether the color value of the set of pixels corresponding to the selected location in the image falls within the color value range associated with the particular content type, and the identification Displays a set of on-image UI controls associated with the content type.

  Various embodiments implement a set of on-image UI controls in various ways. For example, some embodiments of the application implement a set of on-image UI controls as direction-dependent UI controls. In some of these embodiments, the application displays a directional arrow for each UI control to guide the user to initiate a color adjustment operation associated with the UI control. The user initiates various color adjustment operations by providing various direction inputs. Based on the direction of this direction input, the application performs a specific color adjustment on the image. In the second stage 2010, since the user has selected a place on the image where the person's face is to be displayed, the color value of the pixel corresponding to the selected place is related to the skin tone. Judge that there is. As a result, this application displays a set of on-image UI controls 2025 and on-image UI controls 2030 (displayed as two directional arrows along the horizontal axis) for adjusting the skin tone. The set of the on-image UI control 2025 and the on-image UI control 2030 corresponds to the skin tone UI control 356. In some embodiments, the application also highlights the skin tone UI control 356 to indicate that the skin tone UI control 356 is activated. In addition to UI controls for adjusting skin tone, some embodiment applications also display a set of content-independent UI controls for adjusting color values for the entire image. In these embodiments, the content independent set of UI controls is always displayed regardless of the location selected by the user. In this example, the application displays a set of content-independent UI controls 2035 and UI controls 2040 (displayed as two directional arrows along the vertical axis) to adjust the saturation for the entire image. . The set of the on-image UI control 2035 and the on-image UI control 2040 corresponds to the saturation UI control 352. In some embodiments, the application also highlights the saturation UI control 352 to indicate that the saturation UI control 352 is activated. Although not shown in this figure, the user at this second stage 2010 can provide a specific color adjustment (ie, saturation adjustment for the entire image, or color adjustment for the skin tone of the image) by providing a directional input. Can be selected.

  In some embodiments, different sets of on-image UI controls associated with different color adjustments have different appearances. For example, the on-image UI control 2025 and the on-image UI control 2030 may have different colors than the on-image UI control 2035 and the on-image UI control 2040 to provide a distinction to the user.

  The third stage 2015 shows the GUI 300 after the user has selected a different location on the image 355. As shown, the user has selected a location on the image showing the blue sky. The application determines that the color value of the pixel corresponding to this newly selected location falls within the color value range defined for the sky color. As a result, the application displays a set of on-image UI controls 2045 and on-image UI controls 2050 (displayed as two directional arrows along the horizontal axis) for adjusting the sky color. The set of the on-image UI control 2045 and the on-image UI control 2050 corresponds to the empty UI control 354. As shown, the application also highlights the empty UI control 354 to indicate that the empty UI control 354 is activated. Similar to the second stage 2010, in addition to the on-image UI control for adjusting the sky color, this application also provides content-independent UI control 2035 and UI control 2040 for adjusting the saturation for the entire image. Also displays the set. The application also highlights the saturation UI control 352 to indicate that the saturation UI control 352 is activated. Although not shown in this figure, the user at this third stage 2015 may provide a specific color adjustment (ie, saturation adjustment for the entire image, or color adjustment for the sky color of the image) by providing a directional input. Can be selected.

  The fourth stage 2020 shows the GUI 300 after the user has selected a different location on the image 355. As shown, the user has selected a location on the image showing the mountain. The application determines that the color value of the pixel corresponding to this newly selected location falls within the color value range defined for the foliage color. As a result, this application displays a set of on-image UI controls 2055 and on-image UI controls 2060 (displayed as two directional arrows along the horizontal axis) to adjust the foliage color. The set of the on-image UI control 2055 and the on-image UI control 2060 corresponds to the foliage UI control 358. As shown, the application also highlights the foliage UI control 358 to indicate that the foliage UI control 358 is activated. Similar to the second stage 2010 and the third stage 2015, in addition to the on-image UI control for adjusting the foliage color, this application also provides a content-independent UI for adjusting the saturation for the entire image. A set of controls 2035 and UI controls 2040 is also displayed. The application also highlights the saturation UI control 352 to indicate that the saturation UI control 352 is activated. Although not shown in this figure, the user at this fourth stage 2020 can provide a direction input to provide a specific color adjustment (ie, saturation adjustment for the entire image, or color adjustment for the foliage color of the image). ) Can be selected.

  FIG. 20 illustrates an exemplary operation of activating various on-image UI controls to perform various color adjustments on the image. FIGS. 21, 22, 23, and 24 illustrate embodiments in which various color adjustments are performed on an image by manipulating various on-image UI controls. Specifically, FIG. 21 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust saturation for the entire image, at four different stages 2105, 2110, 2115, and stage 2120.

  The first stage 2105 is the same as the first stage 2005 in FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. The second stage 2110 shows the GUI 300 after the user has selected a location on the image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown in the figure, the user selects a place on the image 355 showing a person's face by tapping a finger at the place. As a result of this selection, a set of on-image UI controls 2025-2040 is displayed around the selected location on the image. This set of on-image UI controls corresponds to the saturation UI control 352 for adjusting the saturation on the entire image, and sets the on-image UI control 2035 and the on-image UI control 2040, and the skin tone of the image. It includes a set of on-image UI controls 2025 and on-image UI controls 2030 corresponding to skin tone UI controls 356 for adjustment. The second stage 2110 also shows that the corresponding saturation UI control 352 and skin tone UI control 356 are highlighted to indicate that these two UI controls are activated.

  The third stage 2115 shows the GUI 300 after the user has started adjusting the saturation of the image 355 by providing directional input to the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides directional input by dragging the user's finger upwards toward the top of the image, as indicated by arrow 2125. The application associates this directional input with the on-image UI control 2035 to increase the saturation for the entire image and performs saturation adjustment on the image 355. As shown, the color saturation in image 355 has increased as indicated by the diagonal lines across image 355. The third stage 2115 also allows the deactivated on-image UI control 2025 and on-image UI control 2030 to be removed from the display after the application associates the direction input with one set of on-image UI controls. The corresponding skin tone UI control 356 also indicates that it is no longer highlighted. In some embodiments, the activated on-image UI control 2035 and on-image UI control 2040 follow a directional input. Therefore, an activated on-image UI control is always present around the location on the image where the finger (or cursor) is placed. Furthermore, the third stage 2115 indicates that when the saturation of the color in the image 355 is increased by the on-image UI control 2035, the knob of the corresponding saturation UI control 352 has moved to the right. .

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 342 in the third stage 2115 after the user performs saturation adjustment on the image 355.

  The fourth stage 2120 shows the GUI 300 after the user has further adjusted the saturation of the image 355 by dragging the user's finger further upward as indicated by the arrow 2130. As shown, the colors in the image 355 at this stage 2120 are more saturated than the colors in the image at the third stage 2115, as indicated by the denser diagonal lines throughout the image 355. Is shown increased. The fourth stage 2120 also indicates that the corresponding chroma UI control 352 knob has moved further to the right. In some embodiments, when the user's adjustment of image saturation reaches a limit (eg, the upper or lower end of the possible range of saturation for the image), the arrow in the direction of the user's finger gradually Disappear or disappear. For example, when the user further adjusts the saturation of the image 355 by dragging the user's finger upward from the position of the finger indicated by the stage 2120, the arrow 2035 gradually disappears, Or disappear. Further, the knob of the corresponding saturation UI control 352 reaches the right end and does not move further to the right. In some embodiments, the disappearance or disappearance of this arrow with respect to reaching the limit is applicable to other types of color adjustments described in this application.

  FIG. 21 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust saturation for the entire image. FIG. 22 shows an exemplary operation for operating a set of on-image UI controls to adjust the color temperature of the image. Color temperature is a characteristic of visible light reflected from an object in an image. The warmer light that strikes an object in the image creates a warmer tone (ie, more red and yellow) in the color of the object in the image, while the object in the image The cooler light that hits creates a cooler tone (ie, more blue and cyan) in the color of the object in the image. Therefore, adjusting the color temperature of the image means adding more red / yellow or more blue / cyan to the image. Specifically, FIG. 22 illustrates this color adjustment operation with four different stages 2205, 2210, 2215, and stage 2220.

  The first stage 2205 is the same as the first stage 2005 in FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. Second stage 2210 shows GUI 300 after the user has selected a location on image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown in the figure, the user selects a place on the image 355 showing a person's face by tapping a finger at the place. As a result of this selection, a set of on-image UI controls 2025-2040 is displayed around the selected location on the image. This set of on-image UI controls corresponds to the saturation UI control 352 for adjusting the saturation on the entire image, and sets the on-image UI control 2035 and the on-image UI control 2040, and the skin tone of the image. It includes a set of on-image UI controls 2025 and on-image UI controls 2030 corresponding to skin tone UI controls 356 for adjustment. The second stage 2210 also indicates that the corresponding saturation UI control 352 and skin tone UI control 356 are highlighted to indicate that these two UI controls are activated.

  The third stage 2215 shows the GUI 300 after the user has started adjusting the color temperature of the image 355 by providing directional input for the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides directional input by dragging the user's finger to the right, as indicated by arrow 2225. The application associates this directional input with an on-image UI control 2025 to improve the skin tone in the image and performs a color temperature adjustment on the image 355. As shown, the color temperature of image 355 is rising as indicated by the diagonal lines across image 355. In some embodiments, the application adjusts the color temperature of the image to make the skin tone more pleasant. In some of these embodiments, the application uses a defined skin tone range that is the same as that described above with reference to FIG. 5, and is within that defined skin tone range. The color value of the image is adjusted so that the colors in the image that fall within the range are more comfortable.

  The third stage 2215 also allows the deactivated on-image UI control 2035 and on-image UI control 2040 to be removed from the display after the application associates the direction input with one set of on-image UI controls. The corresponding saturation UI control 352 also indicates that it is no longer highlighted. The third stage 2215 also indicates that the knob of the corresponding skin tone UI control 356 has moved to the right.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 342 in the third stage 2215 after the user performs skin tone adjustment on the image 355.

  The fourth stage 2220 shows the GUI 300 after the user has further adjusted the color temperature of the image 355 by dragging the user's finger further to the right as indicated by the arrow 2230. As shown, the image 355 at this stage 2220 is shown further adjusted than the image at the third stage 2215, as shown by the denser diagonal lines across the image 355. The fourth stage 2220 also indicates that the knob of the corresponding skin tone UI control 356 has moved further to the right.

  FIG. 22 shows an exemplary operation for operating a set of on-image UI controls to adjust the color temperature for the image. FIG. 23 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust the sky color in the image. Specifically, FIG. 23 illustrates this color adjustment operation with four different stages 2305, 2310, 2315, and stage 2320.

  The first stage 2305 is the same as the first stage 2205 in FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. The second stage 2310 shows the GUI 300 after the user has selected a location on the image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As illustrated, the user has selected a place on the image 355 showing the sky by tapping a finger at the place. As a result of this selection, a set of on-image UI controls 2035-2050 is displayed around the selected location on the image. This set of on-image UI controls includes a set of on-image UI controls 2035 and on-image UI controls 2040 corresponding to a saturation UI control 352 for adjusting the saturation of the entire image, and the sky color of the image. It includes a set of on-image UI controls 2045 and on-image UI controls 2050 that correspond to empty UI controls 354 for adjustment. Second stage 2310 also shows that the corresponding saturation UI control 352 and empty UI control 354 are highlighted to indicate that these two UI controls are activated.

  The third stage 2315 shows the GUI 300 after the user has started adjusting the sky color of the image 355 by providing a directional input for the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides directional input by dragging the user's finger to the right as indicated by arrow 2325. The application associates this directional input with the on-image UI control 2045 to adjust the sky color in the image and performs sky color adjustment for the image 355. As shown, the sky color in the image 355 is enhanced as indicated by the diagonal lines across the sky-present region in the image 355. The third stage 2315 also allows the deactivated on-image UI control 2035 and on-image UI control 2040 to be removed from the display after the application associates the direction input with one set of on-image UI controls. The corresponding saturation UI control 352 also indicates that it is no longer highlighted. The third stage 2315 also indicates that the corresponding empty UI control 354 knob has moved to the right.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As shown in the figure, a bar appears above the selectable UI item 342 in the third stage 2315 after the user performs the sky adjustment on the image 355.

  Fourth stage 2320 shows GUI 300 after the user has further adjusted the sky color of image 355 by dragging the user's finger further to the right as indicated by arrow 2330. As shown, the sky color in image 355 at this stage 2320 is shown in the image at third stage 2315 as indicated by the denser diagonal lines across the sky-indicating region in image 355. It is shown more emphasized than the sky color. The fourth stage 2320 also indicates that the corresponding empty UI control 354 knob has moved further to the right.

  FIG. 23 illustrates an exemplary operation for operating a set of on-image UI controls to adjust the sky color for an image. FIG. 24 shows an exemplary operation for manipulating a set of on-image UI controls to adjust the color of the foliage in the image. Specifically, FIG. 24 illustrates this color adjustment operation with four different stages 2405, 2410, 2415 and a stage 2420.

  The first stage 2405 is the same as the first stage 2305 in FIG. As shown, the user has selected a selectable UI item 342 for adjusting the color of the image 355. The second stage 2410 shows the GUI 300 after the user has selected a location on the image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As illustrated, the user has selected a place on the image 355 showing the mountain by tapping a finger at the place. As a result of this selection, a set of on-image UI controls 2035, 2040, 2055 and an on-image UI control 2060 are displayed around the selected location on the image. This set of on-image UI controls includes a set of on-image UI control 2035 and on-image UI control 2040 corresponding to a saturation UI control 352 for adjusting the saturation of the entire image, and the color of the foliage of the image. A set of on-image UI controls 2055 and on-image UI controls 2060 corresponding to the foliage UI controls 358 for adjusting Second stage 2410 also shows that the corresponding saturation UI control 352 and foliage UI control 358 are highlighted to indicate that these two UI controls are activated.

  The third stage 2415 shows the GUI 300 after the user has started adjusting the foliage color of the image 355 by providing directional input for the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides direction input by dragging the user's finger to the right, as indicated by arrow 2425. The application associates this directional input with the on-image UI control 2055 for adjusting the color of the foliage in the image and performs the adjustment of the foliage color for the image 355. As shown, the foliage color in the image 355 is enhanced as shown by the diagonal lines across the region representing the mountain in the image 355. The third stage 2415 also removes the deactivated on-image UI control 2035 and on-image UI control 2040 from the display after the application associates the direction input with one set of on-image UI controls. The corresponding saturation UI control 352 also indicates that it is no longer highlighted.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As shown in the figure, a bar appears above the selectable UI item 342 in the third stage 2415 after the user performs the foliage adjustment on the image 355.

  The fourth stage 2420 shows the GUI 300 after the user has further adjusted the foliage color of the image 355 by dragging the user's finger further to the right as indicated by the arrow 2430. As shown, the foliage color in the image 355 at this stage 2420 is the image at the third stage 2415, as shown by the denser diagonal lines across the area representing the mountain in the image 355. It is shown more emphasized than the color of the inner foliage.

  In some embodiments, after the user begins to provide directional input associated with a set of on-image UI controls, the set of on-image UI controls is always progressively disappeared completely from the image. It begins to disappear. The fourth stage 2420 indicates that the on-image UI control 2055 and the on-image UI control 2060 gradually disappear after the user moves the finger further to the right.

  In the above example shown in FIGS. 20, 21, 22, 23, and 24, the set of superimposed UI controls 2025-2060 is displayed on the image 355 as four arrows, but in other embodiments The set of superimposed UI controls can have different appearances. FIG. 25 shows two alternative embodiments for overlay UI control. As shown in the first embodiment, the set of superposition UI controls 2545, 2550, 2555, and superposition UI control 2560 includes a set of arms extending from a location 2525 on the image 355 selected by the user. Each superimposed UI control also includes an animated directional arrow that traverses the interior of the arm in the direction indicated by the arrow. Further, each overlay UI control is displayed in a color representing an adjustment operation corresponding to the overlay UI control. Overlay UI control 2545 and Overlay UI control 2550 are displayed in one color (eg, black), while Overlay UI control 2555 and Overlay UI control 2560 are in different colors (eg, Displayed (in white) because these two sets of superimposed UI controls are for initiating different adjustments. The second example shows a second alternative appearance for the overlay UI control. Specifically, Example (2) shows a set of overlay UI controls 2543, 2547, 2553, and overlay UI controls 2557 that provide a thumbnail preview to the user. As shown, a set of thumbnails 2570 is displayed on each overlay UI control. This thumbnail provides a preview of the image 355 corresponding to the edit that will be applied based on the location of the thumbnail on its overlay UI control. For example, the thumbnail 2585 provides a preview of how the image 355 will look if the user provides directional input based on the location 2580.

  FIG. 26 provides a set of on-image UI controls for adjusting the saturation, skin tone, sky color, and foliage color in an image based on a user selection of a location on the image. Process 2600 is shown conceptually. In some embodiments, this process is performed by an image editing application. The first three operations 2605, 2610, and 2615 are identical to the first three operations 1905, 1910, and 1915 of the process 1900 of FIG. The process begins by receiving (at 2605) a location selection on the image. The process then obtains (at 2610) the color value of the set of pixels that corresponds to the selected location on the image. In some embodiments, this set of pixels is a pixel located near the selected location.

  The process then performs (at 2615) a series of analyzes on the acquired color values. In some embodiments, each analysis determines whether the selected location is associated with a particular content type based on the obtained color values. For example, this series of analyzes can be used to determine whether a selected location is associated with a skin tone, and to determine whether a selected location is associated with a sky color. Analysis and analysis to determine whether the selected location is associated with the foliage color.

  This process determines (at 2620) whether the selected location is associated with a skin tone based on the results from the series of analyses. If the selected location is associated with a skin tone, the process provides (at 2625) a set of UI controls for adjusting the skin tone in the image. In some embodiments, this set of UI controls for adjusting the skin tone is displayed around a selected location on the image.

  This process determines (at 2630) whether the selected location is associated with a sky color based on the results from the series of analyses. If the selected location is associated with a sky color, the process provides (at 2635) a set of UI controls for adjusting the sky color in the image. In some embodiments, this set of UI controls for adjusting the sky color is displayed around a selected location on the image.

  This process determines (at 2640) whether the selected location is associated with a foliage color based on the results from the series of analyses. If the selected location is associated with the foliage color, the process provides (at 2645) a set of UI controls for adjusting the foliage color in the image. In some embodiments, this set of UI controls for adjusting the foliage color is displayed around a selected location on the image.

  The process then provides (at 2650) a set of UI controls for adjusting the saturation for the entire image. In some embodiments, this process always provides a set of UI controls for adjusting the saturation for this entire image, regardless of the location on the image selected by the user.

  The process then determines (at 2655) whether a directional input is received. If a direction input is received, the process performs (at 2660) a color adjustment operation on the image based on the direction input. The process then determines (at 2665) whether the on-image UI control is deselected. In some embodiments, the set of on-image UI controls can be deselected by lifting a finger (or releasing a button) from a device having a touch or near touch sensitive screen.

  If the process determines (at 2655) that there are no directional inputs to be received, the process also proceeds to operation 2665 to determine whether the on-image UI control is deselected. If the set of on-image UI controls is not deselected, the process returns to 2655 to determine if a directional input is received. On the other hand, if the set of on-image UI controls is deselected, the process ends.

  In some embodiments, the set of UI controls displayed on the image are direction-dependent UI controls. In these embodiments, the user can activate one of the color adjustment operations on the image by providing a directional input (ie, a vector) to the image. This direction input is a vector containing the direction and magnitude. FIG. 27 shows an exemplary set of direction-dependent UI controls 2700 that may be displayed after a user selects a location 2725 in the image. The UI control set 2700 includes UI controls 2705-2720. Each UI control corresponds to a different general direction. For example, UI control 2705 corresponds to a generally upward direction along the vertical axis of the image (ie, between direction 2730 and direction 2735), and UI control 2710 corresponds generally to the right along the horizontal axis of the image. Corresponding to a direction (ie, between direction 2735 and direction 2740), UI control 2715 corresponds to a generally downward direction (ie, between direction 2740 and direction 2745) along the vertical axis of the image, and UI Control 2720 corresponds to a direction generally to the left along the horizontal axis of the image (ie, between direction 2745 and direction 2730). When the user provides a directional input, such as vector 2750, the application determines whether the direction of vector 2750 falls within the general directional range of a particular UI control. In some embodiments, the vector 2750 is created by moving a finger on a device having a touch or near touch sensitive screen, or moving a cursor, from a selected location 2725 to a new location 2760. In some embodiments, the vector 2750 is determined to be a straight line “c” from the selected location 2725 to the new location 2760.

  In some embodiments, the direction of the vector can be expressed as an angle with respect to the axis. For example, an application of some embodiments may choose to use direction 2755 along the horizontal axis as 0 degrees. In these embodiments, each direction can be expressed in terms of an angle that goes clockwise from a 0 degree direction 2755. For example, the direction of the direction input 2750 can be represented as an angle “a”. In these embodiments, the direction input is associated with UI control 2705 if the angle of the direction input is between direction 2735 (45 degrees) and direction 2730 (135 degrees). Similarly, the direction input is associated with the UI control 2720 when the angle of the direction input is between the direction 2730 (135 degrees) and the direction 2745 (225 degrees), and the direction input is the direction input. If the angle is between the direction 2730 (225 degrees) and the direction 2745 (315 degrees), it is associated with the UI control 2715, and the direction input is the direction 2740 (315 degrees). And the direction 2735 (45 degrees) is associated with the UI control 2710. Therefore, a directional input having a 30 degree direction is associated with UI control 2710 and another directional input having a 70 degree direction is associated with UI control 2705. In this example, the angle of the direction input 2750 (ie, angle “a”) is between 315 degrees and 45 degrees, so this application is directed to the direction input as indicated by the UI control 2710 highlighting. Associate 2750 with UI control 2710.

  In some embodiments, each color adjustment operation is associated with a range of adjustment values for application to the image. That is, a larger adjustment value indicates that a greater degree (or degree) of color adjustment operation is applied to the image, and a smaller adjustment value indicates a smaller degree (or degree) of color adjustment operation. Indicates application to an image. In these embodiments, the application determines an adjustment value related to the color adjustment operation based on the magnitude of the direction input. Various embodiments use various techniques to calculate the magnitude of the direction input. In some embodiments, the application determines the magnitude of the direction input 2750 as the vector distance “c” from the selected location 2725 to the new location 2760. In other embodiments, each direction-dependent UI control has a corresponding axis. For example, UI control 2710 corresponds to axis 2755. In these embodiments, the application determines the magnitude of the direction input 2750 as the distance “b”, which is the distance covered by the vector 2750 along the corresponding axis 2755 of the UI control 2710. In any case, the calculated magnitude is used to determine an adjustment value for applying a color adjustment operation to the image.

  Although the set of direction-dependent UI controls 2700 shown in FIG. 27 only includes four direction-dependent UI controls, some embodiments are more than four to provide various adjustments to the image. It should be apparent to those skilled in the art that more or fewer than four direction-dependent UI controls are provided on the image.

  FIG. 28 conceptually illustrates a process 2800 for receiving a directional input from a user and applying a corresponding color adjustment operation to the image based on the direction and magnitude of the input. In some embodiments, the application performs process 2800 after the application receives a selection of a location on the image and provides a corresponding set of UI controls on the image.

  The process begins by receiving a directional input (at 2805) for the image. In some embodiments, the direction input includes direction and magnitude. The process then determines (at 2810) the direction of the direction input. In some embodiments, the operation of determining the direction of the direction input involves determining an angle of the direction input with respect to the image axis.

  After determining the direction of the direction input, the process associates the direction with the UI control (at 2815). In some embodiments, this process associates a direction with a particular UI control using the technique described above with reference to FIG. In these embodiments, the process assigns different directional ranges to different UI controls. The process then determines whether the direction of the input falls within the range associated with the particular UI control.

  The process then determines (at 2820) the magnitude of the direction input. In some embodiments, this process determines the magnitude of the directional input using the technique described above with reference to FIG. The process then calculates (at 2825) an adjustment value based on the magnitude of the direction input. In some embodiments, a larger magnitude corresponds to a larger adjustment value and a smaller magnitude corresponds to a smaller adjustment value.

  Finally, the process uses the calculated adjustment value to apply a color adjustment operation corresponding to the associated UI control to the image. In some embodiments, the process applies color adjustments to only a portion of the image based on UI controls that receive user input. Furthermore, the process can apply maximum color adjustments to a particular set of image pixels and a lesser degree of color adjustments to another set of image pixels. The process then ends.

  FIG. 29 illustrates an image editing application 2900 of some embodiments that provides a set of on-image UI controls for editing image color values. In some embodiments, the image editing application 2900 performs processes 1900, 2600, and process 2800. The image editing application 2900 includes several components that are the same as the image editing application 1800. For example, the image editing application 2900 includes a set of content analysis modules 1845, a color space conversion module 1815, a pixel separation module 1810, and a color adjustment engine 1820. As shown, the image editing application also includes an image processor 2940.

  UI module 1805 receives a user selection of a location on the image and passes information about the selected location to image processor 2940. Upon receipt of the selected location, the image processor 2940 obtains the color values of the set of pixels corresponding to the selected location on the image, and the content analysis module 1845 obtains the set of pixels and their color values. Pass to the set. The set of content analysis modules 1845 performs a series of different analyzes on those color values.

  In some embodiments, each content analysis module 1845 performs a different analysis to determine whether the selected location is associated with a different content type based on the received color values. For example, a particular content analysis module performs an analysis on color values to determine whether the selected location is associated with skin tone, and another content analysis module determines whether the selected location is Perform another analysis on the color values to determine if they are associated with the sky. The set of content analysis modules 1845 then passes the results of those analyzes (ie, the determined specific content type associated with the selected location) to the image processor 2940.

  Based on the results of the analysis received from the content analysis module 1845, the image processor 2940 selects a set of on-image UI controls for adjusting various color values for display on the image. The image processor 2940 then requests the UI module 1805 to display the set of UI controls on the selected image. When the UI module 1805 receives input for the image, the UI module 1805 returns the received input to the image processor 2940. In some embodiments, the received input is a directional input.

  Image processor 2940 associates the direction input with a particular color adjustment operation (eg, saturation adjustment, natural saturation adjustment, skin tone adjustment, etc.). The image processor 2940 then requests the pixel separation module 1810 to identify a set of pixels in the image that have color values that fall within the range of color values associated with that particular color adjustment operation. In some embodiments, the color values of the image are in a color space (eg, RGB color space) that is different from the color space (eg, YCC color space) used to define the range of those color values. Defined. In these embodiments, before requesting the pixel separation module 1810 to identify a set of pixels, the image processor 2940 is used to define the color values of the image and the range of those color values. Requests the color space conversion module 1815 to convert to a color space.

  After the set of pixels is identified, the image processor 2940 sends the image and information about the identified pixels to a color adjustment engine 1820 for performing a series of color adjustments on the image. The color adjustment engine 1820 applies various color adjustments to the identified pixels in the image. As shown, the color adjustment engine 1820 includes a contrast adjustment engine 1825 for applying contrast adjustment to the image, a brightness adjustment engine 1830 for applying brightness adjustment to the image, and a saturation adjustment for applying the saturation adjustment to the image. A saturation adjustment engine 1835 is included. These are only exemplary color adjustments that can be performed on an image, and the color adjustment engine 1820 provides additional color adjustment engines to perform additional types of color adjustments on the image. It should be apparent to those skilled in the art that it can be included. For example, the brightness adjustment engine 1830 uses a user input to determine a brightness adjustment value for adjusting the brightness of the image, and the contrast adjustment engine 1825 uses the user input to adjust the contrast of the image. The saturation adjustment engine 1835 uses a user input to determine a saturation adjustment value for adjusting the saturation of the image. The various adjustment engines 1825-1835 then apply those lightness adjustments, contrast adjustments, and saturation adjustments to the image to produce an edited image.

  In some embodiments, the color adjustment engine 1820 adjusts the color values of the image within the converted color space. In these embodiments, the image processor 2940 sends the adjusted image to the color space conversion module 1815 and converts the color value of the image back to its original color space. The image processor 2940 then sends the adjusted image to the media storage device and to the UI module 1805 for display.

  Providing a set of UI controls on an image has many benefits. For example, the on-image UI control provides a larger scale with finer granularity than the range slider to allow the user to specify the adjustment value within the range of adjustment values. FIG. 30 shows an example of such benefits through the GUI 300. As shown in FIG. 30, see FIG. 7 and FIG. 23 in which the sky UI control 354 for adjusting the sky color of the image 355 and the corresponding on-image UI control 2045 and on-image UI control 2050 are activated. As described above, the user can either manipulate the empty UI control 354 or use the on-image UI control 2045 and the on-image UI control 2050 to provide directional input to the image. The sky color can be adjusted. In this embodiment, the empty UI control 354 is implemented as a range slider so that the user can specify an adjustment value between 0 and 100 by moving the knob along the range slider 354. It becomes. Since the range slider 354 is limited in size, each unit 3005 in which the user moves the knob along the range slider 354 increases or decreases the adjustment value by a large amount (eg, 25). On the other hand, when the user uses the on-image UI control by providing directional input to the image, the user can move from the selected location to both ends of the image. Therefore, each unit 3010 that the user moves on the image increases or decreases the adjustment value only by a small number (eg, 2). Therefore, the on-image UI control provides the user with a much larger scale with finer granularity for specifying adjustment values.

B. On-Image Exposure Control FIGS. 20, 21, 22, 23, and 24 in the above section described several examples of on-image UI control. In those embodiments, the application allows the user to select a location within the image and various to adjust the color of the image depending on the type of content associated with the selected location. Provides a set of UI controls on the image. This section provides some further examples of UI controls on these images. Specifically, FIG. 31 illustrates operations for providing various sets of on-image UI controls for adjusting image exposure settings depending on the type of content associated with the selected location. Specifically, FIG. 31 illustrates operations for activating various sets of on-image UI controls through the GUI 300 at four different stages 3105, 3110, 3115, and stage 3120. Each of these stages is described in more detail below.

  The first stage 3105 is similar to the first stage 2005 of FIG. 20, except that the user can select to adjust the exposure setting of the image 355, as indicated by the highlighting of the selectable UI item 3125. The UI item 3125 is selected. Selection of the selectable UI item 3125 includes performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on the device that has a touch or near-touch sensitive screen that displays the selectable UI item 3125. Or by placing the cursor on the selectable UI item 3125 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown in FIG. 31, a set of exposure adjustment UI controls 3130 to 3150 is displayed in the UI control display area 350. Various embodiments provide various types of UI controls for adjusting the color of the image. In this embodiment, the exposure adjustment UI control is an integrated slider control that includes a plurality of control knobs 3130-3150 that are slidable along a sliding path.

  In some embodiments, using the integrated slider control in a media editing application allows the user to move several different control knobs along the sliding trajectory, thereby moving several different characteristics of the image (eg, , Brightness, contrast, etc.) can be corrected. Each knob (3130 to 3150) on the multi-slider corresponds to a different adjustment operation that can be performed on the image. The UI control 3130 (knob 3130) is for adjusting the shadow of the image, and the UI control 3135 and UI control 3140 (knob 3135 and the knob 3140) are for adjusting the contrast of the image. The control 3145 (knob 3145) is for adjusting the brightness of the image, and the UI control 3150 (knob 3150) is for adjusting the highlight of the image. The user can initiate various color adjustments to the image by selecting one of these knobs and moving it to different positions along the multi-slider. Various operations of this integrated slider control are described in US Provisional Application No. 61 / 607,554.

  The second stage 3110 shows the GUI 300 after the user has selected a location on the image 355. As shown, the user has selected a location on the image to display the sea. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As described above, after the user selects a location on the image, some embodiments of the image editing application may use a series of steps to determine the type of specific content associated with the selected location. Perform analysis. Various embodiments use various techniques to determine the type of content associated with the selected location. The application of some embodiments initially defines a range of different color values associated with different types of content. For example, the application of some embodiments defines a range of color values associated with a shadow, a range of color values associated with a midtone, and a range of color values associated with a highlight. The application then determines whether the color value of the set of pixels corresponding to the selected location in the image falls within the color value range associated with the particular content type, and then Displays a set of on-image UI controls associated with that particular content type.

  Various embodiments implement a set of on-image UI controls in various ways. For example, some embodiments of the application implement a set of on-image UI controls as direction-dependent UI controls. In some of these embodiments, the application displays a directional arrow for each UI control to guide the user to initiate a color adjustment operation associated with the UI control. The user initiates various color adjustment operations by providing various direction inputs. Based on the direction of this direction input, the application performs a specific color adjustment on the image.

  In the second stage 3110, the user has selected a location on the screen displaying the sea that has a dark color (i.e., a low brightness value), so that the application uses the color value of the pixel corresponding to the selected location. Determines that it is associated with the shadow. As a result, the application displays a set of on-image UI controls 3165 and on-image UI controls 3170 (displayed as two direction arrows along the vertical axis) for adjusting shadows in the image. A set of the on-image UI control 3165 and the on-image UI control 3170 corresponds to the shadow UI control 3130. In some embodiments, the application also highlights the shadow UI control 3130 to indicate that the shadow UI control 3130 is activated. In addition to the UI controls for adjusting shadows, some embodiment applications also display a set of content-independent UI controls for adjusting the contrast for the image. In these embodiments, the content independent set of UI controls is always displayed regardless of the location selected by the user. In this example, the application displays a set of content-independent UI controls 3155 and UI controls 3160 (displayed as two directional arrows along the horizontal axis) for adjusting the contrast for the image. The set of the on-image UI control 3155 and the on-image UI control 3160 corresponds to the contrast UI control 3135 and the contrast UI control 3140. In some embodiments, the application also highlights the contrast UI control 3135 and the contrast UI control 3140 to indicate that the contrast UI control 3135 and the contrast UI control 3140 are activated. Although not shown in this figure, the user at this second stage 3110 can adjust the exposure (ie, contrast adjustment or shadow adjustment) by providing a directional input to the image.

  The third stage 3115 shows the GUI 300 after the user has selected a different location on the image 355. As shown, the user has selected a location on the image showing a blue sky that has a bright color (ie, a high brightness value). The application determines that the color value of the pixel corresponding to this newly selected location falls within the color value range defined for the highlight. As a result, the application displays a set of on-image UI controls 3175 and on-image UI controls 3180 (displayed as two directional arrows along the vertical axis) for adjusting the highlights of the image. A set of the on-image UI control 3175 and the on-image UI control 3180 corresponds to the highlight UI control 3150. As shown, the application also highlights the highlight UI control 3150 to indicate that the highlight UI control 3150 is activated. Similar to the second stage 3110, in addition to the on-image UI control to adjust the highlights, this application also has a content-independent UI control 3155 and UI control 3160 set to adjust the contrast for the image. indicate. The application also highlights the contrast UI control 3135 and the contrast UI control 3140 to indicate that the contrast UI control 3135 and the contrast UI control 3140 are activated. Although not shown in this figure, the user at this third stage 3115 can initiate a specific color adjustment (ie, contrast adjustment or highlight adjustment) by providing a directional input to the image. .

  The fourth stage 3120 shows the GUI 300 after the user has selected a different location on the image 355. As shown, the user has selected a location on the image showing a mountain that has a midtone (ie, a medium luminance value). The application determines that the color value of the pixel corresponding to this newly selected location falls within the color value range defined for the intermediate tone. As a result, this application displays a set of on-image UI controls 3185 and on-image UI controls 3190 (displayed as two directional arrows along the vertical axis) for adjusting the brightness of the image. The set of the on-image UI control 3185 and the on-image UI control 3190 corresponds to the lightness UI control 3145. As shown, the application also highlights the brightness UI control 3145 to indicate that the brightness UI control 3145 has been activated. Similar to the second stage 3110 and the third stage 3115, in addition to the on-image UI control for adjusting the brightness, this application also provides a content independent UI control 3155 and UI control for adjusting the contrast for the image. A set of 3160 is also displayed. The application also highlights the contrast UI control 3135 and the contrast UI control 3140 to indicate that the contrast UI control 3135 and the contrast UI control 3140 are activated. Although not shown in this figure, the user at this fourth stage 3120 can select a particular exposure adjustment (ie, contrast adjustment or brightness adjustment) by providing a directional input to the image.

  FIG. 31 illustrates an exemplary operation of activating various on-image UI controls to perform various exposure adjustments on the image. FIGS. 32, 33, 34, and 35 illustrate embodiments in which various exposure adjustments are performed on an image by manipulating various on-image UI controls. Specifically, FIG. 32 shows an exemplary operation for manipulating a set of on-image UI controls to adjust the contrast for the image at four different stages 3205, 3210, 3215, and stage 3220.

  The first stage 3205 is the same as the first stage 3105 in FIG. As shown, the user has selected a selectable UI item 3125 for adjusting the exposure setting of the image 355. The second stage 3210 shows the GUI 300 after the user has selected a location on the image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown in the figure, the user has selected a place on the image 355 showing the sea, which has a dark color, by tapping the finger at the place. As a result of this selection, a set of on-image UI controls 3155-3170 are displayed around the selected location on the image. This set of on-image UI controls includes a set of on-image UI controls 3155 and an on-image UI control 3160 corresponding to a contrast UI control 3135 and a contrast UI control 3140 for adjusting contrast on the image, and an image shadow. It includes a set of on-image UI controls 3165 and on-image UI controls 3170 corresponding to the shadow UI controls 3130 for adjustment. The second stage 3210 also shows that the corresponding contrast UI control 3135 and contrast UI control 3140 and the shadow UI control 3130 are highlighted to indicate that these UI controls are activated.

  The third stage 3215 shows the GUI 300 after the user has started adjusting the contrast of the image 355 by providing directional input for the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides directional input by dragging the user's finger to the right of image 355 as indicated by arrow 3225. The application associates this directional input with an on-image UI control 3155 and an on-image UI control 3160 to increase the contrast for the image 355 to perform contrast adjustment for the image 355. As shown, the color contrast in image 355 is increased. Darker areas (eg, mountain areas and sea areas) of image 355 are darkened and lighter areas (eg, clouds) of image 355, as indicated by the diagonal lines across those darker areas. And the sun region) are lightened, as indicated by the dashed lines that outline their brighter regions. The third stage 3215 also allows the deactivated on-image UI control 3165 and on-image UI control 3170 to be removed from the display after the application associates the direction input with one set of on-image UI controls. The corresponding shadow UI control 3130 also indicates that it is no longer highlighted. The third stage 3215 also indicates that the corresponding contrast UI control 3135 and contrast UI control 3140 have been moved away from each other.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 3215 when exposure adjustment is performed on the image. The top bar of this selectable UI item 3215 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 3215 in the third stage 3215 after the user has performed contrast adjustment on the image 355.

  The fourth stage 3220 shows the GUI 300 after the user has further increased the contrast of the image 355 by dragging the user's finger further to the right as indicated by the arrow 3230. As shown, the colors in the dark area of the image 355 at this stage 3220 are dark areas of the image at the third stage 3215, as indicated by the denser diagonal lines across the dark area of the image 355. It is shown darker than the color inside. Similarly, the colors in the light color region of the image 355 at this stage 3220 outline the light color region of the image 355, as indicated by the finer dashed lines, the light color region of the image at the third stage 3215. It is shown with a lighter color than the inside color. The fourth stage 3320 also indicates that the corresponding contrast UI control 3135 and contrast UI control 3140 have been moved further away from each other.

  FIG. 32 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust image contrast. FIG. 33 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust image shadows. Specifically, FIG. 33 illustrates this exposure adjustment operation with four different stages 3305, 3310, 3315, and stage 3320.

  The first stage 3305 is the same as the first stage 3105 in FIG. As shown, the user has selected a selectable UI item 3125 for adjusting the exposure setting of the image 355. Second stage 3310 shows GUI 300 after the user has selected a location on image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown in the figure, the user has selected a place on the image 355 showing the sea, which has a dark color, by tapping the finger at the place. As a result of this selection, a set of on-image UI controls 3155-3170 are displayed around the selected location on the image. This set of on-image UI controls includes a set of on-image UI controls 3155 and on-image UI controls 3160 corresponding to a contrast UI control 3135 and a contrast UI control 3140 for adjusting the contrast of the image, and an image shadow. It includes a set of on-image UI controls 3165 and on-image UI controls 3170 corresponding to the shadow UI controls 3130 for adjustment. The second stage 3310 also shows that the corresponding contrast UI control 3135 and contrast UI control 3140 and the shadow UI control 3130 are highlighted to indicate that these UI controls are activated.

  The third stage 3315 shows the GUI 300 after the user has started adjusting the shadow of the image 355 by providing a directional input for the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides direction input by dragging the user's finger downwards toward the bottom of the image 355, as indicated by arrow 3325. The application associates this direction input with an on-image UI control 3170 to darken the shadows in the image and perform a shadow adjustment on the image 355. As shown in the figure, a dark color region (for example, a region indicating the sea) in the image 355 is darkened as indicated by a diagonal line over the entire region indicating the sea in the image 355. The third stage 3315 also allows the deactivated on-image UI control 3155 and on-image UI control 3160 to be removed from the display after the application associates the direction input with one set of on-image UI controls. The corresponding contrast UI control 3135 and contrast UI control 3140 also indicate that they are no longer highlighted. The third stage 3315 also indicates that the corresponding shadow UI control 3130 has been moved to the left.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 3125 when exposure adjustment is performed on the image. The bar above this selectable UI item 3125 remains visible to indicate which type of adjustment the user has performed on the image. As shown, a bar appears above the selectable UI item 3125 in the third stage 3315 after the user performs shadow adjustment on the image 355.

  Fourth stage 3320 shows GUI 300 after the user has further adjusted the shadow of image 355 by dragging the user's finger further downward as indicated by arrow 3330. As shown, the dark area of the image 355 at this stage 3320 (eg, the area showing the sea and mountains) is third, as indicated by the denser diagonal lines across the dark area in the image 355. The saturation is shown to be further increased than in the dark area in the image at stage 3315. The fourth stage 3320 also indicates that the corresponding shadow UI control 3130 has been moved further to the left.

  FIG. 33 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust image shadows. FIG. 34 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust image highlights. Specifically, FIG. 34 illustrates this exposure adjustment operation with four different stages 3405, 3410, 3415 and a stage 3420.

  The first stage 3405 is the same as the first stage 3105 in FIG. As shown, the user has selected a selectable UI item 3125 for adjusting the exposure setting of the image 355. The second stage 3410 shows the GUI 300 after the user has selected a location on the image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown, the user has selected a location on the image 355 that has a light color and shows the sky by tapping the finger at that location. As a result of this selection, a set of on-image UI controls 3155, 3160, 3175 and on-image UI controls 3180 are displayed around the selected location on the image. This set of on-image UI controls includes a set of on-image UI controls 3155 and on-image UI controls 3160 corresponding to a contrast UI control 3135 and a contrast UI control 3140 for adjusting the contrast of the image, and an image highlight. A set of on-image UI controls 3175 and on-image UI controls 3180 corresponding to a highlight UI control 3150 for adjusting the image. The second stage 3410 also shows that the corresponding contrast UI control 3135 and contrast UI control 3140 and highlight UI control 3150 are highlighted to indicate that these UI controls are activated.

  The third stage 3415 shows the GUI 300 after the user has started adjusting the highlights of the image 355 by providing directional input for the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides directional input by dragging the user's finger downwards toward the bottom of the image 355, as indicated by arrow 3425. The application associates this direction input with the on-image UI control 3180 for darkening the highlights in the image and performs highlight adjustment on the image 355. As shown, the light areas in the image 355 (eg, areas that show the sun and sky) are darkened, as indicated by the diagonal lines across the areas that show the sun and sky in the image 355. The third stage 3415 also removes the deactivated on-image UI control 3155 and the on-image UI control 3160 from the display after the application associates the direction input with one set of on-image UI controls. The corresponding contrast UI control 3135 and contrast UI control 3140 also indicate that they are no longer highlighted. The third stage 3415 also indicates that the corresponding highlight UI control 3150 has been moved to the left.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 3125 when exposure adjustment is performed on the image. The bar above this selectable UI item 3125 remains visible to indicate which type of adjustment the user has performed on the image. As illustrated, a bar appears above the selectable UI item 3125 in the third stage 3415 after the user performs highlight adjustment on the image 355.

  The fourth stage 3420 shows the GUI 300 after the user has further adjusted the highlights of the image 355 by dragging the user's finger further downward as indicated by the arrow 3430. As shown, the light areas in the image 355 at this stage 3420 (eg, areas showing the sun and sky) as shown by the denser diagonal lines across the light areas in the image 355. In the third stage 3415, the image is further darkened than in the light color area in the image. The fourth stage 3420 also indicates that the corresponding highlight UI control 3150 has been moved further to the left.

  FIG. 34 illustrates an exemplary operation for manipulating a set of on-image UI controls to adjust image highlights. FIG. 35 shows an exemplary operation for manipulating a set of on-image UI controls to adjust the overall brightness of the image. Specifically, FIG. 35 shows this exposure adjustment operation with four different stages 3505, 3510, 3515 and a stage 3520.

  The first stage 3505 is the same as the first stage 3105 in FIG. As shown, the user has selected a selectable UI item 3125 for adjusting the exposure setting of the image 355. The second stage 3510 shows the GUI 300 after the user has selected a location on the image 355. Selection of a location on the image 355 can be performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device with a touch or near touch sensitive screen that displays the image. Alternatively, it can be performed by placing the cursor at the location of the image and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown, the user has selected a location on the image 355 showing a mountain having an intermediate color tone by tapping the finger at that location. As a result of this selection, a set of on-image UI controls 3155, 3160, 3185, and on-image UI controls 3190 are displayed around the selected location on the image. This set of on-image UI controls includes a set of on-image UI controls 3155 and on-image UI controls 3160 corresponding to a contrast UI control 3135 and a contrast UI control 3140 for adjusting the contrast of the image, and the overall image. A set of on-image UI control 3185 and on-image UI control 3190 corresponding to the brightness UI control 3145 for adjusting the brightness. The second stage 3510 also shows that the corresponding contrast UI control 3135 and contrast UI control 3140 and the brightness UI control 3145 are highlighted to indicate that these UI controls are activated.

  The third stage 3515 shows the GUI 300 after the user has started adjusting the overall brightness of the image 355 by providing directional input for the image 355. This directional input can be provided by dragging the user's finger in a specific direction on a device having a touch or near touch sensitive screen, or by dragging a cursor. In this example, the user provides directional input by dragging the user's finger upwards toward the top of the image 355 as indicated by arrow 3525. The application associates this directional input with an on-image UI control 3185 to increase the brightness of the image (ie, increase the brightness value of the pixels of the image) and perform a brightness adjustment on the image 355. As shown, the entire image 355 is lightened as indicated by the dashed line that outlines the object in the image 355. The third stage 3515 also allows the deactivated on-image UI control 3155 and on-image UI control 3160 to be removed from the display after the application associates the directional input with one set of on-image UI controls. The corresponding contrast UI control 3135 and contrast UI control 3140 also indicate that they are no longer highlighted. The third stage 3515 also indicates that the corresponding brightness UI control 3145 has been moved to the left.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 3125 when exposure adjustment is performed on the image. The bar above this selectable UI item 3125 remains visible to indicate which type of adjustment the user has performed on the image. As illustrated, a bar appears above the selectable UI item 3125 in the third stage 3415 after the user performs brightness adjustment on the image 355.

  The fourth stage 3520 shows the GUI 300 after the user has further adjusted the brightness of the image 355 by dragging the user's finger further upward, as indicated by the arrow 3530. As shown, the image 355 at this stage 3520 is shown to be lighter than the image at the third stage 3515, as indicated by the finer dashed lines that outline the object of the image 355. It is. The fourth stage 3520 also indicates that the corresponding brightness UI control 3145 has been moved further to the left.

  FIG. 36 conceptually illustrates a process 3600 for providing a set of on-image UI controls for adjusting contrast, brightness, shadows, and highlights in an image based on a user selection of a location on the image. Show. In some embodiments, this process is performed by an image editing application. The first three operations 3605, 3610, and 3615 are identical to the first three operations 1905, 1910, and 1915 of the process 1900 of FIG. The process begins by receiving (at 3605) a location selection on the image. The process then obtains (at 3610) the color value of the set of pixels corresponding to the selected location on the image. In some embodiments, this set of pixels is a pixel located near the selected location.

  The process then performs (at 3615) a series of analyzes on the acquired color values. In some embodiments, each analysis determines whether the selected location is associated with a particular content type based on the obtained color values. For example, the series of analyzes includes an analysis for determining whether a selected location is associated with a dark color, an analysis for determining whether a selected location is associated with a light color, and , Including an analysis to determine whether the selected location is associated with a midtone.

  The process then determines (at 3620) whether the selected location is associated with a dark color based on the results from the series of analyses. In some embodiments, the process determines that the selected location is associated with a dark color if the acquired color value has a low luminance (ie, lightness) level. If the selected location is associated with a dark color, this process provides (at 3625) a set of UI controls for adjusting shadows in the image. In some embodiments, a set of UI controls for adjusting this shadow is displayed around a selected location on the image.

  The process then determines (at 3630) whether the selected location is associated with a light color based on the results from the series of analyses. In some embodiments, the process determines that the selected location is associated with a light color if the acquired color value has a high luminance (ie, lightness) level. If the selected location is associated with a light color, this process provides (at 3635) a set of UI controls for adjusting highlights in the image. In some embodiments, a set of UI controls for adjusting this highlight is displayed around a selected location on the image.

  The process then determines (at 3640) whether the selected location is associated with a midtone based on the results from the series of analyses. In some embodiments, the process determines that the selected location is associated with a midtone when the acquired color value has an average or median luminance (ie, brightness) level. judge. If the selected location is associated with a midtone, this process provides (at 3645) a set of UI controls for adjusting the brightness in the image. In some embodiments, a set of UI controls for adjusting this brightness is displayed around a selected location on the image.

  The process then provides (at 3650) a set of UI controls for adjusting the contrast for the entire image. In some embodiments, this process always provides a set of UI controls for adjusting the contrast for this entire image, regardless of the location on the image selected by the user.

  The process then determines (at 3655) whether a directional input is received. If a directional input is received, the process performs an exposure adjustment operation on the image (at 3660) based on the directional input. The process then determines (at 3665) whether the on-image UI control is deselected. In some embodiments, the set of on-image UI controls can be deselected by lifting a finger (or releasing a button) from a device having a touch or near touch sensitive screen.

  If the process determines (at 3655) that there are no directional inputs to be received, the process also proceeds to operation 3665 to determine whether the on-image UI control is deselected. If the set of on-image UI controls is not deselected, the process returns to 3655 to determine if a directional input is received. On the other hand, if the set of on-image UI controls is deselected, the process ends.

III. Color Balance Control In addition to providing UI controls for adjusting only the portion of the image associated with the type of content, the image editing application of some embodiments also adjusts the color balance of the image. It also provides a set of UI controls. Very often, capturing an image under various lighting conditions can adversely affect the quality of the image. A common artifact is the overall undesirable color cast in an image created by the lighting conditions in which the image is captured. For example, an image captured under an incandescent lamp often has a yellowish color, so an object that should appear white under neutral lighting conditions appears yellow in the image. Therefore, color balance adjustment (also known as white balance adjustment) is an operation for correcting the overall color of an image so that an object in the image looks natural and comfortable.

  In some embodiments, the application provides a color balance UI control that allows the user to select a location on the screen. The application then obtains the color value of the pixel corresponding to the selected location and changes the color value of the pixel to a set reference color (eg, gray, ideal skin color, etc.) To decide. Based on this color adjustment, the application generates a color space transformation that maps each color in the image to a different color. The application then applies the color space transformation to every pixel in the image.

  Various embodiments provide various reference colors for this color balance adjustment operation. For example, the application of some embodiments sets gray as the reference color. In these embodiments, the user adjusts the color balance of the image through the color balance UI by selecting a location in the image that displays objects that appear to be gray in nature.

  FIG. 37 illustrates an example of adjusting the color balance of an image through a color balance UI tool, with four different stages 3705, 3710, 3715, and stage 3720.

  The first stage 3705 is similar to the second stage 310 of FIG. 3, except that the user at the first stage 3705 can select a color balance to trigger the display of a set of various color balance tools. The UI item 3725 has been selected. As shown, the user has selected an image 355 to be displayed in the image display area 345. In some embodiments, the selectable UI item 3725 for triggering the color balance tool is selected when the user selects the color adjustment view (by selecting the selectable UI item 342 in the menu bar 340). It is displayed inside the UI control display area 350. Selection of selectable UI item 3725 involves performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device having a touch or near-touch sensitive screen that displays selectable UI item 3725 Or by placing the cursor on selectable UI item 3725 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.).

  As illustrated, after the user selects a selectable UI item 3725, a color balance tool display area 3730 appears at the top of the UI control display area 350. The color balance tool display area 3730 includes a number of selectable UI items for initiating various color balance tools. For example, the color balance tool display area 3730 includes a selectable UI item 3735 for starting the “original white balance” tool. This “original white balance” tool uses the white balance settings of the device that captured the image to adjust the white balance of the image. The color balance tool display area 3730 also includes a set of selectable UI items for initiating various preset white balance tools. The preset white balance tool adjusts the white balance of the image based on various predefined lighting conditions, such as daylight, cloudy, flash, shade, incandescent, and fluorescent lighting conditions. Further, the color balance tool display area 3730 also includes two custom color balances, such as a selectable UI item 3745 for initiating a custom white balance tool and a selectable UI item 3740 for initiating a custom skin color balance tool. Also includes a selectable UI item to start the tool.

  The custom white balance tool allows the user to select a location on the image that is assumed to be white or gray under neutral lighting conditions. The image editing application then defines an algorithm or mathematical equation for adjusting the color value of the location so that the color value represents white. The application then applies the same algorithm or mathematical equation to the rest of the pixels in the image.

  The second stage 3710 shows the GUI 300 after the user has selected a selectable UI item 3745 to start the custom white balance tool. Selection of selectable UI item 3745 is performed by performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device having a touch or near-touch sensitive screen that displays selectable UI item 3745 Or by placing the cursor on selectable UI item 3745 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown, the user has selected the selectable UI item 3745 as indicated by the highlighting of the selectable UI item 3745. As a result, a custom white balance UI control 3755 is displayed on the image.

  When custom white balance UI control 3755 appears on the image, the application performs a white balance operation by adjusting the color value of image 355. Specifically, the application samples one or more of the color values of the set of pixels corresponding to or near the location selected through the custom white balance UI control 3755. The application then devises an algorithm or mathematical equation (taking a set of parameters as input) to convert the color value to the nearest gray, and then for every pixel in the image 355, The same algorithm or mathematical equation is applied. In some embodiments, the application converts the color values of the image 355 to a YCC color space (eg, YIQ color space) before applying color balance adjustments to the image.

  Various embodiments implement custom white balance UI control in various ways. In this example, the custom white balance UI control appears as a circle and has a crosshair in the center of the circle. The user moves the custom white balance UI control 3755 to a region of the image 355 so that the crosshairs are over an object in the image 355 that is assumed to be white or gray under neutral lighting conditions. By placing in, a location on the image 355 can be selected.

  In some embodiments, the application may edit the white balance adjustment for the portion of the image outside the custom white balance UI control 3755 (ie, based on the current location of the custom white balance UI control 3755). Provide a preview of this white balance adjustment by displaying the version (if any). In these embodiments, the portion of the image inside the custom white balance UI control 3755 is left unedited so that the user can see the difference between the edited version and the unedited version. It becomes. As shown by this second stage 3710, the portion of the image 355 outside the custom white balance UI control 3755 is adjusted as indicated by the diagonal lines over only the portion of the image 355 outside the custom white balance UI control 3755. On the other hand, the portion of the image 355 that is inside the custom white balance UI control 3755 remains unaffected. In other embodiments, only the portion of the image inside the custom white balance UI control 3755 is edited, and the portion of the image outside the custom white balance UI control 3755 is left unedited. Furthermore, in other embodiments, the entire image is edited.

  In addition to this preview functionality, some embodiment applications also allow the user to accurately select a white or gray location in the image by zooming in on the portion of the image inside the custom white balance UI control 3755. To help. As shown, the portion of the image 355 inside the custom white balance UI control 3755 has been zoomed in so that the area shows a bird that is much larger than what is visible on the first stage 3705.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 342 in the second stage 320 after the user has performed color balance adjustment on the image 355.

  As described above, some embodiments of the application allow the user to move the custom white balance UI control 3755 to various locations on the image. The third stage 3715 is a transitional stage indicating that the user has selected the custom white balance UI control 3755 but has not started moving the custom white balance UI control 3755. The fourth stage 3720 shows the GUI 300 after the user has moved the custom white balance UI control 3755 to a different location on the image. In some embodiments, the user can move the custom white balance UI control 3755 by dragging the user's finger or dragging the cursor to a different location on the image. As shown, the user has moved the custom white balance UI control 3755 to the area of the image 355 showing the cloud, as indicated by the arrow. As a result of the movement of the custom white balance UI control 3755 to this new location, the application will readjust the color values of the image. In some embodiments, the application samples another pixel corresponding to the location selected through this custom white balance UI control 3755. The application then devises an algorithm or mathematical equation (taking a set of parameters as input) to convert the color value to the nearest gray, and then for every pixel in the image 355, The same algorithm or mathematical equation is applied. As shown, the portion of the image outside of the custom white balance UI control 3755 is shown readjusted as indicated by the different set of diagonal lines across that portion of the image 355.

  In the above example shown in FIG. 37, the application performs a color balance operation on the image immediately after the color balance control 3755 appears on the image, and whenever the user moves the color balance UI control 3755, Readjust the image. In some other embodiments, in order to save the processing power of the device on which the application is executed, the application may receive further input from the user (e.g. selection of “Run Color Balance” selectable UI item, image Perform color balance operations on images only when receiving additional taps, etc.). In this way, the image is not adjusted each time the user moves the color balance UI control 3755, but only when the user confirms the location and provides further input.

  FIG. 38 illustrates an example white balance operation for an image performed by the image editing application of some embodiments. As shown, color space 3800 represents a color space in which the color values of the image are defined. This particular color space has three axes: an axis 3820 representing a range of red and green color values, an axis 3830 representing a range of blue and yellow color values, and various shades of gray (i.e. Defined along the vertical axis 3810 representing the range of luminance values, where the bottom surface location 3840 represents black and the top surface location 3850 represents white. The brighter the color, the farther the color appears from the bottom surface of the column 3800 in this color space.

  Region 3860 represents all the color values of the image. Although region 3860 is shown as being located within one small region within color space 3800, the color values of the image may occupy many different regions of color space 3800. When the user selects a location on the image using the white balance UI control, the application obtains the color value of the pixel corresponding to the selected location. In this figure, color 3880 represents the color of the pixel corresponding to the location specified by the white balance UI control. The application then locates the gray (eg, 3890) closest to the selected color value within this color space and determines a color adjustment that modifies the selected color value 3880 to a gray value 3890. . The application then generates a color space transformation based on the determined color adjustment. In some embodiments, this color space transformation is an M × M matrix (eg, 3 × 3 matrix) that is generated based on the determined color adjustment. In some embodiments, this color space conversion maps each color in color space 3800 to another color in the distorted color space. A color space 3805 in FIG. 38 represents a color space after the color space 3800 is converted by this color space conversion. As shown, color 3880 in color space 3800 is mapped to color 385 in color space 3805 and another color 3870 in color space 3800 is mapped to color 3875 in color space 3805. In some embodiments, the application adjusts the white balance of the image based on the selected location by applying this color space transformation to each pixel in the image.

  FIG. 37 illustrates an example of performing a custom white balance operation using a custom white balance UI control. In addition to custom white balance operations, some embodiments of the application also provide custom skin color balance operations. This custom skin balance operation is similar to the custom white balance operation, except that instead of selecting a location on the image that is assumed to be white under neutral light, the user can Select a location on the image to display. In these embodiments, the application defines an ideal facial color set. For example, an application of some embodiments may define an ideal facial color under daylight conditions and an ideal facial color under artificial light conditions. When the user selects a location on the image, this application samples the color value of the pixel corresponding to the selected location and adjusts the color value of that pixel to the ideal facial color. Define adjustments. The application then applies that same adjustment to every pixel in the image.

  FIG. 39 shows an example of performing a custom flesh color balance operation with four different stages 3905, 3910, 3915, and stage 3920. FIG.

  The first stage 3905 is similar to the first stage 3705 of FIG. 37 except that the user has selected another image for editing. As shown, the user has selected an image 3955 to be displayed in the image display area 345. Image 3955 is a photograph of a person 3930 standing in the foreground and another person 3935 standing in the background. Image 3955 also shows mountains and seas on the right side of the image. Similar to the first stage 3705 of FIG. 37, the user opens the white balance tool display area 3730 by selecting the selectable UI item 3725.

  The second stage 3910 shows the GUI 300 after the user has selected a selectable UI item 3740 to start the custom skin color balance tool. Selection of selectable UI item 3740 includes performing a gesture (eg, placing a finger, pointing, or tapping a finger) at a location on a device having a touch or near-touch sensitive screen that displays selectable UI item 3740 Or by placing the cursor on selectable UI item 3740 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.). As shown, the user has selected selectable UI item 3740 as indicated by the highlighting of selectable UI item 3740. As a result, the custom skin color balance UI control 3925 is displayed on the image.

  When the custom skin color balance UI control 3925 appears on the image, the application performs a skin color balance operation by adjusting the color value of the image 3955. Specifically, the application samples one or more of the color values of the set of pixels corresponding to or near the location selected through the custom skin color balance UI control 3925. The application then devises an algorithm or mathematical equation (taking a set of parameters as input) to convert that color value into a predefined ideal facial color, and then in image 3955 The same algorithm or mathematical equation is applied to all the pixels.

  Various embodiments implement custom skin color balance UI control in various ways. In this embodiment, the custom skin color balance UI control 3925 appears as a circle and has a crosshair in the center of the circle. The user places a location on the image 3955 by placing (or moving) the custom skin color balance UI control 3925 within the region of the image 3955 so that the crosshairs are on the person's face in the image 3955. Can be selected.

  In some embodiments, the application uses a face detection algorithm to detect a person's face in the image, and over the first person's face detected in the image, a custom skin color balance UI. A control 3925 is arranged. As shown in the second stage 3910, the custom skin color balance UI control 3925 appears on the face of a person standing in the background.

  Similar to the custom white balance UI control, some embodiments of the application may have edited the image for that portion of the image outside of the custom skin color balance UI control 3925 (ie, the current skin of the custom skin balance UI control 3925). Provide a preview of this skin color balance adjustment by displaying the version (if the image is skin tone balanced) based on location. In these embodiments, the portion of the image inside the custom skin color balance UI control 3925 is kept unedited so that the user can see the difference between the edited version and the unedited version. It becomes. As shown in this second stage 3910, the portion of the image 3955 that is outside the custom skin color balance UI control 3925 is adjusted as indicated by the diagonal lines that span only the portion of the image 3955 that is outside the custom skin color balance UI control 3925. On the other hand, the portion of the image 3955 that is inside the custom skin color balance UI control 3925 remains unaffected. In other embodiments, only the portion of the image inside the custom skin color balance UI control 3925 is edited, and the portion of the image outside the custom skin color balance UI control 3925 remains unedited. Furthermore, in other embodiments, the entire image is edited.

  In addition to this preview functionality, the application of some embodiments also allows the user to zoom in on the portion of the image inside the custom skin balance UI control 3925 so that the user can see where in the image the person's face is displayed. Help you make the right choice. As shown, the portion of the image 355 inside the custom flesh color balance UI control 3925 has been zoomed in so that the area shows more details of the face of the person standing in the background.

  Furthermore, the GUI 300 also displays a bar at the top of the selectable UI item 342 when color adjustment is performed on the image. The top bar of this selectable UI item 342 remains visible to indicate which type of adjustment the user has made to the image. As illustrated, a bar appears above the selectable UI item 342 in the second stage 320 after the user performs a custom skin color balance operation on the image 3955.

  As described above, the application of some embodiments allows the user to move the custom skin color balance UI control 3925 to various locations on the image. With the help of a face detection algorithm, this application is able to locate a person's face in the image. However, if the image contains several faces, the user may not want to make a skin color balance adjustment based on the person's face selected by the application; rather, the user may not want to There is a possibility that skin color balance adjustment is desired based on the face of the person. The second stage 3910 allows the application to automatically place a custom skin color balance UI control 3925 on the background human face (the first face that the application has detected using the face detection algorithm). Show. However, the foreground person, not the background person, is the actual target point in the picture. Therefore, the movable custom skin color balance UI control 3925 allows the user to specify the correct location of the target face.

  The third stage 3915 is a transitional stage indicating that the user has selected the custom skin color balance UI control 3925 but has not started moving the custom skin color balance UI control 3925. The fourth stage 3920 shows the GUI 300 after the user has moved the custom skin color balance UI control 3925 to a different location on the image. In some embodiments, the user can move the custom skin color balance UI control 3925 by dragging the user's finger or dragging the cursor to a different location on the image. As shown, the user has moved the custom skin color balance UI control 3925 to the area of the image 3955 displaying the face of the foreground person. As a result of the movement of the custom skin color balance UI control 3925 to this new location, the application readjusts the color values of the image. In some embodiments, the application samples another pixel corresponding to the location selected through this custom skin balance UI control 3925. The application then devises an algorithm or mathematical equation (taking a set of parameters as input) to convert that color value into a predefined ideal facial color, and then in image 3955 The same algorithm or mathematical equation is applied to all the pixels. As shown, the portion of the image outside of the custom skin color balance UI control 3925 is shown readjusted as indicated by a different set of diagonal lines across that portion of the image 3955.

  In the above example shown in FIG. 39, the application performs a color balance operation on the image immediately after the color balance control 3925 appears on the image, and whenever the user moves the color balance UI control 3925, Readjust the image. In some other embodiments, in order to save the processing power of the device on which the application is executed, the application may receive further input from the user (e.g. selection of “Run Color Balance” selectable UI item, image Perform color balance operations on images only when receiving additional taps, etc.). In this way, the image is not adjusted each time the user moves the color balance UI control 3925, but only when the user determines the location and provides further input.

  FIG. 40 illustrates an exemplary skin color balance operation for an image performed by the image editing application of some embodiments. In some embodiments, the application first changes from a color space (eg, RGB color space) in which color values are defined to a YCC color space (eg, YIQ color space) before performing a skin color balance operation. , Convert the color value of the image. The conversion from the RGB color space to the YIQ color space first applies about 1/4 gamma to the color values in the RGB color space, and then applies a 3 × 3 matrix to convert the color values. This can be done by converting to the YIG color space. As illustrated, the color space 4000 represents a YCC color space in which the color values of the image are converted. This particular color space has three axes: an axis 4020 representing a range of red and green color values, an axis 4030 representing a range of blue and yellow color values, and various shades of gray ranges (ie, Brightness value range) is defined along the vertical axis 4010, where the bottom surface location 4040 represents black and the top surface location 4050 represents white. The brighter the color, the farther it appears from the bottom of the cylinder 4000 in this color space.

  Region 4060 represents all the color values of the image. Although region 4060 is shown as being located within one small region within color space 4000, the color values of some other images may occupy many different regions of color space 4000. When the user selects a location on the image using the skin color balance UI control, the application obtains the color value of the pixel corresponding to the selected location. In this figure, color 4080 represents the color of the pixel corresponding to the location specified by the white balance UI control.

  In some embodiments, this application defines two different ideal skin colors, one is ideal skin color under daylight conditions and the other is ideal under artificial light conditions. It is skin tone. The application of some embodiments performs an analysis on the image to detect whether the image was captured under daylight or artificial light, and for this operation, Select the corresponding ideal skin color.

  This application identifies an ideal skin color 4090 within the YCC color space 4000. The application then determines a color space conversion based on the difference between the ideal skin color value 4090 and the acquired color value 4080. In some embodiments, this color space conversion maps each color in color space 4000 to another color in the distorted color space. This color space conversion adjusts color values in a manner that essentially compresses the color space from all angles towards the white / black axis 4010. In other words, this application performs larger adjustments for color values with higher saturation (ie, color values farther away from the white / black axis 4010) and lower saturation. Make smaller adjustments to the color values you have (ie, color values that are closer to the white / black axis 4010). In some embodiments, the application does not adjust color values located along the white / black axis 4010 (ie, various shades of gray).

  A color space 4005 in FIG. 40 represents a color space after the color space 4000 is converted by this color space conversion. As shown, color 4080 in color space 4000 is mapped to color 4085 in color space 4005, color 4070 in color space 4000 is mapped to color 4075 in color space 4005, and color space 4000 is in color space 4000. Another color 4062 is mapped to color 4065 in color space 4005. As shown, color 4062 is farther from vertical axis 4010 than color 4070 in color space 4000. Therefore, color 4062 is drawn at a greater distance (ie, greater color adjustment) than color 4070. In some embodiments, the application adjusts the flesh color balance of the image based on the selected location by applying this color space transformation to each pixel in the image.

  Custom skin tone balance control 3925 offers many benefits. One of those benefits is that the user specifies a skin color region in the image (especially when there are two or more people in the image) that are intended for skin color balance operations. The ability to allow the selected skin color to be adjusted to the ideal skin color. Another benefit of the custom skin color balance control 3925 is related to the skin tone UI control 356 described above with reference to FIG. 3 and the on-image UI control 2025 and on-image UI control 2030 described above with reference to FIG. To do.

  As described above with reference to FIGS. 6 and 20, the skin tone UI control 356, the image UI control 2025, and the image UI control 2030 adjust the color temperature of the image based on the defined skin tone. By improving the skin tone. In some embodiments, the application defines the skin tone to be a range of color values within the color space. As described above, the range of the color value is broadly defined so as to target the human average skin tone. However, due to various conditions in which the image is captured (for example, lighting conditions, color cast, etc.), the face color of the target person in the image is within the range of color values defined as the skin tone by the application. , May not fit. A target person is a person that a user considers as a target in an image. In that case, the application can adjust the color value of the image based on the color of some other object (eg, the faces of people other than the subject person). The custom skin color balance UI control 3925 allows the user to designate the face of the person to be targeted in the image, so that the user can specify the skin tone UI control 356 or the on-image UI control 2025 and the on-image UI control. When operating 2030, (1) the face color of the target person is adjusted in balance (ie, adjusted to be the ideal face color), and (2) the color temperature of the image is Is adjusted based on the face color of the person. Specifically, after the user performs a skin color balance operation on the image, instead of using a predefined color value range as the skin tone, the application The ideal skin color is used as the skin tone relating to the skin tone adjustment.

  FIG. 41 illustrates an embodiment in which the skin tone UI control 356 is used to adjust the color of only a portion of the face rather than all of the faces in the image. Specifically, FIG. 41 illustrates this skin tone adjustment operation with four different stages 4105, 4110, 4115 and a stage 4120.

  The first stage 4105 is the same as the first stage 3905 of FIG. As shown, the user has selected an image 3955 to be displayed in the image display area 345. As described above, the image 3955 is a photograph of a person 3930 standing in the foreground and another person 3935 standing in the background. In this embodiment, a person 3930 standing in the foreground is a target person. The colors of the faces of the two people appear to be very different depending on the conditions (for example, lighting conditions, color cast, etc.) in which the image 3955 is captured. Specifically, the face color of the person 3935 falls within the range of color values defined by the application as skin tone, but the face color of the person 3930 falls outside the range of color values defined by the application as skin tone. . Image 3955 also shows mountains and seas on the right side of the image.

  As shown, the user has selected a selectable UI item 342 from the toolbar 340 to adjust the color of the image 3955 as indicated by the highlighting of the selectable UI item 342. The second stage 4110 shows the GUI 300 after the user has selected the skin tone UI control 356, as indicated by the highlighting of the skin tone UI control 356. Selection of the skin tone UI control 356 includes performing a gesture (eg, placing a finger, pointing, or fingering) at a location on the device that has a touch or near-touch sensitive screen that displays the skin tone UI control 356. By tapping) or by placing the cursor on the skin tone UI control 356 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.).

  The third stage 4115 shows the GUI 300 after the user has started improving the skin tone of the image by moving the knob of the skin tone UI control 356 to the right as indicated by the arrow 4125. In some embodiments, the user can move the knob of the skin tone UI control 356 by dragging the user's finger (or dragging the cursor) to a different location. Based on this adjustment to the skin tone UI control 356, the application adjusts the color temperature of the image 3955 based on a predefined skin tone, as indicated by the diagonal lines across the image 3955.

  The fourth stage 4120 shows the GUI 300 after the user has further adjusted the skin tone of the image by moving the knob of the skin tone UI control 356 further to the right as indicated by the arrow 4130. As shown, the image 3955 at this stage 4120 is shown more adjusted than the image at the third stage 4115, as shown by the denser diagonal lines across the entire image 3955.

  As shown in the embodiment shown in FIG. 41, the color of this image is simply the skin tone UI because the face color of the subject person cannot fall within the color value range predefined by the application. Only using the control 356 or the on-image UI control 2025 and the on-image UI control 2030 cannot be adjusted based on the skin color of the target person. FIG. 42 shows the same operations performed in FIG. 41 except that the user uses the skin color balance UI tool 3925 to adjust the face of the foreground person 3930 before adjusting the skin tone of the image. Is selected. Specifically, FIG. 42 shows this skin tone adjustment operation with four different stages 4205, 4210, 4215, and a stage 4220.

  The first stage 4205 is the same as the fourth stage 4120 in FIG. As shown in the figure, the user has completed the skin color balance operation by using the skin color balance UI control 3925. Specifically, the user uses the skin color balance UI control 3925 to select the face of the foreground person 3930 for this skin color balance operation. As a result, the face color of the person 3930 is adjusted to the ideal skin color defined by this image.

  The second stage 4210 shows the GUI 300 after the user has selected the skin tone UI control 356 as indicated by the highlighting of the skin tone UI control 356. Selection of the skin tone UI control 356 includes performing a gesture (eg, placing a finger, pointing, or fingering) at a location on the device that has a touch or near-touch sensitive screen that displays the skin tone UI control 356. By tapping) or by placing the cursor on the skin tone UI control 356 and providing input (eg, clicking on a cursor control device, pressing a hot key, etc.).

  Third stage 4215 shows GUI 300 after the user has started adjusting the skin tone of the image by moving the knob of skin tone UI control 356 to the right as indicated by arrow 4225. In some embodiments, the user can move the knob of the skin tone UI control 356 by dragging the user's finger (or dragging the cursor) to a different location. Since the user is performing a skin color balance operation on the image, this application instead of using the range of values defined as the normal skin tone to adjust the color temperature of the image 3955. Use a range of values defined as its ideal skin tone. As a result of user input through the skin tone UI control 356, the color value of the image is adjusted based on the color of the face of the foreground person, as shown by the 45 degree diagonal lines across the image 3955. .

  The fourth stage 4220 shows the GUI 300 after the user has further improved the skin tone of the image by moving the knob of the skin tone UI control 356 further to the right as indicated by the arrow 4230. As shown, the image 3955 at this stage 4220 is shown more adjusted than the image at the third stage 4215, as shown by the denser 45 degree diagonal lines across the image 3955. .

  The GUI of the image editing application shown in the above figure is shown under the assumption that the device on which the image editing application is executed has a sufficiently large screen to display the GUI. However, some of the devices on which image editing applications are executed have limited screen sizes to display UI items in a way that they are displayed in a larger screen on devices with larger items. There may be. Also, a larger screen of a larger device may be considered limited if the device is held in a different orientation (eg, portrait). In some embodiments, the image editing application adapts to this limited screen space by displaying different sets of different UI items at different time instances.

  FIG. 43 conceptually illustrates an example of invoking a set of color balance tools through the GUI 1200 of some embodiments, with three different stages 4301, 4302, and stage 4303. The first stage 4301 is the same as the sixth stage 1206 of FIG. As shown, the tool navigation pane 1225 within the GUI 1200 includes a set of color adjustment tools 1260. This set of color adjustment tools 1260 includes a UI item 1275 for invoking a set of color balance tools.

  Second stage 4302 indicates that the user has selected UI item 1275. In some embodiments, the user performs a gesture (eg, puts a finger, points at, or taps a finger) at a location on a device that has a touch or near touch sensitive screen that displays a UI item 1275. ), The UI item 1275 can be selected. As a result of this selection, as indicated by the third stage 4303, a set of UI items 4315 representing a set of color balance tools appears on the GUI 1200. In some embodiments, the set of UI items is ordered throughout the GUI 1200, as shown in this third stage 4303. In other embodiments, the set of UI items 4315 occupies only a portion of the screen, allowing the user to still view images on the GUI. Although not shown herein, the user selects any one of the color balance tools (including the custom white balance tool and the custom face balance tool) by selecting one of the UI items 4315. Can be called.

  FIG. 44 conceptually illustrates a process 4400 for performing a color balance operation on an image, as shown in FIGS. In some embodiments, this process is performed by an image editing application. The process begins by receiving (at 4405) a color balance tool selection. In some embodiments, the color balance tool can be a custom white balance tool or a custom skin color balance tool. Other embodiments can provide additional color balance tools.

  The process then displays a color balance UI control on the image (at 4410) to select a location on the image. In some embodiments, this process displays a color balance UI control in a default area on the image (eg, the center of the image). In some other embodiments, the process performs a detection algorithm (e.g., a face detection algorithm) to create a color in an area on the image that corresponds to a particular feature (e.g., a human face). Place balance UI controls.

  Various embodiments implement color balance UI control in various ways. In some embodiments, the color balance UI control includes a closed border (eg, rectangle, ellipse, etc.), and within the closed border is an indicator for accurately indicating the location on the image. Have. In some embodiments, the color balance UI control can appear like the color balance UI control 3925 of FIG. In some of these embodiments, the color balance UI control also zooms in (ie, enlarges) the portion of the image that is inside the closed border. This zoomed-in version provides better accuracy when selecting a location on the image.

  The process then adjusts (at 4415) the color balance of the image based on the location indicated by the color balance UI control. In some embodiments, this adjustment may result in a pixel color value corresponding to the selected location being set to a pre-defined set of specific color values (e.g., a color value for white, a color value for gray, a predefined color value). To define an algorithm or mathematical equation to change to a color value for the ideal skin tone. The process then applies the same algorithm or mathematical equation to every pixel in the image.

  In some embodiments, this process provides a preview of the adjusted image by displaying an adjusted version of the portion of the image that is outside the closed border of the color balance UI control. In these embodiments, the portion of the image inside the closed border of the color balance UI control is kept unchanged to show the difference between the edited and unedited version of the image.

  The process then determines (at 4420) whether any movement of the color balance UI control has been received. If a color balance UI control move is received, the process re-adjusts (at 4425) the color balance of the image based on the new location of the color balance UI control. This process repeats operations 4420-4425 until no further movement is received. If no movement for the color balance UI control is received, the process ends.

  FIG. 45 conceptually illustrates a process 4500 for performing a color balance operation on an image. In some embodiments, process 4500 is performed by the image editing application during operations 4415 and 4425 of process 4400 after the user selects a location on the image through the color balance UI control. This process begins by obtaining (at 4505) the pixel corresponding to the selected location on the image. The process then identifies (at 4510) the desired color within the color space for the acquired pixel. In some embodiments, if the user has selected a custom white balance control tool at operation 4405 of process 4400, this process identifies the gray closest to the color of the acquired pixel within the color space. . The operation of specifying the gray color is described above with reference to FIG. If the user has selected a custom skin color balance tool at operation 4405 of process 4400, this process identifies a predefined ideal skin color that is closest to the color of the acquired pixel in the color space. . In some embodiments, the application may have several sets of predefined ideal skin colors for various conditions (eg, ideal skin color set for daylight and ideal skin color for artificial light). Set). In these embodiments, the process performs an analysis to determine the conditions under which the image was captured and identifies the corresponding ideal skin color.

  The process then determines (at 4515) a color space transformation based on the acquired pixel color and the identified color. In some embodiments, this color adjustment operation involves a specific color conversion algorithm or mathematical equation. After determining the color adjustment operation, the process applies (at 4520) the color space transformation to every pixel of the image. The process then ends.

  FIG. 46 illustrates an image editing application 4600 of some embodiments that performs an image color balance adjustment operation. In some embodiments, image editing application 4600 performs process 4400 and process 4500. As shown in FIG. 46, the image editing application 4600 includes an image processor 4610, an image preview module 4615, a color space conversion generator 4625, a content analysis module 4640, and a color adjustment module 4630.

  When the UI module 4605 receives a user selection of a color balance tool (eg, custom white balance tool, custom skin color balance tool, etc.), the UI module 4605 passes the selection information to the image processor 4610. Next, the color balance UI control module 4610 displays the color balance UI control on the image. In some embodiments, the image processor 4610 displays a color balance UI control in a default location for each image (eg, in the middle of the image). In some other embodiments, the color balance UI control module 4610 sends an image to the content analysis module 4640 to perform feature detection operations (eg, face detection operations) on the images, and features are detected. The color balance UI control is displayed at a location (for example, on the face of a person on the image). The image processor 4610 then obtains the color value of the pixel corresponding to the location of the color balance UI control and passes information about the color value to the color space conversion generator 4625.

  In some embodiments, the color space conversion generator 4625 performs the process 4500 of FIG. Specifically, the color space conversion generator 4625 obtains the color value of the pixel corresponding to the location of the color balance UI control. The color space conversion generator then identifies a set reference color (eg, gray, ideal skin color, etc.) associated with the color balance tool selected by the user. The color space conversion generator 4625 determines a color adjustment that changes the color value of a pixel in the image to its specified and set reference color. The color space conversion generator 4625 then generates a color space conversion based on the determined color adjustment. In some embodiments, this color space transformation is an M × M matrix (eg, 3 × 3 matrix) that is generated based on the determined color adjustment. In some embodiments, this color space conversion takes each color defined within the color space and outputs a different color using the determined color value adjustment.

  The color space conversion generator 4625 then passes the generated color space conversion to the image processor 4610. The image processor 4610 then sends the image and color space conversion to the color adjustment engine 4630. The color adjustment engine applies color space conversion to all pixels in the image and sends the adjusted image back to the image processor 4610. The image processor stores the adjusted image in the media storage device 4635.

  In some embodiments, the image editing application provides the user with a real-time preview of the edited image. In these embodiments, the image processor 4610 sends the edited image to the image preview module 4615. Various embodiments provide previews to the user using various techniques. In some embodiments, the image preview module 4615 displays an edited version of the portion of the image outside the color balance UI control while displaying an unedited version of the portion of the image inside the color balance UI control. . Thus, the user can easily visually recognize the difference between the edited version and the unedited version in real time. In other embodiments, only the portion of the image inside the color balance UI control is edited and the portion of the image outside the color balance UI control is kept unedited. Furthermore, in other embodiments, the entire image is edited.

  When the user changes the location of the color balance UI control, the UI module 4605 passes information about the new location of the color balance UI control to the image processor 4610. The image processor 4610 then obtains the color value of the pixel corresponding to the new location and passes the color value to the color space conversion generator 4625. The color space conversion generator 4625 generates a new color space conversion based on the acquired color value, and sends the generated color space conversion back to the image processor 4610. The image processor 4610 then sends the image and color space transformations to the color adjustment engine 4630 to readjust the image. Color adjustment engine 4630 applies color space conversion to the image and sends the adjusted image to image processor 4610.

IV. Image Browsing, Editing, and Configuration Application The above-described figures illustrate various examples of image browsing, editing, and configuration application GUIs in some embodiments. FIG. 47 shows a detailed view of a GUI 4700 of some embodiments for viewing, editing and composing images. The GUI 4700 is described with reference in part to FIG. 48, which conceptually illustrates a data structure 4800 for an image as stored by an application of some embodiments.

  The data structure 4800 includes an image ID 4805, image data 4810, editing instructions 4815, an image cache version 4840, and any additional data 4850 related to the image. The image ID 4805 is a unique identifier for the image that, in some embodiments, is used by the collection data structure to refer to images stored in the collection. Image data 4810 is the actual full-size pixel data (eg, a series of color space channel values for each pixel in the image or encoded version thereof) for displaying the image. In some embodiments, this data can be stored in a database of image viewing, editing, and configuration applications, or can be stored on the same device along with data of another application. In some embodiments, this additional application is another image composition application that runs on the device, on which image viewing, editing, and composition work.

  This data structure can therefore store a pointer to a local file associated with the application, or an ID that can be used to query another application's database. In some embodiments, when an application uses an image in a journal or performs an edit on an image, the application automatically creates a local copy of the image file that includes the image data.

  Edit instructions 4815 include information related to any edits that the user has applied to the image. In this manner, the application stores the image in a non-destructive manner so that the application can easily return from the edited version of the image to the original at any time. For example, a user can apply a saturation effect to an image, leave the application, and then reopen the application at another opportunity to remove the effect. The edits stored in these instructions can be cropping and rotation, full image exposure and color adjustments, local adjustments, and special effects, and other edits that affect the pixels of the image. Some embodiments store these editing instructions in a particular order so that the user can view different versions of the image to which only a particular set of edits have been applied.

  In some embodiments, the edit instruction 4815 is implemented as a list 4860 of edit operations. List 4860 includes editing operations such as edits 4861, 4862, 4863, and edit 4865. Each editing operation in the list 4860 specifies parameters necessary for performing the editing operation. For example, an edit operation 4865 in the list 4860 specifies an edit to the image to which the saturation effect is applied using the color selection parameter θ.

  In some embodiments, list 4860 records a sequence of editing operations initiated by the user to create the final edited image. In some embodiments, the list 4860 provides the image editing application to generate an output image for display because some embodiments define a specific order for the various possible edits provided by the application. Therefore, the editing commands are stored in the order in which editing is applied to the image. For example, some embodiments define the saturation effect as one of editing operations that should be applied later than other editing operations such as crop and rotate, full image exposure, and color adjustment. . In some of these embodiments, the list 4860 is a location where editing instructions relating to saturation effects are applied later than some of the other editing operations (eg, edits 4861 to 1363) ( That is, it is stored in the edit 4865).

  Cache image version 4840 stores versions of images that are often accessed and displayed, so that the application does not need to repeatedly generate these images from full-size image data 4810. For example, the application often stores a thumbnail for the image as well as a display resolution version (eg, a version adapted for the image display area). In some embodiments, each time an edit is applied, it creates a new thumbnail for the image and replaces the previous thumbnail. Some embodiments store multiple display resolution versions, including the original image and one or more edited versions of the image.

  Finally, the image data structure 4800 includes additional data 4850 (eg, face location and size, etc.) that the application can store with the image. In some embodiments, this additional data may include Exchangeable image file format (Exif) data, caption data, shared image data, tags on the image, or any other type of data. Exif data includes various information stored by the camera that captured the image, such as camera settings, GPS data, and time stamps. The caption is a description of the image input by the user. A tag is information that allows an application to associate with an image, such as marking an image as favorite, flagged, hidden, etc.

  One skilled in the art will recognize that the image data structure 4800 is just one possible data structure that an application can use to store the information needed for an image. For example, different embodiments may store more information or less information, store information in a different order, and so on.

  Returning to FIG. 47, the GUI 4700 includes a thumbnail display area 4705, an image display area 4710, a first toolbar 4715, a second toolbar 4720, and a third toolbar 4725. A thumbnail display area 4705 displays thumbnails of images in the selected collection. A thumbnail is a small representation of a full-size image, and in some embodiments represents only a portion of the image. For example, the thumbnails in the thumbnail display area 4705 are all square regardless of the aspect ratio of the full-size image. To determine the portion of the rectangular image to use for the thumbnail, the application identifies the smaller dimensions of the image and uses the central portion of the image in the longer direction. For example, for an image of 1600 × 1200 pixels, this application uses a 4700 × 1200 square. In order to further refine the selection for thumbnails, some embodiments identify the center of all faces in the image (using a face detection algorithm) and are then clipped using that location. Place it at the center of the thumbnail in the direction Therefore, if all the faces in a 1600 × 1200 image are theoretically located on the left side of the image, this application uses the leftmost 4700 columns of pixels rather than truncating 200 columns on both sides.

  After determining the portion of the image to use for the thumbnail, the image viewing application generates a low resolution version of the image (eg, using pixel blending techniques and other techniques). Some embodiments of the application store thumbnails for images as a cached version 4840 of the image. Thus, when a user selects a collection, the application identifies all of the images in that collection (through the collection data structure) and is cached in each image data structure for display in the thumbnail display area. Access thumbnails.

  The user can select one or more images in the thumbnail display area (eg, through the various touch interactions described above, or through other user input interactions). The selected thumbnail is displayed using highlighting or other selection display. In the thumbnail display area 4705, the thumbnail 4730 is selected. Further, as shown, the thumbnail display area 4705 of some embodiments shows several images in the collection that are flagged (eg, with a tag for the flag set to “Yes”). . In some embodiments, this text is selectable to display only flagged image thumbnails.

  The application displays the selected image in the image display area 4710 with a resolution larger than the corresponding thumbnail. Since these images often have a higher resolution than the display device, they are typically not displayed at the full size of the image. Therefore, the application of some embodiments stores a cached version of the image 4840 that is designed to fit within the image display area. The image in the image display area 4710 is displayed with the aspect ratio of the full-size image. When one image is selected, the application displays the image as large as possible within the image display area without truncating any part of the image. If multiple images are selected, this application will maintain their visual weighting by using approximately the same number of pixels for each image, even if they have different aspect ratios. Display images in various ways.

  The first toolbar 4715 displays title information (for example, the name of the collection shown in the GUI, the caption attached to the image currently selected by the user). Further, the toolbar 4715 includes a first set of GUI items 4735-1238 and a second set of GUI items 4740-1243.

  The first set of GUI items includes a “back” button 4735, a grid button 4736, a help button 4737, and a cancel button 4738. A “back” button 4735 allows the user to navigate back to the collection configuration GUI, from which the user can select various collections of images (eg, albums, events, journals, etc.). can do. Depending on the selection of the grid button 4736, the application moves the thumbnail display area onto or out of the GUI (eg, via a slide animation). In some embodiments, the user can also slide the thumbnail display area onto or from the GUI via a swipe gesture. Help button 4737 activates a context-sensitive help function that identifies the current set of tools active for the user and provides the user with a help display for those tools that briefly describes those tools. . In some embodiments, this help display can be selected to access additional information about the tool. Selection of the cancel button 4738 causes the application to remove the latest edit to the image regardless of whether the edit is cropping, color adjustment, or the like. To perform this cancellation, some embodiments remove the latest instruction from the set of edit instructions 4815 stored with the image.

  The second set of GUI items includes a share button 4740, an information button 4741, a “show original” button 4742, and an edit button 4743. The share button 4740 allows the user to share images in a variety of ways. In some embodiments, among other things, the user can send the selected image to another compatible device on the same network (eg, WiFi® or Bluetooth® network) to create an image hosting web Images can be uploaded to a site or social media website and a journal (ie, a presentation of arranged images that can add additional content) can be created from the selected set of images.

  An information button 4741 activates a display area that displays additional information about one or more selected images. The information displayed in the activated display area may include some or all of the Exif data (eg, camera settings, time stamps, etc.) stored for the image. When multiple images are selected, some embodiments display only Exif data that is common to all of the selected images. Some embodiments (i) display a map showing where the image was captured according to GPS data (if this information is available), and (ii) any photo sharing website Above, an additional tab is included in the information display area to display the comment stream for the image. In order to download this information from the website, the application uses the object ID for the image stored with the shared image data for the image and sends this information to the website. A comment stream and possibly additional information is received from the website and displayed to the user.

  “Show original” button 4742 allows the user to switch between the original version of the image and the current edited version of the image. When the user selects this button, the application displays the original version of the image without any editing instructions 4815 applied. In some embodiments, an appropriately sized image is stored as one of the cached versions of the image 4840 so that the image can be quickly accessed. When the user selects button 4742 again, the application displays the edited version of the image to which edit instruction 4815 has been applied.

  Edit button 4743 allows the user to enter or exit edit mode. If the user has selected one of the set of editing tools in the toolbar 4720, the edit button 4743 returns the user to view and configure mode, as shown in FIG. If the user selects edit button 4743 while in view mode, the application returns to the last set of editing tools used in the order shown in toolbar 4720. That is, the items in the toolbar 4720 are arranged and configured in a specific order, and the edit button 4743 activates the right end of those items that have been edited on the selected image.

  Toolbar 4720 includes five items 4745 to 1249 arranged in a particular order from left to right as described above. Crop item 4745 activates a crop and rotate tool that allows the user to align the distorted image and remove unwanted portions of the image. Exposure item 4746 activates a set of exposure tools that allow the user to modify the black point, shadow, contrast, brightness, highlight, and white point of the image. In some embodiments, this set of exposure tools is a set of sliders that work together in various combinations to modify the tone attributes of the image. Color item 4747 activates a set of color tools that allow the user to modify saturation and natural saturation, as well as saturation by color (eg, blue or green pixels) and white balance. Turn into. In some embodiments, some of these tools are presented as a set of sliders. Brush item 4748 activates a set of correction tools that allow the user to localize modifications to the image. Using this brush, the user removes red eyes and spots by performing a rubbing action on the image, and applies saturation and other features to localized portions of the image, or Can be removed. Finally, effect item 4749 activates a set of special effects that the user can apply to the image. These effects include duotone effects, coarse grain effects, gradations, tilt shifts, non-realistic desaturation effects, gray scale effects, and various filters. In some embodiments, the application presents these effects as a set of items that fan out from the toolbar 4725.

  As described above, the UI items 4745 to 1249 are arranged in a specific order. This order follows the order in which the user best applies these five different types of edits. Thus, the edit instructions 4815 are stored in this same order in some embodiments. When the user selects one of items 4745 to 1249, some embodiments apply only edits from the tool to the left of the selected tool to the displayed image (but other edits). Is kept stored in instruction set 4815).

  Toolbar 4725 includes a set of GUI items 4750-1254 as well as setting items 4755. The automatic correction item 4750 automatically executes correction editing (for example, clear red-eye removal, color balance adjustment, etc.) on the image. A rotate button 4751 rotates any selected image. In some embodiments, each time this rotation button is pressed, the image rotates 90 degrees in a particular direction. The automatic correction includes a predetermined set of editing instructions that, in some embodiments, are placed in the instruction set 4815. Some embodiments perform an analysis of the image and then define a set of instructions based on the analysis. For example, the automatic correction tool attempts to detect red eyes in the image, but if no red eyes are detected, no instructions are generated to correct the red eyes. Similarly, automatic color balance adjustment is based on image analysis. The rotation generated by the rotation button is also stored as an edit command.

  The flag button 4752 tags any selected image as flagged. In some embodiments, collection flagged images can be displayed without any flagless images. Favorites button 4753 allows the user to mark any selected image as a favorite. In some embodiments, this tags the image as a favorite and also adds the image to the collection of favorite images. Hide button 4754 allows the user to tag an image as hidden. In some embodiments, the non-displayed image is not displayed in the thumbnail display area and / or is not displayed when the user cycles through the images in the collection in the image display area. As described above with reference to FIG. 48, many of these features are stored as tags in the image data structure.

  Finally, settings button 4755 activates a context-sensitive menu that provides different menu options depending on the currently active toolset. For example, in browse mode, the menus in some embodiments include an option to create a new album, an option to set a primary photo for the album, an option to copy settings between photos, and others Provides options. If various editing tool sets are active, this menu provides options related to the particular active tool set.

  Those skilled in the art will recognize that the image viewing and editing GUI 4700 is just one example of many possible graphical user interfaces for image viewing, editing and composition applications. For example, the various items can be arranged in different regions or in different orders, and some embodiments can include items with additional functionality or different functionality. The thumbnail display area of some embodiments can display thumbnails that match the aspect ratio of the corresponding full-size image.

V. Electronic System Many of the functions and applications described above are implemented as a software process that is specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). If these instructions are executed by one or more computing or processing units (eg, one or more processors, processor cores, or other processing units), the instructions are sent to the processing units by their instructions. The action indicated by is executed. Examples of computer readable media include CD-ROM, flash drive, random access memory (RAM) chip, hard drive, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM) ) And the like, but is not limited thereto. Computer-readable media does not include carrier waves and electronic signals transmitted wirelessly or via wired connections.

  As used herein, the term “software” is intended to include firmware residing in read-only memory, or applications stored in magnetic storage, that can be loaded into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as subdivisions of larger programs while maintaining separate software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of individual programs that together implement the software invention described herein is within the scope of the present invention. In some embodiments, a software program defines one or more specific machine implementations that, when installed to run on one or more electronic systems, perform and execute the operations of the software program. .

A. Mobile Device The image editing and viewing application of some embodiments runs on a mobile device. FIG. 49 is an example of such a mobile computing device architecture 4900. Examples of mobile computing devices include smartphones, tablets, and laptops. As shown, mobile computing device 4900 includes one or more processing units 4905, a memory interface 4910, and a peripheral device interface 4915.

  Peripheral device interface 4915 is coupled to various sensors and subsystems, including camera subsystem 4920, wireless communication subsystem 4925, audio subsystem 4930, I / O subsystem 4935, and the like. Peripheral device interface 4915 allows communication between processing unit 4905 and various peripheral devices. For example, a direction sensor 4945 (eg, a gyroscope) and an acceleration sensor 4950 (eg, an accelerometer) are coupled to the peripheral device interface 4915 to facilitate direction and acceleration functions.

  The camera subsystem 4920 is coupled to one or more photosensors 4940 (eg, charge coupled device (CCD) photosensors, complementary metal oxide semiconductor (CMOS) photosensors, etc.). A camera subsystem 4920 coupled with the optical sensor 4940 facilitates camera functions, such as capturing image and / or video data. The wireless communication subsystem 4925 serves to facilitate communication functions. In some embodiments, the wireless communication subsystem 4925 includes a radio frequency receiver and transmitter, and an optical receiver and transmitter (not shown in FIG. 49). In some embodiments, these receivers and transmitters are implemented to operate via one or more communication networks such as a GSM network, a Wi-Fi network, a Bluetooth network, and the like. Audio subsystem 4930 is coupled to a speaker (e.g., to output various acoustic effects associated with various image manipulations) to output audio. In addition, the audio subsystem 4930 is coupled to a microphone to facilitate voice-enabled functions such as voice recognition and digital recording.

  The I / O subsystem 4935 is responsible for transfer between input / output peripherals such as displays, touch screens, etc., and the data bus of the processing unit 4905 through the peripheral interface 4915. The I / O subsystem 4935 includes a touch screen controller 4955 and other input controller 4960 to facilitate transfer between the input / output peripherals and the data bus of the processing unit 4905. As shown, touch screen controller 4955 is coupled to touch screen 4965. Touch screen controller 4955 detects touch and movement on touch screen 4965 using any of a plurality of touch sensitive techniques. The other input controller 4960 is coupled to other input / control devices, such as one or more buttons. Some embodiments include a near touch sensitive screen and corresponding controller capable of detecting near touch interaction instead of or in addition to touch interaction.

  Memory interface 4910 is coupled to memory 4970. In some embodiments, memory 4970 may be volatile memory (eg, fast random access memory), non-volatile memory (eg, flash memory), a combination of volatile and non-volatile memory, and / or any other Including types of memory. As shown in FIG. 49, the memory 4970 stores an operating system (OS) 4972. The OS 4972 includes instructions for processing basic system services and instructions for executing hardware dependent tasks.

  Memory 4970 also facilitates communication instructions 4974 to facilitate communication with one or more additional devices, graphical user interface instructions 4976 to facilitate graphical user interface processing, image-related processing and functions. Image processing instructions 4978, input processing instructions 4980 for facilitating input related (eg touch input) processing and functions, audio processing instructions 4982 for facilitating audio related processes and functions, and cameras Also included are camera instructions 4984 to facilitate related processing and functions. The instructions described above are merely exemplary, and memory 4970 includes additional instructions and / or other instructions in some embodiments. For example, memory for a smartphone may include telephone instructions to facilitate telephone related processes and functions. The instructions identified above need not be implemented as separate software programs or modules. Various functions of the mobile computing device can be implemented as hardware and / or software, including one or more signal processing circuits and / or application specific integrated circuits.

  The components shown in FIG. 49 are shown as separate components, but those skilled in the art will recognize that two or more components can be integrated into one or more integrated circuits. Let's go. Furthermore, two or more components can be coupled together by one or more communication buses or signal lines. Also, although many of these functions are described as being performed by one component, the functions described in connection with FIG. 49 may be divided into two or more integrated circuits. Those skilled in the art will understand what can be done.

B. Computer System FIG. 50 conceptually illustrates another example of an electronic system 5000 in which some embodiments of the present invention are implemented. The electronic system 5000 can be a computer (eg, desktop computer, personal computer, tablet computer, etc.), telephone, PDA, or any other type of electronic or computing device. Such electronic systems include various types of computer readable media and interfaces for various other types of computer readable media. The electronic system 5000 includes a bus 5005, a processing unit 5010, a graphics processing unit (GPU) 5015, a system memory 5020, a network 5025, a read only memory 5030, a permanent storage device 5035, an input device 5040, and an output device 5045.

  A bus 5005 collectively represents all system buses, peripheral device buses, and chipset buses that connect a large number of internal devices of the electronic system 5000 in a communicable manner. For example, bus 5005 communicatively couples processing unit 5010 to read only memory 5030, GPU 5015, system memory 5020, and permanent storage 5035.

  From these various memory units, processing unit 5010 obtains instructions to execute and data to process in order to execute the process of the present invention. This processing unit may be a single processor or a multi-core processor in different embodiments. Some instructions are passed to the GPU 5015 and executed by the GPU 5015. The GPU 5015 can offload various computations provided by the processing unit 5010 or complement image processing. In some embodiments, such functionality may be provided using the CoreImage kernel shading language.

  Read only memory (ROM) 5030 stores static data and instructions required by processing unit 5010 and other modules of the electronic system. On the other hand, the permanent storage device 5035 is a read / write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 5000 is off. Some embodiments of the present invention use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 5035.

  Other embodiments use removable storage devices (floppy disks, flash memory devices, etc., and their corresponding drives) as permanent storage devices. Similar to permanent storage device 5035, system memory 5020 is a read and write memory device. However, unlike the storage device 5035, the system memory 5020 is a volatile read and write memory, such as a random access memory. System memory 5020 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the processes of the present invention are stored in system memory 5020, permanent storage 5035, and / or read only memory 5030. For example, according to some embodiments, various memory units include instructions for processing multimedia clips. From these various memory units, processing unit 5010 obtains instructions to execute and data to process in order to execute the processes of some embodiments.

  Bus 5005 also connects to input device 5040 and output device 5045. The input device 5040 allows the user to communicate information and select commands to the electronic system. Input device 5040 may include an alphanumeric keyboard and pointing device (also referred to as a “cursor control device”), a camera (eg, a webcam), a microphone for receiving voice commands, or the like. The output device 5045 displays an image generated by the electronic system or outputs data in another manner. Examples of the output device 5045 include a printer, a display device such as a cathode ray tube (CRT) or a liquid crystal display (LCD), and a speaker or a similar audio output device. Some embodiments include devices such as touch screens that function as both input and output devices.

  Finally, as shown in FIG. 50, bus 5005 also couples electronic system 5000 to network 5025 through a network adapter (not shown). In this manner, a computer is considered part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an intranet), or part of a network of multiple networks (eg, the Internet). can do. Any or all of the components of the electronic system 5000 can be used with the present invention.

  Some embodiments include a storage device and memory for storing computer program instructions in a microprocessor, machine readable or computer readable medium (also referred to as computer readable storage medium, machine readable medium, or machine readable storage medium). Including electronic components. Some examples of such computer readable media include RAM, ROM, read-only compact disc (CD-ROM), recordable compact disc (CD-R), rewritable compact disc (CD-RW), read-only. Digital multipurpose disc (eg, DVD-ROM, dual layer DVD-ROM), various recordable / rewritable DVDs (eg, DVD-RAM, DVD-RW, DVD + RW, etc.), flash memory (eg, SD card, mini SD) Cards, micro SD cards, etc.), magnetic and / or solid state hard drives, read-only recordable Blu-Ray® disks, ultra high density optical disks, any other optical or magnetic media, and floppy disks. . The computer readable medium can store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code as created by a compiler and files containing high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter. .

  Although the above discussion primarily refers to microprocessors or multi-core processors executing software, some embodiments may include application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), Performed by one or more integrated circuits. In some embodiments, such an integrated circuit executes instructions stored in the circuit itself. Further, some embodiments execute software stored in a programmable logic device (PLD), ROM, or RAM device.

  As used herein and in any claim, the terms “computer”, “server”, “processor”, and “memory” all refer to an electronic device or other technical device. These terms exclude people or groups of people. For the purposes of this specification, the term “display” or “displaying” means displaying on an electronic device. As used herein and in any claim, the terms “computer-readable medium” and “machine-readable medium” refer to tangible physical storage that stores information in a form readable by a computer. Fully limited to the entity. These terms exclude all wireless signals, signals downloaded over the wire, and any other transient signals.

  Although the invention has been described with reference to numerous specific details, those skilled in the art can embody the invention in other specific forms without departing from the spirit of the invention. Will be recognized. For example, many of the figures show various touch gestures (eg, tap, double tap, swipe gesture, press and hold gesture, etc.). However, many of the illustrated operations are performed via different touch gestures (eg, swipe instead of taps) or by non-touch input (eg, cursor controller, keyboard, touch pad / trackpad, near touch sensitive). (Using a screen etc.). In addition, some figures (including FIGS. 16, 17, 19, 26, 28, 36, 44, and 45) conceptually illustrate the process. The specific operations of these processes may not be performed in the exact order shown and described. Certain operations may not be performed in one continuous sequence of operations, and various specific operations may be performed in various embodiments. Furthermore, this process can be implemented using several sub-processes or as part of a larger macro process. Accordingly, it will be appreciated by one skilled in the art that the present invention is not limited by the foregoing illustrative details, but rather is defined by the appended claims.

  Furthermore, the controls for setting a single adjustment value used to perform various image editing operations are shown in FIGS. 3, 4, 6, 7, 9, 14, 15, 20, 21, 22. , 23, 24, 31, 32, 33, 34 and FIG. 35 are shown as slider controls. The slider in such an embodiment provides a visual display of the set value when the knob is slid along the slider to set a value for that slider. However, in some embodiments, the slider controls shown in any of these figures may include vertical slider controls, pull-down menus, value entry boxes, incremental tools activated by keyboard keys, and other range-related UIs. Any other control that can receive a value (eg, a single value), such as a control (eg, a dial, button, numeric field, etc.) can be substituted. Similarly, the slider controls in those figures may be expressed as set by a finger gesture (eg, placing one or more fingers, pointing, tapping) on the touch-sensitive screen, or The slider control is simply displayed at a specific position without displaying how the slider control is moved to a predetermined position. The controls of FIGS. 3, 4, 6, 7, 9, 14, 15, 20, 21, 22, 23, 24, 31, 32, 33, 34, and FIG. 35 are also cursor control in some embodiments. By a device (eg, mouse or trackball), stylus, keyboard, finger gesture near the touch sensitive screen (eg, placing, pointing, tapping one or more fingers), or any other control system However, those skilled in the art will appreciate that they can also be activated and / or set. Accordingly, it will be appreciated by one skilled in the art that the present invention is not limited by the foregoing illustrative details, but rather is defined by the appended claims.

Claims (42)

A non-transitory machine-readable medium having a computer program for adjusting color values of an image including a set of pixels when executed by at least one processing unit, wherein each pixel is defined in a first color space A computer program comprising:
Receiving user input for a user interface item associated with a type of content in the image;
Converting the color value of the image defined in the first color space to a color value defined in a second color space;
Identifying a subset of pixels having transformed color values that fall within a range of color values associated with the content type;
Performing a color adjustment operation on the identified subset of pixels;
A non-transitory machine-readable medium containing a set of instructions for.   The non-transitory machine-readable medium of claim 1, wherein the range of color values associated with the content type is predefined with respect to the content type.   The non-transitory machine-readable medium of claim 1, wherein the type of content includes one of sky, foliage, and skin tone.   The non-transitory machine-readable medium of claim 1, wherein the identified pixels do not form a continuous area in the image.   The color adjustment operation comprises: an operation for adjusting a saturation of the identified subset of pixels; an operation for adjusting a brightness of the identified subset of pixels; and a contrast of the identified subset of pixels. The non-transitory machine-readable medium of claim 1, comprising adjustments for adjusting.   The non-transitory machine-readable medium of claim 1, wherein the color adjustment operation is performed within the first color space. A non-transitory machine readable medium storing a program that, when executed by at least one processing unit, provides a graphical user interface for editing an image, the graphical user interface comprising:
A display area for displaying different individual images displaying one or more different content types;
(I) receiving user input to adjust the color values of the displayed image; (ii) identifying a set of pixels having color values associated with the type of content in the image; and (Iii) a color adjustment tool for adjusting the color value of the identified set of pixels based on the user input;
A non-transitory machine-readable medium.   The non-transitory machine-readable medium of claim 7, wherein the color adjustment tool includes a range slider.   The non-transitory machine-readable medium of claim 8, wherein the user input moves a knob of the slider along the slider.   The non-transitory machine-readable medium of claim 9, wherein the color adjustment tool adjusts a color value of the identified set of pixels based on a distance traveled by the knob along the slider. An apparatus for adjusting color values of an image represented in a color space, wherein the image includes a set of pixels, each pixel includes a set of color values, and the apparatus includes:
A user interface control for receiving input to adjust the color value of the image having one or more predefined different content types;
A pixel separator for identifying a subset of pixels associated with a type of content in the image;
A color adjuster for adjusting the color values of the identified subset of pixels based on the received input;
Including the device.   If the color value of a particular pixel falls within a predefined color value range for the content type, the particular pixel of the image is associated with the particular content type and the color value range Is defined in an opposite color space different from the color space in which the color values of the image are defined, and the device is a color space converter for converting the color values of the image into the opposite color space The apparatus of claim 11, further comprising:   The apparatus of claim 11, wherein the color adjuster adjusts the color values of the identified subset of pixels within the opposite color space. A non-transitory machine-readable medium having a computer program for adjusting color values of an image including a plurality of pixels when executed by at least one processing unit, wherein each pixel includes a set of color values, The program
Receiving a selection of locations on the image having a set of different content types;
Determining the type of content associated with the selected location on the image;
Selecting a set of image editing operations from a plurality of different image editing operations based on the determined content type;
Displaying on the image a set of user interface controls associated with the set of selected image editing operations;
A non-transitory machine-readable medium containing a set of instructions for.   The non-transitory machine-readable medium of claim 14, wherein the content type is one of human skin color, sky, and foliage. The computer program is
Receiving input for specific user interface controls;
Performing the image editing operation associated with the particular user interface control on the image;
The non-transitory machine-readable medium of claim 14, further comprising a set of instructions regarding.   The non-transitory machine-readable medium of claim 14, wherein the set of user interface controls is a set of direction-dependent user interface controls superimposed on the image. The computer program is
Receiving a direction input after the set of user interface controls is displayed;
Identifying one of the set of user interface controls for initiating a specific image editing operation on the image based on the received directional input;
Applying the specific image editing operation to the image;
The non-transitory machine-readable medium of claim 17, further comprising a set of instructions regarding. A method for adjusting a color value of an image, wherein the image includes a set of pixels, each pixel includes a set of color values, the method comprising:
Providing a user interface control for receiving a selection of a location on the image;
Providing a content identifier for determining the type of content associated with the color value of the set of pixels corresponding to the selected location;
Providing an editing operation selector for determining a set of image editing operations based on the determined type of content;
And wherein the user interface control is also for displaying on the image a set of user interface controls associated with the determined set of image editing operations on the image.   The method of claim 19, wherein the set of image editing operations includes an exposure adjustment operation.   The method of claim 19, wherein the content type is one of shadow, midtone, and highlight.   The method of claim 19, wherein the set of image editing operations includes an image editing operation that adjusts only a portion of the image associated with the determined content type.   The method of claim 19, wherein each of the set of user interface controls is displayed in a different color to indicate the associated image editing operation. A non-transitory machine-readable medium having a computer program for adjusting color values of an image including a plurality of pixels when executed by at least one processing unit, wherein each pixel includes a set of color values, The program
Displaying an image in the display area;
In response to selection of a location on the displayed image, displaying a set of user interface controls in the display area based on the location on the displayed image selected by a user;
Applying an image editing operation associated with the user interface control that received the direction input in response to a direction input to one of the user interface controls;
A non-transitory machine-readable medium containing a set of instructions for.   The non-transitory machine-readable medium of claim 24, wherein the program further comprises a set of instructions for determining an amount of the image editing operation to apply based on the directional input.   Each user interface control is associated with a specific image editing operation, and the program determines a set of user interface controls to display based on the type of content associated with the selected location on the image. 25. The non-transitory machine-readable medium of claim 24, further comprising a set of instructions to do. A non-transitory machine readable medium having a computer program that, when executed by at least one processing unit, performs a color balance operation on the color values of an image represented in a color space, the computer program comprising: ,
Receiving a selection of locations on the image including a plurality of pixels, each pixel including a set of color values;
Identifying a set of parameters for generating a color space transformation that modifies the color space based on a set of color values of a set of pixels corresponding to the selected location of the image;
Using the color space transformation to perform a color balance operation on the image;
A non-transitory machine-readable medium containing a set of instructions for.   28. The non-transitory machine-readable medium of claim 27, wherein the generated color space transformation maps a color value for each pixel of a set of pixels in the image to a different set of color values.   28. The non-transitory machine-readable medium of claim 27, wherein the set of instructions for identifying the set of parameters comprises a set of instructions for using a set of gray color values.   28. The non-transitory machine-readable medium of claim 27, wherein the set of instructions for identifying the set of parameters comprises a set of instructions for using a set of skin color values.   28. The non-transitory machine of claim 27, wherein the set of instructions for using the color space transformation comprises a set of instructions for performing the color balance operation on the subset of pixels of the image. A readable medium.   28. The non-transitory machine-readable medium of claim 27, wherein the program further comprises displaying user interface items around the selected location on the image. The program is
Moving the user interface item to another location on the image;
Identifying another set of parameters for generating another color space transformation that modifies the color space based on a set of color values of another set of pixels corresponding to the other location of the image. When,
Using the other color space transformation to perform another color balance operation on the image;
35. The non-transitory machine readable medium of claim 32, further comprising a set of instructions for.   28. The non-transitory machine-readable medium of claim 27, wherein the program further comprises a set of instructions for adjusting a zooming level of a region of the image specified by the user interface item. A non-transitory machine-readable medium storing a program for performing a color balance operation on an image including a plurality of pixels, wherein the plurality of pixels have color values represented by a color space, and the program Provides a graphical user interface for editing an image when executed by at least one processing unit of the computing device, the graphical user interface comprising:
A display area for displaying the image;
A color balance control movable on the image for selecting a location on the image;
A non-transitory machine-readable medium, wherein a color space transformation is generated based on the selected location on the image and the color space transformation modifies the color space.   36. The non-transitory machine-readable medium of claim 35, wherein the generated color space transformation maps the color value of the pixel to a desired color value.   36. The non-transitory machine-readable medium of claim 35, wherein the color balance control includes a border that surrounds a portion of the image.   38. The non-transitory machine-readable medium of claim 37, wherein the enclosed portion of the image appears enlarged in relation to the portion of the image that is outside the border of the color balance control.   38. The image is adjusted based on the generated color space transformation, and a preview of the adjusted image is displayed for a portion of the image that is outside the border of the color balance control. A non-transitory machine-readable medium according to.   36. The non-transitory machine-readable medium of claim 35, wherein the color balance control is displayed in the display area when a user selects a location on the displayed image. A non-transitory machine readable medium storing a program that, when executed by at least one processing unit, provides a graphical user interface for editing an image, the graphical user interface comprising:
A display area for displaying different individual images displaying one or more different content types;
Multiple color adjustment controls, each with
Identifying a set of pixels associated with different content types in the displayed image;
Adjusting a color value of the identified set of pixels based on user input to the color adjustment control; and a plurality of color adjustment controls.
A non-transitory machine-readable medium.   42. The non-transitory machine-readable medium of claim 41, wherein each user interface control includes a range slider.

Images