Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q10/00—Administration; Management
  • G06Q10/10—Office automation; Time management
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/26—Visual data mining; Browsing structured data
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q30/00—Commerce
  • G06Q30/06—Buying, selling or leasing transactions
  • G06Q30/0601—Electronic shopping [e-shopping]

Definitions

• This specification relates to color-based designs.
• one innovative aspect of the subject matter described in this specification may be embodied in one or more methods that include receiving a first query for a product, receiving an input that comprises a collection of colors, determining, based at least in part on the input, one or more candidate color palettes, presenting the one or more candidate color palettes on a web site, receiving a selection that specifies one of the candidate color palettes as a selected color palette, and providing, on the web site, a list of products based at least in part on the first query and the selected color palette.
• Other embodiments of these aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.
• the first query includes a selection of a product category from a list of product categories.
• the first query includes a text-based query that describes a product.
• Receiving an input includes receiving an uploaded image.
• Determining one or more candidate color palettes includes extracting color attributes from the input. Weights are applied to the color attributes after the color attributes are extracted from the input. Extracting the color attributes includes identifying one or more of a hue, a saturation, and a value of one or more pixels associated with the input.
• the color attributes are stored in a database. A similarity of the candidate color palettes to the input is evaluated. The candidate color palettes are ranked based on the evaluating.
• Default weights are applied to colors in the candidate color palettes and the selected color palette.
• a control for is provided altering weights of colors associated with the selected color palette or the candidate color palettes.
• the input includes a product in a shopping cart associated with a user account.
• the products are ranked on the list of products. Ranking the products includes evaluating the process: [0006]
• FIG. 1 is a diagram of a design system.
• FIG. 2 is a flow chart of a process for selecting products.
• FIGS. 3-6 are screenshots of user interfaces provided by a design system.
• FIGS. 7A and 7B are diagrams of color theory models.
• FIG. 8 is a flow chart of a process for providing color attributes to a database.
• FIG. 9A is a flow chart of a process for providing color palettes.
• FIG. 9B shows controls for adjusting color-related parameters.
• FIG. 10 is a flow chart of a process for ranking products.
• FIGS. 1 1 A and 1 1 B are diagrams of models used to rank products.
• FIG. 12 is a diagram of a computing system. [0017 Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION
• FIG. 1 shows a design system 102.
• the design system 102 is a combination of hardware and/or computer software that enables a user 104 to interact with various aspects of design.
• the design system 102 can provide a web site on which the user 104 can browse for products (e.g., home design products such as wallpaper, furniture, and the like, and/or other products, such as clothing, cosmetics, etc).
• the design system 102 can assist the user 104 in identifying appropriate products for various applications and contexts.
• the design system 102 may assist the user 104 by suggesting a couch of a particular style and color for incorporation into a room having various properties (e.g., a living room with a particular wall color, square footage, and layout).
• the design system 102 can enable the user 104 to browse for products that correspond to one or more color schemes (sometimes referred to as "color palettes"), and may even assist the user 104 in selecting one or more color schemes by combining aspects of color theory, design rules, and information provided by the user 104 or other users (e.g., neural networking, which relies on the collection, collaboration, and organization of information collected from a plurality of users).
• color palettes are groups of two or more colors that are harmonious according to color theory principles and rules, or that are otherwise aesthetically pleasing or desirable. In FIG.
• the design system 102 assists the user 104 by suggesting harmonious color palettes to the user 104 based at least in part on color theory and user input, and then helps the user 104 locate products (e.g., decorative home design products, such as an area rug) based at least in part on the color palettes suggested by the home design system 102 and approved by the user 104.
• products e.g., decorative home design products, such as an area rug
• FIG. 1 shows states A-O of the system 100.
• the states illustrate operations and flows of information that can allow the design system 102 to provide, to the user 104, one or more products (e.g., as a listing of products on a web page) according to a color palette selected by the user from a plurality of color palettes suggested by the design system 102.
• the states A-O may occur in the illustrated sequence, or they may occur in a sequence that is different than the illustrated sequence.
• the user 104 provides a product query (e.g., a query associated with a product that the user 104 is interested in) to the design system 102 via a user interface 106 associated with the design system.
• the user interface 106 is a web site provided by the design system 102
• the product query can be a text search for a product that is entered into a search tool associated with the design system.
• the product query may also be a selection of a product from a list of product categories, and be further include updated, filtered, or refined product queries. For example, if a previous product query returned a list of products, the user 104 could refine the product query based on price, brand, size, style and other attributes and resubmit the refined product query as the product query illustrated in state A.
• Construction of the product query may be based on attributes for certain styles (e.g., "Queen Anne” style) or patterns (e.g., striped, solid, and floral).
• the design system 102 can identify this and other information based on products that the user 104 has already showed interest in (e.g., if the product is in a shopping cart associated with the user 104). For example, if the user 104 has already added a Queen Anne style sofa to a shopping list, the design system 102 can construct the product query based at least in part on this information.
• the design system 102 may also provide product query suggestions based on attributes of previous product selections and on products that are typically used in the same design, such as various types of living room furniture. Examples include “Queen Anne Armchair,” “Queen Anne Side Table,” or “Curved Antique Wall Mirror.”
• the design system 102 may also identify pattern information. For example, if the user 104 has added a striped sofa to a design or shopping cart and is now searching for pillows, the design system 102 may confirm which piece of furniture the user 104 plans to pair with the pillow and whether they plan to add it as a new item or replace an existing item.
• the design system 102 may present the user 104 with the current pattern option of "floral” and may suggest alternative pattern options such as "solid.” If the user is searching for a new product, then the design system 102 can retrieve the existing pattern data for accessories that are already associated with the striped sofa (e.g., accessories such as pillows and throw blankets that are located on the striped sofa in an image or listing of the striped sofa) and then evaluate which patterns to suggest to the user 104. For example, if a striped sofa includes two floral pillows, the design system 102 may suggest a solid color pillow as a new product. This information can all be used to construct the product query in state A. The product query can then be provided to a product selection engine 1 14 associated with the design system 102 in state C. Again, the states A-O may occur in the illustrated sequence, or they may occur in a sequence that is different than the illustrated sequence.
• the states A-O may occur in the illustrated sequence, or they may occur in a sequence that is different than the illustrated sequence.
• the user 104 provides color data to the design system 102 via the user interface 106.
• the color data can be used by the design system 102 to suggest and/or specify one or more color palettes that will be used in the search for products that correspond to the product query.
• the user 104 provides the color data by uploading an image (e.g., a photograph) to the design system 102 via the user interface 106.
• the user 104 may apply weights to regions or colors in the uploaded image that may alter the significance each weighted region or color receives in future calculations. For example, the user 104 may specify portions of the uploaded image as being more important (e.g., more heavily weighted) than other portions.
• the user 104 can provide color data to the design system by identifying one or more products to the design system 102, or by browsing or searching predefined color palettes provided by the design system 102.
• the user 104 may also be presented with an option to select the most important color from the uploaded image, or the design system 102 may automatically select this base color in preparation for constructing the possible harmonious color palettes. This base color helps the design system 102 identify possible query palettes that may be constructed using color theory rules.
• the design system 102 may prompt the user 104 to select one or more products from their shopping list that they wish to match, or that they wish to use as a basis for the selection of a color palette.
• the design system 102 may also pre-select products for the user 104 based on the product query submitted in state A. For example, if the user 104 is searching for a faucet and has already added a chrome towel rod to their shopping list, the design system 102 may preselect the chrome towel rod and save instructions for a later instance in which the user 104 may provide further input on color palettes.
• state D the color data provided in state B (e.g., in the form of an uploaded photograph) is provided to an image evaluation engine 108.
• the image evaluation engine 108 can identify, extract, and/or store one or more color attributes of the uploaded image. For example, the image evaluation engine 108 may evaluate the uploaded image pixel by pixel (or on some other scale, such as a group of pixels), and may identify, extract, and/or store hue, saturation, and value (e.g., color brightness) information that corresponds to the uploaded image.
• state F the image evaluation engine 108 stores the color attributes in the database 1 10 (or otherwise makes the color attributes available for storage in the database 1 10).
• the palette suggestion engine 1 12 is provided with the color attributes stored in the data base 1 10 (e.g., in response to the palette suggestion engine 1 12 calling for the color attributes).
• the palette suggestion engine 1 12 can use the retrieved color attributes to determine and rank possible color palettes.
• the palette suggestion engine 1 12 uses the techniques described in greater detail below in order to both identify one or more candidate color palettes based on the color attributes and to rank the candidate color palettes after they have been identified according to how well the candidate color palettes correspond to the color data provided by the user 104.
• the palette suggestion engine 1 12 provides the candidate color palettes to the user interface 106, which in turn presents the candidate color palettes to the user 104.
• the candidate color palettes can be presented to the user 104 on a web site, and can be ordered according to the rank determined by the palette suggestion engine 1 12.
• the candidate palettes are presented to the user 104 in state J in order to provide the user 104 with an opportunity to select or confirm one of the candidate color palettes as the selected color palette.
• the selected color palette is provided to the user interface 106 in state K.
• the user interface 106 then provides the selected color palette to the product selection engine 1 14 in state L.
• the user 104 may select more than one color palette from among the candidate color palettes, and more than one selected color palette can be provided to the product selection engine 1 14 as a result.
• the product selection engine 1 14 determines and ranks products based at least in part on the selected color palette and the product query.
• the product selection engine 1 14 uses a number of techniques (described in further detail below) in order to both identify products that correspond to the selected color palette in some way, and to rank the identified products according to how well the products correspond to one or both of the product query and the selected color palette. For example, if the product query was a text search for an "8' ⁇ 10' area rug," and the selected color palette included the colors red and green, the product selection engine 1 14 could identify a plurality of area rugs that had dimensions of 8'x10' and that somehow corresponded to the colors in the selected color palette (e.g., a green area rug with a red border). The product selection engine could also rank the identified products based on how well the products match one or more of the product query and the selected color palette.
• the product selection engine 1 14 provides the ranked products to the user interface 106 which, in turn, provides the ranked products to the user 104 in state O.
• the ranked products can be presented to the user 104 in a variety of formats. For example, the products can be arranged on a web page in a sorted list based on the rank provided by the product selection engine 1 14.
• the user 104 can browse the ranked products and may filter the ranked products by additional constraints (e.g., by price, brand, availability, and the like).
• FIG. 2 illustrates a process 200 for providing a list of products.
• the design system 102 receives a first query for a product (202).
• a product query can be a query associated with a product that the user 104 is interested in.
• the product query can be a text search for a product that is entered into a search tool associated with the design system, or can be a selection of a product from a list of product categories.
• a user interface such as the web page 300 shown in FIG. 3, can be provided to the user 104 by the design system 102.
• the web page 300 can include a query box 302 for receiving a product query 304 as a string of text.
• the product query 304 is a text search for an "8 x 10 area rug.”
• the web page 300 may include a control 306 for submitting the product query to the design system 102 when activated by the user 104 (e.g., when the user clicks or otherwise activates the control 306 using an input device such as a mouse or keyboard).
• the web page 300 may also provide constraint controls 308 for filtering the products that would otherwise be returned in response to the submission of the product query 304.
• the user 104 could use the constraint controls 308 to limit the price range of searched products, or to limit the product search to a particular brand.
• the design system receives an input that comprises a collection of colors (204).
• the input can be used by the design system 102 to suggest and/or specify one or more color palettes that will be used in the search for products that correspond to the product query.
• a web site 400 can provide mechanisms for users to upload an image (e.g., a photograph) to the design system 102 to be used as a color palette inspiration.
• the web site 400 includes an upload control 402 that allows the user 104 to choose an image that will influence the candidate palettes provided by the design system 102.
• the upload control 402 includes a control 404 that, when activated, causes the web site 400 to display an image selection pane 406.
• the user 104 can select from several upload options 408, 410, 412 in order to upload an inspirational image.
• the upload option 408 allows the user 104 to select an image from an existing image gallery (e.g., a third party website that stores images).
• the user 104 can also select the upload option 410 to use a shopping list item as the inspirational image.
• Selection of the upload option 410 may not require an image of the shopping list product or products to be uploaded; instead, the design system 102 can draw on attributes of the shopping list product or products that are stored as metadata.
• Selection of the upload option 412 can allow the user 104 to upload an image from a local machine or from another storage location.
• the user 104 can provide color data to the design system by browsing or searching predefined color palettes provided by the design system 102.
• the user 104 may also be presented with an option to select the most important color from the uploaded image (e.g., the user 104 may provide weights to one or more colors in the image), or the design system 102 may automatically select weight the colors in preparation for constructing the possible harmonious color palettes.
• the weighted image helps the design system 102 identify possible query palettes that may be constructed using color theory rules. For example, if the user 104 specifies that a portion of an image that includes the color red is more important than other portions of the image which are green, the design system 102 may more heavily consider the color red when generating or identifying candidate color palettes.
• the design system 102 determines one or more candidate color palettes (206). In some examples, the design system 102 uses aspects of color theory and/or one or more ranking techniques to determine candidate color palettes. Some of these techniques are described in further detail below.
• the design system 102 presents the one or more candidate color palettes (208). As shown in FIG. 5, the design system 102 can provide a web page 500 that presents the candidate color palettes 514, 516 in a color palette suggestion pane 512.
• the candidate color palettes 514, 516 are related in some way to the colors in the color input.
• the image 504 includes two colors (Color 1 and Color 2).
• the candidate color palette 514 includes three colors: (1 ) a shade a Color 1 , (2) a shade of Color 2, and (3) a complement of Color 2.
• the candidate color palettes 514, 516 are merely examples, as various aspects of color theory and color palette ranking can be used by the design system 102 to provide a multitude of candidate color palettes.
• the web page 500 may also present the color input (e.g., the image 504) that forms the basis for the candidate color palettes 514, 516.
• the web page 500 may also include a control 510 for selecting a different color input that, when activated, may cause the design system 102 to determine a second set of candidate color palettes based on the newly selected color input (e.g., a second, different uploaded image).
• the design system 102 can present the web page 500 to the user 104 to allow the user 104 to select one or more of the candidate color palettes to be used in a product search in combination with the first query (e.g., the product query).
• the design system 102 receives a selection that specifies one of the candidate color palettes as a selected color palette (210). For example, the user 104 can use an input device such as a mouse to select (e.g., by clicking) one or more of the candidate color palettes 514, 516. If the user 104 selects the candidate color palette 514, the candidate color palette 514 will be designated as a selected color palette that can be in a product search in combination with the product query (e.g., the product query).
• the product query e.g., the product query
• the design system provides a list of products based at least in part on the first query and the selected color palette (212).
• the design system 102 can provide a web page 600 that includes a selected color palette 602, the product query 304, and one or more matched products 604, 606, 608 that have been identified by the design system 102 based at least in part on both the selected color palette 602 and the product query 304.
• the design system 102 can identify the matched products 604, 606, 608 using the techniques described below.
• the web page 600 may also include constraint controls 514 for applying one or more additional filters to the matched products 604, 606, 608 (e.g., by adjusting a price range of the matched products 604, 606, 608).
• FIGS. 7A and 7B are diagrams that will be used to describe some aspects of color theory which can be used by the design system 102 to extract color attributes from images, to determine candidate color palettes, and to identify and provide products based at least in part on one or more selected color palettes.
• FIG. 7A shows a color wheel 700 which, in general, is an abstract illustrative organization of color hues around a circle that shows relationships between the colors. Color theory originated using red- yellow-blue (RYB) as primary colors. As it was studied how light is perceived by humans, it was discovered that the receptors in a human eye are capable of sensing three colors red, blue and green (RBG). The brain processes the three colors by mixing them together and creating the colors within the visible spectrum.
• Color wheels can be used to illustrate various color schemes.
• the color wheel 700 illustrates a complementary color scheme by showing the relationship between the color red 702 and the color green 704. Because these colors are directly opposite one another on the color wheel 700, the color red 702 and the color green 704 are complementary (e.g., they contrast highly with one another).
• Other color schemes such as monochromatic, analogous, split complementary, triadic, and tetradic color schemes can also be illustrated on a color wheel, but will not be specifically discussed herein.
• Colors can be defined using a number of parameters.
• a color can be defined by its hue (H).
• H hue
• a hue is a color that is measured as an angle within a color wheel.
• the color red 702 has a hue angle of 0 degrees.
• Colors can also be defined in terms of their saturation, which represents how much of a given hue is in a color, ranging from no color to a pure color. Saturation can be measured from the center of a color wheel on a scale of 0 at the wheel's center to 1 at the wheel's edge. For a color with saturation (S) of 0, the color would appear gray, black, or white regardless of its hue.
• a color can be further defined by its value (V), which represents the lightness or darkness of the color.
• V represents the lightness or darkness of the color.
• a color's value can also range from 0 to 1 .
• FIG. 7B is a hue-saturation-value (HSV) color space model 706.
• HSV hue-saturation-value
• the HSV color space model 706 can define a color in terms of its coordinates within the HSV color space model 706.
• a color's hue 712 is based on its angle relative to a central point 712 within the HSV color space model 706,
• a color's saturation 708 is based on its distance from the center of the HSV color space model 706, and the color's value 710 is based on its height with respect to the HSV color space model 706.
• FIG. 8 shows a process 800 for extracting color attributes for an image that will be used to generate one or more candidate color palettes.
• the design system 102 can use elements of color theory (such as those described above) in order to extract, store, and categorize the color attributes of a given image.
• the design system 102 extracts HSV information for pixels in an image (802). For example, an image (e.g., a digital photograph) provided by the user 104 is made up of many pixels, each of which represent a color.
• the design system 102 extracts the color information from each pixel and maps the extracted information to an appropriate color space within a color model (e.g., the HSV color space model 706) for each pixel.
• One purpose of the process 800 is to prepare query images for generating color palettes or to prepare products to be added to an index for image retrieval and searches.
• the design system 102 can divide a hue circle into as many angles as required until a probability threshold is reached that can provide accurate results. For example, the system may store 192 hues angles. However, as a simple example, the hue data can be stored for 12 main hues of the color wheel, as in the following table (the radian angle below is in RYB- based HSV space):
• the design system 102 parses, identifies, and groups pixels to form color attributes (804).
• the design system 102 identifies H
• This process may result in values being stored in formats such as: H
• an image that is 100% red could be stored as 0
• the design system 102 can ignore certain pixels when measuring colors for both query image(s) (e.g., images used to provide color data, as in state B (FIG. 1 )) and for result images. Pixels that can be ignored may be associated wtih image backgrounds, scenes behind a product, and black, white, and/or gray pixels. Edge detection and object recognition technology may be used to detect the meaningful portion of an image. For example, an edge detection process may be used to separate an image of an area rug from a substantially uniform background.
• An advantage of such an approach is that only the red pixels are indexed as contributing to 100% of the image's P value and, as a result, the design system may assign 100% as the weight (W) for a given query H
• Neutral gray color definitions can also be stored, and the design system 102 can then use a query to decide whether to include all or some of the stored neutral color values for H
• Neutral brown colors may range from red to yellow for hue angles. They are also defined as having certain saturation and value ranges depending on the hue angle. Tolerances for hues angles can be retrieved from a database during this process. In design, it possible to match neutral brown hues with other neutral brown hues. However, it may also be useful to ignore neutral brown colors (e.g., the color tan) if a result image only returns matching colors. By excluding the neutral brown colors, the search result can have an improved recall (number of documents returned in a search result) and precision (accuracy). For example, by including a beige color in the ranking calculations, a search result consisting of a beige pillow with small pink dots may have not satisfy color matching tolerances for a given query.
• S pairs can be separately stored in order to increase flexibility by allowing the query to decide whether to include neutral brown colors in the ranking results. If neutral colors are excluded during a ranking process, the design system 102 might adjust the value for P to be relative only to the sum of a color's P value for a given image document.
• S pairs can be stored as: NBH
• the design system 102 can send the information to separate histograms.
• a visual image of a histogram of the image might show H
• the design system 102 may use weighted averages for value and saturation to reduce the size of the index containing the H
• P indexing result images
• W for storing query data (e.g., images that are associated with a user query).
• P indexing result images
• W for storing query data
• This is just for colors not including neutral colors. Probability and number of pixels in images can reduce this number significantly.
• a weighted average saturation can be used for each H
• Neutral brown can be reduced to about 64 H
• Neutral gray may only require the storage of V data, along with either P or W data.
• average saturation s a for each of 192 hues angles for an image result document can be calculated as:
• the average saturation s a for each of 192 hues angles for a query document can be calculated as:
• the design system 102 may divide saturation in three sections, and may then perform an average calculation for each section.
• the design system 102 provides color attributes to a database (806). For example, once extracted, parsed, and/or grouped, the color attributes can be mapped to a histogram where the H
• the index above represents a result image.
• a similar index could be provided by replacing all P values with W values (to produce index entries in the H
• FIG. 9 shows a process 900 for identifying candidate color palettes.
• the design system 102 accesses color attributes (902).
• the palette suggestion engine 1 12 (FIG. 1 ) of the design system 102 can access color attributes stored in the database 1 10 to extract H
• the color attributes can be stored in the database 1 10 using the techniques and formats discussed above. For example, an image may have color attributes that indicate that the image is 40% green, 10% red, 20% Black, 20% White and 10% Blue-Violet.
• the retrieved color attributes could resemble the following:
• the color attributes above are in the H
• the notations NB and NG are used to represent colors that are neutral brown or neutral gray (Black, White, Gray).
• Hue rank is a simplified version of harmonize rank where only the hue angles are used for ranking products or palettes.
• Base hue is the initial hue identified for generating a color palette. For example, a base hue of red result in a complementary hue of green when using the complementary color theory method. If we assume that a tolerance for hue rank is set as 0.06, then red with a hue angle of 0 qualifies for a base hue. However, red with an angle of does not meet the 0.06 tolerance, so the design system 102 may exclude it as a base hue. Next, the design system 102 may evaluate the weighted average for the saturation for each qualifying base hue using the following formula:
• the saturation's weighted average is simply equal to 0.75 or the original saturation. This process can be repeated for all hue increments for all hypothetical queries and all of the color attributes retrieved from the database 1 10.
• the design system 102 generates candidate palettes and initial query settings (906). With a set of base hues generated, the design system 102 can generate candidate palettes from the base hues using color theory schemes such as monochromatic, analogous, complementary, and split complementary schemes. For example, using the base hue of green 180 degrees, the design system 102 can generate palettes for each color theory rule. For example, using the complementary color scheme with the base hue of green, and when H1 is base hue and H2 is the second complementary hue generated from the base hue, the design system 102 calculates H2 as follows:
• H2 H i ⁇ ⁇
• the two colors using the complementary palette are green and red.
• S pairs may require that all H
• the representative color palette may consist of two colors: red and green.
• the design system 102 may associate different regions with one or more colors in the palette. For example if the user 104 is creating a more traditional design, the design system 102 may present the user 104 with regions for saturation and value for each of the major parts of design.
• the primary areas are typically the wall, the floor, and ceiling coverings. These are traditionally the least saturated colors, with floors having the darkest value, walls having a lighter value, and the having ceiling the lightest value.
• Area rugs, furnishings and window treatments are typically secondary areas which typically have a relatively moderate saturation.
• Vases, lamps and wall art are tertiary items that are typically the most saturated.
• the design system 102 can vary the value of V to increase the estimated visual interest of a design.
• the system may identify the current query of "8 x 10 area rug" as an item that may be well-suited for the "secondary area”. However it may also identify it as an accent piece.
• a complementary color scheme a limited amount of the color that is complementary to the base hue can be used to avoid color tension. In this case, a limited amount of red (which is complementary to the base hue of green) should be used.
• the complementary color of red may be presented to the user as a color they should pick for the current query. If the system decides it is not an accent piece then the base color or neutral colors may be selected as the default setting for the user.
• the design system 102 may reduce the value range of the V component to a much darker color within the secondary area, such as by selecting red color with a 15-25% value as: lira - 0.15 to 0.2S.
• the design system 102 ranks the candidate palettes (908). For example, after the design system 102 has determined the candidate palettes, the design system 102 (e.g., the palette suggestion engine 1 12) can rank the candidate palettes against the color attributes retrieved from the database. In some examples, the design system 102 ranks the candidate palettes against color attributes that represent the image uploaded by the user 104 as a design inspiration.
• the design system 102 e.g., the palette suggestion engine 1 12
• GGrreeeenn ccoolloorr : ⁇
• the design system 102 then substitutes query parameters (e.g., the parameters associated with candidate color palettes) into the ranking process.
• the design system 102 then substitutes the result parameters (e.g., the color attributes associated with the image uploaded as a design inspiration) into the ranking process, as follows (the adjusted P numerical value can be used as above):
• Red color is 0
• Green color is ⁇
• one or more items in a user's shopping list can be used as the color inspiration (e.g., in place of an image uploaded by the user 104).
• the design system 104 would evaluate the shopping list and existing color palette; however, instead of using pixels of a single image to calculate P, the design system 102 may use a set of images from the shopping list.
• P stored in the database for this shopping list can be based on weight adjusted versions of P for each H
• the shopping list (adjusted by visible surface area for each product) behaves like a single uploaded image. For example, assume Product 1 has a visible surface area of 10% of the design, and product 2 consists of 90% of the design. The following process can be evaluated:
• red can be calculated as:
• A is the percentage visible surface area of a product relative to the overall surface visible area of the design.
• One definition of visible is the viewable area of a product within a design relative to all other products that can be seen from a given angle.
• S pairs can use the same surface area weighted method for each H
• the design system 102 could reevaluate the base color, generate new color palettes, and rank them in preparation for presentation to the user 104. If the user 104 is searching for a new paint color, the design system 102 may exclude the existing paint color in all calculations for color palette generation and ranking. Surface Area Adjustments for P may also be used in ranking palettes for presentation to the user.
• the design system 102 can present the candidate color palettes to the user 104.
• the user 104 may select one or more of the candidate color palettes for use in a search for products.
• the design system 102 can present the user 104 with options for modifying default settings for each color in the color palettes (including the core palettes generated from the color theory rules and the neutral palettes that coordinate with the color scheme).
• the design system 102 may allow the user 104 to select different saturations and values for each of the hues in the candidate color palettes (e.g., using the controls 903, 905 shown in FIG. 9B).
• the design system 102 may also allow the user 104 to edit default search settings for the current query and for future queries. For example, the user 104 may notify the design system 102 that they are doing a more contemporary design and the design system 102 may suggest different settings for primary, secondary and tertiary areas.
• the user 104 can build palettes dynamically for all colors that they wish to use. They can assign each color to primary, secondary or tertiary areas. A visual indication of each of these palettes might show the primary colors as large swatches with secondary smaller and tertiary the smallest. The user 104 may also adjust initial weights for each color within each candidate color palette.
• FIG. 10 shows a process 1000 for identifying and ranking products based at least in part on a product query and a selected color palette.
• the design system 102 processes a product query and filters results based on categorization data (1002). For example, the design system 102 parses the product query for an "8' ⁇ 10' area rug.”
• An ontology module can be used in the process to match product query terms with categorization data in the text query.
• the design system 102 first identifies "area rugs" as a match via the ontology keyword database to the category called "Area Rug.” With a known category, the design system 102 can now evaluate possible attributes for the category “Area Rug” and can identify matches for refinement options (e.g., a value called “Solid” for pattern). The matched refinement options can be used as filters for the category search "Area Rugs.” "8'x10"' remains part of the text query.
• the design system 102 ranks filtered results based on a selected color palette (1004). For example, assume the user 104 has selected a color palette to be used in the product search that includes the following two colors: Green color 1 : ⁇ hue, 0.75 saturation
• Green color 2 25 ⁇ /24 hue, 0. 5 saturation
• the query associated with the selected color palette would have H
• the design system 102 uses the following process to rank the filtered product results based on how well they correspond to the selected color palette:
• HR represents the Harmonize Rank of a given product image in a search result (e.g., how well the product image matches a query palette or query image). Search results can be ranked between 0.00 and 1 .00, with larger numbers indicating higher rankings.
• the process uses: ⁇ is, - ⁇ : ⁇ - to measure the distance between each n instance of H
• a value of 0.00 may indicate that the product result does not match the query.
• variable A in HR(A) represents the product image in a search result.
• the product result image is stored in an index. It groups pixels in the image together and stores them as H
• n represents the number of H
• a set of pixels in a search result image are assigned to a group using each possible saturation/value combination. Combinations that are within the query hue tolerance are considered, and there are no redundant H
• S pairs are assigned to a single Query Hue based on the shortest distance between the Query H
• the variable h q represents the hue of each query H
• the variable s q represents the saturation of each query H
• the variable h r represents the hue of each matching H
• the variable s r represents the saturation of each matching H
• the variable As n represents the distance between the query saturation and the result saturation within the given hue tolerance t. It can be expressed as j.s 3 ⁇ 4 - s for each instance of n.
• the variable Ah n represents distance between the query hue and the result hue within the given hue tolerance t. It can be expressed as js. B — for each instance of n.
• the variable W n represents the weight assigned to a given query H
• W The W variable and the limits for w H are optional and will typically be used for image correlation purposes instead of image harmony ranking. Typically, this can be a number between 0.00 and 1 .00.
• W may be applied to determine the image correlation between query image(s) and search result images such as identifying the similarity of room photographs, products and clothing outfits. It is also used in some user interfaces where the system or user interface allows user to select colors and assign importance for a given query. When the user is doing a palette search (described below), W may be set to 1 for all colors in the palette by default but may also be adjusted, as described above. For an image correlation search (also described below) W is set to the percentage of the query image(s).
• the variable P n represents the percentage of a search result image with the same H
• the variable t h represents the hue angle tolerance. For example, if there are 12 hues in the color wheel, the tolerance is set to and the query hue angle is 0, then the possible hue in a search result calculation are hue angles that are smaller than ⁇ Hue tolerance angles can be set for each query color independently or globally for all colors within an entire query palette. For example, cyan may have a smaller tolerance than green.
• the design system 102 can store a value for t for each instance of H
• the variable md(s q ) represents the maximum distance from s q to the possible result H
• the design system 102 can measure and store a value for md(s q ) for each instance of H
• the variable range(Vj) represents the range of values of V (darkness/lightness of product in image) and is used to filter results.
• FIGS. 1 1A and 1 1 B illustrate a manner in which the design system 102 may calculate a maximum distance between a query color (QC) and a result color (RC).
• the QC can be represented as a location within the HSV color space model 706 shown in FIG. 7B.
• colors represented within a HSV color space model can be represented in a portion of the HSV color space model in a pie-shaped slice 1 100.
• the pie- shaped slice 1 100 can be transformed (or "stretched") into a rectangular color model 1 102. In some examples, this new rectangular color model can be used for each query color.
• the pie shaped-slice 1 100 can be stretched into a rectangle where saturation approaches 0: ⁇ i
• the design system 102 can now measure the distance (d) from the QC to the RC using the Pythagorean Theorum: [00112] Accordingly, using the example discussed above, the pnocess ⁇ as s i + (A,)- can be evaluated to identify that the distance between all query colors (Green 1 and Green 2) and all result colors. In this example, the distance is shortest when using Green Color 1 as the query color, so green color 1 now acts as the only query color. In the case where there are multiple query colors, only a single H
• the design system 102 also calculates t 1 ⁇ 2d(s ⁇ £ + (t 3 ⁇ 4 )-, which is the largest possible distance between H
• FIG. 1 1 B illustrates how the maximum distance can be calculated.
• the center point QC is the query color, while the result color RC is located in the upper right corner of the rectangular model 1 106.
• the longest possible distance is used. In some examples, the maximum distance can be measured by identifying all possible result colors and determining the longest distance. However, this approach requires the initial retrieval of qualifying all qualifying result colors.
• the design system 102 can evaluate the ranking process.
• An example calculation is as follows:
• the design system 102 can weigh the importance of saturation vs hue in measurement calculations, f is a factor for setting the relative importance of h (hue) to s (saturation).
• the scaling factor f can also be included in the ranking process, as follows:
• a simpler version of the ranking process can be used that uses hue as distance measurement and that filters matches by saturation ranges.
• hue may be the only factor for determining distance.
• the ranking process will only use H
• the design system appends information to low probability results (1006).
• the design system 102 may append low probability results with incomplete categorization and refinement options as the next set of results and follow method 1 or 2 for the remaining results.
• result can be grouped based ranking in the initial document query.
• Each group is re-ranked using one of the harmonize rank processes described herein. This technique can be repeated multiple times where a tolerance for harmonize rank and document search relevance are set for the first set of qualifying results and the process is repeated.
• the design system 102 can present a list of ranked products to the user 104, as shown in FIG. 6.
• the design system 102 can perform an image correlation query in order to substantially exactly match a color palette to an image.
• the user 104 can elect to perform an image correlation search.
• the palette can include neutral brown colors and may include neutral gray colors if they are not an image background color.
• Palettes include W (percentage for each H
• S pair is executed using the exact W for each W
• Harmonize Rank where H
• the techniques described above can be used to provide ranked lists of products in a variety of industries.
• the harmonize search methods can be used in any industry where color, color matching and color coordination is a part of a decision for purchasing or selecting a product, combination of products or a design. It can also be applied to more abstract applications like the art industry in assisting with paint color selection.
• Example market segments include home design and decorating, apparel, clothing, and apparel accessories, cosmetics, jewelry and watches, gardening, and landscaping and other aesthetic markets.
• the techniques above can be used to determine skin tones of a user or colors of one or more cosmetic products and to provide color palettes that can be used to select cosmetics that will complement the user's complexion or other cosmetic products.
• the techniques above can also be used to determine skin tones of a user or colors of one or more clothing items and to provide color palettes that can be used to select clothing that will complement the user's complexion or other clothing items.
• the systems, software, and methods described herein can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device.
• Each computer program can be implemented in a high-level procedural or object oriented programming language, or in assembly or machine language if desired, and in any case, the language can be a compiled or interpreted language.
• Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory and/or a random access memory.
• a computer will include one or more mass storage devices for storing data files, such devices include magnetic disks, such as internal hard disks and removable disks magneto-optical disks and optical disks.
• Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as, internal hard disks and removable disks; magneto- optical disks; and CD_ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits). An example of one such type of computer is shown in FIG.
• the system 51 1 includes a processor 520, a random access memory (RAM) 521 , a program memory 522 (for example, a writeable read-only memory (ROM) such as a flash ROM), a hard drive controller 523, and an input/output (I/O) controller 524 coupled by a processor (CPU) bus 525.
• the system 51 1 can be preprogrammed, in ROM, for example, or it can be programmed (and reprogrammed) by loading a program from another source (for example, from a floppy disk, a CD-ROM, or another computer).
• the hard drive controller 523 is coupled to a hard disk 130 suitable for storing executable computer programs, including programs embodying the present methods, and data including storage.
• the I/O controller 524 is coupled by an I/O bus 526 to an I/O interface 527.
• the I/O interface 527 receives and transmits data in analog or digital form over communication links such as a serial link, local area network, wireless link, and parallel link.
• the system and methods described above have been directed to an implementation of the home design tool for design and layout of rooms for a house, the tool is not limited to home designs, as such, other implementations are possible.
• the home design tool could be used to generate a layout of an office building or other commercial space.
• the home design tool could be used to generate layouts of garden or other landscaping features.
• the home design tool described above is not limited to home design layouts.
• similar design tools can be used to produce, for example, exterior/landscaping layouts, factory equipment layouts, industrial and commercial layouts, clothing ensembles/outfits, office space layouts, restaurant layouts, and the like.
• Other implementations not specifically described herein are also within the scope of the following claims.

Landscapes

• Engineering & Computer Science (AREA)
• Business, Economics & Management (AREA)
• Theoretical Computer Science (AREA)
• Strategic Management (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Entrepreneurship & Innovation (AREA)
• Human Resources & Organizations (AREA)
• Accounting & Taxation (AREA)
• Finance (AREA)
• Databases & Information Systems (AREA)
• Economics (AREA)
• Marketing (AREA)
• Data Mining & Analysis (AREA)
• General Business, Economics & Management (AREA)
• Development Economics (AREA)
• General Engineering & Computer Science (AREA)
• Operations Research (AREA)
• Quality & Reliability (AREA)
• Tourism & Hospitality (AREA)
• Processing Or Creating Images (AREA)

Priority Applications (4)

Applications Claiming Priority (1)

Related Child Applications (1)

Publications (1)

Family

ID=46831032

Family Applications (1)

Country Status (2)

Cited By (2)

Citations (6)

• 2011
  • 2011-03-17 WO PCT/US2011/028848 patent/WO2012125168A1/en active Application Filing
  • 2011-03-17 EP EP11861030.2A patent/EP2686795A4/de not_active Withdrawn

Patent Citations (6)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events