WO2023224663A1

WO2023224663A1 - Color histogram based image selection

Info

Publication number: WO2023224663A1
Application number: PCT/US2022/072345
Authority: WO
Inventors: Jan Morovic; Peter Morovic
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2022-05-16
Filing date: 2022-05-16
Publication date: 2023-11-23

Abstract

Certain examples relate to a method of a method of method of evaluating printing equipment and/or materials. An example method comprises determining a target color histogram, providing a plurality of representative images each having a respective color histogram, and selecting a sub-set of the plurality of representative images using the respective color histograms and the target color histogram.

Description

COLOR HISTOGRAM BASED IMAGE SELECTION

BACKGROUND

[0001] Color printing may result from a number of colorants of different colors being superimposed on top of a substrate, such as paper or other media. A printing system may be associated with a color space, defined by one or more colorants available to the printing system for deposition or application to a print medium. An example of a colorant color space is the Cyan, Magenta, Yellow, BlacK (CMYK) color space, wherein four variables are used in a subtractive color model to represent respective quantities of colorants. Examples of colorants include inks, dyes, pigments, paints, toners and powders.

BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various features of the present disclosure will be apparent from the detailed description which follows, taken in conjunction with the accompanying drawings, which together illustrate features of the present disclosure, and wherein:

[0004] Figure 1 illustrates a color histogram according to an example;

[0005] Figure 2 is a schematic illustration of a printing apparatus according to an example;

[0006] Figure 3 illustrates representative images and corresponding color histograms according to an example;

[0007] Figure 4 is a flow chart illustrating a method of selecting a sub-set of representative images according to an example;

[0008] Figure 5 is a flow chart illustrating a method of selecting from candidate groups of representative images according to an example;

[0009] Figure 6 illustrates sub-sets of representative images and corresponding color histograms according to an example; and

[0010] Figure 7 is a flow chart illustrating a method of selecting from candidate groups of representative images according to another example.

DETAILED DESCRIPTION

[0011] Color can be represented within imaging devices such as print and display devices in a variety of ways. For example, in one case, a color as observed visually by an observer is defined with reference to a power or intensity spectrum of electromagnetic radiation across a range of visible wavelengths. In other cases, a color model is used to represent a color at a lower dimensionality. For example, certain color models make use of the fact that color may be seen as a subjective phenomenon, i.e. dependent on the make-up of the human eye and brain. In this case, a “color” may be defined as a category that is used to denote similar visual perceptions; two colors are said to be similar if they produce a similar effect on a group of one or more people. These categories can then be modelled using a lower number of variables. [0012] Within this context, a color model may define a color space. A color space in this sense may be defined as a multi-dimensional space, with a point in the multi-dimensional space representing a color value and dimensions of the space representing variables within the color model. For example, in a Red, Green, Blue (RGB) color space, an additive color model defines three variables representing different quantities of red, green and blue light. In a digital model, values for these quantities may be defined with reference to a quantized set of values. For example, a color defined using an 8-bit RGB model may have three values stored in a memory, wherein each variable may be assigned a value between 0 and 255. Other color spaces include: a Cyan, Magenta, Yellow and Black (CMYK) color space, in which four variables are used in a subtractive color model to represent different quantities of colorant or printing fluid, e.g. for a printing system; the International Commission on Illumination (CIE) 1931 XYZ color space, in which three variables (X, Y and Z or tristimulus values) are used to model a color; the CIE 1976 (L* a* b* - CIELAB or ‘LAB’) color space, in which three variables represent lightness (L*) and opposing color dimensions (a* and b*); the YUV color space, in which three variables represent the luminance (Y) and two chrominance dimensions (II and V); and the IPT color space, in which the three variables represent a lightness or Intensity dimension (I), a “Protanopia” red-green chroma dimension (P), and a “Tritanopia” yellow-blue chroma dimension (T).

[0013] A simple color space may be defined using ink vectors which represent the proportion of individual inks within a combination such as an NPac vector. For example, an ink vector may be a vector with a predefined set of elements, where each element represents an ink (e.g. a colorant or color output available to the rendering device) and the value of the element represents a quantity of ink (e.g. [C, M, Y, K]). An ink vector may be used to instruct deposit of inks in a printing device, e.g. where the value represents a particular number of drops or quantity of ink to deposit. An ink vector may be used to instruct the printing of a color in an addressable area of a substrate (e.g. a print “pixel”). [0014] A colorant may be a dye, a pigment, an ink, or a combination of these materials. A substrate may include various types of paper, including matt and gloss, cardboard, textiles, plastics and other materials.

[0015] Certain examples described herein address a challenge with evaluating the performance of printing systems and substrates for printing images, given that different types of images may typically be printed for different users or for different applications. In some of these images neutral colors may dominate whereas in other images pastel or saturated colors may dominate. These different dominant color values are affected differently by printing processes. For example, neutral tones are harder to match than saturated colors and maintaining uniform output is more difficult for olive pastel colors than yellow. These performance factors are also influenced by the print materials and printing equipment used. If images with mostly saturated colors are printed for a customer who typically prints more neutrally colored images, the printed images may be misleading in terms of the printer performance that can be expected. This may lead to a poor assessment of printing equipment and/or substrates, or of upgrading these, for different customers and use cases or applications.

[0016] Certain examples described herein utilize color histograms of images to select a sub-set of images using a target color histogram. The target color histogram represents a type of image printed by a particular user or for a particular application. The target color histogram may then be used to select representative images that have color histograms similar to the target color histogram. These representative images may be a sub-set of a larger predetermined set of representative or stock images. The selected representative images can then be printed for the user to allow them to more accurately assess the performance of the printing equipment and/or substrate based on the type of images they would normally print. For example, a user who normally prints vivid vehicle wraps may typically use images with very different color histograms than a user who normally prints more muted neutral tones for engineering design presentations.

[0017] The color values of an image or a set of images may be described in a color histogram which indicates the use of colors at different levels in a color space. An example color histogram is shown in FIG. 1. This example color histogram 100 is represented in the RGB (Red, Blue, Green) color space, however, color histograms having other numbers of dimensions may be used depending on the color space; for example, four dimensions for CYMK (Cyan, Yellow, Magenta, Black), eight dimensions for a bi-level CYMK printing system, various dimensions for Neugebauer Primary area coverage (NPac) color space based systems, device-independent color spaces like CIELAB or CIEXYZ, and so on. Whilst the example color histogram 100 is illustrated, it may be stored and processed as digital values, for example in a matrix or tensor.

[0018] Each color is represented along an axis 105R, 105B, 105G and is divided into color value ranges 105, for example 110R1 , 110R2, ...110R5, 110B1 ... 110B5, 110G1 ... 110G5 - not all of which are illustrated for simplicity. In this example, each color is divided into five ranges having respectively color values 0-50, 51-100, 101-150, 151-200, 2001-255 in an 8-bit color system. A bin is used herein as a combination of a color range for each color in the dimensions of the color space - in this example three dimensions of color. So for example, a lower left bin or cube 120-B1 ,G1 ,R5 corresponds to the color range 0-49 for Blue, 0-49 for Green and 200-249 for Red. In the example color histogram 100 there are 5x5x5=125 bins.

[0019] The number of times colors in each bin are used in the image, or set of images, is indicated by the size of a respective dot 115. The dot, or sphere 115, may indicate a proportion or percentage of times the color combinations within each bin are used in the image or set of images. Some bins will not have a dot such as a bin in the upper left and corresponding to Blue 200-250, Green 0-50 and Red 200-250. This means that colors in this particular combination of color values ranges are not used in the image. By contrast the dot 115-B4,G1 ,R4 indicates that colors in the range Blue 150-200, Green 0-50 and Red 150-200 are used regularly in the image. It will therefore be appreciated that different images, or sets of images, may have different color histograms, having different size dots for each bin. The dot sizes may be represented by numbers of times or frequencies that these colors are used in the image, for example as a value in a matrix. [0020] Fig. 2 illustrates a printing apparatus according to an example. The printing apparatus 200 comprises a controller 205 and a printhead 210. The controller 205 comprises an input/output interface 210, a processor 215 and memory 220 and may implement a HANS color printing pipeline for example. Such a pipeline includes color management which receives a source file 250 comprising image data in a source color space, for example CIE 1976. The color management stage generates a device color output corresponding to colors that the printer apparatus can actually render onto a substrate - for example the image data may be mapped from a CIE 1976 color space to a CMYK color space. The device color output can be processed by a color separation stage that maps input from the device color output to amounts of colorants to be applied to the substrate for each pixel of the image. In an example this can be represented by ink-vectors which may be mapped from the device colors using an ink-vector lookup table (LUT). The ink-vectors may be further mapped to NPac vectors by a further conversion process. A halftoning process then generates instructions for the print head 210 to render the images 255.

[0021] Color histograms for an image may be determined from its pixels’ color values. The total number of image pixels that fall within a bin or range of color values provide respective color value frequencies or bin values corresponding to the size of respective dots of the image’s color histogram. Each bin value of a color histogram is associated with a frequency metric corresponding to the frequency of color values in the range or bin. The frequency metric may be: a number of pixels of an image that fall within the bin; a relative number or percentage of the pixels in the bin compared with the total number of pixels in the image; a likelihood percentage of the number of pixels in the bin.

[0022] The print apparatus 200 may be operated by a print service provider having many different customers that require different types of images to be printed, for example printing onto different substrates and with color sets having different color distributions and properties such as highly saturated vehicle wraps or more neutral engineering design proposal paper documents. The images normally printed for one customer may have similar color histograms whereas the images normally printed for another customer may also have similar color histograms, but which are quite different to those of the first customer.

[0023] The memory or storage medium 220 comprises a set of representative or “stock” images which can be printed on demand and may be used for demonstrating how different substrates, colorants or printer apparatus properties may affect rendered images. This may be useful to a customer when trying to visualize how changing these parameters, for example via an upgrade path, will affect rendered images. However, the set of representative images is typically large in order to contain a wide range of images that may be relevant to a wide range of customers. Some customers may regularly print a certain type of image, such as text and graphs, and so using images such as bright beach scenes to demonstrate possible changes in printing parameters may give a misleading indication of how rendered images will change. Further, it may be difficult for humans to assess the impact of new materials or printing equipment based on a large number of representative images.

[0024] In an example, the memory 220 comprises an instruction set 225 arranged to select a sub-set of the representative images 230 to print for a particular customer or user. This may be based on historical images printed by the user, for example using the print data 250. The memory may be a non- transitory computer-readable storage medium for example, a hard drive, a CD- ROM disc, a USB-drive, a solid-state drive or any other form of magnetic storage device, optical storage device, or flash memory device, maintained locally or accessed remotely, capable of having thereon computer readable code suitable for the functions described herein

[0025] Instruction 235 instructs the processor 215 to determine a target color histogram for user images. The target color histogram may be determined from the historical images printed by or for the user, for example averaging the color histograms of each historical image. In another example, the target color histogram may be stored or provided based on similar users for example.

[0026] Instruction 240 instructs the processor to select a subset of representative images using the target color histogram. This may be implemented by comparing the color histograms of the representative images with the target color histogram. The color histograms of the representative images may be stored with the representative images or may be calculated by the processor as needed. Various example methods of selecting this subset are described in more detail below. Once the sub-set of images has been selected, the subset of images may be rendered using the printhead 210 and provided to the user for assessment.

[0027] Fig. 3 illustrates a set of representative images 300 as well as two separate representative images 310 and 320 from this set 300. Respective color histograms 315 and 325 are shown for separate representative images 310 and 320. The color histograms 315 and 320 are in the RGB color space with five color ranges for each color axis or dimension. It will be appreciated that color histograms could be represented using a different color space, such as CMYK, using different numbers of colors and color value ranges. Bins correspond to a respective color value range for each color dimension used. It can be seen that the color histograms for each representative image are quite different, with the use of colors in bins for one image being different to those used in the other image. For example, the dominant color value ranges for each color dimension are different as can be seen from the different locations of the larger dots.

[0028] Fig. 4 illustrates an example method of selecting a subset of representative images. The method 400 may be implemented by the printing apparatus 200 of Fig. 2 or any other suitable apparatus. For example, the method 400 or parts of the method may be implemented in a computing apparatus separate from a printer.

[0029] At 410, the method determines a color space and resolution for color histograms. In the previously used examples, the RGB color space and five color value ranges per color was used, but different color spaces and number of color ranges could alternatively be used. The number of bins is determined by the number of color ranges per color multiplied by the number of colors - for example 3x3x3=27 bins for a three-color space (e.g. RGB) and three color value ranges per color. The method also determines the number (k) of representative images to include in the subset, as well as the number of pixels for each image. These parameters can be determined using user input or according to a predetermined setting.

[0030] At 420, the method provides a plurality (n) of representative images, each associated with a respective color histogram. The representative images may be provided in memory as a large set of predetermined images of different subjects and color distributions. For example, images may include a wide range such as landscapes, people, animals, plants, human artefacts, abstract shapes and natural phenomena. Respective color histogram data may be stored with the respective images or calculated as previously described.

[0031] At 430, the method determines a target color histogram. The target color histogram may be a predetermined or a default color histogram, for example having a uniform color distribution where all probabilities for each bin or color range are equal and together they sum to one. For example, for a 3x3x3 color histogram having three color value ranges for each of three color dimensions or axes, each bin has a likelihood probability of 1/27 or 3.7%. Each dot or sphere is centered at the center of each bin, and it’s diameter is proportional to its probability. As noted previously, for example with reference to Fig. 1 , in other examples the dots or spheres at the centers of each bin will be of different sizes depending on the probability of the respective color value combinations for that bin.

[0032] At 440, the method selects a subset of the plurality of representative images using the respective color histograms of the representative images and the target color histogram. The method aims to find a subset of representative images having color histograms that approximate the target color histogram. Depending on the target color histogram, this may result in a subset of representative images that favor neutral and skin tones or alternatively a subset of representative images having highly saturated colors as might be used in a vehicle wrap for example.

[0033] Different approaches to selecting the subset of representative images are possible, for example those described with respect to Fig. 5, 7 and 8 below. [0034] At 450, the method prints or renders the subset of images so that these can be assessed either directly by a user or customer or using automated assessment approaches.

[0035] This approach to selecting subsets of representative images reconciles a simultaneous need for using images that are “like” the contents of customer print jobs on the one hand and using colors with predefined probabilities on the other. Each can be addressed independently for example using uniform color patches, however this is time and resource consuming and difficult for a customer to assess as they will typically be able to assess a limited number of such images. The ability to evaluate a small number of images which are similar to images historically printed by the customer, or that the customer wants to print in the future, improves the ability of the customer to assess the impact of changes to different printing parameters such as printing equipment, substrates and colorants. Selecting a small subset of similar images from a large set of representative images and rendering these allows a customer to focus on small differences which yields results that have greater similarity.

[0036] At 455, the method scans the subset of printed representative images and determines a printing metric such as grain. This may be achieved using a pretrained machine learning model.

[0037] Fig. 5 illustrates a method of selecting a subset of representative images according to an example. This method may be performed at 440 of Fig. 4. To find the subset k of images whose joint color histogram best approximates the target color histogram, the method 500 proceeds as follows.

[0038] At 515, the method traverses all possible combinations of candidate k images from the n images. For example, where n=50 and k=5, the number of candidate subsets of k images will be 2118760. For each candidate subset of k images, the method 500 then performs the following.

[0039] At 525, the method computes a joint color histogram for the candidate subset of k images. This may be achieved by summing the frequencies or occurrences of color values (or bin values) for each color value (or bin) in each color histogram of the candidate set of images. For example, if bin 120-B1 ,G1 ,R5 has a bin value or color value frequency of 10 for image 1 in the candidate subset of images, 12 for image 2, 67 for image 3, 1 for image 4 and 98 for image 5, then the sum for this bin is 188. This may then be divided by the subset number k=5 to get an average for this bin of 37.6. The joint color histogram then comprises average color value frequencies for each bin. This may be expressed in a likelihood percentage frequency metric instead of the actual number of color value pixels frequencies.

[0040] At 535, the method computes a difference metric between the joint color histogram and the target color histogram. For example, the absolute difference in color value likelihood percentages or bin values between respective bins of the joint color histogram of the candidate subset and the target color histogram are calculated. These differences for each bin can be summed and divided by the number of bins to calculate an average difference. Alternative difference metrics may be employed such as finding the largest absolute difference between bin values. The first difference metric, average bin value differences, expresses the difference between the two sets of bin values more comprehensively, whilst the second difference metric, the largest difference, relates more to how the human visual system perceives color differences where large differences are given more weight than smaller ones. A combined difference metric may be based on both of these.

[0041] Once all of the candidate subsets are traversed, the method at 555 selects the candidate subset of representative images with the smallest difference metric.

[0042] Fig. 6 illustrates a first subset of representative images 650 is selected on the basis of minimizing the largest difference between bin values of the joint and target color histograms. The joint color histogram 655 for the first subset 650 indicates a 6.8% maximum probability error in this example. A second subset of representative images 660 is selected on the basis of minimizing the average difference between bin values of the joint and target color histograms. The joint color histogram 665 for the second subset 660 indicates a 2.5% mean probability error in this example.

[0043] Various aspects of the method 500 may be varied such as the color space in which the color histograms are computed (e.g. RGB, CMYK, CIELAB, CIECAM16), the number of bins, whether the same number of bins is used per color dimension and/or whether the bins are of a uniform subdivision. For example, smaller bins may be included in regions of a color space where differences are more important, such as neutral colors. Different difference metrics may be computed such as minimizing the standard deviation of differences between bin values of the target and joint color histograms, and/or adding a minimum bin value needed in a bin which may act effectively as a filter against some images. Different frequency metrics or bin values may be used, for example actual color range frequency values instead of likelihood percentages. The target color histograms may be obtained using Big Data analysis of field data and customer usage models or other approaches.

[0044] Fig. 7 illustrates a second method of selecting a subset of representative images according to an example. This method 700 may be performed at 440 of Fig. 4 and is faster than the exhaustive search approach of Fig. 5. To find the subset k of images whose joint color histogram best approximates the target color histogram, the method 700 proceeds as follows.

[0045] At 715, an initial selected subset S is defined which is empty and therefore has a histogram with zeros in all bins.

[0046] At 725, an initial subset of images is selected (e.g. randomly) as a starting candidate subset C, from which sequential selections will be made.

[0047] At 735, the candidate subset C is traversed as many times as the number of images k that are desired and at each iteration the method 700 proceeds as follows.

[0048] At 745, for each image in the candidate subset C, a temporary set X is constructed consisting of the sum of the bin values or frequency metrics for the candidate subset C selected at the previous iteration and corresponding bin values for the candidate subset C of the current iteration.

[0049] At 755, difference metrics between the bin values of the target color histogram or set and those of set X are calculated. These may be any of the difference metrics previously discussed in relation to the method of Fig. 5.

[0050] At 765, the image that yielded the smallest difference is added to the selected subset S and removed from the candidate subset C. The method 700 then continues at 735 to find a next image with the next smallest difference in the difference metric to remove from the candidate set C and add to the selected subset S. This process continues until a termination condition such as a predetermined number k of images being included in the selected subset S, a color histogram difference metric such as a threshold improvement or a predetermined number of iterations.

[0051] Method 700 sequentially identifies images that result in the largest reduction in error that can be made at each step. Increasing the number of images selected from a candidate subset C may further reduce histogram error and will reduce the speed of finding the selected set S.

[0052] In an experimental example compared to the global approach of Fig. 5, where n=41 and k=6, a 7.6% maximum probability error (compared with 4.7% for global) was achieved using a maximum bin frequency difference metric and a 2.3% mean probability difference error (compared with 2.0% for global) using an average bin frequency difference metric. This indicates that the sequential approach of Fig. 7 achieves results similar to the global approach of Fig. 5.

[0053] An advantage of the sequential approach of Fig. 7 is that it is much faster to process (0.007 seconds on a test system) than the exhaustive global approach of Fig. 5 (1 .01 .279 seconds). This advantages further increases as the subset size k increases, with processing time increasing linearly for method 700 and exponentially for method 500.

[0054] A third method of selecting a subset of representative images according to an example uses a scaling approach. In this example, the problem of matching a target color histogram with a subset of color histograms of a plurality of representative images may be formulated as an optimization problem. Given a set of n representative images h with respective color histograms Hi, find a set of weights wsuch that Wi^T*Hi=Htarget.

[0055] This approach finds the linear combination of color histograms that results in the target color histogram. The weights then refer to the proportions of image H in a final composite collage that will have a color histogram matching the target color histogram. Allowing for varying the relative size of images in a printed collage provides more flexibility in matching the target color histogram. Constraints may be introduced that determine a range of scaling values such as +/-10% or +/-20%.

[0056] This calculation may be performed using a pseudo-inverse matrix operation. However as negative weights may be possible a constrained leastsquares (defining a minimum and maximum for each weight) or convex programming (constrains weights to [0,1]) approach can be used which limits the number of images used in the final combined image or collage. This approach is fast and also provides an exact solution or match for the target color histogram when no constraints are applied to the weights.

[0057] The preceding description has been presented to illustrate and describe examples of the principles described. This description is not intended to be exhaustive or to limit these principles to any precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is to be understood that any feature described in relation to any one example may be used alone, or in combination with other features described, and may also be used in combination with any features of any other of the examples, or any combination of any other of the examples

Claims

What is claimed is:

1 . A method comprising: determining a target color histogram; providing a plurality of representative images each having a respective color histogram; selecting a sub-set of the plurality of representative images using the respective color histograms and the target color histogram.

2. The method of claim 1 , wherein the color histograms comprise respective frequency metrics for a plurality of color values.

3. The method of claim 2, wherein the color histograms comprise a plurality of color dimensions having a plurality of color ranges each with a respective frequency metric corresponding to the frequency of color values in said range.

4. The method of claim 2, wherein the target color histogram comprises an average of the frequency metrics of color histograms associated with historically printed images.

5. The method of claim 2, wherein selecting the sub-set of representative images is based on a difference metric between the frequency metrics of the color histograms of the representative images and the frequency metrics of the target color histogram.

6. The method of claim 5, wherein the difference metric comprise one or more of the following: the average differences between frequency metrics of respective color values; the maximum differences between frequency metrics of respective color values; the standard deviations between frequency metrics of respective color values.

7. The method of claim 5, wherein the subset of representative images comprises a predetermined number of representative images and selecting the subset of representative images comprises iterating through all combinations of the predetermined number of representative images to select the combination of the predetermined number of images with the lowest difference metric.

8. The method of claim 5, wherein the subset of representative images comprises a predetermined number of representative images and selecting the sub-set of representative images comprises iterating through an initial set of representative images to transfer representative images having a lowest difference metric from the initial set to the subset of representative images.

9. The method of claim 2, wherein the subset of representative images comprises a number of representative images which are weighted such that the sum of the frequency metrics of the respective color histograms matches the frequency metrics of the target color histogram.

10. The method of claim 1 , comprising printing the sub-set of the plurality of representative images.

11 . The method of claim 10, comprising: scanning the printed sub-set of the plurality of representative images; determining a printing metric of the scanned printed sub-set of the plurality of representative images.

12. A printing apparatus comprising: a printhead assembly; a processor; a memory storing instructions for execution by the processor and a plurality of representative images each having a respective color histogram, wherein the instructions, when executed by the processor, cause the processor to: determine a target color histogram; select a sub-set of the plurality of representative images using the respective color histograms and the target color histogram; print the sub-set of the plurality of representative images.

13. The apparatus of claim 12, wherein the color histograms comprise a plurality of color dimensions having a plurality of color ranges each with a respective frequency metric corresponding to the frequency of color values in said range.

14. The apparatus of claim 12, configured to select the sub-set of representative images using a difference metric between the frequency metrics of the color histograms of the representative images and the frequency metrics of the target color histogram.

15. A non-transitory computer-readable storage medium comprising a set of computer-readable instructions that, when executed by a processor, cause the processor to: determine a target color histogram; provide a plurality of representative images each having a respective color histogram; select a sub-set of the plurality of representative images using the respective color histograms and the target color histogram.