WO2017205066A1

WO2017205066A1 - Method for rendering color images

Info

Publication number: WO2017205066A1
Application number: PCT/US2017/032148
Authority: WO
Inventors: Kenneth R. Crounse
Original assignee: E Ink Corporation
Priority date: 2016-05-24
Filing date: 2017-05-11
Publication date: 2017-11-30
Also published as: ES2812176T3; CN109074672B; US20190182406A1; PT3465628T; JP6599569B2; CN109074672A; KR20180119688A; US11265443B2; TW201804456A; US20170346989A1; JP2019515340A; US10771652B2; CN112331122A; PL3465628T3; EP3465628B1; US10554854B2; US20200137263A1; US20200404121A1; EP3465628A1; CN112331122B

Abstract

An image is rendered on a display having a limited number of primary colors by (104) combining input data representing the color of a pixel to be rendered with error data to form modified input data, determining in a color space the simplex (208 - typically a tetrahedron) enclosing the modified input data and the primary colors associated with the simplex, converting (210) the modified image data to barycentric coordinates based upon the primary colors associated with the simplex and (212) setting output data to the primary having the largest barycentric coordinate. calculating (214) the difference between the modified input data and the output data for the pixel, thus generating error data, applying (106) this error data to at least one later-rendered pixel, and applying the output data to the display and thus rendering the image on the display. Apparatus and computer-storage media for carrying out this process are also provided.

Description

METHOD FOR RENDERING COLOR IMAGES

[Para 1] This application is related to U.S. Publication No. 2014/0340430; U.S. Publication No. 2016/0091770; and United States Patents Nos. 9,383,623 and 9, 170,468. Other related applications and patents will, be discussed below.

[Para 2] This invention relates to a method for rendering color images. More specifically, this invention relates to a method for half-toning color images in situations where a limited set of primary colors are available, and this limited set may not be well structured. The method of the present invention is particularly, although not exclusively, intended for use with color eiectrophoretic displays

[Para 3] Half-toning has been used for many decades in the printing industry to represent gray tones by covering a varying proportion of each pixel of whi te paper with black ink. Similar half-toning schemes can be used with CMY or CMYK color printing systems, with the color channels being varied independently of each, other.

[Para 4] However, there are many color systems in which the color channels cannot be varied independently of one another, in as much as each pixel can display any one of a limited set of primary colors (such systems may hereinafter be referred to as "limited palette displays" or "LPD's"); the BCD patent color displays are of this type. To create other colors, the primaries must be spatially dithered to produce the correct color sensation. It is known to effect such spatial dithering by using, for any desired, color, only the primary colors at the vertices of a tetrahedron which contains the desired color; see, for example:

Arad, N., Shaked, D., Baharav, Z., & Lin, Q. (1999). Barycentric

Screening and

Ostromoukhov, Victor, and Roger D. Hersch. "Multi-color and artistic dithering. " Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesiey Publishing Co., 1999.

Both these documents effect dithering by means of a threshold-array based screening method, which is a simple dithering method that has been found not to give good results in ECD patent displays. [Para 5] Standard dithering algorithms such as error diffusion algorithms (in which the "error" introduced by printing one pixel in a particular color which differs from the color theoretically required at that pixel is distributed among neighboring pixels so that overall the correct color sensation is produced) can be employed with limited palette displays. However, such standard algorithms are typically intended for use with a limited palette which is "well structured", in the sense that the distances in the appropriate color space between the primary colors are substantially constant. There is considerable literature on the problems of designing optimal color palettes that perform well with error diffusion; see, for example :

Kolpatzik, Bernd W., and Charles A. Bouman. "Optimized Universal Color Palette Design for Error Diffusion." Journal of Electronic Imaging 4.2 (1995): 131443.

However, in ECD and similar limited palette displays, in which the limited palette is defined by the colors capable of being generated by the system, the limited palette may not be well structured, i.e., the distances between the various primaries in the color space may differ greatly from one another.

[Para 6] Figure 1 of the accompanying drawings is a schematic flow diagram of a prior art palette based error diffusion method, generally designated 100. At input 102, color values

are fed to a processor 104, where they are added to the output of an error filter 106 (described below) to produce a modified input

The modified inputs are fed to a quantizer 108, which also receives details of the palette {Pk} of the output device. The quantizer 108 determines the appropriate color for the pixel being considered, given by:

and feeds to appropriate colors to the device controller (or stores the color values for later transmission to the device controller). Both the modified inputs

and the outputs are fed to a processor 110, which calculates error values where:

The error values are then fed to the error filter 106, which serves to distribute the

error values over one or more selected pixels. For example, if the error diffusion is being carried out on pixels from left to right in each row and from, top to bottom in the image, the error filter 106 might distribute the error over the next pixel in the row being processed, and the three nearest neighbors of the pixel being processed in the next row down. Alternatively, the error filter 106 might distribute the error over the next two pixels in the row being processed, and the nearest neighbors of the pixel being processed in the next two rows down. It will be appreciated that the error filter need not apply the same proportion of the error to each of the pixels over which the error is distributed; for example when the error filter 106 distributes the error over the next pixel in the row being processed, and the three nearest neighbors of the pixel being processed in the next row down, it may be appropriate to distribute more of the error to the next pixel in the row being processed and to the pixel immediately below the pixel, being processed, and less of the error to the two diagonal neighbors of the pixel being processed.

[Para 7] Unfortunately, it has been found that if one attempts to use conventional error diffusion methods such as that shown in Figure 1 to ECD and similar limited palette displays, severe artifacts are generated which may render the resultant images unusable. For example, in one type of artifact, (hereinafter called a "transient" artifact) when stepping from one input color to a. next very different color, the spatial transient can be so long that the output never settles to the correct average even across the size of object being rendered. In another type of artifact (hereinafter called a "pattern jumping" artifact), for a constant color input image, the output jumps between two different sets of primaries at a seemingly random position in the image. Although both sets of primaries should ideally produce output close to the color being requested, the resultant output is not robust because small changes in the system can. cause these switching between the two sets and the texture change at such a jump is al so noticeable and. unpleasant.

[Para 8] The present invention seeks to provide a method of rendering color images which can be used with palettes which are not well, structured, and may be large, without producing transient and pattern jumping artifacts to which standard error diffusion methods are susceptible.

[Para 9] In one aspect, this invention provides a method of rendering an image on a display, the method comprising:

receiving input data representing the color of a pixel to be rendered: combining the input data with error data generated from a least one pixel previously rendered to form modified input data;

determining in a color space the simplex enclosing the modified input data, and the display primary colors associated with the simplex;

converting the modified image data to barycentric coordinates based upon the simplex and setting output data to the primary having the largest barycentric coordinate;

calculating the difference between the modified input data and the output data for the pixel and thereby generating error data for the pixel;

applying the error data thus generated to at least one later-rendered pixel; and

supplying the output data for a plurality of pixels to the display and thereby rendering the image on the display.

[Para 10] In one form of this process, the modified input data is tested to determine whether it is within the color gamut of the display and, if the modified input data is outside this color gamut, the modified input data is further modified by being projected on to the color gamut. This projection may be effected towards the neutral axis of the color space along lines of constant lightness and hue. Alternatively, the projection may be effected towards to the color represented by the input data for the pixel until the gamut boundary is reached. Typically, the color space used will be three-dimensional, so that the simplex will be a tetrahedron. The error data may, and typically will be, spread over more than one pixel. For example, if the method of the present invention is effected using a top-to-bottom and left-to-right order of pixel processing, the error data will normally be spread over at least the pixel to the right of, and the pixel below, the pixel being rendered. Alternatively, the error data may be spread over the pixel to the right of, and the three pixels below and adjacent the pixel being rendered. Especially, in the latter case, it is not necessary that an equal proportion of the error data be spread over all the pixels to which it is dispersed; for example, when the error is spread over the pixel to the right of, and the three adjacent pixels in the next row, it may be advantageous to assign more of the error data to the two pixels which share an edge with the pixel being rendered, as opposed to the two pixels which only share a vertex. [Para 11] The present invention extends to an apparatus comprising a display device having a plurality of pixels, each of which is arranged to display any one of a plurality of primary colors, and a computing device capable of carrying out the method of the present invention and supplying its output data to the display device, thereby causing the display device to display an image,

[Para 12] The present invention also extends to a non-transitory computer storage medium comprising instructions that when executed by a processor cause the processor to carry out the method of the present invention.

[Para 13] As already noted, Figure 1 of the accompanying drawings is a flow diagram of a prior art method for rendering a color image.

[Para 14] Figure 2 is a similar flow diagram illustrating the method of the present invention.

[Para 15] ^'The present invention is based upon the recognition that the transient and pattern jumping artifacts discussed above result from the fact that the quantizer (108 in Figure 1) has available to it an under-constrained list of primaries. In a three dimensional color space, any color in the device gamut can be rendered by dithering only four primaries, and the present invention is based upon constraining the choice of primaries in an appropriate way to ensure that only a restricted set of primaries are used during quantization.

[Para 16] ^'The subset of primaries that can used in a dither pattern to represent a given color is not unique; for example in a three dimensional color space, any set of four or more primaries which define a volume in the color space enclosing the given color can be used in a dither pattern. Even if one restricts the subset of primaries to only four, any set of four primaries which define a tetrahedron enclosing the given color can be used. However, to avoid pattern jumping artifacts, the assignment of subsets of primaries to particular colors should be made in such a way that any parametric path through color space results in a smooth change in proportions of the various primaries used with respect to the parameter. This can be achieved by decomposing the total gamut of the system (the convex hull of all the primaries) into tetrahedra with primaries as vertices and then assigning to each color to be rendered the subset of primaries corresponding to the vertices of its enclosing tetrahedron. This may be effected by Deiaunay triangularization, which decomposes the convex hull of the primaries into a set of tetrahedra, the circumspheres of which do not enclose any vertex from another tetrahedron. This is convenient, hist other decompositions of the color gamut may also be beneficial; for example, to reduce halftone graininess, the subsets of primaries could be chosen to have low variation in lightness. It will be appreciated that the decomposition methods can be generalized to color spaces of any number of dimensions by the use of the appropriate simplexes for the numbers of dimensions involved instead of using tetrahedra in a three dimensional space.

[Para 17] A preferred embodiment of the process of the invention is illustrated in Figure 2 of the accompanying drawings, which is a schematic flow diagram generally similar to Figure 1. As in the prior art method illustrated in Figure 1 , the method illustrated in Figure 2 begins at an input 102, where color values

are fed to a processor 104, where they are added to the output of an error filter 106 to produce a modified input

(Again, this description assumes that the input values

are such that the modified inputs

are within the color gamut of the de vice. ) If this is not the case, some preliminary modification of the inputs or modified inputs may be necessary to ensure that they lie within the appropriate color gamut.) The modified inputs

are, however, fed to a gamut projector 206.

[Para 18] The gamut projector 206 is provided to deal with the possibility that, even though the input values

are within the color gamut of the system, the modified inputs may not be, i.e., that the error correction introduced by the error filter 1.06 may take the modified inputs

outside the color gamut of the system. In such a case, it would not be possible to choose a subset of primaries for the modified input

since it would lie outside all defined tetrahedra. Although other ways of this problem can be envisioned, the only one which has been found to give stable results is to project the modified value

on to the color gamut of the system before further processing. This projection can be done in numerous ways: for example, projection may be effected towards the neutral axis along constant lightness and hue. However, the preferred projection method is to project towards the input color until the gamut boundary is reached.

[Para 19] The projected inpu

values are fed to a simplex finder, which returns the appropriate subset of primaries { Pks} , to a processor 210, which also received the projected input values, and converts them to barycentric coordinates of the

tetrahedron (or other simplex) defined by the subset of primaries {Pks} . Although it might appear that the subset of primaries {P_ks } should be based on those assigned to the input pixel color

this will not work; the subset of primaries must be based upon the projected input values. The output λ of processor 210 is supplied to a quantizer 212,

the function of which is very different from mat of the quantizer 108 shown in Figure 1. Instead of performing conventional error diffusion, the quantizer 212 chooses the primary associated with the largest barycentric coordinate. This is equivalent to a barycentric thresholding with the threshold (1/3,1/3, 1/3) (see the aforementioned Arad et al. document), which is not equivalent to the minimum distance determination carried out by quantizer 108 in Figure 1. The output jy from quantizer 212 is then sent to the device controller in the usual manner, or stored.

[Para 20] The output

values, and either the modified input values

or the projected input values

(as indicated by the broken lines in Figure 2), are supplied to a processor 214, which calculates error values

by:

(depending upon which set of input values are being used) and passes this error signal on to the error filter 106 in the same way as described above with reference to Figure 1.

[Para 21] in theory, it would appear that the error values should be calculated using the original modified input values uu rather than the projected input values u 'u, since it is the former which accurately represents the difference between the desired and actual colors of the pixel; in effect, using the latter values "throws away" the error introduced by the projection step. Empirically, it has been found that which set of input values is used does not have a major effect on the accuracy of the color representation. Furthermore, in deciding whether to use the input values before or after the projection in the error calculation, it is necessary to take account of the type of projection effected by the gamut projector 206. Some types of projection, for example projection along lines of constant hue and lightness, provide a continuous and fixed extension of the quantizer domain boundaries to the out-of-gamut volume, and thus permit the use of the unprojected input values in the error calculation without risk of instability in the output values. Other types of projection do not provide both a continuous and fixed extension of the quantizer domain boundaries; for example, projection towards the input color until the gamut boundary is reached fails to provide a fixed extension of the quantizer domain boundaries but instead the quantizer domains change with input values, and in these cases the projected input values should be used to determine the error value, since using the unprojected values could result in an unstable method in which error values could increase without limit.

[Para 22] From the foregoing, it will be seen that the present invention can provide improved color in limited palette displays with fewer artifacts than are obtained using conventional error diffusion techniques. The present invention may be used in display systems capable of displaying a continuum of colors (or at least a very large number of colors) but in which the available primaries are not evenly spread throughout the color gamut; for example interference based displays which control a gap width can display a large number of colors at each pixel, but with a pre-determined structure among the primaries, which lie on a one-dimensional manifold. The present invention may also be used with electrochromic display s.

[Para 23] For further details of color display systems to which the present invention can be applied, the reader is directed to the aforementioned ECD patents (which also give detailed discussions of electrophoretic displays) and to the following patents and publications:

U.S. Patents Nos. 6,01 7,584; 6,545,797; 6,664,944; 6,788,452; 6,864,875; 6,914,714; 6,972,893; 7,038,656; 7,038,670; 7,046,228; 7,052,571 ; 7,075,502; 7, 167, 155; 7,385,751; 7,492,505; 7,667,684; 7,684, 108; 7,791,789; 7,800,813; 7,821,702; 7,839,564; 7,910, 175; 7,952,790; 7,956,841 ; 7,982,941; 8,040,594; 8,054,526; 8,098,41 8; 8, 159,636; 8,213,076; 8,363,299; 8,422, 116; 8,441 ,714; 8,441 ,716; 8,466,852; 8,503,063; 8,576,470; 8,576,475; 8,593,721; 8,605,354; 8,649,084; 8,670, 174; 8,704,756; 8,717,664; 8,786,935; 8,797,634; 8,810,899; 8,830,559; 8,873, 129; 8,902, 153; 8,902,491 ; 8,917,439; 8,964,282; 9,013,783; 9, 1 16,412; 9, 146,439; 9, 164,207; 9, 170,467; 9, 170,468; 9, 182,646; 9, 195, 11 1 ; 9, 199,441; 9,268, 191; 9,285,649; 9,293,511 ; 9,341,916; 9,360,733; 9,361,836; 9,383,623; and 9,423,666; and U.S. Patent Applications Publication Nos. 2008/0043318; 2008/0048970; 2009/0225398; 2010/0156780; 201 1/0043543; 2012/0326957; 2013/0242378; 2013/0278995; 2014/0055840; 2014/0078576; 2014/0340430; 2014/0340736; 2014/0362213; 2015/0103394; 2015/0118390; 2015/0124345; 2015/0198858; 2015/0234250; 2015/0268531; 2015/03012/46; 2016/0011484; 2016/0026062; 2016/0048054; 2016/0116816; 2016/0116818; and 2016/0140909.

Claims

1. A method of rendering an image on a display, the method comprising:

receiving input data (102) representing the color of a pixel to he rendered;

combining (104) the input data with error data

generated from a least one pixel previously rendered to form modified input data

converting (210, 21 2) the input data to output data

calculating (214) the difference between the modified input data

and the output data

for the pixel and thereby generating error data for the

pixel;

applying (106) the error data

thus generated to at least one later- rendered pixel; and

supplying the output data

for a plurality of pixels to the display and thereby rendering the image on the display,

the method being characterized by:

determining (208) in a color space the simplex enclosing the modified input data and the display primary colors associated with the simplex;

converting (210) the modified image

data to barycentric coordinates based upon the simplex and setting (212) output data

to the primary having the largest barycentric coordinate.

2. The method of claim 1 further comprising testing (206) the modified input data

) to determine whether it is within the color gamut of the display and, if the modified input data is outside this color gamut, further modifying the modified input data by projecting the modified input data on to the color gamut.

3. The method of claim 2 wherein the projection is effected towards the neutral axis of the color space along lines of constant lightness and hue.

4. The method of claim 2 wherein the projection is effected towards the color represented by the input data for the pixel until the gamut boundary is reached.

5. The method of any one of claims 2 to 4 wherein the modified input data

produced by the projection is used for both the conversion to barycentric coordinates (210) and for the calculation of the error data (214),

6. The method of any one of claim 2 to 4 wherein the modified input data

y produced by the projection is used for the conversion to barycentric coordinates (210) but the modified image data

prior to the projection is used for the calculation of the error data (214).

7. The method of any of the preceding claims wherein the color space is three-dimensional so that the simplex is a tetrahedron.

8. The method of any of the preceding claims wherein the error data is spread over more than one pixel.

9. The method of claim 8 wherein the error data

is spread over at least four pixels.

10. The method of claim 9 wherein the proportion of the error data applied to differing pixels varies.

1 1. The method of any of the preceding claims wherein the display is an electrophoretic display.

12. An apparatus comprising a display device having a plurality of pixels, each of which is aixanged to display any one of a plurality of primary colors, and a computing device capable of carrying out the method of any one of claims 1 to 11 and supplying its output data to the display device, thereby causing the display device to display an image.

13. A non-transitory computer storage medium comprising instructions that when executed by a processor cause the processor to cany out the method of any one of claims 1 to 11.