EP1387319B1

EP1387319B1 - Image processing apparatus

Info

Publication number: EP1387319B1
Application number: EP20030250945
Authority: EP
Inventors: Keith Robert Calder; Nordine Chaibelaine; Stuart Jonathan Levine
Original assignee: Sony United Kingdom Ltd
Current assignee: Sony Europe Ltd
Priority date: 2002-02-22
Filing date: 2003-02-17
Publication date: 2006-09-13
Anticipated expiration: 2023-02-17
Also published as: GB0204239D0; DE60308282T2; EP1387319A2; DE60308282D1; EP1387319A3; CN1440001A; GB2385764A; CN1253016C

Description

Field of the Invention

The present invention relates to image processing apparatus and methods for enhancing colour images.

Background of the Invention

For most applications, an amount of data required to represent colour images demands that some form of compression or bandwidth reduction is performed in order to reduce the amount of data, which must be communicated or stored. This is because colour images require at least a representation of the red, green and blue (RGB) components of the image to be communicated or stored in order to provide a faithful representation of the original colour of the image. Next generation mobile radiotelephones may require a facility to receive and display colour images over a wireless link for display on a screen provided as part of the telephone. Similarly, for applications such as Personal Digital Assistants (PDA) there is a requirement to communicate and store colour images for display. For such applications, the colour images may be either received in compressed form or compressed when received in order to reduce an amount of data, which is required in order to communicate or store the image.
One technique for reducing an amount of information, which is required in order to store a representation of a colour image, is to reduce a dynamic range of the RGB signal samples by reducing a number of bits that are used to represent the RGB components. For example, if each pixel is represented as 18-bits, each RGB component will be allocated a 6-bit value providing a range of 64 possible colour values for each component, producing 262144 different colours. If the number of bits used to represent each component is reduced to 4-bits, each pixel can be represented as 12-bits providing a 33% reduction in the size of a memory required to store the colour image. However, a result of this reduction is to reduce the number of possible colours to 4096, which can be represented by each pixel, thereby reducing the so-called "colour palette" available to represent the image. As a result, the quality of the stored image when reproduced may be affected. For example, certain images such as faces or sun sets may suffer from contour effects, which appear on certain areas of the image due to a limitation on the number of available colours, resulting from the reduced colour palette. Preferably, therefore a reduction in the number of bits used to represent the signal samples should be provided without reducing the quality of the reproduced image.
"Bit Depth" (Stephen Canning) is an article describing how it is possible to reduce the amount of data required to store a digital image by decreasing its bit field depth. The concept of bit field depth is explained, a list showing how varying the bit field depth changes the number of colours that can be represented per pixel is given, and it is pointed out that an alteration in bit field depth can be achieved by saving an image as a GIF file.

Summary of Invention

The present invention is defined by the appended claims.
According to the present invention there is provided an image processing apparatus operable to process input colour component signal samples representative of an input colour image to produce output colour component signal samples representing an enhanced version of the input colour image. The apparatus comprises an enhancing processor operable to increase the dynamic range of the input colour component samples, a low pass filter operable to filter the increased dynamic range colour samples to form the output colour component samples, and a detection processor. The detection processor is operable to detect input colour component samples associated with a representation of detail appearing in the input colour image. Consequent upon detection of the detail in the image, the detection processor is operable to adjust the bandwidth of the low pass filter in correspondence with the detected detail.
Embodiments of the present invention are arranged to provide a facility for reducing an amount of data which is required to represent a colour image, whilst improving or at least maintaining the quality of the image when reproduced. To this end, an image processor embodying the present invention is provided with a colour enhancing processor which is arranged to improve a dynamic range with which the colour components of the image are represented by increasing a number of bits used to represent the RGB colour component samples. The increased dynamic range samples are then filtered with a low pass filter in order to provide a smooth transition between the colours of the reproduced image. However, in order to reduce a likelihood of losing detail within the reproduced image a detection processor is arranged to detect signal samples representative of the detail within the image and to adjust the band width of the low pass filter in correspondence with the detected detail. In one embodiment the samples representing the detail are arranged to bypass the low pass filter. As a result, a likelihood of losing detail in the image is reduced, whilst reducing a likelihood of contour effects which may appear in the image as a result of a reduced colour palette caused by a low dynamic range of the colour component signal samples.
Further advantageous features of the present invention will become apparent from a description of embodiments. In particular, the image processor providing the enhanced dynamic range colour samples may be implemented with a small number of components, providing a cost effective implementation.
Various further aspects and features of the present invention are defined in the appended claims.

Brief Description of Drawings

Embodiments of the present invention will now be described by way of example only with reference to the accompanying drawings, where like parts are provided with corresponding reference numerals, and in which:

Figure 1 is a schematic block diagram of an LCD display module;
Figure 2 is a schematic block diagram of an LCD display module including an image processor according to an embodiment of the invention;
Figure 3 is a representation of an example colour image which suffers from contour effects resulting from a limited colour palette;
Figure 4 is a schematic block diagram of the image processor appearing in Figure 2;
Figure 5 is a schematic block diagram of a dynamic range converter appearing in Figure 4;
Figure 6 is a table providing an illustration of values required to increase signal samples in accordance with an increased dynamic range with respect to a corresponding increase produced by copying most significant bits into least significant bits;
Figure 7 is a schematic illustration of an impulse response of a moving averaging filter of eight taps duration;
Figure 8 is a schematic block diagram of a low pass filter appearing in Figure 4;
Figure 9 is a schematic block diagram of a detection processor appearing in Figure 4;
Figure 10 is a flow diagram representing the operation of the image processor shown in Figure 4; and
Figure 11 is a representation of the example colour image after being processed by the image processor shown in Figure 3.

Description of Preferred Embodiments

An embodiment of the present invention will now be described with reference to an LCD display module. Figure 1 provides an illustration of a typical LCD display module such as that which would be used within a mobile radiotelephone or a personal digital assistant. As mobile radiotelephones move towards the next generation, the so-called third generation (3G), there will be an increasing demand to display digital images on a display module of mobile radiotelephones. A typical display module for this application is an LCD display panel and as shown in Figure 1, such a module comprises a display memory 1 which is arranged to store signal samples representative of a digital image which is to be displayed on an LCD display panel 2. Pixels which are representative of the colour image to be displayed are received on a connecting channel 4 and written into the display memory 1 using write address logic 8. When the image is to be displayed, the signal samples, stored in the memory 1, are read by read address logic 10 and converted into the analogue domain by a digital to analogue converter 12 before being displayed on the LCD panel 2.
As already explained, the present invention provides a facility for reducing the amount of data storage, that is the size of the display memory 1 for the present example, whilst maintaining the quality of the image to be displayed on, for example the LCD panel 2. The reduction in the amount of data storage is achieved by reducing the dynamic range of the signal samples representing the colour components. Alternatively, the colour image may be received with component signal samples having a reduced dynamic range as a result of being truncated, which may be required to communicate the image. As will be appreciated the RGB components will be truncated by the same amount. For both these examples, an image processor embodying the present invention is arranged to improve a representation of the received or stored colour image. As will be explained, the improvement can facilitate a faithful or at least improved reproduction of the original image with a reduced likelihood of contour effects, which may appear in the image.
An embodiment of the present invention is illustrated in Figure 2. Figure 2 shows the LCD display module of Figure 1 in combination with an image processing circuit 20. As shown in Figure 2, the pixels of the colour image as received on channel 4 are represented as only 12 bits rather than 18 bits as shown in Figure 1. The reduction in the resolution of the signal samples may be provided by a colour truncating processor 15. The colour truncating processor 15 may reduce the dynamic range of the signal samples by truncating the signal samples from 18-bits to 12-bits by discarding the two least significant bits of each RGB component. As such, the RGB colour components are represented as 4 bits rather than 6 bits, resulting in a 33% reduction in a storage requirement of the display memory 1.
A result of reducing the dynamic range of the colour signal samples by truncating from 6 bits to 4 bits is to reduce the colour palette available for displaying the reproduced image. As a result a colour image may suffer a reduction in quality which can cause contours or apparent flat sections to appear in the reproduced image. An example of such a colour image suffering from contour effects as a result of a reduced colour palette is shown in Figure 3.
In order to reduce the likelihood of contour effects appearing in the reproduced image, the display module according to the present invention is provided with an image processor 20. The image processor 20 is arranged to increase the dynamic range of the colour component signal samples, which are read out of the display memory 1. As shown in Figure 2, for the example embodiment illustrated, the number of bits, which are used to represent each pixel, is increased from 4-bits to 6-bits. The image processor 20 is shown in more detail in Figure 4.
In Figure 4 the image processor 20 receives the 4 bit RGB colour component signal samples on a connecting channel 22 from the read address logic 10. The colour component signal samples are fed to a colour dynamic range converter 30 and a detection processor 34 in parallel. The dynamic range converter 30 is arranged to increase the number of bits, which are used to represent the colour component signal samples from 4 bits to 6 bits. The enhanced colour component signal samples are then fed to a low pass filter 36 after which the enhanced signal samples are fed to an output channel 24. As shown in Figure 2, the enhanced signal samples are fed to a digital to analogue converter 12 as shown in Figure 4.
The detection processor 34 is arranged to generate a filter selection control signal, which is fed on a channel 38 to the low pass filter 36.
In accordance with the increased dynamic range provided by the dynamic range converter 30, the values of the 4-bit signal samples can be scaled to reflect the enhanced dynamic range provided by increasing the number of bits to six. In order to adjust the colour component sample values in accordance with the increased dynamic range, the values of the input signal samples are multiplied by again factor G. The gain factor G is determined in accordance with the original resolution of n-bits and the increase in the resolution to q-bits as represented by the following equation: $G = \frac{2^{q} - 1}{2^{m} - 1}$
To reduce a number of components required to implement the gain factor G, the gain factor G is applied by copying the two most significant bits from the 4 bit input sample into the two least significant bits appended to the 6-bit output sample. Such an arrangement for applying the gain to the input signal samples is illustrated in Figure 5. In Figure 5, the 4 bit input signal sample is represented as the letters A B C D which are stored in a four stage register 40. A six stage register 42 is provided for forming the output signal sample. Connecting the four stage register to the six stage register are connecting channels 44. As shown in Figure 5 the connecting channels arrange for the 2 bit values A, B in the first two stages of the shift register 40 to be copied into the last two stages of the register 42. Thus, the values of the two least significant bits of the output signal sample are formed from the most significant bits of the input signal samples.
Figure 6 provides a table illustrating all possible values of the 4-bit input signal samples with respect to the gain required to adjust the signal sample values from 4-bits to 6-bits, to take advantage of the increased dynamic range. The gain, as calculated by the above formula, is shown with respect to the gain, which is provided by bit replication in accordance with the arrangement of registers shown in Figure 5. As can be seen in Figure 6 the bit replication arrangement provides a good approximation to the calculated gain factor G but has an advantage of not requiring any multiplier components, thus reducing implementation costs.
The enhanced colour component signal samples are filtered by a low pass filter 36 in order to provide a smooth transition between colour values. Filtering the signal sample values has the effect of reducing the likelihood of contour effects, which may appear in the reproduced image. In preferred embodiments, the low pass filter 36 is implemented as a moving averaging filter having an impulse response as shown in Figure 7. Figure 7 illustrates an impulse response of an eight tap moving averaging filter providing an effective width of T_W.
A moving averaging filter effectively forms an output signal sample from the average values of the input signal samples within a window formed by the impulse response of the moving averaging filter. The filter is arranged to form a smoothing effect on the contours of the image caused by the limited dynamic range of the resolution of each signal sample.
For the present example shown in Figure 7 an eight tap moving averaging filter is used so that the output sample is formed from an average of the values within the eight tap window. As will be explained shortly, in other embodiments, as well as the eight tap moving averaging filter, the low pass filter 36 includes a four tap moving averaging filter and a two tap moving averaging filter. These are used to provide an increase in the pass bandwidth of the low pass filter as the samples representing detail move towards a centre tap of the eight-tap filter. This provides a more gradual amount of filtering to the enhanced signal samples, where the detection processor determines that these signal samples represent detail.
Figure 8 provides an illustration of the low pass filter 36 according to one example embodiment. In Figure 8 the low pass filter 36 has three moving averaging filters, which include the eight-tap filter having an impulse response shown in Figure 7. As shown in Figure 8 an eight-tap delay 50 is used to delay the input signal samples by eight samples. At the output of the eighth tap, adders 52, 54 are used to form the output sample in combination with a one-stage register 56. Effectively, the filter 36 shown in Figure 8 implements a moving averaging filter by accumulating the last eight signal sample values as performed by the adder 54 whilst subtracting the last value to pass thorough and fall outside the eight tap window as implemented by the adder 52. The adders 52, 54 and the register 56 therefore form an accumulation unit 58 to provide the calculation of the moving average output.
Effectively therefore, the accumulation unit 58 shown in Figure 8 implements a moving averaging filter without a requirement for summing and dividing all the signal sample components within an eight tap window to calculate every output signal sample. The circuit shown in Figure 8 therefore provides an advantage in reducing the number of components required to implement the moving averaging filter and for which no multipliers are necessary. Correspondingly, second and third accumulation units 58', 58" form output signal samples representing the four-tap and two-tap moving averaging filters. To this end, the second accumulator 58' is arranged to receive samples from the second stage and the sixth stage to form the four-tap moving averaging filter, whereas the third accumulator 58" is arranged to receive samples from the third stage and fifth stage. Thus the eight, four and the two tap filters are centred with respect to a common temporal reference.
As shown in Figure 8, the control signal is received from the channel 38 by a selection processor 59. The selection processor 59 has four input channels 51, 53, 55, 57. The four input channels 51, 53, 55, 57 are arranged to receive the output sample from the eight tap moving averaging filter 51, the output sample from the four tap moving averaging filter 53, the output sample from the two tap moving averaging filter 55, and the signal sample from the fourth stage of the shift register 50, respectively. The selection of the output from either the two tap, the four tap, or the eight tap output, or the bypass sample from the fourth stage of the shift register is determined in accordance with the control signal received from the detection processor 34.
In order to avoid a likelihood of reducing the clarity of detail such as edges or single pixel objects in the reproduced image, the detection processor is used to detect such detail. The detection processor then generates a control signal in order to select output samples from either the eight-tap, four-tap or two-tap moving averaging filter, or the bypass sample. Effectively, the selection signal adjusts the bandwidth of the low pass filter in accordance with the relative position of pixels representing detail with respect to the centre tap. Figure 9 provides an example implementation of the detection processor 34. As shown in Figure 9 the detection processor 34 comprises a shift register 60 having eight stages which correspond to the stages of the shift register 50 used in the low pass filter 36. As shown in Figure 9, an adder 62 is arranged to receive an input signal sample and a signal sample of the output of the second stage of the shift register 60. The adder 62 is arranged to subtract the newly received signal sample from the signal sample present in the second stage to form a difference at an output 64. A threshold processor 66 then compares the difference between the signal samples with a predetermined threshold and if the difference is greater than the threshold, the processor 66 produces a control signal on the control channel 38. The threshold is determined so that detail is detected if all of the three RGB signal samples are greater than the corresponding values of the RGB samples of a pixel from two sample periods earlier. However, in addition to the adder 62 a further adder 68 is arranged to calculate a difference between the output of the fourth stage and the second stage, which is also fed to the threshold processor 66.
The operation of the detection processor is represented by the flow chart in Figure 10. As shown in Figure 9 the latest signal sample is compared with the contents of the second previous signal sample as performed by the adder 62. The threshold processor 66 determines whether detail has been detected between these two compared signal samples, which is represented in Figure 10 as step S2.
As mentioned above, the threshold processor 66 also receives a comparison between the output of the second stage of the shift register and an output of the fourth stage of the shift register as performed by an adder 68. Therefore, at step S4 in the flow diagram the threshold processor determines whether the detail has been detected between effectively a latest signal sample and the second previous signal sample within a four tap window provided by sample values at the outputs of the third, fourth, fifth and sixth register stages.
If the threshold processor determines that detail has not been detected within the four-tap filter, but is present in the eight-tap filter, from a comparison of the contents of the second and fourth stage of the eight tap delay, then at step S4 the threshold processor determines that the four tap filter should be selected to produce the output pixel samples, at step S5. If the threshold detection processor determines that the detail samples is within the four tap filter but just before the two centre taps forming the two-tap filter the processing proceeds to step S8.
At step S8 the value of the signal sample in the fourth stage is also compared with a second threshold formed between the threshold value for detecting detail and a predetermined offset value, as expressed below: $Threshold < sample value < (Threshold + offset)$
If the signal sample value is greater than the threshold, but less than the threshold plus the offset value, then the control signal is generated on the channel 38 to select the output of the two-tap moving averaging filter 55, as performed by the step S10. If however the value of the signal sample is greater than the threshold plus the offset value, then the bypass sample is selected from the output of the fourth-tap of the shift register 50.
If detail is not detected at either step S2 or S4 then the processing proceeds via the loop S6 to the beginning of step S2 and the control signal is generated to select the output from the eight tap filter.
A further advantage is provided by selecting either the four-tap output or the two tap output independence upon the relative position of a sample representing detail in the colour image and the value of the samples representing the colour image. Effectively, the threshold processor is adjusting the bandwidth of the low pass filter in accordance with the relative frequency bandwidth of the pixels samples passing through the filter. Accordingly, as the detail pixels sample are detected within the eight tap window, but just before the four tap window, then the four tap output is selected. If the detail pixel samples are detected within the four taps window but just before the two-tap window, then the output from the two-tap window is selected. Otherwise the output samples from the eight tap moving averaging filter are selected. By selecting a degree of filtering in accordance with a relative position in the eight-tap window, an improved representation of the colour image is provided in which the likelihood of contour effects is reduced.
Furthermore, it has been found to improve the colour image quality if the two tap moving averaging filter is used to produce output samples, when samples representing detail in the colour image are determined to fall between the threshold and the threshold value plus the predetermined offset. The predetermined offset value is set to detect sample values which exceed the threshold value by a relatively small amount. It has been discovered that filtering such sample values with a two tap moving averaging filter provides an improvement in the reproduced colour image. This is because these sample values which just exceed the threshold but not the offset represent intermediate detail in the reproduced image which requires some limited degree of filtering with respect to the two immediately surrounding samples. Therefore as illustrated by step S7, S8 and S10, if the difference between signal samples is greater than the threshold plus the offset, then the filter is bypassed and the centre sample that is that contained in the fourth stage of the register 60 is passed to the output of the filter.
A result of processing an image represented as 12 bit signal samples shown in Figure 3 with the image processor 20 is shown in Figure 11. As can be seen in Figure 12 the contour effects which appear in the face of the women in Figure 3 have been removed. However the fine detail information has been preserved in this image.
Although in the example embodiment described above, the low pass filter is arranged to filter only one-dimensional signal samples, in alternative embodiments a two-dimensional filter may be used to provide a further improvement in the reproduced image. It will be appreciated therefore that the filter could be arranged to filter vertical or horizontal component samples of the image.
Various modifications may be made to the embodiments of the invention as herein before described without departing from the scope of the present invention. In particular, as will be appreciated a colour image may be compressed by representing the colour samples with an even smaller number of bits. The colour image may then be recovered by increasing the resolution of the bits as the colour signal samples are read out from the memory, to the original resolution in accordance with the embodiments described above. Thus it will be appreciated that resolutions other than 12 bits for storing the colour image, and the enhanced resolution samples of 18 bits are envisaged.

Claims

An image processing apparatus operable to process input colour component signal samples representative of an input colour image to produce output colour component signal samples representing an enhanced version of said input colour image, said apparatus comprising
an enhancing processor (20) operable to increase the dynamic range of the input colour component samples by increasing the number of bits used to represent each sample,
a low pass filter (36) operable to filter the increased dynamic range colour signal samples to form said output colour component samples, and
a detection processor (34) operable to detect input colour component samples associated with a representation of detail appearing in said input colour image, and consequent upon detection of said detail said detection processor is operable to adjust the band width of the low pass filter (36) in correspondence with the detected detail.
An image processing apparatus as claimed in Claim 1, wherein said detection processor (34) is operable to adjust the bandwidth of low pass filter (36) by selecting said increased dynamic range colour samples representing said detail as the output colour component signal samples, without passing through said low pass filter (36).
An image processing apparatus as claimed in Claim 1 or 2, wherein said input colour signal samples are represented as a predetermined number of bits n and said enhancing processor (20) is operable to increase the resolution of said input colour component samples by adding a predetermined number of additional least significant bits l to said input colour signal samples and copying the values of the corresponding number l of most significant bits of said input component signal samples into the l least significant bits.
An image processing apparatus as claimed in Claim 1, 2 or 3, wherein said low pass filter (36) includes a moving averaging filter having a predetermined number of taps d.
An image processing apparatus as claimed in Claim 4, wherein said input colour component samples are passed through said moving average filter to form said output colour component samples, and consequent upon detection of detail the samples of said input colour component samples associated with said detail are passed to said output when said associated samples reach a centre tap of said moving averaging filter.
An image processing apparatus as claimed in any preceding Claim, wherein said low pass filter (36) comprises a plurality of low pass filters, each having a different pass bandwidth, and the detection processor is operable to adjust the bandwidth of low pass filter by selecting the output of one of said low pass filters.
An image processing apparatus as claimed in Claim 6, wherein each of said low pass filters comprises a moving averaging filter, each having a different number of taps, each of said moving averaging filters providing a different length window of input signal samples centred with respect to a common temporal sample position, and said detection processor is operable to detect samples representative of said detail with respect to said common temporal reference and consequent upon said detection, to select the output signal sample from a corresponding one of said moving averaging filters, before said detail samples have entered the window of the filter.
An image processing apparatus as claimed in Claim 6 or 7, wherein said detection processor (34) is operable to detect the samples associated with said detail by comparing successive samples present in the taps of said moving averaging filters.
An image processing apparatus as claimed in Claim 8, wherein said detection processor (34) is operable to detect samples associated with detail by comparing each of the colour component samples of one pixel with the corresponding colour component sample of a successive pixel, and if each of the colour component samples of the successive pixel has changed, detecting detail represented by said successive pixel.
An image processing apparatus as claimed in Claim 8 or 9, wherein said compared samples are separated by two sample periods.
A display device comprising a display (2), a display memory (1) and an image processing apparatus as claimed in any of Claims 1 to 10, wherein input colour component samples representative of a colour image are stored in said display memory (1) prior to being displayed and said input colour component samples are read out from said display memory (1) and processed by said image processing apparatus before being displayed by said display device.
A display device as claimed in Claim 11, comprising
a compressing processor operable to receive said input colour component samples and to reduce a resolution of each of said samples by discarding at least one least significant bit of each said sample, said reduced resolution samples being stored in said memory, wherein a capacity of said memory required to store said colour image is reduced in accordance a number of said least significant bits of each said sample discarded.
A display device as claimed in Claim 11 or 12, wherein said display is a Liquid Crystal Display (LCD) (2).
A display device as claimed in Claim 11, 12 or 13, wherein enhanced colour component signal samples produced by said image processing apparatus are converted to analogue form by an analogue-to-digital converter (12) for display on said LCD display.
A computing or communicating device having a display device according to any of Claims 11 to 14.
A mobile radiotelephone having a display device as claimed in any of Claims 11 to 14.
A method of processing input colour component signal samples representative of an input colour image to produce output colour component signal samples representative of an enhanced version of said input colour image, said method comprising
increasing the dynamic range of the input colour component samples by increasing the number of bits used to represent each sample,
low pass filtering the increased dynamic range colour component signal samples to form said output colour component samples, and
detecting input colour component samples associated with a representation of detail appearing in said input colour image, and consequent upon detection of said detail, adjusting the band width of the low pass filter in correspondence with the detected detail.