WO2004107767A1

WO2004107767A1 - Capturing images from a display device

Info

Publication number: WO2004107767A1
Application number: PCT/IB2004/050724
Authority: WO
Inventors: Michael J. Mcdonald; Josephes A. Janssen
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2003-05-28
Filing date: 2004-05-18
Publication date: 2004-12-09

Abstract

An image acquisition unit (200) for imaging a display device (202) which is arranged to display a sequence of display images is disclosed. The image acquisition unit (200) comprises: image capture means (204) for capturing a first captured image (112), representing a first portion of the sequence of the display images and for capturing a second captured image (114), representing a second portion of the sequence of the display images; and combining means for creating an output image (120) by means of combining the first captured image (112) and the second captured image (114). The combining means (208) are arranged to compute the pixel values of the output image on basis of statistical operations, like minimum or maximum, of corresponding pixels of the captured images (112, 114).

Description

Capturmg images from a display device

The invention relates to an image acquisition unit for imaging a display device which is arranged to display a sequence of display images, the image acquisition unit comprising image capture means for capturing a first captured image, representing a first portion of the sequence of the display images and for capturmg a second captured image, representing a second portion of the sequence of the display images, and combining means for creating an output image by means of combmmg the first captured image and the second captured image

The invention further relates to a camera comprising - the image acquisition unit as described above, and a memory device for temporarily storing the output image

The mvention further relates to an inspection system for inspecting a display device which is arranged to display a sequence of display images, the inspection system compnsmg - the image acquisition unit as described above, and analyzing means for analyzing the output image

The mvention further relates to a method of imaging a display device which is arranged to display a sequence of display images, the method comprising capturing a first captured image, representing a first portion of the sequence of the display images and for capturmg a second captured image, representing a second portion of the sequence of the display images, and creating an output image by means of combining the first captured image and the second captured image

An embodiment of the image acquisition umt for imaging a display device is known from inspection systems which are used in factories for display devices Typically, display devices are tested at the end of an assembly line of display devices A method of testing these display devices compπses providing the display device with a video signal enabling the display device to display a sequence of display images; capturing images of the sequence of display image by means of an image acquisition unit comprising optical means like an image sensor; and - analyzing the captured images, e.g. by means of appropriate image analysis software running on a PC.

By applying this test scheme e.g. the homogeneity of a display device can be analyzed. In many cases the integration period of the image sensor and the display refresh rate of the display device do not match. In other words, the period during which the image sensor is acquiring a captured image differs from the period in which the display device is displaying a complete display image. As a result it might e.g. be that a part of the display device is updated twice during the integration period of the image sensor while the rest of the display device is only update once. That means that in that case, the captured image comprises a first group of pixel values which are based on integration of light corresponding to two display images and a second group of pixel values which are based on integration of light corresponding to a single display image. Typically, the first group of pixels represents a bright bar on the captured image.

Besides a mismatch between the integration period of the image sensor and the display refresh rate of the display device, it might be that there are also no synchronization means between the display device and the camera.

A known solution for the problem of having bright bars in the image, as described above, is applying a relatively long integration period of the image sensor. By applying an integration period which is e.g. 20-30 times longer than the time for displaying a single image, the effect of differences in contributions of the different display images for each of the pixels of the captured image is reduced. It will be clear that this approach is not appropriate in the case that a test is performed for analyzing the homogeneity of a display device.

Another solution for the problem of having bright bars in the image, as described above, is adjusting the integration period of the image sensor to the refresh rate of the display device. However, in many cases the refresh rate of the display device is unknown.

The problem as described above is also known by those who want to make photographs directly from a display device. This might be required by someone who wants to register a Powerpoint™ presentation which is shown by means of the display device during a conference. It will be clear that a relatively long integration period is also not appropriate in that case. The amount of blur caused by movement of the camera will deteriorate the captured image.

It is an object of the invention to provide an image acquisition unit of the kind described in the opening paragraph which is arranged to acquire images without unintended differences in brightness.

This object of the invention is achieved in that the combining means comprises processing means being arranged to determine a first value of a first one of the pixels of the output image on basis of a statistical operation on a second value of a corresponding pixel of the first captured image and a third value of a corresponding pixel of the second captured image. Instead of adding values of pixel, which is implicitly done in the case of applying a relatively long integration time, a selection of values of pixels of the different captured images is applied. The selection is based on a statistical operation. Preferably an order statistical operation. The working of the image acquisition unit according to the invention is based on the assumption that at least one of the values of the pixels of a set of corresponding pixels of the captured images is appropriate to be selected as an output pixel value.

In an embodiment of the image acquisition unit according to the invention, the processing means are arranged to determine the first value of the first one of the pixels of the output image by taking the minimum of the second value of the first captured image and the third value of the second captured image. Determining the minimum of a set of values is relatively easy. An other advantage of this embodiment according to the invention is that it is relatively robust. In another embodiment of the image acquisition unit according to the invention, the processing means are arranged to determine the first value of the first one of the pixels of the output image by taking the maximum of the second value of the first captured image and the third value of the second captured image. Determining the maximum of a set of values is relatively easy. An other advantage of this embodiment according to the invention is that it is relatively robust.

An embodiment of the image acquisition unit according to the invention comprises control means to control the processing means being arranged to determine the first value on basis of an integration period of the image capture means and a display refresh rate of the display device. On basis of the integration period of the image capture means and the display refresh rate of the display device an appropriate order statistical operation can be determined. That means that the image acquisition unit is arranged to select one of the types of available operations.

An embodiment of the image acquisition unit according to the invention is characterized in that the image capture means are arranged to capture a third captured image, representing a third portion of the sequence of the display images; and the combining means comprise processing means being arranged to determine the first value of the first one of the pixels of the output image on basis of a statistical operation on the second value of the corresponding pixel of the first captured image and the third value of the corresponding pixel of the second captured image and a fourth value of a corresponding pixel of the third captured image.

By applying more than two images the robustness of the selection of appropriate pixel values is improved. Besides a minimum or maximum operation also other types of order statistical operations can be applied by this embodiment according to the invention for the selection of the first value of the first one of the pixels of the output image, e. g. a median operation.

It is an other object of the invention to provide a camera of the kind described in the opening paragraph which is arranged to acquire images without unintended differences in brightness. This object of (he invention is achieved in that the combining means comprise processing means being arranged to determine a first value of a first one of the pixels of the output image on basis of a statistical operation on a second value of a corresponding pixel of the first captured image and a third value of a corresponding pixel of the second captured image. It is an other object of the invention to provide an inspection system of the kind described in the opening paragraph which is arranged to acquire images without unintended differences in brightness.

This object of the invention is achieved in that the combining means comprise processing means being arranged to determine a first value of a first one of the pixels of the output image on basis of a statistical operation on a second value of a corresponding pixel of the first captured image and a third value of a corresponding pixel of the second captured image. It is an other object of the invention to provide a method of the kind described in the opening paragraph with which images are acquired without unintended differences in brightness.

This object of the invention is achieved in that a first value of a first one of the pixels of the output image is determined on basis of a statistical operation on a second value of a corresponding pixel of the first captured image and a third value of a corresponding pixel of the second captured image.

Modifications of the image acquisition unit and variations thereof may correspond to modifications and variations thereof of the camera, the inspection system and the method described.

These and other aspects of the image acquisition unit, of the camera, the inspection system and of the method according to the invention will become apparent from and will be elucidated with respect to the implementations and embodiments described hereinafter and with reference to the accompanying drawings, wherein:

Figs. 1 A- ID schematically show the relations between display images and captured images;

Fig. 2 schematically shows an embodiment of the image acquisition unit according to the invention;

Fig. 3 schematically shows an embodiment of the camera according to the invention;

Fig. 4 schematically shows an embodiment of the inspection system according to the invention; and Fig. 5A and 5B schematically show how it is determined whether a minimum or maximum operation should be applied. Same reference numerals are used to denote similar parts throughout the Figs.

Figs. 1 A-1D schematically show the relations between display images and captured images. In Fig. 1 A four refresh periods T_d (1) , T_d (2) , T_d (3) and T_d (4) are depicted of a display device. It is assumed that during a particular refresh period the lines, i.e. rows, of the display device are sequentially provided with data resulting in generation of light by these respective lines. After the last line is provided with data the first of the lines of the display is provided again with new data. This type of image generation is common in e.g. CRT and organic LED displays. Furthermore it is assumed that all lines are provided with the same data. If the display device works appropriate this results in a sequence of homogeneous display images being generated by the display device. The four refresh periods T_d (1) , T_d (2) , T_d (3) and T_d (4) are mutually equal, but they differ from the duration of the integration periods of the image capture means, e.g. image sensor, of the image acquisition unit according to the invention. In Fig. IB three integration periods T_c (A) , T_c (B) and T_c (C) of the image capture means are depicted. The image capture means is not synchronized with the display device. Hence, the first integration period T_c (A) overlaps with a portion 1 of the first refresh period T_d (1) and with a portion 2a of the second refresh period T_d (2) . Furthermore, the second integration period T_c (B) overlaps with a portion 2b of the second refresh period T_d (2) and with a portion 3 a of the third refresh period T_d (3) . The third integration period T_c (C) overlaps with a portion 3b of the third refresh period T_d (3) and with a portion 4 of the fourth refresh period T_d (4) . In Fig. 1C three captured images 112-116 are schematically shown. Each of these captured images 112-116 comprises contributions of multiple display images. The first captured image 112 comprises a first contribution 100 on basis of the overlap between T_c (A) and T_d (2) and a second contribution 102 on basis of the overlap between T_c (A) and T_d (1) .

The second captured image 114 comprises a first contribution 104 on basis of the overlap between T_c (B) and T_d (3) and a second contribution 106 on basis of the overlap between

T_C(B) and T_d(2) . The third captured image 116 comprises a first contribution 108 on basis of the overlap between T_C(C) and T_rf (4) and a second contribution 110 on basis of the overlap between T_c (C) and T_d (3) .

The parts of the display device which have been refreshed twice during the different integration periods T_c (A) , T_c (B) and T_c (C) appear as bright bands in the captured images. It can be seen that the part of the display device which is refreshed twice progresses, i.e. moves, over the display device in successive integration periods T_c (A) , T_c (B) and T_c (C) .

In this case, there is no overlap in doubly refreshed parts in successive captured images 112- 116. Fig. ID schematically shows the resulting output image 120 which is based on a combination of captured images, e.g. the first captured image 112 and the second captured image 114. That means that the values of the pixels of the output image 120 are determined on basis of a minimum operation on corresponding pixels of the first captured image 112 and corresponding pixels of the second captured image 114. Alternatively, the values of the pixels of the output image 120 are determined on basis of a minimum operation on corresponding pixels of the first captured image 112, corresponding pixels of the second captured image 114 and corresponding pixels of the third captured image 116. The result of these minimum operations is in both cases a uniformly refreshed image, i.e. the output image 120 represents the intended image content.

Although in the example described above, and also in other examples below, the displayed images are all homogeneous it will be clear that the image acquisition unit 200 according to the invention can be applied also in the case that the display images are not homogeneous but represent a picture.

Fig. 2 schematically shows an embodiment of the image acquisition unit 200 according to the invention and a display device 202 which is arranged to display a sequence of display images. The display device can be any type of display, e.g. CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), PDP (Plasma Display Panel), organic LED (Light emitting diode). The image acquisition unit 200 comprises: an image capture unit 204 for capturing images, e.g. a first captured image 112, representing a first portion of the sequence of the display images and a second captured image 114, representing a second portion of the sequence of the display images; a memory device 206 for temporarily storage of captured images, e.g. the first captured image 112; and a processing unit 208 for creating an output image by means of combining captured images provided by the image capture unit 204 and the memory device 206. The processing unit 208 is arranged to determine a first value of a first one of the pixels of the output image on basis of a statistical operation on a second value of a corresponding pixel of the first captured image 112 and a third value of a corresponding pixel of the second captured image 114. The output image 120 is provided at the output connector 210 of the image acquisition unit 200. Optionally, the image acquisition unit 200 is arranged to compute the output image 120 on basis of more than two captured images. Preferably the image acquisition unit 200 provides multiple types of statistical operations, e.g. minimum, maximum or median, to combine a number of captured images. Also other types of statistical operations might be supported, e.g. the second value of a list of five values, or the sixth value of a list of seven values. A selection of the appropriate type of statistical operation might be based on visible inspection by an observer of the output images. However, preferably the processing unit 208 is arranged to determine the appropriate type of statistical operation automatically. For determining whether a minimum or maximum operator is to be used, the ratio between the display refresh period T_c and the integration period T_d of the image capture unit 204 is important. The phase relation is less important.

If the integration period T_c is an integer multiple of the display refresh period

T_d , then a uniformly exposed image can be made of the display device using a single captured image. This can be represented as T_c = N *T_d where N is an integer. However, also in that case the image acquisition unit 200 can be applied. The larger N is, the brighter the captured image of the display device. When this relation does not hold, i.e.T_c ≠ N *T_d then bright bars or dark bars will occur in the captured images. In a series of subsequent captured images these bars shift in position vertically over the display device as depicted in Fig. lC. Use is made of this fact by combining the information from consecutive captured images in such a series. The combining being carried out by e.g. a maximum or minimum operation. If less than half the display device is refreshed N + 1 times the majority of the display device is refreshed only N times and there is no overlap of display lines in consecutive captured images which have been refreshed N + 1 times, then a minimum operation is appropriate over two consecutive captured images. If more than half the image is refreshed N + 1 times the minority of the display is refreshed only N times and there is no overlap of display lines in consecutive captured images which have been refreshed N times, then a maximum operation is appropriate over two consecutive captured images.

If the display refresh period T_d is longer than the integration period T_c , then a part of the display device will not have been refreshed in the captured image. If the refresh period T_d is longer than twice the integration period T_c , then more than two captured images are needed to get a filled display in the image. The operation is preferably a maximum operation in these cases.

Examples of the various T_c : T_d relations are given in the following table.

Hence, there is a transition from minimum to maximum operation when T_c just exceeds an odd multiple of T_d and a transition from maximum to minimum operation when 2T_C just exceeds a multiple of T_d In cases where exactly half the display is refreshed N times and the other N + 1 times, either a maximum or minimum operation can be used See also the description in connection with Figs 5 A and 5B

Fig 3 schematically shows an embodiment of the camera 300 accordmg to the mvention and a display device 202 which is arranged to display a sequence of display images The display device can be any type of display, e g CRT, LCD, PDP, organic LED The camera 300 comprises an optical unit 302 for captunng images, e g a combination of a lens and an image sensor, an image acquisition umt 200 as described m connection with Fig 2, - a memory device 304 for temporarily storing output images, and an output connector 306 for providmg output images The camera might be a digital still photo camera. Alternatively, it might be part of a system like a mobile phone or it might be a Webcam. Besides capturing still pictures, it is also possible to combine series of video images, being shown by the display device, into a series of captured video images. Fig. 4 schematically shows an embodiment of the inspection system 400 according to the invention and a display device 202 which is arranged to display a sequence of display images. The inspection system 400 might be part of a production line of display devices or of an assembly line of apparatuses comprising a display device. The inspection system 400 comprises: - an optical unit 302 for capturing images, e.g. a combination of a lens and an image sensor; an image acquisition unit 200 as described in connection with Fig 2; analyzing means 402 for analyzing the output image; and an output connector 404 for providing the results of the analysis. Preferably the analyzing means 402 and the image acquisition unit 200 share the same processing unit 208. The analyzing might include the correct display of data being provided to the display device 202. A typical analysis is whether the different elements of the display device 202 generate the appropriate amount of light. If all elements are provided with the same data the differences between the amounts of generated light should be below a predetermined threshold. This is a so-called homogeneity test. It will be clear that other types of test images might be displayed by the display device 202 in order to test its functioning.

Fig. 5A and 5B schematically show how it is determined whether a minimum or maximum operation should be applied. This is especially relevant in the case of inspection. Suppose that the display device 202 is provided with data having a constant value, i.e. the displayed images should be substantially homogeneous. To automatically determine which operator to use (maximum or minimum), a comparison is made of a two sequences of pixel values Coll and Col2 corresponding to two columns of pixels of the two successive captured images. As explained in connection with Fig. 1C the bar (or band) in the captured image is moving. By determining if the bar is bright or dark the choice of operation can be made. If the bar is bright then a minimum operation should be applied. If the bar is dark then a maximum operation should be applied. The processing steps to determine whether there is a bright bar or a dark bar are the following:

The pixel values Col2 of the column from the second image are subtracted, by means of saturated arithmetic, from the corresponding pixel values Coll of the column in the first image resulting in a first difference signal |Coll-Col2|, representing two series 502, 504 of values being substantially equal to zero and a series 518 of values being unequal to zero. The pixel values Coll of the column from the first image are subtracted, by means of saturated arithmetic, from the corresponding pixel values Col2 of the column in the second image resulting in a second difference signal |Col2-Coll|, representing two series 506, 508 of values being substantially equal to zero and a series 520 of values being unequal to zero.

The two difference signals |Coll-Col2| and |Col2-Coll| are compared with each other. The relative positions of non-zero content values 518 and 520 indicate if the bar is bright or dark: if the non-zero content 518 in the difference signal |Coll-Col2| succeeds the non-zero content 520 in the difference signal |Col2-Coll|, then there is a dark bar. This is depicted in Fig. 5 A. As said above, a maximum operation should be applied then. Otherwise, if the non-zero content 518 in the difference signal |Coll-Col2| precedes the non-zero content 520 in the difference signal |Col2-Coll| then there is a bright bar. This is depicted in Fig. 5B. As said above, a minimum operation should be applied then.

In the description in connection with Fig. 5A and Fig. 5B it is assumed that the display device 202 and the image capture unit 204 of the image acquisition unit 200 are aligned. If the display device is rotated 90 degrees related to the image capture unit 204 then rows would be used. Similarly a rotation of 180 degrees would need to make the opposite choices to what are being made now.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be constructed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitable programmed computer. In the unit claims enumerating several means, several of these means can be embodied by one and the same item of hardware.

Claims

1 An image acquisition umt (200) for imaging a display device (202) which is arranged to display a sequence of display images, the image acquisition unit (200) comprising image capture means (204) for capturing a first captured image (112), representing a first portion of the sequence of the display images and for capturing a second captured image (114), representing a second portion of the sequence of the display images, and combining means (208) for creatmg an output image by means of combmmg the first captured image (112) and the second captured image (114), characterized in that the combining means (208) comprise processing means being arranged to determme a first value of a first one of the pixels of the output image (120) on basis of a statistical operation on a second value of a corresponding pixel of the first captured image (112) and a third value of a corresponding pixel of the second captured image (114)

2 An image acquisition unit (200) as claimed in claim 1, characterized in that the processmg means are arranged to perform an order statistical operation to determine the first value of the first one of the pixels of the output image (1 0)

3 An image acquisition unit (200) as claimed in claim 1, characterized in that the processmg means are arranged to determme the first value of the first one of the pixels of the output image (120) by taking the minimum of the second value of the first captured image (112) and the third value of the second captured image (114)

4 An image acquisition unit (200) as claimed in claim 1, characteπzed in that the processmg means are arranged to determine the first value of the first one of the pixels of the output image (120) by taking the maximum of the second value of the first captured image (112) and the third value of the second captured image (114)

5 An image acquisition unit (200) as claimed in claim 1, characteπzed m that the image acquisition unit (200) comprises control means to control the processing means being arranged to determine the first value on basis of an integration period of the image capture means and a display refresh rate of the display device (202)

6 An image acquisition u t (200) as claimed in claim 1, characteπzed m that the image capture means (204) are arranged to capture a third captured image

(116), representing a third portion of the sequence of the display images, and the combining means (208) comprise processmg means being arranged to determme the first value of the first one of the pixels of the output image (120) on basis of a statistical operation on the second value of the correspondmg pixel of the first captured image (112) and the third value of the correspondmg pixel of the second captured image (114) and a fourth value of a correspondmg pixel of the third captured image (116)

7 An image acquisition unit (200) as claimed in claim 5, characteπzed m that the processmg means are arranged to determme the first value of the first one of the pixels of the output image (120) by taking the minimum, maximum or median from a set of values compπsmg the second value of the first captured image (112), the third value of the second captured image (114) and the fourth value of the third captured image (116)

8 A camera (300) comprising an image acquisition unit (200) for imaging a display device (202) which is arranged to display a sequence of display images, image acquisition umt (200) comprising

- image capture means (204) for capturmg a first captured image (112), representing a first portion of the sequence of the display images and for captunng a second captured image (114), representing a second portion of the sequence of the display images, and

- combining means (208) for creating an output image (120) by means of combmmg the first captured image (112) and the second captured image (114), and - a memory device (304) for temporarily stonng the output image (120), characteπzed in that the combining means compπse (208) processing means be g aπanged to determme a first value of a first one of the pixels of the output image (120) on basis of a statistical operation on a second value of a corresponding pixel of the first captured image (112) and third value of a correspondmg pixel of the second captured image (114)

9. An inspection system (400) for inspecting a display device (202) which is arranged to display a sequence of display images, the inspection system (400) comprising: an image acquisition unit (200) for imaging the display device (202), comprising:

- image capture means (204) for capturing a first captured image (112), representing a first portion of the sequence of the display images and for capturing a second captured image (114), representing a second portion of the sequence of the display images; and - combining means (208) for creating an output image (120) by means of combining the first captured image (112) and the second captured image (114); analyzing means (402) for analyzing the output image (120), characterized in that the combining means (208) comprise processing means being aπanged to determine a first value of a first one of the pixels of the output image (120) on basis of a statistical operation on a second value of a coπesponding pixel of the first captured image (112) and third value of a coπesponding pixel of the second captured image (114).

10. A method of imaging a display device (202) which is aπanged to display a sequence of display images, the method comprising: - capturing a first captured image (112), representing a first portion of the sequence of the display images and for capturing a second captured image (114), representing a second portion of the sequence of the display images; and creating an output image (120) by means of combining the first captured image (112) and the second captured image (114), characterized in that a first value of a first one of the pixels of the output image (120) is determined on basis of a statistical operation on a second value of a coπesponding pixel of the first captured image (112) and a third value of a coπesponding pixel of the second captured image (114).