GB2588673A

GB2588673A - Method and apparatus for verifying a display

Info

Publication number: GB2588673A
Application number: GB1915928.4A
Authority: GB
Inventors: Morse Douglas; Hall Trevor; Alexander Jeacocke Jonathan
Original assignee: DisplayLink UK Ltd
Current assignee: DisplayLink UK Ltd
Priority date: 2019-11-01
Filing date: 2019-11-01
Publication date: 2021-05-05
Anticipated expiration: 2039-11-01
Also published as: GB201915928D0; GB2588673B

Abstract

A method of monitoring a display device for identifying whether a display output of the display device matches an expected output. The method comprises generating a spatial and/or temporal display pattern for output by at least one pixel of the display device; outputting light by the at least one pixel of the display device according to the display pattern; detecting light output from the at least one pixel of the display device; comparing the detected light output from the at least one pixel with the light expected to be output from the at least one pixel when the display pattern is displayed by the at least one pixel; and determining, based on the comparison, whether the light output from the at least one pixel matches, within a predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel. If the light output from the pixel(s) match(es), within the threshold, expected light output when the display pattern is displayed by the pixel(s), it is verified that at least a portion of an image displayed on the device is correct. Non-matching pixel light outputs may provide an indication of incorrect displayed image portions.

Description

Method and Apparatus for Verifying a Display

Background

As display technology improves, users are coming to rely more widely on display systems in varied contexts, including commercial advertising, interactive, and other medical, automotive, and military applications. In turn, it is becoming more important that display devices are updated rapidly and reliably.

It is desirable, and in some cases mandatory, that the images transmitted are actually displayed and visible to the target user. To achieve this currently, some systems include a requirement for a check on a frame buffer each time it is updated and may also require a check that the display panel is powered, but there is no way to check that the contents of the frame buffer are correctly shown on a connected display panel without a user manually checking that the images displayed are correct. It will be appreciated that for systems comprising many displays in different locations (for example, digital timetable displays at bus stop distributed around a city), regular manual checks of all of the displays may be prohibitively time consuming.

The present invention seeks to solve and/or at least partially ameliorate these problems by providing an appropriate check of a display.

Summary

Aspects of the invention are set out in the independent claims and preferred features are set out in the dependent claims.

in one aspect, the invention provides a method of monitoring a display; device for identifying whether a display output of the display device matches an expected display output of the display device, the method comprising: generating a spatial and/or temporal display pattern for output by at least one pixel of the display device; outputting light by the at least one pixel of the display device according to the generated display pattern; detecting light output from the at least one pixel of the display device; comparing the detected light output from the at least one pixel with the light expected to be output from the at least one pixel when the display pattern is displayed by the at least one pixel; determining, based on the comparison, whether the light output from the at least one pixel matches, within a predetemined threshold, the expected light output when the display pattern is displayed by the at least one pixel; and if the light output from the at least one pixel matches, within the predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel, verifying that at least a portion of an image displayed by the display device is correct.

In one embodiment, if the light output from the at least one pixel does not match, within the predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel, the method further comprises providing an indication that at least a portion of the image displayed by the display device is incorrect.

Preferably, providing an indication that at least a portion of the image displayed by the display device is incorrect comprises identifying at least one pixel or group of pixels of the display device that is malfunctioning. The pattern may be a temporal pattern corresponding to a change in brightness of the at least one pixel, and, in some cases, the pattern may be a repeating pattern in time.

in an embodiment, the pattern may be generated for display outside of an area of the display panel that is visible during normal use.

In some cases, the pattern may be a spatial pattern for output by at least one row of pixels of the display device.

Preferably, an image frame may be generated for display at the display device, and the pattern is included in the image frame. The pattern may be added to the image frame after the image frame has been generated.

In some embodiments, the light output from the at least one pixel of the display device is detected using a photosensitive sensor, which may be arranged at a substantially oblique angle to a display surface of the display device, or at a substantially perpendicular angle to a display surface of the display device. The photosensitive sensor may be coupled to the display by a light guide.

In some examples, at least some of the light output from the at least one pixel of the display device is reflected by at least one mirror, and the photosensitive sensor is arranged to detect the reflected light. The at least one pixel behind the mirror is preferably controlled to have an increased brightness relative to other pixels of the display device.

in an embodiment, the method further comprises detecting, using a second photosensitive sensor, ambient light in the vicinity of the display device, wherein the detected light output by the at least one pixel is preferably adjusted based on the detected ambient light.

In some cases, the detected light output by the at least one pixel may be used to determine a particular display pattern that was generated, wherein the determined particular display pattern may be used to identify a particular frame of an image being displayed by the display device.

In a second aspect, the invention provides an apparatus comprising: a pattern generation unit configured to generate a spatial and/or temporal display pattern for display by one or more pixels of a display device: and a determination module configured to receive an indication of light output by the one or more pixels of the display device, the determination module further being configured to compare die light output by the one or more pixels of the display device with light expected to be output from the one or more pixels when the display pattern is displayed by the one or more pixels, and, based on the comparison, to determine whether the light output from the one or more pixels matches, within a predetermined threshold, the expected light output when the display pattern is displayed by the one or more pixels, and, if the light output from the one or more pixels matches, within the predetermined threshold, the expected light output when the display pattern is displayed by the one or more pixels, determining that at least a portion of an image displayed by the display device is correct.

The apparatus is preferably configured to perform the method described above The invention may also provide, according to a further aspect, a system comprising die apparatus described above: a display device comprising a plurality of pixels, the display device being coupled to the pattern generation unit for outputting the display pattern by the one or more pixels of the display device; and a photosensitive sensor for sensing light output by the one or more pixels of the display device displaying the display pattern and providing the indication of the light output by the one or more pixels of the display device to the determination.

The system is preferably configured to perform the method described above.

In an embodiment, the display device comprises the photosensitive sensor.

In a further aspect, the invention provides a system comprising: a non-transitory memory storing (instructions or local data): and one or more hardware processors coupled to the non-transitory memory and configured to execute the instructions from the non-transitory memory to cause the system to perform operations comprising: generating a spatial and/or temporal display pattern for output by at least one pixel of the display device; outputting light by the at least one pixel of the display device according to the generated display detecting light output from the at least one pixel of the display device; comparing the detected light output from the at least one pixel with the light expected to be output from the at least one pixel when the display pattern is displayed by the at least one pixel; determining, based on the comparison, whether the light output from the at least one pixel matches, within a predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel; and if the light output from the at least one pixel matches, within the predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel, verifying that at least a portion of an image displayed by the display device is correct.

According to a -thither aspect, there is provided a system for verifying that a display device is operating correctly, the system comprising: 1. A display device which displays images; 2. A photosensitive sensor which senses light emitted by the display device and 3. A source computing device which: IS a, generates display data; b. incorporates patterns that can be detected by the sensor into the display data; c. receives signals related to the light sensed by the sensor; d. validates correct responses or generates fault information as appropriate; and e. handles fault information The system can therefore operate using a method comprising: 1 The source computing device generates a frame of display data incorporating a distinguishable pattern; 2 The source computing device transmitting the frame to the display device; 3 The display device displaying the frame; 4 The sensor detecting the pattern; S. The sensor transmitting signals related to the detected pattern to a fault detection application; 6 A fault detection application validating the received signals and generating validation information; and 7 The fault detection application handling the validation and/or fault information as appropriate.

The source computing device may be a conventional computing device such as a laptop, mobile device, game console, etc. Alternatively, it may be a camera arranged to transmit a video feed to a display device or a device acting as an intermediary between a computing device or camera and the The pattern may be indistinguishable to the human eye compared to the main display data, or it may be a visible pattern such as a group of pixels of contrasting colours. It may be displayed outside the ordinarily-visible part of a display panel. It may also be a subtle pattern that is visible but is unlikely to be noticed by a user such as localised changes in brightness. Furthermore, it may be a pattern that is not in the visible spectrum of light, such as a collection of pixels that emit light in the infra-red portion of the electromagnetic spectrum. Known stcganographic techniques could also be used; for example, if 8 bits are used to represent each of the three colour values (red, green, and blue, or RGB) of a displayed pixel. A change in one bit could produce a change in the colour of the pixel which is highly unlikely to be detected by a human eye but which could be detected by a photosensitive sensor; this is currently used for transmitting clandestine information and watermarks in image files, but it is not used to determine the display of an image on a display panel.

The pattern may be a collection of pixels in any location in the displayed image, including around the edge, and may comprise the entire image.

The fault detection application may be a separate device, or may be part of the source computing device.

Handling validation or fault information may comprise: * Recording the result in a log file, which can then be used for audit purposes; * Sending an error message to an automated system; * Displaying or otherwise providing an error message to a user; * Sending signals to peripherals to change their behaviour; * Re-transmitting the frame that was not appropriately displayed; or any other appropriate behaviour.

Any system feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure.

Any feature in one aspect of the invention may be applied to other aspects of the invention, in any appropriate combination. Furthermore, any, some and/or all features in one aspect can be applied to any, some and/or all features in any other aspect, in any appropriate combination.

It should also be appreciated that particular combinations of the various features described and defined in any aspects of the invention can be implemented and/or supplied and/or used independently.

Brief Description of the Drawings

Embodiments will now be described, by way of example only and with reference to the accompanying drawings, in which: Figure la shows an overview of a system comprising a source computing device, a display device and a sensor; Figure lb shows an overview of a thither system comprising a source computing device, a display device and a sensor; Figure 2 shows an exemplary sensor configuration; Figure 3 shows a second exemplary sensor configuration; Figure 4 shows a third exemplary sensor configuration; Figure 5 shows an alternative configuration comprising a second sensor; Figure 6a shows an exemplary frame sequence; Figure 6b, shows a further exemplary frame sequence; Figure 7 shows an exemplary process of displaying an image and generating fault data; arid Fig. 8 is a block diagram of a computer system suitable for implementing one or more embodiments of the present disclosure.

Detailed Description of the Drawings

Figure la shows an overview of a system comprising a source computing device, a display device and a sensor. The photosensitive sensor [11] is connected to the source computing device [12], which is in turn connected to the display device [13]. The source computing device [12] includes processors which run applications including a fault-detection application [14] which receives and analyses input from the sensor [II] and a display application [15] which generates display data, which is then transmitted to the display device [13]. It also contains a frame buffer [16] which stores display data before it is sent to the display panel [13]. In this Figure, signalling connections arc shown with dashed lines and data connections are shown with solid lines.

In one example, there may also be a display control device which is connected to the source computing device [12] and the display panel [13] and which contains a frame buffer. It will be 25 appreciated that such a display control device operates in the same way as the frame buffer [16] on the source computing device 1121.

The photosensitive sensor [11] is used to monitor the images displayed on the display device [13] to observe changes in the images. It could be a photodiode or phototransistor -or set of photodiodcs or phototransistors -with appropriate additional circuitry or could be any similar sensor.

The fault detection application [14] is also connected to the display application [15]. It can therefore analyse the information received from the sensor [II], for example by sampling the voltages of a photodiode, and send information on faults in the display process to the display application [15].

Figure lb shows an overview of a further system comprising a sensor [II], a source computing device 1124 and a display device [13], similar to the system shown in Figure la. In this system, however, the fault detection application [14] is a separate device. The fault detection application [14] may be part of another computing device or it may be an independent device in its own right. The sensor [11] is connected to the fault detection application [14] and is not connected to the source computing device [12]. Otherwise, the system operates the same way.

Figure 2 shows an exemplary configuration of the sensor [11]. In this embodiment, the sensor [11] is positioned to "look" along the length of the display panel [13]. It is therefore able to detect changes in the Fight emitted by the display panel [13] even though the light is not shining directly onto the sensor [11]. The display panel [13] may be hilly or partially formed of a transparent material to enable at least some of the emitted light to reach the sensor [11] by means of normal light scattering. Alternatively, or additionally, specific partially reflective structures may be configured within the material forming the display panel [13] to cause at least some of the light to be directed to the sensor [11].

Figure 3 shows an exemplary configuration of a sensor [II] comprising a light waveguide, which may be an optical fibre, for example a fibreoptic cable [31], which is in turn connected to the display panel [13] such that light emitted by part of the display is transmitted directly to the sensor [11]. This could also be a light pipe that follows one or more edges of the display panel [13] to divert light from pixels on the edges of the display panel [13] to the sensor [111, or a conventional transparent plastic block, as long as it provides a conduit between the sensor [11] and the display panel [131 in the system of Figure 3, unlike in the embodiment shown in Figure 2, the light emitted by the display panel [13] is received directly by the sensor [11], which beneficially increases the accuracy of fault detection. Accordingly, in a similar embodiment the sensor [11] could be in contact with the surface of the display panel [13].

Figure 4 shows a schematic diagram of a system in which light is harvested and diverted to the sensor [Il] by a half-silvered mirror [41] which may cover all or part of the display so as to allow some light from the display panel [13] to pass through but reflects some of the light from the display panel [13] to the sensor [11]. This has the benefit that while part or all of the displayed image is sent to the sensor [Il] for analysis, there is no disruption to a user's view of the image on the display panel such as might be caused by a sensor [11] or light pipe [31] as described in Figures 2 and 3.1n a particularly beneficial example, the source computing device [12] may increase the brightness of the affected region of pixels so the half-silvered mirror does not affect the image in that region of the display as viewed by the user.

Figure 5 shows a modification to the system that could be used in any of the systems of Figure 2, Figure 3, or Figure 4. The sensor [11] collects light from the display device [13] as previously described and transmits signals based on that light to the fault detection application [14], but there is also a second

S

photoreceptive sensor [511 connected to the fault detection application [14] which is used to detect ambient light and transmit signals based on such ambient light to the fault detection application [14]. This additional information can then be used by the fault detection application [14] to cancel out any influence from ambient light on the light detected by the sensor [II].

This modification could be especially useful in the embodiments shown in Figure 2 and Figure 4, since they are most likely to be affected by ambient light because they are not directly receiving the light transmitted by the display panel [13]. However, it could also be used in the embodiment shown in Figure 3.

If detection is fast enough, it is possible to 'see' each row of pixels as it is displayed on the display panel [13]: a process known as rastering (named after the past traditional process of activating pixels in a cathode ray screen using an electron beam). This means that any of the systems of Figures la to 5 could be used to detect a pattern of pixels in the field of view of the sensor [11], which can be added by sacrificing a few rows/columns of pixels at the edge of the display panel [13]. Any of these embodiments can be used to produce signalling that provides "proof of life": i.e. a confirmation that the image displayed on the display panel [13] has been updated after a new frame was generated by the display application [15]. For example, the sensor could detect the light emitted from a known pixel or group of pixels which can be amended from frame to frame. It will be appreciated that alternatively, instead of using a group of pixels, the pattern could be generated using a single pixel that, for example, changes through gradients of colours from frame to frame, or flickers between two known colours, for example black and white. Alternatively, a group of pixels can form a special pattern similar to a bar code or QR code.

Two examples of such a group of pixels changing from frame to frame are shown in Figures 6a and 6b, respectively.

Figure 6a shows a sequence of four frames [61] in, for example, a video stream produced by the display application [15] on the source computing device [12]. These frames [61] show a specially-generated pattern of pixels [62] at the bottom of each frame. It will be appreciated that the height of the row of pixels may be a single pixel or may be multiple pixels and accordingly tiles -i.e. geometrically-shaped groups of pixels -may be used instead of individual pixels. It will also be appreciated that whilst the patterns are described in terms of black and white pixels, any suitable colours of pixels may be used. For example, the pattern may comprise blue and red pixels. This pattern changes from frame to frame such that each frame [61] in the sequence has a unique pattern [62], or at least a pattern [62] that is unlikely to be repeated for a large number of frames [61].

In this example,

* Frame 1 [61A] shows an evenly-spaced pattern of black and white pixels [62A]; * Frame 2 [6 IB] shows a pattern of black and white pixels comprising a short row of black pixels, a stretch of evenly-spaced black and white pixels, a row of white pixels, a further pattern of black and white pixels, and a further row of black pixels [62B]; * Frame 3 [61C] shows a row of pixels comprising a single black pixel, a short row of white pixels, a stretch of evenly-spaced black and white pixels, a row of black pixels, a further stretch of black and white pixels, a further stretch of white pixels, and a single black pixel [62C]; and * Frame 4 [6113] shows an evenly-spaced pattern of rows of black and white pixels [6213].

Figure 6b shows the same sequence of frames as were shown in Figure 6a, but the pattern [62] is located on the right-hand side of the frame [61] rather than the bottom of the frame [61]. It will be appreciated that the pattern may also be located along the top edge of the frame, along the left-hand edge of the frame, or at any suitable position in the image frame. It will also be appreciated that the pattern may be distributed over several different areas of the image frame. For example, a part of the pattern may be located in the upper left corner of the frame, and the remaining part of the pattern may be located in the lower right corner of the frame. in this example, the pattern [62] changes in a predictable way, comprising a repeating pattern of: * Row 1: two white pixels; * Row 2: a white pixel and a black pixel; * Row 3: a black pixel and a white pixel; * Row 4: two black pixels; which rotates from frame to frame [61] in a predictable manner, such that the pattern [62A] used for row 2 in Frame 1 [61A] is used for row 1 in Frame 2 [61B]. This pattern [62] can be used as previously described with regard to the pattern [62] in Figure 6a.

In addition, this pattern [62] takes advantage of the conventional method of displaying images on a display panel [13] by updating pixels row by row, as previously mentioned. Because each row ends in part of the pattern [62], an appropriately-configured sensor [11] could detect changes in the pattern [62] shown in different rows so that rows can be specifically identified, allowing the fault detection application [14] to determine particular rows that are not updating properly.

These patterns [62] can be added by the display application [15] as part of the generation of the frame [61], or they can be added once the frame has been generated, potentially as part of a compression process and in some cases by the fault-detection application [14]. If a fault-detection application [14] which is part of the source computing device [12] adds the pattern [62], it may beneficially be better able to detect faults. Alternatively, the display application [15] may pass the pattern [62] used to the fault-detection application [14], or there might be a known predetermined sequence of patterns [62] used.

An example of such a known predetermined sequence of patterns [62] could be based on a numbering system such that each frame 1611 is given a number -for example the first frame [61] in a video feed being numbered 1, the second 2, etc. -and the pattern [62] is based on that numbering system, for example the pattern 1621 being a binary representation of the frame number modulo the maximum number that can be represented by the number of pixels or pixel tiles in the pattern [62] plus 1. For example, in a system such as that shown in Figure Ga where the pattern [62] is a stripe of 19 black and white pixels along the bottom of the frame [61], the pattern [62] could be: * Frame 1: eighteen white pixels and one black pixel (1 in binary notation) * Frame 2: seventeen white pixels, one black pixel, and one white pixel (2 in binary notation) * Frame 524,287: nineteen black pixels (524,287 in binary notation, the maximum number that can be represented by nineteen bits) * Frame 524,288: nineteen white pixels (524,288 modulo 524,287+1 = 0) Similarly, if the pattern [62] were, for example, a square of four pixels or tiles, this could represent any number up to 15 and therefore could return to being an all-white square every 16 frames, and so on for other arrangements. Naturally, black or white could be used to represent 1 or 0 in such a system, or any other appropriate colours could be used as previously mentioned.

The pattern [62] is detected by the sensor [1 It and since each frame has a unique pattern the fault-detection application [14] is able to determine whether the image being displayed has changed at the correct time and, furthermore, whether the image being displayed corresponds to the frame [61] that should be displayed.

Other methods of identifying frames [61] can also be used. For example, in a context such as a digital advertising hoarding, there may be a limited number of frames [61] and therefore the entire frame [61] or a specific, known area could be used as a unique pattern [62]. Under these circumstances, the embodiment shown in Figure 3 could be suitable. Furthermore, instead of sacrificing part of the frame [61] to incorporate an area of specially-generated pixels [62], a changing patch of brightness could be used, such that the colour does not change but the brightness of the backlight of the display panel [13] changes from frame to frame.

If notifications of display faults are not time-sensitive, patterns could be added at periodic intervals rather than to every frame [61].

Naturally, these are only examples of patterns other patterns could be used and operate in a similar way.

A pattern [62] could also be added to the beginning and/or end of a video feed. For example, a pattern [62] consisting of known numbers of solid-coloured complete dummy frames or dummy frames with a known pattern in the appropriate location for the pattern [62] that will be used for the rest of the application [14], which would allow it to be only active when frames [61] which are arranged for use in this system are to be shown. This could reduce power and processing resources used in either the host computing device [12] or -in a system such as that in Figure lb -a separate processor running the fault-detection application [14]. Accordingly, a similar epilogue pattern could also be added to carry out configuration, deactivate the fault-detection application [14] etc Figure 7 shows an exemplary process of displaying an image and generating fault data At Step S71, the display application [151 generates a frame [61] of display data, either in the conventional way or by including a specially-generated pattern [62] during the generation of the frame [61], as previously described. If the pattern [62] is not added during generation of the frame [61], it can be added in the display application [15] when the frame [61] is completed but before it is sent to the frame buffer [16] at step S71 A. Since step S71A will only occur if the pattern is not included when the frame is generated, it is outlined in the Figure with dashed lines. The display application [15] can then pass the pattern [62] which was added to the fault-detection application [14] for use in analysing signals received from the sensor [II]. This is especially straightforward in a system such as that shonm in Figure la where the fault-detection application [14] and the display application [15] are within the same device. Alternatively, if the display application [15] uses a known sequence of patterns [62], the fault-detection application [14] could use the same sequence and the pattern [62] in use could be synchronised by, for example, a common dock. This could be done in a system such as that shown in Figure Ib, where the fault-detection [14] and the display application [15] are in separate devices.

At Step 572, the frame [61] is stored in the frame buffer [16] on the source computing device [12].

At this stage it may be compressed for transmission to the display panel [13] by a further processor running a compression algorithm. The pattern [62] could also be added at this compression stage, for example by adding a watermark. If the pattern [62] is added at the compression stage, it could be notified to the fault-detection application [14] in the same way as if it were added by the display application [15].

The data is then transmitted to the display panel [13] at Step S73. If it was compressed before transmission, it can be decompressed on receipt and may be stored in a further frame buffer either at the display panel [13] or in a connected display control device before being rastered to the display panel [13].

At Step S74, the image [61, 62] is rastered to the display panel [13] for display in the conventional way, including the unique pattern [62]. If the display data is not displayed correctly at the display panel -for example, if the display panel [13] is faulty or data cannot be rastered as quickly as expected -then the pattern [62] will not appear, or will appear differently to expected, and this can be detected by the sensor [11].

At Step S75. the sensor [II] detects the light emitted by the display panel [13]. it could then determine the pattern [62] it -secs", and at Step 576 it transmits that information to the fault detection application [14], or it could transmit information simply indicating the detected light to the fault detection application [14].

At Step S77, the fault-detection application [14] determines whether the frame [61] generated at Step S71 has been displayed correctly. if the fault-detection application [14] has received a signal from the sensor [11] that indicates the pattern [62] that was displayed on the display panel [13], this might mean simply comparing the received pattern [62] to the known pattern [62] which was added to the frame [61] at Step S71 or S71A: as previously mentioned, this pattern [62] could have been received from the display application [15] or any other application that added such a pattern [62] to the frame [61]. If the fault-detection application [14] received information indicating the light detected, such as a raw signal, it could then perform processing internally to determine the pattern [62] detected.

In an embodiment such as that shown in Figure 5, the fault detection application [14] could also receive information on the ambient light from the second sensor [51] and this detected light could be subtracted from the detected light from the display panel [13] to improve the accuracy of the pattern detection.

If the fault-detection application [14] determines that the pattern [62] displayed matches the pattern [62] expected, this means that the frame [61] has been transmitted and displayed correctly and therefore that the system is operating correctly and no action need be taken. It may store a record to this effect in a log, or it may do nothing.

If the fault-detection application [14] determines that the pattern [62] displayed does not match the pattern [62] expected, this means that the frame [61] has not been displayed correctly. indications from other error detection mechanisms in the display pipeline may alert the source computing device [12] if there have been errors or delays elsewhere, for example in the transmission of the frame [61] to the display panel [13], but in order to determine whether a frame [61] has been correctly rastered and displayed, under current art a user must observe the display panel [13] and manually check for errors. Beneficially, the presently described system has the advantage that errors such as skipped or stuck frames can be detected automatically.

The behaviour of the fault-detection application [14] upon determining that the expected frame [61] has not been displayed or has not been displayed correctly can vary depending on context.

For example, in a digital advertising context where it is simply important to know whether a frame [61] was displayed or not, and time at which the frame is displayed is of little or no importance, the fault-detection application [14] could signal the display application [15] to re-transmit the frame [61], possibly with a new specially-generated pattern [62] if appropriate. The fault-detection application [14] could also have a mechanism for transmitting an error report, for example, to a separate monitoring system or user if multiple frames [61] are missed, or if the re-transmitted frame [61] is also not displayed correctly. The fault-detection application [14] could also, or instead, log a record of the error in a local or remote area of memory.

In a second example, in a user interface context where the displayed image is updating relatively rapidly but the exact timing is not critical, the fault-detection application [14] could cause an error message to be sent to the user, possibly through an alternative medium such as speakers or on a different display panel, to notify them that there is potentially a problem with the display panel [13].

In a third example, in an automotive context where the frames [61] being sent from the display application [15] are, for example, a video feed showing a view to the rear of a reversing vehicle, the displayed image is updating rapidly and it is critically important that the display is updated correctly and when expected. in this example, the fault-detection application [14] could, for example, halt the vehicle and send an error message to the user to inform them that it may not be safe to rely on the video feed.

In a fourth example, in a medical context where a camera feed is used during, for example, endoscopic surgery, the fault-detection application [14] could withdraw some equipment from contact with the patient while leaving other equipment in place and send error messages to multiple users, such as the surgeon, nurses, and a technician, who could then perform immediate maintenance operations such as rebooting the display system.

Naturally, in different contexts the fault-detection application [14] could behave in different ways depending on the required implementation.

In a fifth example, in a manufacturing context where the display system is being tested, the fault detection application [14] could retain a log of frames [61] that did not appear or did not appear as or when expected which could then be compared to a log of the operation of the display application [15] and intermediate processes such as compression, encryption, transmission etc. The performance of specific frames [61] could then subsequently be traced through the system to determine the cause of missed, late, or incorrect frames [61].

Fig. 8 is a block diagram of a computer system 600 suitable for implementing one or more embodiments of the present disclosure, including the source computing device [12], in various implementations, the source computing device [12] may include a mobile cellular phone, personal computer (PC), laptop, wearable computing device, etc. adapted for wireless communication, and each of source computing device [12] or the sensor [11] may include a network computing device. Thus, it should be appreciated that these devices may be implemented as the computer system 600 in a maimer as follows.

The computer system 600 includes a bus 612 or other communication mechanism for AM-, nr. A inc,r.",÷;,.., 1,n1-vw,nat. -wrnr^r*Iln e-S+11.2. ne.rnrn-rEar system 600. The components include an input/output (170) component 604 that processes a user action, such as selecting keys from a keypad/keyboard, selecting one or more buttons or links, etc., and sends a corresponding signal to the bus 612. The I/O component 604 may also include an output component, such as a display 602 and a cursor control 608 (such as a keyboard, keypad, mouse, etc.). An optional audio input/output component 606 may also be included to allow a user to use voice for inputting information by converting audio signals. The audio I/0 component 606 may allow the user to hear audio. A transceiver or network interface 620 transmits and receives signals between the computer system 600 and other devices via network 622. In one embodiment, the mummission is wireless, although other transmission mediums and methods may also be suitable. A processor 614, which can be a micro-controller, digital signal processor (DSP), or other processing component, processes these various signals, such as for display on the computer system 600 or transmission to other devices via a communication link 624. The processor 614 may also control transmission of information, such as cookies or IP addresses, to other devices.

The components of the computer system 600 also include a system memory component 610 (e.g., RAM), a static storage component 616 (e.g., ROM), and/or a disk drive 618 (e.g., a solid-state drive, a hard drive). The computer system 600 performs specific operations by the processor 614 and other components by executing one or more sequences of instructions contained in the system memory component 610. For example, the processor 614 could be utilised to perform the fault detection function of the source computing device [12].

Executable logic for performing any described functions may be encoded in a computer readable medium, which may refer to any medium that participates in providing instructions to the processor 614 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media, in various implementations, non-volatile media includes optical or magnetic disks, volatile media includes dynamic memory, such as the system memory component 610, and transmission media includes coaxial cables, copper wire, and fiber optics, including wires that comprise the bus 612. In one embodiment, the logic is encoded in non-transitory computer readable medium, such as a magnetic or optical disk or other magnetic/optical storage medium, or FLASH or other solid-state memory (e.g. integrated into a device or in the form of a memory card). In one example, transmission media may take the form of acoustic or light waves_ such as those generated during radio wave, optical, and infrared data communications.

In various embodiments of the present disclosure, execution of instruction sequences to practice the present disclosure may be performed by the computer system 600. In various other embodiments of the present disclosure, a plurality of computer systems 600 coupled by the communication link 624 to the network (e.g., such as a LAN, WLAN, PTSN, and/or various other wired or wireless networks, including telecommunications, mobile, and cellular phone networks) may perform instruction sequences to practice the present disclosure in coordination with one another.

The above embodiments and examples are to be understood as illustrative examples. Where applicable, various embodiments provided by the present disclosure may be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and/or software components set forth herein may be combined into composite components comprising software, hardware, and/or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and/or software components set forth herein may be separated into sub-components comprising software, hardware, or both without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components and vice-versa.

Software in accordance with the present disclosure, such as program code and/or data, may be stored on one or more computer readable mediums. It is also contemplated that software identified herein may be implemented using one or more general purpose or specific purpose computers and/or computer systems, networked and/or otherwise. Where applicable, the ordering of various steps described herein may be changed, combined into composite steps, and/or separated into sub-steps to provide features described herein.

The various features and steps described herein may be implemented as systems comprising one or more memories storing various information described herein and one or more processors coupled to the one or more memories and a network, wherein the one or more processors are operable to perform steps as described herein, as non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising steps described herein, and methods performed by one or more devices, such as a hardware processor, user device, server, and other devices described herein.

Claims

Claims A method of monitoring a display device for identifying whether a display output of the display device matches an expected display output of the display device, the method comprising: generating a spatial ancUor temporal display pattern for output by at least one pixel of the display device; outputting light by the at least one pixel of the display device according to the generated display pattern; detecting light output from the at least one pixel of the display device; comparing the detected light output from the at least one pixel with the light expected to be output from the at least one pixel when the display pattern is displayed by the at least one pixel; determining, based on the comparison, whether the light output from the at least one IS pixel matches, within a predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel; and if the light output from the at least one pixel matches, within the predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel, verifying that at least a portion of an image displayed by the display device is correct.
2, The method according to claim 1, wherein if the light output from the at least one pixel does not match, within the predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel, the method further comprises providing an indication that at least a portion of the image displayed by the display device is incorrect.
3. The method according to claim I, wherein providing an indication that at least a portion of the image displayed by the display device is incorrect comprises identifying at least one pixel or group of pixels of the display device that is malfunctioning.
The method according to any preceding claim, wherein the pattern is a temporal pattern corresponding to a change in brightness of the at least one pixel.
The method according to claim 4, wherein the pattern is a repeating pattern in time.
6. The method according to any preceding claim, wherein the pattern is generated for display outside of an area of the display panel that is visible during normal use.
The method to any preceding claim, wherein the pattern is a spatial pattern for output by at least one row of pixels of the display device.
The method to any preceding claim, wherein an image frame is generated for display at the display device, and the pattern is included in the image frame.
The method according to claim 8, wherein the pattern is added to the image frame after the image frame has been generated.
10. The method according to any preceding claim, wherein the light output from the at least one pixel of the display device is detected using a photosensitive sensor.
11. The method according to claim 10, wherein the photosensitive sensor is arranged at a substantially oblique angle to a display surface of the display device.
12. The method according to claim 10, wherein the photosensitive sensor is arranged at a substantially perpendicular angle to a display surface of the display device.
13. The method according to any one of claims 10 to 12, wherein the photosensitive sensor is coupled to the display by a light guide.
14. The method according to any one of claims 10 to 12, wherein at least some of the light output from the at least one pixel of the display device is reflected by at least one minor, and the photosensitive sensor is arranged to detect the reflected light.
15. The method according to claim 14, wherein the at least one pixel behind the minor s controlled to have an increased brightness relative to other pixels of the display device.
16. The method according to any one of claims 10 to 15 the method further comprising detecting, using a second photosensitive sensor, ambient light in the vicinity of the display device.
17. The method according to claim 16, wherein the detected light output by the at least one pixel is adjusted based on the detected ambient light.
18. The method according to any preceding claim, wherein the detected light output by the at least one pixel is used to determine a particular display pattern that was generated.
19. The method according to claim 18, wherein the determined particular display pattern is used to identify a particular frame of an image being displayed by the display device.
20. Apparatus comprising: a pattern generation unit configured to generate a spatial and/or temporal display pattern for display by one or more pixels of a display device; and a determination module configured to receive an indication of light output by the one or more pixels of the display device, the determination module further being configured to compare the light output by the one or more pixels of the display device with light expected to be output from the one or more pixels when the display pattern is displayed by the one or more pixels, and, based on the comparison, to determine whether the light output from the one or more pixels matches, within a predetermined threshold, the expected light output when the display pattern is displayed by the one or more pixels, and, if the light output from the one or more pixels matches, within the predetermined threshold, the expected light output when the display pattern is displayed by the one or more pixels, determining that at least a portion of an image displayed by die display device is correct.
21. Apparatus according to claim 20, configured to perform all steps of the method according to any one of claims 2 to 9.
22. A system comprising: the apparatus of either claim 20 or claim 21; a display device comprising a plurality of pixels, the display device being coupled to the pattern generation unit for outputting the display pattern by the one or more pixels of the display device; and a photosensitive sensor for sensing light output by the one or more pixels of the display device displaying die display pattern and providing the indication of the light output by the one or more pixels of the display device to the determination.
23. The system according to claim 22, configured to perform all steps of the method according to any one of claims 10 to 19.
24. The system according to either claim 22 or claim 23 wherein the display device comprises the photosensitive sensor.
25. A system comprising: a non-transitory memory storing (instructions or local data); and one or more hardware processors coupled to the non-transitory memory and configured to execute die instructions from the non-transitory memory to cause die system to perform operations comprising: generating a spatial and/or temporal display pattern for output by at least one pixel of the display device; outputting light by the at least one pixel of the display device according to the generated display pattern; detecting light output from the at least one pixel of the display device; comparing the detected light output from the at least one pixel with the light expected to be output from the at least one pixel when the display pattern is displayed by the at least one pixel, determining, based on the comparison, whether the light output from die at least D one pixel matches, within a predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel; and if the light output from the at least one pixel matches, within the predetermined threshold, the expected light output when the display pattern is displayed by the at least one pixel, verifying that at least a portion of an image displayed by the display device is correct.