WO2010144096A1

WO2010144096A1 - Data processing system and method

Info

Publication number: WO2010144096A1
Application number: PCT/US2009/047220
Authority: WO
Inventors: Bernard J. Thompson
Original assignee: Displaylink Corporation
Priority date: 2009-06-12
Filing date: 2009-06-12
Publication date: 2010-12-16

Abstract

A data processing system comprises a processing device (10), a display device (12) and a user interface device (14). A method of operating the system comprises the steps of running one or more applications on the processing device, displaying an image on the display device, detecting a user's interest with respect to a region of the image, assigning a higher priority to the detected region, and assigning a lower priority to an area outside the detected region. In one embodiment, the method can be used to update the image on the display device, the updating comprising updating the higher priority region before the lower priority area.

Description

DATA PROCESSING SYSTEM AND METHOD

DESCRIPTION

This invention relates to a data processing system and a method of operating the data processing system. In one embodiment, the method can be used for attention-based graphics update prioritization.

In desktop computing, it is now common to use more than one display device.

Traditionally, a user would have a computer with a single display device attached, but now it is possible to have more than one display device attached to the computer, which increases the usable area for the worker. For example, Internationa! Patent

Application Publication VVO 2007/020408 discloses a display system which comprises a plurality of display devices, each displaying respectively an image, a data processing device connected to each display device and controlling the image displayed by each display device, and a user interface device connected to the data processing device. Connecting multiple monitors to a computer is a proven method for improving productivity.

Connections to USB displays, network-based displays, or remote clients have extremely limited bandwidth in comparison to the fast buses used by traditional graphics displays. Simultaneous updates that are required for many regions can saturate the bandwidth of these buses, giving the impression that the computer is running slowly, or as with video, giving unacceptable performance.

It is therefore an object of the invention to improve upon the known art. According to a first aspect of the present invention, there is provided a method of operating a data processing system, the system comprising a processing device, a display device and a user interface device, the method comprising the steps of running one or more applications on the processing device, displaying an image on the display device, detecting a user's interest with respect to a region of the image, assigning a higher priority to the detected region, and assigning a lower priority to an area outside the detected region. According to a second aspect of the present invention, there is provided a data processing system comprising a processing device, a display device and a user interface device, wherein the processing device is arranged to run one or more applications, the display device is arranged to display an image, and the processing device is further arranged to detect a user's interest with respect to a region of the image, assign a higher priority to the detected region, and assign a lower priority to an area outside the detected region.

According to a third aspect of the present invention, there is provided a computer program product on a computer readable medium for operating a data processing system, the system comprising a processing device, a display device and a user interface device, the product comprising instructions for running one or more applications on the processing device, displaying an image on the display device, detecting a user's interest with respect to a region of the image, assigning a higher priority to the detected region, and assigning a lower priority to an area outside the detected region.

Owing to the invention, it is possible to provide attention-based prioritization, for example as a means to prioritize display updates based on where the user's attention may be focused. Users like computers to feel responsive, and their impression of responsiveness is often based on how quickly the display reacts as they work. It is therefore advantageous for the system to give higher priority to updating the region of the display where the user's attention is focused, thus updating it more quickly. Updates to other parts of the display can take place, unnoticed, as the user's attention is elsewhere. This prioritization is even more applicable to computers with multiple displays, as users are often focused on only one display at a time. If software and hardware can prioritize those updates, it is possible to better manage bandwidth, and thus give the user better responsiveness. The system provides a method of prioritizing updates.

There are a number of means to determine where the user's attention lies. The position of the mouse or text cursors could be monitored, for example. The system would prioritize updates of the display, or even the sub-region of the display where the cursor is currently located. This method extends to any technique for detecting human attention, such as an eye or face tracking system. Many webcams today come with software for face tracking. This method is equally applicable to multi- touch or multi-cursor systems as well. Display updates are usually performed on a first come, first served basis, or by prioritizing video (a high demand application) first. The novelty of the method described above is to prioritize updates based on the user's attention, using whatever means is available to determine the user's attention.

Performance and responsiveness are the key buying criteria for displays, and essential to convincing the market to adopting multiple display systems. It is an absolute requirement as well for thin clients, both in terms of end user performance, but scaling to massive numbers of thin clients. User productivity and patience, thus satisfaction, all come down to perceived performance. By prioritizing updates on attention, there is an inherently priority for perceived performance. This method can also be used in other ways to tune a user interface based on attention. For example by de-prioritizing updates outside of the attention region. This might be by reducing the frame-rate of a video from 60 frames per second to 10 if it is on a different display than the one the user is working in. The system could also be used for muting or reducing the volume of alerts or other sonic disturbances made by applications outside of the user's attention. The priority system could also be used for dimming battery-powered displays outside the user's attention. Another application is adjust the operating system priority of applications based on the user's attention.

The prioritization may not be based directly on indicators of user attention, but on secondary indicators of activity. For example, the system may monitor the frequency of updates on each display and give extra priority, beyond the natural priority of having so many updates, to displays with frequent updates. The assumption would be that busy screens are the ones users use most. This example is just one of any number of other prioritization schemes that can be used to prioritize based on implied user activity. Preferably, the system further comprises a second display device, and the step of displaying an image on the display device comprises displaying the image on both display devices. In this case, the step of detecting a user's interest with respect to a region of the image advantageously comprises detecting the user's interest with respect to the first or second display device. The method of detecting the user's interest works particularly well when the user has a data processing system with multiple display devices. In this case, the system will prioritise the display device on which the user is currently working, which provides a better experience of the overall system for the user, as, for example, display updates will occur first on the display device on which the user is currently working.

Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:-

Figure 1 is a schematic diagram of a data processing system, _;

Figure 2 is a schematic diagram, similar to Figure 1 , of the data processing system,

Figure 3 is a schematic diagram of a further embodiment of the data processing system,

Figures 4 and 5 are schematic diagrams of yet further embodiments of the data processing system, and Figure 6 is a flowchart of a method of operating the data processing system.

A data processing system is shown in Figure 1. The system comprises a processing device 10, a display device 12 and user interface devices 14. The user interface devices are a keyboard 14a and a mouse 14b. The data processing system shown in Figure 1 is a standard desktop computer composed of discrete components that are locally located but could equally be a single device such as a laptop computer or suitably enabled handheld device such as a mobile phone or pda

(personal digital assistant). Similarly, the data processing system may comprise part of a networked or mainframe computing system, in which case the processing device

10 may be located remotely from the user input devices 14 and the display device 12 or indeed may have its function distributed amongst separate devices. The display device 12 shows an image 16, and the display of that image 16 is controlled by the processing device 10. One or more applications are running on the processing device 10 and these are represented to the user by windows 18, with which the user can interact in a conventional manner. A cursor 20 is shown, and the user can control the movement of the cursor 20 about the image 16 shown on the display device 12 using the computer mouse 14b, again in a totally conventional manner. The user can perform actions with respect to the running application via the user interface device 14 and these actions result in corresponding changes in the image 16, displayed by the display device 12. The data processing system of Figure 1 is configured to detect a user's interest with respect to a region of the image 16, to assign a higher priority to the detected region, and to assign a lower priority to an area outside the detected region. The processing device 10 is prioritizing specific regions on the display device 12 over others on that display device 12. One simple implementation is cursor-based display prioritization. The graphic user input system will process display updates in regions. When an application changes something in its window 18, the operating system run by the processing device 10 does not redraw the entire image 16. Instead, operating system components determine which specific rectangular regions of the image 16 need to be redrawn and then send those regions to a display driver. The region prioritization implementation prioritizes those regions which contain the cursor 20 over those regions that do not.

The principle of the system is that the display region where the user's interest is focussed is assigned a higher priority, for example, for the purpose of assigning a higher priority to the display updating priority with respect to the region where the user's interest is located. Figure 2 shows how an image 16 will have different priorities assigned to different regions and areas of the image 16, following the detection of the user interest in a specific portion of the image 16. In the image 16, the dashed rectangles represent screen update regions. The user has just clicked the I'm Feeling Lucky button, causing the button to be redrawn with a highlight. The use of such a highlight is optional, and can be user configured. The processing device 10 is arranged to detect the position of the cursor 20 within the image 16. Since the cursor 20 is in the region of the button, that region 22 will have a higher priority over the stock ticker and dock, marked as lower priority areas 24. If bandwidth to the display device 12 is at a premium, the region 22 which corresponds to the button will get painted as quickly as possible, and the other areas 24 will get painted whenever bandwidth allows, A single region 22 of high priority has been created within the image 16 and multiple areas 24 of lower priority have also been created, as a result of the user action of locating the cursor 20 and performing an action in relation to that cursor 20. Although the methodology of assigning different priorities to different display areas is applicable to a data processing system that uses a single display device 12, the method can be implemented at many levels. For example, it can be used to prioritize one display device 12 out of many. This implementation will still take advantage of any technique for detecting the user's attention. Multiple display devices 12 are now common in many desktop computing environments, as the addition of the one or more additional display devices 12 increases the productivity of the user who is working with the processing device 10 and multiple display devices 12. Point of sale systems and systems in use in, for example, airports often use multiple display devices 12 connected to a single processing device 10. Figure 3 illustrates a data processing system that is using two display devices

12, both connected to the same processing device 10. Each display device 12 may be controlled by its own graphics driver on a dedicated chip within the processing device 10, or more likely, the main display device 12a is controlled by a single dedicated video chip, via a VGA connection, within the processing device 10, and the secondary display device 12b is connected via a USB connection and controlled by software running on the main chip of the processing device 10. The use of USB as a connection technology for additional displays 12 provides a low cost and simple method by which a user's available display area can be expanded, although with a reduced bandwidth for the graphics data. In the discussion below, "cursor" should be taken as synonymous with "touch" for muiti-touch devices. The system of Figure 3 uses a cursor based display prioritization, which is the simplest implementation. Whichever display device 12 the cursor 20 is on gets highest priority over any other display 12. When a user moves the cursor 20, for example, from the current display device 12 to another display device 12, then the associated high priority, in relation to the display area, moves with the cursor 20. If there are multiple cursors 20, or multiply touched regions, any display device 12 with a cursor 20 or having just received a touch input gets the highest priority. Specific implementations may choose to further prioritize those high- priority displays using other techniques. For example, in the case of the mouse and text cursors 20 being on different display devices 12, if a mouse click was just received, then prioritize the display device 12 with the mouse cursor 20 over that of the text cursor, or the reverse if a key was just pressed.

A further embodiment of the data processing system is shown in Figure 4. In this embodiment, the system further comprises two audio devices 26 which are connected to the display device 12, but are controlled by the processing device 10. Once a region of interest to the user is determined, the processing device 10 is arranged to adjust the audio output of the audio devices 26, the audio output of the applications of the higher priority region being greater than the audio output of the applications of the lower priority area. In this, the system is also being used for muting or reducing the voiurne of alerts or other sonic disturbances made by applications outside of the user's attention.

In this way, distractions to the user are minimised, by reducing or removing the audio output from applications on which the user is not currently focussing. For example, the cursor 20 is currently located at the window 18b. This may be enough to trigger the designation of the image region of the window 18b as having a higher priority, or the processing device 10 may only make this decision once the user has clicked on the relevant window 18b. However, once this region is designated as high priority and the area associated with the window 18a is designated as a low priority, then any audio output that is originating with the application linked to the window 18b will be muted or removed entirely from the audio output of the speakers 26.

The use of the higher priority function in relation to the control of the audio output of the speakers 26 can be used in conjunction or as an alternative to the display updating priority. Other uses of the priority designation of the image region are also possible. For example, it is possible that the processing device 10 will adjust the operating system priority of the applications being run by the processing device 10. In this case, the operating system priority of the applications of the higher priority region will be greater than the operating system priority of the applications of the lower priority area. This will result in more hardware and/or software resources being used for the application associated with the higher priority region when compared to the applications associated with the lower priority area(s).

In the above embodiments, the decision about the user's interest is made based upon the detection of the cursor's position (optionally additionally including a user action such as pressing a mouse button). An alternative method of detecting a user's interest with respect to a region of the image 16 comprises tracking one or more physiological features of a user. This is shown in Figure 5. In this Figure, the user 28 is using a system that includes two display devices 12. The primary display device 12a is additionally equipped with a webcam 30, which is capable of monitoring the eye focus (shown as the arrows 32) of the user 28.

The webcam 30 is connected to the processing device 10 and provides a continuous feedback to the processing device 10 of the monitored physiological feature of the user 28. Software being run by the processing device 10 is able to determine which of the display devices 12 is attracting the user's attention, or indeed on which part of a display device 12 a user is focussing. This determination of the region of interest can made either be as a point in time measurement or as a percentage over a period of time above a certain threshold, in order to be able to compensate for the user's attention occasionally wandering to other objects.

The detection of the user's interest via the monitoring of the eye focus 32 (or indeed other physiological feature such as face direction) then leads to a region of the displayed image 16 being assigned as the high priority region. This region designation could be the entirety of the output of a display device 12. In this case, the image 16 is being displayed across both display devices 12 and the sub-area of this image 16 which is designated as the high priority region is the entire output of one of the display devices 12. The use of the eye or face tracking can be combined with any other method of the determination process such as the cursor monitoring.

The method of operating the data processing system is summarised in the flowchart of Figure 6. The method comprises the steps of, step S1 , running one or more applications on the processing device 10, step S2, displaying the image 16 on the display device(s) 12, step S3, detecting a user's interest with respect to a region of the image 16, step S4, assigning a higher priority to the detected region, step S5, assigning a lower priority to an area outside the detected region, and finally step S6, perform different actions with respect to the region and area of the image 16. The first three steps are taking place simultaneously, and the assigning of the priorities in steps S4 and S5 can be revised at any time according to the output of the detection step S3.

The final step S6 can take many different forms. In general, the nature of the action is such that in relation to the operation of the processing device 10 in more technical areas, the priority of the operation of the processing device 10 follows the priority assigned to the display region and area. Different operations, discussed above, include the updating of the display device(s) 12, the setting of the audio level of the speakers 26, the assignment of the operating system resources (software and/or hardware) and also the use of power resources with respect to different display devices 12. One or more of these actions can be taken in response to the detection of the user's area of interest in the image 16.

The detection step S3 can take the form of one or more different methods, such as the detection of user inputs on the user interface devices 14, for example relating to the location of the cursor 20 in the image 16. Other methods include tracking physiological parameters of the user 28 such as eye tracking and/or head/face tracking. These can be considered as decisions about direct data on the user. More indirect cues can also be used to determine the user's interest, such as the user's history of working with specific applications on specific displays 12. For example, an incoming email may be received which relates to an email application that the user is not currently viewing, but from historic data it may be assumed that the user 28 is likely to switch their attention to this application in the near future. Therefore a higher priority can then be assigned to this display region, at the expensive of other areas of the displayed image 16.

Claims

1. A method of operating a data processing system, the system comprising a processing device, a display device and a user interface device, the method comprising the steps of: o running one or more applications on the processing device, o displaying an image on the display device, o detecting a user's interest with respect to a region of the image, o assigning a higher priority to the detected region, and o assigning a lower priority to an area outside the detected region.

2. A method according to claim 1 , and further comprising updating the image on the display device, the updating comprising updating the higher priority region before the lower priority area.

3. A method according to claim 1 or 2, and further comprising adjusting the frame rate of the display device, the frame rate of the higher priority region being greater than the frame rate of the lower priority area.

4. A method according to claim 1 , 2 or 3, wherein the system further comprises an audio device, and the method further comprises adjusting the audio output of the audio device, the audio output of the applications of the higher priority region being greater than the audio output of the applications of the lower priority area.

5. A method according to any preceding claim, and further comprising adjusting the operating system priority of the applications, the operating system priority of the applications of the higher priority region being greater than the operating system priority of the applications of the lower priority area.

6. A method according to any preceding claim, wherein the system further comprises a second display device, and the step of displaying an image on the display device comprises displaying the image on both display devices.

7. A method according to claim 6, wherein the step of detecting a user's interest with respect to a region of the image comprises detecting the user's interest with respect to the first or second display device.

8. A method according to claim 7, and further comprising adjusting the power consumption of the display devices, the power consumption of the display device of the higher priority region being greater than the power consumption of the display device of the lower priority area.

9. A method according to any preceding claim, wherein the step of detecting a user's interest with respect to a region of the image comprises detecting the position of a cursor within the image.

10. A method according to any preceding claim, wherein the step of detecting a user's interest with respect to a region of the image comprises tracking one or more physiological features of a user,

11. A data processing system comprising a processing device, a display device and a user interface device, wherein: o the processing device is arranged to run one or more applications, o the display device is arranged to display an image, and o the processing device is further arranged to detect a user's interest with respect to a region of the image, assign a higher priority to the detected region, and assign a lower priority to an area outside the detected region.

12. A system according to ciaim 11 , wherein the processing device is further arranged to update the image on the display device, the updating comprising updating the higher priority region before the lower priority area.

13. A system according to claim 11 , or 12, and further comprising an audio device, wherein the processing device is further arranged to adjust the audio output of the audio device, the audio output of the applications of the higher priority region being greater than the audio output of the applications of the lower priority area.

14. A system according to claim 11 , 12 or 13, and further comprising a second display device, both display devices arranged to display the image.

15. A computer program product on a computer readable medium for operating a data processing system, the system comprising a processing device, a display device and a user interface device, the product comprising instructions for: o running one or more applications on the processing device, o displaying an image on the display device, o detecting a user's interest with respect to a region of the image, o assigning a higher priority to the detected region, and o assigning a lower priority to an area outside the detected region.