GB2383510A

GB2383510A - Mouse control emulation for web browser via television

Info

Publication number: GB2383510A
Application number: GB0126557A
Authority: GB
Inventors: Andrew Hodgkinson; Mark Moir; Alistair Rodger; John Beranek; Paul Skirrow
Original assignee: Pace Micro Technology PLC
Current assignee: Arris Global Ltd
Priority date: 2001-11-06
Filing date: 2001-11-06
Publication date: 2003-06-25
Anticipated expiration: 2021-11-06
Also published as: GB2408435A; GB2408435B; GB0126557D0; GB0503576D0; GB2383510B

Abstract

An embedded web browser, typically as part of a broadcast data receiver of the type provided for reception of data transmitted from a remote location. In addition to the processing of data for the generation of television programmes and the like, the embedded web browser allows the selection of internet pages which can be displayed on a television screen. The invention provides a conventional remote control device to be used to allow navigation about the displayed internet page on the display screen and also the selection of selectable options. Two forms of navigation can be provided dependent on the location on the display screen, a first which involves the highlighting of portions of the display screen to indicate the particular location on the screen. The second involves the generation on-screen of a cursor, the movement of both being controlled and moved by selective depression of keys on the remote control device, simulating the movement according to the use of a mouse.

Description

Mouse Control Emulation For Web Browser Devices The invention which is the subject of this application relates to an internet web browser service and particularly, although not necessarily exclusively, to the web browser device which is known as an embedded web browser, which can be provided as part of apparatus such as a broadcast data receiver. A broadcast data receiver can be provided as part of, or connected to, a television set and used to provide the web browser service which is viewable via the display screen of the television set.

One known problem with embedded web browser devices is that there is only limited user controllability of the mechanisms used to navigate the web pages. This is in contrast to a conventional desktop PC, where a mouse may be used to move a cursor around the screen in any direction, at differing speeds and with a large degree of user control. With the embedded browser, typically two dimensional"cursor highlighting"navigation systems are provided such as those which can be controlled by the remote control device for the broadcast data receiver. While this has been accepted to date as this type of web browser has been treated as an additional rather than core service of the apparatus, the increasing utilisation of this type of web browser and appeal of being able to use the web browser in a user's living room rather than study, means that there is an increasing need for improved user navigation capabilities for users of embedded web browsers.

The web pages when displayed can contain regions designed to be"filled in"by pieces of software that are not part of the web browser itself. These extra pieces of software are called plugins and provide a way in which the range of functions that a web page normally provides can be greatly extended. Examples

include older implementations of Java virtual machines, Macromedia Flash movies and embedded video/movie players.

In some software architectures for the browser, the plug-ins are completely independent applications in their own right, running alongside the web browser and communicating with it through some messaging scheme, but not being in any other way part of the browser software itself. Such architectures keep any faults and upgrade issues very localised to the plug-in applications themselves, but create problems for the end-user when they try to navigate around the web page being presented.

As stated, embedded web browser devices of this nature are often controlled using infra-red remote control handsets.

These usually have some kind of cursor controls (up/down/left/right arrows) for use when entering data into text fields and for moving around the display. A"Select"key is typically provided to"activate"an item, e. g. follow a hypertext link.

Conventionally, the movement about the display by pressing the keys is illustrated to the user by highlighting the selectable items on the display. The highlight can be in the form of some brightly coloured box that surrounds the selectable region, though this is implementation defined. The cursor keys in this case always act like cursor keys; the highlight jumps between any selectable items. It is easy to see how this makes navigating lists of items on a web page a simple and fluid affair-no accurate"pointing"is needed and, as the web browser has rendered the web page onto the television display, it is relatively easy for the highlight to be moved around the page as the browser knows which regions are selectable. It can control entry and exit of editable text fields, selection of hyperlinks on the page, scrolling the page if parts of it are hidden off the edge

of the television display, and so-on. There are plenty of devices already available that use such an approach.

However, when plug ins are added to the display, the same have to develop their own navigation schemes if mouse control devices are not available as the plug ins are typically provided to run on mouse controlled systems. This can lead to inconsistent navigation approaches and duplication of software code.

The aim of the present invention is to provide an improved navigation system which achieves, at least partially, the effect of a mouse controlled navigation in an embedded web browser.

In a first aspect of the invention there is provided an internet web browser system, said system allowing the generation on a display screen of at least one display page from the internet, said system including control means to allow user interaction with the display and navigation by the user to user specified locations on the display and wherein the system includes two navigation methods, a first based on the indication of position by highlighting regions of the display and the second allowing movement of a cursor across a defined region of the display and selection can be made between the first and second navigation systems dependent upon the status of the web browser system.

In one embodiment the switch to the mouse emulation occurs when an additional or plug-in service is utilised and uses systemlevel mouse emulation to perform"drag-and-drop"operations in a manner that requires no special knowledge or extra code in the browser plug-ins and involves no extra software or change in software.

Preferably the mouse emulation navigation system is such that if a mouse control device is subsequently added to the web

browser, the navigation system will operate in the same manner as before.

In one embodiment, the highlighting based navigation system utilises the user selection of a series of keys on a remote control device such as up/down/left/right and, dependent upon the key depressions made and the infra red signals generated and transmitted from the remote control device to the receiving apparatus, such as a broadcast data receiver, regions of the display are selectively highlighted. Typically the cursor position is indicated by the highlighting or otherwise differentiating, the appearance of the particular portion of the screen with the highlighting moving as the key presses are made. If the user wishes to select a highlighted portion there is a particular key designated for the task.

If the mouse emulation navigation system is activated, the cursor appears on screen as a small line or symbol such as a "crosshair"and this is moved by repeated key presses. When the "select key"is pressed a mouse control device button"click"is emulated at the control level of the system and with the select key depressed, or until a release signal is sent by another key press, the cursor, or other remote control device keys emulate mouse device controls when pressed.

When the item selected is a plug-in, the bounding box of the area under the plug-in's control is highlighted using the highlighting navigating system and over the defined region a symbol such as a cross hair is illustrated. While in this region use of the cursor keys move this pointer under control of the web browser. When the"Select"key is pressed again, however, a mouse click is emulated at the operating system level. At this point, the operating system switches control of the cursor keys to stop acting like key presses, and start emulating a mouse.

This behaviour continues until the selection key is released, at which point the cursor keys revert to normal actions. The plugin applications see a normal mouse click event, just as if a real mouse was present, or, if the user moves the cursor keys during the selection, plug-ins see a normal mouse drag and drop operation. The cursor keys do not always emulate a mouse, nor do they always act like cursor keys; but the transition is invisible both to the user and higher level software.

In a further aspect of the invention there is provided a web browser system, said browser system including a display screen and a means for navigating a cursor around the display screen using a remote control device incorporating a number of user selectable keys, said remote control device generating configure signals to allow the control of the cursor in response to the key presses made and wherein the cursor is represented on the display by a symbol.

In one embodiment the symbol can move across the entire area of the display. Typically, when the symbol lies on a selectable portion of the display a further key depression can cause the selection to be made.

Typically the navigation system is one of two interchangeable navigation systems, both controllable via the remote control device with the other navigation system utilising the highlighting of only those regions on the display which are user selectable as a means of indicating the position to the user as they navigate across the display.

The advantages which can be obtained are that the control of the plug in feature can be achieved using the existing software architecture of the web browser and independent plug-in items.

The web browser user can continue to use the same cursor key input and a"Select"keys from some control source such as an infra-red handset to perform both the highlight-based navigation and the mouse emulation navigation systems.

In addition to performing mouse like navigation, in one embodiment, the further depression of a control key can allow the use of the cursor keys to have a further operating characteristic such as, for example, the performance of drag and select functions and the plug in software cannot distinguish between a real mouse and the emulated version of the invention.

Typically, depending on the protocols used by the software in the device, the operating system may send a mouse click event for the first time it is told to start mouse emulation.

Depending on the length of time the"Select"key is held for and whether or not the cursor keys are used to move the emulated mouse pointer around, events for drag start and movement will be sent-the exact criteria and event schemes are implementation defined.

Preferably, when the user releases the"Select"or other key or depresses a"release"key, mouse emulation at the operating system level stops and subsequent"Select"and cursor key events are sent to the web browser as before. There may be a drag end event or similar, the details being implementation defined, sent to the plug-in prior to this as the final stage of mouse emulation, should there have been a drag operation considered to be in progress by said operating system.

Specific embodiments of the invention will now be described; wherein

Figure 1 illustrates a display page from an internet web browser using the first navigation system in accordance with the invention ; Figure 2 illustrates the display page of Figure 1 utilising the second navigation system in accordance with the invention; and Figure 3 illustrates a conventional remote control device.

Figure 1 illustrates a typical web browser display of a web page, CtT) in this case the web page of BBC Online, which is of the type which can be generated on a display screen of a television set. The page comprises a number of portions and selectable items are indicated by the reference numeral 4. Each of these items can be selected by the user navigating using the cursor keys 6 on the remote control device shown in Figure 3. To allow the user to navigate between the selectable items a particular selectable item location is highlighted 8 as shown. This therefore means that the user can then press the keys on the remote control device with respect to the highlighted portion to select the same and also to move the highlighting to other particular selectable items 4 as they wish. When the highlighting appears on the required item 8 the user can select the same by pressing the "select"key 10 on the remote control. When the selection is made, the web browser is then directed to the new link represented by the selected item and may, for example, generate a new display from a new internet site.

Problems with this approach can arise when an item in the web page displayed is not under direct control of the web browser such as a plug in or additional service 12. In this instance when the user selects the plug in portion, the co-ordinates of the specific part of the portion on which the selection was made are sent to the remote web server. The server's actions will differ

depending on the region selected however the web browser has no idea which regions are significant so the browser cannot utilise an intelligent, highlight based navigation method and this is where the second navigation scheme as described with reference to Figure 2 can be selected to be used.

When the location of the plug in portion is identified using the highlighting method, the edge of the plug in portion 12 on the display is highlighted in the usual fashion. If the user then presses the"Select"key on their remote control device, the different navigation method is entered. The web browser displays a symbol, such as the cross hair 14 shown and the cursor keys are now used to move this around over the region covered by the portion in the web page and so the user can move the crosshair throughout the region using the cursor keys and this represents a first part of the method.

If the user then presses the"Select"key a second time, typically to activate a specific location on the plug in portion that the symbol is over then the web browser, which is handling the cursor key presses, must now let the plug-in know about the selection event. This can be achieved by emulating a"click" event so that the plug-in believes a mouse device has actually been clicked over the region in the web page it is controlling, and acts accordingly. There is no need to add code to the plugin to deal with special cases since real mouse events and those "faked"from the web browser are indistinguishable.

If the user then wants to leave the mouse emulation navigation mode, they simply move the symbol to"go over"the edge 16 of the item being navigated and normal highlight based navigation is resumed.

At all times, the operating system is delivering key press events for the cursor and"Select"keys from the infra-red handset to the web browser, which is controlling all actions based on these events.

There is then the question of how the symbol is moved or "dragged"to move features on the region across the region of the display. Some plug-ins allow drag-and-drop operations. In a normal operation, the user if they had a mouse control device click on an item, keep the button held down, move the mouse somewhere, and release the mouse button. Macromedia Flash movies can require dragging operations, as can some Java applets, so there are certainly existent, common cases where this is required.

In accordance with the invention the crosshair symbol can be moved by the activation of an operating system feature for mouse emulation. When the web browser sees the"Select"key pressed event during the crosshair navigation"mode"as illustrated above, it tells the operating system that this has happened. At this point, the operating system takes control of the cursor keys and at a low level makes them look like a mouse device. The web browser no longer has to send a"fake"mouse click event to the plug-in; the operating system does this. Whatever rules the operating system has for messages relating to mouse drags now operate in a completely normal fashion with the cursor keys on the remote control handset being used to move the mouse pointer, and the"Select"key on the handset acting as the mouse click button so that the select and drag operation can be achieved.

This mouse emulation continues only for as long as the"Select" key is held down. If the user briefly presses"Select"and lets go, for a simple"click"operation, the operating system only

very briefly takes over the cursor keys, and just delivers a normal mouse click event to the plug-in being navigated. If the user keeps the"Select"key held down and now activates the cursor keys to perform a drag operation, the mouse emulation is still in force, and the plug-in sees the events it would expect for any mouse drag. When the user releases the"Select"key on the handset, the drag operation finishes, any drop-related events are sent, and the operating system stops performing mouse emulation. Subsequent cursor key presses are sent to the web browser for it to handle in the normal fashion.

In one practical implementation a broadcast data receiver with a RISC OS system includes a web browser providing highlight base navigation on a television display. The user visits a web page with a list of links to various Java games. By default, the first item in the list is highlighted. The highlight consists of a green box which surrounds the selectable item.

Using simple up/down/left/right/Select controls on a low cost infra-red handset, the user moves to the fourth item in the list (by pressing"down"three times) and follows the link using "Select". A new web page is fetched, which includes a Java applet. The applet is a demonstration game of Chess. Instructions on the web page explain that pieces are moved by dragging them from their current position to the intended position. The page contains only the Java applet as a selectable item, so by default this is highlighted by the web browser by a large green box drawn around the edges of the area being redrawn by the Java plug-in.

The user presses"Select". A crosshair symbol is shown. This is actually the system hardware mouse pointer, but its position is being controlled by the web browser as there is no actual real mouse plugged into the set-top box. The user presses the

cursor keys and the crosshair symbol moves in the appropriate direction. The user moves the pointer over the"Start game" button in the applet. The user presses"Select"again.

Up until now, the web browser has been controlling the navigation based on the cursor, key events passed to it by the operating system. At this point, however, the web browser asks the operating system to start mouse emulation until"Select"is released. The operating system does so, and starts by emulating the clicking of the left mouse button (a"mouse down"event). This makes another part of the system send a normal mouse click event to the Java applet. The applet immediately starts a new game, and redraws its area of the web page to reflect this. The user sees this and releases the"Select"button. The operating system emulates a mouse button release (a"mouse up" event). Mouse emulation stops, and cursor key input is restored to the web browser. The browser is now back at its starting state, with the Java applet highlighted, highlight-based navigation in force, and no crosshair pointer showed.

The computer player in the Java applet makes its move. The user chooses their response. The user once again presses "Select"and the web browser displays the crosshair pointer. The user moves over the piece they want to move and presses "Select"again. As before, the operating system starts mouse emulation and emulates a left mouse button click; higher level software sends a click event to the applet, which this time ignores it, as it requires a drag for game piece movement. The user keeps"Select"held down and uses the cursor keys to move what appears to be the same crosshair pointer as before; indeed it is the hardware pointer, but now that the operating system is turning cursor key movements into mouse movements, other parts of the operating system believes a real mouse to be in use and move the mouse pointer accordingly. Higher level software

sees that the minimum drag distance after the first click has been exceeded, and sends a drag started message to the Java applet, which then starts to track mouse pointer locations in a normal fashion and plots the piece the user is moving at the relevant location.

Eventually the user reaches the place where they want to drop the piece, and releases"Select". The operating system emulates the release of the mouse button and higher level software sends a drag finished event to the Java applet as a result. As before, mouse emulation then finishes and control returns to the web browser.

The Java applet drops the piece in the right place, checking it is a valid move, and the computer player then makes its move.

The user responds, and so the game continues, until a winner is found.

At no point would the user have seen any inconsistency between the browser highlight movement, browser pointer emulation and operating system mouse emulation; the Java applet did not need any special code as it only ever saw standard mouse click and drag related events from the operating system; if a real mouse were added to the device, no software changes would be needed. The same would be true if the user later interacted with a Macromedia Flash movie, or navigated a client-side image map, where temporary operating system level pointer emulation would not be used.

Thus, web pages may be intuitively and swiftly navigated using a web browser controlled highlight system based around simple cursor key and"Select"key controls on (for example) an infrared handset. There is no need for a separate mouse-like device. When attempting to navigate around independent plug-in items,

there is a seamless switch to real mouse emulation based on the "Select"key being held down and cursor key movement, which allows plug-in authors to write completely normal code for a mouse-based environment with no extra knowledge of any kind about the highlighting schemes employed. Standard operating system rules for dragging are maintained, so consistency across different components is ensured. There is no unnecessary code duplication between plug-in components.

Claims

Claims 1. An internet browser system, said system comprising an embedded web browser device connected to a display screen and allowing the generation on the display screen of at least one display page generated and selected from the internet, said system including remote control means to allow user interaction with the display and web browser device to allow the user to navigate to specified locations on the display and characterised in that the system includes two navigation methods, a first based on the indication of position by highlighting regions of the display and a second allowing movement of a cursor display across a defined region of the display and selection can be made between the first and second navigation systems with reference to the status of the internet browser system.
2. A system according to claim 1 characterised in that the selection of the second navigation method occurs when an additional or plug-in service is utilised in the selected display page.
3. A system according to claim 1 characterised in that the first navigation system utilises the user selection of a series of keys on a remote control device and, via infrared signals generated and transmitted from the remote control device to the receiving apparatus, a region of the display is selectively highlighted.
4. A system according to claim 3 characterised in that the highlighted portion of the display is changed in accordance with any additional key presses being made on the remote control device.

<Desc/Clms Page number 15>
5. A system according to claim 1 characterised in that the second navigation system is activated when a region of the display generated via a plug-in is to be navigated and a cursor is displayed on that region of the display screen and the position of same is moved by key presses of a remote control device.
6. A system according to claim 5 characterised in that if a pre-designated button on a remote control device is selected, the function on the display on which the cursor key is positioned at that time is activated.
7. A system according to claim 1 characterised in that the selection and switch between navigation methods is achieved by the selection and depression of a predetermined key of the remote control device and the generation of a signal from the remote control device to the receiver.
8. A system according to claim 1 characterised in that the selection between first and second navigation methods occurs automatically when an interface between display portions of the displayed internet page respectively generated by the embedded web browser or a plug-in is crossed.
9. A web browser method, said browser method utilising a display screen, an embedded web browser connected to the display screen, a remote control device for emitting control signals to the web browser and a cursor generated on at least a portion of the display of the display screen, and characterised in that the remote control means navigate the cursor around the display screen using the remote control device incorporating a number of user selectable keys, said remote control device generating configured signals to allow the control of the cursor in response to the key presses made and the cursor is represented on the display by a symbol.

<Desc/Clms Page number 16>
10. A method according to claim 9 characterised in that the navigation system is one of two interchangeable navigation systems, both controllable via a remote control device, the particular navigation system used dependent upon the location on the display screen.
11. A method according to claim 9 characterised in that the web browser device is provided in a broadcast data receiver, said data receiver connected to a television set on the display screen of which the selected page and navigation system is displayed.