GB2408435A - Mouse control emulation for web browser via television - Google Patents

Abstract

The invention relates to the provision of an embedded web browser, typically as part of a broadcast data receiver of the type provided for reception of data transmitted from a remote location. In addition to the processing of data for the generation of television programmes and the like, the embedded web browser allows the selection of internet pages which can be displayed on a television screen. The invention provides the facility for a conventional remote control device to be used to allow navigation about the displayed internet page on the display screen and also the selection of selectable options. The remote control device generates signals, which may be from arrow keys, to allow control of an on screen cursor.

Description

Mouse Control Emulation For Web Browser Devices lithe invention which is

the subject of this application relates to an internet web browser service and particularly, although not necessarily exclusively, to the web browser device which is known as an embedded web browser, which can be provided as part of apparatus such as a broadcast data receiver. A broadcast data receiver can be provided as part of, or connected to, a television set and used to provide the web browser service which is viewable via the display screen of the television set.

One known problem with embedded web browser devices is that there is only limited user controllability of the mechanisms used to navigate the web pages. 'this is in contrast to a conventional desktop 1'(, where a mouse may be used to' move a cursor around the screen in any direction, at differing speeds and with a large degree of user control. With the embedded browser, typically two dimensional "cursor highlighting" navigation systems are provided such as those which can be controlled by the remote control device for the broadcast data receiver. While this has been accepted to date as this type of web browser has been treated as an additional rather than core service of the apparatus, the increasing utilisatioi1 of this type of web browser and appeal of being able to use the web browser in a user's living roon1 rather than study, means that there is an increasing need for improved user navigation capabilities for users calf embedded web browsers.

Lyle web pages when displayed can contain regions designed to be "filled in" by pieces of software that are not part of the web browser itself. '-I'hese extra pieces of software are called plug- ins and provide a way in which the range of functions that a web page normally provides can be greatly extended. E:xamples include older implementations of Java virtual machines, Macrcmeclia l'lash movies and embedded video/movie players.

In sc>me software architectures for the browser, the plug-ins are completely independent applicatic>ns in their own right, running alongside the web browser and communicating with it thtc>ugh some messaging scheme, but not being in any tither way part of the browser software itself. Such architectures keep any faults and upgrade issues very localised to the plug-in applications themselves, but create problems for the end-user when they try tc> navigate around the web page being presented.

As stated, embedded web browser devices of this nature are often controlled using infra-red remote control handsets.

lihesc usually have some kind of cursor controls (up/down/left/right arrows) for use when entering data into text fields and for moving around the display. 1\ "Select" key is typically provided tc> "activate" an item, e.g. follow a hypertext link.

( onventionally, the movement about the display by pressing the keys is illustrated to the user by highlighting the selectable items on the display. 'Lyle highlight can be in the form of some brightly c->loured box that surrounds the selectable region, though this is implementation defined. 'lihe cursc>r keys in this case always act lice cursor keys; the highlight jumps between any selectable items. It is easy to see how this makes navigating lists of items on a web page a simple and fluid affair no accurate "pointing" is needed and, as the web browser has rendered the web page onto the television display, it is relatively easy fr>r the highlight to be moved around the page as the browser knows which regions are selectable. It can control entry and exit of editable text fields, selection of hyperlinks on the page, scrolling the page if parts Of it are hidden off the edge of the television display, and so-on. There are plenty of devices already available that use such an approach.

However, when plug ins are added to the display, the same have to develop their own navigation schemes if mouse control devices are not available as the plug ins are typically provided to run on mouse controlled systems. 'I'his can lead to inconsistent navigation approaches and duplication of software code.

The aim of the present invention is to provide an improved navigation system which achieves, at least partially, the effect of a mouse controlled navigation in an embedded web browser.

In a hrst aspect of the invention there is provided an internet /r web browser system, said system allowing the generation on a ': I display screen of at least one display page from the internet, said i J) system including control means to allow user interaction with the display and navigation by the user to user specified locations on the display and wherein the system includes two navigation rneth<-'ds, a first leased on the indication of position by highlighting regions of the display and the second allowing mcverncnt Off a cursor across a defined region of the display and selection can be made between the first and second navigation systems dependent upon the status of the web browser system.

In cane embodiment the switch to the mouse emulation occurs when an additional or plup,--in service is utilised and uses system- level mouse emulation to perform "drag-and-drop" operations in a manor that recluires no special knowledge or extra code in the browser plug-ins and involves no extra software or change in s o ftware.

Preferably the mouse emulation navigation system is such that if a mouse control device is subsequently added to the web browser, the navigation system will operate in the same manner as before.

In sync embodiment, the highlighting based navigation system utilises the user selection of a series of keys on a remote control device such as up/down/left/ right and, dependent upon the key depressions made and the infra red signals generated and transmitted from the remote control device to the receiving apparatus, such as a broadcast data receiver, regions of the display are selectively highlighted. 'typically the cursor position is indicated by the highlighting or otherwise differentiating, the appearance of the particular portion of the screen with the highlighting moving as the key presses are made. If the user wishes to select a highlighted portion there is a particular key designated for the task.

If the mouse emulation navigation system is activated, the cursor appears on screen as a small line or symbol such as a crossllair'' and this is moved by repeated key presses. When the "select key" is pressed a mouse control device button "click" is emulated at the control level of the system and with the select key depressed, or until a release signal is sent by another key press, the cursor, or other remote control device keys emulate mouse device controls when pressed.

When the item selected is a plug-in, the bounding box of the area under the plug-in's control is highlighted using the highlighting navigating system and over the defined region a symbol such as a cross hair is illustrated. While in this region use of the cursor keys move this pointer under control of the web browser. When the "Select" key is pressed again, however, a mouse click is Undulated at the operating system level. At this point, the Operating system switches control of the cursor keys to stop acting like key presses, and start emulating a mouse.

This behaviour continues until the selection key is released, at which point the cursor keys revert to normal actions. The plug- in applications see a normal mouse click event, just as if a real mouse was present, or, if the user moves the cursor keys during the selection, plug-ins see a normal mouse drag and drop ->peration. The cursor keys do not always emulate a mouse, nor do they always act like cursor keys; but the transition is invisible both to the user and higher level software.

In a further aspect of the invention there is provided a web browser system, said browser system including a display screen and a means for navigating a cursor around the display screen using a remote control device incorporating a number of user selectable keys, said remote control device generating configure signals to allow the control of the cursor in response to the key presses made and wherein the cursor is represented on the display by a symbol.

In one embodiment the symbol can move across the entire area of the display. '[typically, when the symbol lies:' n a selectable portion:' f the display a further key depression can cause the selection to be made.

typically the navigation system is one of two interchangeable navigation systems, both controllable via the remote control device with the other navigation system utilising the highlighting of only those regions on the display which are user selectable as a means of indicating the position to the user as they navigate across the clisplay.

I'he advantages which can be obtained are that the control of the plug in feature can be achieved using the existing software architecture of the web browser and independent plug-in items.

G

The web browser user can continue to use the same cursor key input and a "Select" keys from some control source such as an infra-red handset to perform both the highlight-based navigation and the mouse emulation navigation systems.

in addition to performing mouse like navigation, in one embodiment, the further depression of a control key can allow the use of the cursor keys to have a further operating characteristic such as, for example, the performance of drag and select functions and the plug in software cannot distinguish between a real mouse and the emulated version -,f the invention.

Typically, depending on the protocols used by the software in the device, the operating system may send a mouse click event for the first time it is told to start mouse emulation.

Depending on the length of time the "Select" key is held for and whether or not the cursor keys are used to move the emulated mouse pointer around, events for drag start and movement will be sent - the exact criteria and event schemes are implementation defined.

Preferably, when the user releases the "select" or other key or depresses a "release" key, mouse emulation at the operating systc m level stops and subsequent "Select" and cursor key events are sent to the web browser as before. there may be a drag end event or similar, the details being implementation defined, sent to the plug-in prior to this as the final stage of mouse emulation, should there have been a drag operation considered to be in progress by said operating system.

Specific embodiments of the invention will now be described; wherein Figure I illustrates a display page from an interoet web browser using the first navigation system in accordance with the invention; I7igurc 2 illustrates the display page of Figure 1 utilising the second navigation system in accordance with the invention; and ligure 3 illustrates a conventional remote control device.

I-'igure 1 illustrates a typical web browser display of a web page, in this case the web page of BB(: Online, which is of the type which can be generated on a display screen of a television set.

The page comprises a number of portions and selectable items arc indicated by the reference numeral 4. F,ach of these items can be selected by the user navigating using the cursor keys G can the remote control device shown in Figure 3. 'l-'o allow the user to navigate between the selectable items a particular selectable item location is highlighted 8 as shown. 'I'his therefore means that the user can then press the keys on the remote control device with respect to the highlighted portion to select the same and also to move the highlighting to other particular selectable items 4 as they wish. When the highlighting appears on the required item 8 the user can select the same by pressing the "scleet" key 10 on the remote control. When the selection is made, the web browser is then directed to the new link represented by the selected item and may, for example, generate a new clisplay from a new internet site.

ProlJlems with this approach can arise when an item in the web page displayed is not under direct control calf the web browser such as a plug in or additional service 12. In this instance when the user selects the plug in portion, the co-ordinates of the specific part of the portion on which the selection was made arc sent to the remote web server. The server's actions will differ depending on the region selected however the web browser has no idea which regions are significant so the browser cannot utilise an intelligent, highlight based navigation method and this is where the second navigation scheme as described with reference to Figure 2 can be selected to be used.

When the location of the plug in portion is identified using the highlighting method, the edge of the plug in portion 12 on the display is highlighted in the usual fashion. If the user then presses the "elect" key on their remote control device, the different navigation method is entered. The web browser displays a symbol, such as the cross hair 14 shown and the cursor keys are now used to move this around over the region covered by the portion in the web page and so the user can move the crosshair throughout the region using the cursor keys and this represents a first part of the method.

If the user then presses the "Select" key a second time, typically tc' activate a specific location on the plug in portion that the symbol is over then the web browser, which is handling the cursor key presses, must now let the plug-in know about the selection event. This can be achieved by emulating a "click" event so that the plug-in believes a mouse device has actually been cliched over the region in the web page it is controlling, and acts accordingly. There is no need to add code to the plug- in to deal with special cases since real mouse events and those "faked" from the web browser are indistinguishable.

If the user then wants to leave the mouse emulation navigation mode, they simply move the symlol tc' "go over" the edge 16 of the item leing navigated and normal highlight based navigation is resumed.

At all times, the operating system is delivering key press events for the cursor and "Select" keys from the infra-red handset to the web browser, which is controlling all actions based on these evcuts.

There is then the question of how the symbol is moved or "clraggcd" to move features on the region across the region of the display. S;-'me plug-ins allow drag-and-drop operations. In a normal - 'pcration, the user if they had a mouse control device click on an item, keep the button held down, move the mouse somewhere, and release the mouse button. Macromedia Flash movies can require dragging -,perations, as can some Java applets, so there are certainly existent, common cases where this is required.

In accordance with the invention the crosshair symbol can be moved by the activation of an operating system feature for mouse emulation. When the web browser sees the "Select" key pressed event during the crosshair navigation "mode" as illustrated above, it tells the operating system that this has happened. lit this point, the operating system takes control of the cursor keys and at a knew level makes them look like a mouse device. 'lathe web browser no longer has to send a "fake" mouse click event to the plug-in; the operating system does this.

Whatever rules the operating system has for messages relating to mouse drags now operate in a completely normal fashion with the cursor keys on the remote control handset being used to move the mouse pointer, and the "Select" key on the handset acting as the mouse click button so that the select and drag cooperation can be achieved.

This mouse emulation continues only for as long as the "Select" key is held dowrl. if the user briefly presses "Select" and lets go, for a simple "click" operation, the operating system only very briefly takes over the cursor keys, and just delivers a normal mouse click event to the plug-in being navigated. If the user keeps the "select" key held down and now activates the cursor keys to perform a drag operation, the mouse emulation is still in force, and the plug-in sees the events it would expect for any mouse drag. When the user releases the "Select" key on the handset, the drag operation finishes, any Jrop-related events are sent, and the -'perating system stops performing mouse emulation. Subsequent cursor key presses are sent to the web browser for it tc> handle in the normal fashion.

In one practical implementation a broadcast data receiver with a 1tIS( ()S system includes a web browser providing highlight base navigation -' n a television display. The user visits a web page with a list of links to various Java games. Fly default, the htst item in the list is highlighted. The highlight consists of a green box which surrounds the selectable item.

Using simple up/down/left/right/Select controls on a low cost infra-red handset, the user moves to the fourth item in the list (by pressing "down" three times) and follows the link using "Select". A new web page is fetched, which includes a Java apples. the apples is a demonstration game of (:hess.

Instructions on the web page explain that pieces are moved by dragging them from their current position to the intended position. '1-he page contains only the Java apples as a selectable item, so by default this is highlighted by the web browser by a large green box drawn around the edges of the area being redrawn by the J ava plug-in.

the user presses "Select". A crosshair symbol is shown. This is actually the system hardware mouse pointer, but its position is being controlled by the web browser as there is no actual real mouse plugged into the set- top box. 'the user presses the 1 1 cursor keys and the crosshair symbol moves in the appropriate direction. The user moves the pointer cover the "Start game" button in the apples. The user presses "Select" again.

Up until now, the web browser has been controlling the navigation based on the cursor, key events passed to it by the operating system. At this point, however, the web browser asks the operating system to start mouse emulation until "Select" is released. 'I'he operating system does so, and starts by emulating the clicking of the left mouse button (a "mouse down" event).

This makes another part of the system send a normal mouse click event to the Java apples. The apples immediately starts a new gamy and redraws its area of the web page to reflect this.

The user sees this and releases the "Select" button. The c>perating system emulates a mouse button release (a "mouse up" event). Mouse emulation stops, and cursor key input is restored to the web browser. The browser is now back at its starting state, with the Java apples highlighted, highlight-based navigation in force, and no crosshair pointer showed.

I'he computer player in the Java apples makes its move. The user chooses their response. The user once again presses "Select" and the web browser displays the crosshair pointer.

The user moves over the piece they want to move and presses "Select" again. As before, the operating system starts mc>use emulation and emulates a left mouse button click; higher level software sends a click event tc> the apples, which this time ignores it, as it requires a drag for game piece movement. The user keeps "Select" held down and uses the cursor keys to move what appears to be the same crosshair pointer as before; indeed it is the hardware pointer, but now that the operating system is turning cursor key movements into mouse movements, other parts of the operating system believes a real mouse tc> be in use and move the mouse pointer accordingly. Higher level software sees that the minimum drag distance after the first click has been exceetlcd, and sends a drag started message to the Java apples, which then starts to track mouse pointer locations in a normal fashion and plots the piece the user is moving at the relevant location.

It ventually the user reaches the place where they want to drop the piece, and releases "Select". The operating system emulates the release of the mouse button and higher level software sends a drag finished event to the Java apples as a result. As before, mouse emulation then finishes and control returns to the web browser.

lihe Java apples drops the piece in the right place, checking it is a valid we've, and the computer player then makes its move.

l he user responds, and so the game continues, until a winner is found.

At no point would the user have seen any inconsistency between the browser highlight movement, browser pointer emulation and operating system mouse emulation; the lava apples did not need any special code as it only ever saw standard mouse click and drag related events from the operating system; if a real mouse were added to the device, no software changes would be needed.

The same would be true if the user later interacted with a Macromcdia Flash movie, or navigated a client- side image map, where temporary operating system level pointer emulation would not be used.

Thus, web pages may be intuitively and swiftly navigated using a web browser controlled highlight system based around simple cursor key and "Select" key controls on (for example) an infra- red handset. There is no need for a separate mouse-like device.

When attempting to navigate around independent plug-in items, there is a seamless switch to real mouse emulation based on the "Select" key being held down and cursor key movement, which allows pl ug- in authors to write completely normal code for a mouse-based environment with no extra knowledge of any kind about the highlighting schemes employed. Stanclard operating system rules for dragging are maintained, so consistency across different components is ensured. There is no unnecessary code duplication between plug-in components.

Claims (3)

1. Claims 1. A web browser method, said browser method utilising a display

   screen, an embedded web browser connected to the display screen, a remote control device for emitting control signals to the web browser and a cursor generated can at least a portion of the display of the display screen, and characterised in that the remote control means navigate the cursor around the display screen using the remote control crevice incorporating a number of user selectable keys, said remote control device generating configured signals to allow the control of the cursor in response to the key presses made and the cursor is represented on the display by a symbol.
2. 2. A method aeecrding to claim 9 eharacterised in that the navigation system is one of twc> interchangeable navigation systems, both controllable via a remote control device, the particular navigation system used dependent upon the location on the display screen.
3. 3. A method according to claim 9 eharaeterised in that the web lr'wser crevice is provided in a broadcast data receiver, said data receiver connected to a television set Oil the display screen calf which the selected page and navigation system is displayed.