BE1025593B1 - METHOD AND DEVICE AND SYSTEM FOR PROVIDING DOUBLE MOUSE SUPPORT - Google Patents

Abstract

A pointing device for use in a computer system that provides dual mouse support. The pointing device (M2 *) comprising: a motion detection mechanism; and a control unit connected to the motion detection mechanism, and adapted to send a motion message (mov2) including displacement parameters (dx, dy) when a motion is detected; at least one button (M2b1) and a button detection mechanism to detect whether the at least one button (M2b1) is pressed or released; and optionally a scroll wheel and a scroll wheel mechanism to detect whether the scroll wheel is being rotated; wherein the control unit is connected to the button detection mechanism, and if present also to the scroll wheel mechanism, and is adapted to first send a dummy movement message (dmov) before sending a button-pressed message (press2) or a scroll wheel rotation- message. The dummy displacement values dx, dy are preferably very small. The pointing device can be a wired USB mouse or a wireless mouse.

Description

METHOD AND DEVICE AND SYSTEM FOR PROVIDING DOUBLE MOUSE SUPPORT

Domain of the invention

The present invention relates generally to the domain of computers with a graphical user interface and multiple pointing devices. Specifically, the present invention relates to a computer system, a computer device, a computer-implemented method, and a computer program product for providing dual mouse support, and to a mouse device, and an adapter or hub or dongle, and to a display device, and to a display device, and video interface device, and on a set of parts.

BACKGROUND OF THE INVENTION

FIG. 1 shows a schematic block diagram of a classical computer system 100 comprising a computer device 101 (e.g., a desktop computer) with a keyboard input connected to a keyboard device 102, and with a mouse input connected to a mouse device 103, and with a video output connected to a display screen or monitor or screen 104. The computer device 101 of FIG. 1 comprises one or more central processing units (CPU) which execute an operating system O / S with a graphical user interface GUI, such as for example Windows 7 or Windows 10 from Microsoft Corporation, or mac OS 10.12 Sierra or mac OS 10.13 High Sierra from Apple Inc. In the example of FIG. 1, the computer 101 also executes two applications: a PDF document viewer 107a (e.g., Acrobat Reader) displayed on the left-hand side of the screen 104, and a text editor 107b (e.g., Microsoft Word) shown on the right-hand side of the screen 104.

Such a system 100 can be used, for example, by a translator or proofreader for simultaneous viewing of a source document, for example a granted European patent document with English, French and German claims, and a translation of the claims into another language (e.g. Dutch or Spanish or Italian). Although the translation itself will typically be performed using a CAT tool (computer-assisted translation), the final proof reading step is preferably performed by comparing the original PDF document and the final text document, or if the final document is also converted to a PDF document, by comparing the two PDF documents.

When proofreading (or verifying and / or correcting the translation), the translator or proofreader must constantly shift his focus to and fro between the visual representations of the two documents 107a, 107b, and must continually acquire the current context in each of these documents, which is time-consuming and mentally demanding. This cognitive burden can be reduced by, for example, physically pointing to the words of the source text, using a finger of the non-dominant hand, and by indicating corresponding words of the target text

BE2017 / 5896 using the mouse cursor 105, which can be moved on the screen by physically moving the mouse device 103, which is held in the dominant hand, as shown in FIG. 1 for a legal person. A left-handed person will typically move the mouse with his left hand, and will point the screen with his right hand.

But this process is not ideal, among other things because the arm cannot always rest on the table, in particular with larger screen dimensions, or with screens mounted above the table. Furthermore, when a translation error is found, and a correction is to be made, both hands are typically brought to the keyboard 102 for typing, such that the context is lost again and needs to be acquired again after the correction is made. It would be advantageous if the computer system 100 provided a mechanism that allows a user to easily switch the focus between two reading locations on the screen 104, which locations can be moved simply and intuitively.

FIG. 2 shows another problem. When reading a document, a user typically uses the mouse cursor to point to the words he is currently reading. If the person wants to highlight or underline some words or add comments, the mouse cursor should typically be moved to the top of the screen, where the user must click to choose a highlight color, or a typewriter tool, or a comment field, etc., but by doing this, the context from which he was reading is lost, and the context must be acquired again, which is time-consuming. It would be advantageous if the computer system 200 provided a mechanism that would allow a user to easily switch the focus between a reading location and a work location, which locations can be moved simply and intuitively.

FIG. 3 shows another problem. When writing or checking certain texts (for example claims of a patent application), it may be desirable to compare two different parts of a single text document, for example a first part containing product claims and a second part containing method claims. In practice, the user sometimes wants to change both text portions, which is fairly simple for the lower text portion of the example shown in FIG. 3, because that is where the mouse cursor 305 is located, but is not simple for the upper text portion, where the finger is located. The user can swap the position of the finger and mouse cursor, but that is not easy in practice without re-acquiring the context at both locations, which is time-consuming. It would be advantageous if the computer system 300 provided a mechanism that would allow a user to easily switch between two different locations on the screen, to read and / or to edit, which locations could be moved easily and intuitively.

FIG. 4 shows another problem. When transcribing (transferring) a sound fragment (i.e., converting spoken text to a written document), a user may

BE2017 / 5896 use an audio player application (e.g., VLC Media Player) to play a sound clip using a start / stop button, and a text editor application for typing the corresponding text information. Using known computers, the user would typically move the mouse cursor to the audio player, click on the audio player window 417 to activate the audio player application, click on the start button, let the audio player play for a few seconds, click on the start / pause button to pause the audio player, move the mouse cursor to the text editor, click on the text editor window to activate the text editor application, optionally click again to reposition the mouse cursor in the text document, type the corresponding text using the keyboard , move the mouse cursor 405 back to the audio player, and repeat the same steps again and again until the audio clip is fully transcribed. Moving the mouse cursor 405 back and forth between the start / stop button of the audio player 417 and the current position in the text editor 407 is time-consuming, especially since the mouse cursor needs to be positioned accurately so as not to click on the wrong button, for example on a rewind button, because an unwanted click on this button would take even more time, because the user will have to find the audio clip. It would be advantageous if the computer system 400 provided a mechanism that allows the user to work at two different screen locations by using both hands, especially when one of these locations has a substantially fixed position on the screen.

Summary of the invention

It is an object of the present invention to provide a computer system that supports two pointing devices, and / or components thereof, such as, for example, an overlay application, a computer program product, a computing device, a set of components, a pointing device, a hub, a display device and a video adapter device.

It is an object of embodiments of the present invention to provide a mechanism to improve or extend the functionality of a computer device that runs an operating system (O / S) with a graphical user interface (GUI) to support two pointing devices instead of from just one.

It is an object of embodiments of the present invention to provide a computer-implemented method to improve or extend the functionality of an O / S and GUI, for example an existing O / S and GUI that provides a single native mouse cursor or mouse pointer and maintains a single native cursor position to display two visible objects and to keep track of two object positions, which objects are individually movable by means of two pointing devices.

It is an object of embodiments of the present invention to provide a computer-implemented method for enhancing or extending the functionality of an O / S and GUI, for example an existing O / S and GUI that a single native mouse cursor or

BE2017 / 5896 provides a mouse pointer and maintains a single native cursor position to display at least two visible objects and to maintain at least two object positions, which objects are individually movable by means of at least two pointing devices.

It is an object of certain embodiments of the present invention to provide such a computer-implemented method, where one of the visible objects is the native mouse cursor or mouse pointer provided by the O / S and the GUI.

It is an object of certain embodiments of the present invention to provide such a computer-implemented method that responds to movements of the at least two pointing devices.

It is an object of certain embodiments of the present invention to provide such a computer-implemented method that responds to movements of the at least two pointing devices, and / or to pressing buttons and releasing buttons and clicking buttons of one or both of the pointing devices, and / or on scroll wheel movements of one or both pointing devices.

It is an object of certain embodiments of the present invention to provide such a computer-implemented method for improving the functionality of the Microsoft Corporation's Windows operating system, headquartered in Redmond, Washington, e.g., Windows 7 or Windows 10 or later versions of Windows.

It is an object of certain embodiments of the present invention to provide such a computer-implemented method for improving the functionality of the Mac OS from Apple Inc. with headquarters in Cupertino, California, US, for example, mac OS Sierra or mac OS High Sierra or later.

It is an object of certain embodiments of the present invention to provide such a computer-implemented method, wherein the dual mouse functionality is provided as a graphical overlay application.

It is also an object of the present invention to provide a computer program product for performing the method, and to provide a computer device that includes such a computer program product and / or performs such a method, and to provide a computer system that includes such a computer device, and to provide a pointing device (e.g., a USB mouse device or an RF mouse device) that is particularly suitable for use in such a computer system, and to provide an adapter or a hub or a dongle (e.g., a USB hub or RF dongle) ) that is particularly suitable for use in such a computer system.

It is also an object of certain embodiments of the present invention to provide a computer system that includes a computer device operatively connected to a first pointing device, and a display device operatively connected to a second pointing device.

BE2017 / 5896

It is also an object of certain embodiments of the present invention to provide a computer system that includes a computer device operatively connected to a first pointing device, and a display device, and a video adapter device operatively connected to a second pointing device.

These and other objects are achieved by a computer-implemented method, a computer program product, a computer device, a computer system, a pointing device, an adapter or hub or dongle, a display device, and a video adapter device according to embodiments of the present invention.

According to a 1st aspect, the present invention provides a computer-implemented method for providing at least two visible objects comprising a first visible object and at least one other visible object, the first visible object being movable through a first pointing device and the at least one another visible object is separately movable by a second pointing device, wherein the first pointing device and the second pointing device are connected to a computer device that executes a control system with a graphical user interface that provides a native mouse cursor; the method comprising the steps of: a) configuring the operating system with the graphical user interface to provide the native mouse cursor as the first visible object; b) providing a graphic overlay window that includes at least one other visible object; c) configuring the graphic overlay window in click-through mode; e) adjusting a position of the first visible object in accordance with movements of the first pointing device; and adjusting a position of the at least one other visible object in accordance with movement of the second pointing device.

The first pointing device can be operated by a dominant hand of a user, the second pointing device can be operated by a non-dominant hand of the same user.

Configuring the O / S to provide the single native mouse cursor may include allowing the O / S to provide the single native mouse cursor, e.g., by not hiding the native mouse cursor.

It is an advantage that this method in combination with an O / S and GUI (with only a single native mouse cursor or pointer) provides two (or more) visible objects that can be moved individually or separately by two separate pointing devices. In other words, it is an advantage that this method extends the functionality of an O / S and GUI to support two pointing devices.

BE2017 / 5896

Providing an overlay window means that the window must be located substantially above other applications (such as most desktop applications) that are not configured as being at the top.

The overlay window is configured in click-through mode, meaning that the O / S is instructed not to offer pointer events such as, for example, mouse movements and mouse clicks to the overlay application, but to an underlying application, ie to an application that has a window with a lower Z order than the overlay window. This offers the advantage that the underlying application (s) respond to mouse events as if there was no overlay window.

It is an advantage of showing the native mouse cursor as one of the visible objects and that the graphic overlay window is configured in click-through mode, because doing this makes retro compatible behavior (backward compatibility) possible for most or all existing applications that have the shape ( change the look & feel of the native mouse cursor. For example, as is known in the art, the mouse cursor can take different forms depending on the context, for example an arrow when moving over an image, or a so-called crosshair (which looks like a roman numeral I) when moving over a text area, or a hand when moving over an object that can be dragged, etc. This is a very big advantage over an application that would hide the native mouse cursor or replace this cursor with another object with a fixed shape.

It is a very big advantage that this method works with an existing O / S with GUI (e.g. Windows 7 or Windows 10), and that no special O / S and GUI need be provided.

It is an advantage that this method does not require any special mouse control (driver).

It is an advantage that this method works with existing applications (such as word processors, worksheets, web browsers, etc.), and that these applications do not need to be modified in any way.

Given the huge amount of computer users, and the large amount of software companies, it is surprising that a fairly simple method can transform a computer device with an O / S and GUI developed for only a single mouse cursor into a computer device with an O / S and GUI with dual mouse support that can help make life easier for many professional users, such as translators, transcribers, reviewers, proofreaders, technical writers, teachers, etc., by offering a technical solution to regain the present context and switch again between two documents, or when switching between two different locations within the same document.

The idea of attaching multiple pointing devices to a single computer has been known for more than two decades, and many people have tried this in different ways.

BE2017 / 5896 problem. But only a few of them succeeded, and those who succeeded seem to have solved the problem in the device layer, not in the application layer. There seems to be a bias in the art that this problem must be solved in the device layer, and cannot be solved in the application layer, or at least that a good and satisfactory solution cannot be found in the application layer.

This method is particularly useful for professional computer users who work with both hands on a single computer device.

* Some embodiments of the method provide a mechanism that allows a user to easily switch the focus between two reading locations on one or more screens, which locations can be moved easily and intuitively * Some embodiments of the method provide a mechanism that makes it possible allows a user to easily switch the focus between a reading location and a work location on one or more screens, which locations can be moved simply and intuitively * Some embodiments of the method provide a mechanism that allows a user to easily switch the focus between two different locations on a screen, for reading and / or for editing, which locations can be moved simply and intuitively * Some embodiments of the method provide a mechanism that allows the user to work at two different screen locations , using both hands where reading means that the user should only be able to easily see and / or find and move this location by moving the pointing device (s), and where working means that the user should not only be able to move pointing device (s) but must also be able to click and / or scroll with one or both pointing device (s).

In other words, games (games) that enable the simultaneous use of two pointing devices are not the main purpose of the present invention, nor are network applications where several users each control a respective pointing device. The present invention is primarily intended for a single user using both hands, one hand typically being the dominant hand, and the other hand being the non-dominant hand.

This method can be implemented, for example, as a standalone overlay application that includes the graphical overlay window, and optionally further includes a user interface window, although the latter is not absolutely required.

BE2017 / 5896

It is known that some versions of the Windows operating system (Windows 10 or lower) allow multiple USB mice to be connected to the computer, e.g., two USB mice, if the computer device has at least two USB ports, but these Windows versions only provide a single mouse cursor. If multiple pointing devices are connected to the computer, the Windows O / S directs all movement events from any pointing device to the same mouse cursor. However, when combined with the method of the present invention, it appears to a user that it supports Windows O / S two pointing devices. The method of the present invention can thus be used as an extension or as an add-on to provide dual mouse functionality, not just for one specific application (such as a game application), but for most existing desktop applications.

It is an advantage that this overlay application only needs to adjust the position of the native mouse cursor of the O / S, but does not have to implement other functionality such as clicking, dragging, scrolling the mouse wheel, etc.

It is an advantage that switching between the two locations is done automatically merely by moving the corresponding pointing device, without having to take special actions. This makes switching very quick and very intuitive.

In one embodiment, the present invention provides a computer-implemented method for providing a first visible object, movable by a first pointing device, and a second visible object, movable separately by a second pointing device.

In this embodiment, the second visible object can be considered as the at least one other visible object.

In this embodiment, the first visible object is preferably moved substantially independently of movements of the second pointing device, apart from some transition effects. Similarly, the second visible object is preferably moved substantially independently of movements of the first pointing device, apart from some transient effects.

In one embodiment, the present invention provides a computer-implemented method for providing a first visible object and a second visible object and a third visible object, the third visible object being movable through a first pointing device; the second visible object movable separately by a second pointing device; the first visible object is selectively movable by one of the first pointing device and the second pointing device, which was most recently moved and / or operated; and wherein step b) comprises: providing a graphic overlay window that includes the second visible object and the third visible object.

BE2017 / 5896

With movements is meant, for example, horizontal and / or vertical movements of the pointing device. By operation is meant, for example, that a button of the pointing device is pressed or released, or that a scroll wheel of the pointing device is scrolled.

In one embodiment, step e) comprises: adjusting a position of the first visible object in accordance with movements and / or controls of one of the first pointing device and the second pointing device, which was most recently moved or operated; and adjusting a position of the second visible object in accordance with movement of the second pointing device; and adjusting a position of the third visible object in accordance with movements of the first pointing device.

In this embodiment, the first and third visible object are preferably moved substantially independently of movements of the second pointing device, apart from some transition effects. Likewise, the second visible object is preferably moved substantially independently of movements of the first pointing device, apart from some transition effects.

In one embodiment, the method further comprises step d) of: configuring the operating system to send raw input messages related to the first and second pointing devices, and providing a message processor to process these raw input messages; and includes step e): adjusting a position of the first visible object based on the raw input messages; and adjusting a position of the at least one other visible object based on the raw input messages.

Thanks to the message processor, the overlay window also receives the mouse events, allowing it to extract motion information from any mouse device. Although this approach has advantages, it also has disadvantages such as, for example, being unable to stop click events.

In one embodiment, the positions of the native mouse cursor as well as the second and optionally further visible object (s) within the message processing routine are updated.

This offers the advantage of a very low response time between movements of the pointing devices and the actual display on the screen, however, at the expense of a higher CPU load.

In one embodiment, only the position of the native mouse cursor in the message processing routine itself is updated, while the position of the second and / or further visible object is performed outside of the message processing routine, i.e. asynchronously, e.g., based on a timer interrupt.

In this way, the response time of movements of the native mouse cursor is very low, while the response time between movements of the pointing devices and the other visible

BE2017 / 5896 item (s) is somewhat larger. This offers the advantage of a reduced CPU load, which can be particularly advantageous if the second and / or the further visible object (s) occupies more than approximately 50 x 50 = 2500 pixels on the screen, or more than 70 x 70 = 4900 pixels, or more than 100 x 100 = 10000 pixels on the screen.

In one embodiment, step e) comprises:

* receiving an input message;

* if the received input message is related to the first pointing device, updating a first pointing position; and if the received input message is related to the second pointing device, updating a second pointing position;

* adjusting a position of the native mouse cursor and a position of the at least one other visible object in one of the following ways:

a) adjusting the position of the native mouse cursor in accordance with the first position, and adjusting the position of the at least one other visible object in accordance with the second position; or

β) adjusting the position of the native mouse cursor in accordance with the second position, and adjusting the position of the at least one other visible object in accordance with the first position; or

y) depending on whether the input message is related to the first pointing device or the second pointing device:

* adjusting the position of the native mouse cursor in accordance with the first position and adjusting a position of the at least one other visible object in accordance with the second position if the received input message is related to the first pointing device;

* adjusting the position of the native mouse cursor in accordance with the second position and adjusting a position of the at least one other visible object in accordance with the first position if the received input message is related to the second pointing device.

This embodiment described in more detail how input messages are treated to update a first position associated with the first visible object, and the second position associated with the second or third visible object. The input message can be a rough input message related to a motion event, or related to an event where a button is pressed, or an event where a button is released, or related to an event where a scroll wheel is turned.

BE2017 / 5896

In one embodiment, step e) comprises: receiving an input message, and in the case that the input message is a motion message, obtaining displacement values related to said motion message.

In one embodiment, the computer-implemented method further comprises: configuring the graphic overlay window as a semi-transparent window that has an alpha blending value (or alpha transparency value) in the range of 1% to 99% or from 2% to 98% or from 5 % to 95%.

In this embodiment, the second (and / or further) visible object is semi-transparent, meaning that the user can see through the object (e.g., a line or a cross or a rectangle or a hand), and thus the underlying information can (eg, textual or alphanumeric or graphical information represented by an underlying text application or spreadsheet application or PDF document viewer or web browser, etc.).

It is an advantage that the second (and / or further) visible object does not completely conceal underlying text or numerical information, which generally makes it easier for a user to read the underlying information, and does not require accurate movements of the pointing device, which usually harder for the non-dominant hand.

In one embodiment, only a minority portion of the area of the overlay window is occupied by the second visible object, and a majority portion of the area of the overlay window is occupied by fully transparent pixels.

For example, the second visible object may occupy less than 20% or less than 10% or less than 5% of the area of the overlay window.

The second visible object can be or contain an opaque image.

The second visible object may contain a bitmap with a first plurality of pixels that are opaque and a second plurality of pixels that are completely transparent, the first and second plurality of pixels being alternately positioned (interleaved), e.g. according to a regular pattern, e.g. according to a checkerboard pattern (as described in more detail in the pending perforated bitmap application, see e.g. FIG. 37 and FIG. 39 thereof).

A majority portion of the pixels of the overlay window, for example at least 70% of these pixels, can be completely transparent pixels. In an embodiment where the overlay window comprises only the second visible object (and no further visible objects), preferably all pixels that do not belong to the second visible object are completely transparent pixels.

In one embodiment, the method is implemented in an overlay application, and the overlay application has an input mode selected from or selectable from a first input mode

BE2017 / 5896 (e.g. Right Handed) and / or a second input mode (e.g. Left Handed) and / or a third input mode (e.g. Two Handed); and step e) includes one or two or all of the following options:

* in the case that the overlay application is configured in the first input mode (e.g. Right Hand) adjusting a position of the native mouse cursor in accordance with the first position;

* in the case that the overlay application is configured in the second input mode (e.g. Left Handed) adjusting a position of the native mouse cursor in accordance with the second position;

* in the case that the overlay application is configured in the third input mode (e.g. Two-Handed) selectively adjusting a position of the native mouse cursor in accordance with the first position if the received input message is related to a motion message from the first pointing device, and adjusting said position in accordance with the second position if the received input message is related to a motion message from the second pointing device.

In some embodiments, the input mode may be predetermined, for example, hard-coded for right-handed people or for left-handed people.

In some embodiments, the input mode may be configurable, for example selectable by a user, explicitly (see e.g. FIG. 31) or implicitly (see e.g. FIG. 28 (a) and FIG. 30).

The first input mode may be particularly suitable for legal users. Expressed in simple terms, the native mouse cursor in this mode follows movements of the first pointing device, e.g., the pointing device designated by the user as the right pointing device, usually controlled by the right hand. In this mode, the native mouse cursor does not jump between the first and second position.

The second input mode may be particularly suitable for left-handed users. Expressed in simple terms, the native mouse cursor in this mode follows movements of the second pointing device, e.g., the pointing device designated by the user as the left pointing device, usually controlled by the left hand. In this mode, the native mouse cursor does not jump between the first and second position.

The third input mode may be particularly suitable for users who wish to click and / or scroll with both pointing devices. Expressed in simple terms, the native mouse cursor in this mode follows movements of the pointing device that was most recently moved and / or clicked and / or scrolled. In this mode, the native mouse cursor jumps between the first and second position.

In one embodiment, the overlay application has a display mode selected from or selectable from a first group of display modes associated with the first input mode (e.g., Right Handed), and / or a second group of display modes associated with the second input mode (e.g., Left Handed), and a third group of display modes associated with the third input mode (e.g., Two-Handed); and includes step e) performing at least one of the following steps:

BE2017 / 5896 * in the case that the overlay application is configured in a display mode selected from the first or third group of display modes, adjusting a position of the second visible object in accordance with the second position;

* in the case that the overlay application is configured in a display mode selected from the second group of display modes, adjusting a position of the second visible object in accordance with the first position.

The first group of display modes may include, for example, a mode with only two visible objects: the native mouse cursor as the first visible object that is movable through the right pointing device, and a second visible object that is movable through the left pointing device, and that e.g. has the shape of a black arrow (see icon 2860a of FIG. 28) or an image of a left hand (see icon 2860b) or a square (see icon 2860c) or a line (see icon 2860d) or a cross (see icon 2860e ), but other modes have also been considered.

The second group of display modes may include, for example, a mode with only two visible objects: the native mouse cursor as the first visible object movable through the left pointing device, and a second visible object movable through the right pointing device, and which, for example, is in the form of a black arrow (see icon 2850a of FIG. 28) or an image of a right hand (see icon 2850b) or a square (see icon 2850c) or a line (see icon 2850d) or a cross (see icon 2850e), but others modes are also considered.

The third group of display modes may include, for example, a mode with only two visible objects: the native mouse cursor as the first visible object, and a second visible object that is, for example, in the form of a black arrow (see icon 2870a of FIG. 28) wherein the first visible object is selectively movable by the right pointing device or the left pointing device, depending on which is being moved, and the second visible object is in the other maintained position.

In one embodiment, step b) comprises: providing a third visible object contained in the graphic overlay window; and step e) comprises performing the following steps at least once or repeatedly: * adjusting a position of the second visible object in accordance with the second position, and adjusting a position of the third visible object in accordance with the first position.

The first group, second group, and third group of display modes can include, for example, a mode with three visible objects: the native mouse cursor as the first visible object, and a second and third visible object each having the shape of a square (see icons 2860g, 2850g, 2870g) or a line (see pictograms 2860h, 2850h, 2870h) or a cross (see pictograms 2860Î, 2850Î, 2870Î), one of which is movable by the left pointing device, the other by the right pointing device.

BE2017 / 5896

The third group of display modes may include, for example, a mode with three visible objects: the native mouse cursor as the first visible object selectively movable by the right or left pointing device, depending on which pointing device was most recently moved and / or clicked and / or scrolled and a second visible object (e.g., having an image of a left hand) movable through the left pointing device, and a third visible object (e.g., having an image of a right hand) movable through the right pointing device (see icon 2870b), but other modes are also considered.

In one embodiment, step b) further comprises: providing a vertical visible line contained in the graphic overlay window; and includes step c): repeatedly performing the following steps: checking whether a position of the native mouse cursor is on the vertical visible line, and if a result of the test is false, configuring the graphic overlay window in click-through mode; and if a result of the test is true, configuring the graphic overlay window in non-click-through mode.

The vertical visible line makes it possible to split the screen into two parts: a left part and a right part. By repeatedly checking where the native mouse cursor is located, and by dynamically configuring the overlay window in click-through or non-click-through mode, dragging the vertical line can be implemented, which is a very convenient and intuitive way of moving the vertical line. No applications are known where a window is dynamically configured in and out of click-through mode, depending on a position of the native mouse cursor.

The second visible object can be a horizontal line that extends across the width of the left portion of the screen, or the right portion of the screen.

If both a second and a third visible object are available, the second object can be a left horizontal line, and the third object can be a right horizontal line. The vertical line can be conveniently used to define the width of a left horizontal line and the right horizontal line.

In one embodiment, the graphic overlay window occupies at least 80% or at least 90% of the area occupied by the graphical user interface (GUI).

In one embodiment, the graphic overlay window occupies substantially the entire screen area, e.g., the entire screen area.

In one embodiment, the graphic overlay window occupies substantially the entire work area, e.g., the entire work area, which in a Microsoft Windows context may mean the area of the desktop minus the area of a taskbar.

In one embodiment, the second visible object and / or any further visible object each occupies 1000 to 20000 pixels, or 1500 to 15000 pixels, or 2000 to 12000 pixels, or 2500 to

10000.

BE2017 / 5896

It is an advantage that only a small portion of the screen can be updated rather quickly, and has no significant impact on CPU performance. At the same time, it is an advantage to display a fairly large visible object, because such an object is much easier to find on the screen. As an example, the native mouse cursor (in the form of an arrow) typically occupies only about 10 x 10 or about 100 pixels, but is hardly visible, and hard to find on a screen. Showing a bitmap of a hand that has about 20 x 20 pixels or about 400 pixels is better, but still quite small, especially on a screen with a resolution of about 1920 x 1080 pixels or more. On such a screen of this resolution, it is much easier to locate the visible object (s) if they are approximately 30x30 or 50x50 to approximately 100x100 pixels, i.e. approximately 900 pixels or approximately 2.5K to approximately 10K pixels.

It is a major advantage of embodiments showing three visible objects: a first visible object being the native mouse cursor with a size of about 100 to 400 pixels (e.g., an arrow of about 10 x 10 pixels, or an arrow of about 16 x 25 pixels) because this can be used for accurate positioning, and at the same time a second and third object, one of which is substantially in the same position as the native mouse cursor, optionally with a constant shift, each with much larger size, e.g., each occupying approximately 1.5K to 15K pixels (e.g., a hand of approximately 34 x 44 pixels, or approximately 68 x 88 pixels, or approximately 102 x 132 pixels). In particular, the combination of a fairly large hand and a fairly small native mouse cursor is extremely useful because it offers the best of both worlds: good visibility and accurate positioning.

In one embodiment, the first pointing device comprises at least one button that can be pressed or released by a user, and the second pointing device comprises at least one button that can be pressed or released by a user, and comprises step e):

* testing whether the received input message is related to a motion message from the first pointing device, and if a result of this test is true, obtaining displacement information and updating a first pointing position; and * testing whether the received input message is related to a motion message from the second pointing device, and if a result of this test is true, obtaining displacement information and updating a second pointing position; and * testing whether the received input message is related to pressing or releasing the at least one button of the first pointing device, and if a result of this test is true, setting a first button state variable to TRUE or FALSE respectively, and * testing whether the received input message is related to pressing or releasing the at least one button of the second pointing device, and if a result of this test is true, setting a second button state variable to TRUE or FALSE respectively;

BE2017 / 5896 and wherein step e) further comprises:

* in the case that the overlay application is configured in the first input mode (e.g. Right Hand), adjusting a position of the native mouse cursor in accordance with the first position;

* in the case that the overlay application is configured in the second input mode (e.g. Left Handed), adjusting a position of the native mouse cursor in accordance with the second position;

* in the case that the overlay application is configured in the third input mode (e.g. Two-Handed) and if the received input message is related to a movement of the first pointing device and if the second button state variable is FALSE, adjusting a position of the native mouse cursor in accordance with the first position, and otherwise ignoring the input message;

* in the case that the overlay application is configured in the third input mode (e.g. Two-Handed) and if the received input message is related to a movement of the second pointing device and if the first button state variable is FALSE, adjusting a position of the native mouse cursor in accordance with the second position, and otherwise ignoring the input message.

This behavior is expressed in pseudo code in FIG. 18 (a). This algorithm is an improvement on the first algorithm described above. Expressed in simple terms for two mouse devices: the method forces the native mouse cursor to jump according to the most recent mouse movement (left mouse or right mouse), except when a button of the other mouse device is pressed. The attentive reader will note that this algorithm permits dragging, using the pointing device whose button is pressed.

It is noted that simply ignoring the motion event is not a good solution, because the O / S with GUI will move the mouse cursor anyway. The method of the present invention corrects this behavior to make the behavior intuitive for the user. For example, if a user presses a button on the right mouse device, the windows cursor should not move even if the left mouse device is moved as long as the right mouse button is pressed. Neither this problem, and consequently neither the requirement nor the solution is known in the art.

In one embodiment, step e) comprises: under predetermined conditions, transforming the displacement values to correspond to a displacement in a direction selected from a limited group of directions, e.g. containing fewer than nine directions, e.g. containing only four directions (North, West, South, East), or containing only eight directions (N, W, Z, O, NW, SW, SE, NO), or containing only two horizontal directions (West, East).

BE2017 / 5896

It is an important advantage of this feature that it can drastically help the user to better control movements of at least one of the pointing devices when performing certain tasks. This is particularly useful for the pointing device controlled by the non-dominant hand when performing text-based tasks (such as reading a text document, proofreading a translation, pointing to individual words on the same line during a Webinar or presentation, etc.)

In one embodiment, step e) comprises: under predetermined conditions, transforming the displacement values to correspond to a horizontal displacement or a vertical displacement.

In one embodiment, step e) comprises: under predetermined conditions, transforming the displacement values to correspond to a horizontal displacement or a vertical displacement, or a displacement at a predetermined angle in the range of 30 ° to 60 °, e.g. , equal to approximately 45 °.

In one embodiment, step e) comprises: testing whether an angle or an angular distance between a displacement vector defined by the displacement values and a horizontal line is less than a predetermined angle (e.g., less than 30 °, or less than 40 °) and if a result of this test is true, transforming the displacement values into horizontal displacement values; or testing whether a ratio of the vertical displacement value and the horizontal displacement value is less than a predetermined value, and if a result of this test is true, transforming the displacement values into horizontal displacement values.

The predetermined value can be a constant value selected in the range of 0.2 to 0.75, for example equal to about 0.20 = 1/5 or 0.25 = 1/4 or 0.33 = 1/3 or 0.50 = 1/2 or 0.66 = 2/3 or 0.75 = 3/4.

Transforming the displacement values into horizontal displacement values may mean, for example, setting the vertical displacement value dy to 0.

In one embodiment, the first and second pointing devices are HID-compatible devices, and step d) comprises: configuring the operating system to send raw input data to the overlay application.

More specifically (in the case of a Microsoft Windows environment), step d) may include configuring the application to receive raw input messages of the WM_INPUT type using the WinAPI function RegisterRawinputDevices.

The term HID is a well-known term and stands for the Human Interface Device.

In one embodiment, the operating system is Windows 7 or Windows 10 from Microsoft Corporation.

BE2017 / 5896

Methods of the present invention are particularly suitable for use in conjunction with the Microsoft Windows operating system.

In one embodiment, the operating system is mac OS Sierra or mac OS High Sierra or a later mac OS version of Apple inc.

Methods of the present invention can also be advantageously used in conjunction with operating systems for a mac computer.

The present invention is also directed to the use of a method according to the first aspect, by a single user, for pointing to two different locations on a single screen, or on multiple screens connected to a single computer device, using two pointing devices.

The present invention is also directed to the use of this method by a single user, for underlining alphanumeric information at two different locations on one or more screens connected to a single computer device, using two pointing devices.

The present invention is also directed to the use of this method by a single user, for operating at two different locations on one or more screens connected to a single computer device, using two pointing devices, both of which can be clicked and / or scrolled.

Such use is particularly suitable for professional users, specifically translators, proofreaders, technical writers, lawyers, patent attorneys, transcribers, and teachers.

In a 2nd aspect, the present invention provides a computer program product that includes executable instructions that perform the method of the first aspect when executed on a computer device that has a display screen or is connected to it, and that has a first pointing device or is connected to it, and which has a second pointing device or is connected thereto, the computer device comprising an operating system with a graphical user interface that provides the native mouse cursor.

The O / S with GUI can be Windows 7 or Windows 10 or later, from Microsoft Corporation, or can be mac OS Sierra or mac OS High Sierra or a later mac OS version of Apple Inc. to be.

According to a 3rd aspect, the present invention provides a computer device comprising: an operating system with a graphical user interface that provides a native mouse cursor, and a computer program product according to the second aspect.

The computer device may be a personal computer (PC) or a desktop computer or a laptop computer or a server computer or the like.

The computer device may have at least one processor, and a storage medium or memory that stores an overlay application for performing the overlay method described above.

BE2017 / 5896

The computer device may include two pointing devices, or may include one pointing device (e.g., a touch pad) and be connected to one pointing device (e.g., a mouse device), or may be connected to two pointing devices (e.g., two mouse devices).

The computer device may have an internal display screen or may be connectable to an external display screen.

The computer can have an integrated keyboard or can be connected to an external keyboard.

The computer device may further comprise one or more applications selected from the group consisting of: a word processor, a PDF document viewer, a PDF document editor, a web browser, a spreadsheet application, a PowerPoint application, an audio player, a video player, etc.

The computer program product enhances the functionality of the control system to provide support for two visible objects, individually movable by two pointing devices.

According to a 4th aspect, the present invention provides a pointing device comprising: a motion detection mechanism for detecting a motion, e.g., a two-dimensional motion; a control unit connected to the motion detection mechanism and adapted to send a motion message when a motion is detected; at least one button depressable and release by a user; and a button detection mechanism for detecting whether the at least one button is pressed or released; wherein the control unit is connected to said button detection mechanism and adapted to send a button-pressed message when it is detected that the at least one button is being pressed, and to send a push-button release message when it is detected that the at least one at least one button is released; and wherein the control unit is adapted for, when it is detected that the at least one button is pressed, to send (first) a dummy motion message, and to (subsequently) send the button-pressed message a predetermined period after the sending the dummy motion message.

This pointing device is particularly suitable for use in combination with a method according to the first aspect, or in combination with a computer program product according to the second aspect, or in combination with a computer device according to the third aspect.

The control unit may comprise a programmable processor, programmed for reading the motion detection mechanism and / or the button detection mechanism and / or a scroll wheel mechanism (if present), and for sending a corresponding message to the computer device (host).

It is an advantage of this pointing device that the dummy message will cause the overlay application described above to make the native mouse cursor jump if it is

BE2017 / 5896 was at the position associated with the other pointing device (which was not clicked), thereby reducing the risk of the native mouse cursor being in the wrong position at the time the actual button-pressed message at the computer device arrives, as shown in FIG. 22. Or in other words, it is an advantage of this pointing device that the risk that the native mouse position does not match the position of the mouse device being clicked is drastically reduced. It is noted that this problem does not exist in the prior art because the operating system directs all pointer messages to the same cursor position.

Or in other words, it is an advantage of this pointing device that the user does not have to move the pointing device manually before clicking, but that this is done automatically.

This pointing device is preferably retro compatible, such that it can also be used in conventional computer devices that do not perform the overlay application according to the first aspect, albeit with a slight delay when clicking a button, which is considered to be perfectly acceptable. for desktop applications, but is probably unacceptable for gaming applications.

The delay can be based on a timer, an internal clock signal, an external clock signal, etc.

In one embodiment, the processor is further adapted for, when it is detected that the at least one button is released, to send (first) a dummy movement message, and to (subsequently) send the push-button released message for a predetermined period after sending the dummy motion message.

This pointing device is particularly useful in combination with a position updating routine as shown in FIG. 18 (b), wherein motion messages from one pointing device are processed even if a button of the other pointing device is pressed.

In one embodiment, the pointing device further comprises at least one scroll wheel rotatable by a user; and a scroll wheel detection mechanism for detecting whether the at least one scroll wheel has turned; and wherein the control unit is connected to the scroll wheel detection mechanism, and adapted to send a scroll wheel rotation message when it is detected that it has turned at least one scroll wheel; and wherein the control unit is adapted for, when it is detected that it has rotated at least one scroll wheel, to send a dummy motion message, and to send the scroll wheel rotation message a predetermined period after sending the dummy motion message .

This pointing device is particularly suitable for use in combination with a method according to the first aspect, or in combination with a computer program product according to the second aspect, or in combination with a computer device according to the third aspect.

BE2017 / 5896

In practice, the scroll wheel rotation message would typically include a value that is characteristic of how much the scroll wheel has turned.

In a similar manner as for the button-pressed message, it is an advantage that instead of immediately sending the scroll wheel rotation message to the host, this pointing device first sends a dummy motion message and then waits a certain time so that the The overlay application according to the first aspect has time to force the native mouse cursor to jump if it was at the position associated with the other pointing device. Again, this drastically reduces the risk of the wheel rotation command being interpreted at the incorrect position.

In one embodiment, the predetermined period is a value in the range of 1 to 250 ms or in the range of 1 to 100 ms; in combination with the following options:

* wherein the dummy movement message has a horizontal movement value in the range of -5 to +5 and a vertical movement value in the range of -5 to +5; or * wherein the dummy movement message has a horizontal movement value in the range of -1 to +1 and a vertical movement value in the range of -1 to +1, or * wherein the dummy movement message has a horizontal movement value equal to zero and a vertical displacement value equal to zero.

It is an advantage of using a rather small displacement value, in order not to move the native mouse cursor much. The smaller the displacement, the better.

The predetermined period is a period in the range of 1 ms to 250 ms, or in the range of 5 ms to 200 ms, and may be, for example, equal to about 8 ms, or about 16 ms, or about 24 ms, or about 32 ms, or about 40 ms, or about 48 ms, or about 56 ms, or about 64 ms, or about 72 ms, or about 80 ms, or about 88 ms, or about 96 ms, or about 104 ms, or about 112 ms, or about 120 ms, or about 128 ms, or about 136 ms, or about 144 ms, or about 152 ms, or about 160 ms, or about 168 ms, or about 176 ms, or about 184 ms, or about 192 ms, or about 200 ms, or about 208 ms, or about 216 ms, or about 224 ms, or about 232 ms, or about 240 ms, or about 248 ms. About 8 ms is a usual time between two HID messages, so a delay of about 8 ms would mean that the dummy motion message and the button-pressed message are sent in two adjacent time slots.

In one embodiment, the control unit is configured to send a subsequent dummy motion message after a subsequent detection of pressing the at least one button, or after a subsequent detection of turning the at least one scroll wheel, the dummy motion message displacement values includes chosen such that a cumulative horizontal displacement and a cumulative vertical displacement is a value in the range of -5 to +5, or in the range of -1 to +1.

BE2017 / 5896

Simply put, instead of sending the same dummy motion message every time the at least one button is clicked, for example with (dx = + 1 and dy = 0), it is better to send another dummy motion message the next time that the button is clicked, for example with displacement values dx, dy in the opposite direction, for example with (dx = -l and dy = 0).

It is an advantage of this embodiment that the pointer position does not float away, even if the button is clicked several times.

The same effect can also be obtained by not sending dummy messages back and forth with a dummy displacement, but by sending dummy messages with a dummy displacement that form a small loop.

In one embodiment, the pointing device further comprises an RF transceiver connected to the control unit, and the control unit is further configured to send messages to a computer device and to receive messages from the computer device via said RF transceiver, optionally via a dongle connected to the computer device.

This embodiment is directed to a wireless pointing device, such as, for example, a wireless mouse.

The messages can be received by an internal RF transceiver present within the computer device, or via an external RF transceiver, also known as a dongle. The dongle can have a male USB connector to be connected into a female USB slot of a computer device.

In one embodiment, the pointing device is a USB HID class device or is compatible with USB HID class devices.

According to a 5th aspect, the present invention also provides a computer system comprising a computer device according to the third aspect; and a first and a second pointing device according to the fourth aspect; and optionally also a display device; and optionally also a keyboard.

In one embodiment of this computer system, the first and second pointing devices are USB HID class devices or are compatible with USB HID class devices; and the computer device comprises at least a first USB port for connection to the first pointing device, and a second USB port for connection to the second pointing device.

According to a 6th aspect, the present invention also provides a set of components comprising: a computer program product according to the second aspect; and two indicating devices according to the fourth aspect.

This set of parts (or package) can be very suitable for users who already have a computer device with an operating system and a keyboard and a display screen.

BE2017 / 5896

By using the computer program product of the second aspect in combination with two special pointing devices, the risk of accidental clicking or scrolling in the wrong position is greatly reduced, without the user having to physically move the pointing device before clicking or scrolling.

According to a 7th aspect, the present invention provides a hub, comprising: a first interface to communicate with a first pointing device, and a second interface to communicate with a second pointing device, and a third interface to communicate with a computer device; and a control unit connected to the first interface and adapted to receive first messages from the first pointing device; wherein the control unit is connected to the second interface and adapted to receive second messages from the second pointing device; wherein the control unit is connected to the third interface and adapted to send the first and second messages to the computer device;

and wherein the control unit is further adapted for:

* upon receipt of a button-pressed message from the first pointing device via the first interface, to (first) send a first dummy movement message to the computer device via the third interface, and to press the button-pressed a predetermined period later; send message to the computer device via the third interface;

* upon receipt of a button-pressed message from the second pointing device via the second interface, to send a second dummy motion message to the computer device via the third interface, and to send the button-pressed message later a predetermined period to the computer device via the third interface.

Or in other words, in this way the hub prevents the message stream being sent to the computer device from containing a button-pressed message or a scroll wheel rotation message from the first pointing device preceded by a message from the second pointing device, and vice versa.

This hub is particularly suitable for use in combination with a method according to the first aspect, or in combination with a computer program product of the second aspect, or in combination with a computer device according to the third aspect, although the hub can also be used with other applications with dual mouse support, for example educational games or the like, not configured in click-through mode.

By first / second dummy motion message is meant a dummy motion message formatted as if it was sent by the first / second pointing device.

The use of this hub offers at least the same advantages as the use of the special pointing devices according to the fourth aspect, and may in some embodiments even further

BE2017 / 5896 offer advantages, because the hub can see the messages from other pointing devices (and optionally block or delay or change).

In one embodiment, the control unit is further adapted for:

* upon receipt of a push button release message from the first pointing device via the first interface, to send a first dummy motion message to the computer device via the third interface, and to send the push button release message a predetermined period later to the computer device via the third interface;

* upon receipt of a push button release message from the second pointing device via the second interface, to send a second dummy motion message to the computer device via the third interface, and to send the push button release message a predetermined period later to the computer device via the third interface.

This hub is particularly advantageous when used in conjunction with an overlay method that has a position updating routine (as shown, for example, in FIG. 18b) in which motion messages from one pointing device are not simply ignored, but are processed even if a button of the other pointing device is pressed .

In one embodiment, the control unit is further adapted for:

* upon receipt of a scroll wheel rotation message from the first pointing device via the first interface, to send a first dummy movement message to the computer device via the third interface, and to send the scroll wheel rotation message a predetermined period later to the computer device via the third interface;

* upon receipt of a scroll wheel rotation message from the second pointing device via the second interface, to send a second dummy movement message to the computer device via the third interface, and to send the scroll wheel rotation message a predetermined period later to the computer device via the third interface.

In one embodiment the control unit is adapted for:

* upon receiving a first button-pressed message, determining whether a most recent motion-related or button-related or scroll-wheel message sent to the computer device via the third interface was a message related to the first pointing device, and if the result of this test is true, sending the first button-pressed message to the computer device via the third interface without prior transmission of a dummy movement message, and if the result of this test is false, sending a first dummy movement message to the computer device through the third interface, and a predetermined period of time sending the first button-pressed message to the computer device via the third interface;

* upon receipt of a second button-pressed message, determining whether a most recent motion-related or button-related or scroll wheel message sent to the computer device

BE2017 / 5896 via the third interface was a message related to the second pointing device, and if the result of this test is true, sending the second button-pressed message to the computer device via the third interface without prior transmission of a dummy movement message, and if the result of this test is false, sending a second dummy motion message to the computer device via the third interface, and a predetermined period later sending the second button-pressed message to the computer device through the third interface;

* upon receipt of a first scroll wheel rotation message, determining whether a most recent motion related or button related or scroll wheel message sent to the computer device via the third interface was a message related to the first pointing device, and if the result of this test is true, sending the first scroll wheel rotation message to the computer device via the third interface without prior transmission of a dummy motion message, and if the result of this test is false, sending a first dummy motion message to the computer device through the third interface, and a predetermined period of time sending the first scroll wheel rotation message to the computer device via the third interface;

* upon receipt of a second scroll wheel rotation message, determining whether a most recent motion-related or button-related or scroll-wheel message sent to the computer device via the third interface was a message related to the second pointing device, and if the result of this test is true, sending the second scroll wheel rotation message to the computer device via the third interface without prior transmission of a dummy movement message, and if the result of this test is false, sending a second dummy movement message to the computer device through the third interface, and a predetermined period of time later sending the second scroll wheel rotation message to the computer device via the third interface.

Expressed in simple terms, when a button-pressed message arrives, the hub verifies whether the most recent message sent to the computer device came from the same pointing device, and if so, the button-pressed message can be sent without first send a dummy motion message, otherwise the hub first sends a dummy motion message and sends the button-pressed message some time later.

This can be seen as an improved version of the hub described above, because a dummy motion message is not always sent unconditionally, but is only sent if necessary.

In one embodiment, the control unit further maintains a first button state variable as an indication of whether a button of the first pointing device is depressed, and a

BE2017 / 5896 second button state variable as an indication whether a button of the second pointing device is pressed; and the control unit is further configured for:

* upon receipt of a first motion message from the first pointing device, testing whether the second button state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the first motion message, and if the result of this test is FALSE, sending the first motion message to the computer device via the third interface;

* upon receipt of a second motion message from the second pointing device, testing whether the first button state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the second motion message, and if the result of this test is FALSE, sending the second motion message to the computer device via the third interface;

* upon receipt of a first scroll wheel rotation message from the first pointing device, testing whether the second button state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the first scroll wheel rotation message, and if the result of this test is FALSE, sending the first scroll wheel rotation message to the computer device via the third interface;

* upon receipt of a second scroll wheel rotation message from the second pointing device, testing whether the first button state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the second scroll wheel rotation message, and if the result of this test is FALSE, sending the second scroll wheel rotation message to the computer device via the third interface;

* upon receipt of a first button-pressed message from the first pointing device, testing whether the second button state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the first button-pressed message, and if the result of this test is FALSE, perform the following steps:

- setting the first button state variable to TRUE,

- sending a first dummy movement message to the computer device via the third interface,

and a predetermined period later, sending the first button-pressed message to the computer device via the third interface;

* upon receiving a first push-release message from the first pointing device, testing whether the second button-state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the first push-release message, and if the result of this test is FALSE, perform the following step:

- setting the first button state variable to FALSE,

- sending the first push-release message to the computer device via the third interface;

η

172017/5896 * upon receipt of a second button-pressed message from the second pointing device, testing whether the first button-state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the second button-pressed message , and if the result of this test is FALSE, perform the following steps:

- setting the second button state variable to TRUE,

- sending a second dummy movement message to the computer device via the third interface,

and a predetermined period later, sending the second button-pressed message to the computer device via the third interface;

* upon receipt of a second push button release message from the second pointing device, testing whether the first button state variable is TRUE, and if the result of this test is TRUE, ignoring or delaying the second push button release message, and if the result of this test is FALSE, perform the following step:

- setting the second button state variable to FALSE;

- sending the second push button release message to the computer device via the third interface.

Expressed in simple terms, this hub keeps track of whether the at least one button of the first and second pointing devices is pressed, and if so, ignoring messages sent by the other pointing device, such as shown in FIG. 26 and FIG. 27.

In one embodiment, the predetermined period is a value in the range of 1 to 250 ms or in the range of 2 to 100 ms; and the dummy movement message has a horizontal movement value in the range of -5 to +5 and a vertical movement value in the range of -5 to +5; or the dummy movement message has a horizontal movement value in the range of -1 to +1 and a vertical movement value in the range of -1 to +1; or the dummy movement message has a horizontal movement value equal to zero and has a vertical movement value equal to zero.

This offers the same advantages as a pointing device according to the fourth aspect which has the same or similar characteristics.

In one embodiment, the control unit is configured to send subsequent first dummy motion messages with displacement values selected such that a first cumulative horizontal displacement and a first cumulative vertical displacement are values in the range of -5 to +5, or in the range of - 4 to +4, or in the range of -3 to +3, or in the range of -2 to +2, or in the range of -1 to +1; and the control unit is configured to send subsequent second dummy motion messages with displacement values selected such that a second cumulative horizontal displacement and a second cumulative vertical displacement

BE2017 / 5896 displacement values are in the range of -5 to +5, or in the range of -4 to +4, or in the range of 3 to +3, or in the range of -2 to +2, or in the range from -1 to +1.

This offers the same advantages as a pointing device according to the fourth aspect which has the same or similar characteristics.

In one embodiment, the first interface comprises a first connector connectable to the first pointing device, and the second interface comprises a second connector connectable to the second pointing device, and the third interface comprises a third connector connectable to the computer device.

In a preferred embodiment, the first and second connectors are female USB connectors, and the third connector is a male USB connector.

In one embodiment, the hub further comprises a first RF transceiver for receiving messages from and sending messages to the first pointing device; and the hub further comprises a second RF transceiver for receiving messages from and transmitting messages to the second pointing device, wherein the second RF transceiver is the same as the first RF transceiver, or is a separate RF transceiver; and the first interface includes the first RF transceiver; and the second interface includes the second RF transceiver; and the third interface comprises a connector connectable to the computer device.

In one embodiment, the first and second transceiver are a single RF transceiver, capable of communicating with both the first and the second pointing device, and the third interface is a male USB connector.

In one embodiment, the first and second transceivers are two physically separate RF transceivers, and the third interface is a male USB connector.

Such a hub is typically called a dongle.

It is noted that in this case the first and second pointing devices also need an RF transceiver, but they do not need the special behavior described above. They can be conventional RF pointing devices.

According to an 8th aspect, the present invention provides a computer system comprising: a computer device according to the third aspect; and two pointing devices; and a hub according to the seventh aspect, wherein the hub is operatively connected between the computer device and the first and second pointing device; and optionally also a display device; and optionally also a keyboard.

With this computer system, the user can use both hands to move two visible objects.

This computer system provides a technical solution to a technical problem, which can be used to significantly reduce the cognitive burden of the user, and to speed up

BE2017 / 5896 between different locations on a screen, which is particularly relevant for translators or reviewers or proofreaders who often switch back and forth between two documents.

According to a 9th aspect, the present invention also provides a set of components comprising: a computer program product according to the second aspect, and a hub according to the seventh aspect.

This set of components (or package) can also be very suitable for users who already have a computer device with a control system and a keyboard and a display screen and two mouse devices. By using the computer program product of the second aspect in combination with the special hub, the risk of accidental clicking or scrolling in the wrong position is greatly reduced or even completely eliminated, without the user having to physically move the pointing device before clicking or clicking. scroll.

In one embodiment, the set of parts may further comprise one or two indicating devices.

This set of parts (or package) may be better suited for users who already have a computer device with an operating system and a keyboard and a display screen and a single mouse device.

According to a 10th aspect, the present invention provides a display device comprising: an input port for receiving first graphic image data from a computer device; an input buffer for storing at least a portion of the first graphic image data received at the input port; a processor connected to the input buffer, and adapted to generate second graphic image data based on the first graphic image data, and to store at least a portion of the second graphic image data in a frame buffer; wherein the frame buffer is operatively connected to the processor, and adapted to store at least a portion of second graphic image data to be displayed on a display panel; a readout circuit for reading at least a portion of the second graphic image data from the frame buffer, and for sending the at least a portion to a display panel; the display panel for generating a visible image corresponding to the second graphic image data; wherein the display device further comprises a wired and / or a wireless interface for communicating with a wired or wireless pointing device; wherein the processor (of the display device) is further adapted for: * communicating with the pointing device to receive displacement information related to movements of the pointing device; and * update a position based on the received movement information; and * generating a visible object; and * generating the second graphic image data by overlaying the first graphic image data with the generated visible object at said position.

BE2017 / 5896

In practice, the input buffer will typically store only a fraction of a video frame contained in the first graphic image data (stream) at any time in time, and the frame buffer will typically store at least one entire field at any time in time (e.g., in an interlaced video display screen, such as a CRT display screen) or at least one whole frame (e.g., in a non-interlaced video display screen, such as an LCD display screen).

In some embodiments, the frame buffer may store at least two entire fields or at least two entire frames to allow the processor 4233 to prepare the next field or frame to be displayed.

The pointing device may be an HID-compatible pointing device, e.g., an HID-compatible mouse device.

The input buffer and the frame buffer may be part of a memory component contained in the display device, although they are shown as separate blocks in FIG. 42.

This display device has the advantage that it not only displays the graphic image provided by the computer device, but also provides at least one visible object, for example with the shape or shape of an arrow, or a hand, or a small line, or a large line, or a square, or a rectangle, or a small cross, or a large cross, or any other suitable shape or shape, which at least one visible object is movable by a pointing device operatively connected to the display device in place of with the computer device.

As can be understood from FIG. 41 in comparison with FIG. 5 (a) and FIG. 5 (b), the use of this special display device in conjunction with a conventional computer device also solves the problem of FIG. 1 to FIG. 3 issue shown.

Although retrospectively the solution presented in FIG. 41 and FIG. 42 may seem easy to implement, does not mean that this embodiment is an obvious solution to the problem of FIG. 1 to FIG. 3 to solve. It is pointed out in this respect that there is a widespread belief that a software solution is always cheaper than a hardware solution, especially in consumer electronics, and therefore that the problem must be solved in the computer device, rather than in the display device.

The solution presented in FIG. 41 and FIG. 42 elegantly solves the problem, moreover without significantly increasing the CPU load of the computer device, and without requiring the special overlay application.

The visible object may be a predetermined visible object, or may have a bitmap obtained from a removable memory, for example from a memory stick with a file comprising said bitmap. In such an embodiment, the display device may have a further connector for inserting a non-volatile memory card or a flash card or a memory stick or the like.

BE2017 / 5896

In one embodiment, the interface for communicating with the pointing device comprises a physical connector, in particular a USB connector.

A physical connector allows direct connection of the pointing device to the display device.

In one embodiment, the interface for communicating with a pointing device comprises a wireless RF transceiver, e.g., a Bluetooth transceiver. The RF receiver can be integrated in the display device, or can be externally connected to the display device, for example in the form of a so-called dongle, e.g. a USB dongle. The dongle can be connected to the physical connector of the display device.

In one embodiment, the generated visible object comprises or consists of monochrome pixels, and the processor is adapted to overlay the first graphic image data with the monochrome pixels by replacing pixel values of the first graphic image data with a value of the monochrome pixels.

It is an advantage of this embodiment that the overlay operation can be as simple as replacing pixel values of the first graphic image with pixel values of the visible object. The result is a non-transparent, visible movable object overlaid over the first graphic image data.

In one embodiment, the generated visible object comprises or consists of monochrome pixels, and the processor is adapted to overlay the first graphic image data with the monochrome pixels by alpha gene, for example using an alpha transparency in the range of 5% to 95% or of 25% to 75%.

It is an advantage of this embodiment that the overlay alpha gene processing involves a mature technique, and although technically somewhat more complex, the result is a semi-transparent visible movable object overlaid over the first graphic image data. This offers the user the advantage that he can see through the movable object, and that less accurate positioning is required, for example, when underlining alphanumeric information, which is particularly useful since this pointing device is typically operated by a non-dominant hand.

In specific embodiments, the alpha transparency is 1/8 or 2/8 = 1/4 = 25% or 3/8 or 4/8 = 1/2 = 50% or 5/8 or 6/8 = 3/4 or 7 / 8, in which case the mixing can be performed by simple addition and shift operations, and thus no multiplication or division operation is required. This operation can easily be performed in hardware, for example in programmable logic, and does not require a powerful DSP.

In one embodiment, the generated visible object comprises a first plurality of fully transparent pixels and a second plurality of opaque pixels, the first plurality of pixels and the second plurality of opaque pixels being interleaved (alternately positioned), and

BE2017 / 5896 the processor is adapted to overlay pixels of the first graphic image data with completely transparent pixels by leaving the pixel values of the first graphic image data unchanged, and to overlay pixels of the first graphic image data with opaque pixels by the pixel values of the first graphic image by the pixel values of the visible object.

In this embodiment, the visible object has a so-called perforated bitmap as explained in more detail in the pending perforated bitmap application.

In this way, the movable object (spatially) is partially transparent, giving the visual impression of being semi-transparent, but implemented in a manner that does not require mixing of color values. It offers the same visible benefits to the user without requiring a fast DSP.

In one embodiment, the first plurality of pixels and the second plurality of pixels of the visible object are organized in a regular pattern according to one of the following options: i) the regular pattern is a 2x2 pattern, and exactly one of four pixels is completely transparent; ii) the regular pattern is a 2x2 pattern and exactly two of four pixels are completely transparent, the two fully transparent pixels preferably being diagonally opposite each other; iii) the regular pattern is a 2x2 pattern and exactly three out of four pixels are completely transparent.

Tests have shown that these patterns provide a very beautiful (aesthetically attractive) result, without significantly reducing the easy readability of underlying alphanumeric information.

In one embodiment, the generated visible object comprises or consists of monochrome or opaque pixels, and the processor is adapted to generate the second graphic image data at first moments in time by overlaying the first graphic image data with data of the visible object of the generated object; and for * generating the second graphic image data at second moments in time by selecting the first graphic image data without using the data of the visible object.

The first moments in time can correspond to even-numbered frames (T = 2xN, where N is an integer representing the frame number), and the second moments in time can correspond to odd-numbered frames (T = 2xN + 1). Expressed in simple terms, this means that the first graphic image data is overlaid with the movable object at even moments in time, and that the first graphic image data is displayed as it is at odd moments in time, without being overlapped with the movable object. , resulting in a time-multiplexed overlay, which gives the visual impression of being semi-transparent.

BE2017 / 5896

In one embodiment, the processor is further adapted to adjust the displacement values such that under predetermined conditions, the displacement is adjusted to be a horizontal displacement.

It is an important advantage of this feature that it can help the user better control movements of at least one of the pointing devices when performing certain tasks. This is particularly useful for the pointing device controlled by the non-dominant hand when performing text-based tasks (such as reading a text document, proofreading a translation, pointing to individual words on the same line during a Webinar or presentation, etc.)

In one embodiment, the predetermined conditions include: testing whether an angle or angular distance between a displacement vector defined by the displacement values and a horizontal line is less than a predetermined angle (e.g., less than 30 °, or less than 40 °).

In one embodiment, the predetermined conditions include: testing whether a ratio (rico) of the vertical displacement value and the horizontal displacement value is less than a predetermined value.

In certain embodiments, the predetermined value may be a constant value selected in the range of 0.2 to 0.7, e.g. equal to about 0.2 or 0.25 or 0.3 or 0.33 = 1/3 or 0.35 or 0.40 or 0.50 = 1/2 or 0.60 or 0.66 = 2/3 or 0.7.

In other embodiments, the predetermined value may be dynamically adjusted, and may be selected, for example, slightly larger (e.g., a value in the range of 0.5 to 0.7) if an amplitude of the horizontal displacement dx is greater than a predetermined threshold value , for example abs (dx)> 10, and can be chosen to be slightly smaller (for example, a value in the range of 0.3 to 0.5) if an amplitude of the horizontal displacement dx is smaller than or equal to the predetermined threshold value , for example abs (dx) <= 10.

According to an 11th aspect, the present invention provides a computer system comprising: a computer device; and a first pointing device connected to the computer device; and a display device according to the tenth aspect; and a second pointing device connected to the display device.

According to a 12th aspect, the present invention provides a set of components comprising: a computer program product according to the second aspect; and a display device according to the 10th aspect; and optionally a mouse device connectable to the display device.

According to a 13th aspect, the present invention provides a video adapter device comprising: an input port for receiving first graphic image data from a computer device; an input buffer for storing at least a portion of the first graphic image data received at the input port; an output port for providing

BE2017 / 5896 second graphic image data to a display device; an output buffer for storing at least a portion of the second graphic image data to be provided at the output port; a processor connected to the input buffer and adapted to generate second graphic image data based on the first graphic image data; and wherein the video adapter device further comprises a wired and / or a wireless interface for communicating with a wired or wireless pointing device; wherein the processor is further adapted for: * communicating with the pointing device to receive displacement information related to movements of the pointing device; * updating a position based on the received movement information; * generating a visible object; * generating the second graphic image data by overlaying the first graphic image data with the generated visible object at the said position.

A frame buffer is not absolutely required, for example if the output frame frequency and the output frame resolution are the same as the input frame frequency and the input frame resolution, and the overlay on-the-fly can be performed, as represented by path P2 in FIG. 44.

In practice, the input buffer will typically store only a fraction of a video frame contained in the first graphic image data (stream) at a given time in time, and the output buffer will typically store only a fraction of a video frame contained in the second graphic image data ( current) at a certain point in time.

The pointing device may be an HID-compatible pointing device, e.g., an HID-compatible mouse device.

This video adapter device has the advantage that it not only displays the graphic image provided by the computer device, but also provides at least one visible object, for example with the shape or shape of an arrow, or a hand, or a small line, or a large line, or a square, or a rectangle, or a small cross, or a large cross, or other suitable shape or shape, which at least one visible object is movable by a pointing device operatively connected to the video adapter device in place of with the computer device.

As can be understood from FIG. 43 compared to FIG. 5 (a) and FIG. 5 (b), the use of this special video adapter device together with a conventional computer device also solves the problem of FIG. 1 to FIG. 3 issue shown.

Although retrospectively the solution presented in FIG. 43 and FIG. 44 may seem easy to implement, it does not mean that this embodiment is an obvious solution to the problem of FIG. 1 to FIG. 3 to solve. It is pointed out in this respect that there is a widespread belief that a software solution is always cheaper than a hardware solution, especially in consumer electronics, and therefore that the problem must be solved in the computer device, rather than in the display device.

BE2017 / 5896

The solution presented in FIG. 43 and FIG. 44 elegantly solves the problem, moreover without significantly increasing the CPU load of the computer device, and without requiring the special overlay application.

The visible object may be a predetermined visible object, or may have a bitmap obtained from a removable memory, for example from a memory stick with a file comprising said bitmap. In such an embodiment, the video adapter device may have a further connector for inserting a non-volatile memory card or a flash card or a memory stick or the like.

In one embodiment, the video adapter device further comprises: a frame buffer operatively connected to the processor, and adapted to store at least a portion of the first graphic image data received from the input buffer, and to store at least a portion of the second graphic data to be sent to the output buffer.

In this embodiment, the video adapter device comprises a frame buffer, which may be contained in a separate memory component, or in the same memory component that also comprises the input buffer and / or the output buffer.

The data typically flows from the input port via the input buffer to the frame buffer (path P1 of FIG. 44), and from the frame buffer and via the output buffer to the output port (path P1b of FIG. 44). The overlay is typically performed by the processor by reading data from and / or writing data to the frame buffer (not explicitly shown in FIG. 44).

The frame buffer may comprise a complete video frame, or more than one video frame, for example at least two video frames, which, for example, allows one video frame to be ready to be output via the output buffer while the other video frame is generated, in manners known per se are in the art.

In one embodiment, the interface for communicating with the pointing device comprises a physical connector, in particular a USB connector.

A physical connector allows direct connection of the pointing device to the video adapter device.

In one embodiment, the interface for communicating with a pointing device comprises a wireless RF transceiver, e.g., a Bluetooth transceiver.

The RF transceiver may be integrated in the video adapter device, or may be externally connected to the video adapter device, for example in the form of a so-called dongle, e.g. a USB dongle. The dongle can be connected to the physical connector of the video adapter.

In one embodiment, the generated visible object comprises or consists of monochrome pixels, and the processor is adapted to overlay the first graphic

BE2017 / 5896 image data with the monochrome pixels by replacing pixel values of the first graphic image data with a value of the monochrome pixels.

It is an advantage of this embodiment that the overlay operation can be as simple as replacing pixel values of the first graphic image with pixel values of the visible object. The result is a non-transparent, visible movable object overlaid over the first graphic image data.

In one embodiment, the generated visible object comprises or consists of monochrome pixels, and the processor is adapted to overlay the first graphic image data with the monochrome pixels by alpha gene, for example using an alpha transparency in the range of 5% to 95% or of 25% to 75%.

It is an advantage of this embodiment that the overlay alpha gene processing involves a mature technique, and although technically somewhat more complex, the result is a semi-transparent visible movable object overlaid over the first graphic image data. This offers the user the advantage that he can see through the movable object, and that less accurate positioning is required, for example, when underlining alphanumeric information, which is particularly useful since this pointing device is typically operated by a non-dominant hand.

In specific embodiments, the alpha transparency is 1/8 or 2/8 = 1/4 = 25% or 3/8 or 4/8 = 1/2 = 50% or 5/8 or 6/8 = 3/4 or 7 / 8, in which case the mixing can be performed by simple addition and shift operations, and thus no multiplication or division operation is required. This operation can easily be performed in hardware, for example in programmable logic, and does not require a powerful DSP.

In one embodiment, the generated visible object comprises a first plurality of fully transparent pixels and a second plurality of opaque pixels, wherein the first plurality of pixels and the second plurality of opaque pixels are interleaved (alternately positioned), and the processor is adapted to overlay pixels of leaving the first graphic image data with the fully transparent pixels by the pixel values of the first graphic image data unchanged, and for overlaying pixels of the first graphic image data with opaque pixels by replacing the pixel values of the first graphic image with the pixel values of the visible object.

In this embodiment, the visible object has a so-called perforated bitmap as explained in more detail in the pending perforated bitmap application.

In this way, the movable object (spatially) is partially transparent, giving the visual impression of being semi-transparent, but implemented in a manner that does not mix

BE2017 / 5896 of color values required. It offers the same visible benefits to the user without requiring a fast DSP.

In one embodiment, the first plurality of pixels and the second plurality of pixels of the visible object are organized in a regular pattern according to one of the following options: i) the regular pattern is a 2x2 pattern, and exactly one of four pixels is completely transparent; ii) the regular pattern is a 2x2 pattern and exactly two of four pixels are completely transparent, the two fully transparent pixels preferably being diagonally opposite each other; iii) the regular pattern is a 2x2 pattern and exactly three out of four pixels are completely transparent.

Tests have shown that these patterns provide a very beautiful (aesthetically attractive) result, without significantly reducing the easy readability of underlying alphanumeric information.

In one embodiment, the generated visible object comprises or consists of monochrome or opaque pixels, and the processor is adapted to generate the second graphic image data at first moments in time by overlaying the first graphic image data with data of the visible object of the generated object; and for * generating the second graphic image data at second moments in time by copying the pixel values of the first graphic image data as they are.

The first moments in time can correspond to even-numbered frames (T = 2xN, where N is an integer representing the frame number), and the second moments in time can correspond to odd-numbered frames (T = 2xN + 1). Expressed in simple terms, this means that the first graphic image data is overlaid with the movable object at even moments in time, and that the first graphic image data is displayed as it is at odd moments in time, without being overlapped with the movable object. , resulting in a time-multiplexed overlay, which gives the visual impression of being semi-transparent.

In one embodiment, the processor is further adapted to adjust the displacement values such that under predetermined conditions, the displacement is adjusted to be a horizontal displacement.

It is an important advantage of this feature that it can help the user better control movements of at least one of the pointing devices when performing certain tasks. This is particularly useful for the pointing device controlled by the non-dominant hand when performing text-based tasks (such as reading a text document, proofreading a translation, pointing to individual words on the same line during a Webinar or presentation, etc.)

In one embodiment, the predetermined conditions include: testing whether an angular distance between a displacement vector defined by the displacement values (dx, dy) and a

BE2017 / 5896 horizontal line is less than a predetermined angle, (for example less than 30 °, or less than 40 °) or where the predetermined conditions include testing whether a ratio (of the vertical displacement value and the horizontal displacement value is less than a predetermined value.

In certain embodiments, the predetermined value may be a constant value selected in the range of 0.2 to 0.75, e.g. equal to about 0.2 = 1/5 or 0.25 = 1/4 or 0.3 or 0.33 = 1/3 or 0.35 or 0.40 or 0.50 = 1/2 or 0.60 or 0.66 = 2/3 or 0.75 = 3/4.

In other embodiments, the predetermined value may be dynamically adjusted, and may be selected, for example, slightly larger (e.g., a value in the range of 0.5 to 0.7) if an amplitude of the horizontal displacement dx is greater than a predetermined threshold value , for example abs (dx)> 10, and can be chosen to be slightly smaller (for example, a value in the range of 0.3 to 0.5) if an amplitude of the horizontal displacement dx is smaller than or equal to the predetermined threshold value , for example abs (dx) <= 10.

According to a 14th aspect, the present invention provides a computer system comprising: a computer device; and a first pointing device connected to the computer device; and a display device; and a video adapter device according to the thirteenth aspect, operatively connected between the computer device and the display device; and a second pointing device operatively connected to the video adapter device.

The computer device can be a conventional computer device, or a computer device according to the second aspect. The display device can be a classical display device, or a display device according to the 10th aspect.

According to a 15th aspect, the present invention provides a set of components comprising: a computer program product according to the second aspect; and a video adapter device according to the thirteenth aspect.

In one embodiment, the set further comprises a pointing device, operably connectable to the video adapter device.

The present invention also provides a set of components comprising: a video adapter device according to the thirteenth aspect; and a pointing device operably connectable to the video adapter device.

Specific and preferred aspects of the present invention are included in the appended independent and dependent claims. Features of the dependent claims can be combined with features of the independent claims and with features of other dependent claims as appropriate and not merely as explicitly stated in the claims.

BE2017 / 5896

These and other aspects of the invention will be apparent from and clarified with reference to the embodiment (s) described below.

Brief description of the drawings

FIG. 1 shows an example of how a translator or proofreader can compare an original document and a translation thereof, using a classical computer system.

FIG. 2 shows an example of how a person reading and editing an electronic text document or editing or commenting on it can use his finger to remember the position he is reading when moving the mouse cursor to the menu bar to select a tool or color, or the like, using a conventional computer system.

FIG. 3 shows an example of how a person reading or editing a single text document can use his finger to point to one position in the document, and use the mouse cursor to point to another position within that one document , using a traditional computer system.

FIG. 4 shows an example of how a person can transcribe an audio file using a conventional computer system. FIG. 4 (a) shows how an audio player can be started to play an audio clip. FIG. 4 (b) shows how the audio player is temporarily stopped, and text is typed into a text document.

FIG. 5 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and a second pointing device, and performs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointing position in accordance with movements of the first pointing device, and maintains a second pointing position in accordance with movements of the second pointing device, and adjusts the position of the native mouse pointer / mouse cursor, e.g., overwrites it, according to with movements of the first pointing device. In addition, the overlay application provides a second visible object (e.g., a line or rectangle) that moves in accordance with movements of the second pointing device. FIG. 5 (a) shows movements of the first visible object in accordance with movements of the first pointing device. FIG. 5 (b) shows movements of the second visible object in accordance with movements of the second pointing device. FIG. 5 (c) is a schematic representation of a so-called Z-order of windows or image planes and a cursor plane as can be used in the computer system of FIG. 5 (a). FIG. 5 (d) is a high-level flow diagram illustrating a computer-implemented method according to an embodiment of the present invention, as may be used in the computer system of FIG. 5 (a). FIG. 5 (e) is a more detailed flow diagram that is a computer-implemented method

BE2017 / 5896 according to an embodiment of the present invention, as can be used in the computer system of FIG. 5 (a). FIG. 5 (f) shows a simplified high-level block diagram of hardware and software components located in the computer system of FIG. 5 (a). FIG. 5 (g) shows some lines of pseudo-code illustrating a portion of a simplified message processor as can be used by the overlay application. FIG. 5 (h) shows some lines of pseudo code illustrating a portion of a simplified object repositioning routine as may be used by the overlay application to reposition the second visible object.

FIG. 6 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and a second pointing device, and performs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointing position in accordance with movements of the first pointing device, and maintains a second pointing position in accordance with movements of the second pointing device, and adjusts the position of the native mouse pointer / mouse cursor, e.g., overwrites it, according to with movements of the first pointing device. In addition, the overlay application provides a second visible object (e.g., a line or rectangle) that moves in accordance with movements of the second pointing device, and provides a third visible object, which moves in accordance with movements of the first pointing device. FIG. 6 (a) shows movements of the first and third visible object in accordance with movements of the first pointing device. FIG. 6 (b) shows movements of the second visible object in accordance with movements of the second pointing device. FIG. 6 (c) shows some lines of pseudo code illustrating a portion of a simplified position updating routine, such as can be used by the overlay application to reposition the second and third visible object.

FIG. 7 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and a second pointing device, and performs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointing position in accordance with movements of the first pointing device, and maintains a second pointing position in accordance with movements of the second pointing device, and adjusts the position of the native mouse pointer / cursor, e.g., overwrites it in in accordance with movements of the pointing device that was most recently moved. In addition, the overlay application provides a second visible object (e.g., a line or rectangle) that moves in accordance with movements of the second pointing device, and provides a third visible object, which moves in accordance with movements of the first pointing device. FIG. 7 (a) shows movements of

BE2017 / 5896 the first and third visible object in accordance with movements of the first pointing device. FIG. 7 (b) shows movements of the first and second visible object in accordance with movements of the second pointing device. FIG. 7 (c) is a high-level flow diagram illustrating a computer-implemented method according to an embodiment of the present invention, as can be used in the computer system of FIG. 7 (a). FIG. 7 (d) is a more detailed flow diagram illustrating a computer-implemented method according to an embodiment of the present invention, as may be used in the computer system of FIG. 7 (a). FIG. 7 (e) shows some lines of pseudo-code illustrating a portion of a simplified message processor as can be used by the overlay application.

FIG. 8 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and a second pointing device, and performs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointing position in accordance with movements of the first pointing device, and maintains a second pointing position in accordance with movements of the second pointing device, and adjusts the position of the native mouse pointer / mouse cursor, e.g., overwrites it, according to with movements of the first pointing device. In addition, the overlay application provides a second visible object (e.g., a black arrow) that moves in accordance with movements of the second pointing device. FIG. 8 (a) shows movements of the first visible object in accordance with movements of the first pointing device. FIG. 8 (b) shows movements of the second visible object in accordance with movements of the second pointing device. FIG. 8 (c) is a schematic representation of a so-called Z-order of windows or image planes and a cursor plane as can be used in the computer system of FIG. 8 (a).

FIG. 9 shows a computer system according to an embodiment of the present invention, and is a variant of the system shown in FIG. 8. Where the embodiment of FIG. 8 is particularly useful for legal people, the embodiment of FIG. 9 particularly useful for left-handed people. FIG. 9 (a) shows movements of the second visible object (e.g., a black arrow) in accordance with movements of the first pointing device. FIG. 9 (b) shows movements of the first visible object (e.g., the native mouse cursor / pointer, here in the form of a white arrow) in accordance with movements of the second pointing device. FIG. 9 (c) shows a few lines of pseudo-code illustrating a portion of a simplified message processor as can be used by the overlay application.

FIG. 10 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and a second

BE2017 / 5896 pointing device, and performing an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointing position in accordance with movements of the first pointing device, and maintains a second pointing position in accordance with movements of the second pointing device, and adjusts the position of the native mouse pointer / cursor, e.g., overwrites it in in accordance with movements of the pointing device that was most recently moved. In addition, the overlay application provides a second visible object (e.g., a black arrow) that occupies the other maintained position. FIG. 10 (a) shows movements of the first visible object (e.g., the native mouse cursor / pointer) in accordance with movements of the first pointing device (the pointing device being moved), while the second visible object is at the other maintained position. FIG. 10 (b) shows movements of the first visible object in accordance with movements of the second pointing device (the pointing device being moved), while the second visible object (the black arrow) is in the other maintained position.

FIG. 11 shows a simplified high level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, wherein the computer system comprises a computer device with a special overlay application according to an embodiment of the present invention, and two classical pointing devices connected directly with the computer device.

FIG. 12 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, wherein the computer system comprises a computer device with a special overlay application according to an embodiment of the present invention, and two classical pointing devices connected to the computer device via a traditional hub.

FIG. 13 shows some lines of pseudo-code illustrating a portion of a first message processing routine as may be used in overlay applications according to embodiments of the present invention.

FIG. 14 (a) to FIG. 14 (c) and FIG. 15 show exemplary sequences of signals or messages from two classical mouse devices, and show how these signals are satisfactorily handled by the first message processing process shown in FIG. 13.

FIG. 16 and FIG. 17 show two exemplary sequences of signals or messages coming from two classical mouse devices, and show how these signals are treated unsatisfactorily by the message processing process shown in FIG. 13.

FIG. 18 (a) shows some lines of pseudo-code illustrating a portion of a second message processing routine as may be used in overlay applications according to embodiments of the present invention.

BE2017 / 5896

FIG. 18 (b) shows a variant of the routine of FIG. 18 (a), as can be used in overlay applications according to embodiments of the present invention.

FIG. 18 (c) shows a more complete version of the routine of FIG. 18 (a), as can be used in overlay applications according to embodiments of the present invention.

FIG. 19 and FIG. 20 shows two exemplary sequences of signals or messages coming from two classical mouse devices, and shows how these signals are treated satisfactorily by the second message processing process shown in FIG. 18 (a).

FIG. 21 shows an exemplary sequence of signals or messages coming from two classical mouse devices, and shows how these signals are treated unsatisfactorily by the second message processing process shown in FIG. 18 (a).

FIG. 22 (a) and FIG. 22 (b) show two exemplary sequences of signals coming from two special pointing devices according to an embodiment of the present invention, and show how these signals or messages are treated satisfactorily by the second message processing process shown in FIG. 18 (a).

FIG. 23 (a) shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, wherein the computer system comprises a computer device with a special overlay application according to an embodiment of the present invention, and two special mouse devices according to an embodiment of the present invention.

FIG. 23 (b) shows an exemplary block diagram of a special mouse device according to an embodiment of the present invention, as may be used in the system of FIG. 23 (a).

FIG. 24 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, wherein the computer system comprises a computer device with a special overlay application according to an embodiment of the present invention, and two special mouse devices according to an embodiment of the present invention connected to the computer device via a conventional hub.

FIG. 25 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, wherein the computer system comprises a computer device with a special overlay application according to an embodiment of the present invention, and comprises two classical mouse devices connected with the computer device via a special adapter or hub or dongle according to an embodiment of the present invention.

FIG. 26 (a) to FIG. 26 (d) show exemplary sequences of signals or messages coming from the two classical pointing devices of FIG. 25, and show how these signals or

BE2017 / 5896 messages are handled by a message processing process shown in FIG. 18, after passing through the special adapter or hub or dongle.

FIG. 27 (a) shows an exemplary block diagram of a hub (or dongle or adapter) according to an embodiment of the present invention, as may be used in the system of FIG. 25, which has at least two physical connectors for connection to two conventional wired indicating devices.

FIG. 27 (b) shows an exemplary block diagram of a wireless hub (or dongle or adapter) according to an embodiment of the present invention, as may be used in the system of FIG. 25, which has one or two RF transceivers for operative connection to two conventional wireless pointing devices.

FIG. 28 (a) shows an exemplary user interface window, referred to herein as the mini user interface, which supports a plurality of input modes and a plurality of display modes.

FIG. 28 (b) shows an exemplary pop-up menu such as can be used in embodiments of the present invention.

FIG. 29 shows an exemplary user interface window, referred to herein as the main user interface, as may be used in embodiments of the present invention, e.g., to select an input mode and / or a display mode, and / or a size and / or a color. of the second or further visible object, and / or to choose or tune a semi-transparency level, and / or to choose a semi-transparent textured background.

FIG. 30 and FIG. 31 (a) show an enlarged view of a portion of FIG. 29.

FIG. 31 (b) is a variant of the portion of FIG. 31 (a), where an attribute has been added to force cursor movements to be only horizontal (East / West) or vertical (North / South) or diagonal (North-East, North-West, South-East, South-West) .

FIG. 31 (c) shows some lines of pseudo-code that can be used in the position update routine to achieve this effect.

FIG. 31 (d) and FIG. 31 (e) illustrate a variant of the displacement transformation routine in which near-horizontal movements are transformed into purely horizontal movements.

FIG. 32 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and two applications (e.g., a PDF document viewer and a word processor), and executes an overlay application according to an embodiment of the present invention. The overlay application is configured to reposition the native mouse cursor and to show two movable lines that move in accordance with movements of the pointing devices.

BE2017 / 5896

FIG. 33 shows a variant of the computer system of FIG. 32, wherein the overlay application is configured to reposition the native mouse cursor and show two movable squares that move in accordance with movements of the pointing devices.

FIG. 34 shows another variant of the computer system of FIG. 32, wherein the overlay application is configured to reposition the native mouse cursor and display two visible objects, one object containing an image of a left hand, and the other object containing an image of a right hand, which images move in accordance with movements of the pointing devices .

FIG. 35 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and an exemplary application (e.g., a PDF document editor), and executes an overlay application according to an embodiment of the present invention. The overlay application is configured to reposition the native mouse cursor and display a movable line in accordance with movements of the pointing devices.

FIG. 36 shows a variant of the computer system of FIG. 35, wherein the overlay application is configured to reposition the native mouse cursor and display a movable cross, in accordance with movements of the pointing devices.

FIG. 37 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and an exemplary application (e.g., a text editor), and executes an overlay application according to an embodiment of the present invention. The overlay application is configured to reposition the native mouse cursor and display two movable lines in accordance with movements of the pointing devices.

FIG. 38 shows a variant of the computer system of FIG. 37, wherein the overlay application is configured to reposition the native mouse cursor and show two movable crosses, in accordance with movements of the pointing devices.

FIG. 39 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and an exemplary application (e.g., a text editor), and executes an overlay application according to an embodiment of the present invention. The overlay application is configured to reposition the native mouse cursor and display a vertical line to split the screen into a left portion and a right portion, and to display two horizontal lines that are vertically movable in accordance with movements of the pointing devices .

FIG. 40 shows a variant of the computer system of FIG. 39, wherein the overlay application additionally provides two squares or rectangles within the horizontal lines, which squares are also movable horizontally, in accordance with movements of the pointing devices.

BE2017 / 5896

FIG. 41 shows a computer system according to an embodiment of the present invention comprising a special monitor or display screen according to an embodiment of the present invention, connectable directly to a pointing device, the special display screen or monitor being adapted to overlay the image provided by the computer device with at least one visible object that is movable in accordance with movements of the pointing device connected to the display screen. The O / S and GUI of the computer device shows the native mouse cursor / pointer, which is movable through the pointing device connected to the computer.

FIG. 42 shows an exemplary block diagram of a display device used in the system of FIG. 41.

FIG. 43 shows a computer system according to an embodiment of the present invention comprising a video adapter device or video interface device according to an embodiment of the present invention, which has an input port for receiving first graphic image data from a computer device to which the computer device is connected with a first pointing device. The video adapter further comprises an output port for providing second graphic image data to a monitor or display screen, the video adapter further comprising a wired or wireless interface to a second pointing device, the video adapter being adapted to generate at least at least one visible object, and for overlaying the first graphic image data with the at least one visible object, and for moving the visible object in accordance with movements of the second pointing device connected to the video adapter.

FIG. 44 shows an exemplary hardware block diagram of the video adapter device of FIG. 43.

It is noted that the timing diagrams shown herein are high-level timing diagrams, and many details have been omitted, which are not relevant to understanding the principles of the present invention.

Detailed description of illustrative embodiments

The present invention will be described with respect to particular embodiments and with reference to certain drawings, however, the invention is not limited thereto but is only limited by the claims. The figures are only schematic and non-limiting. In the figures, the dimensions of some parts may be exaggerated and not represented to scale for illustrative purposes. The dimensions and the relative dimensions sometimes do not correspond to the current practical embodiment of the invention.

Furthermore, the terms first, second and the like in the description and in the claims are used to distinguish similar elements and not necessarily for describing

BE2017 / 5896 of a sequence, neither in time, nor spatially, nor in ranking, or in any other way. It is to be understood that the terms used in such a manner are suitable under interchangeable conditions and that the embodiments of the invention described herein are capable of operating in a different order than described or depicted herein.

It is to be noted that the term comprising, as used in the claims, is not to be interpreted as being limited to the means described thereafter; this term does not exclude other elements or steps. It can therefore be interpreted as specifying the presence of the listed characteristics, values, steps or components referred to, but does not exclude the presence or addition of one or more other characteristics, values, steps or components, or groups thereof. Thus, the scope of the term a device comprising means A and B should not be limited to devices that consist only of components A and B. It means that with respect to the present invention, A and B are the only relevant components of the device.

Reference throughout this specification to "one embodiment" or "an embodiment" means that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, occurrence of the expressions "in one embodiment" or "in an embodiment" at various places throughout this specification may not necessarily all refer to the same embodiment, but may do so. Furthermore, the specific features, structures, or characteristics may be combined in any suitable manner, as would be apparent to those skilled in the art based on this disclosure, in one or more embodiments.

Similarly, it should be appreciated that in the description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof for the purpose of streamlining disclosure and assisting in understanding one or several of the various inventive aspects. In any case, this method of disclosure should not be interpreted as a reflection of an intention that the invention requires more features than explicitly mentioned in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all the features of a single prior disclosed embodiment. Thus, the claims following the detailed description are hereby explicitly included in this detailed description, with each independent claim as a separate embodiment of the present invention.

Furthermore, while some embodiments described herein include some, but not other, features included in other embodiments, combinations of features of different embodiments are intended to be within the scope of the invention, and

BE2017 / 5896 constitute these different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the described embodiments can be used in any combination.

Numerous specific details are set forth in the description provided here. It is, however, understood that embodiments of the invention can be practiced without these specific details. In other cases, well-known methods, structures and techniques have not been shown in detail to keep this description clear.

In this document, the term alpha mixing or mixing (of images or bitmaps) is used as synonyms. In the context of graphic overlay, there is an upper image (that has a higher Zorde) and a lower image (that has a lower Z order). The mixing can be done in hardware and / or software, and is based on a parameter called alpha mixing value a, which refers to the level of mixing of the two images, where a = 0% means that the original image (with the lower Z- order) is not mixed with the overlay image (or that the overlay image is completely transparent), and where a = 100% means that the resulting pixel value is that of the overlay image (or that the overlay image is completely opaque except for fully transparent pixels). The alpha mixing value can be expressed on a scale from 0 to 255, where a = 255 corresponds to a = 100% and a = 0 corresponds to a = 0%. Alpha mixing can for example be used for fade-in or fade-out of a graphic image. In the context of the present invention, there may also be a second parameter involved in mixing, called TransparentColorValue. If a pixel of the overlay image has this predetermined value, this pixel is treated as completely transparent. Fully transparent pixels are typically used on the circumference of a bitmap, for example, to display a rectangular image with rounded edges. Alpha blending and fully transparent pixels are both well known in the art.

In this document, the word processor or text editor expressions should not be interpreted too narrowly, because many so-called word processors (such as, for example, Microsoft Word) also have drawing options. Similarly, the term spreadsheet or worksheet should not be interpreted too narrowly, because many so-called worksheet applications, such as Microsoft Excel, also have graphic options.

The expression the overlay window is configured in click-through mode or the overlay window is configured in pass-through mode means that the overlay window is configured in such a way that events from input devices (such as, for example, a mouse, a keyboard, a trackball, a touch pad, a stylus) , a joystick, etc.) are sent by the operating system to one or more underlying application windows or to objects thereof, despite the higher Z-order of the overlay application, and despite the native mouse cursor being located above the area defined by the overlay window . The message

BE2017 / 5896 passed to lower windows applies, for example, to events caused by a button press, and to a button release, and to a button click, and to a scroll event of a mouse wheel, and to movements of the pointing device. In the context of the present invention, the overlay application typically comprises one window configured in click-through mode, and optionally but preferably also includes a second, substantially opaque window with user interface elements, not configured in click-through mode.

In this document, the expression the window is configured in non-click-through mode means that the window is not configured in click-through mode.

In this document, the term work area refers to a portion of the pixel area of a screen, namely, the area that will typically be occupied by an application in case it is maximized. In a Microsoft Windows environment, the work area means the entire pixel area or the area of the desktop minus the so-called taskbar, which is typically located at the bottom of the screen.

In this document, the mouse pointer and mouse cursor terms are used as synonyms, and refer to a symbol or graphic image on a computer monitor or other display device that displays movements of a pointing device, e.g., a mouse, touchpad, or stylus pen. This pointer or cursor can take various forms, such as an arrow or a crosshair (which looks like a Roman numeral I) or as a hand, etc.

In this document, the terms display screen and monitor and screen are used as synonyms.

In this document, the terms line or bar or line segment are used as synonyms. They refer to an elongated, for example a rectangular, object, optionally with rounded edges.

In this document, the expression the mouse cursor or pointer means movements of the pointing device means that the position of the cursor or pointer is adjusted in accordance with movements of the pointing device.

In this document, the terms first maintained position and position related to movements of the first pointing device mean the same. This position is also referred to herein as Pos1 or the coordinates (x1, yl) or the coordinates (xR, yR).

In this document, the first pointing device usually refers to the pointing device operated by a user's right hand unless explicitly stated otherwise.

In this document, the terms second maintained position and position related to movements of the second pointing device mean the same thing. This position is also referred to herein as Pos2 or the coordinates (x2, y2) or the coordinates (xL, yL).

In this document, the second pointing device usually refers to the pointing device to be operated by a user's left hand, unless explicitly stated otherwise.

BE2017 / 5896

In this document, the expression shifting focus between two documents means shifting focus between two locations on the screen, for example, between a first window associated with a first application (e.g., a PDF viewer) showing a portion of a first document, and a second window associated with a second application (e.g., a word processor) that displays a portion of a second document, unless explicitly stated otherwise.

In this document, the term host means a computer device, unless explicitly stated otherwise.

In this document, the expressions signal and message sent by a pointing device mean the same thing.

In this document, the term underlining or marking a text fragment means the apparent or virtual underlining or marking of the text fragment by means of an overlaid object.

As described in the background portion and as shown in FIG. 1 to FIG. 4, there are many occasions when a computer user would like the computer system to have a mechanism that allows the user to read and / or write and / or click in two different locations on a screen, while today, in all major Operating systems for personal computers with a graphical user interface, referred to as a single visible object, mouse pointer or mouse cursor, are supported.

While the present invention is not limited to Windows-compatible computers that are running a version of the Windows operating system from Microsoft Corporation (Redmond, USA), the present invention will be primarily explained with such computers in mind, in order to simplify the description. But the principles of the present invention will also work for other computers with an operating system O / S and a graphical user interface GUI that provides only a single mouse pointer, such as, for example, Mac OS from Apple Inc.

Although the present invention is not limited to working with two mouse devices, the present invention will be explained with two mouse devices in mind in order to simplify the description. But the principles of the present invention will also work with other pointing devices, such as, for example, a touch pad or a trackball or a stylus or the like.

The problems encountered by a translator or proofreader or writer or transcriber in the cases shown in FIG. 1 to FIG. 4, are related to a characteristic (or shortcoming) of operating systems with a GUI that only maintain a single mouse cursor or mouse pointer. From a commercial point of view, one might think that there is one

BE2017 / 5896 there is a simple solution for these problems, namely connecting a second pointing device to the computer device and hoping that somehow a second mouse cursor will appear that follows movements of the second pointing device. Unfortunately, conventional operating systems for personal computers such as, for example, Windows and mac OS do not work that way, since they only keep a single cursor position, and although multiple pointing devices can be physically connected to the computer device, all signals coming from these pointing devices are sent to the one cursor position. For example, when two USB mouse devices are connected to a laptop computer running Windows 7 or Windows 10, the Windows native mouse cursor can be moved to the left by moving each of these mouse devices to the left. No second mouse cursor will appear automatically.

In contrast, the present invention provides a computer system that supports two pointing devices and provides two or at least two visible objects that are individually movable through the two pointing devices, e.g., a first mouse device and a second mouse device.

Although it would be technically possible to develop a new operating system with inherent support for multiple pointing devices and multiple Cursors, such a solution is not a realistic option. The inventors of the present invention have therefore attempted to find a solution based on an existing operating system O / S with a graphical user interface GUI that provides a single native mouse cursor, and to extend or improve its functionality, ideally in such a way that most or all existing applications can benefit from this without changing these applications.

As will be further explained in more detail, the present invention proposes to approach the problem not in the device driver layer but as a graphical overlay application that includes a window configured in click-through mode. This is one of the underlying principles of the present invention. This solution is considered not to be trivial, among other things because:

i) the logical place to resolve a problem related to input devices is the device control layer, not by overlaying an application;

ii) it is not logical to configure a window of the overlay application in click-through mode because it means that the O / S does not have any messages related to the pointing devices (e.g., a motion message when the pointing device is moved, a button-pressed message when a button is pressed, a push button release message when a button is released, a scroll wheel rotation message when a scroll wheel is turned, etc.) to this

BE2017 / 5896 overlay application will send, preventing the overlay application from filtering or blocking or changing these messages.

Instead, the O / S is configured to send raw input messages, and the overlay application of the present invention is configured to receive these raw input messages from the O / S, and the overlay application provides a message processor to send these messages. processing, but (as the inventors only discovered during testing) the overlay application only receives these messages after the O / S has already processed them.

Another underlying principle of the present invention is that use is made of the native mouse cursor provided by the O / S and the GUI as one of the at least two visible and movable objects. This solution is also deemed not to be trivial because the O / S and GUI will receive the input messages from the pointing devices via the device driver layer, and the native mouse cursor will want to position in its own way, namely by the native move the mouse cursor in accordance with movements of each pointing device. As will be further explained, the overlay application of the present invention will overwrite this position.

Using the native mouse cursor of the O / S and the GUI as one of the visible objects offers the great advantage that the appearance and appearance of this mouse cursor usually remains under the control of the O / S and the GUI and the other applications, except when the mouse cursor moves over a user interface window of the overlay application. Therefore, the solution provided by the present invention allows the mouse pointer to change shape depending on the context of the application under the overlay application, e.g., to a arrow pointer or a pointer with an 1-dash or an hourglass, etc.

Referring now to the figures.

FIG. 1 to FIG. 4 have already been described in the background section, and illustrate several practical examples when a single user (more specifically a professional user) wants to use a personal computer with two pointing devices, one operable with his / her dominant hand, the other operable with his / her non- dominant hand.

In order to quantify the requirements of a potential solution with two pointing devices, it is assumed that the user of FIG. 1 to FIG. 4 is legal, and that the computer device of FIG. 1 to FIG. 4 is connected to two pointing devices: a right mouse device for the right hand, and a left mouse device for the left hand, then:

(1) the user of FIG. 1 and FIG. 2 and FIG. 3 wants to move the right mouse device and / or click and / or scroll, and wants to move the left mouse device, but does not necessarily have to click or scroll the left mouse device;

BE2017 / 5896 (2) alternatively, the user of FIG. 1 move the right mouse device and / or click and / or scroll, and want to move and / or drag the left mouse device (i.e., press and move and release) and / or scroll;

(3) the user of FIG. 4 wants to move the right mouse device and / or click and / or scroll, and wants to move the left mouse device and / or click, but does not necessarily have to scroll the left mouse device.

FIG. 5 shows a computer system 500 according to an embodiment of the present invention, comprising a computer device 501 and two pointing devices 503R, 503L and a display screen 504 and optionally a keyboard 502. The computer device 501 is connected to a first pointing device 503R and to a second pointing device 503L, and executes an O / S and GUI (e.g., Windows 10 from Microsoft) that provides a native mouse pointer / mouse cursor 505 (shown here in the form of a white arrow), and executes an overlay application according to an embodiment of the present invention, and optionally executes one or more other applications, such as, for example, a word processor, a PDF document viewer, a web browser, etc. The overlay application maintains two positions Pos1, Pos2 in accordance with movements of the first and second pointing device, respectively, and repeatedly overwrites the position of the native mouse pointer / mouse cursor 505 to match with movements of the first pointing device 503R. The overlay application allows the native mouse pointer / mouse cursor 505 of the O / S and the GUI to be displayed as a first visible object instead of hiding them. The overlay application additionally provides a second visible object 506L that moves in accordance with movements of the second pointing device 503L.

FIG. 5 (a) shows movements of the first visible object 505, being the native mouse cursor provided by the O / S and the GUI, in accordance with movements of the first pointing device 503R. The second visible object 506L shows the second maintained position Pos2, but is stationary in FIG. 5 (a).

FIG. 5 (b) shows movements of the second visible object 506L provided by the overlay application, in accordance with movements of the second pointing device 503L. The first visible object 505 shows the first maintained position Pos1, but is stationary in FIG. 5 (b).

FIG. 5 (c) is a schematic representation of a so-called Z-order of five windows or image planes 581-585 and a cursor plane 586 as may be used in the computer system 500 of FIG. 5 (a) and FIG. 5 (b).

BE2017 / 5896

The plane 581 can be a background plane (in the case of Windows known as the desktop), the image plane 582 can be the visible representation of a window associated with a text editor application, for example, the image plane 583 can be the visible representation of a window associated with a PDF viewer application, for example, the image plane 584 can be the visible representation of a window associated with the graphic overlay application according to the present invention, the cursor plane 586 is the image plane associated with the native mouse cursor provided by the O / S and the GUI , whose form can be changed by an underlying application. The graphic overlay application of the present invention may also have an optional user interface window 585, an example of which will be shown in FIG. 28 to FIG. 31 are described. The user interface window 585 is not configured in click-through mode, but is configured in non-click-through mode, meaning that its visible components (e.g., buttons, radio buttons, checkboxes, etc.) are able to receive (among other things) mouse clicks.

The window 584 of the overlay application can have a size equal to the visible screen, or equal to the area of the desktop surface 581, or equal to the area of the so-called work area (ie the area of the screen minus the area occupied by the so-called taskbar), or another suitable size. The window 584 comprises at least one visible object 506L, for example in the form of a rectangular object. The object 506L can be opaque or semi-transparent. The object 506L can have monochrome pixels, for example red or blue pixels, or can have a bitmap with a color gradient, etc. The other pixels 594 of the window 584 can be fully transparent pixels. Alternatively, the window 584 may be configured as a semi-transparent window, for example with an alpha transparency in the range of 1% to 99%, for example in the range of 5% to 95%. In this case, the pixels of the window 584 can have a color, for example a monochrome color, but the window 584 can also comprise a perforated bitmap or a texture bitmap or a perforated texture bitmap. These aspects are described in greater detail in a co-pending patent application filed by the same applicant on the same day, entitled METHOD, DEVICE AND COMPUTER PROGRAM FOR OVERLAYING A GRAPHICAL IMAGE, with 42 claims, which document is herein by reference in its entirety and further referred to herein as the pending perforated bitmap application. In the event of a conflict between this application and the pending application, the present document prevails.

In the example of FIG. 5 (c) the graphic overlay application has two windows:

- a window 584 that is configured in click-through mode and is essentially transparent or semi-transparent (e.g., with an alpha transparency in the range of 5% to 95%),

and another window 585 which is preferably not configured in click-through mode and which is mainly opaque.

BE2017 / 5896

FIG. 5 (d) shows a high-level flow diagram of a computer-implemented method 570 according to an embodiment of the present invention, for providing a first and a second visible object 505, 506L separately movable by a first pointing device 503R and a second pointing device 503L.

The method can be implemented in an overlay application 551 (see FIG. 5f) that can be executed on the computer device 501 (see FIG. 5a), which computer device 501 also comprises an operating system O / S with a graphical user interface GUI that has a single native mouse cursor 505. The computer device 501 is connected to the first and the second pointing device 503R, 503L. The method maintains two positions Pos1 and Pos2 (see also FIG. 5f). The method shown in FIG. 5d includes the following steps:

a) configuring 571 the operating system O / S with graphical user interface GUI to provide the native mouse cursor 505 as the first visible object, e.g., by allowing the O / S to display the mouse cursor or not explicitly using the mouse cursor. hide;

b) providing 572 a graphic overlay window 584 comprising the second visible object 506L;

c) configuring 573 of the graphic overlay window 584 in click-through mode;

e) adjusting a position of the native mouse cursor 505 in accordance with movements of the first pointing device 503R, and adjusting a position of the second visible object 506L in accordance with movements of the second pointing device 503L.

In one embodiment, the position of the first visible object 505 is updated to the same frequency as the position of the second visible object 506L.

In another embodiment, the position of the first visible object 505 is updated to a different frequency than the position of the second visible object 506L. For example, the position of the native mouse cursor can be updated in a message processing routine (see, e.g., FIG. 5g), and the position of the second visible object can, for example, be updated based on a timer interrupt.

FIG. 5 (f) shows a more detailed example of a computer-implemented method according to an embodiment of the present invention, which can be seen as a special case of the method shown in FIG. 5 (d). The method of FIG. 5 (e) further comprises step d):

d) configuring the operating system O / S to send raw input messages 574;

In the case that the O / S is a Microsoft Windows variant, this step may include registering the application with the O / S for receiving raw input data, e.g., in the form of input messages known as WM_INPUT messages, e.g. using the WinAPI function RegisterRawlnputDevices (). It is noted that applications do not automatically receive raw input messages.

BE2017 / 5896

And wherein step e) comprises: repeatedly performing:

f) receiving 575 a raw input message;

g) determining whether the raw input message is related to a movement of the first or second pointing device 503R, 503L, and if the result of this test is true, obtaining displacement information dx, dy related to the displacement, and updating the corresponding first or second position Pos1, Pos2; and if the result of this test is false, skip step h);

h) adjusting 577 the position of the native mouse cursor 505 based on the first maintained position Pos1, or the second maintained position Pos2, or the position Pos1 or Pos2 associated with the pointing device that was most recently operated. In the example of FIG. 5 (a) and FIG. 5 (b) the position of the native mouse cursor 505 is always updated in accordance with movements of the first pointing device 503R, but as will be further explained, this position may also be updated in a different manner.

And wherein step e) further comprises:

i) repeatedly adjusting 578 a position of the second visible object 506L in accordance with the second position (Pos2). This step i) can also be performed asynchronously with respect to steps f) to h), for performance reasons. Alternatively, step i) can be performed after each execution of step h).

Step g) in combination with step h) is also referred to herein as the position update routine, it is a part of the message processing routine. Step i) is also referred to herein as article repositioning routine.

FIG. 5 (f) shows a simplified high-level block diagram of hardware and software components located in the computer system 500, comprising a computer device 501 with a special overlay application according to an embodiment of the present invention, and two classical pointing devices M1, M2.

The overlay application maintains two pointer positions: a first position Pos1 that has coordinates (x1, yl) and a second position Pos2 that has coordinates (x2, y2). A message processor 552 of the overlay application 551 will update the first and second positions Pos1, Pos2 based on movements of the two pointing devices, further discussed in FIG. 5 (g).

The graphic overlay application of the system 500 repeatedly adjusts the position of the native mouse cursor 505 to the first serviced position Pos1, according to movements of the right mouse device 503R, and repeatedly adjusts the position of the second visible object 506L to match with the second maintained position Pos2, corresponding to movements of the left mouse device 503L, or vice versa.

BE2017 / 5896

FIG. 5 (g) shows some lines of pseudo-code to illustrate a simplified version of a message processing routine such as can be used in the overlay application 551 that is in the computer device 501 of FIG. 5 (a) and FIG. 5 (b) is performed. The message processor 552 is triggered by incoming raw messages from input devices (such as, e.g., keyboard and / or pointing devices) in a manner known per se in the prior art. The message processor can obtain or request further information related to each message, for example to get a device handle to determine which pointing device has sent the message, and / or to get a horizontal and vertical movement dx, dy as an indication of a movement in the horizontal and / or vertical direction, and / or to get a value as an indication of how much a scroll wheel has been scrolled, etc.

As can be understood from the pseudo-code, if the message was sent by the left pointing device 503L, the second position Pos2 will be updated using the displacement values dx, dy, and the position of the native mouse cursor 505 will be set (or reset) to the first position Pos1 are related to the right pointing device 503R. If the message was sent by the right pointing device 503R, the first position Pos1 will be updated using the displacement values dx, dy, and the position of the native mouse cursor 505 will be adjusted to move accordingly. Tests have shown that this routine also correctly handles simultaneous movements of the left and right pointing devices 503L, 503R.

In the example shown in FIG. 5 (g), the X and Y coordinates of the first and second positions Pos1, Pos2 are increased by the displacement values dx, dy as received from the pointing devices, but in practice, the displacement values dx, dy can be first transformed, linear or not -linear, to make the movements less or more sensitive (see the left / right mouse speed buttons and the speed curve in FIG. 31a).

As in FIG. 31 (b) and FIG. 31 (c) to be discussed further, other adjustments to the displacement values are possible.

FIG. 5 (h) shows some lines of pseudo-code to illustrate a simplified version of a routine that updates the position of the second visible object 506L, in the example being a rectangle, but the present invention is not limited thereto. This routine can be based on a timer interrupt. In this way, updating the position of the second visible object 506L outside of the message processor can be handled, which may be preferred for performance reasons. The period of the timer can be selected, for example, in the range of 1 ms to 250 ms, preferably in the range of about 1 ms to about 100 ms, e.g. equal to about 1 ms or about 5 ms or about 10 ms or about 20 ms or about 25 ms or about 30 ms or about 35 ms or about 40 ms or about 50 ms or about 75 ms or

BE2017 / 5896 approximately 100 ms. The value of this period is not critical. The smaller this time period value, the faster the response, but the higher the CPU load. The greater this time period value, the slower the response, but the lower the CPU load. The skilled person can easily find a suitable value by routine experimentation. Some embodiments may allow the end user to choose a preferred timer value.

FIG. 5 (a) to FIG. 5 (h) illustrate a first embodiment of the present invention, particularly suitable for legal people. The O / S and GUI provide the native mouse cursor 505, but the overlay application repeatedly adjusts its position such that the position is only defined by movements of the right mouse device 503R. The right mouse device 503R can be moved, its buttons can be pressed or released or clicked, and its scroll wheel can be scrolled as in the prior art. In addition, the overlay application shows a second visible object 506L, which in the example of FIG. 5 (a) is in the form of a line or rectangle, and the overlay application repeatedly adjusts its position such that the position is defined only by movements of the left mouse device 503L. This embodiment successfully addresses the data shown in FIG. 1 to FIG. 3 problems shown. The user can use his non-dominant hand to move the second object 506L to underline or highlight text he or she wants to read, and can use his dominant hand to edit a document, select text, choose menus, etc. using the native mouse cursor 505. Thus, the overlay application improves the O / S and GUI by providing one additional visible object 506L that is movable through the left pointing device 503L, while maintaining most or all of the functionality of the native mouse cursor. The additional visible object 506L can, for example, be used for underlining or marking text fragments.

In one embodiment, the overlay application may be a specially provided application that only displays the specific behavior shown in FIG. 5 (a) and FIG. 5 (b), wherein the native mouse cursor 505 is moved by the right pointing device, and a second visible object 506L that is in the form of a line movable by the left pointing device, in which case the user interface window 585 can be omitted.

In another embodiment, the overlay application can implement various options, and a user can choose a mode from a plurality of modes. In this case, a user interface window is useful, see for example the exemplary user interface of FIG. 28 (a), e.g., mode 2860 (d), and the exemplary user interface of FIG. 30, e.g., mode 3060 (d). The large right mouse shown on the left of FIG. 28 (a) is a hint to the user that the right mouse is the dominant mouse for the 3060 modes, that is, the right mouse device controls the native mouse cursor 505. The small left mouse is a hint to the

BE2017 / 5896 user that the left mouse device is the non-dominant mouse controlling the second visible object, in the case of mode 2860 (d) being a line or rectangle.

FIG. 6 (a) to FIG. 6 (c) shows a computer system 600 according to another embodiment of the present invention, which can be seen as a variant of the computer system 500 of FIG. 5, wherein a computer device 601 is connected to a first and second pointing device 603R, 603L, and executes an O / S and GUI (e.g., Windows 10) that provides a native mouse pointer 605 (e.g., in the form of a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first and a second position Pos1, Pos2 in accordance with movements of the first and second pointing devices 603R, 603L, respectively, and adjusts the position of the native mouse pointer 605, for example, overwrites it in accordance with movements of the first pointing device 603R. The overlay application provides a second visible object 606L that moves in accordance with movements of the second pointing device 603L, and also provides a third visible object 606R that moves in accordance with movements of the first pointing device 603R. In the example of FIG. 6, the second and third objects are 606L and 606R rectangles that have the same size and color, but this is not absolutely required, and the two visible objects can have a different size and / or a different shape and / or a different color. Thus, the overlay application used in the computer system 600 of FIG. 6 improves the O / S and the GUI by providing two additional visible objects 606L, 606R that are movable through the two pointing devices 603L, 603R, and which can be used, for example, to underline or highlight text fragments of classic applications that are on the computer for example: a word processor, a web browser, a slideshow, a PowerPoint presentation, a PDF document viewer, etc. The objects 606L, 606R can be opaque or semi-transparent or can contain a perforated bitmap as described in the co-pending perforated bitmap application.

FIG. 6 (a) shows movements of the first visible object 605 (the native mouse cursor) and movements of the third visible object 606R in accordance with movements of the first pointing device 603R.

FIG. 6 (b) shows movements of the second visible object 606L in accordance with movements of the second pointing device 603L.

FIG. 6 (c) shows an exemplary object repositioning routine that can be used in the overlay application to move the two visible objects 606L, 606R, in pseudocode. Simply put, the coordinates of the object 606L (called rectangleL) are updated according to the second position Pos2 associated with the second mouse device 603L, and the coordinates of the object 606R (called rectangleR) are updated according to the first

BE2017 / 5896 position Pos1 associated with the first mouse device 603R. These steps are performed repeatedly, for example on the basis of a timer.

It is to be noted that, in FIG. 6 (a) and FIG. 6 (b), the first visible object 606R is shown on the right-hand side of the screen 604, and the second visible object 606L is shown on the left-hand side of the screen 604, but that is not absolutely required, and both objects can be displayed anywhere on it screen can be moved, as shown for example in FIG. 37. In practice, however, a user will typically find it more intuitive to position the object related to the left pointing device on the left side of the screen. In other embodiments, the movement of the second and optionally further object (s) 3906R, 3906L is limited, as shown, for example, in FIG. 39, which will be discussed later, but even then, the movement of the native mouse cursor 3905 is preferably not limited.

The computer system 600 of FIG. 6 can be used, for example, by proofreaders for the apparent underlining of text portions in two documents which are shown side by side on the screen 604, without actually changing the content of the documents themselves. It is an important advantage that text fragments can appear to be underlined in two different documents at the same time. This can dramatically reduce the cognitive burden of the user when switching focus between the two documents. It is an important advantage that the text fragments can be independently underlined (unlike, for example, word processors that only allow synchronous navigation through two text documents), not only because sentences in one language can be longer or shorter than sentences in another language, but also because the two documents can have a different formatting, for example, a different layout and / or a different font and / or a different line spacing, etc. It is an advantage that the second mouse and the second object can be used in combination with almost any underlying application, for example, a word processor, a PDF document reader, a worksheet, a web browser, a PowerPoint application, etc. without having to change any of these applications. For example, the object 606L can be moved over a PDF document, and the object 606R can be moved over a text document, and vice versa. In the example of FIG. 6 (a) and FIG. 6 (b), the native mouse cursor 605 moves with the object 606R.

The user can use the native mouse cursor 605 to click and / or drag anywhere on the screen and / or select menu options, for example to select text in the text document that can then be deleted or replaced with other text using the keyboard 602, or to open new applications in the taskbar (not explicitly shown) in the same way as without the graphical overlay application can be done.

In the exemplary user interface of FIG. 28 (a), the behavior of FIG. 6 (a) and FIG. 6 (b) correspond to mode 2860 (h). (the word can be used because various variations are possible, for example for the message processor, as will become clear further).

BE2017 / 5896

FIG. 7 shows a computer system 700 according to another embodiment of the present invention, which can be seen as a variant of the computer system 600 of FIG. 6, wherein a computer device 701 is connected to a first pointing device 703R and a second pointing device 703L, and executes an O / S and GUI that provides a native mouse pointer or mouse cursor 705 (e.g., a white arrow), and an overlay application according to a An embodiment of the present invention that provides a second and a third visible object 706L, 706R.

The overlay application maintains a first and a second pointer position Pos1, Pos2 in accordance with movements of the first and second pointer 703R, 703L, respectively, and adjusts the position of the native mouse pointer / cursor 705, e.g., overwrites it, in accordance with movements of the pointing device that was most recently moved. Thus, if the right pointing device 703R is moved and the left pointing device 703L is stationary, as shown in FIG. 7 (a), the native mouse cursor 705 follows movements of the right pointing device 703R, and the buttons and / or the scroll wheel of the right pointing device 703R can be used. If the left pointing device 703L is moved at a different time in time and the right pointing device 703R is stationary, as shown in FIG. 7 (b), then the native mouse cursor 705 follows movements of the left pointing device 703L, and the buttons and / or the scroll wheel of the left pointing device 703L can be used. As can be understood from FIG. 7 (a) and FIG. 7 (b) the native mouse cursor follows movements of the pointing device that has been moved until the other pointing device is moved, causing the native mouse pointer to jump to the other maintained position. In this way the user can use the native mouse pointer at two different locations of the screen, albeit at different times in time. Or in other words, in this embodiment, the native mouse cursor 705 provided by the O / S and the GUI is temporarily associated with one of the pointing devices 703L, 703R at any time in time, so the native mouse pointer 705 is shared in time between the two indicating devices.

FIG. 7 (c) is a high-level flow diagram showing a computer-implemented method according to an embodiment of the present invention, as can be used in the computer system of FIG. 7 (a).

FIG. 7 (c) shows a computer-implemented method for providing a first visible object 705 and a second visible object 706L and a third visible object 706R. The third visible object 706R is movable through a first pointing device 703R. The second visible object 706L is movable separately by a second pointing device

703L.

BE2017 / 5896

The first visible object 705 is selectively movable by one of the first pointing device 703R and the second pointing device 703L, depending on which was most recently moved and / or operated. In FIG. 7 (a), the first pointing device 703R is moved, and the native mouse cursor 705 and the third visible object 703R both move in accordance with movements of the right pointing device, or more specifically, in accordance with a first position Pos1. In FIG. 7 (b), the second pointing device 703L is moved, and the native mouse cursor 705 and the second visible object 706L both move in accordance with movements of the left pointing device 703L. It can be understood that native mouse cursor 705 moves smoothly as long as only one pointing device is moved, but typically suddenly jumps from Pos1 to Pos2 or vice versa when the other pointing device is moved.

FIG. 7 (c) shows a flow diagram illustrating a computer-implemented method comprising the following steps:

a) configuring, for example, allowing an operating system O / S with a graphical user interface GUI to provide a native mouse cursor 705 as a first visible object;

b) providing a graphical overlay window comprising a second visible object 706L and a third visible object 706R;

c) configuring the graphic overlay window in click-through mode;

e) adjusting (774) a position of the native mouse cursor 705 in accordance with movements of one of the first pointing device 703R and the second pointing device 703L, depending on which movement was most recent, and adjusting (775) a position of the second visible object 706L in accordance with movements of the second pointing device 703L, and adjusting a position of the third visible object 706R in accordance with movements of the first pointing device 703R.

FIG. 7 (d) is a more detailed flow diagram showing a computer-implemented method according to an embodiment of the present invention, as can be used in the computer system of FIG. 7 (a). The method comprises the following steps:

a) configuring, for example, allowing an operating system O / S with a graphical user interface GUI to provide a native mouse cursor 705 as a first visible object;

b) providing a graphical overlay window comprising a second visible object 706L and a third visible object 706R;

c) configuring the graphic overlay window in click-through mode;

d) configuring the O / S to send raw input messages, and providing a message processor to process these raw input messages;

e) repeatedly performing the following steps:

BE2017 / 5896

f) receiving a raw input message;

g) if the received raw input message is related to a movement of the first / second pointing device 703R, 703L, obtaining displacement information dx, dy and updating a first and second position Pos1, Pos2 respectively;

h) adjusting a position of the native mouse cursor 705 in accordance with the first or second position Pos1, Pos2 depending on the most recent input message;

and repeatedly performing the following step:

i) adjusting a position of the second and third visible object 706L, 706R in accordance with the first position Pos1 and the second position Pos2, respectively.

FIG. 7 (e) shows an exemplary position updating routine (as part of the message processing routine), in pseudo-code, that can be used to obtain this behavior. As can be seen, the coordinates (xR, yR) and (xL, yL) of the first and second positions Pos1, Pos2 are updated in accordance with movements of the first and second pointing devices 703R, 703L (albeit indirectly, by use) device handle which assigns the O / S to each pointing device), and the position of the native mouse cursor 705 is adjusted depending on the most recent motion message.

An advantage of this embodiment is that a user can use two mouse devices to simultaneously display two visible objects, e.g., two lines or two rectangles for underlining or highlighting text fragments at two different locations of the screen, and in addition the user can also click or scroll the mouse device that is most recently moved. This can be used, for example, for scrolling up and down the respective documents during proofreading.

FIG. 8 shows another computer system 800 according to an embodiment of the present invention, which can be seen as a variant of the computer system 500 of FIG. 5, wherein the second visible object 806 is in the form of a black arrow, but of course the invention is not limited thereto, and an arrow of a different color can also be used, but the drawing is not limited to an arrow, and also other shapes can be used, see for example FIG. 34 where two hands are used.

As shown in FIG. 8 (c) includes the overlay application used in the computer system 800 of FIG. 8 a click-through window 884 similar to the window 584 of FIG. 5 (c), but wherein the rectangular object 506L has been replaced by a visible object (e.g., an image component)

BE2017 / 5896 containing a suitable bitmap. The image component 806L may include a bitmap containing pixels representing a black arrow surrounded by fully transparent pixels 895 (of which only a few are shown), which is appropriate in the case that the other pixels 894 of the window 884 are also fully transparent pixels. In the case that the window 884 is configured as a semi-transparent window and the pixels 894 have a certain color (e.g., light gray), then the black arrow is preferably surrounded by semi-transparent pixels 895 having the same color (e.g., light gray) . In the case that the window 884 is configured as a semi-transparent perforated bitmap or a semi-transparent perforated texture bitmap as described in more detail in the pending perforated bitmap application, the black arrow is preferably also surrounded by the same or similar perforated bitmap. In the latter case, the object repositioning routine will preferably locate the movable object 806L so that the perforated background of the movable image is aligned with the perforated bitmap of the window 884, which in the case of a 2x2 bitmap with a checkerboard pattern can be implemented by position the movable object 806L, for example, only at locations with even X and even Y coordinates, or only at locations with odd X and odd Y coordinates, depending on where or how the pattern of the pixels 894 is located.

FIG. 8 (a) shows movements of the first visible object 805 in accordance with movements of the first pointing device 803R.

FIG. 8 (b) shows movements of the second visible object 806 in accordance with movements of the second pointing device 803L. For reasons that will further become apparent, the left mouse device 803L should only be moved, but not clicked or scrolled in the system 800. For that reason, the representation of the second visible object 806 in the form of an arrow can be confusing for an arrow. user, and a line or a rectangle or a square may be preferred as a visible indication of the second position, since it is intuitively clear that a line or square should not be clicked. In the system 800, the native mouse cursor 805 follows movements of the right mouse device 803R, and is therefore probably better suited for right-wing people.

FIG. 9 shows a computer system 900 according to an embodiment of the present invention that can be seen as a variant of the computer system 800 shown in FIG. 8, wherein the native mouse cursor 905 follows movements of the left mouse device 903L. The embodiment of FIG. 9 is therefore probably better suited for left-handed people. In this embodiment, the black arrow 906 can be moved by the right pointing device 903R, but this pointing device 903R should not be clicked or scrolled.

FIG. 9 (a) shows that the position of the black arrow 906 is adjusted in accordance with movements of the right pointing device 903R.

BE2017 / 5896

FIG. 9 (b) shows that the position of the native mouse cursor 905 is adjusted in accordance with movements of the left pointing device 903L. Both arrows are visible on the screen at all times.

FIG. 9 (c) shows an exemplary position updating routine as part of a message processing routine, in pseudo-code, that can be used to obtain this behavior. As can be seen, the first and second positions Pos1, Pos2 are updated in accordance with movements of the first and second pointing devices 903R, 903L, respectively. In the case that the left pointing device 903L was moved, the position of the native mouse cursor 905 is adjusted accordingly. In the case that the right pointing device 903R was moved, the position of the native mouse cursor 905 is forced back to the second position Pos2 maintained by the overlay application. Thus, the two arrows 905, 906 are individually movable through the two pointing devices.

FIG. 10 shows a computer system 1000 according to another embodiment of the present invention, which can be seen as a variant of the computer system 700 of FIG. 7 for aspects related to positioning of the native mouse cursor 1005, and which can be seen as a variant of the computer system 800 of FIG. 8 and 900 of FIG. 9 for aspects related to the shape of the second visible object 1006.

The computer system 1000 includes a computer device 1001 connected to a first pointing device 1003R and to a second pointing device 1003L and connected to a display screen 1004. The computer device 1001 has at least one processor that performs an O / S and GUI performing a native mouse pointer / cursor 1005 provided (e.g., a white arrow), and executing an overlay application according to an embodiment of the present invention. The overlay application maintains a first and a second pointer position Pos1, Pos2 in accordance with movements of the first and second pointing devices 1003R, 1003L, and adjusts the position of the native mouse pointer / cursor 1005, e.g., overwrites it, in accordance with movements of the pointing device that was most recently moved.

FIG. 10 (a) shows movements of the first visible object, being the native mouse pointer 1005, in accordance with movements of the first pointing device 1003R because this is the pointing device that is being moved, while the second visible object 1006 (e.g., a black arrow) assume another maintained position Pos2. In this situation, the user can press / release / click the buttons and / or move the scroll wheel of the first mouse pointer 1003R, for example to activate the text editor application, select text, add comments, etc.

FIG. 10 (b) shows movements of the first visible object 1005 in accordance with movements of the second pointing device 1003L, which is the pointing device that moved

BE2017 / 5896, while the second visible object 1006 (e.g., a black arrow) becomes the other maintained position Pos1. In this situation, the user may press / release / click the buttons and / or move the scroll wheel of the second mouse pointer 1003L, for example to activate the audio player application 1017, click on the play or pause button, etc.

As can be understood from FIG. 10 (a) and FIG. 10 (b), the user can see both locations on the screen 1004, and can operate at two different locations on the screen, albeit not simultaneously, merely by moving the respective pointing device. The user can click on any mouse device after making sure that the native mouse cursor 1005 is at the desired maintained position Pos1 or Pos2, as visually indicated by the white arrow.

FIG. 11 shows a specific example of the simplified high level block diagram of hardware and software components shown in FIG. 5 (f) in the case that the pointing devices M1, M2 are HID-compatible devices (abbreviation for the Human Interface Device), and in the case that the computer device comprises an HID interface with at least two connection ports, as is typically the case among other things for computer devices that contain USB interfaces. As is known in the art, such mouse devices send signals or messages to the computer device, e.g., in the form of HID packets, containing motion or displacement information dx, dy and / or information about a button being pressed or released, and / or information about a scroll wheel being turned. In fact, the communication between the computer device and the mouse devices is bidirectional, but the overlay application does not have to handle that portion of the communication, which is therefore not further discussed.

As schematically indicated by the asterisk *, only the overlay application 1151 is new in FIG. 11, all other components (without an asterisk) can be classical components, in particular the pointing devices M1, M2, the operating system O / S and GUI 1160, the mouse device drivers (1161) and the other applications 1162. This again illustrates that the overlay application can be used to extend or improve the functionality of an existing O / S and GUI that originally only provides a single native mouse cursor.

The mouse devices M1 and M2 can be wired mouse devices, or wireless mouse devices, or a combination of both.

FIG. 12 shows a variant of the high level block diagram of FIG. 11, wherein two conventional pointing devices M1, M2 are connected to the computer device via a traditional hub or adapter or dongle 1220 instead of being directly connected to the computer device. The hub 1220 can be a wired hub, or a wireless hub. The interface can be a HID-compatible interface, and the pointing devices can be HID-compatible devices, but that is not absolutely required for the present invention to work.

BE2017 / 5896

FIG. 13 shows some lines of pseudo-code illustrating a portion of a message processor capable of operating in one of three input modes, which are mentioned herein as follows:

Two-Handed (FIG. 13 (a)) as explained in FIG. 7 (c), wherein the native mouse cursor follows movements of the pointing device that was most recently moved, or

Right Hand (FIG. 13 (b)) as explained in FIG. 5 (g), wherein the native mouse cursor follows movements of the right pointing device, and

Left Handed (FIG. 13 (c)) as explained in FIG. 9 (c), wherein the native mouse cursor follows movements of the left pointing device.

Some embodiments of an overlay application according to the present invention may possibly implement only one of these input modes (e.g., hard coding).

Other embodiments of an overlay application according to the present invention may possibly implement two of these input modes, or all three of these input modes, and enable the user to choose which input mode to use, for example explicitly, as shown in FIG. 31 (a), by enabling a user to select a Dominant mouse, in which case, for example, the display modes of groups 2850 and 2870 (see, e.g., FIG. 28) can be hidden, or implicitly by the user to select a display mode of any group 2850 to 2870, in which case the application can change the corresponding input mode itself, if necessary.

FIG. 14 (a) is a schematic representation of an exemplary sequence of signals or messages from two classical mouse devices M1, M2, and shows how these signals are correctly handled by the message processing process shown in FIG. 13, which is assumed to be configured in the Two-handed input mode. In reality, many more signals or messages can be exchanged between the computer device and the mouse devices, but these are not shown in order not to overload the drawings. FIG. 14 only how the position of the native mouse cursor NC is updated, and does not explicitly mention how the position of the second and / or further visible object is updated, again not to overload the drawings.

At time t1, mouse device M2 is moved and sends a motion message mov2 to the computer device, and in response, the overlay application updates Pos2 and sets the position of the native mouse cursor NC to Pos2. Depending on where the native mouse cursor was located before fluorescent, this can occur to the user as a small movement or as a jump.

At time t2, mouse device M1 is moved and sends a motion message movl to the computer device, and in response, the overlay application updates Pos1 and adjusts the position

BE2017 / 5896 from the native mouse cursor NC to Pos1, which will typically occur to the user as a jump from Pos2 to Pos1.

At time t3, mouse device M1 is moved again and sends a motion message movl to the computer device, and in response the overlay application updates Pos1 and sets the position of the native mouse cursor NC to Posl, which will typically occur to the user as a small movement.

At time t4, mouse device M2 is moved and sends a motion message mov2 to the computer device, and in response, the overlay application updates Pos2 and sets the position of the native mouse cursor NC to Pos2, which will typically occur to the user as a jump from Posl to Pos2.

At time t5, mouse device M1 is moved and sends a motion message movl to the computer device, and in response, the overlay application updates Pos1 and sets the position of native mouse cursor NC to Posl, which will typically occur to the user as a jump from Pos2 to Posl.

FIG. 14 (b) shows the same sequence of signals or messages as that of FIG. 14 (a), and shows how an overlay application that the message processor of FIG. 13, when configured in the Right-Handed input mode, would update the NC position of the native mouse cursor. As can be seen, when the right pointing device M1 is moved, the first position Pos1 is updated, and when the second pointing device M2 is moved, the second position Pos2 is updated, but in all these cases the position of the native mouse cursor NC is set at the first position Pos1, associated with movements of the right pointing device M1.

FIG. 14 (c) shows the same sequence of signals or messages as that of FIG. 14 (a), and shows how an overlay application that the message processor of FIG. 13, when configured in left-handed input mode, would update the NC position of the native mouse cursor. As can be seen, when the right pointing device M1 is moved, the first position Pos1 is updated, and when the second pointing device M2 is moved, the second position Pos2 is updated, but in all these cases the position of the native mouse cursor NC is set at the second position Pos2, associated with movements of the left pointing device M2.

As can be understood from FIG. 14 (a) to FIG. 14 (c), movements of both mouse devices M1, M2 are correctly handled, even simultaneous movements of the two mouse devices M1, M2.

FIG. 15 is a schematic representation of another exemplary sequence of signals or messages from two classical mouse devices M1, M2, and shows how these

BE2017 / 5896 signals are correctly handled by the message processing process shown in FIG. 13, which is assumed to be configured in the Two-Hand input mode.

At time t1, mouse device M1 is moved and sends a motion message movl to the computer device, and in response, the overlay application updates Pos1 and sets the position of the native mouse cursor NC to Posl. Depending on where the native mouse cursor NC was located before fluorescent, this can occur to the user as a small movement or as a jump.

At time t2, mouse device M2 is moved and sends a motion message mov2 to the computer device, and in response, the overlay application updates Pos2 and sets the position of native mouse cursor NC to Pos2, which will typically occur to the user as a jump from Posl to Pos2.

At time t3, a button (e.g., a left button M2B1) of the second mouse device M2 is pressed, and the mouse device M2 sends a button-pressed message press2 to the computer device. The message processor of the overlay application receives the message, but since it is not a motion message, step h) of FIG. 5 (e) skipped, and does not update the overlay application Pos1 or Pos2, but allows them to handle the O / S and GUI pressing the M2B1 button.

At time t4, the button M2B1 is released from the second mouse device M2 and the second mouse device M2 sends a push-button release message release2 to the computer device, and again the overlay application does not change a position, but leaves it the O / S and the GUI handle releasing the button. As is known in the art, under certain conditions the O / S will consider pressing a button and releasing a button as a so-called mouse click.

At time t5, mouse device M1 is moved and sends a motion message movl to the computer device, and in response, the overlay application updates Pos1 and sets the position of native mouse cursor NC to Posl, which will typically occur to the user as a jump from Pos2 to Posl.

As can be understood from FIG. 15, the click on a button M2B1 of the second mouse M2 is correctly handled by the message processor of FIG. 13, configured in the Two-Hand input mode, despite movements of the first mouse M1 before and after the click.

However, tests with further sequences have shown that the message processor of FIG. 13 does not always provide the expected behavior. Further investigation has shown that the exemplary sequence of signals or messages shown in FIG. 16 is treated unsatisfactorily.

At time t1, mouse M1 is moved and sends a motion message movl to the computer device, and in response, the message processor updates the first position Pos1 and positions the native mouse cursor accordingly. Likewise at time t2.

BE2017 / 5896

At time t3, a button (e.g., a left button M1B1) of the first mouse device M1 is pressed and the mouse sends a button-pressed message pressi to the computer device, for which the message processor of FIG. 13 takes no action.

At time t4, the second mouse device M2 is moved and sends a motion message mov2 to the computer device, and the message processor of FIG. 13 the native mouse cursor NC to the second maintained position Pos2.

At time t5 the button M1B1 is released from the first mouse device M1 and the mouse device M1 sends a push-button release message to the computer device, but the O / S considers that this release takes place at Pos2, which is usually different from Pos1, which does not result in the expected click behavior of the first mouse M1, but in a drag behavior. This is not the intended behavior.

In a first attempt to solve this problem, the message processor of FIG. 13 modified by adding and maintaining two variables, a boolean variable Mlpressed as an indication of whether a button of the first mouse device M1 has been pressed or released, and a variable M2pressed as an indication whether a button of the second mouse device M2 has been pressed or released, and by not responding to the motion signal mov2 transmitted by the second mouse M2 at t4 during the time that a button of the first mouse M1 is pressed, as shown in FIG. 17, but for some reason this modified message processor did not provide the correct result either.

In a second attempt to solve this problem, the message processor of FIG. 13, but instead of ignoring the mov2 signal (as in FIG. 17), the message processor was modified to actively reposition the native mouse cursor NC at the same position as Pos1 where the native mouse cursor was at time t3 when the button was pressed. Tests have shown that such a message processor works for a click on a button of the first mouse M1, but does not work when the first mouse M1 is dragged. Instead of (at t4) freezing the Pos1 position at the same position where the native mouse cursor NC was located when the button was pressed (at t3), the position of the native mouse cursor NC was allowed to move together with movements of the first mouse M1, even when a button of the same device M1 was pressed.

FIG. 18 (a) shows a few lines of pseudo-code, illustrating a portion of an improved message processing routine to implement this behavior, (only a portion of the Two-Handed input mode is shown).

BE2017 / 5896

FIG. 18 (b) shows a variant of the routine of FIG. 18 (a), as can be used in overlay applications according to embodiments of the present invention, (again only a portion of the Two-Handed input mode is shown). The main differences with the routine of FIG. 18 (a) are that the second position Pos2 is updated even if a button of the first pointing device is pressed, and conversely, that the first position Pos1 is updated even if a button of the second pointing device is pressed. This routine will be further discussed when discussing FIG. 26 (a) and FIG. 26 (b).

FIG. 18 (c) shows a more complete version of a position updating routine, such as can be used in overlay applications according to embodiments of the present invention. This can be seen as a combination of the code portions shown in FIG. 5 (g) and in FIG. 9 (c) and in FIG. 18 (a).

FIG. 19 shows the same exemplary sequence of signals or messages of FIG. 16, but processed by the message processor of FIG. 18 (a). As can be seen, in response to the motion signal mov2 of the second mouse M2 at time t4, the improved message processor of FIG. 18 (a) the second position Pos2 at in accordance with the motion signal mov2, but actively repositions the native mouse cursor to Pos1, so that at time t5, the button M1B1 of the first mouse device M1 is deemed to be released at the same position Pos1 as where it was was pressed. The O / S interprets this as a click of the button M1B1 of the mouse device M1, which has been treated satisfactorily, despite the motion signal mov2 of the second mouse M2 at time t4.

It is noted that updating Pos2 to t4 is not required for the correct interpretation of the click command sent by the first pointing device M1, and may possibly be omitted. However, if omitted, the visible object (s) associated with the second position Pos2 will not move as long as a button of the first pointing device M1 is pressed, even if the second pointing device is moved. To make such movements possible, the position Pos2 is preferably updated even if a button of the first pointing device is pressed.

FIG. 20 shows a variant of the sequence of signals or messages of FIG. 19, and shows that dragging one mouse M1 works correctly even when the second mouse M2 is moved.

At time t1, the first mouse M1 is moved, the first mouse M1 sends a first motion message movl to the computer device, and in response, the overlay application Pos1 updates and positions the native mouse cursor NC on Posl.

At time t2, a button (e.g., a left button M1B1) of the first mouse M1 is pressed, the first mouse M1 sends a button-pressed message pressi to the computer device, and in response, the message processor sets an internal variable Mlpressed to TRUE .

BE2017 / 5896

At time t3, the second mouse M2 is moved, the second mouse M2 sends a second motion message mov2 to the computer device, and in response, the message processor Pos2 updates and positions the native mouse cursor NC on Pos1.

At time t4, the first mouse M1 is moved, the first mouse M1 sends a first motion message movl to the computer device, and in response, the overlay application Pos1 updates and positions the native mouse cursor NC on Posl.

At time t5 the button M1B1 of the first mouse device M1 is released, causing the O / S and the GUI to perform a drag operation, and in response the message processor sets the variable Mlpressed to FALSE.

As can be understood from FIG. 20, the dragging of the first mouse M1 is satisfactorily handled by the message processor shown in FIG. 18 (a) configured in the Two-Hand input mode, even in combination with movements of the second mouse M2.

Referring to FIG. 21, further testing was performed using the message processor of FIG. 18 (a) in the Two-Handed input mode, which revealed yet another error condition, represented by the exemplary sequence of signals or messages shown in FIG. 21, wherein:

At time t1, the second mouse M2 is moved, a motion message mov2 is sent, the second position Pos2 is updated taking into account the displacement values dx, dy, and the native mouse cursor is set to Pos2.

At time t2, the first mouse M1 is moved, a moving message mov1 is sent, the first position Pos1 is updated taking into account the displacement values dx, dy, and the native mouse cursor is set to Pos1.

At time t3, the first mouse M1 is moved again, another motion message movl is sent, the first position Pos1 is updated, and the native mouse cursor NC is set to the updated position Pos1.

At time t4, a button (e.g., a left button M2B1) of the second mouse device M2 is pressed, a button-pressed message press2 is sent, and the message processor of FIG. 18 (a) set the value M2pressed to TRUE, and change the position of the native mouse cursor NC to Pos2.

At time t5, the button M2B1 of the second mouse device M2 is released, a release message release2 is sent to the computer device, and in response, the message processor sets the variable M2pressed to FALSE.

Despite the fact that a button of the second mouse device M2 was pressed and released at the same physical position, the press and release messages of FIG. 21 not interpreted as a click. Although the inventors do not wish to be bound by theory, analysis seems to indicate this

BE2017 / 5896 that this sequence goes wrong because at the time t4 at which the button M2B1 of the second mouse M2 is pressed, the native mouse position NC is still at the first position Pos1 (x1, yl), and therefore it will be O / S and the GUI assume that the button press takes place at the position of Pos1 (x1, yl) instead of Pos2, and only some time later (then t4) will the overlay application reposition the native mouse cursor NC at the second position Pos2 ( x2, y2), but that's too late. This problem is not known in the prior art because there is only one cursor position.

No technical solution to this problem seems possible, because it is impossible for the overlay application to predict when a M2 mouse button will be pressed, and because the O / S and GUI will always be faster than the overlay application. application. Desiring to keep the mouse driver layer 561, 1161, 1261 of the O / S and the GUI intact, the inventors surprisingly came up with a different solution to this problem.

Referring to FIG. 22 (a), the inventors decided to provide a special mouse device M2 * which, when a button thereof is pressed, sends a dummy movement signal some time ΔΤ before sending an actual button-pressed signal. This effectively circumvents the problem of FIG. 21, because, as can be understood from FIG. 22 (a):

At time t1 the second mouse M2 * is moved, a (real) motion message mov2 is sent to the computer device, in response the overlay application updates the second position Pos2 and sets the native mouse cursor NC to position Pos2.

At time t2, the first mouse M1 * is moved, a (real) motion message mov1 is sent to the computer device, in response, the overlay application updates the first position Pos1 and sets the native mouse cursor NC to position Pos1.

At time t3 a button (e.g. a left button M2B1) of the special second mouse device M2 * is pressed, which causes the mouse device M2 * to first send a dummy movement message dmov2 (preferably with small displacement values dx, dy), and for some time ΔΤ sends a button-pressed message press2 later. As shown, in response to the dummy movement signal dmov2, the overlay application will set the native mouse cursor NC to the second position Pos2, well before the button-pressed message press2 is sent by the mouse device M2 * at time Ϊ3 + ΔΤ, such that the O / S and the GUI will assume that the button depression takes place at position Pos2 (or more precisely: very close to the original position Pos2, if dx and dy are small, but not exactly zero).

At time t4 the button M2B1 is released, and in response the message processor of the overlay application sets a variable M2pressed to FALSE.

It is noted that a dummy motion message is not required when the button is released, only when a button is pressed.

BE2017 / 5896

FIG. 22 (b) shows a variant of the sequence of FIG. 22 (a) wherein a scroll wheel of the second pointing device M2 * is rotated instead of a button being pressed and released. In a similar manner as described in FIG. 22 (a), the special pointing device M2 * does not merely send a scroll wheel rotation message, but first sends a dummy movement message dmov2, and only some time ΔΤ later does the pointing device M2 * send the actual scroll wheel rotation message rotate2, to possibly causing the overlay application to reposition the native mouse cursor NC before the actual scroll wheel rotation message arrives at the computer device,

In FIG. 22 (b), a scroll wheel of the pointing device M2 * is physically rotated at time t3, at which the pointing device M2 * sends a dummy movement message dm2 to the computer device. In response, the overlay application (configured in Two-Handed input mode) will update the second position Pos2 and set the native mouse cursor to Pos2.

At time Ϊ3 + ΔΤ, the pointing device M2 * sends a scroll wheel rotation message rotate2, which the overlay application can simply ignore, because this message will be handled by the O / S and the GUI.

Although the solution presented in FIG. 22 (a) and FIG. (B) requires a special mouse device, it is an advantage that this special mouse behavior does not require any hardware modification, but only a firmware modification of the mouse device.

It is also an advantage that the proposed solution is retro compatible, in the sense that this special mouse with the move-before-click function and / or with the move-before-scroll function can also be used in traditional computer systems that do not have special have overlay application. The main difference with existing mouse devices is that the button-pressed signal and / or the scroll wheel rotation signal is slightly slowed down, which is probably totally unacceptable in the gaming world, and for that reason alone is not evident, but this causes no significant problem for professional users such as translators, proofreaders, transcribers, technical writers, lawyers, etc. who are used to working with tools (eg CAT tools, local or online dictionaries, local or online data files, local or online translation memories) , etc.) which in any case have rather slow response times (typically in the order of at least 500 ms).

There is some design freedom in the choice of the three parameters ΔΤ, dx, dy.

In one embodiment, the delay ΔΤ is a value in the range of 1 ms to 250 ms or from 2 ms to 100 ms, and dx is a value in the range of -5 to +5, and dy is a value in the range of -5 to +5.

In one embodiment, the delay ΔΤ is a value in the range of 1 ms to 250 ms or from 2 ms to 100 ms, and dx is a value in the range of -2 to +2, and dy is a value in the range of -2 to +2.

BE2017 / 5896

In one embodiment, the delay ΔΤ is a value in the range of 1 ms to 250 ms or from 2 ms to 100 ms, and dx is a value in the range of -1 to +1, and dy is a value in the range of -1 to +1.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = -l and dy = -l.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = -1 and dy = 0.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = -l and dy = +1.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = 0 and dy = -1.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = 0 and dy = 0.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = 0 and dy = + 1.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = + 1 and dy = -1.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = + 1 and dy = 0.

In one embodiment, the delay ΔΤ is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = + 1 and dy = + 1.

In all of the above embodiments, the value of ΔΤ can be, for example, at least 1 ms, or at least 2 ms or at least 8 ms, or at least 16 ms, or at least 24 ms, or at least 32 ms, often at least 40 ms, often at least 48 ms, often at least 56 ms, often at least 64 ms, or at least 72 ms, or at least 80 ms, or at least 88 ms, or at least 96 ms, or at least 104 ms, or at least at least 120 ms, or at least 144 ms, or at least 160 ms, or at least 176 ms, or at least 192 ms.

For example, in all of the above embodiments, the value of ΔΤ may be about 8 ms or about 16 ms, or about 24 ms, or about 32 ms, or about 40 ms, or about 48 ms, or about 56 ms, or about 64 ms, or about 72 ms, or about 80 ms, or about 88 ms, or about 96 ms, or about 104 ms, or about 112 ms, or about 120 ms, or about 128 ms, or about 136 ms, or about 144 ms, or about 152 ms, or about 160 ms, or about 168 ms, or about 176 ms, or about 184 ms, or about 192 ms, or about 200 ms, or about 208 ms, or about 216 ms, or about 224 ms, or about 232 ms, or about 240 ms, or about 248 ms.

BE2017 / 5896

On the one hand, the value of ΔΤ should be large enough to increase the likelihood that the computer device has sufficient time for the dummy movement signal to be processed by the overlay application before the button-pressed signal arrives. On the other hand, the value of ΔΤ should be as small as possible in order to reduce the delay, and to reduce the risk that a real motion signal or a dummy motion signal from the other mouse device might come in between. The skilled person can find a suitable compromise.

In one embodiment, the dummy motion signal or message contains the displacement values dx = 0 and dy = O. If this message is not filtered out by the O / S and the GUI, such a message would be ideal, because it does not adjust the coordinates of the first and second position Pos1 or Pos2, but still allows for the overlay application to signaling to reposition the native mouse cursor (or in other words: it allows the overlay application to force a jump if necessary). If the dummy motion message has a zero displacement (dx = 0 and dy = 0), the native mouse cursor NC will not shift or drift due to multiple dummy motion messages being sent, regardless of how many times the button is clicked, and / or the scroll wheel becomes scrolled.

In another embodiment, the dummy motion message is not a single predetermined message with dx = 0 and dy = 0, but subsequent dummy motion messages are different from the first dummy motion message, with dx and dy values selected such that a cumulative horizontal displacement defined as Δχ = ^ dx and a cumulative vertical displacement defined as Ay = ^ dy are smaller than a predetermined maximum distance. Preferably, the predetermined maximum distance is less than 10 pixels, more preferably, less than 4 pixels, or less than 3 pixels, or less than 2 pixels.

For example, the dummy motion messages can be alternately or periodically selected from a group consisting of two predetermined dummy motion messages, including a first message with (dx = + 1, dy = 0) and a second message with (dx = -1, dy = 0), which leads to a small left / right movement, but other groups of two messages are also possible, for example:

- the group consisting of two dummy movement messages containing a first message with (dx = 0, dy = + 1) and a second message with (dx = 0, dy = -1), which leads to a small up / down movement, or

- the group consisting of two dummy movement messages containing a first message with (dx = + 1, dy = + 1) and a second message with (dx = -1, dy = -1), which leads to a small diagonal movement, or

- the group consisting of two dummy movement messages containing a first message with (dx = +1, dy = -1) and a second message with (dx = -1, dy = +1), which leads to a small diagonal movement, or

BE2017 / 5896

- the group consisting of three dummy motion messages containing a first message with (dx = + 1, dy = 0), and a second message with (dx = 0, dy = + 1), and a third message (dx = -1) , dy = -1), or

- the group consisting of four dummy movement messages, containing a first message with (dx = + 1, dy = 0), a second message with (dx = 0, dy = -1), a third message with (dx = -1) , dy = 0), and a fourth message with (dx = 0, dy = + 1), etc.

In this way, the mouse position remains substantially stationary, even after multiple clicks and / or scroll wheel rotations, and does not drift away from the original position unless the pointing device is physically moved.

FIG. 23 (a) shows a simplified high-level block diagram of hardware and software components located in an exemplary computer system according to an embodiment of the present invention, comprising a computer device 2301 with a special overlay application 2351 according to an embodiment of the present invention, and two special mouse devices M1 *, M2 * according to an embodiment of the present invention. The special mouse devices M1 *, M2 * have the aforementioned move-before-click function shown in FIG. 22 (a) and / or the aforementioned move-before-scroll function shown in FIG. 22 (b), which means that these mouse devices M1 *, M2 *, upon detection of a button that is pressed and / or a scroll wheel that is turned, first send a dummy movement message and only a time ΔΤ later a button-pressed message or a scroll wheel rotation message.

As suggested by the asterisks, only the overlay application 2351 and the mouse devices M1 *, M2 * are special. All other components, specifically the O / S and the GUI 2360, the device drivers 2361, the other applications 2362, etc. can be classic components.

The mouse devices M1 * and M2 * can be wired mouse devices, or wireless mouse devices, or a combination of both.

But the present invention is not limited to special mouse devices, and the proposed solution can also be applied to other pointing devices that have at least one button and / or at least one scroll wheel.

FIG. 23 (b) shows an exemplary block diagram of a special pointing device according to an embodiment of the present invention, as may be used in the system of FIG. 23 (a) or FIG. 24 (which will be discussed below). The pointing device 2303 comprises at least:

- a motion detection mechanism 2321 for detecting a two-dimensional movement;

- a control unit 2324 connected to said motion detection mechanism 2321;

- at least one button BI can be pressed and released by a user;

BE2017 / 5896

- a button detection mechanism 2322 for detecting whether said at least one button B1 is pressed or released;

- wherein the control unit 2324 is connected to the button detection mechanism 2322;

- wherein the control unit 2324 is adapted for (e.g., programmed for):

* when a motion is detected, to send a motion message mov2;

* when it is detected that the at least one button B1 is pressed, for sending a dummy movement message dmov2, and for sending a button-pressed message press2 a predetermined period ΔΤ after sending the dummy movement message dmov .

The motion message may include displacement parameters dx, dy related to the two-dimensional motion.

The pointing device 2303 may further comprise at least one scroll wheel W1 rotatable by a user, and a scroll wheel detection mechanism 2323 for detecting whether the at least one scroll wheel W1 has been turned, and the control unit 2324 may be connected to the scroll wheel detection mechanism 2323, and may further adapted, if it is detected that the at least one scroll wheel has been rotated, to send a dummy movement message by means of, and to send a scroll wheel rotation message a predetermined period ΔΤ after sending the dummy movement message dmov.

The pointing device 2303 may further comprise an RF transceiver 2325 connected to the control unit 2324, and the control unit 2324 may further be configured to send messages to a computer device 2301, and to receive messages from the computer device 2301 via said RF transceiver 2325, for example via a dongle 2311 connected to the computer device 2301.

The control unit 2324 can be a programmable control unit, e.g. a microprocessor.

FIG. 24 shows a variant of the high level block diagram of FIG. 23 (a), wherein the two special pointing devices M1 *, M2 * are connected to the computer device 2401 via a conventional hub or adapter or dongle 2420 instead of being directly connected to the computer device. The hub 2420 can be a wired hub, or a wireless hub.

FIG. 25 shows a variant of the high level block diagram of FIG. 24, wherein in FIG. The problem shown in Fig. 21 is solved in a special hub 2520 instead of in special pointing devices. Thus, the computer device 2501 of FIG. 25 preferably includes a classic O / S and GUI 2560, classic applications 2562, classic device drivers (device drivers) 2561,

BE2017 / 5896 for example, mouse drivers, classical pointing devices M1, M2, but a special overlay application 2551 and a special hub or adapter or dongle 2520.

The functionality of a special hub 2520 will be explained in more detail in FIG. 26 (a) to FIG. 26 (d). Exemplary hardware block diagrams to implement this functionality will be described in FIG. 27 (a) and FIG. 27 (b).

As can be understood from FIG. 21 to FIG. 22 (b), the hub H * should prevent the computer device from receiving a button-pressed message from pointing device M2 preceded in time by a message from the other pointing device M1 (e.g., a motion message, a button-click) message, a push-button release message, a scroll wheel message), and vice versa. Since the messages from both pointing devices must pass through the hub H *, the hub is able to prevent this situation.

In the exemplary sequence of FIG. 26 (a), this situation is avoided as follows.

At time t1, the first pointing device M1 sends a moving message movl, and the hub H * sends the moving message to the computer device (also called host).

At time t2, a button M2B1 of the second pointing device M2 is pressed, and the pointing device M2 sends a button-pressed message to the hub H *. The hub H * intercepts this message, and sends a dummy motion message dmov2 to the host (preferably with small motion parameters as explained above), preferably formatted as if this dummy motion message was sent by the second pointing device M2.

Some time later, at time t2 + AT, the hub 2520 sends the button-pressed message to the host, formatted as if this message came from the second pointing device M2.

In the embodiment shown in FIG. 26, the hub H * ignores all messages sent by the other pointing device M1 until the button M2B1 of the pointing device M2 is released at time t5. Therefore, in the example shown, the motion message mov1 sent by the pointing device M1 at t3 is ignored.

At time t4, the second pointing device M2 sends a push button release message to the hub H *, and the hub H * sends this message to the host.

At time t5, the first pointing device M1 sends a motion message to the hub H *, and the hub H * sends this message to the host.

In other words, the special hub H * that controls the behavior of FIG. 26 (a), transmits all messages coming from the pointing devices to the host, except when a button-pressed message is received from one of the pointing devices (in the example M2), in which case the hub H * temporarily moves to a special behaves, namely by first sending a dummy movement message, and for some time ΔΤ later sending a button-pressed message press2, and by ignoring all messages from the other pointing device (in the example: M1) to a

BE2017 / 5896 push button-release message is received from the pointing device M2 whose button M2B1 was pressed.

FIG. 26 (b) shows the same sequence as FIG. 26 (a), but treated differently by a variant of the special hub H *. In this embodiment, the hub does not ignore the movl message sent on t3, but delays this message until after the button-pressed press2 message is sent to the host on t2 + AT, say ΐχ = ΐ2 + ΔΤ + ε, with ε preferably is less than 33 ms, for example about 8 ms or about 16 ms or about 24 ms. As explained above, the special overlay application can optionally update Posl, but needs some time to reposition the native mouse cursor on Pos2, otherwise the O / S can interpret the button as being released on Posl instead of Pos2. Therefore, in this example, the push button release message sent by the second pointing device M2 on t4 must also be delayed to tx + ΔΤ. Of course, if t4 was already larger than tx, then the push-release message release2 can easily be forwarded to the host.

If the behavior of the signals and how the overlay application will respond to it, shown in FIG. 26 (b), it is well understood that the position updating routine of FIG. 18 (b), which allows the second object to be moved while a button of the first pointing device is pressed, may go wrong for the sequence shown in FIG. 26 (b), in case the user releases the M2B1 button before tx + ΔΤ. The in FIG. 18 (a) does not suffer from this problem because it ignores the motion signal in much the same way as the hub of FIG. 26 (a) does this.

Thus, when the special hub H * is used, both the position updating routine of FIG. 18 (a) as well as those of FIG. 18 (b) both are used in the Two-Hand input mode, because the hub provides the timing between the messages from different devices. But if the dedicated hub is not used in the computer system, the position updating routine is shown in FIG. 18 (a) more secure.

FIG. 26 (c) shows an exemplary sequence to explain how the special hub H * shown in FIG. 26 (a) has specified behavior, may respond in response to a mouse wheel message.

At time t1, the first pointing device M1 sends a moving message movl, and the hub H * sends the moving message to the computer device (also called host).

At time t2, a scroll wheel M2W1 is rotated from the second pointing device M2, and the pointing device M2 sends a scrolling wheel rotation message to the hub H *. The hub H * intercepts this message, and sends a dummy motion message dmov2 to the host (preferably with small motion parameters as explained above), preferably formatted as if this dummy motion message was sent by the second pointing device M2.

BE2017 / 5896

Some time later, at time t2 + AT, the hub H * sends the scroll wheel rotation message to the host, formatted as if it came from the second pointing device M2.

In the embodiment shown in FIG. 27, the hub H * ignores all messages sent by the other pointing device M1 to the hub H * arriving between t2 and Ϊ2 + ΔΤ. Therefore, in the example shown, the motion message mov1 sent by the pointing device M1 at t3 is ignored.

At time t4, the first pointing device M1 sends a motion message movl to the hub H *, and the hub H * sends this message to the host.

At time t5, the second pointing device M2 sends a motion message to the hub H *, and the hub H * sends this message to the host.

FIG. 26 (d) shows the same sequence as FIG. 26 (c), but treated differently by a variant of the special hub H *. In this embodiment, the hub H * does not ignore the movl sent message at t3, but delays this message until after the scroll wheel rotation message scroll2 has been sent to the host at Ϊ2 + ΔΤ, say ΐχ = ΐ2 + ΔΤ + ε. The motion message mov1 on t4 sent by the first pointing device M1, and the motion message mov2 sent by the second pointing device M2 are simply sent to the host.

When comparing the behavior of the special hub H * as specified in FIG. 26 (a) to FIG. 26 (d) with the behavior of the special pointing devices, as specified in FIG. 22 (a) and FIG. 22 (b), it can be seen that the special hub H * addresses the problem of FIG. 21 approaches ΔΤ on the basis of the same principle, namely by inserting a dummy movement message before transmitting the actual button-pressed message or scroll-wheel rotation message. Implementing this behavior in a special hub instead of in special pointing devices provides the additional advantage that the hub H * can actually prevent the other pointing device from sending an intermediate message to the host between this dummy motion message and the actual button-pressed or scroll wheel rotation message, and therefore can guarantee that the situation of FIG. 21 cannot occur. This is not possible with the special pointing devices.

Or expressed in other words, although the move-before-click and move-before-scroll function presented in FIG. 22 reduces the risk that the computer device will misinterpret the messages, the risk is not completely eliminated, because one pointing device has no influence over the messages sent by the other pointing device. In contrast, a special hub has H * with the features described in FIG. 26 (a) to FIG. 26 (d) complete control over the messages sent to the computer device, and can filter and / or adjust the flow of messages to and from both pointing devices. In this way, unwanted combinations of conditions or unwanted timing between the signals / messages can be avoided or

BE2017 / 5896, so that the O / S and the GUI and the special overlay application 2551 will function correctly.

Smart hubs that interpret and / or modify messages from multiple pointing devices and / or insert and / or delay to make double mouse support on a computer device in general, let alone facilitate double mouse support provided by an overlay application that is performed on a computer device with an O / S and GUI that provides only a single native mouse cursor, to reposition the native mouse cursor, to provide an O / S and GUI that provides virtual double mouse support, do not exist in the prior art .

FIG. 27 (a) shows an exemplary block diagram of a hub or dongle or adapter according to an embodiment of the present invention, as may be used in the system of FIG. 25, which has at least two physical connectors CON1, CON2 for connection to two conventional wired indicating devices M1, M2, and at least one connector CON3 (optionally at the end of a cable) for connection to a computer device 2701.

FIG. 27 (b) shows an exemplary block diagram of a wireless hub (or dongle or adapter) according to an embodiment of the present invention, as may be used in the system of FIG. 25, which has one or two RF transceivers for operative connection to two conventional wireless pointing devices.

The hub 2720 of FIG. 27 (a) includes:

- a first interface CON1 for communicating with a first pointing device M1;

- a second interface CON2 for communicating with a second pointing device M2;

- a third interface CON3 for communicating with a computer device 2701;

- a control unit 2744 which is connected to the first interface for receiving first messages from the first pointing device M1, and which is connected to the second interface for receiving second messages from the second pointing device M2, and which is connected to the third interface for sending the first and second messages to the computer device (2701);

- and wherein the control unit 2744 is adapted to send a dummy movement message a predetermined period (ΔΤ) before sending a button-pressed message and / or a scroll wheel rotation message.

More specifically, the control unit 2744 can be adapted to behave as follows:

* upon receipt of a first button-pressed message pressi from the first pointing device via the first interface, to send a first dummy movement message by means of dll to the

BE2017 / 5896 computer device via the third interface, and to press the first button-pressed message lateri a predetermined period Δ press to the computer device via the third interface;

* upon receipt of a second button-pressed message press2 from the second pointing device via the second interface, to send a second dummy movement message dmov2 to the computer device via the third interface, and to a predetermined period ΔΤ later, the second button send printed message press2 to the computer device via the third interface;

* upon receipt of a first scroll wheel rotation message from the first pointing device via the first interface, to send a first dummy movement message through the third interface to the computer device, and to a predetermined period ΔΤ later, the first scroll wheel send rotation message to the computer device via the third interface;

* upon receipt of a second scroll wheel rotation message from the second pointing device via the second interface, to send a second dummy movement message dmov2 to the computer device via the third interface, and to a predetermined period ΔΤ later, the second scroll wheel send rotation message to the computer device via the third interface.

The hub 2721 of FIG. 27 (b) includes:

- a first RF transceiver RFX1 for receiving messages from and sending messages to the first pointing device M1; and

- a second RF transceiver RFX2 for receiving messages from and sending messages to the second pointing device M2, wherein the second RF transceiver is the same as the first RF transceiver, or is a separate RF transceiver; and

- a connector CON3 that can be connected to a computer device 2701;

a control unit 2745 which is connected to the first RF transceiver RFX1 for receiving first messages from the first pointing device M1, and which is connected to the second RF transceiver RFX2 for receiving second messages from the second pointing device M2, and connected to the connector CON3 for sending messages to (and typically also receiving messages from) the computer device 2701, wherein the control unit is adapted to send a dummy movement message a predetermined period ΔΤ before sending a button pressed message to the computer device and / or for sending a dummy movement message a predetermined period (ΔΤ) before sending a scroll wheel rotation message to the computer device.

What has been described above with regard to the parameters dx, dy, ΔΤ when discussing special pointing devices is not repeated here, but applicable to the special hub.

The control unit 2744, 2745 can be a programmable control unit, e.g. a microprocessor.

BE2017 / 5896

FIG. 28 (a) shows an exemplary user interface window, referred to herein as the mini user interface, as may be used in embodiments of the present invention to configure the overlay application, e.g., to select an input mode and / or a display mode, and / or choose a size and / or a color from the second or further visible object, and / or choose a semi-transparency level of the click-through window using one or more pop-up menus, an example of which is shown in FIG. 28 (b).

The user interface window, e.g., window 585 in FIG. 5c or window 885 in FIG. 8c, is preferably opaque and is not configured in click-through mode, and preferably has a Z order higher than that of the click-through window 584, 884 in FIG. 5c and FIG. 8c. The pop-up menu of FIG. 28 (b) may, for example, appear when clicking a right button of the dominant pointing device when the native mouse cursor is positioned above one of the icons representing a specific display mode.

The overlay application can implement one or more input modes selected from the group consisting of: a One Handed Pointing Mode (referred to as OneH in FIG. 28a), a Double Pointing Mode where the left pointing device is the dominant pointing device (referred to as LeftH in Fig. 28a), a Dual Pointing mode where the right pointing device is the dominant pointing device (referred to as RightH in FIG. 28a), and a Double Pointing Mode where both pointing devices are dominant (referred to as TwoH in FIG. 28a).

In an embodiment of the present invention that is configurable in one of several input modes, a message processor of the overlay application may use the pseudo code of FIG. 18 (c). It is noted that testing whether a button of the dominant pointing device is depressed or not is not required in the case of the Right Hand and Left Hand input mode, since in these cases the native mouse cursor does not jump from one maintained position Posl to another, Pos2. Thus, for example, in the case of the Legal input mode, a button of the right pointing device can be safely pressed and moved (e.g., when dragging or selecting text in a word processor) while the left pointing device is also moved, without conflict (assuming that no button of the left pointing device is pressed, and that the scroll wheel of the left pointing device is not turned).

It is noted in this regard that some embodiments of the special hub have the behavior shown in FIG. 26 (a) and FIG. 26 (c), or as shown in FIG. 26 (b) and FIG. 26 (d) can implement, assuming that both pointing devices may be moved and / or clicked and / or scrolled.

BE2017 / 5896

Other embodiments of the special hub can unconditionally ignore all button-pressed messages and all push-button released messages and all scroll wheel rotation messages received through one of its input ports, and only allow motion messages to pass. Such an embodiment may be particularly useful in combination with a Left Handed or Right Handed input mode. In this way it is guaranteed that such message sent by the non-dominant pointing device cannot reach the computer device even if the user accidentally clicks or scrolls the non-dominant pointing device.

Some embodiments of the special hub are configurable, e.g., via dip switches or in software, e.g. via a command from the overlay application, or otherwise, to allow either (i) to allow all motion and impression and release and scroll wheel rotation messages from all pointing devices, or (ii) allowing passage of motion messages from all pointing devices, but only allow impression and release and scroll wheel messages from a single pointing device.

The overlay application can implement one or more display modes, for example as represented by the icons shown in FIG. 28 (a).

FIRST GROUP:

A first group of 2840 display modes works with the One-Hand Point input mode, and offers the following display modes, which can be summarized as follows:

- icon 2840 (a) represents a mode with the single (native) mouse pointer provided by a classic O / S and GUI,

- icons 2840 (b) to 2840 (f) represent a mode with the single (native) mouse pointer, but also a second and / or third visible object in the form of (b) a hand; (c) a square or line segment or rectangle; (d) a line or rectangle that extends the entire width of the overlay window; (e) a small cross; (f) a large cross formed by a horizontal line or rectangle that preferably extends over the entire width of the overlay window 584, 884 and by a vertical line or rectangle that preferably extends over the entire height of the overlay window 584, 884;

- icon 2840 (j) representing a mode of a horizontal line or bar similar to mode 2840 (d), but with an additional visible object in the form of a small block e.g. a square or rectangle movable within the line or bar;

- icon 2840 (k) represents a mode with a vertical line that splits the screen into a left-hand section and a right-hand section, this vertical line is preferably movable by dragging (after temporarily setting the overlay window 584, 884 in non- click-through mode as explained in more detail in the pending perforated bitmap application, and with a horizontal line on the left-hand side of the vertical line, which horizontal line moves with the

BE2017 / 5896 mouse pointer when the mouse pointer is on the left side of the vertical line, and which horizontal line freezes when the mouse pointer is on the right side of the vertical line. This mode can be particularly useful for legal translators or reviewers who work with just a single mouse pointer;

- icon 2840 (1) represents a mode similar to mode 2840 (k) but the horizontal line is on the right side of the vertical line, and moves up / down when the native mouse pointer is on the right side of the vertical line, and freezes when the mouse pointer is on the left side of the vertical line. This mode can be particularly useful for left-handed translators or reviewers who work with just a single mouse pointer.

In all modes 2840 (a) - (1), the native mouse cursor is visible on the screen, and moves with the at least second visible object in the form of a hand or a line or a cross or a block as described above, except in the frozen line modes represented by pictograms 2840 (k) and 2840 (1), the visible object moving with the pointing device only under certain conditions.

SECOND GROUP:

A second group of 2850 display modes works with the Left Handed Dual Pointing Input Mode, and offers the following display modes, which can be summarized as follows:

icon 2850 (a) represents a mode with the single (native) mouse pointer provided by a classic O / S and GUI, and also a second visible object in the form of an arrow, e.g., a black arrow. The native mouse cursor is controlled by the left pointing device. Aspects of mode 2850 (a) were also described in FIG. 9;

icons 2850 (b) to 2850 (f) represent a mode with the single (native) mouse pointer controllable through the left pointing device, and with an additional second or further visible object in the form of (b) a hand; (c) a square; (d) a line or rectangle; (e) a small cross; (f) a large cross adjustable by the other pointing device, namely the right pointing device;

- icon 2850 (g) to icon 2850 (i) represents a mode similar to mode 2850 (c) to 2850 (e), but the overlay application further shows an additional (g) square or (h) line or (h) small cross that moves with the native mouse pointer, and is therefore movable through the left pointing device;

- icon 2850 (j) represents a mode in which the overlay application provides a vertical line that splits the screen into a left portion and a right portion, and with a first horizontal line located on the left side of the vertical line moving in accordance with movements of the left pointing device, and with a second horizontal line located on the right side of the vertical line moving in accordance with movements of the right

BE2017 / 5896 pointing device. The native mouse cursor is movable over the entire screen, and moves with the left pointing device. This mode can be particularly useful for left-handed translators or reviewers who compare two texts;

- icon 2850 (1) represents a variant of the mode 2850 (j) without showing the left horizontal line;

In all 2850 (a) - (l) modes, the native mouse cursor is visible on the screen and moves in accordance with left-pointing devices, which is why these modes are probably best suited for left-handed people who are native Control the mouse cursor with their dominant (left) hand, and move a second visible object with their non-dominant (right) hand. The user should not click or scroll with the non-dominant pointing device, but only move it (for example, to underline textual information on the screen).

THIRD GROUP:

A third group of 2860 display modes works with the Right Handed Dual Pointing Input mode, and provides display modes similar to the second group 2850 of display modes, except that the function of the left and right pointing devices are swapped, and that in mode 2860 (k) the left horizontal line becomes shown instead of the right horizontal line.

Aspects of mode 2860 (a) were also described in FIG.

Aspects of mode 2860 (d) were also described in FIG.

Aspects of mode 2860 (h) were also described in FIG.

In all 2860 (a) - (k) modes, the native mouse cursor is visible on the screen and moves in accordance with movements of the right pointing device, which is why these modes are probably best suited to legal-minded people who are native Control the mouse cursor with their dominant (right) hand and move a second visible object with their non-dominant (left) hand. The user should not click or scroll with the non-dominant pointing device, but only move it (for example, to underline textual information on the screen).

FOURTH GROUP:

A fourth group of 2870 display modes works with the Two-Handed Double Pointing Input mode, and offers display mode 2870 (a) with two visible objects: the native mouse cursor, e.g., as a white arrow, and a second arrow (e.g., black arrow), already described in FIG . 10

icon 2870 (b) represents a mode similar to mode 2850 (b), except that two bitmaps are shown with one hand, one movable with the left pointing device, the other movable with the right pointing device;

BE2017 / 5896

- icon 2870 (g) to icon 2870 (j) represents a mode similar to modes 2850 (g) to 2850 (j), except that the native mouse cursor will follow the position associated with the pointing device that was most recently moved , and / or whose button was pressed most recently, and / or whose scroll wheel was turned most recently, as explained above. These modes are particularly suitable for users who want to move both pointing devices and click and / or scroll, for example, the transcribers of FIG. 4, but also, for example, the translators or proofreaders of FIG. 1 who wish to scroll and / or edit through both documents, using both pointing devices.

As described above (see, e.g., FIG. 14 and FIG. 15 and FIG. 19 and FIG. 20), the modes of this fourth group 2870 work well even with conventional pointing devices M1, M2 provided that the pointing device being clicked and / or scrolled is the one that was moved most recently, which requires special attention from the user. A fairly simple solution would be to always move a mouse manually before clicking it. But as explained above, these modes work even better when using special pointing devices (see, e.g., FIG. 22) that automatically take care of moving before clicking and / or moving before scrolling, or with a special hub or adapter or dongle as described above, see e.g. FIG. 26 (a) to FIG. 26 (d).

FIG. 28 (b) shows an exemplary pop-up menu that can be accessed by clicking with a right button of the dominant pointing device on some of the display modes in the mini-interface window of FIG. 28 (a). In the example shown, the menus allow you to change the color and size of the second and / or further visible object (s), as well as to choose a transparency level of the click-through overlay window 584 (FIG. 5c) or 884 (FIG. 8c).

FIG. 29 shows another exemplary user interface window, referred to herein as the main user interface, as may be used in embodiments of the present invention to configure the overlay application.

The user interface (UI) window shown in FIG. 29 has an area with the title Mouse / Touchpad (English: Mouse / Touchpad) for selecting the input mode (e.g. Left-Handed or Right-Handed or Two-Handed), and for selecting which pointing device (or actually, which handle is assigned by the operating system) on each pointing device) should be considered as the dominant pointing device (or handle thereof). The UI also includes components such as, e.g., radio buttons for selecting relative speed of movement, etc. In FIG. 31 (a), this area is shown in an enlarged view.

The interface window shown in FIG. 29 also has an area with the title Background) (Dutch: Background), which allows the click-overlay window 584, 884 to

BE2017 / 5896 comprises at least one second visible object 506L, 806L, further comprises a bitmap comprising pixels 594, 894, the majority of which are fully transparent pixels, or semi-transparent monochrome pixels, or a semi-transparent textured bitmap as in more detail described in the pending perforated bitmap application.

The exemplary user interface window shown in FIG. 29 also has an area with the title Transparency, which makes it possible to select or choose an alpha transparency level from the overlay window 584. For example, if the background pixels 584 and 884 are fully transparent pixels, the degree of transparency of the movable visible object can be changed in this way. The user can choose a compromise between on the one hand showing an object that is very different from the background pixels, and on the other hand, that is sufficiently transparent to be able to read underlying textual information.

The user interface window shown in FIG. 29 also has an area with title Mode (Dutch: Mode), already described in FIG. 28. Depending on which display mode is selected, the main window can show an additional area such as the block area where the user can, for example, select a color, height, width and vertical shift.

The interface window shown in FIG. 29 also has an area with the title Examples, for using the display modes with default settings, but default settings can of course also be implemented differently. In FIG. 30, this area is shown in an enlarged view.

It is to be noted that embodiments of the present invention are by no means limited to this specific user interface.

FIG. 30 is an enlarged view of a portion of FIG. 29. It provides a visible overview of various possible embodiments of the present invention, however, the invention is not limited thereto. These standard modes largely correspond to the modes shown in the mini interface of FIG. 28 (a). Some of the modes are described in more detail in this document, as indicated by the Roman numerals referring to the respective figures.

FIG. 31 (a) is an enlarged view of a portion of FIG. 29. It shows, for example, how a user can choose an input mode, eg Left Handed; Righthanded; Two-handed.

It is noted that, for example, a touchpad of a laptop computer is also considered a pointing device, hence the indication # 3 in the title, meaning that the operating system of the computer device from which this screenshot was taken has found three input devices: two physical mouse devices and one touch pad.

BE2017 / 5896

It is noted that the present invention, for example, will also work with a laptop computer to which only a single physical mouse is connected, e.g., by using the touch pad as the non-dominant pointing device and the physical mouse as the dominant pointing device.

FIG. 31 (b) is a variant of the portion of FIG. 31 (a), wherein an option has been added to adjust the orientation of the displacement vector defined by the displacement values (dx, dy) to a preferred orientation selected from a first group of predetermined directions consisting of the directions: North, West, South, East, or a second group of predetermined directions consisting of the directions: North, North West, West, South West, South, South East, East, North East. This transformation can be applied to movements of only the dominant pointing device, or to movements of only the non-dominant pointing device, or to both pointing devices.

Tests have shown that this function can greatly help the user to better control movements of one or both pointing devices when performing certain tasks. This function may, for example, be particularly useful for moving the non-dominant hand-driven pointing device horizontally across a text line, e.g., when performing text-based tasks such as reading a text document, proofreading a translation, pointing to individual words on the same line during a Webinar or presentation, etc.

FIG. 31 (c) shows some lines of pseudo-code that can be used in the position update routine to transform the displacement values dx, dy to be located in an orientation of the second group of eight allowed orientations mentioned above (North, North-West, etc.). In the example of FIG. 31 (c),

- if dx = 0, this means that the movement was already vertical, therefore no adjustment is required;

- if dy = 0, this means that the movement was already horizontal, therefore no adjustment is required;

- if dx = dy, this means that the movement was already diagonal, therefore no adjustment is required,

- otherwise, it is determined whether the angle of the displacement vector is in the range of (-30 ° to + 30 °) or in the range of (150 ° to 210 °), in which case the absolute value of the tangent of said angle is less than 0.5, in which case the movement is forced to be horizontal, and / or it is determined whether the angle of the displacement vector is in the range of (60 ° to 120 °) or in the range of (240 ° up to 300 °), in which case the absolute value of the tangent of said angle is greater than 2.0, in which case the movement is forced to be vertical,

BE2017 / 5896 otherwise the movement is forced to be diagonal (oriented at + 45 ° or 135 ° or 225 ° or 315 °).

It is noted that trigonometric functions are not used in this routine, keeping the implementation fairly simple and fairly fast, and thereby keeping the CPU load rather low. The careful reader will recognize that the amplitude of the displacement vector may increase or decrease slightly due to the adjustment, in order to keep the transformation simple and fast. This effect is hardly noticeable.

Tests have shown that the transformation of FIG. (C) does not affect slow movements of the pointing device (for which dx and dy are very small). For a user, this feels like fairly slow movements can be made in any direction, which is desirable to enable fine tuning of a position on the screen. However, if the speed at which the pointing device is moved is increased, the amplitude of the displacement vector will become greater than a predetermined threshold value (in the example of FIG. 31c greater than 0), in which case the displacement is forced to move horizontally or vertically. or be diagonal. This behavior is highly desirable for keeping almost horizontal movements perfectly horizontal, which is difficult to achieve manually, especially at a relatively high speed.

But, of course, the present invention does not require this feature to be present (it can be enabled or disabled by a user). Furthermore, if present, the present invention is not limited to the specific routine or algorithm shown in FIG. 31 (c), and other algorithms for refocusing the pointer movements have also been considered, as described, for example, in the following variants.

FIG. 31 (d) and FIG. 31 (e) show a variant of the displacement transformation routine in which near-horizontal movements are transformed into purely horizontal movements.

More specifically, FIG. 31 (d) a graphical representation of the displacement vector V corresponding to the physical movement of the pointing device associated with the displacement values (dx, dy). The displacement vector V forms an angle α with respect to the X axis. If this angle is in the segments indicated in gray, which means that the angle a is in the range [-amax .. + amax] or in the range [180 ° -amax .. 180 ° + amax], then the displacement vector transformed into a purely horizontal displacement, by setting the value of dy equal to zero. Instead of working with angles, this can be codified by calculating the absolute value of the angle tangent, that is abs (dy / dx), and testing whether this value is less than, or less than or is equal to a predetermined constant. For example, amax = 30 ° corresponds to abs (dy / dx) <0.577, or if another example, testing whether abs (dy / dx) <0.5 corresponds to testing whether the angle α is in the range of about

BE2017 / 5896 [-26 ° to + 26 °] or about [154 ° to 206 °], but of course the present invention is not limited to only this range, and other ranges can also be used.

FIG. 31 (e) shows exemplary lines of pseudo-code implementing the above-described displacement transformation, which can be used in the position updating routine (see, for example, step g) of FIG. 5 (e), when updating Pos1 and Pos2). But of course a different angle range can be used, or in other words, the invention will also work with other values for MAXTAN, for example values in the range of 0.2 to 0.75, for example smaller than 0.25 = 1/4, or less than 0.33 = 1/3, or less than 0.5 = 1/2, or less than 0.66 = 2/3, or less than 0.75 = 3/4. The person skilled in the art having the advantage of the present invention can easily define other suitable ranges.

FIG. 32 shows a computer system 3200 according to an embodiment of the present invention, comprising a computer device 3201 according to an embodiment of the present invention which performs an overlay application according to an embodiment of the present invention, and two pointing devices, e.g., mouse devices 3203L, 3203R connected to the computer device, and optionally a keyboard 3202. The mouse devices may be conventional mouse devices, or may be special mouse devices with the move-before-click function and / or the move-before-scroll function as explained in FIG. 22 (a) and FIG. 22 (b). The mouse devices can be directly connected to the computer device (e.g., as shown in FIG. 11 and FIG. 23 (a)), or via a conventional hub or adapter or dongle (e.g., as shown in FIG. 12 and FIG. 24). or via a special hub (e.g., as shown in FIG. 25).

The overlay application used in the system of FIG. 32 allows the O / S and GUI (e.g., Windows 7 or Windows 10 from Microsoft Corporation, or mac OS Sierra or mac OS High Sierra from Apple Inc.) to display the native mouse cursor 3205, but adjusts its position, for example, it overwrites. The overlay application further provides a second visible object 3206L and a third visible object 3206R, in the example of FIG. 32 being two lines or rectangular areas. These rectangular areas can be opaque or semi-transparent. They can have a single color, or a color gradient, or can contain a perforated bitmap. The other pixels of the overlay window can be fully transparent pixels, or can be all semi-transparent pixels, or the overlay window can include a perforated bitmap or a texture bitmap as explained in the pending perforated bitmap application.

In the case that the overlay application is designed (e.g., hard-coded) for, or configured in, display mode 2860 (h), i.e., in Right Hand input mode, the object 3206L follows movements of the left pointing device 3203L (or actually the pointing device configured) or referred to as the left pointing device) and following the native mouse cursor 3205 and

BE2017 / 5896 the object 3206R moves the right pointing device 3203R (or actually the pointing device configured or referred to as the right pointing device).

In the case that the overlay application is designed (e.g., hard-coded) for, or configured in, display mode 2870 (h), i.e., in the Two-Hand input mode, then the item 3206L follows movements of the left pointing device 3203L (or actually the pointing device configured) or referred to as the left pointing device) and the object 3206R follows movements of the right pointing device 3203R (or actually the pointing device configured or referred to as the right pointing device), and the native mouse cursor 3205 will move and / or jump depending on which pointing device is most recent was moved and / or clicked and / or scrolled.

The embodiment shown in FIG. 32, and variants thereof, optionally using a wired or wireless hub or adapter or dongle, and optionally using a motion transformation as explained in FIG. 31 (b) and FIG. 31 (c), may be particularly useful for addressing the problem described in FIG. 1 and FIG. 3, wherein it is desirable to underline two text fragments.

FIG. 33 shows embodiments of the present invention that implement display modes 2850 (g), 2860 (g), and 2870 (g), wherein the O / S and GUI show the native mouse cursor 3305, the shape of which can and will typically change depending on the context, and the overlay application additionally shows two visible and movable squares 3306L, 3306R. Display modes 2860 (g) and 2870 (g) appear to be particularly suitable for each of the cases shown in FIG. 1 to FIG. 4 for legal people, while display modes 2850 (g) and 2870 (g) appear to be particularly suitable for similar problems as shown in FIG. 1 to FIG. 4, but for left-handed people.

FIG. 34 shows embodiments of the present invention implementing display mode 2870 (b), wherein the O / S and GUI show the native mouse cursor 3405 (in the example a white arrow), and the overlay application additionally shows two visible objects, a second object 3406L being or containing a bitmap or image of a left hand, and a third object 3406R being or containing a bitmap or image of a right hand. This mode, too, appears to be particularly suitable for all the functions shown in FIG. 1 to FIG. 4, in particular in combination with a motion transformation routine as explained in FIG. 31 (b) and FIG. 31 (c) to help the user to make horizontal movements easily, and especially if the hands are semi-transparent. Preferably in this case the position of the bitmap containing the image of the hand is slightly shifted relative to the position of the native mouse cursor 3405, such that the native mouse cursor is located at or near the end of the respective index fingers. Tests have shown that the two hands can be extremely useful in proofreading.

BE2017 / 5896

Although the present invention can also be used with bitmaps other than a hand, it is noted that the shape of a hand, particularly a hand with an extended index finger, feels very natural and intuitive, and the user's attention does not seem to be lost perhaps because it looks a lot like the real one, in FIG. 1 to FIG. 3 hand and finger shown. That being said, a triangular shape, or a polygonal shape, preferably with a pointed end at the top, can also be used.

FIG. 35 shows embodiments of the present invention implementing display modes 2850 (d) and 2860 (d), showing the native mouse cursor 3505 as a first visible and movable object, and one line 3506 or rectangle as the second visible and movable object. These modes can be particularly useful for approaching the problem shown in FIG. 2 and FIG. 3. It may be an advantage of this embodiment that there is no third item following the native mouse cursor that might be slightly distracting or slightly annoying when selecting menus or colors or the like.

FIG. 36 shows embodiments of the present invention implementing display modes 2850 (e) and 2860 (e), showing native mouse cursor 3605 as a first visible and movable object and one small cross 3606 as a second visible and movable object. These modes can also be particularly useful for approaching the problem shown in FIG. 2 and FIG. 3.

Display modes 2850 (j) and 2860 (j) showing a large cross extending over the entire height and width of the overlay window can be seen as a variant of the embodiment shown in FIG. 36. These modes can be particularly useful when reading or extracting information from a worksheet with the non-dominant hand, while working with the native mouse cursor in another document, for example, a text document. The large cross in overlay offers the advantage that a specific cell can be easily found, and that the row headings and column headings of a particular cell are also extremely easy to find, as explained in more detail in the pending perforated bitmap application.

For the sake of completeness, it is noted that single mouse mode 2840 (f) may be better suited for editing a worksheet, because in this embodiment the native mouse cursor is located at the intersection of the large cross. That being said, the person skilled in the art can easily add a display mode in which the cross of display mode 2840 (f) is used as the third visible object, and another object, for example, a line or a square or a hand becomes added as a second visible object.

BE2017 / 5896

FIG. 37 is a variant of FIG. 32 which implements the same modes, but is used to underline different parts of a single document.

FIG. 38 illustrates embodiments of the present invention that implement display modes 2850 (i) to 2870 (i) showing the native mouse cursor and two small crosses 3806L, 3806R.

FIG. 39 shows embodiments of the present invention wherein the overlay application is configured in one of the display modes 2850 (j), 2860 (j) and 2870 (j). The horizontal line or beam 3906L located on the left side of the vertical line 3907 is automatically moved up / down when moving the left pointing device 3903L. The horizontal line or beam 3906R located on the right side of the vertical line 3907 is automatically moved up / down when moving the right pointing device 3903R. Depending on which input mode is selected or implemented, the native mouse cursor 3905 follows movements from the left pointing device 3903L (in the Left Handed input mode), or from the right pointing device 3903R (in the Right Handed input mode), or from the pointing device that was most recently moved and / or clicked and / or scrolled (in the Two-Handed input mode). An advantage of this embodiment for proofreaders is that the non-dominant hand does not require accurate positioning, since an entire line of text can be underlined by only moving the non-dominant pointing device up / down. Left and right movements of the non-dominant pointing device are not required. This is especially true for simple texts that can be read rather quickly, where it is not required or desirable to underline each word separately, as opposed to, for example, translations of patent claims, where each word must be carefully checked.

FIG. 40 shows a variant of the one shown in FIG. 39, wherein the overlay application, in addition to the horizontal line or beam on the left and right side of the vertical line 4007, also shows two blocks or squares or rectangles 4009L, 4009R within the horizontal line or beam 4006L, 4006R, which blocks moving along the lines in the vertical direction (up / down), but also being movable in the horizontal direction (left / right) in accordance with movements of the respective pointing devices 4003L, 4003R. The line or bar will typically have a different color than the blocks or squares or rectangles within the line or bar so that they are clearly visible. In a specific embodiment or configuration, the lines have a monochrome color and the blocks are completely transparent, giving a visual impression of a beam with a hole.

BE2017 / 5896

The inventors also came up with the idea of solving the problem shown in FIG. 1 to FIG. 3 to be solved in a completely different way, namely outside the computer device, for example in a special display device 4104, as shown in FIG. 41.

FIG. 41 shows a computer system 4100 according to an embodiment of the present invention, comprising a special monitor or display screen 4104 according to an embodiment of the present invention, and a computer device 4101, which may be a conventional computer device. The computer system 4100 comprises at least two pointing devices, including a first pointing device 4103R connected to the classical computer device 4101, and a second pointing device 4103L connected to the special display screen 4104, in a wired or wireless manner.

Expressed in simple terms, the display device 4104 has a processor adapted to communicate with the pointing device 4103L and to maintain a position xL, yL corresponding to movements of the pointing device 4103L, and adapted to overlay the graphic image received from the computer device 4101 with a locally generated visible object 4106L, which is, for example, in the form of a square or a rectangle or a rectangle with rounded edges, or a circle or an arrow or a hand or a small cross or a large cross extending over the entire height and width of the display screen, or other suitable shape, and for moving that object in accordance with movements of the pointing device 4103L by positioning the object 4106L at the maintained position xL, yL. In other words, in this embodiment, the display device 4104 generates the object 4106L and displays it on the screen at a position movable by the left pointing device 4103L, while the computer device handles movements of the right pointing device 4103R as usual.

FIG. 42 shows an exemplary block diagram of an embodiment of the display device 4104 as may be used in the computer system 4100 of FIG. 41. The display device 4104 comprises:

an input port 4231, e.g., a VGA port or an HDMI port or a DVI port or other suitable port for receiving graphic image data from a computer device 4101;

- an interface or port for communicating with the pointing device 4203L, e.g., a physical connector 4237 such as a USB connector, or a wireless RF interface 4238;

- a display panel 4236, e.g., an LCD panel for generating a visible image;

an input buffer 4232 for storing at least a portion of the graphic image data received at the input port 4231;

BE2017 / 5896

- a frame buffer 4234 for storing at least a portion of the image data to be displayed on the display panel 4236;

- and a processor adapted for:

* communicating with the pointing device 4203L for receiving displacement values dx, dy related to movements of the pointing device;

* maintaining a position xL, yL based on the received displacement values dx, dy;

* generating image data from the visible object 4106L;

* generating the image data to be stored in the frame buffer 4234 by overlaying the graphic image data received at its input port with the generated image data of the visible object 4106L at the maintained position xL, yL.

In the case of a VGA or other analog input, the incoming signal must be digitized by means of one or more ADC (analog-to-digital) converters, in ways that are known per se in the prior art. Similarly, if the output signal is analog, then the digital signal leaving the output buffer must be converted to analog signals using one or more DAC (digital-to-analog) converters, in ways known per se in the prior art . The display device may optionally include input signal conversion circuits for converting analog signals to digital signals, and / or may optionally include output signal conversion circuits for converting digital signals to analog output signals.

The pointing device 4203L can be a wired mouse device or a wireless mouse device. The pointing device 4203L can be an HID-compatible pointing device, e.g., an HID-compatible mouse device. The pointing device can be directly connected to the physical connector via wires, or via a conventional hub (not shown).

The pointing device may be a wireless pointing device, e.g., a wireless mouse device, which communicates via a dongle connected to, e.g., inserted into the physical connector of the display screen.

The display screen may include an internal or external wireless RF transceiver 4238 to communicate with a wireless pointing device, e.g., via a Bluetooth protocol.

The generated visible object 4106L may be in the form of a square, or a square with rounded edges, or a rectangle, or a rectangle with rounded edges, or an elongated object, or an arrow or a hand, or other suitable shape.

The object 4106L may consist of monochrome pixels, e.g., red or blue or black or yellow pixels, or other suitable color. Alternatively, the object may have exactly two colors, for example an inner, inner color and a peripheral color (e.g., black), e.g.

BE2017 / 5896 to form greater contrast. But the visible objects can also contain more than two colors.

The object 4106L can be opaque, which means that the user cannot see through the object 4106L. In this case, the processor can generate the overlayed image by simply replacing part of the graphic image data from the computer with image data from the locally generated visible object, without having to mix data.

The article 4106L can be a semi-transparent monochrome article that has, for example, an alpha transparency α in the range of 10% to 90%, e.g., about 25% or about 50% or about 75%. In this case, the processor will either mix the graphic image data or mix it with the object data. In the case of 50% alpha gene, the red, blue and green color components R1, G1, B1 of the graphic image data and the red, blue and green color components R2, G2, B2 of the pixels of the movable object can be simply averaged, which is a simple arithmetic operation, for example equivalent to the following calculation: R: = (R1 shr 1) + (R2 shr 1), and similarly for the other color components.

The item 4106L can be or include a so-called perforated bitmap, as explained in the pending perforated bitmap application. This bitmap may comprise a first plurality of fully transparent pixels and a second plurality of monochrome pixels, the first plurality of pixels and the second plurality of pixels being interleaved (alternately positioned), for example, according to a checkerboard pattern.

* This perforated bitmap can be overlayed in an opaque way, as follows: In the event that a graphic image pixel from the computer image is overlaid with a completely transparent pixel, the result will be the original graphic image pixel. In the case that a graphic image pixel is overlaid with a monochrome pixel, the result will be the monochrome pixel of the movable object.

* This perforated bitmap can be overlayed in a semi-transparent manner, as follows: In the event that a graphic image pixel is overlaid by a fully transparent pixel, the result will be the original graphic image pixel. In the case that a graphic image pixel is overlaid with a monochrome pixel, the result will be the alpha mixing or mixing of the graphic image pixel from the computer image and the monochrome pixel of the movable object.

It is also possible to make the item 4106L appear semi-transparent by time multiplexing between the graphic image pixels (coming from the computer device), and the image pixels of the movable object (generated within the display screen). For example, by using / displaying the monochrome pixels of the object 4106L at a first moment in time (e.g., on even-numbered frames) and by using / displaying the graphic image data at a second moment in time (e.g., with odd-numbered ones frames). This would

For example, BE2017 / 5896 means that half of the frames would effectively display the visible object 4106L, and half of the frames would not, creating the impression of being semi-transparent (average over time).

Comparison of FIG. 41 with FIG. 5 (a) and FIG. 5 (b) teaches that the images on the screen can look exactly the same or very similar (assuming the same color and / or the same semi-transparency level are used). Therefore, the embodiment of FIG. 41 also the one shown in FIG. 1 to FIG. 3, but the computer device 4101 does not require the above-described special overlay application and / or the special mouse and / or the special hub, only the display device 4104 must be special.

The computer system 4100 of FIG. 41 allows the user to move the left pointing device 4103L to move the position of the visible object 4106 on the screen, but of course clicking on that pointing device or scrolling its wheel will not have the effect of selecting text or scrolling up or down a document. In other words, the embodiment of FIG. 41 does not solve the problem shown in FIG. 4.

In a variant of the display device shown in FIG. 42, the display device comprises at least two connectors for connecting at least two pointing devices, and the processor 4233 is adapted to maintain two pointing positions Pos1, Pos2, and to generate two visible objects, for example two lines similar to FIG. 32, or two squares similar to FIG. 33 or two hands similar to FIG. 34, and for positioning these two objects at the two tracked locations. With a computer system comprising such a display device, for example, a user could use two specially designed pointing devices for comparing two text documents (e.g., proofreading a translation), and could use a third pointing device connected to the computer, for editing or correcting.

The display device may have a further connector for connection to a removable, non-volatile memory device, such as a memory card or a USB stick or the like, which may contain a file and a bitmap of the visible objects to be displayed.

The inventors came up with yet another idea of the problem shown in FIG. 1 to FIG. 3, also outside the computer device 4301, namely by providing a special video adapter or video interface device 4330, connected between the computer device 4301 and the display device 4304, as shown in FIG. 43 and FIG. 44.

Many types of video adapters are known in the art, for example a video adapter for converting an HDMI signal to a VGA signal or vice versa. The video

100

BE2017 / 5896 adapter can contain one or more high-speed analog-to-digital ADC converters and / or one or more high-speed digital-to-analog converters DAC.

None of the existing video adapters maintains a position related to movements of a pointing device connected to said adapter, and none of these shows a visible object that is movable in accordance with these movements. And preferably, according to embodiments of the present invention, the video adapter 4330 comprises a digital video input connector (e.g., an HDMI input connector) and a digital video output connector (e.g., an HDMI output connector), in which case the incoming video or graphic image data is digital, and also the outgoing video or graphic image data is digital, and the DAC (s) and ADC (s) can be omitted.

FIG. 43 shows a computer system 4300 according to an embodiment of the present invention, comprising a video adapter device 4330 according to an embodiment of the present invention. The video adapter 4330 has a video input port A, 4431 (e.g., a DVI or an HDMI port or a VGA port) for receiving first graphic image data from a computer device 4301. The computer device 4301 is connected to a first pointing device 4303R. The video adapter 4330 further comprises a video output port C, 4436 (e.g., a DVI or an HDMI port or a VGA port) for providing second graphic image data to a monitor or display screen 4304. The video adapter 4330 comprises furthermore at least one interface (e.g. a physical connector such as a USB port, or an RF transceiver) for communicating with a second pointing device 4303L operatively connected to the video adapter. The video adapter 4330 is adapted to generate at least one visible object 4306L, for example with the shape of a square or a rectangle or a rectangle with rounded edges, or a circle or an arrow or a hand or a small cross or a large cross that extends over the entire height and width of the display screen, or other suitable shape, and for generating the second graphic image (to be sent to the display screen) by overlaying the first graphic image data (coming of the computer device) with the at least one visible object 4306L (internally generated), and for moving that at least one visible object 4306L in accordance with movements of the second pointing device 4303L.

The at least one visible object 4306L can have a monochrome color. The monochrome color may be predetermined, for example red or green or blue or black or another suitable color. Optionally, the video adapter 4330 may have one or more buttons 4441 for selecting the monochrome color from a number of predetermined colors.

The video adapter is adapted to generate at least one visible object 4306L and to overlay the at least one visible object with the incoming graphic image data, e.g., by alpha mixing or otherwise mixing or combining the first

101

BE2017 / 5896 graphic image data from the computer with the image data of the movable object. The at least one visible object may be an opaque or semi-transparent object, or may have a so-called perforated bitmap, as described in more detail in the co-punctured perforated bitmap application. It can be understood by comparing FIG. 41 and FIG. 43 that the embodiment of FIG. 43 the problems described in FIG. 1 to FIG. 3 dissolves.

FIG. 44 shows an exemplary hardware block diagram of a video adapter 4330 as may be used in the computer system 4300 of FIG. 43, but the invention is not limited thereto. The video adapter 4330 of FIG. 44 includes the following components:

- a video input port 4431 for receiving first graphic image data from a computer device;

- a video output port 4436 for providing second graphic image data to a display device;

- an input buffer 4432 for storing at least a portion of the first graphic image data received at the input port;

- a processor 4433 connected to the input buffer 4432, and adapted to generate second graphic image data based on the first graphic image data;

- a wired and / or wireless interface 4437, 4438 for communicating with a wired or wireless pointing device 4403L;

- wherein the processor 4433 is further adapted for:

* communicating with the pointing device 4403L for receiving movement information dx, dy related to movements of the pointing device 4403L;

* updating a position Pos2 based on the received displacement information dx, dy;

* generating a visible object 4306L;

* generating the second graphic image data by overlaying the first graphic image data with the generated visible object 4306L at the Pos2 position.

The video adapter may further comprise a frame buffer 4434 operably connected to the processor 4433 adapted to store at least a portion of the first graphic image data received from the input buffer 4432, and to store at least a portion of the second graphic data to be sent to the output buffer 4435. However, a frame buffer is not absolutely required, e.g., in the case that the incoming video stream and the outgoing video stream have the same frame rate and resolution.

102

BE2017 / 5896

Optionally, the video adapter 4330 may further comprise one or more buttons 4441, for example to enable the user to choose a color and / or shape and / or size of the movable object 4306L.

Optionally, the video adapter 4330 may further comprise one or more indicators, for example in the form of LEDs 4442 to provide information to the user.

The video adapter 4330 may include a memory device (not shown) that includes the input buffer and the output buffer and (if present also) the frame buffer.

However, the present invention is not limited to this specific embodiment, because other architectures can fulfill the same functionality. For example, if the clock speed of the incoming video data and the outgoing video data is the same, then the second graphic image can be generated in real time, and then the frame buffer can be omitted, and the generation of the visible object 4306L can in practice be implemented in real time, together with the overlay or alpha blending operation. The person skilled in the art of designing video chips knows how to implement this functionality.

The video interface device 4330 may have additional interfaces, for example an additional USB port 4439 for connection to, for example, a USB memory stick which may, for example, comprise a bitmap of the movable object, and / or further may comprise a texture bitmap to be overlaid with the incoming video data, for example in a manner as described in the pending perforated bitmap application.

In one embodiment, the video interface device 4330 has an integrated RF transceiver 4438 to communicate with a wireless mouse and / or to communicate with the personal computer device, for example for configuring the video adapter device 4330, and / or for selecting the size or color of the object, etc. But such a transceiver can also be provided externally, e.g., as a dongle connected to the connector 4437 or 4439.

In a variant of the video adapter device shown in FIG. 43 and FIG. 44, the video adapter includes at least two connectors for connecting to at least two pointing devices, and the processor 4433 is adapted to maintain two pointing positions Pos1, Pos2, and to generate two visible objects, e.g., two lines as in FIG. 32, or two squares as in FIG. 33 or two hands as in FIG. 34, and for positioning these two objects at the two tracked locations. For example, with a computer system comprising such a video adapter, a user could use two specially designed pointing devices to compare two text documents (e.g., proofreading a translation), and use a third pointing device connected to the computer , for editing or correcting.

103

BE2017 / 5896

In summary, the present invention provides, inter alia:

* A computer-implemented method for providing at least one first visible object 505, movable by a first pointing device 503R, and at least one other visible object 506L, movable separately by a second pointing device 503L, the first and second pointing device being connected with a computer device executing an operating system O / S with a graphical user interface GUI that provides a native mouse cursor 505, the method comprising: a) configuring the O / S and the GUI to provide the native mouse cursor as the first visible object ; b) providing a graphic overlay window 584 comprising the at least one other visible object; c) configuring the graphic overlay window 584 in click-through mode; e) adjusting a position of the first / second visible object in accordance with movements of the first / second pointing device. A computer program product for performing the method. A computer device comprising the computer program product. A computer system.

* A pointing device for use in a computer system that provides dual mouse support. The pointing device (M2 *) comprising: a motion detection mechanism; and a control unit connected to the motion detection mechanism, and adapted to send a motion message (mov2) including displacement parameters (dx, dy) when a motion is detected; at least one button (M2bl) and a button detection mechanism to detect whether the at least one button (M2bl) is pressed or released; and optionally a scroll wheel and a scroll wheel mechanism to detect whether the scroll wheel has been turned; wherein the control unit is connected to the button detection mechanism, and if present also to the scroll wheel mechanism, and is adapted to first send a dummy movement message (dmov) before sending a button-pressed message (press2) or a scroll wheel rotation- message. The dummy displacement parameters dx, dy are preferably very small. The pointing device can be a wired USB mouse or a wireless mouse.

* A hub for use in a computer system that provides dual mouse support. A hub (2520) configured to exchange messages between two pointing devices and a computer device, the hub comprising a processor which, upon receipt of a button-pressed message (pressi, press2) from a pointing device, first a dummy movement message (dmovl) sends to the computer device before sending the button-pressed message (pressi) to the computer device for some time (ΔΤ), and which, upon receipt of a scroll wheel rotation message from a pointing device, first receives a dummy movement message (dmovl) before sending the scroll wheel rotation message to the computer device some time (ΔΤ) later. The hub can ignore messages from one pointing device when a button of the other pointing device is pressed. The dummy displacement parameters dx, dy are preferably very small. The hub can have connectors and / or an RF transceiver.

104

BE2017 / 5896 * A display device (4104) for use in a computer system that provides dual mouse support, the display device (4101) comprising: a video input port for receiving first graphic image data from a computer device (4101); a display panel for generating a visible image; an input buffer and optionally a frame buffer; an interface to communicate with a pointing device (4103L), and a processor adapted to: receive motion information (dx, dy) from the pointing device; for updating a position (Pos2) based on the received motion information (dx, dy); for generating visible object data (4106L); and for generating second graphic image data to be displayed on the display panel by overlaying the first graphic image data with the generated data of the visible object (4106L) at the position (Pos2).

A video adapter device (4330) for use in a computer system (4300) that provides dual mouse support, the video adapter device comprising: a video input port (A) for receiving first graphic image data from a computer device (4301); a video output port (C) for providing second graphic image data to a display device (4304); an input buffer and optionally a frame buffer; an interface for communicating with a pointing device (4303L); and a processor adapted for: receiving motion information (dx, dy) from the pointing device; for updating a position (Pos2) based on the received motion information (dx, dy); for generating data from the visible object (4306L); and for generating second graphic image data to be displayed on the display panel by overlaying the first graphic image data with the generated data of the visible object (4306L) at the position (Pos2).

Finally, although individual features are explained in different drawings and in different embodiments of the present invention, it is possible that features of different embodiments may be combined, as would be logical to those skilled in the art when reading this document.

Although the drawings are shown with only one display screen or monitor, the present invention also works with multiple monitors, wherein the first position Pos1 may be located on a first monitor, and the second position Pos2 may be located on a second monitor. In this way a user can easily switch back and forth between two documents that are displayed on different screens.

It is explicitly pointed out that embodiments of the present invention (primarily directed at movable visible objects controlled by multiple pointing devices) can be combined with embodiments of the co-defined

105

BE2017 / 5896 treatment being perforated bitmap application (primarily directed to graphic overlay with a perforated bitmap, textured bitmap, movable line, movable cross, etc.) in any suitable manner.

For example, the overlay window 584 of FIG. 5c and the overlay window 884 of FIG. 8c of the present application may further comprise a perforated bitmap or a semi-transparent textured bitmap as explained in the co-pending application.

As another example, the movable / movable object (s) in embodiments of the present application may include a perforated bitmap as explained in the co-pending application (e.g., FIG. 37 and FIG. 39).

Although the present invention is explained for systems that have only two pointing devices, the present invention will also work with more than two pointing devices connected to the same computer, for example with three pointing devices arranged to move two or three visible objects, in addition to the native mouse cursor.

Claims (11)

Conclusions: A pointing device (2303) comprising: - a motion detection mechanism (2321) for detecting a two-dimensional movement; - a control unit (2324) connected to the motion detection mechanism (2321) and adapted to send a motion message (mov2) when a motion is detected; - at least one button (butl) can be pressed and released by a user; - and a button detection mechanism (2322) for detecting whether the at least one button (butl) is pressed or released; - wherein the control unit (2324) is connected to the button detection mechanism (2322) and is adapted to send a button-pressed message (press2) when it is detected that the at least one button (but1) is pressed, and to send of a push button release message (release2) when it is detected that the at least one button (but1) is released; characterized in that - the control unit (2324) of the pointing device (2303) is adapted for, when it is detected that the at least one button (butl) is pressed, for sending a dummy movement message (dmov), and for sending the button-pressed message (press2) at least a predetermined period (ΔΤ) after sending the dummy movement message (dmov). A pointing device (2303) according to claim 1, wherein the control unit is further adapted for when it is detected that the at least one button (butl) is released, for sending a dummy movement message (dmov), and for sending of the push-release message (release) at least a predetermined period (ΔΤ) after sending the dummy movement message (dmov). A pointing device (2303) according to claim 1 or 2, further comprising at least one scroll wheel rotatable by a user; and - a scroll wheel detection mechanism (2323) for detecting whether the at least one scroll wheel has turned; and - wherein the control unit (2324) is connected to the scroll wheel detection mechanism (2323) and is adapted to send a scroll wheel rotation message when it is detected that it has turned at least one scroll wheel; characterized in that - the control unit (2324) is adapted for, when it is detected that it has turned at least one scroll wheel, to send a dummy movement message (dmov), and for 107 BE2017 / 5896 sending the scroll wheel rotation message at least a predetermined period (ΔΤ) after sending the dummy movement message (dmov). A pointing device (2303) according to any one of the preceding claims, - wherein the predetermined period (ΔΤ) is a value in the range of 1 to 250 ms or in the range of 1 to 100 ms; and - one of the following options: * wherein the dummy movement message (dmov) has a horizontal movement value (dx) in the range of -5 to +5 and a vertical movement value (dy) in the range of -5 to +5; or * wherein the dummy movement message (dmov) has a horizontal movement value (dx) in the range of -1 to +1 and a vertical movement value (dy) in the range of -1 to +1, or * wherein the dummy movement movement message (dmov) a horizontal movement value (dx) has zero and a vertical movement value (dy) has zero. A pointing device (2303) according to any one of the preceding claims, wherein the control unit (2324) is configured to send a next dummy movement message (dmov) after a subsequent detection of the at least one button that is pressed, or after a subsequent detection of the at least one scroll wheel being rotated, the dummy motion message including displacement values (dx, dy) selected such that a cumulative horizontal displacement (£ dx) and a cumulative vertical displacement (£ dy) is a value in the range from -5 to +5, or in the range of -1 to +1. A pointing device (2303) as claimed in any preceding claim, further comprising an RF transceiver (2325) connected to the control unit (2324), the control unit (2324) being further configured to send messages to a computer device (2301) and for receiving messages from the computer device (2301) via the RF transceiver (2324), optionally via a dongle (2311) connected to the computer device (2301). A pointing device (2303) as claimed in any preceding claim, wherein the pointing device is a USB HID class device or is compatible with USB HID class devices. 108 BE2017 / 5896 8. A computer system comprising: - a computer device (2301); - a first and a second pointing device (2303) according to any one of the preceding claims, connected to the computer device (2301); - optionally comprising a display device; - optionally comprising a keyboard. A computer system according to claim 8, wherein the computer device comprises a processor that executes an operating system with a graphical user interface that provides a native mouse cursor, the processor further executing an overlay application, the overlay application providing a window configured in click-through mode, which window contains at least one other visible object, and wherein the overlay application adjusts a position of the native mouse cursor in accordance with movements of the first pointing device or the second pointing device which was most recently moved and / or operated, and a position adjusts the at least one other visible object in accordance with movements of the second pointing device. A computer system according to claim 8 or 9, wherein the first pointing device and the second pointing device are USB HID class devices or are compatible with USB HID class devices; and wherein the computer device comprises at least a first USB port for connection to the first pointing device, and a second USB port for connection to the second pointing device. 11. A set of components comprising: - a computer program product for providing an overlay application, wherein the overlay application provides a window configured in click-through mode, which window contains at least one other visible object, and wherein the overlay application adjusts a position of the native mouse cursor in accordance with movements of the first pointing device or the second pointing device which was most recently moved and / or operated, and adjusts a position of the at least one other visible object in accordance with movements of the second pointing device; - two indicating devices according to one of claims 1 to 7.