EP0923759A2

EP0923759A2 - Apparatus and method for creating and controlling a virtual workspace of a windowing system

Info

Publication number: EP0923759A2
Application number: EP98921679A
Authority: EP
Inventors: Waldemar J. Siskens
Original assignee: Koninklijke Philips Electronics NV
Current assignee: Koninklijke Philips NV
Priority date: 1997-07-03
Filing date: 1998-06-05
Publication date: 1999-06-23
Also published as: WO1999001838A2; JP2001506389A; WO1999001838A3

Abstract

An apparatus and method for creating and controlling a virtual workspace of a windowing system, the windowing system comprising hardware and software components, including a computer, a display device and an operating system supporting windowing. The virtual workspace has associated logical dimensions exceeding the physical dimensions of the display device's screen so that only a portion of the virtual workspace can be displayed at any time. The apparatus comprises a workspace input device coupled to the computer, the workspace input device providing, responsive to the end-user's activity, data relevant to the operations of the virtual workspace. The apparatus also comprises a workspace control mechanism responsive to the data provided by the workspace input device, the workspace control mechanism actuating the operations associated with such data and controlling the display on the screen of the portions of the virtual workspace. The method comprises the steps of (i) acquiring, at a workspace input device, data of types relevant to virtual workspace operations, said data being representative of end-user activity associated with said operations; (ii) providing data representative of the acquired data, to the workspace control mechanism; (iii) initiating, responsive to the provided data, interaction between the workspace control mechanism and one or more other software components, the interaction being directed to actuating the virtual workspace operations, and (iv) actuating virtual workspace operations by initiating interaction between the software components and the hardware components.

Description

Apparatus and method for creating and controlling a virtual workspace of a windowing system.

BACKGROUND OF THE INVENTION

This invention relates to an apparatus and method for use in a windowing system, particularly an apparatus and method for creating and controlling a virtual workspace comprising workspace units, each workspace unit containing program windows in number and arrangement as selected by the end-user from time to time. Windowing systems comprise hardware and software which, together, provide the end-user with a multi-use environment. As to hardware, windowing systems typically comprise a general purpose computer having a processor and memory, together with a display device (e.g. , a monitor having a screen) and one or more input devices (e.g. , a keyboard and one or more pointing devices, such as a mouse). As to software, windowing systems typically comprise one or more application programs and an operating system, the operating system itself generally comprising a collection of programs that, among other things, monitor the use of the system and supervise the execution of the application programs. In particular, the windowing system's operating system provides for executing each application in respective, dedicated portions of the display device's screen, the portions being virtual screens referred to as windows. Numerous such operating systems are known, including: System 7 by Apple Computer, Inc. , Cupertino. California; OpenWindows by Sun Microsystems, Inc. , Palo Alto, California; X Windows by the Massachusetts Institute of Technology, Cambridge, Massachusetts; and Windows 3.x, 95 and NT by Microsoft Corporation, Redmond, Washington. Windowing systems engender a paradigm shift from operating systems based on command line input. Under conventional command line systems, for example, the display device typically is a bland tablet at which the end-user labors, often deemed weighted down by arcane commands and constrained by alpha-numeric input particular to the application at hand. By comparison, windowing systems transform the computer into a multi-media device, enlivening the alphanumerics with graphics, sound, and video, all of which seem capable of being mixed together in near-limitless variety.

The paradigm shift to windowing systems not only has liberated the end- user, but also has revolutionized the computer industry. Windowing systems' impact is evidenced by their market share in the personal computer segment. Over the approximate ten years of these systems' commercial life, their market share has been marked by inexorable growth. Today, one would be well-challenged to find a new personal computer offered for purchase by a reputable retailer that did not support, and come packaged with, one or more operating systems supporting windowing.

Windowing systems' advantages over command line systems are many. One advantage can be found in the use of graphical user interfaces ("GUIs"). GUIs provide, among other things, for identifying applications and system resources by respective icons disposed in a root window, the root window being that window which fills the display device's screen upon booting up the computer. Icons tend to be real world analogues with which the typical end-user is likely familiar. For example, a graphic symbol representing a garbage receptacle is likely to be employed to indicate a location for disposing of objects, while a graphic symbol resembling a laser printer can be employed to show that such a resource is available. Similarly, a file folder can be used to indicate a location where various documents are electronically stored.

Windowing systems' advantages are also found in the reduction of time needed by the typical end-user both (i) to become comfortable with the system and (ii) to become adept with any new application. To reduce the learning time, the systems typically minimize the need to remember an application's commands. Instead, the systems typically make commands available to the end-user via menus that can be made to appear, as necessary, at the end-user's will or by intercession on the part of the application or system. Moreover, the learning time is reduced because applications generally adopt the style of the GUI, such as buttons, task bars, scrolling indicators and other interface components and modes of operation, thereby providing the end-user with consistency and familiarity across applications. To speed comfort with the system, GUIs are designed to be operated in conformity with the end-user's intuition. For example, a file can be printed by asserting its file folder icon (e.g. , by pointing with the mouse) and then associating the asserted icon with the printer icon (e.g. , by placing the folder icon on the printer icon, again with the mouse). In addition to the above advantages, windowing systems' have a powerful advantage that is referenced in their name: they provide windows. Windows allow the end- user to view the output of, provide input to and otherwise interact with a user-selectable number of applications, seemingly all at once and with each application available in its own window. In doing so, windowing systems encourage the end-user to exploit a larger illusion wherein the display device's screen becomes the workspace over which (i) applications are represented by icons that transform the applications into tools and (ii) documents associated with applications become various papers and folders arranged alone or stacked on the workspace, which papers and folders the end-user has wide latitude to edit, combine, rearrange, re-size, re-file, forward, toss or otherwise dispose or revise. Notwithstanding the advantages of windowing systems, these systems are not without limitations. One such limitation is that the workspace illusion is burdened by the physical dimensions of the display device's screen. For example, conventional monitors based on cathode ray tube technology have dimensions, measured on the diagonal, that peak at just over 20 inches. By comparison, real world desktops typically have diagonal dimensions that reach to over 70 inches.

With the workspace illusion confined to a relatively small screen, tools, papers and folders are increasingly stacked one on top of each other in the confined desktop space provided by the screen. In short, there is no free space on the desktop space. Thus, the promise of the windowing system —such as enhanced efficiency of and pleasure for the end-user, may be unrealized in practice.

Accordingly, a need exists for creating and controlling a virtual workspace in a windowing system, particularly an apparatus and method for overcoming the limitations associated with the conventional workspace, including those imposed by the physical dimensions of the display device's screen.

SUMMARY OF THE INVENTION

An object of this invention is to provide a virtual workspace for a windowing system so as to overcome the limitations associated with conventional workspaces, including the limitations imposed by the physical dimensions of the display device's screen.

Another object of this invention is to provide an apparatus and method for creating and controlling a virtual workspace comprising workspace units, each workspace unit structured to contain windows in number and arrangement as selected by the end-user from time to time. According to one aspect of the invention, an apparatus is provided that controls a virtual workspace of a windowing system, the windowing system including a computer, a display device and an operating system supporting windowing. The display device includes a screen having predetermined physical dimensions, while the virtual workspace has associated logical dimensions exceeding the physical dimensions of the screen so that only a portion of the virtual workspace can be displayed at any time.

The apparatus comprises a workspace input device coupled to the computer, the workspace input device providing, responsive to the end-user's activity, data relevant to the operations of the virtual workspace. The apparatus also comprises a workspace control mechanism responsive to the data provided by the workspace input device, the workspace control mechanism actuating the operations associated with such data and controlling the display on the screen of workspace units of the virtual workspace.

The apparatus preferably employs, as the display device, a head mounted display, while using a head/eye tracking device as the workspace input device. The workspace control mechanism preferably comprises a workspace control program.

In another aspect of the invention, a method is provided for creating and controlling a virtual workspace in a windowing system. The method comprises the steps of (i) acquiring, at a workspace input device, data of types relevant to virtual workspace operations, said data being representative of end-user activity associated with said operations; (ii) providing data representative of the acquired data, to a workspace control program; (iii) initiating, responsive to the provided data, interaction between the workspace control program and one or more other software components (e.g. , the operating system and application programs), the interaction being directed to actuating the virtual workspace operations; and (iv) actuating virtual workspace operations by initiating interaction between the software components and the hardware components.

The method preferably includes the step of configuring the windowing system. The configuring step comprises at least one of: establishing the logical dimensions of the display device's screen; establishing a shape and orientation for the surface on which the virtual workspace is disposed; establishing the number of dimensions of the virtual workspace; establishing the dimensions of the workspace units comprising the virtual workspace; establishing support for dynamic change in the number and dimensions of workspace units and virtual workspaces; mapping one or more data types to respective virtual workspace operations; establishing precisions and tolerances for initiating virtual workspace operations responsive to end-user activity; and establishing a compensation form.

The various features of novelty which characterize the invention are pointed out with particularity in the claims annexed to and forming a part of this specification. For a better understanding of the invention, its operating advantages and specific objects attained by its use, reference should be made to the accompanying drawings and descriptive matter in which its preferred embodiments are illustrated and described, wherein like reference numerals identify the same or similar elements.

BRIEF DESCRIPTION OF THE DRAWINGS In the drawings:

Fig. 1 is a block diagram of a windowing system, detailing hardware components thereof, according to the present invention;

Fig. 2 is a block diagram of the windowing system of Figure 1, detailing software components thereof, according to the present invention; Fig. 3 is a schematic diagram of an embodiment of a windowing system, according to the present invention, the system employing a head mounted display and tracking mechanisms, each in communication with a computer;

Fig. 4 is a perspective view of a virtual workspace, the virtual workspace comprising workspace units, according to the present invention; Fig. 5 is a top view of a virtual workspace, according to the present invention;

Fig. 6 is a perspective view of a virtual workspace, the virtual workspace comprising workspace units and providing multiple dimensions, according to the present invention; Fig. 7 is a flow-chart showing steps generally associated with creating and controlling a virtual workspace of a windowing system, according to the invention;

Fig. 8 is a flow-chart showing steps associated with enrolling program windows with workspace units, according to the invention; and

Fig. 9 is a flow-chart showing steps associated with moving program windows among workspace units, according to the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention contemplates an apparatus and method for creating and controlling a virtual workspace 52 in a windowing system 10, the virtual workspace 52 comprising workspace units 54, each workspace unit 54 structured to contain windows in number and arrangement as selected by the end-user from time to time.

As shown in Figures 1 and 2, a windowing system 10, as contemplated by this invention, comprises hardware components 11 and software components 21.

As shown in Figures 3, 4, 5 and 6, the windowing system 10 provides the virtual workspace 52 so that its virtual dimensions are enabled to exceed the physical dimensions of a screen 19 of a display device 18. In that regard, each workspace unit 54 of the virtual workspace 52 preferably has dimensions substantially equal to those of the screen 19. As shown in Figures 7, 8, and 9, the end-user 40 preferably is enabled to control the virtual workspace 52. As an example, the end-user 40 preferably can open one or more application programs 38 and associate ("enroll") such programs' program windows 60 with one or more workspace units 54. As another example, the end-user 40 preferably can also move program windows 60 among units 54. As yet another example, the end-user 40 can also traverse among workspace units 54 of the virtual workspace 52, thereby navigating around the workspace 52. The end-user's control over navigation is control of what the system 10 makes visually accessible to the end-user 40, it being understood that the virtual workspace 52 typically extends beyond what can be displayed within the confines of the physical screen 19. In that regard, navigating around the virtual workspace at least in the plane of the display device's screen extends beyond the visual portion of the workspace being displayed. If each workspace unit 54 has dimensions substantially equal to those of the screen 19, this navigating preferably engenders the end-user 40 viewing one workspace unit 54 at any given time, including the unit's associated program windows. The windowing system 10 preferably also provides for operations in multiple dimensions. One-dimensional operations can be provided, for example, by having a virtual workspace 52 comprising a row of workspace units 54. Two-dimensional operations can be provided, for example, by having a virtual workspace 52 comprising rows and columns of workspace units 54. Three-dimensional operations can be provided, for example, by having a plurality of virtual workspaces 52, e.g. , structured akin to stacked planes, concentric spheres or cylinders, or otherwise. Four-dimensional operations can be provided, for example, by having a plurality of workspace units 54 that, for any one virtual workspace 52, are arranged at a common coordinate (e.g., one position in the row/column coordinate system).

Hardware Components.

The hardware components 11 typically include (i) a general purpose computer 12 having a processor 14 and memory 16, (ii) a display device 18 having a screen 19, (iii) one or more workspace input devices 20, and (iv) other devices 17. The general purpose computer 12, the processor 14, the memory 16 are well known. Similarly, the other devices 17 (which typically comprise printers, plotters, storage drives (e.g. , disk and tape drives), CD-ROMs, DVDs, document scanners, communication devices, and the like) are well known. As shown in Figure 3, the display device 18 preferably comprises a head mounted display ("HMD") 42. The HMD 42 is coupled to the computer 12 via a communication channel 48, the channel 48 preferably being implemented using a wireless technology. HMDs are commercially available and have various characteristics, including being "see-through" or "non- see-through", being monocular or binocular, supporting various video modes (e.g. , VGA or SVGA), and having fixed or adjustable fields of view. It is to be understood that no particular HMD, or HMD characteristic, is required to remain within the principles of the invention; rather, in any particular implementation of the windowing system 10 according to the invention, the employed display device 18 preferably is selected responsive to engineering issues and challenges that are peculiar to that implementation. It is also to be understood that the display device 18 can be implemented using technologies other than HMDs (e.g., monitors, projectors and the like), without departing from the principles of the invention.

The workspace input devices 20 preferably include data acquisition devices responsive to the end-user's head and/or eye movements. However, it is to be recognized that the devices can include one or more of a keyboard, a microphone, pointing devices (e.g. , a mouse, trackball, touch pad, and/or a joystick), gesture detecting devices, virtual reality gloves, and other data acquisition devices, without departing from the principles of the invention. In any case, the devices 20 preferably provide data representative of the end-user's activity respecting operations of the virtual workspace, such activity including interaction with the virtual workspace 52 (e.g. , traversing therealong), interaction with workspace units 54 (e.g., enrolling program windows 60 to one or more workspace units 54), interaction with one or more program windows 60 (e.g. , opening, closing, minimizing, restoring, re-sizing, and moving).

Software Components.

As shown in Figure 2, the software components 21 of the windowing system 10 typically comprise an operating system ("OS") 22, a workspace control program 39 and some number of application programs 38. The OS 22 supports windowing and typically comprises a collection of component programs. The OS component programs generally include (i) workspace input device drivers 24 which are associated with the respective input devices 20, (ii) display device drivers 26 which are associated with the display devices 18, (iii) other device drivers 27 which are associated with the other devices 17, (iv) subsystem programs for executing system calls, which programs include, among others, a memory manager 28, a file manager 30, a windows manager 32, and a process/interprocess manager 34, and (v, an application programming interface ("API") 36 that provides for system calls to the subsystem programs. (The term "system calls" is generally used herein to indicate calls to and effected by the subsystem programs. Morever, the term "function calls" is generally used elsewhere herein to indicate calls to the API 36 from application programs 38. However, it is to be understood that these terms are interchangeable herein and, indeed, either term can be used to indicate both operations, without departing from the principles of the invention.)

So as to support windowing, the OS 22 typically relies on the windows manager 32. The windows manager 32, among other things, conventionally provides for execution of application programs 38 in respective windows, while also creating and maintaining how all windows are organized for display. Typically, such window maintenance entails two hierarchies: (i) organizing the primary windows of various application programs 38 (the term "'primary" window is generally used in this application to mean the first program window created for display by an application program 38, the creation typically occurring when the application is initially opened), and (ii) organizing the windows associated with each one of the respective application programs 38.

As to the latter hierarchy, windows associated with a single application program 38 typically are described, relative to each other, as being either parents, children or siblings. Generally, every so-associated window has a parent, including the application program's primary window, as the primary window can be considered a child of a root window, the root window being that window which fills the display device's screen upon booting up the computer 12. Moreover, children windows typically are restricted by the windows manager 32 to be inside the region demarcated by the parent window. Indeed, if a child window is positioned and sized such that, when displayed in full, some part of it would extend outside the region demarcated by parent window, conventional windows managers 32 typically clip the child window to the respective parent's region. This determines, in practice, that all children of a primary window are restricted in their position, size, and relocation options.

In either hierarchy, conventional windows managers 32 typically provide for arranging windows for display under one or more policies, including tiling and cascading. Under the tiling policy, windows are arranged adjacent one another. If the root window has its primary windows in tiled arrangement such that it has no available space for displaying an additional primary window, the opening of an application program typically triggers the windows manager 32 to re-size and re-arrange the existing windows so that all windows are displayed, including the new primary window.

Under the cascading policy, windows are displayed overlapping one another. Like pieces of paper, windows can be stacked so as to partially or completely obscure the windows therebelow. This stacking of windows engenders an ordering of windows in a dimension perpendicular to the screen, i.e. , into the screen. In Microsoft's Windows 95-brand OS ("95OS"), for example, top-level windows associated with open application programs are stacked on the "desktop" , the "desktop" being a root window that is sized to cover the entire area of the display device's screen 19. The stacking follows the "Z-order": when a window becomes the "foreground" window (i.e. , is being worked on by the end-user 40) or is first created, it is at the top of the Z-order and, accordingly, is displayed on top of all other open windows. The Z-order determines the sequence in which windows come to the foreground if the end-user 40 were to work through that order using keystroke commands such as ALT-ESC and ALT-TAB. The term "Z-order" is derived from the Cartesian coordinate system , the x and y axes are ascribed to the surface associated with the display device's screen 19 so that the z-axis becomes the window stack direction.

Under the 95 OS, all windows not in the foreground are characterized as being in the background. Moreover, because the 950S implements pre-emptive multitasking, windows may be active in the background at various times. Nevertheless, windows may be described herein as active with the meaning that they are in foreground. Conventional windows managers 32 typically provide for un-mapping windows from display and, thereafter, re-mapping the removed window for re-display. In re-mapping, the previously un-mapped window typically is re-displayed in size and position substantially equivalent to that prevailing at un-mapping. In Microsoft's Windows-brand OS's, such as 95OS, the un-mapping is referred to as "minimizing" and the re-mapping is referred to as "restoring" . The Windows brand OS's also support so-called "maximizing" of windows, which process provides for re-sizing the window to predetermined dimensions, the dimensions being such as to substantially fill the screen 19 of the display device 18.

The OS 22 admits a broad range of architectures. In that regard, no particular commercial OS, or architectural characteristics, are preferred for implementing this invention. Moreover, it is to be recognized that the OS 22 can be architectured other than as described and illustrated above, without departing from the principles of the invention. As an example, the architecture can omit, combine or re-arrange various of the above components, with or without adding new components. Indeed, the OS 22 can be re- architectured in a kernel/shell form, wherein its operations are reduced to a minimum size in the kernel, the user interface is provided by the shell, and many of the conventional OS subsystem programs (e.g., windows and networking managers ) are implemented as application programs 38, residing in the software level associated with the programs 38. In that case, rather than calling the kernel, the application programs 38 interact with the subsystem programs as to the subsystem's functions, and the subsystem programs, in turn, interact with other subsystem programs and the kernel, so as to effect system and/or function calls.

In a particular example of the range of the OS 22, the 950S has an architecture that responds, among other things, to Microsoft's understood desire to support the scores of legacy application programs written for MS-DOS-brand and Windows 3.x- brand OS's. The architecture includes several layers, including, a shell layer (i.e., Windows Explorer), a system virtual machine layer, an MS-DOS virtual machine layer, a base operating system layer and a device drivers layer. The system virtual machine layer comprises an API layer (i.e., referred to as the Win32 API) which, in turn, comprises Windows subsystem libraries. The system virtual machine layer supports (i) a single virtual machine for all 16-bit (e.g. , Windows 3.x-based) application programs, and (ii) a respective virtual machine for each 32-bit (e.g. , Windows-95-based) application programs. The MS- DOS virtual machine layer provides a respective virtual machine for each executing MS- DOS -based application program. The 95OS's system virtual machine layer provides the context under which all application programs run. The context includes, among other things, virtual memory, processor registers, and operating system resources. The operating system resources are provided by the Windows subsystem libraries of the API layer, which libraries run in the context of the virtual machine. The Windows subsystem libraries include, among others, the Kernel, GDI and User dynamic-link libraries, wherein: the Kernel libraries handle memory management, the loading and execution of programs and scheduling; the GDI libraries manage graphics operations (e.g., painting windows and printing documents); and the User libraries provide for window management. If an application program needs to obtain services from the operating system, it obtains the services, at runtime, through the appropriate module of the Windows subsystem libraries. The module either provides the functions or calls to the applicable subsystem in the base operating system layer or the device driver layer. For example, the GDI library is enabled to call the video driver so as to display selected subject matter. In the latter operation, then, the Windows subsystem libraries provide the interface between the application programs and the base operating system layer.

Each virtual machine is managed by the virtual machine manager subsystem of the base operating system layer. The virtual machine manager subsystem ("VMMS") creates, runs, monitors and terminates virtual machines, providing services that manage, among other things, memory, task scheduling, program loading and termination, interrupts and intertask communication. The VMMS also controls the shell and device drivers layers, the device drivers including virtual device drivers ( "VxDs"), miniport drivers, class drivers, minidrivers (i.e. , extensions of class drivers), and conventional real- mode drivers. In addition to Microsoft's 95OS, numerous other windowing OS's are commercially available. Accordingly, the OS 22, as well as its windows manager 32, API 36 and device drivers 24,26, 27, have been described herein, and are further described herein, so as to illustrate operation of the windowing system 10 according to the invention.

Workspace Input Devices.

Referring to Figure 3, the workspace input devices 20 preferably comprise at least one of a head tracking mechanism 44, an eye tracking mechanism 46, a microphone 47, a pointing device (e.g. , a mouse) 49, and a keyboard 51. Each of these devices 20, and other acceptable devices (e.g. , virtual reality gloves, gesture detection devices, and other data acquisition devices) are responsive to the activities of an end-user 40 of the windowing system 10, so as to provide data representative of the end-user's activity respecting operations of the virtual workspace. Moreover, each of the devices 20 preferably is coupled to the computer 12 via a respective communication channel 53, such communications channels typically comprising appropriate wires or cables. Although wires or cables are sufficient, it is preferred that, as to the head and eye tracking mechanisms 44, 46, the respective communication channels 53 be established via wireless technologies 50 (e.g. , infrared technologies).

Suitable head tracking mechanisms 44, and their operation, generally are known. For example, one form of operation comprises detecting the relative movement of the end-user's head 41 in an applicable environment 43. The environment 43 is typically established within a Cartesian coordinate system 45 having a selected origin. In such environment, the detected movement typically includes one or more of: linear displacement along the x, y and z axes of the Cartesian coordinate system 45; rotational displacement about such axes (i.e., pitch, roll and yaw); and movement characteristics, such as accelerations.

These movements generally are detected based on principles of magnetic fields. Typically, field transmitters and receivers 44A and 44B are employed. In that regard, one or more magnetic field transmitters 44A produce lines of flux having components associated with the axes of the Cartesian coordinate system, while one or more receivers 44B sense the flux directions and/or its field strengths. The sensed signals are sufficient to calculate the relative head movement in light of known factors, e.g. , the known position of either the transmitters 44A or the receivers 44B.

The receiver 44B provides data representative of the end-user's activity respecting operations of the virtual workspace. To do so, the receiver 44B can be implemented in various ways. For example, the receiver 44B can comprise hardware for processing the sensed signal associated with the magnetic field, such processing providing head movement data and/or the system calls implicate by such data. However, it is preferred that hardware processing be limited (e.g. , to analog to digital conversion of the sensed signal) so that the data can be processed by the computer 12 using one or more of the software components 21, particularly the workspace control program 39 and the workspace device driver 24.

Although, as shown, a field transmitter 44A is disposed in association with the end-user's head and a field receiver 44B is disposed in association with the computer 12, it is to be understood that the associations can be reversed, without departing from the principles of the invention. It is also to be understood that head tracking mechanisms 44 having different operations can be used, without departing from the principles of the invention. In the event that a magnetic field implementation is used, however, the particular implementation preferably responds to the engineering challenges particular to the that use. For example, in the described implementation, the HMD 42 preferably is shielded from the locally-produced magnetic field so as to minimize any interference with the quality of the displayed subject matter.

It is also to be understood that neither the transmitters 44A nor the receivers 44B need be disposed in close physical proximity to the computer 12, which disposition is shown as a matter of convenience in illustrating the invention. At the same time, it is preferred to dispose either the transmitters 44A or the receivers 44B in selected proximity to the computer 12, including to enable the end-user to readily use other resources of the computer 12, such as the other workspace input devices 20 and the other devices 17. Moreover, the transmitters 44A and receivers 44B preferably are disposed relative to each other so that the environment 43 both is readily established respecting the hardware 11 and is conveniently oriented for the end-user 40.

While either a head tracking mechanism 44 or an eye tracking mechanism 46 can be employed, it is preferred to employ both mechanisms. For example, it is known that individuals acquire peripheral targets through a coordinated sequence between eyes and head, as follows: eye rotation toward the target, followed by head rotation toward the target, followed by compensatory eye movement in opposite direction of the head rotation. Accordingly, the two mechanisms 44 and 46 can be employed to trigger traverse operations among adjacent workspace units 54, the triggering event being the combination of the end- user's eye approaching an edge of the display device's screen 19 and the end-user's head 40 following. An advantage of the combination, then, is to provide enhanced control in the traverse operations of the virtual workspace 52, e.g. , more precise triggering of traverse among the workspace units 54 responsive to end-user activity.

Use of both mechanisms 44 and 46 is also preferred so as to enhance the facility of the windowing system 10. In particular, the number of available data types generally is limited. Accordingly, it is preferred to use each available data type in controlling numerous virtual workspace operations. For example, eye movement data can be used in association with virtual workspace operations separate from traverse operations. In that regard, eye movement data (with or without head movement data) can be used to traverse among workspace units 54, but, once the end-user 40 has settled on a particular workspace unit 54, eye movement data can be used (with or without data from other workspace input devices 20) for virtual workspace operations particular either/both to workspace units 54 or/and to the program windows associated therewith (e.g., enrolling program windows to one or more workspace units, and opening, closing, minimizing, restoring, re-sizing, and/or moving, one or more program windows).

Head and eye tracking mechanisms 44 and 46 are commercially available. For example, HMDs featuring head tracking mechanisms are marketed by Kaiser Electro- Optics, Inc., Carlsbad, California (e.g. , the Pro View-brand line). In addition, HMDs featuring eye tracking mechanisms are marketed by n-vision, inc., McLean, Virginia (e.g., the Datavisor-brand line).

The workspace input devices 20 provide, responsive to the end-user's activity, data relevant to the operations of the virtual workspace 52. The data preferably includes information as to the source of the activity (e.g., the eye tracker, a particular key on the keyboard, a particular button on the mouse), the nature of the activity (e.g., head roll, eye rotation, button pushed, button released, key depressed), the window or portion thereof associated with the activity, and, in some cases, the relative or absolute time of the activity.

The data preferably is packaged by the windowing system 10 into data structures known as events. Events comprising relevant data are passed by the OS 22 to the workspace control program 39. So-passed events are those that fall within the interest list of the program 39, as specified to the OS 22 by the program 39. The interest list can be dynamic: the program 39 preferably can change its interest list from time to time, e.g., at any given time, change that list responsive to changes in the windowing system's configuration. So-passed events preferably are maintained by the OS 22 in association with the control program 39 in a respective event queue.

Because the interest list defines the data that is relevant to the program 39 respecting the virtual workspace operations, the interest list is a measure of which input devices comprise workspace input devices 20. For example, the opening of a program window 60 in a workspace unit 54 is a virtual workspace operation. However, to open a window, an individual mouse click on an icon is generally sufficient. The OS 22, in response to the interest list, passes that event to the program 39. Accordingly, even if the mouse is associated with no other event passed to the program 39, this one event and the data provided by the mouse in connection therewith renders the mouse a workspace input device 20. It is also to be understood that data associated with one or more of the input devices 20 can be irrelevant to the operation of the virtual workspace 52. Events containing only irrelevant data preferably are not passed to the workspace control program 39. Such events can arise without departing from the principles of the invention, as workspace input devices 20 can be general purpose devices. For example, when the end- user is working with an application program 38 providing for word processing, the entry of text into a document engenders numerous events associated with the keyboard input device, which events typically have data that is irrelevant to the operation of the virtual workspace 52. On the other hand, predetermined key combinations can be relevant, including keystroke combinations and sequences that trigger traverse of the virtual workspace 52 (e.g., ALT-CURS OR_UP) or key-strokes combinations and sequences that open, close, minimize, restore, resize, or move program windows 60, such key combinations providing data representative of the end-user's activity respecting operation of the virtual workspace 52.

Display Device.

Figure 3 shows an implementation of a windowing system 10, according to the present invention, wherein the system's display device comprises an HMD 42 having a screen 19. The HMD 42 has field of view ("FOV") characteristics. For a rectangular screen 19, the HMD 42 has FOVs associated with each of the vertical and horizontal axes of the screen 19. The FOVs correspond to the angle subtended by the HMD's virtual image, as viewed by the end-user 40 for the HMD's focal length. By way of comparison, the human binocular FOV is approximately 200 degrees in both the vertical and horizontal directions, while typical commercial HMDs have FOVs that range from 24 to 180 degrees along the horizontal axis and from 18 to 90 degrees along the vertical axis. In any case, HMDs 42 are available which provide FOVs correlative to those associated with standard computer monitors, when such monitors are disposed at normal working distances from the end-user 40.

The Display Device and the Virtual Workspace. Turning to Figures 4 and 5, the virtual workspace 52 is shown, respectively, in perspective and top views to comprise workspace units 54, as provided by the windowing system 10. The workspace units 54 comprise root windows and, accordingly, the virtual workspace 52 comprises a root window. Workspace units 54 can be populated or empty: populated workspace units 56 being characterized by the association therewith of one or more program windows 60; and empty windows 58 being characterized by the absence of any such association. Populated workspace units 56 preferably provide for the end-user 40 to selectably organize associated windows according to any and all of the policies conventional to the OS, including by sizing the windows and by having cascade arrangements 62 and tiled arrangements 64. The units 54 are virtually disposed over a theoretical surface 55 in the environment 43. As shown, each workspace unit 54 is associated with a particular portion of the surface 55. The surface 55 preferably is curved in the event that rotational displacements of the end-user's head 41 in the environment 43 are mapped to traversing operations. In that regard, the surface 55 can be planar, without departing from the principles of the invention. Planar surfaces 55, in particular, are associated with the units' virtual disposition being constrained to two-dimensions, such as by the available workspace input device 20 and/or the display device 18.

The Figures 4 and 5 illustrate the FOVs of the HMD 42 as such FOVs are employed in the system 10. As illustrated, the vertical FOV of the HMD 42 is the angle I, while the horizontal FOV is the angle J. The FOV angles can be equal, but typically are not. In that regard, these angles preferably reflect the conventional aspect ratios of computer monitors. Moreover, the FOVs preferably are functionally related to the rotational displacements of the end-user's head 41 so that the end-user 40 is enabled to readily control, through rotational displacement's functional relationship to what is displayed in the FOV, the traversing of the virtual workspace 52.

Each of the workspace units 54 preferably has dimensions that are substantially equal to the dimensions associated with the FOV of the HMD 42. However, it is to be understood that the workspace units 54 can be otherwise dimensioned, without departing from the principles of the invention.

It is also to be understood that, although the workspace units 54 are shown with curved sides, the curvature is used to illustrate that the units 54 are virtually disposed over the surface 55, and that surface, as illustrated, is curved. However, whether the surface 55 is curved or planar, each workspace unit 54 preferably is planar or, at least, substantially planar. Planarity is preferred, among other reasons, because the workspace unit 54 —a logical structure — is displayed on the screen 19 - a physical structure — which screen 19 typically has a planar or substantially planar surface. The planarity of the units 54 in the case of a curved surface 55 is illustrated in Figure 5.

The virtual workspace 52 comprises workspace units 54 which, in number, generally are capable of changing dynamically during runtime. The change typically is responsive to the end-user's interaction with the system 10. In particular, the change often reflects the configuration of the system 10.

In Figure 6, for example, the system configuration includes the surface 55 which comprises sections 55A, 55B and 55C of the interior walls of concentric right cylinders. The sections are disposed in an environment 43A established within a coordinate system 45A, the system 45A comprising a longitudinal axis 70A, a tangential axis 70B and a radial axis 70C. The axes 70A-C have origin selected at the head 41 of the end-user 40. The system 10, so-comprised, supports stacked virtual workspaces 52A, 52B and 52C, each such workspace being associated with one of the respective sections 55A-C (i.e., stacked along the radial axis 70C) and each such workspace having an associated number of workspace units 54. The system 10 also supports having stacked workspace units 54A, wherein the units 54 A are arranged at a common coordinate (e.g. , a longitudinal-tangential pair associated with what otherwise would be one workspace unit 54) of any of the sections 55A-C, such units 54A preferably being ordered along the radial axis 70C. (NB.: System support for the above-described stacks engenders an infinite number of potential workspace units 54; that number, however, is finite due to, e.g. , memory limitations of the hardware

11.)

The configuration of Figure 6, as illustrated, provides for dynamic change in the number of workspace units 54 either by adding/ removing virtual workspaces 52 or by adding/removing workspace units 54A in stacks. Although the virtual workspaces 52 are shown to be three in number, fewer or greater numbers can be used, without departing from the principles of the invention. In addition, although the workspace units 54 A are shown in association with only one workspace unit 54, it is to be recognized that stacks of such units 54 A can be associated with any one or more of the workspace units 54 of any one or more of the virtual workspaces 52, without departing from the principles of the invention. In that regard, the number of workspace units 54 is a theoretical infinite if the system 10 is configured to support either or both stacks of units 54 or multiple. For example, the system preferably provides the capability of supporting stacked workspace units in association with any one position relative to the virtual workspace 52, while also providing for stacking a plurality of virtual workspaces. (These theoreticals preferably are limited only by the hardware capabilities of the computer 12, particularly the memory 16 and other storage components.)

The system 10 can also be implemented to support a finite number of workspace units 54. For example, Figure 4 depicts a system 10 comprising a single spherical surface 55, and omitting support of workspace units 54A in stacks. The surface 55 has a selected radius, the radius being determined, e.g. , as a function of the HMD's FOV and rotational displacements triggering traverse operations. Accordingly, in this implementation, the system 10 supports workspace units 54 comprising, in number, that which fills the surface 55. However, because in operation the virtual workspace 52 may not put in use all of its units 54, the system 10 having a finite number of units 54 generally need not, and typically will not, access all such units 54. That is, windowing systems 10 according to the invention are capable not only of changing the absolute number of workspace units 54 during runtime, but also the number of populated workspace units 56. As shown by Figure 6, the windowing system 10 preferably also provides for operations in multiple dimensions. One-dimensional operations can be provided, for example, by having only the virtual workspace 52A comprising a tangential row of workspace units 54. Two-dimensional operations can be provided, for example, by having only the virtual workspaces 52A and 52B, each comprising one tangential row of workspace units 54. Two-dimensional operations «.an also be provided, for example, by having only the virtual workspace 52C, that workspace comprising two tangential rows of workspace units 54, the rows being disposed along the longitudinal axis 70 A. Three-dimensional operations can be provided, for example, by having a two-dimensional virtual workspace, e.g. workspace 52C, in combination with a one-dimensional virtual workspace, e.g., 52A or 52B. Three-dimensional operations can also be provided, for example, by having a three- dimensional virtual workspace, e.g. workspace 52C, in combination with stacked units 54A. Four-dimensional operations can be provided, for example, by adding a one-dimensional virtual workspace to the three-dimensional virtual workspace. Subject to the topology of the virtual workspace 52, the windowing system 10 preferably is implemented to support previewing operations. The previewing operations give the end-user 40 visual access to the entire virtual workspace 52 within the physical dimensions of the screen 19. For example, where the virtual workspace 52 is the above-described one-dimensional topology, the previewing operations can display, within the boundaries of the display device's screen 19, downscaled versions of the workspace units 54 of the tangential row, the versions being arranged in a row across the screen or in a snakelike rendering. The downscaled units 54 preferably include an indication of what program windows 60, if any, are associated with each unit. In doing so, the end-user 40 is enabled to navigate the virtual workspace 52 by selecting a particular unit 54 on the basis of this indication, rather than on the basis of reading the actual text or scanning the actual image of any one or more program windows 60. (See, e.g., published PCT-EPO Application 91201741.5.)

Configuration. The windowing system 10 of Figure 3 provides for virtual workspace operations, preferably according to hardware and software configuration properties of the virtual workspace 52. The hardware configuration properties include, for example, (i) the nature of the workspace input devices 20, (ii) the data types such devices 20 acquire, and (iii) the nature of the display device 18. The software configuration properties include, for example, (i) selecting the shape of the surface 55 and its orientation in the environment 43 (e.g., a plane disposed substantially horizontally —like a table top— from the end-user 40; a plane disposed substantially vertically —like a theatre screen- from the end-user 40; the inside surface of a cylinder, the cylinder's longitudinal axis being disposed either substantially vertically or substantially horizontally), (ii) mapping the data types acquired by the workspace input devices 20 to the respective virtual workspace operations (e.g., rotational displacement of the end-user's head 41 mapped to traversing the virtual workspace 52), and (iii) selecting precisions and tolerances for initiating virtual workspace operations responsive to end-user activity. Configuration properties preferably are selectable by the end-user 40. As to hardware, the end-user 40 generally can select between various components 11, including display devices 18 and workspace input devices 20. As to software, however, the end-user 40 typically is limited in the selection by whether the hardware and software components 11 , 21 support the configuration being sought. In that regard, the selection of a spherical surface 55 can remain unsupported if the display device 18 is a standard computer monitor and the workspace input device 20 is a mouse. On the other hand, a spherical surface 55 generally is supported, and can be selected, if (i) the display device 18 comprises the HMD 42 and (ii) the workspace input device 20 comprises the head tracking mechanism 44. In implementation using an OS 22 supporting plug-and-play and menuing-based configuration, e.g. 95OS, it is preferred that the end-user is provided with a menu of configuration options that lists those that are supported.

Processing acquired data generally responds to the configuration properties, which properties often include selections that are interdependent. An illustration is found in traversing operations, which operations comprise displaying the respective workspace units 54 interposed between, and including, the first and the last workspace units 54 of the traverse. The traverse operations typically respond to configuration properties of mapping, surface shape/ orientation, and precisions/tolerances, such as those described above.

For purposes of the illustration, it is assumed that: traverse operations are mapped to the rotational displacement of the end-user's head 41; the surface 55 is an inside surface of a sphere, the sphere being substantially centered at the end-user's head 41; and the precision/tolerances include movement differentials that trigger display of the sequence of traversed workspace units 54.

The assumptions have various implications. The mapping selection, for example, contemplates using a head tracking mechanism 44 to detect the rotational displacements. In addition, the mapping selection can contemplate omitting linear displacements and characteristics (e.g., accelerations) associated with head movement. Moreover, the mapping selection can contemplate omitting combinations with other data, such as eye movement data, which combinations often provide advantages, including those described above.

The assumed mapping selection also can give rise to interdependence with other configuration properties. For example, the mapping selection is interdependent with the selection of precisions/tolerances for the traversing operations. As shown in the Figures, with the mapping selection, the precisions/tolerances become movement differentials that comprise rotational displacements of the head 41. Such differentials are identified as angles \ and L. While these differentials can be absolute, they preferably are relative to the respective FOV angles I and J of the HMD 42 so that, traversing between adjacent workspace units 54 is triggered by a selected combination of the operationally-relevant differentials with the respective FOV angle or angles. In one configuration, a vertically- directed traverse can be triggered by rotationally displacing the head 41 beyond the FOV angle I by an amount at least equal to the differential's angle \. An advantage of this combination is that it typically enhances the stability of the system 10 by reducing the frequency of undesired traverses between adjacent workspace units 54. In another configuration, the vertically-directed traverse can be triggered by displacing the head 41 within the differential's angle \ of the top or bottom of the FOV. An advantage of this latter combination is that it typically enhances the responsiveness of the system 10 by anticipating the traverse — it being noted that, in working within a particular workspace unit 54, the end- user's head movements tend to be minimal relative to eye movements. Although this description is particular to vertical traverses, it is to be recognized that the description applies substantially to horizontal and diagonal traverses, with diagonal traverses typically entailing one of the possible combinations of the vertical and horizontal precisions/tolerances, all without departing from the principles of the invention.

The assumptions also can engender interdependence between the shape/orientation of the surface 55, on the one hand, and each of the mapping selection and the precisions/ tolerances selection, on the other hand. As stated, the assumptions minimize any engineering issues associated with the interdependence. For example, because the surface 55 is the inside surface of a sphere and the sphere is substantially centered at the end-user's head 41, the mapping's rotational displacement and the precisions/ tolerances' movement differentials (i.e., as relevant to traverse operations) substantially correspond to the internal angles among the sphere's radii. In that regard, then, the assumed configuration properties are in agreement with one another.

However, the shape/orientation of the surface 55 can be selected to upset the agreement. For example, the surface 55 can be selected, in the illustration, to be a plane disposed substantially vertically —like a theatre screen— in front of the end-user 40. In that case, considerations of perspective phenomena generally arise. More specifically, perspective phenomena have the workspace units tapering in size relative to the FOV as traverse operations are performed through large angular displacement relative to the environment's origin, i.e. , as traverse operations move to units 54 that are remotely associated with the origin. Such tapering, in turn, entails tapering of the angles associated with and describing the units' edges, as drawn from the end-user's head 41. Moreover, such tapering makes preferable selected compensatory adjustments.

In the absence of some compensation, rotational displacements of the head 41 can also cause irregular traversing operations. For example, if the differential's angles are selected to be sufficient to trigger traverse between two adjacent workspace units 54 which are proximately associated with the environment's origin, use of such angles can trigger traverses between greater than two units 54 if such units are remotely associated with the origin. Compensation can take various forms. As an example, no compensation can be provided. As another example, the dimensions of the virtual workspace 52 can be limited so that perspective phenomena become insignificant. As another example, the differential's angles can be adjusted to compensate for the variable size of workspace units 54. As yet another example, the workspace units 54 can have associated buffer bands that are sized inversely to, and circumscribe, the workspace units 54 in the FOV, thereby compensating for the variable size of the units 54 to meet the fixed size of the FOV. These buffer bands preferably can have selected displayed subject matter, including, for example, (i) a single color (e.g. , black) to simulate open space around the program windows 60 of the workspace unit 54 or (ii) portions of the windows 60 associated with adjacent workspace units. Although each of these compensation forms has been described individually, it is to be recognized that combinations of these and other forms can be employed, without departing from the principles of the invention.

As respects the illustration set forth above and in other discussions above, a broad range of configurations are contemplated, each within the principles of the invention. Several additional exemplary configurations are set forth below. These configurations, being examples, are not intended to exhaust all the possibilities.

It is contemplated to map data types provided by the workspace input devices 20 to greater than one virtual workspace operation. Previous discussions set forth mapping of eye movement data not only to traverse operations, but also to virtual workspace operations within a particular workspace unit 54. Similarly, head movement data acquired by the head tracking mechanism 44 can be used not only for traverse operations, but also with other virtual workspace operation. As an example, linear displacement into or out of any workspace unit 54 preferably can be mapped to virtual workspace operations involving program windows 60 of a workspace unit 54, including, for example, to sequence through the order of the windows 60 (e.g. , the Z-order in 950S) and/or to enlarge or reduce the size or windows (e.g. , moving toward the unit 60 enlarges the active window). As another example, linear displacement data into or out of any workspace unit 54 preferably can be mapped to sequence through a stack of workspace units 54 sharing that relative position on the virtual workspace (i.e. , like a virtual in-out box). As yet another example, linear displacement data into or out of the surface 55 can be mapped to sequence through a stack of virtual workspaces 52 (e.g. , like the slidably-overlapping blackboards once found in university lecture halls).

As still another example, accelerations of the head at or above a characterized threshold and/or in prescribed directions, can be mapped to various virtual workspace operations. In that regard, a series of nods of the head 41 can be mapped to sequence through program windows 60, workspace units or virtual workspaces 52, as such operations are described above.

To map any one data type to greater than one virtual workspace operation, it is contemplated to combine the data type with one or more data types. For example, head movements can be combined with one or more keystrokes, pointing device manipulations or voice commands so as to provide respective operations described above.

Operation. The windowing system 10 provides virtual workspace operations, the operations preferably being controlled by the workspace control program 39. As previously described, the operations preferably are initiated by one or more end-user activities that are mapped to the respective operations. The end-user activities are associated with one or more types of data relevant to the operations. The data types are acquired by the respective workspace input devices 20, are packaged as events by the OS 22 and are passed to the program 39 for processing. The program 39, responsive to such data, actuates the mapped operations. The actuation typically requires processing the data and generally is performed in conjunction with other software components 21, including the OS 22. Virtual workspace operations fall into various categories of interaction between the end-user 40 and the windowing system 10. The categories include, for example, interaction with the virtual workspace (e.g. , traversing therealong), interaction with workspace units (e.g., enrolling program windows to one or more workspace units), interaction with one or more program windows (e.g. , opening, closing, re-sizing, moving, etc.).

Referring to Figure 7, the steps generally associated with the provision of virtual workspace operations are shown. Step 100 comprises the end-user's activity that initiates the respective operation. Step 102 comprises acquisition, by respective workspace input devices 20, of data relevant to the respective operation. Step 104 comprises the receipt of relevant data by the OS 22. Step 106 comprises the packaging, by the OS 22, of relevant data as an event. Step 108 comprises the passing of the event to the workspace control program 39. Step 110 comprises initiation of interaction between the program 39 and the end-user 40. For example, this step can comprise opening a communication to the end-user, e.g., a dialogue window, so as to trigger additional end-user activity respecting the operations. Step 112 comprises interaction between the program 39 and one or more of the other software components 21. Generally, this step 112 includes function calls to the OS 22 to actuate the virtual workspace operations. In that regard, it typically entails interaction between software components 21 to effect the display of a dialogue window initiated in Step 110. Step 114 comprises interaction between one or more of the software components 21 (including the program 39) and the hardware components 11. This step can include the display of a dialogue window initiated in step 110. This step also includes (i) storing configuration selections, (ii) displaying workspace units 54 and/or program windows 60 associated therewith, (iii) storing information about virtual workspaces, including where more than one such workspace is supported, (iv) calculating and effecting changes, such as for compensations, and (v) otherwise actuating the virtual workspace operations.

Although these steps are generally associated with the provision of the virtual workspace operations, it is to be recognized that one or more of these steps can be omitted in providing any particular operation, without departing from the principles of the invention. For example, step 110 can be omitted whenever the relevant data further determines the operation.

The following sections describe specific virtual workspace operations within the context of the above-described steps.

Referring to Figure 8, steps are described for enrolling a program window 60, enrollment being the association of such window to a workspace unit 54. In step 200, the application is opened. Typically, opening the application requires the end-user 40 to perform some predetermined activity respecting the OS 22. In Microsoft's 95OS, an application can be opened, among other ways, by manipulating a pointing device with respect to the application's associated icon on the desktop or by finding the application listing in the menus under the "start" button of the desktop window. For the purposes of this description, then, the end-user's activity is a physical movement of the end-user's hand to interact with the OS GUI, as prescribed to open the application. Although a pointing device is here discussed, it is to be recognized that other workspace input devices 20, e.g., a microphone, can be used in place of the pointing device, without departing from the principles of the invention.

In step 202, the relevant data about the end-user's activity, e.g. , hand movement, is acquired by the respective workspace input device 20. As stated above, the device 20 typically is a pointing device and the data includes the nature of the pointing device, the button pushed and the location of the logical pointer on the screen at the time the button was pushed. This data corresponds to end-user's hand movement to position the logical pointer on the screen 19 and, once so positioned, to click the appropriate device button to open the application.

In step 204, the data acquired by the pointer device is received by the OS 22. In step 206, the OS 22 packages the data as an event which event is passed to the workspace control program in step 208.

In step 210, the program 39 preferably initiates interaction with the end- user 40. Whether interaction occurs and, if so, its nature, depends on the configuration of the windowing system 10.

In one configuration example, the system 10 enrolls new program windows automatically in a predetermined workspace unit, e.g. a unit 54 proximately associated with the environment's origin or, e.g., the current workspace unit 54 (the current workspace unit being that unit 54 displayed on the screen 19 when the end-user's activities initiate a virtual workspace operation). In that case, the interaction step 210 preferably is omitted (i.e., no interaction is initiated). In the alternative, the interaction step 210 can be initiated toward confirming that the end-user 40 desires to enroll the program window 60 to the predetermined workspace unit 54. Confirmation, however, can instead be implied by subsequent end-user activity. Implied enrollment can arise, for example, by the end-user's traversing to another workspace unit 54 without closing the program window 60. By contrast, that implication can be negated by subsequent end-user activity. Negated enrollment can be implied, for example, by activity associated with moving the program window 60 to a new workspace unit 54.

As another configuration example, however, the system 10 can be configured merely to open new program windows, without automatic enrollment. Again, the window preferably is opened in a predetermined workspace unit 54, e.g. the current workspace unit or one proximately associated with the environment's origin. In this example, step 210 can comprise opening a communication to the end-user, e.g. , a dialogue window, including to query whether the program window is to be enrolled and, if so, in which workspace unit 54.

Step 210 is followed by step 212 in which the program 39 interacts with the other software components 21. If the absence of a step 210, step 212 preferably is to provide for storage of the enrollment information in a file associated with the respective workspace unit 54. To do so, the workspace control program 39 interacts with the appropriate device drivers 27 and manager subsystems 28, 30, 32 and 34 of the OS 22, typically through system calls handled by the API 36, so as to effect the storage operation. In Microsoft's 95OS, the interaction implicates the system virtual machine layer (including its API layer), the base operating system layer and the device drivers layer.

If step 210 is provided to initiate further interaction with the end-user 40, step 212 preferably directs the communication to the end-user and, as necessary, readies the implicated device drivers. In the event that a dialogue window is implicated, for example, step 212 includes system calls to the OS 22 associated with displaying the window, including interaction with the display device drivers 26.

Step 212 is followed by step 214. In the absence of step 210, this step 214 comprises the storage of the enrollment information in the appropriate location, whether the location be the memory 16, a disk drive among the other devices 17, or both. The virtual workspace operation is then complete.

If step 210 initiates further interaction, step 214 actuates that interaction, e.g., by displaying one or more dialogue windows on the screen 19. In this latter case, control loops back to step 200, wherein additional end-user activity occurs, responsive to the dialogue window.

In one case, the end-user's additional activity is either (i) to affirmatively accept enrollment with the workspace unit 54 in which the program window 60 is opened or (ii) to affirmatively reject any enrollment. For both activities, the subsequent steps preferably correspond to the steps of Figure 7. In the step corresponding to step 102, the data indicating affirmative acceptance/ rejection is acquired by a workspace input device 20. In the steps corresponding to steps 104-108, the data is provided to the workspace control program 39 in predetermined form. In the step corresponding to step 110, it is preferred to initiate no additional interaction with the end-user 40. For affirmative acceptance, the steps corresponding to steps 112 and 114 preferably follow steps 212 and 214 set forth above for the case of no initiation in step 210. For affirmative rejection, the steps corresponding to steps 112 and 114 can be omitted or, preferably, provide for closing the program window 60 (and, thus, generally closing the application program 38) upon any one or more, predetermined, end-user activities. Such predetermined activities can include, for example, the end-user's traverse to another window. Such predetermined activities preferably are configurable by the end-user.

In another case, the end-user's activity is non-responsive to the dialogue window. The non-responsive activities can be of two types: those having system-understood implications and those that do not. As to the latter, it is preferred to maintain the state of the virtual workspace operations. Such activities include, for example, the end-user shutting down the windowing system 10 and/or user-initiated transition to operations outside the windowing system 10. As to those non-responsive activities that have system-understood implications (e.g. , via end-user configurations), however, the windowing system 10 preferably performs the implied virtual workspace operations. Such activities include, for example, the end-user's closing of the program window 60 from the OS's GUI. Such activities also include the end-user's indication, again via the OS's GUI, that the program window 60 is to be moved, which indication triggers the virtual workspace operations for moving a program window among workspace units 54. For both such activities, enrollment preferably is deemed to be rejected.

As to any non-responsive activity, however, other alternatives are contemplated, without departing from the principles of the invention. One alternative is to send a reminder dialogue window. Another alternative is to terminate the original dialogue window upon a predetermined time out. Yet another alternative is to terminate the original dialogue window upon a predetermined time out from the reminder dialogue window, the latter window warning the end-user of the impending operation.

In yet another case, the end-user's additional activity is to indicate, in response to the dialogue window, that the program window 60 is to be enrolled, but in a workspace unit 54 other than that in which the program window 60 is opened. This activity engenders the ..teps for rejecting enrollment, while triggering the virtual workspace operations for moving a program window to a user- selected workspace unit 54.

The operations for moving a program window, in one embodiment, include the steps illustrated in Figure 9. The steps preferably correspond to those of Figure 7. In step 300, the end-user performs activity that initiates the virtual workspace operations for moving a program window 60. As previously described, various activities can so initiate these operations, including responding to a dialogue window of the windowing system 10 or by interaction with the OS's GUI. As to the latter, the interaction with the OS's GUI typically is indicated by drag and drop activity. However, it is to be recognized that the activity can take other forms, including a voice command picked up by microphone, without departing from the principles of the invention.

In step 302, the relevant data about the end-user's activity is acquired by the respective workspace input device 20. As stated above, the device 20 typically is a pointing device and the data includes the nature of the pointing device, the button pushed, the location of the logical pointer on the screen at the time the button was pushed, and the vector associated with the mouse movement. The location data corresponds to the end-user's selection of the program window 60 to be moved. The vector data indicates to which adjacent workspace unit 54 the window 60 is to be moved.

The workspace input device 20 that is mapped to the moving operations can be other than a pointing device, without departing from the principles of the invention. For example, once moving operations are indicated, the end-user 40 can direct movement using either or both the head tracking mechanism 44 or the eye tracking mechanism 46. Indeed, head tracking can provide an acceleration characteristic, which characteristic can be mapped to indicate the arrival at the target workspace unit 54. In step 304, the data acquired by the applicable device 20 is received by the OS 22. In step 306, the OS 22 packages the data as an one or more events which events are passed to the workspace control program in step 308.

In step 310, the program 39 preferably initiates interaction with the end- user 40. Whether interaction occurs and, if so, its nature, again depends on the configuration of the windowing system 10. Several configuration selections are associated with moving operations, including, among others: (i) selecting whether or not and, if so, how to display the respective workspace units traversed in the moving operations; (ii) selecting whether or not to automatically enroll the program window with the target workspace unit 54; (iii) selecting the relationship (e.g., 95OS's Z-order) and arrangement (e.g., tiled, cascaded, or otherwise) o the moved program window 60 to previously enrolled windows of the target workspace unit 54; and (iii) selecting how to accommodate, or not, the boundaries of the virtual workspace 52 and/or the surface 55.

In each case, the configurations preferably are user-selectable. However, as to the display selection, it is preferred not only to display each traversed workspace unit 54 but also to do so by panning, rather than by switching. As to the enrollment selection, the options include, among others, those described above, with the preferred option being to automatically enroll based on implication from the end-user^'s initiation of movement operations. As to the relationship and arrangement selection, it is preferred to place the moved window at the top of the Z-order (i.e., in the foreground) and in a cascaded arrangement. As to the boundary selection, it is preferred to identify empty workspace units 58 within the boundary and, if none, to allow the end-user to choose between using stacked workspace units 54 A and/or stacked virtual workspaces 52, as described above with reference to Figure 6. As to each of the respective selections, the configuration determines whether the interaction step 310 is omitted (i.e. , no interaction is initiated) or whether the program 39 directs to the end-user 40 one or more queries or requests for additional activity.

Step 310 is followed by step 312 in which step the program 39 interacts with the other software components 21. Step 312 preferably provides for actuation of all aspects of the move operations, including, as appropriate, initiating the communication of the query or request for additional activity, managing display of the traversed workspace units 54, managing storage of the enrollment information in a file associated with the respective workspace unit 54, etc. To do so, the workspace control program 39 interacts with the appropriate device drivers 27 and manager subsystems 28, 30, 32 and 34 of the OS 22, typically through function calls to, or system calls handled by, the API 36.

Step 312 is followed by step 314 in which step the hardware actuates the operations. In the event that step 310 initiates further interaction, step 314 actuates that interaction, e.g. , by displaying one or more dialogue windows on the screen 19. After that, control loops back to step 300, wherein additional end-user activity occurs, responsive to the dialogue window.

Virtual workspace operations also comprise the operations of traversing among the workspace units 54. As with the other operations, it preferably corresponds to the steps of Figure 7. The end-user activities which trigger these operations are configurable and/or reflect use of the OS's GUI (e.g., directing the mous pointer beyond an edge of the screen 19, or using system-created scroll bars and/or buttons to traverse units 54 and/or workspaces 52). Various configuration options have been described above respecting the mapping of data types to traverse operations. In addition, several configuration selections that typically are associated with traverse operations have been discussed in respect to move operations.

OS Implementation of Workspace Units to the Virtual Workspace.

As previously described, the virtual workspace 52 comprises workspace units 54. As also previously described, the virtual workspace 52 and the workspace units 54 comprise root windows. In implementing the virtual workspace and its workspace units, the capabilities of the OS 22 are significant.

In one exemplary implementation under Microsoft's 950S, the virtual workspace comprises a choreography of that OS's desktop with the loaded application programs 38. More specifically, each of the workspace units 54 comprises the OS's desktop, together with an associated (i.e. , enrolled) subset of open, primary program windows 60 of the loaded application programs 38, all as associated with some position on (or relative to) the surface 55. the position being determined with respect to the applicable coordinate system 45 in the environment 43. Because the set of loaded application programs 38 applies to the virtual workspace 52 as a whole, and because the subset of open program windows associated with any given workspace unit 54 is taken from that set, the creation of a virtual workspace 52 reduces to providing a data base of logical workspace units, each having an associated data structure. The data structure, in that case, preferably includes, among others, records as to (i) the subset of program windows enrolled with the unit, (ii) the program windows' arrangement (e.g. , tiled, cascaded, etc.), (iii) any other windows associated with the unit (e.g. , windows provided as part of the windowing system, such as inter-unit scroll bars), (iv) the windows' order (e.g., 95OS's Z-order), (v) the size and relative locations of the windows associated with the workspace unit, and (vi) a specific position on or relative to a surface 55, the surface 55 having known attributes of shape and orientation within the environment 43.

Although the database is described above as being sorted by workspace units 54, it is to be recognized that the database can be sorted according to other indexes, or according to any record within the database, or combinations thereof, all without departing from the principles of the invention. For example, the database can be sorted according to the set of loaded application programs, including the enrolments of each such prognms' primary window.

The characteristics and operations of the virtual workspace 52 depend on the particular implementation. For example, traverse operations involve detection of the traverse data, particularly the sequence of positions on or relative to the surface 55. Having such data, the workspace control program 39, generally by system calls to the OS, sequences through the traversed workspace units 54. That sequence is effected by displaying each unit 54 in turn, including by minimizing and restoring and/or opening and closing program windows 60. This window management is directed to reflect the differences in enrolments between adjacent units and preferably is performed at a frequency undetectable to the end- user.

To illustrate the traverse operations, it is assumed that (i) the set of loaded application programs 38 includes an Internet browser program, an e-mail program, a word processing program, a presentations program and a spreadsheet program, (ii) a first workspace unit 54 has program windows open for the browser and e-mail programs, (iii) a second workspace unit 54 is disposed adjacent to the first unit, and has program windows open for the e-mail and word processing programs, and (iv) a third workspace unit 54 is disposed adjacent the second unit but not the first, and has program windows open for the presentations and spreadsheet programs. Accordingly, to traverse from the first to the second workspace units, the system 10 preferably minimizes the browser program and restores the word processing program, while leaving unchanged the e-mail program. In turn, to traverse between the second and third workspace units, the system 10 preferably minimizes both the e-mail and word processing programs, while restoring the presentations and spreadsheet programs. In each inter-unit traverse, the arrangement and order information of the windows associated with the arrived-at unit preferably is reflected in the displayed subject matter.

In the above illustration, traverse operations can be implemented as an organizing and re-organizing of the displayed subject matter, based on the above-described database. In such operations, it is preferred that the workspace control program 39 support a panning function. The panning function contemplates, from time to time, displaying contiguous portions of two or more adjacent workspace units, such portions being known from and the function preferably being implemented using, the above-described database. When panning is supported, the windowing system 10 preferably provides, upon the end- user's arrival at or indication of the selected target unit, for automatically centering that unit in the screen 19.

As with traversing operations and panning, other virtual workspace operations also can be implemented as an organizing or re-organizing of the data structures of the database, or otherwise employing the data thereof.

In the exemplary implementation using Microsoft's 95OS, the virtual workspace 52 comprises workspace units 54, each of which units comprises the Microsoft desktop, displayed substantially in full. In another exemplary implementation using 95OS, the virtual workspace 52 comprises workspace units 54, each of which units displays only that portion of the Microsoft desktop that corresponds to the position of the unit relative to the surface 55. Accordingly, the second exemplary implementation provides a virtual workspace that comprises but one desktop, albeit capable of containing one or more workspace units 54, each of which units can have dimensions that fill the screen 19 (NB.: if compensation is provided, as described above, buffer bands can be used to yet bring the display of each workspace unit 54 to fill the screen 19).

In Microsoft's 95OS, the enlarged desktop can be supported by providing the operating system with appropriate information about the screen 19 of the display device 18. The information includes the resolution of the screen (total dots by total dots for a rectangular screen) and the dot pitch. The information typically is provided when the display device 18 is registered with the 95OS's Registry, in accordance with the system's Plug and Play procedures and the operation of its Configuration Manager. At least two categories of approaches are contemplated: static screen configuration and dynamic screen configuration. In the static category, the display devices 18 are registered with information providing logical screen dimensions that exceed the physical screen dimensions, while preferably using the screen's actual pitch. In one embodiment, the logical screen dimensions can be provided substantially co-extensive to the range of the logical coordinate system supported by the 95OS. It being understood that the 95OS supports a maximum of 4000x3000 pixels, this co-extensive configuration provides a virtual workspace 52 comprising approximately 36 workspace units 54 for screens supporting a 640x480 physical resolution. Once the virtual workspace 52 is established as having some number of units 54 (e.g., 36 units), however, that number remains unchanged until set-up operations re-configure the windowing system 10.

In the dynamic category, the Plug and Play procedures are exploited to change, from time to time, the logical dimensions associated with the screen. In one embodiment, various logical dimensions are stored in the Registry so that, at run-time, the Configuration Manager is enabled to instantly update the dimensional configuration responsive to events specified to trigger such update. For example, an event leading to a dimensional increase can be the moving or enrollment of a program window to a workspace unit 54 expected to be outside the prevailing logical dimensions of the screen. Similarly, an event leading to a dimensional decrease can be the closing or other removal of all program windows enrolled in a workspace unit 54.

Implementations under the dynamic category preferably employ the 95 OS's VxD capabilities, while taking advantage of the 95OS's support for dynamically loading and unloading device drivers in response to events. VxDs are virtual display drivers which, as supported by the 95 OS, have broad utility, including being able (i) to read from and write to the entire Registry, (ii) to trap hardware and software interrupts, (iii) to receive messages from the operating system respecting events, and (iv) to access I/O ports and memory. Accordingly, a VxD can implement the embodiment by, among other things, entering the Registry during run-time to change the logical dimensions associated with the display device^'s screen, such entry typically being based on receipt of a prescribed event from the 950S.

A VxD preferably is used to provide not only the re-configuration function, but also the functions of the display device driver 26. In that regard, the functions of the workspace control program 39 preferably are implemented as a VxD. It is to be understood that the VxDs implementing these functions can be one or more in number, including one VxD implementing all functions, without departing from the principles of the invention.

VxDs can also be used to implement features not directly supported by system calls to the 95OS. As an example, VxDs can be used to enable the end-user 40 to enter a prescribed keystroke pattern so as to preview the entire desktop contents within the physical dimensions of the screen 19 (e.g. , to configure the logical screen dimensions to equal the physical dimensions of the screen 19). As another example, VxDs can be used to enable arrangements (tiled, cascaded, etc.) of program windows specific to each workspace unit 54.

As yet another example, VxDs can be used to provide that, notwithstanding any dynamic reconfiguration of 95OS's desktop dimensions, the size of the workspace units 54 and the program windows 60 enrolled therein remains unchanged. To illustrate, the VxDs can be used to maintain the workspace units' size substantially equal to the physical size of the display device's screen, while also maintaining the size of the enrolled program windows relative to the respective units 54. Otherwise, as is apparent when viewing adjacent 20" and 14" monitors, the reconfiguration to a larger logical desktop size would entail larger units 54 and windows 60 relative to the fixed physical screen dimensions (e.g. , windows sized to exceed that which the screen 19 can show in entirety). Respecting the lattermost example, the VxD preferably supports the maintenance of program windows 60 seamlessly and transparently. The support preferably is through API function calls. Under 950S, for example, the workspace control program 39 (implemented as a VxD or otherwise) is enabled to manipulate windows running under the operating system, including those of the application programs 38. In that regard, the range of window-specific API function calls of the 95OS is wide, including, among others: functions for manipulating the window hierarchy (e.g. , changing the Z-order and changing the parent window for a child window); functions for organizing windows (e.g, finding primary windows); and functions for manipulating selected windows' location and size.

While the invention has been described in connection with preferred embodiments, it will be understood that modifications thereof within the principles outlined above will be evident to those skilled in the art and thus the invention is not limited to the preferred embodiments but is intended to encompass such modifications.

Claims

CLAIMS:

1. An apparatus for controlling a virtual workspace of a windowing system, the windowing system including a computer, a head mounted display and an operating system supporting windowing, the head mounted display including a screen having predetermined physical dimensions associated with a pair of axes, the virtual workspace including workspace units and having operations initiated by an end-user's activity, the virtual workspace having logical dimensions correlative to the pair of axes of the screen, such logical dimensions of the virtual workspace exceeding the physical dimensions of the screen so that only a portion of the virtual workspace can be displayed, the apparatus comprising: a workspace input device coupled to the computer, the workspace input device providing, responsive to the end-user's activity, data relevant to the operations of the virtual workspace; and a workspace control mechanism responsive to the data provided by the workspace input device, the workspace control mechanism actuating the operations associated with such data and controlling the display on the screen of workspace units of the virtual workspace, wherein: the workspace input device comprises at least one of a head tracking mechanism and an eye tracking mechanism; and the workspace control mechanism comprises a workspace control program accessible to the computer, the workspace control program controlling the logical dimensions associated with the virtual workspace.

2. An apparatus as claimed in claim 1 , wherein the workspace control mechanism is arranged to control the display on the screen of the workspace units in a direction perpendicular to the pair of axes of the screen in order to select the display of overlapping workspace units.

3. An apparatus as claimed in claim 1, further comprising a wireless communication system, the wireless communication system coupling the workspace input device to the computer.

4. A windowing system providing a virtual workspace, the virtual workspace having associated logical dimensions, the virtual workspace including workspace units and having operations initiated by an end-user's activity, the windowing system comprising: a computer having an operating system supporting windowing; a head mounted display coupled to the computer, the head mounted display having a screen, the screen having predetermined physical dimensions associated with a pair of axes, the physical dimensions being exceeded by the logical dimensions of the virtual workspace that are correlative to the pair of axes so that only a portion of the virtual workspace can be displayed on the screen; a workspace input device coupled to the computer, the workspace input device providing, responsive to the end-user's activity, data relevant to the operations of the virtual workspace; and a workspace control mechanism responsive to the data provided by the workspace input device, the workspace control mechanism actuating the operations associated with such data and controlling the portion of the virtual workspace on the screen being displayed, wherein the workspace input device comprises at least one of a head tracking mechanism, an eye tracking mechanism, a virtual reality glove, a gesture detecting device, a pointing device, a keyboard, and a microphone.

5. A system as claimed in claim 4, wherein the workspace control mechanism is arranged to control the portion of the virtual workspace on the screen being displayed in a direction perpendicular to the pair of axes of the screen, in order to select the select the display of overlapping workspace units.

6. A method for creating and controlling a virtual workspace in a windowing system, the windowing system including hardware components and software components, the hardware components comprising a computer providing for storage of information associated with the virtual workspace, one or more workspace input devices providing data types, and a head mounted display for displaying subject matter associated with the virtual workspace, the software components comprising a workspace control program, an operating system supporting windowing and one or more application programs, each application program having associated program windows, the head mounted display including a screen having predetermined physical dimensions and having associated logical dimensions stored in the computer, the virtual workspace including workspace units and having, associated logical dimensions exceeding the physical dimensions of the screen so that only a portion of the virtual workspace can be displayed at any time, the virtual workspace being disposed on a surface, the method comprising the steps of: acquiring, at a workspace input device, data of types relevant to virtual workspace operations, said data being representative of end-user activity associated with said operations; providing data representative of the acquired data, to the workspace control program; initiating, responsive to the provided data, interaction between the workspace control program and one or more other software components, the interaction being directed to actuating the virtual workspace operations; and actuating virtual workspace operations by initiating interaction between the software components and the hardware components; configuring the windowing system, the configuring step comprising at least one of: establishing the logical dimensions of the screen; establishing a shape and orientation for the surface; establishing the number of dimensions of the virtual workspace; establishing the dimensions of the workspace units; establishing support for dynamic change in the number and dimensions of workspace units and virtual workspaces; mapping one or more data types to respective virtual workspace operations; establishing precisions and tolerances for initiating virtual workspace operations responsive to end-user activity; and establishing a compensation form, wherein the virtual workspace operations comprise traversing the virtual workspace, and the step of mapping data types to the operations comprises mapping data types associated with one or both head and eye movement to the traverse operations.

7. A method as claimed in claim 6, wherein the mapping step comprises mapping selected of the data types to traversing in each of the dimensions of the virtual workspace.

8. A method as claimed in claim 6, wherein the virtual workspace operations comprise enrolling a program window with a selected workspace unit, and the actuating step comprises storing information about the enrollment, and further comprising the step of configuring the windowing system to provide at least one of: (i) automatically enrolling of the program window with a predetermined workspace unit; (ii) enrolling the program window with a selected workspace unit; (iii) enrolling the program window upon affirmative confirmation of enrollment by the end-user; (iv) enrolling the program window based on end- user activity selected to imply confirmation of enrollment; and (v) negating enrollment based on selected end-user activity.

9. A method as claimed in claim 6, wherein the virtual workspace operations comprise moving a program window between original and target workspace units; and the actuating step comprises storing information about movement, the storing being in association with both the original and target workspace units, and further comprising the step of configuring the windowing system to provide at least one of: (i) displaying by switching sequentially among the workspace units interposed between the original and target workspace units in the virtual workspace; (ii) displaying by panning among the workspace units interposed between the original and target workspace units in the virtual workspace; and (iii) selecting the order, size, location and arrangement of the moved program window as respects other program windows enrolled with the target workspace unit.

10. A method as claimed in claim 6, wherein the virtual workspace operations comprise traversing between an original workspace unit and a target workspace unit, the actuating step comprising controlling the portion of the virtual workspace unit being displayed by minimizing program windows associated with the original workspace unit but not the target workspace unit, restoring program windows associated with the target workspace unit but not the original workspace unit, and adjusting the arrangement, size, relative locations and order of the program windows.