JP2001136504A - System and method for information input and output - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide information environment which is dynamically constituted by combination of a computer group of a table-type, a wall-type, etc., which are installed in an actual space and a group of portable computers that respective users carry. SOLUTION: A table and a wall surface function as work environment where a portable computer is extended spatially. Objects (paper documents, cards, video tapes, etc.), in the real world are specified by marker recognition and can freely exchange information. A user is able to operate information by direct operating technique functioning exceeding the border of a closed work space on a single computer, without being conscious of the addresses and IDs of individual devices and bodies by making good use of the spatial position relation among the objects in the real world. For example, 'hyperdragging' operation which takes over dragging operation on the computer display of one user on a plane (table or wall surface), where a computer is placed as is, becomes possible.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a proposal of a user interface environment for operating a digital object or inputting a command or the like to a computer. The present invention relates to a proposal of a user interface environment that can easily perform object operations and command inputs.

More specifically, the present invention relates to a proposal of a user interface environment capable of intuitively and intuitively expressing a cooperative operation between a plurality of computers, and more particularly, to a desktop interface native to each computer. The present invention relates to a proposal of a user interface environment capable of intuitively and intuitively expressing individual work of each user and cooperative work between users on an information space in which a screen is extended to the real world.

[0003]

2. Description of the Related Art Along with recent technological innovation, workstations and personal computers (PCs) have been called.
A relatively small, low-priced, high-value-added, highly-functional general-purpose computer system has been developed and marketed, and is deeply embedded in universities and other research institutes, companies and other offices, and even in everyday life in ordinary households. Has penetrated.

One of the recent trends in computer technology
One is miniaturization. Particularly in recent years, large scale integration (LSI) associated with the improvement of semiconductor technology.
Along with the miniaturization of chips and the remarkable improvement of the processing capability of processors, portable (portable) computers (or mobile computers) that are small, lightweight and highly portable have become widespread. .

[0005] An example of a portable computer is a so-called "notebook computer", and another example is a "PC".
DA (Personal Digital Assist)
a smaller, almost palm-sized computer called a "tance." For example, a notebook computer includes a system body having a keyboard on an upper surface and an LCD (Liquid Crystal).
l Display) has a so-called “clam shell” structure in which a lid having a screen on the inner surface is rotatably connected.

Recently, computer resources (CPUs (Central Processes) of notebook computers have been developed.
The computing power and hard disk space of a single computer (i.e., ing Unit) have been significantly improved and are as powerful as those of desktop computers. For example, a document file being edited, a spreadsheet file, a PIM (Person)
Almost all personal data, such as al Information Manager data, can be carried along with the computer while being stored on the local disk of the notebook computer. In such a case,
This eliminates the need to copy data from a local disk to a floppy disk, and from the floppy disk to a destination local disk.

[0007] These portable computers can operate a portable and portable system in a mobile environment, that is, outdoors or at a destination. With the advent of computers with excellent portability, computers have appeared in various situations of daily life of computer users, and opportunities for supporting business execution have been increasing more and more.

For example, when a participant enters a notebook computer or a PD in a conference room, a presentation room, or the like.
Bringing in A is becoming an everyday sight. In particular, a notebook computer has a hard disk capacity sufficient to hold a large amount of files such as a large amount of document data owned by an individual (several GB to several tens GB: for example, storing all presentation materials performed in the past). It is also possible to freely extract information from the local disk as the conference progresses, and share, exchange, and distribute information with other participants.

On the other hand, computerization of the office environment itself is progressing, and the user interface for inputting commands and the like to the information environment, particularly to the computer, is undergoing a major transformation. It can also be called an "electronic meeting room". For example, a technology has emerged that uses not only a liquid crystal projector and an electronic blackboard (whiteboard) but also a conference room table itself as a computer display.

For example, Remesh Raskar,
Greg Welch, MattCutts, Ad
am Lake, Lev Stesin, and He
nry Fuchs's paper "The office"
of the future: A unified
approach to image-basedmo
deling and spatially imme
rsive displays. "(In SIGGR
APH '98 Proceedings, pp. 179
-188, 1998) describe that the surface of a building in the real world can logically function as a computer display. Also, Beth M.
Language, Mark A. Jones and J
ames L.A. "Insight," written by Meyers.
t Lab: Animative team
environmental linking paper
r, displays, and data "(In
CHI '98 Proceeding, pp. 55
0-557, 1998).
It is disclosed that a large screen composed of displays supports not only individual user activities but also collaboration of the entire team.

[0011] As a natural consequence of the development of the respective technologies of the spread of the portable computer and the evolution of the user interface environment, in the near future computing environment, there will be a portable computer having a powerful arithmetic function owned by an individual, A room such as a conference room and computers installed in the environment dynamically cooperate, and the entire workplace such as the conference room is integrated with a logical space on the computer (for example, a table or wall of the conference room is Integrated with the desktop).

However, in the current electronic conference room,
Only a user interface using a fixed device configuration is mainly considered. Therefore, there is no idea that “the environment is dynamically configured by the portable computer and the installed computer”.

In a conference in the real world, actual documents can be circulated and exchanged between participants. Also, the spreadsheet can be easily posted on a wall for explanation.

On the other hand, even if a simple operation such as information exchange (exchange of an object such as a document file) between computers carried by each participant is performed, a network is set up and each device is connected. Work is needed. In addition, target (transfer destination)
It is not enough to visually recognize the physical location of the device (for example, where the device is located in the table), and it is necessary to check the address assigned to the device. In other words, it places a burden on the user and causes frustration. For example, it is not easy to distribute objects displayed on a screen using a projector to other participants, and it is not possible to write annotations (comments, notes, notes, etc.) on presentation materials on such a projector screen.

For example, a paper "A multi" written by the present inventor.
ple-device approach fors
upporting whiteboard-base
interactions "(In Processed
ings of ACCMHI '98, pp. 344
-351, February 1998) describes an information system in which an electronic wall in an electronic conference room and a PDA are linked.

Also, a paper written by Naohiko Khotake, “InfoStick: An interac”.
Tion device for inter-app
liance computing ”(In Work
shop on Handheld and Ubiq
uitous Computing (HUC'9
9), 1999) describe a portable device that can function as an integrated commander between a plurality of information home appliances.

In the above two papers, the execution area or concept of direct manipulation using a mouse or the like, which was available only in a single computer, was extended from the logical space on the computer to the real world. He proposes an interactive technique that crosses the boundaries of computers and other devices. The present inventors describe such an interactive technique as “Augmented Direct”.
This is called "t Manipulation" (extended direct operation).

In an electronic conference room, there is a demand for an information space capable of dynamically expanding a user operation environment by introducing the concept of such extended direct operation.

In order to extend the face-to-face interactive operation, it is often assumed that a computer having a fixed system configuration is installed in advance. For example, “Colab” ^[10] is composed of a projector screen and a computer for each participant on the table. Participants cannot bring in new computers. However, given the recent trends in mobile computing, the ability to support dynamic connections between portable and installed computers is an important issue.

Some systems for projecting digital information on a physical desk have been proposed. "VID
EODESK " ^[19] consists of a light table and a video camera. The user can perform an interactive operation through the silhouettes of other users projected on the table.

According to "Digital Desk" ^[11,4] , interactive operation can be performed between a printed document and digital information projected on a desk. Digi
Recent versions of talDesk have added a document identification function based on OCR ^[12] .

"Luminous Room" ^[5] (and the concept of "I / O valve" included therein) is as follows:
The projection area is varied using a video projector mounted on a computer controlled gimbal. According to “Illuminating Light” as an application example, the holographic designer can perform layout design of a physical optical device on a desk in a short time.

Stritz et al. Developed a set of computer extensions, including walls, chairs, tables, etc. ^[3] According to this, the table becomes a table-sized computer, or "InteracTable", to support discussion between those surrounding the table. Further, information carried by a physical block called "Passage" is also displayed. This system mainly focuses on interactive operations between non-electronic objects and projected digital information. On the other hand, the present invention described later aims at interactive operation between a portable computer, a table, a wall surface, and other physical objects.

"ZoomieBoard" ^[13] is a system for scanning on a whiteboard. Zoom
ieBoard controls the pan / tilt camera to capture the mosaic that is part of the image on the whiteboard. By concatenating these mosaic images, a high-resolution image over the entire whiteboard is generated. Also, “BrightBoard” ^[14]
Is another example of a whiteboard system extended by a camera, and can recognize a command input by handwriting with a marker pen.

A conventional example of interactive operation of a multi-computer is disclosed in "Hybrid User Interf."
aces " ^[15] . This is an application example of a see-through type display mounted on a user's head and can generate a larger virtual screen around the screen of a desktop computer.

The PDA-ITV system ^[7] uses a palmtop computer such as "Newton" of Apple Inc. as an interactive TV system. This system is based on the assumption that the device configuration is fixed, and is designed mainly for a single user.

"Ariel" ^[16] and "transBO"
ARD " ^[17] supports the connection between digital content and a document or card on which a barcode is printed. Also," InsightLab "
^[18] is a computer-supported conference room that can be used by extending bar-coded tags into physical and digital links and commands. These systems typically require manual scanning of each printed barcode, which is an overwhelming burden for the user. Further, since these systems do not recognize even the position information of each object, another mechanism is required to realize a spatially continuous operation.

[0028]

SUMMARY OF THE INVENTION It is an object of the present invention to provide an excellent user interface that expands the working space of a computer into the real world and allows the user to input commands to the computer intuitively and easily. To provide an interface environment.

A further object of the present invention is to provide an excellent user interface environment capable of intuitively and intuitively expressing a cooperative operation between a plurality of computers.

A further object of the present invention is to provide an excellent user who can intuitively and intuitively express object operations and command inputs on an information space in which each desktop screen native to each computer is extended to the real world.・ To provide an interface environment.

A further object of the present invention is to dynamically create an intuitive and easy-to-understand user interface environment by cooperation of a portable computer moving in an information space and an environment-type computer fixedly installed in the information space. Consists in configuring.

A further object of the present invention is to dynamically expand the indoor operation environment by using a group of computers installed in a room environment such as a conference room and a group of portable computers brought in by each user such as a conference participant. It is to provide an information space that can be used.

A further object of the present invention is to dynamically extend the operating environment by means of a portable computer group brought by each user such as a conference participant, and to set up a computer or a computerized conference room table or wall. An object of the present invention is to provide an "easy-to-use" information space in which information can be smoothly exchanged with a computer.

[0034]

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and a first aspect of the present invention is to provide a digital space in an information space in which the digital space of a computer is extended to the real world. An information input / output system for performing a user operation on an object, comprising: at least one work surface disposed in the information space; display means for displaying an image on the work surface; and imaging on the work surface And an environment-type computer fixedly installed in the information space, wherein the environment-type computer obtains position information and / or an object on the work surface based on an image captured by the imaging unit. Executing a process of recognizing a user operation on the object, and a process of controlling display of the digital object by the display unit according to the recognition result. It is an information input and output system.

According to a second aspect of the present invention, there is provided an information input / output system for performing a user operation on a digital object in an information space in which a digital space of a computer is extended to the real world. At least one work surface disposed in a space, display means for displaying an image on the work surface, imaging means for taking an image of the work surface, and an environmental computer fixedly installed in the information space And one or more portable computers movable in the information space and capable of exchanging entities of digital objects with other computers, wherein the environment-based computer is an image captured by the imaging means. Processing for recognizing position information of an object on the work surface and / or a user operation on the object based on the recognition result; Processing for controlling the display of the digital object by the indicating means, and dynamically installing the portable computer in the information space by recognizing the identification information and the position information of the portable computer installed on the work surface. The information input / output system is characterized by executing a process to perform

Here, the information space is formed in a predetermined room. The table surface and / or at least one wall surface installed in the room is a work surface of the information space. The present inventors refer to these work surfaces as "InfoTable".
And "InfoWall" (described later).

The imaging means is a CCD (Char).
ge Coupled Device: charge-coupled device)
Camera can be applied. For example, it is composed of a fixed camera that has a fixed visual field and can capture a relatively wide area, and a variable camera that has a variable visual field and can capture a relatively narrow range with high resolution. When such a configuration is adopted, the fixed camera captures a work surface in a wide area, and the variable camera captures a region where a difference has occurred between the current and past captured images of the fixed camera. A change in the object on the work surface can be detected.

[0038] The physical object placed on the work surface may include a visually identifiable visual marker on the surface. One example of a visual marker is
This is a two-dimensional barcode such as a cyber code (described later). In such a case, the environment-type computer can recognize the identification information and / or the position information of the physical object based on the visual marker in the image captured by the imaging unit.

Further, the environment type computer may further execute a process of recognizing an operation of the digital object on a display screen of the portable computer based on a captured image.
The manipulation of digital objects involves the use of computer GUs, such as so-called "drag and drop".
I (Graphical User Interface)
e) Includes mouse cursor operations that are already well known and established by many users in the environment.

Further, the environment type computer further tracks the drag operation of the digital object on the display screen of the portable computer based on the captured image, and the cursor reaches the end of the display screen. In response, the digital object may be displayed at a corresponding position on the work surface, and a process of continuing the drag operation on the work surface may be executed. Inheriting a drag operation onto a work surface in the real world such as a table is called "hyperdrag" by the present inventors.

As a result of supporting hyperdrugs, familiar GUI operations in the digital space of computers are also applied in the real world.

For example, in response to dropping a digital object on the display screen of the portable computer onto the work surface, transferring the entity of the digital object from the portable computer is performed. Be executed.

Also, the correspondence between the digital object dropped on the work surface and the original portable computer may be displayed on the work surface. We call such a display an "anchor cursor". In cases where multiple users are simultaneously performing a hyper-drag operation on the same work surface, the anchor cursor clearly indicates the correspondence between who is operating which object, and prevents work surface confusion. can do.

When two or more work surfaces (for example, a combination of a table and a wall (or whiteboard)) exist in the information space, the hyper-drag of the digital object between different work surfaces may occur. Permissible. For example, by once dropping an object related to a presentation material on a table and then hyper-drag it onto a wall (or a projector screen or the like), a presentation can be made on the wall as it is.

Further, the environment type computer further tracks the drag operation of the digital object on the work surface based on the captured image, and responds to the fact that the cursor reaches the boundary with the portable computer main body. Performing a process of transferring the entity of the digital object to the portable computer, while the portable computer performs a process of receiving the digital object and continuing the drag operation on the display screen. You may. As a result, operations such as moving, distributing, and exchanging digital objects between users can be performed easily and intuitively through a work surface such as a table. For example, each user drops information, or digital objects, he wants to share from his or her portable computer onto the table using hyperdrag. In addition, when a desired object is found among the digital objects scattered on the table, it can be picked up into their portable computer using hyperdrag.

Further, the environment-type computer uses an object aura (described later and FIGS. 12 to 14) indicating a data space of a physical object recognized on the work surface.
May be executed. By displaying the object aura, the user can
Physical objects installed in the information space can be easily confirmed visually. The physical object only needs to be attached with identification information such as a cyber code (see FIG. 17), and need not be a data processing device such as a computer at all. For example, a VCR tape cartridge or a document on which a cyber code is attached can be installed in the information space.

Further, the environment-type computer may execute a process of associating the entity of the digital object dropped on the object aura with the corresponding physical object. As a result,
Digital objects that are virtual and logical entities are tentatively assumed to exist physical objects. In addition, digital data can be moved or delivered via a physical object (such as a VCR tape cartridge) that does not originally have a function of storing and transporting digital data.

Further, the environment-type computer executes processing of the digital object associated with the physical object in response to detecting that the physical object is placed on the work surface based on the captured image. It may be executed. For example, by placing a VCR tape cartridge on a table as a work surface, a voice note attached to the VCR tape cartridge is reproduced.

Further, one or more computer peripheral devices (for example, a printer, a display, a scanner, etc.) for executing predetermined object processing may exist in the information space. In such a case, in response to the digital object placed on the work surface being further hyperdragged and dropped on the computer peripheral device, a predetermined process related to the object is executed. Is also good. For example, a document file is printed out by dropping a digital object having the document file as an entity onto a printer. Alternatively, an image is displayed on the monitor screen by dropping a digital object having a still image or a moving image as an entity on a television monitor.

A third aspect of the present invention is an information input / output method for performing a user operation on a digital object in an information space obtained by extending a digital space of a computer into the real world, wherein the information space is , One or more work surfaces, and display means for displaying an image on the work surface;
An image capturing unit that captures an image of the work surface, and an environment-type computer that is fixedly installed, and based on an image captured by the image capturing unit, position information of an object on the work surface and / or a user operation on the object Recognizing, and controlling the display of the digital object by the display means according to the recognition result;
An information input / output method characterized by comprising:

A fourth aspect of the present invention is an information input / output method for performing a user operation on a digital object in an information space in which a digital space of a computer is extended to the real world. The space includes at least one work surface, and display means for displaying an image on the work surface;
Imaging means for imaging on the work surface; an environment-type computer which is fixedly installed; and at least one which can move in the information space and exchange digital object entities with other computers. (A) recognizing position information of an object on the work surface and / or user operation on the object based on an image captured by the image capturing means; and Controlling the display of the digital object by the display means;
(C) dynamically installing the portable computer in the information space by recognizing identification information and position information of the portable computer installed on the work surface. Information input / output method.

Here, the information space may be formed in a predetermined room, and a work surface may be a table surface and / or at least one wall surface installed in the room.

The imaging means comprises a fixed camera having a fixed visual field and capable of capturing a relatively wide range, and a variable camera having a variable visual field and capable of capturing a relatively narrow range with high resolution. a) calculating a difference between current and past images captured by the fixed camera, and changing an object on the work surface by further capturing an image of a region where the difference exceeds a predetermined threshold using a variable camera; And a sub-step of detecting

The physical object placed on the work surface includes a visually identifiable visual marker on the surface, and in the step (a), the visual marker in the image picked up by the image pickup means is displayed. The identification information and / or the position information of the physical object may be recognized based on the physical information.

Further, the method may include a step of recognizing an operation of a digital object such as a drag and drop operation on a display screen of the portable computer based on a captured image.

[0056] Further, the step of tracking the drag operation of the digital object on the display screen of the portable computer based on the captured image, and in response to the cursor reaching the end of the display screen, Displaying the digital object at a corresponding position on the work surface and continuing the drag operation on the work surface.

Further, in response to dropping a digital object on the display screen of the portable computer onto the work surface, the entity of the digital object is transferred from the portable computer to the environmental computer. The method may further include the step of transferring to

[0058] The method may further comprise displaying on the work surface the correspondence between the digital object dropped on the work surface and the original portable computer.

When two or more work surfaces exist in the information space, a step of dragging the digital object between different work surfaces may be provided.

A step of tracing a drag operation of the digital object on the work surface based on the picked-up image; The method may further include transferring the entity of the digital object to the portable computer, and continuing a drag operation of the digital object on a display screen of the portable computer.

[0061] The method may further include a step of displaying an object aura indicating a data space of the physical object recognized on the work surface.

The method may further include associating the entity of the digital object dropped on the object aura with the corresponding physical object.

Further, in response to detecting that the physical object has been placed on the work surface based on the picked-up image, a step of executing processing of the digital object associated with the physical object is further provided. You may.

In the case where one or more computer peripheral devices for executing a predetermined object process exist in the information space, the digital object on the work surface is dropped on the computer peripheral device. The method may further include performing a predetermined object process in response.

[0065]

According to the present invention, there is provided an information environment dynamically configured by a combination of a group of environmental computers such as a table type or a wall type installed in a real space and a group of portable computers carried by each user. can do.

In the information environment according to the present invention, for example, a camera-based object recognition system is introduced. The computer display screen is projected by a projector onto a surface such as a table or a wall surface. Further, both the projected image, that is, the object in the logical space of the computer, and the real world (that is, an object existing on a table or the like) are recognized based on the image captured by the camera, and the movement of these objects is tracked.

In such an information environment, tables and walls in conference rooms and presentation rooms are computerized, and function as a work environment (desktop) for spatially expanding a portable computer.
It also identifies objects in the real world (paper documents, cards, videotapes, etc.) by marker recognition, etc.
Enables free exchange of information between them.

Further, a user can easily integrate a portable computer carried by the user into an information environment in which several computers are already installed.

[0069] By using a single computer, a direct manipulation technique that works across the boundaries of a closed workspace, the user can operate without knowing the addresses and IDs of individual devices and objects, and can use the spatial and physical properties of each object in the real world. The information can be manipulated by using a simple positional relationship.

For example, it is possible to perform a "hyperdrag" operation in which a drag operation of an object on a computer display by a certain user is directly inherited by a plane (table or wall surface) on which a computer is installed.

According to the hyper-drag, the user can operate the object only by intuitive knowledge of the physical positional relationship of the object in the real world (for example, "an object is near a computer"). be able to. Also, in the background of the operation of the object in the front end, the movement of the entity of the object such as a document file is executed by arithmetic processing (for example, network transfer) between the related computers.

According to the hyper-drag, it is necessary to learn the normal mouse operation (drag and drop) and to know the physical location of the target device (for example, the position placed on the table). Can be moved. In other words, when moving or exchanging digital objects between adjacent portable computers, the user can use a hyperdrag (see, for example, FIG. 16) to perform operations such as network setting and connection, and target devices. There is no need to know the address of the server.

Further, in the information environment according to the present invention, cooperation between a physical real space and a logical space or a digital space on a computer is performed. Thus, computer data can be associated with physical objects. For example, any document folder can be attached to a physical object such as a video tape placed on a table (see, for example, FIGS. 12-14: described below).

Still other objects, features and advantages of the present invention are:
It will become apparent from the following more detailed description based on the embodiments of the present invention and the accompanying drawings.

[0075]

Embodiments of the present invention will be described below in detail with reference to the drawings.

The present invention is based on the combination of a group of computers such as a table type or a wall type installed in a real space and a group of portable computers carried by each user.
Is dynamically configured. In such an information environment,
Tables and walls in conference rooms, presentation rooms, and the like function as a work environment (desktop) that spatially expands the digital space of each computer in the real world. An example of a real space to which the present invention is applied is a meeting room where one or more users perform individual work or joint work.

1. System Configuration The information space provided by the present invention is cooperative between a physical space composed of tables, walls, and other items placed on the table, and a digital space logically configured on a computer. It provides an information environment in which the relationships are structured.

FIG. 1 schematically shows a configuration example of the information space 1 according to the present invention. As shown in the figure, an LCD (Liquid Crystal Disp) for displaying digital data on each of the table 11 and the wall surface 12 is provided.
(Layer) Projectors 13 and 14 are provided (however, the output image of the LCD projector 13 is
3A is projected on the table 11). The projection images of the projectors 13 and 14 constitute a user output device in the information space 1, that is, an “extended display”.

The content displayed by each LCD projector 13 and 14 is controlled by the cooperative operation of computer systems 16 and 17 interconnected by a network 15. Each computer system is, for example, a PC / AT (Personal Comp.
uter / Advanced Technology)
A compatible machine or its successor may be used, and OADG (Open A
rchitecture Developers Gr
(up) specification. Each computer 16 and 17
Each other has a NIC (Network Interface)
Card) via a network 15. Alternatively, they may be connected by other means such as a wireless LAN.

The fixed camera 18 and the pan / tilt camera are used to capture the display contents of the digital data on the table 11 and the wall surface 12 and the portable computer and other physical objects placed on the table 11. 19 are installed. The cameras 18 and 19 are, for example, CCD (Charge Coupled Device)
ce: charge-coupled device) A camera may be used.

Each of the cameras 18 and 19 constitutes a user input device in the information space 1. Image processing server 20
Performs image processing on these captured images, and further performs image recognition on the positions of the digital objects and the placed physical objects displayed on the table 11 and the wall surface 12 and user operations applied to each object. I do.

The image processing server 20 is a computer for controlling the display of the LCD projectors 13 and 14.
The systems 16 and 17 are networked. Therefore, the user operation in the information space 1 whose image has been recognized by the image processing server 20 is reflected on the projected image, and an interactive information environment via the extended display is realized.

The image processing server 20 and the computer systems 16 and 17 are provided with an “environmental computer” (environmental computer) which is pre-installed in the environment of the information space 1, ie, the information environment.
mputer). In the example shown in FIG. 1, the environmental computer is composed of a plurality of computer systems 16, 17, and 20 connected to a network. Of course it doesn't matter.

In such an information space 1, each user carries his / her own portable computer... (Illustration) to perform individual work and cooperative work between users.

The information space 1 is a complex digital space formed by cooperation of one or more computers. On each of the table 11 and the wall surface 12, digital objects existing in the digital space are displayed. As a result, physical objects and digital objects are mixed in the information space 1. Further, various user operations (described later) on the digital object can be performed on the surfaces of the table 11 and the wall surface 12 to constitute an “extended display” that expands a work space (desktop) on a computer screen. The present inventors refer to the table 11 and the wall surface 12 as the extended display as “InfoTable” and “InfoWal”, respectively.
Also called l "

The portable computers 21 used in the information space 1 require some preconditions.

In this embodiment, a cyber code is provided on the surface of each portable computer or other physical object.
Visual markers 31, 32, such as ^{/ * /} , which can be visually identified, are attached. Therefore,
The environment-type computers installed in advance are the cameras 18 and 1
Nine captured images can be recognized.

The portable computers 21 are interconnected by communication means such as a wireless LAN.
Digital object entities (eg, document files and digital content) between portable computers or between portable and environmental computers
Can be transferred in the background of the extended display.

Next, a mechanism for recognizing and identifying an object operation in the information space 1 using the cameras 18 and 19 will be described.

The cameras 18 and 19 used as object recognition sensors generally have a trade-off relationship between resolution and visual field. The cameras 18 and 19 must have sufficiently high resolution to identify the minimal pattern of visual markers affixed to the physical object. The high-resolution captured image is also useful for creating a record of the table 11.

However, most of the video cameras currently on the market do not have specifications for realizing both the required resolution and the viewing angle for imaging the entire surface of the table 11. "Digital Desk"
^[4] attempted to solve this problem by installing a second camera. In this case, the second camera is used to capture only a specific portion on the desk with higher resolution than the first camera. The user has been instructed to place the document within the field of view of the second camera.

On the other hand, the present inventors have solved the above trade-off problem by combining two cameras. The first camera 19 is a motor-controlled video camera (for example, EVI-D3 manufactured by Sony Corporation).
0), so that pan, tilt, zoom and the like can be remotely controlled according to a command from an external computer system. According to the camera 19, the table 11
Not only can the entire image be captured, but only a partial area can be captured at high resolution (up to 120 dpi) by zooming. Normally, a pan / tilt camera 19
The entire surface of the table 11 is scanned by periodically changing the direction and direction of the head. For example, the surface of the table 11 is divided into 6 × 6 blocks,
Control is performed so that the tilt camera 19 regularly visits (ie, captures) all 36 regions. The present inventors describe such camera operation as Landsat.
(Land-satellite) "Desk
The time required to visit all areas is about 30 seconds including camera control and image processing time.

On the other hand, the second camera 18
It is a fixed camera for always looking at the entire surface. The change on the table 11 is analyzed based on the difference between the current and past images captured by the camera 18. Next, it is determined which of the 36 blocks has changed, and an event of “area changed” is transmitted to the pan / tilt camera 19 (see FIG. 15). Using the event information, the pan / tilt camera 19 can immediately re-search the area where the change has occurred, and update the captured image.

In order to prevent the detection by the fixed camera 18 from being affected by the fluctuation of the projection image, it is necessary to set a predetermined threshold value for the difference detection.

In this embodiment, some empirical rules are used to determine the order of searching for the change area.
Since a human usually has a habit of using a table from the outside to the inside, it can be said that an object changes easily inside the table 11. Therefore, the priority can be given to the inner side than the outer area. For example, when the fixed camera 18 captures changes in a plurality of blocks at the same time, the pan / tilt camera 19 searches for changed blocks from inside to outside.

According to such an object recognition technology composed of two cameras, the user can
When an object is placed on or moved (and re-moved), such an event can be recognized by the environmental computer 20 within seconds. Such a response time is referred to as "Illuminatin".
Although not always sufficient for continuous, real-time object tracking, such as g Light " ^[5], it may be appropriate in an information environment that operates intermittently.

Next, a description will be given of a mechanism in which the environment-type computer recognizes the visual marker based on the image captured by the camera.

[0098] All physical objects to be installed in the information space 1 and other portable computers are provided with visually identifiable visual markers. An example of this visual marker is "Cyber
Code "is (see FIG. 17), it is possible to identify 2 ²⁴ objects by a combination of the sequences of printed pattern (provided that sequence code described in the literature [6] In the present embodiment It uses a slightly different version than the system). According to the Desksat architecture described above, 2 cm placed on table 11
A small two-dimensional marker having a size of 2 cm can be recognized by the pan / tilt camera 19.

The visual marker is used not only for acquiring identification information but also for recognizing the position of the physical object on which the marker is attached. For example, while the user performs a hyper-drag operation (described later) in the information space 1, the position of the cursor on the table 11 is changed based on the current position and orientation of the visual marker attached on the portable computer. The calculated value is reflected on the image projected by the LCD projector 13.

[0100] Since the visual markers are provided as printed matter, there is virtually no cost. Also, there are some uses that cannot be achieved with other identification systems.
For example, a Post-it containing a visual marker
Can be used. A digital object (for example, a voice note or a photographic image) attached to the identification information indicated by the visual marker is transferred as the Post-it moves.

2. Information Environment Next, features of the information environment provided in the information space 1 according to the present embodiment will be described.

2-1. Cooperation between environmental type computer and portable computer In an information environment provided in the information space as shown in FIG. 1, "environmental type computer" fixedly installed in a real space such as a conference room and participation in a conference Cooperation is realized with a “portable computer” carried by each user such as a user.

The portable computer possessed by each user is:
It has a complete user interface on its own. Therefore, when cooperating with the environment-type computer, the interface in the environment-type computer or the information space 1 is not completely replaced with the conventional user interface, but is naturally extended from the user interface unique to the portable computer. Should work.

For example, simply placing a portable computer on a table allows the table surface or wall surface to be used as a virtual extended display or desktop for the portable computer.

Furthermore, even when a plurality of users place their portable computers on the same table, the table surface can be used as a shared work space for a group of users.

FIGS. 2 to 4 illustrate a process in which each user's portable computer is extended by an environmental computer.

(A) The user works on his or her portable computer (see FIG. 2). In this case, the user uses a conventional GUI (Graphical Us) provided in a stand-alone environment called a portable computer.
The user can drag and drop an object using a mouse cursor on the (er Interface) screen.

(B) When the portable computer is placed on the table in the information space, the surface of the table or wall can be used as a spatial extension of the desktop of the portable computer. More specifically, when the user arrives at the table 11 and places his or her portable computer on it, the camera 1
8 images this. The image processing server 20 uses the visual image attached to the portable computer based on the captured image.
By recognizing the marker 31, the owner user of the portable computer is specified. Further, the image processing server 20 also recognizes the installation position of the portable computer. For example, a digital object representing an information item such as a document file or a folder is hyper-dragged from a display screen of a portable computer and placed on a table or a wall surface (actually, an LCD projector 1).
3 or 14) (see FIG. 3).

(C) When a plurality of users arrive at the same table, each user can work while placing the portable computer. The surface of the table or wall also provides a work space that can be shared between user groups. For example, digital objects can be transferred via extended displays, such as tables and wall surfaces.
It can be freely transferred, exchanged, and distributed (see FIG. 4).

[0110] In short, the user can use digital objects or application objects handled on a computer.
A window or the like can be moved on an extended display configured on a table or a wall surface. In addition, a wider logical work space can be formed around the portable computer brought in by the user.

In the information space according to this embodiment, the GUI (G
Various interactive techniques developed for a graphical user interface can be introduced on an extended display deployed in the real world. That is, on an extended display composed of a table, a wall, or the like, a normal GUI such as a point, a drag, and a menu is selected using a user input device mounted on a portable computer such as a trackball, a touchpad, and a keyboard. Digital objects can be manipulated according to the operations of the format.

FIG. 5 illustrates a state of a user operation performed on the extended display prepared in the information space 1.

(A) The user can connect a digital object on a computer display to a pointing object.
Drag according to the normal GUI operation using the device.

(B) As the cursor reaches the periphery of the display screen and continues to be dragged, the cursor jumps from the screen onto the table 11 (actually, an object is projected onto the table by the projector). . At this time, the entity of the digital object (for example, a corresponding document file or folder) is transferred from the portable computer to the environmental computer.

(C) The user can further continue the drag operation to move the digital object to a surface other than the table, for example, to the wall surface 12. If an information carrier such as a calendar exists on the wall surface 12, the object can be moved to that location and attached to the calendar (actually, a link between the calendar and the information held by the object is formed).

(D) The user can also drop a digital object on a real physical object placed on a table such as a VCR tape. The effect of the drop on the physical object is to form a link between the real object and the digital or virtual object.

When performing an object operation as shown in FIG. 5, it is sufficient for the user to pay attention only to the physical arrangement between the objects, such as each computer and other physical objects. . The information environment according to this embodiment provides an integrated spatial metaphor for manipulating information.

Therefore, a digital object stored in a portable computer, a digital object appearing on an extended display such as a table or a wall, or a digital object linked to a physical object on a table can be exchanged. Intuitive and seamless handling between them.

As described with reference to FIG. 5, when the user wants to transfer data stored in the portable computer onto an extended display such as a table, the user can transfer the corresponding digital object from the computer screen. All you have to do is drag on the table, that is, across the boundaries of the computer. mouse·
When the cursor reaches the edge of the computer screen, the cursor automatically moves from the computer screen to the table. Of course, by simply dragging and dropping the digital object onto the physical object on the table, the digital object can be attached to the physical object, that is, a link can be formed.

In the present embodiment, the interactive technique developed for the GUI is adopted instead of the interactive technique such as hand gesture recognition because the notebook computer already has an interactive device having abundant applications. Relies on the fact that they are integrated. With the built-in dialogue device, the user remains in the user interface style already established and familiar,
Digital content can be handled on tables and walls.

In recent years, notebook computers have often been equipped with audio input / output devices, so that it is possible to use an interactive method of generating sound while working on the extended display.

Further, an input device such as a microphone and a camera provided in the portable computer can be used as a means for creating a voice memo during a meeting.

2-2. Spatial continuity of information operation In an information environment provided in the information space 1 as shown in FIG. 1, spatial continuity of information operation is realized.

Assuming an environment where a variety of information devices including computers are dynamically added, it is an excessive burden for the user to accurately grasp the network address and device name of each device. It is even annoying. On the other hand, in a closed work space such as a conference room, the situation is different from telecommunications, and most devices are in the range visible to the user, so the object to be operated can be grasped in real space. It should be possible to directly manipulate information without using indirect or symbolic concepts such as address, address, etc.

In the information space 1 according to the present embodiment, for example,
Instead of indirect designation methods such as “transfer files from computer ABC to computer DEF” and “send slide data to projector C”, “right computer to left computer”, “left screen”, etc. The device can be intuitively and directly specified by such a spatial positional relationship in the real world.

In this embodiment, physical operations performed by the user, such as "putting a computer on a table" and "placing a PDA close to a blackboard", are reflected on the network connection.

2-3. Cooperation with Objects Existing in the Real World Further, in an information environment provided in the information space as shown in FIG. 1, cooperation with objects existing in the real world is realized.

In everyday life, non-electronic / non-digital objects other than computers (for example, actual documents and video tapes) are frequently used. Preferably, such physical objects can be captured as part of the information environment. For example, you could select a three-dimensional model from a printed brochure and digitize it and bring it to a table (ie, a luxurious desktop) or portable computer, or carry the relevant slide data with other meeting materials. It is preferable that this can be performed easily and smoothly.

By keeping the above-mentioned principle of “spatial continuity”, the load on the user can be reduced. For example,
In a user interface in which information, that is, a digital object can be directly extracted from a brochure by dragging and dropping, the user does not need to be particularly aware of the connection relationship (mapping) between the brochure and the digital content.

In the present embodiment, cooperation with objects in the real world is realized by camera-based object recognition technology.

For example, a visual marker that is easy to visually recognize and identify is attached to each physical object in the real world. An example of a visual marker is "Cyber Code" ^{/ * /} (see FIG. 17). The camera recognition system can identify the physical object by imaging and analyzing the visual marker, and can derive a digital object linked to the physical object. Also, by dragging and dropping a digital object onto a physical object, a link can be newly formed between the physical object and the digital object.

For research on linking real world objects with electronic information such as digital contents, Info.
Binder ^[1] , mediaBlocks ^[2] , Pas
sage ^[3] . However, the cases proposed in these papers only use physically embodied icons for the movement of digital objects, and the role of such physical icons in the real world It does not confer any or use.

On the other hand, in the information environment according to the present invention,
It is possible to form a link relationship between a digital object and an object having a specific role / use in the real world. Document folders, documents, cards, videos,
A physical object such as a tape itself can be an object combined with electronic information. For example, a digital photo of the contents can be virtually attached to a video tape and carried around, or editorial instructions can be attached as a digital voice note. Further, it is also possible to attach slide data related to a meeting to a document folder.

3. Object Operation Functions Hereinafter, the object operation functions provided in the information space 1 according to the present embodiment will be listed.

3-1. Installation of Portable Computer in Information Space When a user arrives at table 11 and places his portable computer on it, the portable computer is installed in information space 1. This installation operation actually includes the following processing procedure.

(1) The cameras 18 and 19 take images of the portable computer. (2) The image processing server 20 recognizes the visual marker 31 attached to the portable computer based on the images captured by the cameras 18 and 19, and specifies the owner user of the portable computer. (3) The image processing server 20 also recognizes the position information where the portable computer is placed.

3-2. Hyperdrug (Hyperd
If the user wants to show the data held in his or her portable computer to other users, a "hyperdrag" (H
An interactive user operation called hyperdragging can be used. “Hyper drag” refers to an operation of inheriting a drag operation of an object on a computer display of a certain user to a plane on which a computer is installed as it is.

Hyper-drag is realized by the following processing procedure.

(1) The user inputs a desired item (display object) on the computer screen with the mouse.
Select with the cursor and drag toward the perimeter of the screen (see FIG. 6).

(2) When the cursor reaches the periphery of the computer screen, the mouse cursor jumps out of the computer screen and moves to the table 11 on which the computer is placed (see FIG. 7). During this time, if the mouse cursor is holding the object, the drag operation itself is inherited on the table 11 (ie, InfoTable) (ie, “hyperdrag”), and the digital operation is performed.
The object is displayed on the table 11. There is an "anchor" between the original portable computer and the digital object hyper-draged into InfoTable.
Cursor ”(described later) is the LCD projector 13 or 14
Displayed by

(3) Further, the user continues the drag operation while the mouse cursor is holding the digital object, and the table 11, ie, InfoTable,
When the edge of e is reached, the wall surface 1
2, a hyperdrag to InfoWall is performed (see FIG. 8). An "anchor cursor" (described below) is displayed by the LCD projectors 13 and 14 between the original portable computer and the digital object hyper-dragged into InfoWall.

The InfoWall is the InfoTable.
Together, they form an extended display of the information space 1.

A combination of two types of displays, a high-resolution small screen on a portable computer and a low-resolution large screen composed of InfoTable and InfoWall, is positioned as an information space to be watched by the user and an information space around the information space. The user can spread a number of data items (digital objects) around the portable computer while keeping an eye on the screen (see FIG. 9). Also, if one of the diffused data items is needed, the relevant digital object can be hyper-draged and instantly returned to the computer screen (see FIG. 10). ).

In such a user interface operation, the cameras 18 and 19 capture an image of the mouse cursor, and
This is enabled by the image processing server 20 performing image processing and image recognition on the captured image and tracking the movement of the mouse cursor. With hyperdrag, in the background, entities of digital objects (such as document files and digital content) move from the portable computer to the environmental computer (eg, data transfer via network 15 occurs). ).

The hyperdrag technology supports a metaphor that “the surrounding environment of a portable computer such as a table or a wall surface is a spatially continuous work space”. Users can place data items such as text and graphics around a portable computer as a virtual magnified computer desktop.

According to the hyperdrag, the user can intuitively operate the object only by intuitive knowledge of the physical positional relationship of the object in the real world (for example, “an object is near a computer”). And it can be easily realized.

According to the hyperdrag, information can be moved only by learning a normal mouse operation (drag and drop) and knowing the physical location (for example, the position on the table) of the target device. Can be done. That is, the user does not need any complicated work such as setting and connection of the network and knowing the address of the target device.

In order to implement a hyperdrag operation (see FIG. 16) for moving the mouse cursor between two or more computer screens, it is necessary to select a mouse sensing area. The area is selected according to the four corners of each screen of the portable computer installed in the information space 1.

In the information space 1, when the cursor enters the screen as the mouse sensing area, the position of the cursor is remapped on the screen. At the time of this remapping, an offset is calculated so that the cursor position on the table 11 is maintained. While the real (ie, original) cursor is near the edge of the screen of the portable computer, the user controls the position of the virtual cursor projected on the table 11 by continuing to operate the pointing device. Can be.

In order to accurately calculate the position of the cursor on the table 11, it is necessary to recognize the position and orientation of the portable computer placed on the table 11.
This is obtained by recognizing the visual markers (see above).

As a result of performing the hyperdrag operation, data as an entity of the dragged object must be transferred between computers (for example, from a portable computer to an environmental computer or between portable computers).

Application for realizing information space 1
By writing the program in the Java programming language and employing Java's object serialization mechanism, such object transfer can be performed using a remote method call (RMI: Remote).
e Method Invocation). For example, text, sound (voice note), URL (Uniform Resource)
Locator, file shortcuts, images and other digital data.

3-3. Anchor cursor (Ancho
While the user is operating the cursor outside the computer screen, that is, while working on the extended desktop, a line segment connecting the portable computer and the cursor position is projected (FIGS. 7 and 8). checking). Such visual feedback is called "anchor cursor" by the present inventors.

The anchor cursor is a visual and spatial indicator for indicating the owner of the cursor.

In a conference or the like, when a plurality of users simultaneously operate digital objects on the extended display, a large number of cursors are moved to the table 11 or the wall 12.
Appear and become complicated. Using the anchor cursor as a clue, all conference participants can identify the owner of each cursor.

The anchor cursor can represent a semantic relationship between different displays. For example, when using a large map projected on the table 12 for route guidance, travel guidance, and the like, the portable computer can constantly display detailed information at a point corresponding to the current cursor pointing position. Yes (Fig. 11
checking). That is, the anchor cursor means a visual connection between the two.

3-4. Information Sharing Space The surfaces of InfoTable and InfoWall also have aspects as an integrated information space shared between users. If two or more users are in the InfoTable
The user can freely place digital objects on the table 12 from his or her portable computer.

Conventional computer screens and “au”
Unlike the “gmented desk system” ^[4] , “top” and “bo” on the computer screen
There is no absolute concept such as “tom”. For this reason, the multi-user function of InfoTable has to distinguish between the displayed digital object and the orientation of the user interface. Design issues are introduced.

In this embodiment, the InfoTable uses the spatial position of each portable computer recognized through the cameras 18 and 19 to determine which side is closer to each user. For example, if a user brings a digital object representing a diagram from a distance to a front on the InfoTable, the system (actually, an environmental computer) is oriented so that the user can easily read the information. The object is automatically rotated.

3-5. Object Aura (Object
ct aura) In the information space 1 according to the present embodiment, association (binding or link formation) between a physical object and a digital object is also possible.

When a physical object such as a VCR tape cartridge is placed on the table 11,
The captured images of the CCD cameras 18 and 19 are recognized and identified by the environmental computer. And FIG.
As shown in (1), a substantially elliptical shape surrounding the recognized physical object is displayed.

The present inventors call this oval display an "object aura". By visual feedback via the object aura, the user can know that the physical object has been correctly recognized by the environmental computer (or installed in the information space 1).

The object aura displays the data space of the corresponding physical object. Thus, the user can hyper-drag a digital object on table 11 (see FIG. 13) and drop it into the object aura (FIG. 14).
), The digital object can be attached or linked to a physical object.

Object of digital object
The drop operation into the aura is imaged by the camera 18 or 19 and recognized by an environmental computer such as the image processing server 20. Further, the entity of the dropped digital object is transferred from the portable computer to the environmental computer, which stores the entity in association with the physical object.

For example, the VC placed on the table 11
When a voice memo is to be attached to the R tape cartridge, it is realized according to the following processing procedure. (1) The user, for example, on a portable computer,
Create and edit desired voice notes. (2) Digital data with a voice note as an entity
Hyper-drag an object onto the table 11 from the screen of the portable computer. (3) Further, the digital object is stored in table 1.
1. Hyper-drag on 1 to move into the object aura of the VCR tape cartridge. (4) If you release the mouse button in the object aura and drop it, the voice note will be recorded on the VCR tape.
Linked to the cartridge. (5) When someone physically moves the VCR tape cartridge, the entity of the digital object associated therewith is an environmental computer (eg, server 2).
0). (6) Insert the VCR tape cartridge into the table 11 again.
When placed on top, playback of the digital object (ie, the associated voice note) is performed.

<< Note >> * Cyber code: A cyber code is a kind of two-dimensional bar code. As shown in FIG. 17, a "guide / bar display area" for indicating the location of the cyber code,
And a "code pattern display area" for displaying a dimensional code pattern. The code / pattern display area is composed of cells arranged in an n × m matrix (7 × 7 in FIG. 1), and identification information can be given by expressing each cell in white or black. . However, the corner cells at the four corners of the code pattern display area are not used as identification information but are registered (registrati).
on) The pattern is always a black pattern. The cyber code recognition procedure includes a step of binarizing the captured image, a step of finding a guide bar candidate from the binary image, and a step of searching for a corner cell based on the position and direction of the guide bar. And decoding an image bitmap pattern in response to detecting a guide bar and a corner cell. Further, by checking the error bit, it is possible to confirm that a correct cyber code is included in the captured image, and to derive identification information and position information of the code. Further, based on the positions of the corner cells, it is possible to calculate and compensate for the distortion caused by the inclination of the camera or the object. For details of the cyber code, see, for example, Japanese Patent Application No.
-184350 (“Image processing apparatus and method,
As well as providing media ”).

[Supplement] The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0168]

As described above in detail, according to the present invention,
It is possible to provide an excellent user interface environment capable of intuitively and intuitively expressing a cooperative operation between a plurality of computers in an information space extended to the real world.

Further, according to the present invention, an excellent user / user who can intuitively and intuitively express cooperative work on an information space in which each desktop screen native to each computer is extended to the real world. An interface environment can be provided.

Further, according to the present invention, a portable computer brought into a conference room and an environmental computer installed in the same room dynamically configure a user interface environment that inherits a conventional GUI operation. Can be.

According to the present invention, the computer operating environment is dynamically extended by a group of computers installed in a room environment such as a conference room and a group of portable computers carried by each user such as a conference participant. It is possible to provide an information space that can be used.

Further, according to the present invention, the operating environment is dynamically expanded by a portable computer group brought by each user such as a conference participant, so that a computer or a computerized table or wall of a conference room can be used. It is possible to provide an “easy-to-use” information space in which information can be smoothly exchanged with a computer.

In the information space according to the present invention, the table and the wall function as a work environment (that is, an “extended display”) in which the portable computer is extended in the real world space. In addition, physical objects (paper documents, cards, video tapes, etc.) in the real world are specified by marker recognition, and free information exchange between these objects is enabled. Users can utilize the spatial relationship of things without having to be aware of the addresses and IDs of individual devices and objects, using direct manipulation techniques that work across the boundaries of the closed workspace of a single computer. Information can be manipulated.

According to the present invention, a drag operation performed in digital space on a user's computer screen is performed.
"Hyper-drag" operation, which extends the and-drop operation to the real world space as it is, becomes possible. For example, object operations on a screen are performed on a table (InfoTable) on which a computer is installed or on a wall surface (InfoWal) of a room where a computer is brought.
l) and so on. Also, by hyper-drag,
Digital objects can be freely moved between multiple computers. Further, the user can move the object to a place away from his / her seat, such as a wall surface, while staying at the table.

In addition, according to the present invention, a digital object can be attached to a physical object and moved.

For example, when a plurality of users are simultaneously operating an object on a table (especially when operating an object at a distant position), the "anchor cursor" is effective.

Further, computer peripheral devices such as a printer, a display, and a scanner may also support the hyperdrag function. For example, the printout of the digital object may be executed by hyper-drag and dropping the digital object onto a printer installed in the information space.

[0178] Some users find it difficult to move objects at distant places. The technology proposed in document [7] can be incorporated in the information space according to the present invention. Cursor movement and pointing
Mapping the scale between devices will also improve the user experience. Since the resolution of the LCD projector (20 dpi) is coarser than the resolution of the portable computer (100-110 dpi), mapping without scaling will result in a discontinuous change in cursor movement speed at the boundary between the computer screen and the table. .

The “hyperdrug” according to the present invention
Is different from a multi-device interactive operation technique called "pick-and-drop" ^[8,9] . Pick-a
nd-Drop uses the stylus of the digitizer when picking up display objects from one screen and dropping them on the other screen. Pick-and-Dr
The physical metaphor is more direct than hyperdrag in that op is similar to picking up a physical object. However, according to the hyperdrag, it is possible to operate the object even in a place where the user's hand cannot actually reach,
Not possible with -and-Drop. Also, Pick
While realizing -and-Drop requires a stylus-sensitive surface, hyperdrag works on any display screen or projected image surface.

A suitable operation environment differs depending on the pointing device and the style of the interactive operation. Pic
Obviously, k-and-Drop is compatible with pen operation. On the other hand, hyperdrag is not suitable for pen operation because it is not possible to directly map between the pen position and the cursor position. On the other hand, hyperdrugs are extremely compatible with trackballs and pointing devices called trackpoints, which are standard equipment in portable computers.

The information space 1 according to the present invention is applied not to a multi-user environment such as a conference room but to an individual user environment, and a digital object is moved from a computer to a wall surface of a partition to write a post-it memo. It can be set as follows. Also, when you want to attach a To-Do item on the schedule,
You may hyper-drag the digital object to the calendar on the wall.

References [1]: Itiro Sio. InfoBinde
r: a pointing device for
a virtualdesktop system.
In 6th International Conf
erence on Human-Computer
Interaction (HCI Interna
temporal '95), pp. 261-264,
July 1995. [2]: Brygg Ullmer, Hiroshi
Ishii, and Dylan Glas. Me
diaBlocks: Physical container
inners, transports, and co
ntlsfor online media. I
n SIGGRAPH '98 Proceeding
s, pp. 379-386, 1998. [3]: Torsten Holmer Norber
t A. Streetz, Jorg Geisle
r. Roomware for cooperati
ve buildings: Integrated
design of architectural s
paces and information spa
ces. In Norberts A. Strei
tz and Shin'ichi Konomi,
editors, Cooperative Build
dings-Integrating Info
nation, Organization, and
Architecture, 1998. [4]: Pierre Wellner. Inter
acting withpaper on the D
digitalDesk. Communication
of the ACM, Vol. 36, no. 7,
pp. 87-96, August 1993. [5]: John Underkoffler and
Hiroshi Ishii. Illuminat
ing Light: An opticaldesi
gn tool with a luminous-t
angible interface. In CHI '
98 Proceedings, pp. 542-54
9, 1998. [6]: Jun Rekimoto. Matrix:
A realtimeobject identifier
ication and registration
method for augmented real
ity. InProc. Of Asia Paci
fic Computer Human Intera
ction (APCHI'98), July 19
98. [7]: Jorg Geisler. Shuffl
e, throw ortake it! Worki
ng efficiently with anint
eractive wall. In CHI'98
summary, February 1998. [8]: Jun Rekimoto. Pick-an
d-Drop: A Direct manipula
Tion Technique for Multiple
le Computer Environments.
In Processeds of UIST'9
7, pp. 31-39, October 199
7. [9]: Jun Rekimoto. A multi
ple-deviceapproach for su
reporting whiteboard-based
interactions. In Proceed
ings of CHI'98, February 1
998. [10]: M.P. Stefik, G .; Foster,
D. Bobrow, K .; Khan, S .; Lann
ing, and L. Suchman. Beyon
d the challengeboard: compute
r support for collaborati
on and problem solving in
meetings. Communication
of the ACM, Vol. 30, no.
1 pp. 32-47, 1987. [11]: Pierre Wellner. The
DigitalDeskcalculator: Ta
NGILE MANIPULATION ON A
desdk top display. In Pro
seedings of USIT'91 ACM S
ymposium on User Interfac
e Software and Technology
y, pp. 27-34, November 19
91. [12]: Peter Robinson, Dan
Sheppard, Ricard Watts, R
obert Harding, and Steve
Lay. Animated Paper Docum
ents. In7th International
Conference on Human-Comp
Uter Interaction, HCI'97,
1997. [13]: Eric Sound. ZoomieBo
ard project description. h
http: / www / parc. xerox. com /
spl / members / sound / zoomieb
oard-public. html [14]: Questin Stafford-Fra
ser and Peter Robinson. B
rightboard: A video-augume
nted environment. In CHI '
96 processes, pp. 134-1
41, 1996. [15]: Steven Feiner and A.
Shamash. Hybrid user int
erfakeds: Breeding virtua
lly bigger interfaces for
physically smallr compu
ters. InProceedingsof UIS
T'91, ACM Symposium on Us
er Interface Software and
Technology, pp. 9-17, No
venber 1991. [16]: W. E. FIG. Mackay, D.M. S.
Pagani, L .; Faber, B.S. Inwood
d, p. Launiainen, Brenta,
and V. Pouzol. Ariel: au
gmenting paper engineerin
g drawings. In CHI'95 Con
reference Companion, pp. 42
-0422, 1995. [17]: Hiroshi Ishii and Br
ygg Ullmer. Tangible Bits:
Towards seasons interfa
ces bewee people, bits
and atoms. In CHI'97 Proc
eatings, pp. 234-241, 199
7. [18]: Beth M. Language, Mark
A. Jones, and James L. Me
yers. Insight Lab: Animme
rsive team environment li
nking paper, displays, an
d data. InCHI'98 Proceedi
ngs, pp. 550-557, 1998 [19]: Myron W.S. Krueger. A
rtificial Reality II. Add
Ison-Wesley, 1990.

[Brief description of the drawings]

FIG. 1 is a diagram schematically showing a configuration example of an information space according to the present invention.

FIG. 2 is a diagram showing a process in which a portable computer of each user is extended by an environmental computer;
More specifically, it is a diagram showing a state where a user is working on his or her portable computer.

FIG. 3 is a diagram showing a process in which a portable computer of each user is extended by an environmental computer;
More specifically, it is a diagram illustrating a state where a table or a wall surface of a conference room is used as a spatial extension of a desktop of a portable computer.

FIG. 4 is a diagram showing a process in which a portable computer of each user is extended by an environmental computer;
More specifically, it is a diagram illustrating a state in which a plurality of users work together on a digital object on an extended display such as a table or a wall surface.

FIG. 5 is a diagram schematically showing a state in which a drag-and-drop operation performed on a computer desktop is expanded on an extended display outside the desktop.

FIG. 6 is a diagram for explaining a hyper-drag operation, more specifically, a state in which a desired item on a computer screen is selected with a mouse cursor and dragged toward the periphery of the screen. FIG.

FIG. 7 is a diagram for explaining a hyperdrag operation, more specifically, a diagram showing a state in which a mouse cursor that has reached the periphery of the computer screen moves from the computer screen to the table 11; is there.

FIG. 8 is a diagram for explaining a hyperdrag operation, and more specifically, a table 11 (InfoTabl);
FIG. 11E is a diagram showing a state in which hyperdrag is executed from e) to the wall surface 12 (InfoWall).

FIG. 9 is a diagram for explaining a hyper-drag operation, and more specifically, a diagram showing a state in which several digital objects are scattered around a portable computer.

FIG. 10 is a diagram for explaining a hyper-drag operation, more specifically, a diagram showing a state in which a digital object is hyper-draged and instantly returned to a computer screen.

FIG. 11 is a diagram for explaining the function of an anchor cursor. More specifically, as an example showing a semantic relationship between different displays, route guidance is performed using a large map projected on a table 12. FIG. 5 is a diagram illustrating a state in which the portable computer displays detailed information at a point corresponding to the current cursor pointing position during the operation.

FIG. 12 is a diagram showing a state in which a physical object placed on the table 11 is recognized by an environment-type computer, and more specifically, a substantially elliptical shape (“object aura”) surrounding the physical object; ) Is a diagram showing a state where is displayed.

FIG. 13 is a diagram illustrating a state where a digital object is attached to a physical object, and more specifically, a diagram illustrating a state where a digital object on a table 11 is hyper-dragged toward an object aura. is there.

FIG. 14 is a diagram illustrating a state in which a digital object is attached to a physical object, and more specifically, a diagram illustrating a state in which a digital object hyper-dragged on a table 11 is dropped into an object aura. It is.

FIG. 15: Fixed camera 18 and pan / tilt camera 1
FIG. 9 is a diagram showing a mechanism for detecting a change on the table 11 using No. 9;

FIG. 16 illustrates a hyperdrag operation performed between two or more computer screens.

FIG. 17 is a diagram schematically showing the structure of a cyber code.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Information space 11 ... Table (InfoTable) 12 ... Wall surface (or white board) (InfoWal)
l) 13, 14 LCD projector 15 Network 16 17, Computer system (environmental computer) 18 Fixed camera 19 Tilt / pan camera 20 Image processing server (environmental computer) 31, 32 Visual marker

Claims (35)

[Claims] 1. An information input / output system for performing a user operation on a digital object in an information space in which a digital space of a computer is extended to the real world, wherein at least one of the information input / output systems is provided in the information space. A work surface; display means for displaying an image on the work surface; imaging means for taking an image of the work surface; and an environment-type computer fixedly installed in the information space. A process of recognizing position information of an object on the work surface and / or a user operation on the object based on an image captured by the imaging unit, and a process of controlling display of a digital object by the display unit according to the recognition result And an information input / output system. 2. The information input system according to claim 1, wherein the information space is formed in a predetermined room, and a table surface and / or at least one wall surface installed in the room is used as a work surface. 3. The fixed camera comprises a fixed camera having a fixed visual field and capable of capturing a relatively wide range, and a variable camera having a variable visual field and capable of capturing a relatively narrow range at a high resolution. The information input / output system according to claim 1, wherein a change in an object on the work surface is detected by further capturing an area in which a difference has occurred between a current and a past captured image by a variable camera. . 4. A physical object placed on the work surface includes a visually identifiable visual marker on a surface, and the environment-type computer uses the visual marker in the image captured by the image capturing means based on the visual marker. 2. The information input / output system according to claim 1, wherein the information input / output system recognizes identification information and / or position information of a physical object. 5. An information input / output system for performing a user operation on a digital object in an information space in which a digital space of a computer is extended to the real world, wherein at least one of the at least one information space provided in the information space is provided. A work surface, display means for displaying an image on the work surface, image pickup means for taking an image of the work surface, an environmental computer fixedly installed in the information space, and movable in the information space And one or more portable computers capable of exchanging entities of digital objects with other computers, wherein the environment-based computer is an object on the work surface based on images taken by the imaging means. Location information and / or
Or a process of recognizing a user operation on an object, and a digital
A process for controlling display of an object and a process for dynamically installing the portable computer in the information space by recognizing identification information and position information of the portable computer installed on the work surface are executed. , An information input / output system. 6. The information input system according to claim 5, wherein the information space is formed in a predetermined room, and a table surface and / or at least one wall surface installed in the room is used as a work surface. 7. The fixed camera comprises a fixed camera having a fixed visual field and capable of capturing a relatively wide range, and a variable camera having a variable visual field and capable of capturing a relatively narrow range at a high resolution. 6. The information input / output system according to claim 5, wherein a change in an object on the work surface is detected by further capturing an area where a difference has occurred between the current and past captured images by a variable camera. . 8. A physical object placed on the work surface includes a visually identifiable visual marker on a surface thereof, and the environment-type computer uses the visual marker in the image captured by the image capturing means based on the visual marker. The information input / output system according to claim 5, wherein identification information and / or position information of the physical object is recognized. 9. The environment type computer according to claim 5, further comprising executing a process of recognizing an operation of the digital object on a display screen of the portable computer based on a captured image. Information input and output system. 10. The environment-type computer further executes a process of recognizing a drag operation of a digital object on a display screen of the portable computer based on a captured image. Information input and output system described. 11. The environment-based computer further tracks a drag operation of a digital object on a display screen of the portable computer based on a captured image, and a cursor reaches an end of the display screen. 6. The information input / output system according to claim 5, wherein a digital object is displayed at a corresponding position on the work surface, and a process of continuing a drag operation on the work surface is executed in response to the information. 12. The portable computer, in response to dropping a digital object on the display surface of the portable computer onto the work surface, causes the entity of the digital object to be transferred to the portable computer. 6. The information input / output system according to claim 5, wherein the information input / output system executes a process of transferring from a computer. 13. The work environment according to claim 5, wherein the environment type computer displays the correspondence between the digital object dropped on the work surface and the original portable computer on the work surface. Information input and output system. 14. The information input / output system according to claim 5, wherein two or more work surfaces exist in the information space, and a drag operation of a digital object can be performed between different work surfaces. . 15. The environment type computer further tracks a drag operation of the digital object on the work surface based on the captured image, and responds to a cursor reaching a boundary with the portable computer main body. Performing a process of transferring the entity of the digital object to the portable computer, wherein the portable computer performs a process of receiving the digital object and continuing a drag operation on a display screen. The information input / output system according to claim 5, wherein 16. The information input device according to claim 5, wherein the environment-type computer executes a process of displaying an object aura indicating a data space of the physical object recognized on the work surface. Output system. 17. The information input / output according to claim 5, wherein the environment type computer executes a process of associating the entity of the digital object dropped on the object aura with the corresponding physical object. system. 18. An environment-based computer, in response to detecting that a physical object has been placed on a work surface based on a captured image, performs processing of a digital object associated with the physical object. The information input / output system according to claim 5, wherein the information input / output system is executed. 19. There is one or more computer peripherals in the information space for performing predetermined object processing, and in response to a digital object on the work surface being dropped on the computer peripheral. 6. A predetermined object process is executed.
Information input and output system described in. 20. An information input / output method for performing a user operation on a digital object on an information space obtained by expanding a digital space of a computer into the real world, the information space comprising one or more work surfaces, Display means for displaying an image on a work surface, image pickup means for taking an image of the work surface, and an environment-type computer which is fixedly installed. Recognizing the position information of the object and / or a user operation on the object, and controlling the display of the digital object by the display means according to the recognition result. 21. An information input / output method for performing a user operation on a digital object in an information space in which a digital space of a computer is extended to the real world, wherein the information space includes at least one work surface. Display means for displaying an image on the work surface, imaging means for taking an image of the work surface, and an environment-type computer fixedly installed,
It is comprised of one or more portable computers movable in the information space and capable of exchanging entities of digital objects with other computers, and (a) the work is performed based on images taken by the imaging means. Recognizing positional information of the object on the surface and / or user operation on the object; (b) controlling display of the digital object by the display means according to the recognition result; and (c) displaying the digital object on the work surface. Dynamically installing the portable computer in the information space by recognizing the identification information and the position information of the installed portable computer. 22. The information space is formed in a predetermined room, and a table surface and / or at least one of the information spaces is installed in the room.
22. The information input method according to claim 21, wherein one wall is used as a work surface. 23. The imaging means comprises a fixed camera having a fixed visual field and capable of capturing a relatively wide area, and a variable camera having a variable visual field and capable of capturing a relatively narrow area with high resolution. a) calculating a difference between current and past images captured by the fixed camera; and changing an object on the work surface by further capturing an image of the area where the difference exceeds a predetermined threshold value using a variable camera. 22. The information input / output method according to claim 21, further comprising a sub-step of detecting 24. The physical object placed on the work surface includes a visually identifiable visual marker on the surface, and in the step (a), the visual marker in the image picked up by the image pickup means is displayed. 22. The information input / output method according to claim 21, wherein identification information and / or position information of the physical object is recognized based on the information. 25. The information input / output method according to claim 21, further comprising a step of recognizing an operation of the digital object on a display screen of the portable computer based on a captured image. 26. The information input / output method according to claim 21, further comprising the step of recognizing a drag operation of a digital object on a display screen of the portable computer based on a captured image. 27. A method of tracking a drag operation of a digital object on a display screen of the portable computer based on a captured image, and responsive to a cursor reaching an end of the display screen, 22. The information input / output method according to claim 21, further comprising: displaying a digital object at a corresponding position on a work surface and continuing a drag operation on the work surface. 28. In response to dropping a digital object on the display surface of the portable computer onto the work surface, the digital object
22. The information input / output method according to claim 21, further comprising a step of transferring an entity of an object from the portable computer to the environment type computer. 29. The method according to claim 21, further comprising the step of displaying on the work surface the correspondence between the digital object dropped on the work surface and the original portable computer. Information input / output method. 30. The information input device according to claim 21, wherein there are two or more work surfaces in the information space, and further comprising a step of dragging a digital object between different work surfaces. output method. 31. A step of tracing a drag operation of a digital object on the work surface based on a captured image, and in response to a cursor reaching a boundary with the portable computer main body, receiving a signal from the environment type computer. 22. The method of claim 21, further comprising: transferring the entity of the digital object to the portable computer; and continuing a drag operation of the digital object on a display screen of the portable computer. Information input / output method described in. 32. The information input / output method according to claim 21, further comprising a step of displaying an object aura indicating a data space of the physical object recognized on the work surface. 33. The information input / output method according to claim 21, further comprising associating an entity of the digital object dropped on the object aura with a corresponding physical object. 34. In response to detecting that a physical object has been placed on a work surface based on a captured image,
22. The information input / output method according to claim 21, further comprising a step of executing processing of a digital object associated with the physical object. 35. One or more computer peripherals for performing predetermined object processing in said information space, and further responsive to a digital object on said work surface being dropped on said computer peripheral. 22. The information input / output method according to claim 21, further comprising a step of executing a predetermined object process.