JP2000172402A - Image display device and computer-readable recording medium where program thereof is recorded - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image display device which is intuitive and easy to understand by pointing one of a real body in the real world or virtual world information displayed on a display and interlocking them each other according to pointing. SOLUTION: The device includes the display 2, a light pointer 1, a pointing device 3, an object position attitude sensor 4 which measures the position attitude of an object, and a computer 100 which controls them and is stored with electronized data made to correspond to the object according to a predetermined rule and the projection light position of the light pointer 1 is made to always correspond to electronized data displayed on the display 2, when the projection light position of the light pointer 1 is moved by the pointing device 3, the electronized data are scrolled and when the electronized data are scrolled by the pointing device 3, the projection light position of the light pointer 1 is moved.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image display apparatus and a computer-readable recording medium storing the program, and more particularly to a method for displaying digitized data obtained by digitizing an object such as a document or a three-dimensional object. In addition, the present invention relates to a pointing or scrolling image display device and a computer-readable recording medium storing the program.

[0002]

2. Description of the Related Art First, an outline of the present invention will be described. The image display device of the present invention and a computer-readable recording medium storing the program (hereinafter simply referred to as “image display device”)
It is described. ) Is such that the pointing position with respect to the object in the real world is linked with the pointing position on the scroll or scroll of the display image which is the virtual world displayed on the display. Specifically, Linking the scrolling of the display image or the movement of the pointing with respect to the display image with other inputs, and directly pointing the object in the real world, and directly pointing the object in the real world, and the pointing information and An image displayed on the display, that is, a correspondence with the virtual world is performed. Therefore, some examples of the prior art will be described below from these three viewpoints.

(Prior Art Related to Characteristics) When browsing a document manuscript being created, image data such as a map or a photograph, etc. on a display such as a personal computer, a portion desired to be viewed in detail is enlarged, or the entire state is viewed. The operation of shrinking to confirm is frequently performed. For this reason, Japanese Patent Application Laid-Open No. Hei 7-146951 (Japanese Patent Application Laid-Open No. H07-146571) discloses a conventional technique for providing an excellent image display by simultaneously displaying an enlarged display and a reduced display and linking them. Device)) and JP-A-1-265292 (the title of the present invention, "image display device").

(Japanese Patent Application Laid-Open No. Hei 7-146951: FIG. 10) FIG. 10 is a schematic diagram showing the movement of a display screen of the document generating apparatus disclosed in Japanese Patent Application Laid-Open No. Hei 7-146951. The display 302 has a window 3
00 and an enlarged window 301 are displayed. The window 300 includes “$ 123-4567” and “Name:
The image data obtained by reading the document manuscript described as "Taro Patent" by the image input device is reduced and displayed so that the entire document manuscript is displayed. The user operates the pointing device 303 to move the cursor to a portion of the window 300 described as “$ 1” of “$ 123-4567”. At this time, enlarged window 3
At 01, an enlarged window display area indicated by a dotted line is displayed in the window 300, and at the same time, a cursor is displayed at the position of "$ 1" (FIG. 10A).

Next, the user operates the pointing device 3.
03, the cursor in the window 300 is moved to the part of "Kota" of "Name: Kotaro", the display of the enlarged window 301 is scrolled in conjunction therewith, and at the same time, the cursor in the enlarged window 301 is moved. Are also displayed at the position of “Kota” (FIG. 10B).

That is, in the prior art disclosed in Japanese Patent Application Laid-Open No. 7-146951, a scroll operation of the enlarged window 301 is reduced by pointing a desired position in the window 300 displayed in a reduced scale. I have.

In order to make it easier to compare the prior art with the present invention, a document manuscript in which "$ 123-4567" and "Name: Taro Taro" are displayed as examples to be displayed in the description of the embodiment of the present invention. Was also used in the description of the prior art. This does not hinder the essence of the prior art.

(Japanese Patent Application Laid-Open No. 1-265292: Image Display Device) FIG. 11 is a schematic diagram showing the movement of the display screen of the image display device disclosed in Japanese Patent Application Laid-Open No. 1-265292. The display 402 shows “$ 123-45
Image data obtained by reading a document manuscript described as “67” and “Name: Taro Patent” by an image input device is displayed. Further, an enlarged window 401 displaying image data obtained by enlarging a part of the display image data is superimposed and displayed at a position corresponding to the part of the image data being enlarged and displayed. A user not shown can operate the pointing device 403 to move the enlarged window 401.

Referring to FIG. 11, an enlarged window 40 is shown.
Reference numeral 1 denotes image data obtained by enlarging the portion at the position "67" of "$ 123-4567" (FIG. 11).
(A)). Next, the user operates the pointing device 40.
3 is operated to move the enlargement window 401 to the “ro” portion of “name: patent taro”.
In FIG. 11, "ro" is enlarged and displayed (FIG. 11).
(B)).

That is, in the prior art disclosed in Japanese Patent Application Laid-Open No. 1-265292, a desired position can be enlarged and displayed by moving the enlargement window 401.

As described above, Japanese Patent Application Laid-Open No. Hei 7-146951 and Japanese Patent Application Laid-Open No. 1-26529
Japanese Patent Application Laid-Open No. 4-145587 (Title of Invention, "Picture Picking Method of Image Processing Apparatus") and Japanese Patent Application Laid-Open No. 4-52779 (Title of Invention, Business Card Recognition Apparatus "), JP-A-2-252
No. 064 (title of the invention, "cad").
However, in terms of linking images,
JP-A-7-146951 and JP-A-1-2652
It is the same as in the '92 publication and a detailed description is omitted.

(Prior Art Regarding Features) As a conventional technique for directly pointing a real object using laser light, for example, Japanese Patent Application Laid-Open No. 9-46776 (name of the invention,
"Remote pointing device and method") and Japanese Patent Laid-Open No. 5-83705
An outline of the official gazette (name of the invention, "still image input device") will be described.

(Japanese Unexamined Patent Publication No. 9-46776 "Remote indication device and method") FIG. 12 shows a configuration of a remote indication device proposed in Japanese Unexamined Patent Application Publication No. 9-46776. Pointing device 5
00 and a remote device 505 are connected via a network 504. The remote device 505 captures an image of the document 506 at a remote location, and transmits the image to the pointing device 500 via the network 504. The pointing device 500 displays the image of the document sent from the remote device 505 on the display 501. The presenter uses the keyboard 502 or the mouse 503 of the pointing device 500 to point a desired position of a remote document on the display 501. The pointing information is sent to the remote device 505 via the network 504. Remote device 505
Projects a laser beam on the actual position on the document 506 corresponding to the position indicated by the presenter on the display 501, based on the pointing information sent from the pointing device 500.

(Japanese Patent Laid-Open No. 5-83705 "Still Image Input Device") FIG. 13 shows the configuration of a still image input device proposed in Japanese Patent Laid-Open No. 5-83705. This device relates to a pointing device in a document camera for presenting materials in a video conference or the like. When the pointing device 605 is operated, the pointing device 605 sends the pointing information to the CPU 601. The CPU 601 controls the laser beam emitting device 603 based on the sent pointing information, and projects a laser beam on the document table 604. This allows
A presenter in a conference can directly point on a real object such as a conference material. Further, in the related art, an area for instructing image reading execution or the like is provided in advance on the document table, and when the area is pointed with a laser beam and clicked, the camera 602 reads the image. is there.

(Prior Art Regarding Features) "Augume
nted Reality "The image display device of the present invention
It is characterized in that not only the virtual world displayed on the display but also a real object existing in the real world such as a document manuscript are directly pointed and corresponded to each other. In other words, “Augmented” that fuses the real world and the virtual world
Reality ".

As a background art of the present invention, “Augmen
Ted Reality ”will be introduced, and a related art example will be described.

"Augmented Reality"
(Hereinafter abbreviated as “AR”) is a concept of a user interface proposed in 1993, and is described in, for example, Reference (1) (“Communications of
he ACM "Vol. 36, no. 7, pp. 24
-97).

In contrast to "Virtual Reality" (virtual reality), which attempts to virtually realize the entire real world, the real world is enhanced with a computer (Augume).
nt) to realize an interface that is easy for humans to understand. Among the systems developed based on this AR, "Digital Desk" and "Interacti" are related to the present invention.
veDESK ", which will be described below.

("Digital Desk") "Di
"Gital Desk" is an interactive desktop system proposed by PierreWellner of Rank Xerox, and its concept is described in, for example, Reference (2) ("Communications of the
e ACM "Vol. 36, no. 7, pp. 87-
96). This system is shown in FIG.
As shown in FIG. 4, a camera 701 and a projector 702 are installed above a work desk 704, and the projector 702 uses the work desk 704 to output an image created by the computer 703.
Projected upward (see projected image 705), camera 701
Captures human hand movements, handwritten characters, handwritten pictures, documents, and the like. The user operates the work desk 704 on which the image created by the computer 703 and the actual document are mixed,
Interactive work with the computer 703 becomes possible, and a so-called real world interface is realized. When a number portion in a document is designated by a finger, the portion becomes bright and a mark is displayed, and an application in which the designated number is pasted on a calculator formed by light projection is introduced.

Further, "D" by Rank Xerox
For example, Japanese Patent Application Laid-Open No. 7-168949 (title of the invention, "interactive copying apparatus") and Japanese Patent Application Laid-Open No. 9-23734
No. 9 (Title of Invention, "Interactive Desktop System").

Japanese Patent Application Laid-Open No. 7-168949 (title of the invention, "interactive copying apparatus") discloses a video projector for displaying information of a virtual world on a work desk,
A camera for monitoring the work desk is provided. It discloses how interaction with projection information by a video projector and creation of actual documents and pictures on paper can be realized by a user's manual operation. The intuitive operation of the interaction between the projected image of the video projector in the virtual world and the paper on the work desk in the real world can be performed by manual operation.

Japanese Patent Application Laid-Open No. 9-237349 (title of the invention, "interactive desktop system") also discloses a video projector for displaying information of a virtual world on a work desk and a video projector for monitoring the work desk. Equipped with a camera.
The camera captures an image of a real object or the like according to an instruction from a user, and can display and output the captured image via a computer with a video projector. That is, it is possible to copy and paste a real object such as a document manuscript on a work desk. Although a method using a mouse is described as a method for specifying an area to be imaged, a cursor indicating a pointing position is projected on a work desk as display information from a video projector.

("Interactive DESK")
“InteractiveDESK” is a system in which the entire desk is a dialog environment, and its concept and features are described, for example, in Reference (3) (Technical Report of the Institute of Television Engineers of Japan, Vol. 19,
No. 65 pp. This is described in detail in 1-6).

FIG. 15 shows this "Interactive"
1 shows a schematic configuration of “DESK”. As shown in FIG. 15, a work desk 804 whose work surface is a large display capable of pen input, and an upright display 80 in front of the user.
2, a camera 801 for monitoring the work desk, and a keyboard 805 and a mouse 806.
The camera 801 recognizes a user's operation and a position of a document or the like on a desk, and can support work according to the recognition.
For example, the user switches the display between the upright display 802 and the display with built-in desk 803 using input means (keyboard 805 or mouse 806 or pen) or extracts electronic information related to a paper document placed on the desk. be able to.

Further, "InteractiveDES"
Patent applications relating to “K” include, for example, JP-A-6-26.
No. 6498 (Title of Invention, "Information Processing Apparatus"). JP-A-6-266498 discloses that a display state of a display built in a work desk is determined based on at least one of a state of a user, a state on a display surface, and a display state. A method of determining is disclosed.

For example, when a document or the like is placed on a display, information is displayed while avoiding the document.
Real objects such as documents do not interfere with virtual worlds such as display information. Further, a method of switching the display by the above-mentioned input means is also disclosed.

[0027]

In Japanese Patent Application Laid-Open No. Hei 7-146951, which is a prior art relating to features, a window 300 in which digitized data desired by a user to be displayed is reduced.
And the display contents of the enlarged window 301 in which the cursor position is enlarged. However, in this method, it is necessary to open two windows in one display so as not to overlap, and the display area per window is reduced. Japanese Patent Application Laid-Open No. H2-252064 discloses a method of preparing two displays and linking the displayed images to each other in order to enlarge one window. In this case, however, the entire device is large and It will be expensive.

On the other hand, in Japanese Patent Application Laid-Open No. 1-265292, the area desired by the user can be enlarged and displayed by moving the enlargement window 401.
The size of the enlarged window 401 is limited, and a more detailed display cannot be performed.

In Japanese Patent Application Laid-Open Nos. Hei 7-146951 and Hei 1-265292, digitized data displayed in a window are displayed in conjunction with each other. That is, only the information of the virtual world is displayed in association with each other. Therefore, for example, it is impossible to display the real object in front of the eyes and the virtual world information displayed on the display in association with each other.

On the other hand, Japanese Patent Application Laid-Open No. Hei 9-46776 directly projects a laser beam onto a real document.
Furthermore, it seems that the cursor displayed on the display in the pointing device and the laser light projected from the remote device are linked. However, an image of a document at a remote location is displayed on a display in the pointing device, and a cursor is moved on the image to project a laser beam at a corresponding position on the document at a remote location. Thus, the display only acts as a console for controlling the remote device, so to speak, it is the same as controlling the remote device directly through the display. Therefore, there is a predetermined correspondence between the real document and the display content on the display, and when the user points at one, a series of movements in which the other is linked to it is not realized. Absent.

Also, in the above-mentioned Japanese Patent Application Laid-Open No. 5-83705, the user merely operates the pointing device to point on the document table. No display for displaying information of the virtual world is provided, and, of course, no means for connecting any virtual world to a manuscript table in the real world and such a concept are described at all.

This means that “Digital Desert”
k "and a related publication, JP-A-7-16
8949 and JP-A-9-237349,
The same applies to "Interactive DESK" and JP-A-6-266498, which is a related publication. In particular, JP-A-6-266498 does not describe anything about interlocking an upright display and a built-in desk type display.

Accordingly, the present invention has been made in view of the above-mentioned problems of the prior art, and has as its object to combine a real object such as a document manuscript or a three-dimensional object in the real world with a virtual object displayed on a display. Of digital data, which is world information,
It is an object of the present invention to provide an image display device which can realize intuitive and easy-to-understand information presentation for a user by operating one of them in response to a pointing operation.

[0034]

An image display device for achieving the above object comprises a light pointer capable of projecting light of an arbitrary shape at an arbitrary position on an object, and a display for displaying desired information. A pointing device capable of indicating a desired position, an object position and orientation sensor for measuring the position and orientation of the object, and controlling the optical pointer, the display, the pointing device, and the object position and orientation sensor. And a computer for storing digitized data that is associated with the object by a predetermined rule, and is displayed on the display with the projection light position by the light pointer. The digitized data is such that it always satisfies the association, and the pointing device Even if the position of the projection light by the pointer is moved, the digitized data displayed on the display is scrolled accordingly, and even if the digitized data displayed on the display is scrolled with the pointing device, The object of the present invention can be achieved by a configuration in which the position of the light projected by the optical pointer is moved accordingly.

[0035]

DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Invention) An embodiment of the present invention will be described. According to a first aspect of the present invention, there is provided a light pointer capable of projecting light of an arbitrary shape at an arbitrary position on an object, and a display for displaying digitized data associated with the object according to a predetermined rule. A pointing device that can indicate the position on the object to which the light pointer projects light, and an object position and orientation sensor for measuring the position and orientation of the object, and the object to which light is projected from the light pointer. The digitized data corresponding to the upper position is displayed on the display, and when the user operates the pointing device to move the projected light position, the digitized data corresponding to the projected light position is always displayed on the display in conjunction with the movement. It is like that.

(Second Invention) In the second invention, contrary to the first invention, the pointing device can point or scroll the digitized data displayed on the display, and the electronic device displayed on the display can be used. When the user operates the pointing device to point or scroll the digitized data, the light displayed by the light pointer is projected onto the position on the object corresponding to the area of the digitized data. The light from the optical pointer is always projected on a position on the object corresponding to the area of the coded data.

(Third Invention) In a third invention, a pointing device is used to indicate a projection light position using an optical pointer to link display display, and a display device is operated by a pointing device to control light. A display function switching means for switching the embodiment in which the position of the projection light by the pointer is linked is provided so that both functions are freely switched to display an image.

(Fourth Invention) According to a fourth invention, when the projection light position of the optical pointer is operated on the display with a pointing device, the position of the digitized data corresponding to the projection light position is On the other hand, when operating the image displayed on the display with the pointing device,
The cursor is displayed at the indicated position, and the solid angle viewed from the light pointer of the area on the object corresponding to the area displayed on the display and the maximum solid angle at which the light pointer can project light If the ratio is equal to or less than a predetermined value, the position of the light projected by the light pointer on the object and the area displayed in the center of the display are linked to correspond to each other. When the cursor is displayed outside the display area, the cursor is always displayed by scrolling the displayed image if the cursor is displayed outside the display area. Like that.

(Fifth invention) A fifth invention is provided with an imaging device for imaging an object, digitizing it, and displaying the digitized data on a display, generating digitized data, displaying a display, and operating with a pointing device. , So that a series of operations can be efficiently performed.

(Sixth invention) In the sixth invention, the position and orientation of the object are measured by the imaging device, and in this case, the object position and orientation sensor is not required.

(Seventh Invention) According to a seventh invention, an object setting means for setting an object at a position whose relative position and orientation with respect to an optical pointer is known in advance is provided, By installing by the installation means, there is no need to measure the position and orientation of the object, and the object position and orientation sensor is not required.

Hereinafter, the present invention will be described in detail with reference to examples.

[0043]

FIG. 1 is a diagram showing the configuration of a first embodiment of the present invention. In the first embodiment of the present invention, the object is a document, and the digitized data is image data obtained by reading the document with an imaging device such as an image scanner or a camera. The case where the correspondence is a simple geometric correspondence will be described. The digitized data is stored in the image data storage unit 10.
3 is stored.

Referring to FIG. 1, a first embodiment of the present invention is an optical pointer 1 capable of projecting light of a desired shape to a desired position or direction on an object (document original), and an object (document). A display 2 for displaying a read image of a document document);
A pointing device 3 for operating a user to point a desired position, an object position and orientation sensor 4 for measuring the position and orientation of an object (document document), and an optical pointer control for controlling the optical pointer 1 Means 200
And light pointer control means 200, display 2, pointing device 3, and object position and orientation sensor 4.
And a computer 100 connected thereto.

Further, the computer 100 converts the pointing information sent from the pointing device 3 into optical pointing information and outputs it to the optical pointer control means 200.
And a display image generating unit 102 for generating an image to be displayed on the display 2, and an image data storage unit 103 for storing a read image of an object (document original).

An object (document manuscript) is placed on a mounting table (not shown) next to the display 2, and the relative position and orientation with respect to the optical pointer 1 are measured by the object position and orientation sensor 4. .

The light pointer 1 can project light of a desired shape on a subject at a desired position or direction. The light intensity is such that the user can check the light projected on the subject.

As the optical pointer 1, for example, a liquid crystal projector using a transmissive liquid crystal device or a laser pointer used for presentations can be applied.

However, when a laser pointer is used, some device is required to project light having a desired shape, but it is not mentioned because it has no relation to the essence of the present invention.

A method of projecting light at a desired position with a laser pointer is disclosed in, for example, JP-A-6-14920.
The method disclosed in Japanese Unexamined Patent Publication No. 2 (title of the invention, "pointer display device") and the like can be applied. That is, two-dimensional scanning can be realized using two reflecting mirrors that can be driven to rotate in one axis direction.

The display 2 can display a virtual world created by the computer 100. The virtual world is an operating system using a graphical user interface, computerized image data, or a word processing screen. The display 2 may be a CRT or a liquid crystal display using a normal cathode ray tube, or may be a screen on which an image is projected by a projector. Further, the display magnification of the digitized data displayed on the display 2 can be freely changed by the user.

The pointing device 3 comprises position input means (not shown) for inputting position information, and operation determining means (not shown) for deciding to start or end a desired operation. Pointing information based on the user's operation is transmitted to the computer 1
Transfer to 00. Although the pointing device 3 shown in FIG. 1 is assumed to be a mouse, a keyboard, a trackball, a joystick, or the like may be used instead of the mouse.

The object position / orientation sensor 4 can take various forms depending on the object. For example, as shown in this embodiment, the object is a document and is placed as shown in FIG. In such a case, a one-dimensional contact image sensor can be applied. That is, as shown in FIG. 1, a bar code including information such as the size of the document document is printed at a predetermined position of the lower end portion of the document document with which the object position and orientation sensor 4 is in contact. By reading the barcode with the one-dimensional contact image sensor, the position of the barcode with respect to the one-dimensional contact image sensor, that is, the position of the document document and the size of the document document can be obtained. Therefore, if the relative positions of the object position and orientation sensor 4 and the optical pointer 1 are measured in advance, the position and orientation of the document with respect to the optical pointer 1 and the solid angle of the document as viewed from the optical pointer 1 can be calculated. .

Although it is considered difficult to realize a sensor capable of measuring the position and orientation of any object, in applying the present invention, the object is limited to some extent and certain conditions are established, so that the existing technology can be used. It is assumed that the object position and orientation sensor 4 can be implemented.

The case where an image pickup device such as a camera is used as the object position / posture sensor 4 is described as a fifth embodiment. In addition, the object position / posture sensor 4 is provided by installing the object at a predetermined position. A case in which it is unnecessary will be described later as a sixth embodiment.

[Description of Operation] Next, the operation of the first embodiment of the present invention will be described.

When the user operates the pointing device 3, the pointing device 3 outputs the pointing information to the computer 100,
Is sent to the pointing information conversion means 101 in the inside. The pointing information conversion unit 101 converts the pointing information into optical pointing information, and transfers it to the optical pointer control unit 200. The optical pointer control means 200 controls the optical pointer 1 based on the optical pointing information sent from the pointing information conversion means 101, and projects light on an object (document original). Therefore, when the user operates the pointing device 3, the projection light moves accordingly.

On the other hand, pointing information conversion means 101
Sends the optical pointing information to the display image generating means 102 as well. The display image generating means 102 first processes the optical pointing information and the position and orientation information sent from the object position and orientation sensor 4 so that the projection light from the optical pointer 1 Determine which part is being projected. Next, the head address and data length of the image data to be displayed on the display 2 are calculated so that the portion of the image data corresponding to the projection light position is displayed in the center of the display 2, and the image data storage means 103
Send to The data length is determined from the magnification that displays the image data. If the magnification is large, the data length is shortened,
When the enlargement ratio is small, the data length becomes long. The enlargement ratio of the display image is set separately by the user, and the setting method is not related to the present invention, and therefore the description is omitted.

The image data storage means 103 stores necessary image data based on the image data address and data length sent from the display image generation means 102.
Send to 02. The display image generation means 102 transfers the received image data to the display 2 and
To be displayed. For example, a magnetic disk or the like may be applied to the image data storage unit 103.

Although the projected light is a spot light in FIG. 1, for example, a linear or circular light may be projected. However, in this case, as described above, the optical pointer 1
It is necessary to make a device for projecting the light of the shape.

The area on the object corresponding to the image data displayed on the display 2 is indicated by a dotted line as the display area.

FIG. 2 is a schematic diagram for explaining the movement of the projection light projected on the object by the optical pointer 1 and the screen of the display 2 in the first embodiment of the present invention. With reference to FIG. 2, the projection light and the movement of the screen according to the first embodiment of the present invention will be described.

The document manuscript as an object includes “$ 123-
4567 "and" Name: Taro Patent ". A user (not shown) operates the pointing device 3 to change the projection light from the optical pointer 1 to “$ 12”.
3-4567 "to the" 3 "portion. At this time, the image data of “$ 123-4567” is displayed on the display 2 at a certain magnification so that the portion of “3” is located at the center of the image. Next, pointing device 3
Is operated to move the projection light to the “special” part of “name: patent taro”, and accordingly, the “special” part of “name: patent taro” is displayed on the display 2 in the center of the image. Is displayed.

As described above, when the user operates the pointing device 3 to move the projection light to a desired position on the object (document original), image data corresponding to the position is displayed on the display 2. Is displayed. Therefore, the user can freely view a desired portion of the image data on the display 2 without scrolling the screen by moving the projection light while watching the target object (document document).

(Second Embodiment) FIG. 3 is a diagram showing a configuration of a second embodiment of the present invention. In the second embodiment of the present invention, similarly to the first embodiment, the object is a document document, and the digitized data is image data obtained by reading the document document with an image pickup device such as an image scanner or a camera. A case will be described.

Referring to FIG. 3, similarly to the first embodiment of the present invention, the configuration of the optical pointer 1
A display 2, a pointing device 3, an object position and orientation sensor 4, an optical pointer control unit 200,
And a computer 100.

Further, the computer 100 includes a pointing information conversion unit 101 and a display image generation unit 102
And an image data storage unit 103.

An object (document manuscript) is placed on a mounting table (not shown) next to the display 2, and the relative position and orientation with respect to the optical pointer 1 are measured by the object position and orientation sensor 4. .

In the configuration described above, the second embodiment and the first embodiment of the present invention are the same, but the flow of data and control signals is different as can be seen from FIG. That is, the operation in the second embodiment of the present invention is as follows.
The operation is different from that of the first embodiment of the present invention. Hereinafter, the operation of the second embodiment of the present invention will be described.

[Explanation of Operation] When the user operates the pointing device 3, the pointing device 3 outputs the pointing information and sends it to the display image generating means 102 in the computer 100. Display image generating means 1
In step 02, the head address and data length of the image data to be displayed on the display 2 are calculated based on the transmitted pointing information, and sent to the image data storage unit 103. The image data storage unit 103, based on the image data address and data length sent from the display image generation unit 102,
The necessary image data is sent to the display image generating means 102. The display image generation unit 102 transfers the transmitted image data to the display 2 and displays the image data on the display 2.

On the other hand, the display image generation means 102 sends display image information, which is information indicating which part of the image data is displayed on the display 2, to the pointing information conversion means 101. Pointing information conversion means 10
1 processes the transmitted display image information and the position and orientation information sent from the object position and orientation sensor 4 to convert the light pointer 1 into light pointing information which is information of a position where light is to be projected. It is converted and sent to the optical pointer control means 200. The optical pointer control means 200 controls the optical pointer 1 based on the optical pointing information, and projects light on an object (document original). Therefore, when the user operates the pointing device 3 to scroll the image displayed on the display 2, the projection light moves accordingly.

FIG. 4 is a schematic diagram for explaining the movement of the screen of the display 2 and the projection light projected on the object by the optical pointer 1 in the second embodiment of the present invention. Referring to FIG. 4, the movement of the screen and the projection light according to the second embodiment of the present invention will be described.

As in the case of FIG. 2, the document document as the object has portions where "$ 123-4567" and "Name: Taro Tokkyo" are described. On the display 2, a cursor indicating the position pointed to by the pointing device 3 is displayed. The user (not shown) operates the pointing device 3 to display “$ 123-4” on the display 2.
567 "is displayed. At this time, the projection light from the light pointer 1 is moved to the position “3” on the object corresponding to the center of the display image on the display 2.
Next, when the pointing device 3 is operated to scroll the screen displayed on the display 2 to display the image data of the part of “Name: Taro Taro”, the projection light is accordingly displayed on the display 2. The object is moved to a “special” position on the object corresponding to the center of the image.

As described above, when the user operates the pointing device 3 to scroll the display image displayed on the display 2, the light pointer moves to the position on the object (document original) corresponding to the center of the display image. 1 projects light. Therefore, the user can confirm which part of the image displayed on the display 2 corresponds to which part of the entire object (document document) without changing the display enlargement ratio of the image data.

(Embodiment 3) A third embodiment of the present invention will be described.

The third embodiment of the present invention has the functions of both the first embodiment and the second embodiment of the present invention described above, and furthermore, determines which of these functions the user uses. It has a function switching button that can be selected.

The configuration of the first embodiment and the configuration of the second embodiment are the same in terms of hardware. Therefore, the first program for bringing the image display device of the present invention into a state in which the functions described in the first embodiment are realized (hereinafter abbreviated as “first state”), and the second program will be described. A second program for realizing the function (hereinafter, abbreviated as “second state”) is installed in the computer 100, and by activating one of the programs, the first state or the second state is obtained. Can be realized.

The first program and the second program need only realize the functions described in the first and second embodiments.
Detailed description is omitted.

FIG. 5 shows the states of the light projected from the optical pointer 1, the image displayed on the display 2, and the cursor displayed on the display 2 in the first state and the second state.

As shown in FIG. 6, the function switching button can transition between the first state and the second state like a toggle switch.

As for the third embodiment of the present invention, the structure of the third embodiment is not shown. Therefore, the function switching button is not shown, but if incorporated in the pointing device 3, for example, Pointing operation and switching operation can be performed efficiently.

(Embodiment 4) In the first and second embodiments of the present invention, it is arranged that the projection light from the optical pointer 1 is linked to the center position of the display image.

However, when the display enlargement ratio on the display 2 is small, that is, when the area on the object corresponding to the displayed image data is relatively large, the projection light from the optical pointer 1 is linked. In some cases, it is possible to realize an image display that is easier to understand when the cursor is displayed on the display instead of the center of the display image.

In the first to third embodiments, it is assumed that the user can freely change the enlargement ratio of the image data displayed on the display 2.

Therefore, in the fourth embodiment of the present invention, the interlocking system can be automatically switched according to the display magnification when displaying image data on the display 2. That is, when the display enlargement ratio is large and the area on the object corresponding to the image data is small, the projection light from the optical pointer 1 and the display on the display 2 are displayed as described in the first and second embodiments. Link the center of the image.

On the other hand, when the display enlargement ratio is small and the area on the object corresponding to the image data is wide, the light projected from the optical pointer 1 and the cursor are linked.

FIG. 7 is a schematic diagram for explaining the movement of the screen of the display 2 and the projection light projected on the object by the optical pointer 1 when the cursor is linked with the projection light from the optical pointer 1. FIG. FIG. 7 shows a case where the display 2 is in the first state described in the first embodiment.
The movement of the projection light from the light pointer 1 will be described.

The document manuscript as an object includes “$ 123-
4567 "and" Name: Taro Patent ". The display 2 displays image data corresponding to the display display area indicated by the dotted line on the target object. Since the display enlargement ratio is small, “# 123-4567”
And "Name: Patent Taro" are displayed.

A user (not shown) operates the pointing device 3 to move the projection light from the optical pointer 1 to the “3” portion of “$ 123-4567”. At this time, the cursor displayed on the display 2 indicates “$ 1
23-4567 ". Next, when the pointing device 3 is operated to move the projection light to the “special” portion of “name: patent taro”, the cursor moves accordingly to the “special” portion of “name: patent taro”. .

The linkage between the projection light and the cursor described above is as follows.
Although the first state is taken as an example, it can be realized in the second state described in the second embodiment of the present invention.

The configuration of the fourth embodiment is the same as the hardware configuration of the first embodiment or the second embodiment, and the interlocking mode switching program for realizing this function is incorporated in the computer 100. By doing so, it can be realized in both the first state according to the first embodiment and the second state according to the second embodiment. Obviously, it can be realized also in the third embodiment.

The interlocking mode switching program includes first to third programs.
It is sufficient that the interlocking method described in the embodiment and the interlocking method described with reference to FIG. 7 can be switched according to the display enlargement ratio of the display 2, and the detailed description thereof will be omitted. .

(Embodiment 5) FIG. 8 is a diagram showing a configuration of a fifth embodiment of the present invention. Referring to FIG. 8, a fifth embodiment of the present invention relates to a light pointer 1 capable of projecting light of a desired shape onto a target (document original) at a desired position or direction, and a target (document original). , A pointing device 3 for a user to operate to point a desired position, an image of an object (document document), and measurement of the position and orientation of the object (document document) Imaging device 10 for controlling an optical pointer 1 and an optical pointer control means 2 (not shown) for controlling the optical pointer 1
00, light pointer control means 200 and display 2
And a computer 100 (not shown) connected to the pointing device 3 and the imaging device 10.

It is considered that the configuration of the fifth embodiment of the present invention is the same as the configuration of the first or second embodiment except that the object position / orientation sensor 4 is replaced with an imaging device 10. Therefore,
The flow of data and signals is basically the same as in the first or second embodiment, and the interlocking function between the optical pointer 1 and the display 2 implements the functions described in the first to fourth embodiments. can do.

However, the object position and orientation sensor 4 in the first to fourth embodiments only measures the position and orientation of the object, whereas the imaging device 10 in the fifth embodiment is At the same time as measuring the position and orientation of the object, the object can be imaged and digitized data can be generated. Therefore, in the fifth embodiment, a series of operations of capturing an object, generating digitized data, displaying the data on the display 2, and linking the optical pointer 1 and the display 2 to the pointing operation can be continuously performed.

(Embodiment 6) FIG. 9 is a diagram showing a configuration of a sixth embodiment of the present invention. Referring to FIG. 9, a sixth embodiment of the present invention provides a light pointer 1 capable of projecting light of a desired shape to a desired position or direction on an object (document manuscript) and an object (document manuscript). , A pointing device 3 for operating a user to point a desired position, and an object setting means 11 for setting an object at a predetermined position.
A light pointer control means 200 (not shown) for controlling the light pointer 1;
And a computer 100 (not shown) connected to the display 2, the pointing device 3, and the imaging device 10.

In the sixth embodiment, the object position / orientation sensor 11 is provided instead of the object position / posture sensor 4 and the imaging device 10. Object position setting means 1
If the relative position and orientation of the optical pointer 1 and the optical pointer 1 are known in advance, the object (document original) is set by the object position setting means 11 so that the relative position between the object (document original) and the optical pointer 1 is increased. There is no need to measure the position and orientation, and neither the target portion position and orientation sensor 4 nor the imaging device 10 is required.

In the first to sixth embodiments of the present invention described above, as described first in the description of the first embodiment, the object is a document and the digitized data is its image data. Was explained as an example. However, the combination of the object and the digitized data in the present invention is not limited to this. For example, as the object, digitized data, a paper map, names such as place names and landmarks described on the map, etc. Characteristic information, a three-dimensional object, a model, a three-dimensional object, a model design drawing, a paper design drawing, a CG image of an object described in the design drawing, and the like can be considered.

Further, the present invention controls an optical pointer or a display with one pointing device,
Point the real and virtual worlds. Accordingly, Japanese Patent Application No. 10-2284 filed by the present applicant earlier has
Use of the invention described in No. 64 "Pointing device" enables more efficient pointing operation,
An easy-to-use image display device can be provided.

Further, according to the present invention, a program for realizing the functions of the pointing information conversion means 101, the display image generation means 102, and the image data storage means 103 is added to the computer 100 described in the present embodiment and examples. The present invention may be applied to an optical disk medium represented by a CD-ROM or a DVD-ROM, a magnetic recording medium represented by a floppy disk or a hard disk, or a recording medium such as a semiconductor storage device.

[0101]

According to the present invention, the following effects can be obtained.

The first effect of the present invention is that, by linking an optical pointer capable of directly pointing an object and a display image on a display, a real object in the real world can be obtained.
It can be displayed in association with digitized data that is virtual world information. Further, in the linked display, the display can be linked by pointing to the target object, or the optical pointer can be linked by pointing to digitized data as virtual world information on the display.

Thus, if the actual object is a document document and the digitized data is image data of the document document, a place to be viewed on the document document is indicated by an optical pointer, and the vicinity of the optical pointer is indicated. Is displayed on the display. If the real object is a paper map and the digitized data is a place name, building, landmark or other name described on the map, the user indicates a specific place on the map, You can know the name of the place or building. Further, if the real object is a three-dimensional object and the digitized data is the design drawing, it is possible to easily refer to which part of the design drawing corresponds to a specific part of the three-dimensional object.

The second effect of the present invention is that, by providing the function switching button described in the third embodiment, whether the object directly operated by the pointing device is an object in the real world or displayed on the display You can switch between virtual worlds. Therefore, the user can select an interlocking method that is easier to understand in the image display.

A third effect of the present invention is that the movement of the optical pointer is linked to the size of the area on the object corresponding to the digitized data displayed on the display.
It is possible to automatically switch between scrolling the display and using the cursor displayed on the display. This can prevent the image from being scrolled more than necessary, so that it is possible to provide an image display that is easy for the user to see.

A fourth effect of the present invention is that an imaging device for digitizing an object to generate digitized data is provided, and a series of operations from generation of digitized data to image display and pointing operation are efficiently realized. it can. Furthermore, the imaging device also has a role of measuring the position and orientation of the object, and eliminates the need for an object position and orientation sensor, thereby simplifying the entire device.

The fifth effect of the present invention is that the object is provided with an object setting means for setting the object, and the object is set at a predetermined place, thereby eliminating the need to measure the position and orientation of the object. The object position and orientation sensor is not required.

In the sixth embodiment, the case where the imaging device described in the fifth embodiment is not provided has been described. However, in the sixth embodiment, the imaging device described in the fifth embodiment is different from the imaging device described in the fifth embodiment. In the case of a configuration having both of the described object setting means, if the object setting means is adjusted so that the object is set in an area that can be imaged by the imaging device, the position of the object for imaging is adjusted. There is also an effect that alignment is easy.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is a diagram showing a state of pointing according to the first embodiment of the present invention.

FIG. 3 is a diagram showing a configuration of a second exemplary embodiment of the present invention.

FIG. 4 is a diagram showing a state of pointing according to a second embodiment of the present invention.

FIG. 5 is a diagram showing a state of image display and pointing according to a third embodiment of the present invention.

FIG. 6 is a diagram showing a state transition in a third embodiment of the present invention.

FIG. 7 is a diagram showing a configuration of a fourth embodiment of the present invention.

FIG. 8 is a diagram showing a configuration of a fifth embodiment of the present invention.

FIG. 9 is a diagram showing a configuration of a sixth embodiment of the present invention.

FIG. 10 is a schematic diagram showing movement of a display screen of a document creation device described in Japanese Patent Application Laid-Open No. 7-146951.

FIG. 11 is a schematic diagram showing movement of a display screen of an image display device described in Japanese Patent Application Laid-Open No. 1-265292.

FIG. 12 is a diagram showing a part of the configuration of a system described in JP-A-9-46776.

FIG. 13 is a diagram showing a part of the configuration of a system described in Japanese Patent Application Laid-Open No. 5-83705.

FIG. 14 is a diagram showing a configuration of a digital desk.

FIG. 15 is a diagram schematically showing a configuration of an Interactive DESK.

[Description of Signs] 1 Optical pointer 2 Display 3 Pointing device 4 Object position and orientation sensor 10 Imaging device 11 Object installation means 100 Computer 101 Pointing information conversion means 102 Display image generation means 103 Image data storage means 200 Optical pointer control means 300 Window 301 Enlarged window 302 Display 303 Pointing device 401 Enlarged window 402 Display 403 Pointing device 500 Pointing device 501 Display 502 Keyboard 503 Mouse 504 Network 505 Remote device 506 Document 601 CPU 602 Camera 603 Laser beam emitting device 604 Document table 605 Pointing device 701 Camera 702 Projector 703 Compiler Over data 704 workbench 801 camera 802 displays 803 the desk embedded display 804 workbench 805 keyboard 806 mouse

Claims (18)

[Claims] 1. An object existing in the real world is pointed by projection light projected by an optical pointer operable by a pointing device, an image near the pointed portion is displayed on a display, and the pointing device is operated. And moving the pointing location to display an image near the pointing location on the display. 2. An optical pointer for projecting projection light of an arbitrary shape at an arbitrary position on an object, a display for displaying desired information, image data storage means for storing image data of the object, A pointing device capable of indicating the position of the projection light projected by the optical pointer on the object, and an object position and orientation sensor for measuring the position and orientation of the object, wherein the object position and orientation sensor measures Using the position and orientation of the object and the position information on the object of the projection light instructed via the pointing device, the light pointer corresponds to the position of the projection light projected by the light pointer on the object. The digitized data is displayed on the display with reference to the image data stored in the image data storage means, and the user operates the pointing device to throw the digitized data. An image display device wherein, when a light emitting position is moved, digitized data corresponding to the projected light position is displayed on the display. 3. An object existing in the real world is pointed by projection light projected by an optical pointer, an image near the pointed portion is displayed on a display, and the displayed image can be pointed or scrolled. An image display device characterized in that, when the pointing or scrolling is performed via a simple pointing device, the pointing position of the optical pointer moves in conjunction with this operation. 4. An optical pointer for projecting projection light of an arbitrary shape at an arbitrary position on an object, a display for displaying desired information, image data storage means for storing image data of the object, A pointing device capable of pointing or scrolling an image displayed on a display, and an object position and orientation sensor for measuring the position and orientation of the object, comprising: a projection light projected by the optical pointer on the object. The digitized data corresponding to the position is displayed on the display with reference to the image data stored in the image data storage means, and when pointing or scrolling via the pointing device, the operation is linked to this operation. An image display device wherein a pointing position of an optical pointer moves. 5. An object existing in the real world is pointed by projection light projected by an optical pointer, an image near the pointed portion is displayed on a display, and the displayed image can be pointed or scrolled. When the pointing or scrolling is performed via a pointing device, a first function in which the pointing position of the optical pointer moves in conjunction with this operation, and the position of the projection light projected by the optical pointer via the pointing device can be operated. When the pointing device is operated to move the position of the projection light projected by the light pointer, a second function of displaying an image near the projection light on the display is provided. And the second function can be appropriately switched. Image display device for. 6. An optical pointer for projecting projection light of an arbitrary shape at an arbitrary position on an object, a display for displaying desired information, image data storage means for storing image data of the object, A pointing device that can indicate the position of the projection light projected on the object by the optical pointer, and an object position and orientation sensor for measuring the position and orientation of the object, and the projection light projected by the optical pointer. When the digitized data corresponding to the position on the object is displayed on the display with reference to the image data stored in the image data storage means, and the pointing or scrolling is performed via the pointing device, this operation is performed. A first function in which the pointing position of the optical pointer moves in conjunction with Using the position and orientation of the object and the position information on the object of the projection light instructed via the pointing device, the light pointer corresponds to the position of the projection light projected by the light pointer on the object. The digitized data is displayed on the display with reference to the image data stored in the image data storage means, and when the user operates the pointing device to move the projection light position, the position corresponding to the projection light position is obtained. An image display device, comprising: a second function of displaying digitized data on the display; wherein the first function and the second function can be appropriately switched. 7. The image display device according to claim 1, wherein an enlargement ratio of an image displayed on said display can be changed by a user operation. 8. The display displays a cursor indicating a position in the digitized data corresponding to a projection light position on the object projected from the optical pointer or a pointing position of the pointing device, wherein the cursor is displayed on the display. The solid angle viewed from the light pointer of the area on the object corresponding to the area displayed on the display, and the ratio of the maximum solid angle at which the light pointer can project light,
If it is equal to or less than a predetermined value, the position of the light projected by the light pointer on the object and the area displayed in the center of the display are linked so as to correspond to each other. If the value is equal to or greater than a certain value, the position of the light projected by the light pointer on the object and the cursor displayed on the display are linked to correspond to each other. 6. The image display device according to claim 1, wherein an image is scrolled so that a cursor is always displayed. 9. A cursor indicating a position in the digitized data or a pointing position of the pointing device corresponding to a projection light position on the object projected from the optical pointer, is displayed on the display; The solid angle viewed from the light pointer of the area on the object corresponding to the area displayed on the display, and the ratio of the maximum solid angle at which the light pointer can project light,
If it is equal to or less than a predetermined value, the position of the projection light by the light pointer on the object and the area displayed in the center of the display are linked so that the solid angle ratio is If the value is equal to or greater than a certain value, the position of the light projected by the light pointer on the object and the cursor displayed on the display are linked to correspond to each other. 7. The image display device according to claim 2, wherein the cursor is always displayed by scrolling the image.
Or the image display device according to 7. 10. An imaging device for imaging the object, wherein the digitized data is image data obtained by imaging the object with the imaging device. The image display device according to claim 1. 11. An object position and orientation sensor, an imaging device for imaging the object, and an object position for processing image data acquired by the imaging device to measure the position and orientation of the object. The image display device according to claim 2, wherein the image display device includes at least: a posture measurement unit. 12. An object setting means for setting the object at a position whose relative position and orientation with respect to the optical pointer are known in advance, wherein the object is set by the object setting means. 10. The image display device according to claim 2, wherein it is not necessary to measure the position and orientation of the object, and the object position and orientation sensor is not required. 13. An optical pointer for projecting projection light of an arbitrary shape at an arbitrary position on an object, a display for displaying desired information, and a position of the projection light projected by the optical pointer on the object. A pointing device capable of instructing, an object position and orientation sensor for measuring the position and orientation of the object, and image data storage means having a function of storing image data of the object in storage means provided in a computer, at least An image display device comprising:
A recording medium storing a program for controlling the optical pointer, the display, the pointing device, and a computer that controls the object position and orientation sensor, wherein the computer measures the object by the object position and orientation sensor. Using the position and orientation of the object and the position information on the object of the projection light instructed via the pointing device, digitized data corresponding to the position of the projection light projected by the optical pointer on the object. Is displayed on the display with reference to the image data stored in the image data storage means, and when the user operates the pointing device to move the projection light position, digitized data corresponding to the projection light position That stores a program for realizing the function of displaying Over data-readable recording medium. 14. An optical pointer for projecting projection light of an arbitrary shape at an arbitrary position on an object, a display for displaying desired information, and a pointing device capable of pointing or scrolling an image displayed on the display. An image display device comprising at least an object position and orientation sensor for measuring the position and orientation of the object, and image data storage means having a function of storing image data of the object in storage means provided in a computer. A storage medium storing a program for controlling the optical pointer, the display, the pointing device, and a computer that controls the object position and orientation sensor, wherein: a projection light that the optical pointer projects on the computer. Digitized data corresponding to the position on the object A function of moving the pointing position of the optical pointer in conjunction with this operation when pointing and scrolling through the pointing device while displaying on the display with reference to the image data stored in the image data storage means. A computer-readable recording medium on which a program for realizing the program is recorded. 15. An optical pointer for projecting projection light of an arbitrary shape at an arbitrary position on an object, a display for displaying desired information, and a position on the object of the projection light projected by the optical pointer. A pointing device capable of instructing, an object position and orientation sensor for measuring the position and orientation of the object, and an image data storage unit having a function of storing image data of the object in a storage unit provided in a computer. An image display device comprising: the optical pointer, the display, the pointing device,
In a recording medium that records a program for controlling by a computer that controls the object position and orientation sensor, the computer includes: The image data stored in the image data storage means is displayed on the display with reference to the display, and when pointing or scrolling is performed via the pointing device, the pointing position of the optical pointer moves in conjunction with this operation. 1, the position and orientation of the object measured by the object position and orientation sensor, and the position information on the object of the projection light instructed via the pointing device, the light pointer, Digitizing data corresponding to the position of the projection light to be projected on the object; The image data stored in the storage means is displayed on the display, and when the user operates the pointing device to move the projection light position, the computerized data corresponding to the projection light position is displayed on the display. A computer-readable recording medium storing a program for realizing a second function to be performed, and a third function capable of switching between the first function and the second function. 16. A program according to claim 13, wherein a program for causing the computer to further realize a function of enabling a magnification of an image displayed on the display to be changed by a user operation is recorded. Computer-readable recording medium. 17. A cursor indicating a position in the digitized data corresponding to a projection light position on the object projected from the optical pointer or a pointing position of the pointing device is displayed on the display, and the display device displays a cursor on the display. The ratio of the solid angle of the area corresponding to the displayed area on the object viewed from the light pointer to the maximum solid angle at which the light pointer can project light is equal to or less than a predetermined constant value. In some cases, the projection light position by the light pointer on the object and the area displayed in the center of the display are linked so as to correspond to each other, and when the solid angle ratio is equal to or more than the certain value, The projection light position by the light pointer on the object and the cursor displayed on the display are linked so as to correspond to each other, and the cursor position is displayed. The computer according to any one of claims 13 to 16, wherein a program for causing the computer to further realize a function of scrolling a display image and always displaying a cursor when the display image is out of the ray display area is recorded. A readable recording medium. 18. An image pickup device for picking up an image of the object, wherein the object position and orientation sensor processes the image data acquired by the image pickup device to cause the computer to process the position and orientation of the object. The object position and orientation measurement function for measuring
14. A program for further realizing the program is recorded.
18. The computer-readable recording medium according to any one of claims 17 to 17.