JP4491598B2 - Image display system - Google Patents

Description

The present invention measures the relative position and orientation of a reference object with respect to a display device using a three-dimensional information input device, calculates a three-dimensional stereoscopic image based on a coordinate system fixed with respect to the reference object, and It relates to an image table 示Shi stem to display.

  Conventionally, a commonly used device for displaying a three-dimensional image has mainly been a method of displaying an image for the left and right eyes on a fixed screen of a theater and viewing it with glasses that decompose by polarization or color. It was. This method requires a large screen to actually display the screen and is not suitable for carrying. Also, the audience cannot see the object from any direction.

  Some computer displays have the same function, but when changing the direction of an object, it was necessary to use a pad with a dial, a keyboard, or a data glove that measures the movement of the hand. Pads and keyboards require special operation and cannot be operated to handle real objects. A data glove is an input device that is often used in virtual reality. It is in the shape of a glove and measures the position and posture of a hand and the bending angle of a finger in real time. The data glove enables a fairly realistic operation, but it is necessary to be familiar with the operation because the physical force cannot be reproduced by measuring only the position. As a matter of course, the target object is limited to a small object on a computer display, and it is not possible to view a large three-dimensional image.

  In virtual reality, a position measurement device using a magnetic field or the like is used to measure the position of a display device worn on the user's head or a data glove for operation, and the result is projected into a virtual three-dimensional space. The main method was (FIG. 11). In this figure, the user wears a display device (HMD = Head Mount Display) on his face and puts a data glove on his hand. The position and orientation of both the HMD and the data glove room are measured and used to display a composite image. This also requires measurement of the position of the face and hand in the entire operation space, so a large-scale device for the entire room is required, and there is no small device. In addition, this method also measures only the position of the hand when handling, so that it is necessary to be familiar with the user, unlike an actual operation feeling.

  In order to solve such a problem, the present inventor previously made a cubic system that does not require a huge three-dimensional measuring apparatus as conventionally required and does not require measurement of the hand itself, and does not depend on a viewpoint. In addition to displaying original stereoscopic images on a display device, there has been proposed an image display method and system that is realistic in operation and does not require user familiarity (see Patent Document 1). However, this proposed technique calculates two-dimensional images for the left and right eyes in order to show the calculated three-dimensional image to the user.

This proposed technique measures a coordinate transformation matrix that represents the relative positional relationship between the coordinate system of a binocular display device and the coordinate system of a three-dimensional input device for measurement, and provides a three-dimensional shift in the user's field of view. A complicated initial calibration is required to prevent the occurrence of
JP 2003-141574 A

An object of the present invention is to solve such problems and to make the display image for one eye easier than the method of displaying a stereoscopic image in both eyes.
In addition, the present invention eliminates the need for a huge three-dimensional measuring apparatus as conventionally required, and eliminates the need for measurement of the hand itself, and displays a three-dimensional stereoscopic image independent of the viewpoint on the display device, An object of the present invention is to provide an image display method and system that are realistic in operation and do not require user familiarity.

  FIG. 3 is a diagram illustrating the basic configuration of the system of the present invention. The system of the present invention includes a reference object, a three-dimensional information input device, a coordinate conversion device, a display image composition device, and a display device. Further, a display data recording device and a communication device can be included as necessary. The reference object is an object that gives a coordinate system that serves as a reference when displaying a stereoscopic image to be displayed. Anything can be used as long as the coordinates can be calculated uniquely when measured with a three-dimensional information input device. Specifically, there may be a rectangular frame that can be held by hand. When the displayed three-dimensional image is large, the reference object may be a plurality of patterns that can be distinguished from each other.

  The display device is a device that displays a three-dimensional image synthesized in one eye of a user, and is a small display. This display displays a composite video and shows the calculated 3D video to the user. At the same time, a see-through function for displaying an image that is actually visible to the user as it is is required in a portion where the composite image is not displayed. The real part may be a display element that is transparent and transparent, or may display a front image taken by a camera.

  The three-dimensional information input device is a device that observes a reference object placed in the field of view and measures its position and posture in real time. Currently, there is a stereo vision system using two cameras that can be used for such a purpose. This three-dimensional information input device needs to be fixed to a display device (stereo vision is a method of calculating the three-dimensional coordinates of the contour of a symmetric object from the displacement of two or more television camera images) .

  The coordinate conversion device includes coordinates fixed to the reference object and coordinates fixed to the three-dimensional information input device based on the measured position and orientation of the reference object (which are relative to the three-dimensional information input device and the display device). Is converted in real time, and coordinate conversion of the 3D image to be displayed is performed based on the calculation matrix. The display image synthesizing apparatus is a calculation apparatus that calculates a two-dimensional image that can be seen from the position of the eye wearing the coordinate-converted three-dimensional image display apparatus.

According to the present invention, the display image is for one eye, which is simpler than the method for displaying a stereoscopic image in both eyes.
In the present invention, the position and orientation of the reference object are measured by a visual three-dimensional measuring device attached to the face, and the relative positional relationship between the display system and the object system is measured in real time with reference to the coordinates of the face. By measuring, the huge three-dimensional measuring device required in the above-described conventional technology can be eliminated. In addition, the operation of holding the displayed three-dimensional object by hand and looking at the back side also measures the reference object actually moved by hand in the present invention. However, there is a sense of reality and the user's familiarity is not necessary. In addition, since coordinate conversion is possible as long as one of the reference objects is in the field of view of the three-dimensional measuring apparatus, it is possible to walk in a three-dimensional image of the size of a room or a building.

  Hereinafter, a method and system for performing a three-dimensional display for realizing the mixed reality of the present invention will be described based on examples. FIG. 1 shows an example embodying the system of the present invention illustrated in FIG. In this example, the apparatus is composed of a glasses-like stereo image display apparatus 1 and two cameras 2. Moreover, the target object 4 which gives a reference | standard coordinate as a required accessory is required. In this figure, a rectangular plate sized like a PDA is used.

  The display device 1 can superimpose and display the actual scenery actually seen by the wearer and the composite image. It is possible to display a three-dimensional composite image that can be seen from the position of the eye wearing the display device. Stereo stereoscopic vision is stereoscopic vision that humans routinely perform with both eyes. The most common way to display stereoscopic images with glasses is to display the images of both eyes on a large screen or to shift them in time, and then separate them by using color filters, liquid crystal shutters, polarizing filters, etc. It is a method to do. The apparatus used here synthesizes an image directly on the surface of the glasses without using such a huge apparatus.

  Further, unlike a normal stereoscopic video display device, a composite image is displayed only on one eye to display a stereoscopic image. In normal stereoscopic vision, humans recognize the shape of a 3D object based on the images of the left and right eyes of the object, but it is not necessary for all objects to be visible with both eyes, and some objects are hidden from one eye. However, it can be recognized as a three-dimensional object. Therefore, when using an optical system that shows the real object visible to both eyes and the composite image displayed on the one-eye display to the user, it is displayed only on one eye as long as the positional relationship between the real object and the virtual object is fixed. Even if it is a virtual object, a sense of distance can be obtained by supplementing an image in which the brain cannot be seen. In the present invention, in order to fix the positional relationship between the real object and the virtual object, the coordinates of the reference object are measured using a three-dimensional information input device.

  The camera 2 is a small video camera. These can be used to input moving images electronically, and most of them use CCD image elements at present. The shooting interval is usually 1/30 second / frame for NTSC. However, for the purpose of tracking an object, it is preferable that the amount of movement for each frame is as small as possible because recognition errors are reduced. Recently, a commercially available camera capable of high-speed photography has begun to appear. “Object tracking” means inputting the position and orientation of an object moving in a three-dimensional object space in real time, and stereo image processing can be performed every time, but the amount of movement for each screen is usually small. Rather than searching for an object (in this case, a reference object) from the whole, it is possible to perform high-speed calculation by reducing the amount of calculation by searching for the vicinity of the previous position in the screen.

  The present invention performs stereo image processing to restore the shape of a three-dimensional object from images taken from two or more cameras. Basically, based on the principle of triangulation, the three-dimensional coordinates of the same feature point reflected on the left and right cameras are calculated from the camera arrangement. However, in actuality, (1) when a general object is targeted, feature points are not clearly displayed on the image. (2) If there is a figure with a shape similar to the feature point, a miscorrespondence occurs, and the three-dimensional shape is accurately determined. There are difficult points in implementation, such as being unable to calculate. When the target object has a known shape, you can use not only point correspondences but also global information such as boundaries and their combinations, so that the corresponding point search can be performed more smoothly and accurately. become. In the case of the present invention, it is possible to further reduce the probability of mishandling by making the reference object have a shape suitable for stereo image processing or by adding a mechanical mechanism (such as attaching an LED) to emphasize features. is there.

  The calculation device is a computer that performs image processing, coordinate conversion, display, and the like. Specifically, it is a computer. If specializing in coordinate transformation, it is possible to reduce the size and speed by using dedicated hardware. The reference object 4 is an object that serves as an index for giving reference coordinates. The form may be any object that can accurately calculate the position and orientation in three-dimensional measurement. The size may be small, but if it is too small, the error in measurement affects the posture accuracy and the reference coordinates may become unstable. On the other hand, if it is too large, it will not enter the field of view of the camera that performs the three-dimensional measurement, and there is also a problem in this case. Also, since three-dimensional measurement is performed by image processing, colors and shapes that are easy to distinguish from the surroundings are desirable.

Next, the operation of the system example illustrated in FIG. 1 will be described.
(1) As shown in FIG. 12, the image of the reference object 4 is input using the two cameras 2, three-dimensionally measured by the calculation device, and the relative position of the reference object 4 as viewed from the camera coordinate system C 2. And measure posture. The shape, color, and the like of the object 4 are stored as data in advance so that no mistake is made even if there is another object in the field of view.

(2) As shown in FIG. 14, a transformation matrix R42 from the coordinate system C4 fixed on the object 4 to the camera coordinate system C2 is calculated from the position and orientation information of the object 4.
(3) Further, as shown in FIG. 13, a conversion matrix R21 from the camera coordinate system C2 to the coordinate system C1 of the display device is also calculated in advance. Since the camera and the display device are fixed, this is not changed once calibration is performed.
(4) Three-dimensional image data to be synthesized is defined and prepared on the coordinate system C4. This is converted into the coordinate system C2 by the conversion matrix R42 calculated in (2), and further converted into the coordinate system C1 by the conversion matrix R21 in (3).

(5) An image that can be seen by the user's eyes is created from the three-dimensional data thus converted, and is displayed on each display device of the display device 1.
Through the above processing, the user can see a three-dimensional stereoscopic image following the movement and rotation of the object 4. The three-dimensional image does not need to be a still image, and can be a moving image if the capability of the arithmetic unit permits.

  FIG. 4 is a diagram illustrating a second system of the present invention different from FIG. In FIG. 3, the three-dimensional data used for displaying the video is recorded inside the arithmetic unit, but in FIG. 4, it is held outside and acquired by communication. FIG. 2 shows an example embodying the system illustrated in FIG. As shown in FIG. 2, a display operation can be performed by adding buttons to the reference object and communicating between the reference object and the system. The operation that can be performed here is, for example, an operation in which the playback speed of an image such as playback, stop, fast forward, and rewind is performed like a video when displaying a 3D video of a moving image. In addition, operations such as enlargement, reduction, and color change can be considered for a stationary 3D image. In this way, if the button is used, operations other than the change in the position and orientation of the object that can be performed by the reference object can be performed on the 3D image. Thus, it can also be used as an operation for playing and stopping a moving image and a game interface.

  FIG. 5 is a diagram showing a first application example in which the present invention is applied to a display device for displaying a three-dimensional image and moving it freely to change the angle. Since the reference object is a hand-sized frame, and even if there are a plurality of users, the same reference coordinates are defined for the same reference object, it is possible to have a discussion in front of a three-dimensional image.

  FIG. 6 is a diagram showing a second application example in which the present invention is applied to a virtual large-screen display that moves accompanying a mobile phone. The reference object is a mobile phone or an antenna protruding therefrom.

  FIG. 7 is a diagram showing a third application example in which the present invention is applied to a desktop virtual large-screen display. The reference object is a frame attached to the keyboard.

  FIG. 8 is a diagram showing a fourth application example to which the present invention is applied. In this example, the entire table becomes the virtual display of the computer. The reference object is a pattern placed on a desk.

  FIG. 9 is a diagram showing a fifth application example to which the present invention is applied. Video advertisements can be displayed on clothes and cars. The reference object is a fixed pattern drawn on clothes or a car. In this case, the data of the three-dimensional image to be displayed is on the person wearing the clothes on which the reference object is drawn or on the car side, and is sent to a nearby wearer by communication.

  FIG. 10 is a diagram showing a sixth application example to which the present invention is applied. It enables three-dimensional decorations such as clothes and costumes on the stage. The reference object is a pattern fixed to the body. In this figure, the performer has a large number of reference objects attached to the body, each of which can be distinguished, and by overlaying the composite images displayed for the coordinates of each limb, the panda walks to the observer. Looks like you are.

The example which actualizes the system of this invention illustrated in FIG. 3 is shown. FIG. 5 illustrates an example in which the system illustrated in FIG. 4 is embodied. It is a figure which illustrates the basic composition of the system of the present invention. FIG. 4 is a diagram illustrating a second system of the present invention different from FIG. 3. It is a figure which shows the 1st application example to which this invention is applied. It is a figure which shows the 2nd application example to which this invention is applied. It is a figure which shows the 3rd application example to which this invention is applied. It is a figure which shows the 4th application example to which this invention is applied. It is a figure which shows the 5th application example to which this invention is applied. It is a figure which shows the 6th application example to which this invention is applied. It is a figure for demonstrating the method to project on virtual three-dimensional space based on a prior art. It is a figure for demonstrating the method to input the image | video of a reference | standard object using two cameras, and to measure the position and attitude | position. It is a figure for demonstrating conversion from the camera coordinate system C2 to the coordinate system C1 of a display apparatus. It is a figure explaining the conversion from coordinate system C4 fixed on the object to camera coordinate system C2.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Eyeglass-like stereo image display apparatus 2 Camera 4 Object

Claims (1)

In an image display system for displaying a three-dimensional stereoscopic image on a display device,
A reference object that provides a reference coordinate system for displaying a 3D stereoscopic image to be displayed;
A 3D information input device that observes a reference object placed in the field of view and measures its position and orientation in real time;
Based on the measured position and orientation of the reference object, a transformation matrix between the coordinates fixed to the reference object and the coordinates fixed to the 3D information input device is calculated in real time, and based on this, the coordinate conversion of the 3D stereoscopic image to be displayed is performed. A coordinate transformation device for performing
A display image synthesizing device that calculates a two-dimensional image that can be seen from only the right or left eye position of the coordinate-converted three-dimensional image;
A display device having a see-through function that displays a composite image only on one eye and shows a calculated three-dimensional image to the user, and displays an image that is actually visible to the user as it is in a portion where the composite image is not displayed And an image display system.

Images