JP2003256876A - Device and method for displaying composite sense of reality, recording medium and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To present a virtual real world where the harmony of an appearance such as the size or color of a real object and the surrounding environment is realized without disposing any real object in a real space. <P>SOLUTION: An object indicating index where a plurality of markers which have respective identification information and the mutual positional relations of which are already known are disposed is photographed with a camera, the object indicating index is specified on the basis of the photographed image, and three-dimensional position/attitude disposed in a real space is obtained. Furthermore, a three-dimensional model corresponding to the detected identification information is disposed at the three-dimensional position/attitude obtained from the object indicating index in the virtual space so that the composite real space can be prepared. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mixed reality display device and method, a storage medium, and a computer program for incorporating aspects of the real world in a virtual space implemented on a computer or a network, and more particularly to a user. A virtual reality display device and method, a storage medium, and a computer that extend the virtual space to the real world by positively utilizing the real world situation such as the place where the object exists and the position and orientation of the real world object Regarding the program.

More specifically, the present invention provides a mixed reality for presenting a virtual reality world in which the appearance of the real object such as size and color and the surrounding environment are harmonized without arranging the real object in the physical space. The present invention relates to a sensation display device and method, a storage medium, and a computer program, and more particularly, to a mixed reality display device and method, a storage medium, and a computer program for synthesizing and displaying a virtual object that imitates a real object in a real space. .

[0003]

2. Description of the Related Art In the modern age of advanced information processing technology and information communication technology, information devices such as personal computers and personal digital assistants are ubiquitous in the real world such as offices and homes. To do. In such an environment, actively connect to "Ubiquitous Computing," which connects devices to obtain desired information anytime and anywhere, and the real-world situation (real-world things, user positions, etc.). It is expected that the augmented reality system (Augmented Reality: AR) used for

The concept of ubiquitous computing is that the environment of computers that can be used regardless of where a person moves is the same. In other words, because it is "anytime, anywhere", the ultimate ubiquitous computing is not necessarily a computer or PDA (Personal Dig
Ital Assistant) and information terminals such as mobile phones are not required.

Further, according to the augmented reality system, it is possible to provide a service utilizing the real world situation such as the position of the user. In this case, the user simply holds the mobile terminal, and the system presents information according to real-world things in the vicinity of the user or in the field of view, and uses the vast amount of information on the network to carry out daily life. Can support all aspects. For example, when a user visits a CD shop by holding a camera-equipped mobile terminal in a shopping mall, a recommended new song is displayed on the terminal. Also, looking at the signboard of the restaurant, the impression of the food is displayed.

[0006] For example, when purchasing household furniture, electrical appliances, interiors, etc., whether they match the size of the room or whether they match the colors of the room or other furniture, etc. There are many things I want to check before purchasing. Also, when redecorating a room, you can
I would like to try various arrangements such as the interior.

When the user lives only in the real world, it is difficult to actually arrange the real thing in the room at home before purchasing. In addition, furniture, electric appliances, interiors, etc. are generally large and heavy, so it is not realistic to actually move the objects or try various arrangements indoors because it is hard work.

On the other hand, a three-dimensional virtual house imitating an actual house is constructed on a computer, and three-dimensional models such as furniture, electric appliances, and interiors are arranged in the virtual house, and various virtual houses are arranged. Applications have been devised that allow the house to be viewed from a perspective. These applications are used to improve the processing speed of computers in recent years and to improve 3D computer graphics (3D).
With the development of CG) technology, it is possible to have expressive power full of reality.

However, there are some problems in actually using these virtual reality applications.

First, one has to build a model of the room in which one lives. It is rare for anyone to be able to create the floor plan of the room in which they live in an instant and accurately.Generally, the room is actually measured with a tape measure or the like, and the floor plan obtained when purchasing the house is used as a reference. There must be. A rough room model is sufficient if you want to express the general atmosphere of the room, but it is essential to build an accurate room model to check the size of furniture, appliances, interiors, etc.

Further, in order to imitate a house, it is necessary to have a three-dimensional model of furniture, electric appliances, interiors, etc. owned, but it is difficult to create or obtain these complete models, and it is difficult to obtain a plurality of models. At present, the most approximate model is selected from the three-dimensional model materials. Therefore, a virtual house different from the actual house will be constructed.

Further, although the computer processing speed has been improved and the CG technology has been developed and the expressive power has been remarkably improved, it is difficult to express it as if it is mistaken for reality, and even if possible, a great amount of calculation time is required. Mostly it costs.

[0013]

SUMMARY OF THE INVENTION It is an object of the present invention to provide an excellent mixed reality display device and method capable of incorporating aspects of the real world in a virtual space implemented on a computer or a network, and a computer. To provide a program.

A further object of the present invention is to extend the virtual space to the real world by positively utilizing the real world situation such as the place where the user exists and the position / orientation of the real world object in the real space. To provide an excellent mixed reality display device and method, and a computer program.

A further object of the present invention is to present the user with a virtual reality world in which the appearance such as the size and color of the real object and the surrounding environment are harmonized without disposing the real object in the real space. To provide an excellent mixed reality display device and method, and a computer program.

A further object of the present invention is to provide an excellent mixed reality display device and method, and a computer program, which can display a virtual object that imitates a real object by combining it in a real space. .

[0017]

The present invention has been made in consideration of the above problems, and a first aspect thereof is a mixed reality display device for synthesizing and displaying a virtual object in a real space. Or a method, an image input section or step for capturing a scene in a real space including an object teaching index,
A position / orientation calculation unit or step for calculating the spatial position and orientation of the object teaching index based on the input image by the image input unit or step; and the object teaching index based on the input image by the image input unit or step. An object recognition unit or step for recognizing identification information possessed by the object recognition unit or step for recognizing a corresponding physical object; and a virtual object consisting of a three-dimensional model of the physical object recognized by the object recognition unit or step for the position / orientation calculation unit. Or an image synthesizing unit or step of synthesizing on an input image by rotating / moving in accordance with the spatial position and orientation calculated in step, or a mixed reality display device or method. .

According to the mixed reality display device or method according to the first aspect of the present invention, the object teaching index is photographed by the camera, and the identification information possessed by the object teaching index is acquired based on the photographed image. , The three-dimensional position / orientation in which the object teaching index is arranged in the physical space is acquired. Furthermore, by arranging the three-dimensional model corresponding to the detected identification information in the three-dimensional position / orientation obtained from the object teaching index in the virtual space, the mixed reality space can be created in a relatively short time and at low cost. can do.

For example, when purchasing household furniture, electric appliances, interiors, etc., whether they match the size of the room or the colors of the room and other furniture, etc. There are many things I want to check before purchasing. Also, when redecorating a room, you can
I would like to try various arrangements such as the interior. However, it is difficult to actually place the real thing in a room at home before purchasing. In addition, furniture, electric appliances, interiors, etc. are generally large and heavy, so it is not realistic to actually move the objects or try various arrangements indoors because it is hard work.

On the other hand, according to the mixed reality display device or method according to the first aspect of the present invention, furniture, home electric appliances,
If you want to place a real object such as an interior, you can simply place an object teaching index instead of the real object, and look and feel the size and color of the real object without placing it in the physical space. It is possible to present the user with a virtual reality world that is in harmony with the environment.

Therefore, the user can determine whether the furniture, electrical appliances, interior, etc. in the home match the size of the room based on the mixed reality image in which the virtual object is combined with the scene in the real space. It is possible to judge whether or not it is in harmony with the color of other furniture etc. before actually purchasing. In addition, for indoor scenes and objects other than virtual objects such as furniture, electrical appliances, and interiors, the actual images captured by the camera are used as they are. That is, the augmented reality image can be easily generated without the need to create or obtain a complete model for the physical space.

Here, the image composition unit or step is
You may make it further synthesize | combine the charge and specification regarding the physical object recognized by the said object recognition part, or other relevant information on an input image.

Further, a distance measuring unit or step for measuring the distance to the object teaching index may be further provided. In such a case, the image composition unit or step
By applying hidden surface processing to the virtual object based on the distance obtained by the distance measuring unit or step, a more natural composite image can be obtained.

Optical identification information is provided in the object teaching index. The optical identification information includes, for example, a unique color, shape, or a visual code such as a barcode, or a blinking pattern of light in which identification information or other data is encoded.

In such a case, the object recognition unit or step can extract the optical identification information from the input image and acquire the identification information of the object.

Further, the object teaching index may be provided with a plurality of optical identification information whose positional relationship is known. In such a case, the position / orientation calculation unit or step extracts the optical identification information from the input image and determines the spatial position and orientation based on the detected position of each optical identification information in the input image. It can be calculated.

For example, Robert M. Haralick, Chung-nan
Lee, Karsten Ottenberg, and Michael Nolle, "Analysis and Solutions of The Three Point Persp"
ective Pose Estimation Problem "(In Proceedings of
the Conference on ComputerVision and Pattern Reco
gnition, Maui, Hawaii, USA, pp.592-598, 1991) describes a method for measuring the three-dimensional position and orientation of an object from the known three points on the object. There is. Therefore, if three or more pieces of optical identification information are arranged in the object teaching index, the position / orientation calculation section or step uses the method of Robert et al.
It is possible to mathematically calculate the corresponding three-dimensional position and orientation based on the detected positions of the three pieces of optical identification information extracted on the input image.

Also, a paper entitled "Augmented Reality System Based on Marker Tracking and Its Calibration" by Hirokazu Kato, Mark Billinghurst, Koichi Asano, and Keihachiro Tachibana (Journal of the Virtual Reality Society of Japan, Vol.4, No.4) , 1999)
Describes a method of measuring a three-dimensional position and orientation of a parallelogram based on four markers arranged at each vertex of the parallelogram. Therefore, if four or more pieces of optical identification information forming the vertices of a parallelogram are arranged in the object teaching index, the position / orientation calculation unit or step can be input using the method of Kato et al. 4 extracted on the image
It is possible to mathematically calculate the corresponding three-dimensional position and orientation based on the detected positions of the one piece of optical identification information.

When the optical identification information is represented as a blinking pattern of light, the object recognizing unit or step extracts the blinking pattern from each blinking light source based on the input image and decodes them. The identification information of the object teaching index can be recognized.

Further, in such a case, the position / orientation calculation section or step extracts the blinking patterns from each blinking light source based on the input image, decodes these to identify each, and identifies each of them. The spatial position and orientation of each blinking light source can be calculated based on the detected position.

The second aspect of the present invention is a computer-readable physical form of computer software written so as to execute a process for synthesizing and displaying a virtual object in a real space on a computer system. A computer-readable storage medium in which the computer software captures a scene in real space including an object teaching index, and the object teaching index of the object teaching index based on the input image in the image input step. A position / orientation calculating step of calculating a spatial position and orientation; an object recognizing step of recognizing the identification information of the object teaching index based on the input image in the image input step to recognize a corresponding physical object; The position / orientation of the virtual object, which is a three-dimensional model of the physical object recognized in the object recognition step, A storage medium characterized by comprising a, an image combining step of combining on the input image by rotating, moving in accordance with the calculated spatial position and orientation in calculation step.

The storage medium according to the second aspect of the present invention is, for example, a medium that provides computer software in a computer-readable format to a general-purpose computer system capable of executing various program codes. Such a medium is, for example, a DVD (Digital Vers
atile Disc), CD (Compact Disc), FD (Flexible Disc)
Disk), MO (Magneto-Optical disc), and other removable storage media. Alternatively, it is technically possible to provide computer software to a specific computer system via a transmission medium such as a network (whether the network is wireless or wired).

Further, the storage medium according to the second aspect of the present invention is a computer system having a predetermined computer
A computer for realizing software functions
It defines a structural or functional collaborative relationship between software and a storage medium. In other words, the second aspect of the present invention
By installing the predetermined computer software in the computer system via the storage medium according to the first aspect, the cooperative action is exerted on the computer system, and the mixed reality display according to the first aspect of the present invention. It is possible to obtain the same effect as that of the device or the method thereof.

The third aspect of the present invention is a computer program written in a computer-readable format so as to execute a process for synthesizing and displaying a virtual object in a real space on a computer system. An image input step of capturing a scene in real space including the object teaching index, and a position / orientation calculating step of calculating a spatial position and orientation of the object teaching index based on the input image in the image input step. An object recognition step of recognizing the corresponding physical object by recognizing the identification information of the object teaching index based on the input image in the image input step, and a three-dimensional model of the physical object recognized in the object recognition step The virtual object consisting of
An image synthesizing step of rotating and moving in accordance with the spatial position and orientation calculated in the orientation calculating step and synthesizing on an input image.

A computer according to the third aspect of the present invention
The program defines a computer program written in a computer-readable format so as to realize a predetermined process on a computer system. In other words, by installing the computer program according to the third aspect of the present invention in the computer system, a cooperative action is exerted on the computer system, and the mixed reality according to the first aspect of the present invention. The same effect as that of the display device or the method thereof can be obtained.

Further objects, features and advantages of the present invention are as follows.
It will be apparent from the embodiments of the present invention described later and the more detailed description based on the accompanying drawings.

[0037]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below in detail with reference to the drawings.

The mixed reality display system according to the present invention is
A plurality of optical identification information is attached to a physical object existing in the real world, this is photographed by using a two-dimensional image sensor, and the object is identified by image processing and image recognition. It realizes an augmented reality (AR) service based on the real world situation of objects such as three-dimensional positions and postures. For example, mixed reality in which a part of the real space is expanded into a virtual world. Provide a feeling image.

As the optical identification information attached to the real-world object, an optical signal consisting of a blinking light source such as an LED can be used in addition to the visual code encoded in the color space pattern. In the latter case,
The data can be transmitted after being encoded into a time-series optical signal such as a blinking pattern in which the data does not change according to the distance.

The image sensor is, for example, a CMO.
S (Complementary Metal Oxide Semiconductor) sensor and CCD (Charge Coupled)
Device: Charge-coupled device) This is a structure in which an infinite number of light-receiving elements, that is, pixels are arranged in a two-dimensional array such as a sensor, and an optical signal and its spatial information are decoded by all pixels. The image sensor can capture the scene as a normal camera and receive optical signals located in the field of view of the image sensor from long distances. Therefore, the real world object provided with the optical identification information can also be used as a transmitter for transmitting the real world situation.

It is assumed that the plurality of optical identification information attached to the real world object are not only identifiable to each other, but also their spatial positional relationship is known in advance. In such a case, by recognizing the real-world object with a measuring device having a spatial resolution, it is possible to not only identify the real-world object but also measure its three-dimensional position, posture, and the like. The spatial position and orientation of a real-world object is, so to speak, a real-world situation.

Therefore, by positively using the measurement result of the three-dimensional position / orientation of the real world object as the real world situation (the real world thing or the position of the user, etc.), the augmented reality system (Augmented Reality: AR) is used. ) Can be used for realization.

According to the augmented reality system, it is possible to provide a service using real world information such as the position of the user. In this case, the user simply holds the mobile terminal, and the system presents information according to real-world things in the vicinity of the user or in the field of view, and uses the vast amount of information on the network to carry out daily life. You can enjoy support in all aspects. For example, if you visit a CD shop in a shopping mall while holding your camera-equipped mobile device,
The recommended new song is displayed on the terminal. Also, looking at the signboard of the restaurant, the impression of the food is displayed.

In the specification of Japanese Patent Application No. 2001-325356 already assigned to the present applicant, an optical signal for carrying or transmitting identification information and other data in an optical format and a high-speed signal for capturing the optical signal. A data communication system that realizes data transmission that has spatial resolution and can be used over a long distance by using a two-dimensional image sensor is disclosed. In the data communication system described in the specification, a blinking light source such as an LED is used as an optical signal, and data is not changed according to a distance, instead of being encoded into a color space pattern like a visual code. Data is transmitted after being encoded into a time-series optical signal such as a pattern. The image sensor has a configuration in which an infinite number of light receiving elements, that is, pixels, such as a CMOS sensor or a CCD sensor, are arranged in a two-dimensional array, and an optical signal and its spatial information are decoded by all pixels. Thus, the image sensor can capture the scene as a normal camera and receive optical signals located in the field of view of the image sensor from long distances.

A. Configuration of Mixed Reality System FIG. 1 schematically shows the configuration of a mixed reality system according to an embodiment of the present invention. As shown in the figure, this system includes a camera 101 and a mixed reality device 102.
And an image display device 103 and an object teaching index 104.

A plurality of distinguishable markers are attached to the object teaching index 104 in order to measure the three-dimensional position and orientation of the object (described later). It is assumed that each marker has visual identification information including a unique color, shape, pattern, or blinking pattern of light, and can be identified for each marker. In addition, it is assumed that the visual identification information of each marker and the spatial positional relationship between the markers are known in the mixed reality system.

The camera 101 uses, for example, an ideal pinhole lens with no lens distortion, and a CCD or C
A two-dimensional image sensor such as a MOS can capture a landscape including the object teaching index 104 and other scenes. The image frame imaged by the camera 101 is received data having a spatial resolution as the mixed data 1
It is input to 02.

The mixed reality apparatus 102 is composed of a computer system such as a personal computer (PC), and is connected to the camera 101 via a USB (Universal Serial Bus) or other external interface to take a photographed image by the computer. -Can be imported in the form of a file. On the device 102, an application that executes three-dimensional position / orientation measurement processing based on marker detection in the image frame is running, and the object teaching index 1
Based on the three-dimensional position of the marker and the position on the input image, the real world situation such as the three-dimensional position and posture of the marker is measured from the scene image including 03. Then, the mixed reality apparatus 102
Provide augmented reality service (described later) according to the real world situation.

In the example shown in FIG. 1, the object teaching index 104.
Is placed on a TV stand sandwiched between speakers in a living room, and virtually acts as a TV receiver (hereinafter referred to as "TV"). At this time, an area (scene) including the object teaching index 104 is photographed using the camera 101.

The image data acquired by the camera 101 is
It has a spatial resolution and is input to the mixed reality apparatus 102, and the identification information of the object teaching index 104 and its spatial position and orientation are extracted. Then, the extracted object teaching index 1
The virtual object 105 (here, the three-dimensional model of the television) corresponding to the identification information 04 is synthesized with the input image data and output to the image display device 103. At this time, the three-dimensional model of the television is mapped to the coordinate system of the real space based on the real world situation such as the spatial position and orientation obtained from the object teaching index 104.

In this way, the virtual object 105 (television) that does not actually exist on the display screen of the image display device 103 that displays and outputs the input image from the camera 101.
Appears on the image of the real space (superimposed display). Therefore, the user can display whether or not the appearance such as the size and the color of the real object is in harmony with the surrounding environment without arranging the real objects such as furniture and appliances in the real space.

FIG. 2 schematically shows the hardware configuration of the mixed reality apparatus 102 according to this embodiment.

CPU (Cent) which is the main controller
The ral processing unit) 1 executes various applications under the control of an operating system (OS).

In the present embodiment, the CPU 1 uses, for example, a three-dimensional position for measuring the spatial position and orientation of the object teaching index 104 based on data having a spatial resolution such as a captured image of the camera 101. Executes a posture measurement application, an AR service application that provides an augmented reality (AR) service that uses the real-world situation of a real-world object such as the spatial position and orientation of the object teaching index 104, etc. be able to.

As shown in the figure, the CPU 1 is interconnected with other devices (described later) by a bus 8.

The main memory 2 is a storage device used for loading a program code executed in the CPU 1 and temporarily storing work data of an execution program, and is a semiconductor such as a DRAM (Dynamic RAM). Memory is used. For example, a three-dimensional position / orientation measurement application for measuring the spatial position and orientation of the object teaching index 104 based on data having a spatial resolution such as a captured image of the camera 101, and the object teaching index 10
Augmented Reality (A) according to the real-world situation of the real-world object including the spatial position and posture of 4
R) An AR service application or the like that provides a service is loaded into the main memory 2. Also, the camera 10
The calculation result regarding the spatial position and orientation of the object teaching index 104 based on the captured image by No. 1 and various AR service contents (for example, furniture and electric appliances) generated according to the real world situation obtained from the calculation result. , A three-dimensional model of a virtual object such as an interior) is temporarily stored in the main memory 2 as work data.

The ROM (Read Only Memory) 3 is
A semiconductor memory that permanently stores data, such as a self-diagnosis test (POST: Power On Sel) at startup.
f Test) and hardware input / output program code (BIOS: Basic Input / Output System).

The display controller 4 is a CPU
1 is a dedicated controller for actually processing the drawing command issued by 1. The drawing data processed by the display controller 4 is once written in, for example, a frame buffer (not shown), and then the CRT (Cathod).
e Ray Tube: Cathode ray tube display and LCD (Liquid
The image is displayed on the screen by the image display device 103 such as a crystal display.

The display screen of the image display device 103 generally has a role of visually feeding back the contents of input from the user, the processing result thereof, or an error or other system message to the user. Further, in the present embodiment, on the display screen of the image display device 103, an AR such as an augmented reality scene in which the virtual object 105 corresponding to the object teaching index 104 is combined with the image captured by the camera 101.
It is used to display and output services and contents. The AR service content referred to here can be, for example, a mixed reality image in which a three-dimensional model of a virtual object such as furniture, an electric appliance, or an interior is incorporated in an image captured by the camera 101. Details of the mixed reality image generation method will be given later.

The input device interface 5 is a device for connecting user input devices such as the keyboard 12 and the mouse 13 to the mixed reality apparatus 102.

The keyboard 12 and the mouse 13 have a role of taking user input such as data and commands into the system. In this embodiment, the keyboard 12 and the mouse 13
Starts the 3D position / orientation measurement application, A
It is used by the user to instruct activation of R service application, designation of AR service, and the like.

The network interface 6 is E
According to a predetermined communication protocol such as Internet (registered trademark), the system 102 is connected to a LAN (Local Area Net).
work) and even wide area networks such as the Internet.

On the network, a plurality of host terminals (not shown) are connected in a transparent state to construct a distributed computing environment. Distribution services such as software programs and data contents can be provided on the network. For example, a three-dimensional position / orientation measurement application for measuring the spatial position or orientation of the object teaching index 104 based on data having a spatial resolution such as the captured image of the camera 101, or detected from the captured image. A plurality of algorithms (for example, in a library) for three-dimensional position / estimation based on a plurality of markers, object teaching index 10
Augmented Reality (A) according to the real-world situation of the real-world object including the spatial position and posture of 4
R) AR services and applications that provide services can be downloaded via the network. Also, AR service contents used in AR service applications (for example, three-dimensional models of virtual objects such as furniture, appliances, interiors, etc.)
Can be distributed / distributed to / from other host devices via a network. Further, when handling products such as furniture, electric appliances, and interiors as an AR service that provides a mixed reality image, the price (or the usage fee for the augmented reality service), product specifications, the number of stocks, and other real objects Additional information regarding may be downloaded via the network.

The external device interface 7 is a hardware
It is a device for connecting an external device such as a disk drive (HDD) 14 and a media drive 15 to the mixed reality device 102.

The HDD 14 is an external storage device in which a magnetic disk as a storage carrier is fixedly mounted (well known),
It is superior to other external storage devices in terms of storage capacity and data transfer speed. Placing the software program on the HDD 14 in an executable state is called "installing" the program in the system. Usually HDD14
The operating system program code to be executed by the CPU 1, application programs, device drivers, etc. are stored in a non-volatile manner. For example, a three-dimensional position / orientation measurement application for measuring the spatial position or orientation of the object teaching index 104 based on data having a spatial resolution such as the image captured by the camera 101, or the space of the object teaching index 104. HDD 14 includes an AR service application that provides an augmented reality (AR) service according to the real world situation of real world objects such as physical position and posture.
Can be installed on. In addition, a plurality of algorithms (for example, in a library) that perform three-dimensional position / orientation estimation based on a plurality of markers detected from a captured image and AR service contents (for example, a library) used in an AR service application (for example, , 3D models of virtual objects such as furniture, appliances, interiors)
Etc. may be stored in the HDD 14. Further, when handling products such as furniture, electric appliances, and interiors as an AR service that provides a mixed reality image, the price (or the usage fee for the augmented reality service), product specifications, the number of stocks, and other real objects Additional information regarding the information may be stored.

The media drive 15 is a CD (Compact
Disc), MO (Magneto-Optical disc), DVD (Dig)
Ital Versatile Disc) is a device for loading a portable medium such as a disc and accessing the data recording surface thereof.

The portable medium mainly backs up software programs, data files, etc. as computer-readable data, and moves them between systems (that is, including sales, distribution and distribution).
It is used for the purpose. For example, a three-dimensional position / orientation measurement application for measuring the spatial position and orientation of the object teaching index 104 based on data having a spatial resolution such as a captured image of the camera 101, and the object teaching index 10
Augmented Reality (A) according to the real-world situation of the real-world object including the spatial position and posture of 4
R) AR services and applications that provide services can be physically distributed / distributed among a plurality of devices using these portable media. In addition, based on the markers of multiple points detected from the captured image,
A plurality of algorithms for estimating the posture (for example, in a library) and AR service contents used in the AR service application (for example, a three-dimensional model of a virtual object such as furniture, electric appliances, and interiors) are stored in another Portable media may be utilized to exchange with the device. As an AR service that provides mixed reality images, furniture and appliances,
When a product such as an interior is handled, additional information about the actual object such as the price (or usage fee of the augmented reality service), product specifications, and the number of stocks may be distributed via a portable medium.

The camera interface 9 is the camera 1
01 is a device for connecting, and is composed of, for example, a USB interface. Alternatively, it may be composed of an interface for acquiring a moving image from the camera 101 like a video capture card. Camera 101
Is a configuration in which pixels are arranged on a two-dimensional array, and an optical signal from the object teaching index 104 and its spatial information are decoded by all pixels.

Note that the mixed reality apparatus 1 as shown in FIG.
An example of 02 is a personal computer of IBM Corp.
PC / AT (Personal Computer / Advanced Technolog
y) "compatible machine or successor machine. Of course, a computer configured with another architecture can also be applied as the mixed reality apparatus 102 according to the present embodiment.

FIG. 3 shows, in the form of a flow chart, a mixed reality display processing procedure executed on the mixed reality apparatus 102. This processing procedure is actually the CPU 1
Launches the given mixed reality processing application,
This is realized by processing an input image from the camera 101 and displaying and outputting the processing result on the screen display device 103. Hereinafter, the mixed reality display process will be described in detail with reference to this flowchart.

When image data is input from the camera 101 (step S1), a separately defined "extraction of object teaching index, identification information thereof, acquisition of three-dimensional position / orientation information" processing routine (described later) The object teaching index 104 in the imaged image data is extracted, and its identification information and three-dimensional position / orientation information are acquired (step S2).

In the augmented reality service according to the real world situation (steps S7 and S8), when the two-dimensional image corresponding to the identification information is simply displayed at the position of the object teaching index 104 in the image data, In the processing routine in step S2, the acquisition of the three-dimensional position / orientation information of the object teaching index 104 may be omitted. Object teaching index 1
The identification information 04 and the method for acquiring the three-dimensional position / orientation information are processes that depend on the aspect of the object teaching index 104, and will be described in detail later together with the embodiment of the object teaching index 104.

Next, it is determined whether the object teaching index 104 is extracted from the image input from the camera 101 (step S3). If it is extracted, the process proceeds to the subsequent step S4, and an attempt is made to acquire a three-dimensional model of the real object corresponding to the extracted identification information of the object teaching index 104.

Here, the three-dimensional model is created by imitating the real object by a seller of the real object or a third party who performs a three-dimensional model creation service.
It may be stored in advance on the disk device 14 or may be sequentially downloaded from a predetermined information providing server (for example, a three-dimensional model database) via a network. In addition, if the real object is a product such as furniture, electrical appliances, interior, etc., additional information about the real object such as its price (or usage fee of augmented reality service), product specifications, and inventory quantity, and other real-time contents Etc. may be downloaded at the same time.

Next, it is determined whether the three-dimensional model has been successfully acquired (step S5).

If the three-dimensional model can be successfully acquired, the process proceeds to the next step S6, and the acquired three-dimensional model is acquired.
The dimensional model is rotated / moved in accordance with the three-dimensional position / orientation information of the object teaching index 104 acquired in the preceding step S2. Then, the mixed reality space image is created by synthesizing the image data which is the scene in the real space input from the camera 101 in step S1 and the three-dimensional model (step S7). At this time, additional information relating to the real object such as the charge related to the AR service, the specification of the product, the number of stocks, and the image data may be further combined.

Next, the mixed reality space image synthesized in the previous step S7 is displayed and output on the image display device 103 (step S8). Based on the mixed reality image in which a virtual object is combined with a scene in the real space, whether home furniture, electrical appliances, interior, etc. fit the size of the room,
It is possible to judge whether or not it is in harmony with the color of the room or other furniture before actually purchasing.

When it is determined in step S3 that the object teaching index 104 is not extracted from the input image, or when the three-dimensional model cannot be acquired in step S5, the image acquired in step S1 is acquired. The data may be displayed and output on the image display device 103 as it is.

Then, it is determined whether or not to continue the mixed reality display processing (step S9). Then, if this processing routine is continued, the process returns to step S1 and the same operation as described above is repeatedly executed. On the other hand, if it is not continued, the entire processing routine is ended here.

The end condition of this processing routine may be, for example, a user input via the keyboard 12 or the mouse 13, or a rule (for example, game over in a game) predetermined in the application. . Further, a restriction on hardware or software such as memory full may be used as the end condition.

B. Configuration Example of Object Teaching Index (1) Configuration Example 1: FIG. 4 shows one configuration example of the object teaching index 104. As shown in the figure, the object teaching index 104 has a plate-like triangle, and optically identifiable markers 105a to 105c are arranged at respective vertices thereof. The corresponding real object can be specified by the combination of the identification information held by each of the markers 105a to 105c. Further, in the object teaching index 104, information (for example, character information of the product model number) related to the corresponding real object is specified (printed).

Each of the markers 105a to 105c has optically unique identification information consisting of a unique color, shape, pattern, etc., and can be identified for each marker. Further, it is assumed that the visual identification information of the markers 105a to 105c and the spatial positional relationship between the markers are known in the present system. Therefore, the object teaching index 10
When attached to a certain real object, 4 gives information for identifying this object by a combination of identification information held by the respective markers 105a to 105c, and also provides information on the real world situation such as spatial position and posture. Can be given.

On the other hand, the camera 101 connected to the mixed reality display device 102 is, for example, a CMOS sensor or CC.
An image sensor (not shown) in which an infinite number of light receiving elements, that is, pixels is arranged in a two-dimensional array, such as a D sensor, is provided, and spatially with respect to each of the markers 105a to 105c on the object teaching index 104. Has a high resolution.

Here, the three-dimensional positional relationship of all the markers 105a to 105c arranged on the object teaching index 104 is known. The object teaching index 104 is, for example, X
The rotation matrix R of the measurement target object 101 is expressed as follows, where α is rotated around the axis, β is rotated around the Y axis, and γ is rotated around the Z axis.

[0085]

[Equation 1]

If the object teaching index 104 is moved to the position (X _t , Y _t , Z _t ) in the camera coordinate system, the translation matrix T is expressed as follows.

[0087]

[Equation 2]

Here, the point (X _m ,
When the marker 105 located at Y _m , Z _m ) is rotated and translated, and then converted into a point (X _c , Y _c , Z _c ), these relationships are expressed as follows.

[0089]

[Equation 3]

In the above equation, if three markers having known spatial positional relationships can be detected from the image of the object teaching index 104, R and T, that is, the object teaching index 104, can be obtained as a solution of simultaneous equations. It is shown that the three-dimensional position and orientation of can be obtained. For example, Robert M. Ha
ralick, Chung-nan Lee, Karsten Ottenberg, and Mich
ael Nolle co-authored paper "Analysis and Solutions of The
Three Point Perspective Pose Estimation Problem "
(In Proceedings of the Conference on ComputerVisi
on and Pattern Recognition, Maui, Hawaii, USA, pp.
592-598, 1991), a known 3 mounted on the target object.
A method for measuring the three-dimensional position and orientation of the object from the position of a point is described.

The coordinate values (X, Y) of the marker 105 moved to the point (X _c , Y _c , Z _c ) on the two-dimensional captured image are as follows.

[0092]

[Equation 4]

Therefore, if it is possible to detect three markers whose spatial positional relationship is known from the picked-up image, R and T, that is, the three-dimensional position and orientation of the object teaching index 104, can be determined as the solution of the simultaneous equations. It can be obtained by mathematical calculation.

Also, Hirokazu Kato, Mark Billinghurst, Koichi Asano, and Keihachiro Tachibana, "Augmented Reality System Based on Marker Tracking and Its Calibration" (The Virtual Reality Society of Japan, Vol.4, No.4) , 1999)
Describes a method of measuring a three-dimensional position and orientation of a parallelogram based on four markers arranged at each vertex of the parallelogram. Therefore, the object teaching index 10
By arranging the four markers 105a to 105d in a parallelogram shape in No. 4, the position / orientation of the object teaching index 104 can also be calculated mathematically according to the method described in the paper by Kato. It should be fully understood that more accurate measurement can be expected by using the information of 4 points rather than 3 points.

FIG. 5 shows the object teaching index 10 shown in FIG.
4 shows the processing procedure for extracting the object teaching index 104 and acquiring the identification information and the three-dimensional position / orientation information thereof in the case of using 4 in the form of a flowchart. This processing procedure corresponds to step S2 of the processing routine shown in FIG. 3, and in actuality, the CPU 1 extracts a predetermined object teaching index, and activates the identification information and 3D position / orientation information acquisition processing application. Then, the input image from the camera 101 is processed. Less than,
With reference to this flowchart, the object teaching index 10
The recognition process of No. 4 will be described in detail.

When the markers are extracted from the input image from the camera 101 (step S11), it is determined whether there are three extracted markers (step S12). this is,
This is based on the fact that the mathematical calculation method described in the paper by Robert et al. (Described above) can estimate the three-dimensional position and orientation from the image coordinates of three points. When three or more markers are set on the object teaching index 104, the number of required markers may be set to three or more depending on the algorithm for estimating the three-dimensional position and orientation.

If it is determined in step S12 that the number of extracted markers is three, then the Robe
Estimate the three-dimensional position and orientation according to the mathematical calculation method described in the paper by rt (above) (step S1).
3).

Further, as an alternative process of the three-dimensional position and orientation in steps S12 and S13, four markers are arranged so as to form a parallelogram on the object teaching index 104, and the camera 101 is used. When four markers having a parallelogram shape can be extracted from the input image, the three-dimensional position and orientation of the object teaching index 104 is calculated by mathematical calculation according to the method described in the article by Kato Soto (above). To estimate. Further, in this case, when only three markers can be extracted, the three-dimensional position and orientation may be estimated according to the mathematical calculation method described in the article by Robert et al.

Then, the identification information of the object teaching index 104 is recognized (step S14). Identification information such as a model number is specified in the object teaching index 104, for example, in the form of character information, and thus this is recognized using a character recognition technique or the like. Alternatively, if the identification information is specified in the object teaching index 104 in the form of a bar code or other visual code instead of characters, the identification information can be recognized by the pattern recognition.

On the other hand, if it is determined in step S12 that the number of the extracted markers is less than three, the result that the object teaching index 104 cannot be identified is returned, and this processing routine ends. .

As described above, according to the system including the camera 101 and the mixed reality display device 102 according to the present embodiment, the identification information and 3 of the object teaching index 104 are acquired.
Dimensional position / orientation information can be acquired at the same time.

As shown in FIG. 4, the plurality of markers 105a to 105c are installed on the object teaching index 104 because it is desired to estimate the three-dimensional position and orientation of the object teaching index 104 with respect to the image picked up by the camera 101 (in other words, in other words). ,
This is because it is desired to utilize the spatial resolution of the camera 101 in object recognition). Therefore, the object teaching index 10
In the case where the 3D position and orientation are not required in the recognition of 4, for example, when the 2D image corresponding to the identification information is simply displayed at the position of the object teaching index 104 in the image data in an overlapping manner, the identification information is obtained. It is sufficient to dispose only a single marker in the object teaching index 104.

Further, a marker 105 is added to the object teaching index 104.
Even if there is only one, by using a plurality of cameras 101, the object teaching index 104 can be obtained from the principle of triangulation (well known).
It is possible to estimate the three-dimensional position and orientation of the.

(2) Configuration Example 2: FIG. 6 shows another configuration example of the object teaching index 104. In the figure, the markers arranged at the respective vertices of the object teaching index 104 are
Instead of static optical identification information such as color, shape, pattern, etc., a light source 105a ′ such as an LED capable of pulse transmission
105c 'is installed. Each of these light sources has an ID so that they can be identified, and the light sources are caused to blink according to a blinking pattern corresponding to each ID. Further, each of the light sources 105a 'to 105c' can also encode transmission data other than the ID into a blinking pattern for transmission.

Further, in the approximate center of the object teaching index 104,
Information (for example, character information of the product model number) related to the corresponding real object is specified (printed). However, it is also possible to encode both the IDs of the light sources 105a ′ to 105c ′ and the identification information of the object teaching index into a blinking pattern and transmit them simultaneously. For example, when transmitting 8-bit data, the upper 6 bits are assigned to the identification information of the object teaching index 104, and the lower 2 bits are assigned to each of the light sources 105a'-10.
It can be assigned to the ID of 5c '. In such a case, printing of information on the body of the object teaching index 104 (and image recognition of print information) can be omitted.

The camera 101 connected to the mixed reality display device 102 has an image sensor (not shown) in which an infinite number of light receiving elements, that is, pixels are arranged in a two-dimensional array, such as a CMOS sensor or a CCD sensor. )
And has a spatial resolution with respect to each of the markers 105a ′ to 105c ′ formed of blinking light sources on the object teaching index 104. Then, spatially recognizing the blinking pattern of each of the light sources 105a ′ to 105c ′ based on the moving image captured by the camera 101, and recognizing the light receiving position of each blinking pattern and the ID represented by the blinking pattern. You can

In this way, each of the markers 105a 'to 105a
When c ′ is composed of a device that blinks light such as an LED, each of the markers 105a ′ to 105c ′ can encode data and transmit it in a time-series optical signal such as a blinking pattern. In such a case, each marker 1
The object teaching index 104 including 05a ′ to 105c ′ is
It can function not only as identification information of a real object or as an index for measuring a spatial position and orientation, but also as a transmitting device. Further, since the data of the optical signal does not change according to the distance, it is possible to perform robust data transmission with respect to the distance. A data communication system using such a blinking pattern of light is disclosed in the specification of Japanese Patent Application No. 2001-325356 (mentioned above) already assigned to the present applicant.

In FIG. 7, the camera 101 and the mixed reality display device 1 when the object teaching index 104 is equipped with three light emitting diodes as the respective markers 105a 'to 105c'.
The operation characteristics on the 02 side are schematically shown. Each of the light emitting diodes 105a 'to 105c' can encode data to be transmitted, such as identification information, into a blinking pattern of light and send it as an optical signal. Of course, the baseband signal composed of the transmission data may be represented as it is by a light blinking pattern, or the optical signal may be output after performing modulation processing such as frequency modulation or amplitude modulation.

In this case, the blinking pattern of each light emitting diode is controlled by a condenser lens system (not shown).
An image is formed on the light receiving surface of the one-dimensional two-dimensional image sensor and detected by the light receiving element at a position corresponding to the position or orientation of the object teaching index 104 in the real space.

For example, each of the light emitting diodes 105a 'to 105c on the two-dimensional light receiving surface of the image sensor.
The coordinate values of the light receiving element (pixel) on which the lit light of 'is formed are (10, 10), (90, 90), (40, 7), respectively.
0), the blinking pattern of the corresponding light emitting diode is detected as a temporal change in the received light intensity (brightness) at each pixel position. By binarizing the received light intensity with a predetermined threshold value, the 1/0 bit string corresponding to the original transmission data can be restored.

As described above, the mixed reality display device 102
Decodes the captured image of the scene including the measurement target object 101 by all the pixels by the camera 101 to obtain the optical signal transmitted from each of the markers 105a ′ to 105c ′ and the spatial information of the object teaching index 104. be able to. That is, it is possible to perform data transmission with the object teaching index 104 by an optical signal, and it is possible to provide an augmented reality service based on the real world situation that spatial information means.

FIG. 8 shows the object teaching index 10 shown in FIG.
4 shows the processing procedure for extracting the object teaching index 104 and acquiring the identification information and the three-dimensional position / orientation information thereof in the case of using 4 in the form of a flowchart. This processing procedure corresponds to step S2 of the processing routine shown in FIG. 3, and in reality, the CPU 1 extracts a predetermined object teaching index and activates an identification information and 3D position / orientation information acquisition processing application. Then, the input image from the camera 101 is processed. Less than,
With reference to this flowchart, the object teaching index 10
The recognition process of No. 4 will be described in detail.

First, the input image from the camera 101 is subjected to image processing to extract a blinking pattern and decode transmission data represented by the blinking pattern (step S21).

Then, it is determined whether or not there are three extracted blink patterns (step S22). This is Rob
This is based on the fact that the mathematical calculation method described in the paper by Ert et al. (described above) can estimate the three-dimensional position and orientation from the image coordinates of three points.

If it is determined in step S22 that the number of extracted markers is three, then the Robe
The three-dimensional position and orientation are estimated according to the mathematical calculation method described in a paper by rt (above) (step S2).
3). In addition, the object teaching index 104 in the form of a blinking pattern
When the data is sent from the device, it may be integrated with the decoding result to estimate the three-dimensional position and orientation.

Further, as an alternative process of the three-dimensional position and posture in steps S22 and S23, four blinking light sources are arranged so as to form a parallelogram on the object teaching index 104, and the camera 101 is operated. When four blinking patterns in a parallelogram shape can be extracted from the input image of, the three-dimensional position of the object teaching index 104 by mathematical calculation according to the method described in the paper by Kato, et al. And the attitude is estimated.

Then, the identification information of the object teaching index 104 is recognized (step S24). Identification information such as a model number is specified in the object teaching index 104, for example, in the form of character information, and thus this is recognized using a character recognition technique or the like. Alternatively, if the identification information is specified in the object teaching index 104 in the form of a bar code or other visual code instead of characters, the identification information can be recognized by the pattern recognition.

If data is sent from the object teaching index 104 in the form of a blinking pattern, step S
At 24, the identification information may be recognized by integrating with the decoding result.

Alternatively, each of the light sources 105a 'to 105c'
If the information is transmitted only by the ID transmitted in the form of a blinking pattern by and the information is not printed on the object teaching index 104, in step S24, each of the light sources 105a 'to 105
The identification information is recognized only by the decoding result of the blinking pattern in c '. For example, each light source 105a'-1
From 05c ', the upper 6 bits are assigned to the identification information of the object teaching index 104, and the lower 2 bits are assigned to each light source 105
The 8-bit data assigned to the IDs a'to 105c 'is transmitted, and is decoded.

FIG. 9 shows the object teaching index 10 shown in FIG.
4 shows a modification of No. 4. In the figure, light sources 105a 'to 105c' such as LEDs capable of pulse transmission are installed at each vertex of the object teaching index 104. Also,
Instead of printing information (for example, character information of the product model number) related to the corresponding real object in the approximate center of the object teaching index 104, a light source 106 that encodes and transmits this information in a blinking pattern of light. Are additionally provided.

In such a case, the mixed reality display device 10
On the second side, the light source 1 is based on the input image from the camera 101.
By spatially recognizing the 06 blinking pattern,
The identification information in the object teaching index 104 can be decoded.

(3) Configuration Example 3: FIG. 10 shows another configuration example of the object teaching index 104. The optical identification information of the object teaching index 104 shown in the figure is composed of "Cybercode".

This cyber code is a kind of two-dimensional visual code, and includes a "guide bar display area" for indicating the whereabouts of the cyber code and a "code. Code bar" for displaying a two-dimensional code pattern. Pattern display area "
It is composed of and forms a mosaic pattern. The code pattern display area has an n × m matrix (7 × in the figure).
It is composed of cells arranged in 7), and the identification information can be added by expressing each cell in binary with white or black.
However, the corner cells at the four corners of the code pattern display area are aligned (Image Regi
stration) pattern is always a black pattern.

The cyber code recognition procedure includes the steps of binarizing a captured image and the guide bar 1 from the binary image.
002 candidate discovery step and guide bar 10
02, the step of searching for the corner cell 1003 based on the position and the direction of the cell, and the step of decoding the image bitmap pattern in response to the detection of the guide bar 1002 and the corner cell 1003. It Furthermore, by checking the error bit, it is possible to confirm that the captured image contains the correct cyber code, and derive the identification information and the position information of the code. Further, based on the position of the corner cell 1003, the distortion caused by the tilt of the camera or the object can be calculated and compensated.

For details of the cyber code, for example, Japanese Patent Laid-Open No. 2000-82 already assigned to the present applicant.
No. 108 (“two-dimensional code recognition processing method, two-dimensional code recognition processing device, and medium”).

FIG. 11 is a flowchart showing a processing procedure for extracting the object teaching index 104 when the object teaching index 104 shown in FIG. 10 is used, and acquiring its identification information and three-dimensional position / orientation information. It is shown in the format. This processing procedure is step S of the processing routine shown in FIG.
2, the CPU 1 actually activates an application for extracting a predetermined object teaching index and an acquisition processing application for its identification information and three-dimensional position / orientation information,
It is realized by processing the input image from. Hereinafter, the recognition process of the object teaching index 104 will be described in detail with reference to this flowchart.

First, the input image from the camera 101 is binarized (step S31), and then the candidates for the guide bar 1002 are extracted from the binarized image (step S32). Then, it is determined whether or not the guide bar 1002 is extracted (step S33).

If the guide bar 1002 is extracted, the process proceeds to the subsequent step S34, and the guide bar 1002 is extracted.
Corner cell 1 based on the position and direction of 2
Search for 003. Then, it is determined whether the corner cell 1003 has been extracted (step S35).

If the corner cell 1003 can be extracted, the process proceeds to the subsequent step S36 and the guide bar 1002 and the corner cell 10
In response to the detection of 03, the image bitmap pattern is decoded and the identification information of the object teaching index 1001 is acquired.

Next, the guides detected in the input image
The three-dimensional position and orientation of the object teaching index 1001 with respect to the camera 101 is estimated from the positions of the bar 1002 and the corner cell 1003 (step S37).

Here, in order to estimate the three-dimensional position and orientation from the image coordinates of four corner cells 1003 of the cyber code, for example, Hirokazu Kato, Mark Billinghu
A paper co-authored by rst, Koichi Asano and Keihachiro Tachibana "Augmented Reality System Based on Marker Tracking and Its Calibration"
(Journal of Virtual Reality Society of Japan, Vol.4, No.4,
1999) (supra), a method of mathematically calculating the three-dimensional position and orientation based on the four vertices of an identifiable parallelogram can be applied.

The object teaching index 1 having the configuration shown in FIG.
In 04, each corner cell 1003 cannot be identified by itself, but it can be identified by using the positional relationship with the guide bar 1002.
It is possible to estimate the dimensional position and orientation.

On the other hand, if the guide bar 1002 cannot be found in step S33 and the corner cell 1003 cannot be recognized in step S35, the three-dimensional position of the object teaching index 104 is determined. Then, the result indicating that the posture cannot be acquired is returned, and the entire processing routine ends.

C. By combining the image data with the method shown in the section B, the identification information of the object teaching index 104 arranged in the physical space and the three-dimensional position / orientation with respect to the camera 101 can be acquired. further,
By arranging the three-dimensional model corresponding to the detected identification information in the virtual space at the three-dimensional position / posture obtained from the object teaching index 104, a mixed reality space can be created.

It is desirable to create the object teaching index 104 for each product such as furniture, home electric appliances, and interiors. For example, the object teaching index 1202 is attached to a part of the product brochure 1201 as shown in FIG.

The consumer only needs to bring back the product brochure 1201 at the store he / she visited, for example, without purchasing the product. Then, the object teaching index 1202 for the corresponding product is arranged at a desired position in the room at home, and the camera 101 captures an image of the indoor scene.

This photographed image is displayed on the mixed reality display device 102.
, The object teaching index 104 placed in the physical space
And the three-dimensional position / orientation with respect to the camera 101 are acquired. Furthermore, in the virtual space, the three-dimensional model of the product corresponding to the detected identification information is set as the object teaching index 1
By arranging in the three-dimensional position / orientation obtained from 04, a mixed reality space image is created, and the screen display device 1
Output to 03. As a result, consumers can confirm the harmony with the room, such as the size and color of the product, before purchasing, based on the display output of the mixed reality space image without actually arranging the product indoors. Become.

The process of synthesizing a three-dimensional model in such a virtual space corresponds to the process performed in step S7 of the flowchart shown in FIG. Below, image data and 3
The method of synthesizing the dimensional model will be described with reference to FIG. 3 again.

First, the image data input from the camera 101 is drawn in the frame buffer corresponding to the output image (step S1). At this time, if the size of the input image and the size of the output image are different, the input image is reduced or enlarged to match the size of the output image.

Next, by the extraction processing of the object teaching index, the identification information thereof, and the acquisition processing routine of the three-dimensional position / orientation information,
The object teaching index 104 in the photographed image data is extracted (steps S2 and S3), and the real object (furniture, home electric appliance, interior, etc.) corresponding to the identification information of the extracted object teaching index 104 is extracted. A three-dimensional model is acquired (step S4).

Next, the three-dimensional model acquired in step S4 is placed in the three-dimensional position / orientation acquired in step S2 in the real space indicated by the input image of the camera 101 (step S6) and rendered. . At this time,
The attributes of the camera in the virtual space, such as the angle of view, must match the attributes of the camera in the real world (type of camera, lens distortion, etc.).

In this way, it is possible to create a mixed reality space image by superimposing and displaying a virtual three-dimensional model on the real space represented by the input image.

In the above method, the three-dimensional model is simply drawn on the input image. Therefore, for example, as shown in FIG. 13, the camera 101 and the object teaching index 10
When there is an obstacle such as the hand of the user between 4 (however, it is assumed that the markers 105a to 105c of the object teaching index 104 are not hidden by the hand), the composite image A03 is on the hand as illustrated. The virtual object A02 is displayed on the screen, resulting in an unnatural image.

On the other hand, for example, "Entertainment Vision Sens
When using an image sensor equipped with a three-dimensional measurement function that can detect distance information from the camera such as "or",
Since the distance information of the object with respect to the camera can be acquired at the same time as the image data, the above problem can be avoided. Alternatively, in the space in which the positional relationship of the object teaching index 104 with respect to the camera 101 is known, the camera 10
Since the object to be drawn can be determined based on the distance information from 1, the above problem can be similarly avoided.

That is, when each pixel is drawn, by selecting and drawing the object closest to the camera, it is possible to obtain a natural composite image B01 as shown in FIG.

"Entertainment Vision Sensor"
Is a high-performance CMOS image sensor capable of acquiring both a color moving image and three-dimensional distance information with one chip at 15 or 30 frames per second. By using these moving images and distance information at the same time, it is possible to realize functions such as image recognition, object recognition, and motion detection that previously required complicated information processing in a small system. This allows users of PCs, games, etc.
Various applications can be provided in various fields such as interface and 3D modeling, robot obstacle detection / person detection, personal authentication in security system, and image extraction function of videophone (http: // www. sony.co.jp/SonyInfo/News/Press
/).

[Supplement] The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the scope of the present invention. That is, the present invention has been disclosed in the form of exemplification, and the contents of this specification should not be construed in a limited manner. In order to determine the gist of the present invention, the section of the claims described at the beginning should be taken into consideration.

[0148]

As described above in detail, according to the present invention,
It is possible to provide an excellent mixed reality display device and method, a storage medium, and a computer program that can incorporate aspects of the real world in a virtual space implemented on a computer or a network.

Further, according to the present invention, the virtual space can be expanded to the real world by positively utilizing the real world situation such as the place where the user exists and the position / orientation of the real world object in the real space. It is possible to provide an excellent mixed reality display device and method, a storage medium, and a computer program that can be performed.

Further, according to the present invention, it is possible to present the virtual reality world in which the appearance such as the size and color of the real object and the surrounding environment are harmonized without arranging the real object in the real space. It is possible to provide an excellent mixed reality display device and method, a storage medium, and a computer program that can be performed.

Also, according to the present invention, a virtual object imitating a real object can be synthesized and displayed in the real space,
An excellent mixed reality display device and method, a storage medium, and a computer program can be provided.

According to the mixed reality display device of the present invention, the furniture, electric appliances, interiors, etc. in the home are the size of the room based on the mixed reality image in which the virtual object is combined with the scene in the real space. It is possible to judge whether it is suitable or not and whether it is in harmony with the color of the room or other furniture before actually purchasing. In addition, for indoor scenes and objects other than virtual objects such as furniture, electrical appliances, and interiors, the actual images captured by the camera are used as they are.
That is, the augmented reality image can be easily generated without the need to create or obtain a complete model for the physical space.

[Brief description of drawings]

FIG. 1 is a diagram schematically showing a configuration of a mixed reality system according to an embodiment of the present invention.

FIG. 2 is a diagram schematically showing a hardware configuration of a mixed reality apparatus 102 according to the present embodiment.

FIG. 3 is a flowchart showing a mixed reality display processing procedure executed on the mixed reality apparatus 102.

FIG. 4 is a diagram showing one configuration example of an object teaching index 104.

5 is a flowchart showing a processing procedure for extracting an object teaching index when using the object teaching index 104 shown in FIG. 4, and acquiring its identification information and three-dimensional position / orientation information.

FIG. 6 is a diagram showing another configuration example of the object teaching index 104.

FIG. 7 shows the object teaching index 104 as markers 105a ′ to 1
FIG. 10 is a diagram schematically showing operating characteristics on the side of the camera 101 and the mixed reality display device 102 when three light emitting diodes are equipped as 05c ′.

8 is a flowchart showing a processing procedure for extracting the object teaching index 104 when using the object teaching index 104 shown in FIG. 6, and acquiring its identification information and three-dimensional position / orientation information.

9 is a diagram showing a modified example of the object teaching index 104 shown in FIG.

10 is a diagram showing another configuration example of the object teaching index 104. FIG.

11 is a flowchart showing a processing procedure for extracting the object teaching index 104 when the object teaching index 104 shown in FIG. 10 is used, and acquiring its identification information and three-dimensional position / orientation information.

FIG. 12 is a diagram showing a configuration example of a mixed reality image.

FIG. 13 is a diagram showing a configuration example of a mixed reality image.

FIG. 14 is a diagram showing a configuration example of a mixed reality image.

[Explanation of symbols]

1 ... CPU 2 ... Main memory, 3 ... ROM 4 ... Display controller 5: Input device interface 6 ... Network interface 7 ... External device interface 8 ... Bus, 9 ... Camera interface 11 ... Display 12 ... Keyboard, 13 ... Mouse 14 ... Hard disk drive 15 ... Media drive 101 ... Camera 102 ... Mixed reality device 103 ... Image display device 104 ... Object teaching index 105a to 105c ... Markers 105a 'to 105c' ... Marker (light source) 106 ... Light source

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 2F065 AA04 AA37 BB05 BB29 FF04                       JJ03 JJ26 QQ31 UU05                 5B050 AA10 BA07 BA08 BA11 BA13                       BA18 CA07 CA08 DA02 DA04                       DA10 EA06 EA07 EA19 FA02                       FA05 FA08 FA13 FA19 GA04                 5B057 BA02 CA01 CA08 CA12 CA13                       CA16 CA17 CB01 CB08 CB13                       CB16 CC01 CE08 CH07 CH08                       CH11 CH12 CH18 DA02 DA07                       DA16 DB03 DB06 DB09 DC08                       DC09                 5C023 AA03 AA04 AA10 AA11 AA38                       BA02 DA02 DA03 DA08

Claims (18)

[Claims] 1. A mixed reality display device for synthesizing and displaying a virtual object in a real space, comprising: an image input unit for capturing a scene in the real space including an object teaching index; and an input by the image input unit. A position / orientation calculation unit that calculates the spatial position and orientation of the object teaching index based on an image, and a corresponding physical object that recognizes the identification information of the object teaching index based on the image input by the image input unit Of the physical object recognized by the object recognition unit and the physical object recognized by the object recognition unit.
An image synthesizing unit configured to rotate and move a virtual object composed of a dimensional model in accordance with the spatial position and orientation calculated by the position / orientation calculating unit and synthesize the image on an input image. Mixed reality display device. 2. The image synthesizing unit further synthesizes a charge, a specification, or other related information regarding the physical object recognized by the object recognizing unit on an input image. Mixed reality display device. 3. A distance measuring section for measuring a distance to the object teaching index, wherein the image synthesizing section applies hidden surface processing to the virtual object based on the distance obtained by the distance measuring section. The mixed reality display device according to claim 1, wherein: 4. The object teaching index is provided with optical identification information, and the object recognition unit extracts the optical identification information from an input image to acquire the identification information of the object. The mixed reality display device according to claim 1. 5. The object teaching index is provided with a plurality of optical identification information whose positional relationship is known to each other, and the position / orientation calculation unit extracts the optical identification information from the input image. 2. The mixed reality display device according to claim 1, wherein the spatial position and orientation of the optical image are calculated based on the detected position of each optical identification information in the input image. 6. The object teaching index is provided with a plurality of blinking light sources whose positional relationship is known to each other, and each blinking light source encodes identification information or other data into a blinking pattern of light and transmits it. The mixed reality display device according to claim 1, wherein: 7. The object recognition section extracts blinking patterns from each blinking light source on the basis of an input image and decodes them to recognize identification information possessed by the object teaching index.
The mixed reality display device according to claim 6, wherein: 8. The position / orientation calculation unit extracts blinking patterns from each blinking light source based on the input image, decodes these to identify each, and identifies the blinking light source detection position in the input image. The mixed reality display device according to claim 6, wherein the spatial position and orientation thereof are calculated based on the spatial position and orientation. 9. A mixed reality display method for synthesizing and displaying a virtual object in a real space, comprising: an image input step of capturing a scene in the real space including an object teaching index; and an input in the image input step. A position / orientation calculation step of calculating a spatial position and orientation of the object teaching index based on an image, and a real object corresponding to recognition information identified by the object teaching index based on the input image in the image input step Recognizing the object, and rotating / moving a virtual object consisting of a three-dimensional model of the physical object recognized in the object recognizing step according to the spatial position and attitude calculated in the position / orientation calculating step. An image synthesizing step of synthesizing the image on the input image. 10. The image synthesizing step further synthesizes, on an input image, a charge, a specification, or other related information regarding the real object recognized in the object recognizing step. Mixed Reality Display Method. 11. A distance measuring step for measuring a distance to the object teaching index, wherein in the image synthesizing step, hidden surface processing is applied to the virtual object based on the distance obtained in the distance measuring step. The mixed reality display method according to claim 9, wherein: 12. The object teaching index is provided with optical identification information, and in the object recognition step, the optical identification information is extracted from an input image to obtain the identification information of the object. The mixed reality display method according to claim 9. 13. The object teaching index is provided with a plurality of optical identification information whose mutual positional relationship is known, and in the position / orientation calculation step, the optical identification information is extracted from the input image. 10. The mixed reality display method according to claim 9, further comprising calculating the spatial position and orientation of the optical image based on the detected position of each optical identification information in the input image. 14. The object teaching index is provided with a plurality of blinking light sources whose positional relationships are known to each other, and each blinking light source encodes identification information or other data into a blinking pattern of light and transmits it. The mixed reality display method according to claim 9, wherein: 15. The object recognizing step is characterized by extracting blinking patterns from each blinking light source based on an input image and decoding these to recognize identification information possessed by the object teaching index. Item 15. The mixed reality display method according to Item 14. 16. The position / orientation calculating step extracts a blinking pattern from each blinking light source based on an input image, decodes these to identify each, and determines a detection position of each blinking light source in the input image. 15. The mixed reality display method according to claim 14, wherein the spatial position and orientation are calculated based on the spatial position and orientation. 17. A storage medium which physically stores computer software in a computer-readable format, which is written so as to execute a process for synthesizing and displaying a virtual object in a physical space on a computer system. , The computer software includes: an image input step of capturing a scene in a real space including an object teaching index; and a position for calculating a spatial position and orientation of the object teaching index based on the input image in the image input step. An orientation calculation step; an object recognition step of recognizing the corresponding physical object by recognizing the identification information of the object teaching index based on the input image in the image input step; and a physical object recognized in the object recognition step. A virtual object consisting of a three-dimensional model of An image synthesizing step of synthesizing on an input image by rotating / moving in accordance with the spatial position and orientation of the storage medium. 18. A computer program written in a computer-readable format so as to execute a process for synthesizing and displaying a virtual object in a real space on a computer system, the reality including an object teaching index. An image input step of capturing a scene in space, a position / posture calculation step of calculating a spatial position and a posture of the object teaching index based on the input image in the image input step, and an input image in the image input step. An object recognition step of recognizing the identification information of the object teaching index to recognize a corresponding physical object based on the basis, and a virtual object including a three-dimensional model of the physical object recognized in the object recognition step, Rotate / move according to the spatial position and orientation calculated in the calculation step An image synthesizing step of synthesizing, a computer program.