JP2010118002A - Information processing apparatus and information processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To precisely prevent an index displayed on a display screen of a display device present in a real space from being detected wrongly as an index present in the real space. <P>SOLUTION: An index detection unit 1040 acquires an index and image coordinates from an captured image data outputted from an imaging device 1010, and a position and orientation calculation unit 1050 calculates a position and an orientation of the imaging device 1010 using the acquired image coordinates, u<SP>Qkn</SP>, and world coordinates x<SB>W</SB><SP>Qkn</SP>which is retained as known information in advance. Furthermore, a wrong detection prevention processing unit 1060 searches candidate index regions using the calculated position and orientation of the imaging device 1010 and the world coordinates x<SB>W</SB><SP>Qkn</SP>which is retained in advance, detects an index which has not been detected in the index detection unit 1040, and overlays a graphic or a small image on the index portion. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an information processing apparatus, an information processing method, a program, and a storage medium, and more particularly to a technique suitable for use in preventing erroneous detection of an index in an image.

  A technique for measuring the position and orientation (position and orientation) of an imaging unit (hereinafter referred to as a camera) such as a camera that captures a real space is, for example, in a mixed reality system that displays a fusion of a real space and a virtual space. Necessary. Such conventional techniques are disclosed in Patent Document 1, Patent Document 2, and Non-Patent Document 1, for example. Specifically, a position / orientation sensor that measures the position / orientation of the camera by using a known marker that is a position in the real space or using a known feature point that is a position in the real space. The error is corrected. Hereinafter, the marker and the feature point are collectively referred to as an index.

  These methods are methods for estimating the position and orientation of the camera using a position and orientation sensor that measures the position and orientation of the camera and an index captured by the camera. As indices used in such a method, there are a center of gravity of a color region, a concentric circle, and the like.

  In addition, a plurality of indices are often used at the same time, and it is necessary to determine which of the indices arranged in the real space corresponds to the indices detected from the image captured by the camera. As one of the means, the relationship between the coordinates of the index detected from the image and the coordinates of the index on the image plane obtained by projecting based on the measurement value of the position and orientation sensor as the absolute position of the index It is known to use.

  In such a method for estimating the position and orientation of the camera, a small circular sheet-like object having a specific color can be used as an index. In this case, the information included in the index is a three-dimensional position (coordinates) and a color. Using the measured values of the position and orientation sensor, the three-dimensional position of the index is projected onto the image plane of the camera, and on the other hand, color area detection processing is performed to detect the color of the index from the image, and the position of the center of gravity in the image is determined. calculate. Then, the index in the image can be identified by comparing the three-dimensional position projected on the image plane with the barycentric position calculated from the image and determining that the closest one is the same index, for example.

  On the other hand, for example, as disclosed in Non-Patent Documents 2 and 3, there is also known a method for estimating the position and orientation of a camera using only an index imaged by the camera without using a position and orientation sensor. In these Non-Patent Documents 2 and 3, the position and orientation of the camera are estimated based on the coordinates of the four vertices of the square using a square index.

  However, since the square is rotationally symmetric at every 90 ° with the axis passing through the center point (intersection of diagonal lines) perpendicular to the plane as the rotation axis, it is not possible to make an up / down / left / right determination only from the coordinates of the vertex. . Therefore, a further image feature for performing up / down / left / right determination is provided inside the square index. Furthermore, when using a plurality of indices, it is necessary to identify which of the plurality of indices is captured based only on the image captured by the camera. For this reason, graphic information such as a unique pattern or code that is different for each index is embedded in the index.

  In a method that uses an index having a graphic expansion such as a square marker, it is necessary to completely identify each marker from only the image. For this reason, code information and symbol information that can be a template are embedded in the index. FIG. 8 is a diagram illustrating an example of a specific square marker that is an index in which symbol information or the like is embedded.

  Further, the image display device that presents the mixed reality as described above superimposes and draws a virtual space image generated according to the position and orientation of the camera on the real space image captured by the camera. Then, a video see-through method is used in which the superimposed and drawn image is displayed on a display attached to the observer's head. Examples of the virtual space include a virtual object drawn by computer graphics and character information.

  Furthermore, in order to present the scene that the observer is observing to a third party, the real space image captured by the camera, or an image in which the virtual space image is superimposed on the real space image, is displayed in the scene. In some cases, the image is displayed on a display device arranged on the screen.

  In any of the conventional techniques described above, in principle, if there is something other than an index and the same color or figure as the index in the real space photographed by the camera, these are erroneously identified as the index. May end up. For example, when a display device that displays an image of a real space photographed by a camera or an image obtained by superimposing a virtual space image on the image is arranged in a scene, the display screen is photographed by the camera. There was a situation that would occur. Therefore, there is a problem that the index displayed on the display screen arranged in the scene is erroneously detected as the index in the scene.

  Therefore, various proposals have been made to prevent erroneous detection of an index displayed on the display screen of a display device existing in the real space as an index existing in the real space. For example, a method of superimposing and displaying a graphic or small image on the image area of the detected index, or a graphic or small image at the position of the index estimated from the set of world coordinates of the index and information on the position and orientation of the camera Japanese Patent Application Laid-Open No. 2004-228688 discloses a method for displaying in a superimposed manner.

Japanese Patent Laid-Open No. 11-84307 JP 2000-41173 A JP 2007-18426 A A. State, G. Hirota, D. T. Chen, B. Garrett, and M. Livingston: Superior augmented reality registration by integrating landmark tracking and magnetic tracking, Proc. SIGGRAPH'96, pp. 429-438, July 1996. Kato, Billinghurst, Asano, Tachibana: Augmented reality system based on marker tracking and its calibration, Transactions of the Virtual Reality Society of Japan, vol.4, no.4, pp.607-616, Dec.1999. X. Zhang, S. Fronz, N. Navab: Visual marker detection and decoding in AR systems: A comparative study, Proc. Of International Symposium on Mixed and Augmented Reality (ISMAR'02), 2002. RMHaralick, C. Lee, K. Ottenberg, and M. Nolle: Review and analysis of solutions of the three point perspective pose estimation problem, International Journal of Computer Vision, vol.13, no.3, pp.331-356, 1994. D.G.Lowe: Fitting parameterized three-dimensional models to images, IEEE Transactions on PAMI, vol.13, no.5, pp.441-450, 1991. Hirofumi Fujii, Noriyuki Kanbara, Hidehiko Iwasa, Haruo Takemura, Naokazu Yokoya, alignment with stereo camera combined with gyro sensor for augmented reality, IEICE Technical Report PRMU99-192 (Science Tech Vol.99) , no.574, pp.1-8)

  However, in the method described in Patent Document 3, a situation that is not necessarily effective occurs.

  For example, in the method of superimposing and displaying a figure or a small image on the image area of the detected index, when the index can be detected, superimposed display for preventing erroneous recognition of the index is performed on the display device. However, when an index photographed by the camera is not detected for some reason such as an illumination environment, a figure or a small image is not superimposed and displayed. For this reason, there is a problem that, when an environment in which an index can be detected again is obtained, the index displayed on the display device is erroneously detected as the index. In addition, there is a problem that an index that is not detected as an index in a captured image due to a dark scene is detected as an index when displayed on a display device.

  Furthermore, in the method of superimposing and displaying a figure or a small image on the position of the index estimated from the set of world coordinates of the index and the position and orientation information of the camera, part or all of the index is hidden by the real object, Even when shooting is not performed, superimposed display is performed to prevent erroneous recognition of the index. That is, unnecessary superimposed display may be performed. Also, when the camera position and orientation deviates from the original values for some reason, superimposed display is performed on the display device to prevent misrecognition of the index at a location different from the position where the index exists. There is a problem that an index is erroneously detected as an index.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to accurately prevent an indicator displayed on a display screen of a display device existing in a real space from being erroneously detected as an indicator existing in the real space. It is said.

An information processing apparatus according to the present invention includes an image input unit that inputs image data acquired by an imaging unit that captures a scene, and a first index that detects an index included in the scene from the image data input by the image input unit. Detection means, storage means for storing world coordinate information of the index included in the scene, information on world coordinates of the index stored in the storage means, and the index detected by the first detection means Based on information on image coordinates on the image, position and orientation calculation means for calculating the position and orientation of the imaging means, information on world coordinates of the index stored in the storage means, and calculation by the position and orientation calculation means An index included in the scene different from the index detected by the first detection unit based on the information on the position and orientation of the imaging unit that has been performed is used as the image data. Second detection means for detecting the image data, and image generation means for generating an image in which the index detected by the first detection means and the second detection means included in the image data input by the image input means is corrected. And display control means for displaying on the display means the image generated by the image generating means.
Another feature of the information processing apparatus according to the present invention is that an image input unit that inputs image data acquired by a plurality of imaging units that capture a scene, and a plurality of images that are input by the image input unit. First detection means for detecting an index included in the scene from one of the image data, information on world coordinates of the index included in the scene, and positions and orientations between the plurality of imaging means Storage means for storing relationship information, information on world coordinates of indices included in the scene stored in the storage means, information on relationship between position and orientation between the plurality of imaging means, and the first Position and orientation calculation means for calculating the positions and orientations of the plurality of imaging means based on the information of the image coordinates on the image of the index detected by one detection means; and the index stored in the storage means Based on field coordinate information and position and orientation information of the plurality of imaging means calculated by the position and orientation calculation means, the scene is included in the scene different from the index detected by the first detection means. Second detection means for detecting an index from the plurality of image data, and correction of the indices detected by the first detection means and the second detection means included in each image data input by the image input means The image generation means for generating each of the images, and the display control means for displaying each image generated by the image generation means on a plurality of display means corresponding to the images.

An information processing method according to the present invention includes an image input step of inputting image data acquired by an imaging unit that captures a scene, and a first indicator that detects an index included in the scene from the image data input in the image input step. The position and orientation of the image pickup means based on the detection step, the world coordinate information of the index stored in the storage means, and the image coordinate information on the image of the index detected in the first detection step Based on the position and orientation calculation step of calculating the position, the world coordinate information of the index stored in the storage means, and the position and orientation information of the imaging means calculated in the position and orientation calculation step. A second detection step of detecting from the image data an index included in the scene different from the index detected in the detection step, and input in the image input step An image generation step for generating an image in which the index detected in the first detection step and the second detection step included in the image data is corrected, and a display for displaying the image generated in the image generation step on a display means And a control step.
Another feature of the information processing method according to the present invention is that an image input step for inputting respective image data acquired by a plurality of imaging means for shooting a scene, and a plurality of images input in the image input step. A first detection step of detecting an index included in the scene from one of the image data, information of world coordinates of the index included in the scene stored in a storage unit, and a plurality of imaging units Position and orientation calculation for calculating the positions and orientations of the plurality of imaging means based on the information on the relationship between the positions and orientations between them and the information on the image coordinates on the image of the index detected in the first detection step Based on the step, the world coordinate information of the index stored in the storage means, and the position and orientation information of the plurality of imaging means calculated in the position and orientation calculation step. A second detection step of detecting from the plurality of image data an indicator included in the scene different from the indicator detected in the output step; and the first detection step included in each image data input in the image input step. An image generation step for generating an image in which the index detected in the detection step and the second detection step is corrected, and each image generated in the image generation step is displayed on a plurality of display means corresponding to the images. And a display control step.

The program according to the present invention includes an image input step for inputting image data acquired by an imaging means for photographing a scene, and a first detection for detecting an index included in the scene from the image data input in the image input step. Calculating the position and orientation of the imaging means based on the step, the information on the world coordinates of the index stored in the storage means, and the information on the image coordinates on the image of the index detected in the first detection step The first detection based on the position and orientation calculation step, the information on the world coordinates of the index stored in the storage means, and the position and orientation information of the imaging means calculated in the position and orientation calculation step A second detection step for detecting an index included in the scene different from the index detected in the step from the image data; and an image input in the image input step. An image generation step for generating an image in which the index detected in the first detection step and the second detection step included in the data is corrected, and display control for displaying the image generated in the image generation step on a display means The process is executed by a computer.
Another feature of the program of the present invention is that an image input step for inputting respective image data acquired by a plurality of image pickup means for shooting a scene, and a plurality of image data input in the image input step. A first detection step of detecting an index included in the scene from one of the information, information on world coordinates of the index included in the scene stored in a storage unit, a position between the plurality of imaging units, and A position and orientation calculation step of calculating the positions and orientations of the plurality of imaging means based on the information on the relationship of the posture and the information of the image coordinates on the image of the index detected in the first detection step; Based on the information on the world coordinates of the index stored in the storage means and the information on the positions and orientations of the plurality of imaging means calculated in the position and orientation calculation step, the first detection step A second detection step of detecting, from the plurality of image data, an index included in the scene that is different from the index detected in the first detection, and the first detection included in each image data input in the image input step An image generation step for generating an image obtained by correcting the index detected in the step and the second detection step, and a display for displaying the images generated in the image generation step on a plurality of display means corresponding to the images. The control process is executed by a computer.

  A storage medium according to the present invention stores any one of the programs described above.

  According to the present invention, it is possible to accurately detect that an index displayed on the display device is detected as an index existing in the real space even when the index cannot be directly detected or the position and orientation of the imaging device are shifted. Can be prevented.

(First embodiment)
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The information processing apparatus according to the present embodiment prevents erroneous detection of an index by locally superimposing and displaying a graphic or a small image on an index area and an index candidate area detected from an image. The index candidate area is not treated as an index used for calculating the position and orientation of the imaging apparatus, but refers to an area having features close to the index. Hereinafter, an information processing apparatus and an information processing method according to the present embodiment will be described.

FIG. 1 is a block diagram illustrating a configuration example of the information processing apparatus 1000 according to the present embodiment.
As shown in FIG. 1, the information processing apparatus 1000 according to the present embodiment includes an image input unit 1020, a data storage unit 1030, an index detection unit 1040, a position / orientation calculation unit 1050, an erroneous detection prevention processing unit 1060, an image generation unit 1070, And an image output unit 1080. The information processing apparatus 1000 is connected to the imaging apparatus 1010 and the display apparatus 1090.

A plurality of indices Q _k (k = 1, 2,..., K _Q ) are arranged at a plurality of positions in the real space as indices for photographing with the imaging device 1010. Note that the position information of the plurality of indices Q _k is known. In the position information, the position x _W ^Qk is defined by a coordinate system (world coordinate system) in which one point in the real space is defined as the origin, and three orthogonal axes are defined as the X axis, the Y axis, and the Z axis, respectively. Yes.

In the present embodiment, it is desirable that the index Q _k is set so that at least three or more indices are always observed on the captured image acquired by the imaging device 1010. In the example shown in FIG. 1, four indexes Q ₁ , Q ₂ , Q ₃ , and Q ₄ are set, and three of the indexes Q ₁ , Q ₃ , and Q ₄ are within the field of view of the imaging device 1010. Indicates an included situation.

Note that the index Q _k may be configured by, for example, circular markers having different colors, or may be configured by feature points such as natural features having different texture characteristics. Further, a square index formed by a rectangular single color area having a certain area may be used. Any form may be used as long as the coordinates of the image projected on the captured image can be detected and the index can be identified by any method. Furthermore, the indicator may be intentionally set, or may be a characteristic originally existing in the scene that is not intentionally set.

Hereinafter, each configuration of the information processing apparatus 1000 in the present embodiment will be described.
When image data output from the imaging device 1010 (hereinafter referred to as captured image data) is input to the image input unit 1020, the image input unit 1020 converts the captured image data into digital data, and a data storage unit 1030. Save to.

  The index detection unit 1040 functions as a first detection unit, reads captured image data from the data storage unit 1030, and detects image coordinates of the index based on the subjective viewpoint included in the image. For example, when each index is composed of markers of different colors, an area corresponding to each marker color is detected from the image, and the position of the center of gravity is used as the image coordinates of the index. When each index is composed of feature points having different textures, template matching is performed on the image using a template image of each index that is held in advance as known information. Thereby, the position of the index is detected.

  In addition, when a quadratic index is used, labeling is performed after binarizing the image data, and an area formed by four straight lines is detected as an index candidate area. Further, by detecting whether or not there is a specific pattern in the index candidate area, false detection is prevented and an index identifier is acquired. In this embodiment, the quadrangular indices detected in this way are four indices formed by four vertices.

The index detection unit 1040 further outputs information about the image coordinates of the detected index and the identifier of the index to the data storage unit 1030. Hereinafter, an index detected on the captured image is denoted as Q _kn using an identifier n (n = 1, 2,..., N) attached to each detected index. Here, N represents the number of indices detected on the captured image. Further, the image coordinate of the detected index Q _kn is expressed as u ^Qkn . For example, in the example shown in FIG. 1, N = 3, index identifiers k ₁ = 1, k ₂ = 3, k ₃ = 4 and corresponding image coordinates u ^Qk1 , u ^Qk2 , u Information about ^Qk3 is output from the index detection unit 1040.

The position / orientation calculation unit 1050 reads, from the data storage unit 1030, information on the image coordinates u ^Qkn of each index detected by the index detection unit 1040 and world coordinates x _W ^Qkn held in advance as known information. Based on these pieces of information, the position and orientation of the imaging device 1010 are calculated. Calculated position and orientation information, for example, is output and the three-dimensional vector x _W ^C representing the position, the data storage unit 1030 as information of the combination of the 3 × 3 matrix R _WC representing the orientation.

  Note that a method for calculating the position and orientation of an imaging device from a set of world coordinates and image coordinates of an index has been proposed in the field of photogrammetry and the like (see, for example, Non-Patent Document 4 and Non-Patent Document 5). Therefore, the detailed description is abbreviate | omitted.

The erroneous detection prevention processing unit 1060 locally superimposes a graphic or a small image on the index position of the captured image detected by the index detection unit 1040 when the display device 1090 is imaged by the imaging device 1010. This is performed in order to prevent the index (for example, Q ₁ ′, Q ₃ ′, Q ₄ ′ shown in FIG. 1) displayed on the display device 1090 from being detected by the index detection unit 1040.

  Further, the erroneous detection prevention processing unit 1060 functions as a second detection unit, and performs an area search for an index candidate area for an index that is not detected by the index detection unit 1040. If the index candidate area exists, a graphic or small image is locally superimposed on the index candidate area. Here, the small image refers to an image having a size smaller than the captured image. The superimposed image may be another figure such as a three-dimensional virtual object.

  The image generation unit 1070 generates display image data to be displayed on the display device 1090. The image output unit 1080 functions as a display control unit, converts display image data generated by the image generation unit 1070 into data for output to the display device 1090, and outputs the data to the display device 1090.

The data storage unit 1030 stores the captured image data output from the image input unit 1020 and the image coordinates and identifier information of each index output from the index detection unit 1040. In addition, information on the world coordinates x _W ^Qk of the index Q _k as known information, and data such as camera parameters of the imaging device 1010 are also stored. And these data are output as needed.

  Note that each configuration of the image input unit 1020, the data storage unit 1030, the index detection unit 1040, the position / orientation calculation unit 1050, the false detection prevention processing unit 1060, the image generation unit 1070, and the image output unit 1080 shown in FIG. May be treated as an independent device. Alternatively, the function may be realized by installing the software on one or a plurality of computers and executing the software by the CPU of each computer. In this embodiment, the image input unit 1020, the data storage unit 1030, the index detection unit 1040, the position / orientation calculation unit 1050, the erroneous detection prevention processing unit 1060, the image generation unit 1070, and the image output unit 1080 are combined into one computer. It is treated as software to be executed in

FIG. 2 is a diagram illustrating a basic configuration example of the information processing apparatus 1000 that executes each of the image input unit 1020 to the image output unit 1080 in FIG. 1 as software.
In FIG. 2, reference numeral 2001 denotes a CPU, which controls the entire information processing apparatus 1000 using programs and data stored in a RAM 2002 and a ROM 2003. Furthermore, the execution of software of each of the image input unit 1020, the data storage unit 1030, the index detection unit 1040, the position / orientation calculation unit 1050, the erroneous detection prevention processing unit 1060, the image generation unit 1070, and the image output unit 1080 is controlled. Realize the function of each part.

  A RAM 2002 includes an area for temporarily storing programs and data read from the external storage device 2007 and the storage medium drive 2008, and also includes a work area necessary for the CPU 2001 to perform various processes. . In the present embodiment, the function of the data storage unit 1030 is realized by the RAM 2002.

  Reference numeral 2003 denotes a ROM that stores a storage program, setting data, and the like of the information processing apparatus 1000. Reference numeral 2004 denotes a keyboard, and 2005 denotes a mouse. An operator can input various instructions to the CPU 2001 using the keyboard 2004 and the mouse 2005.

  Reference numeral 2006 denotes a display unit, which includes a CRT, a liquid crystal screen, and the like. For example, a message to be displayed for measuring the position and orientation of the imaging apparatus 1010 can be displayed.

  An external storage device 2007 is a device that functions as a large-capacity information storage device such as a hard disk. The external storage device 2007 stores an OS (Operating System), software programs, and the like. In the present embodiment, other necessary information is stored in the external storage device 2007 and is read into the RAM 2002 as necessary.

  Reference numeral 2008 denotes a storage medium drive that reads programs and data stored in a storage medium such as a CD-ROM or DVD-ROM in accordance with instructions from the CPU 2001 and outputs them to the RAM 2002 or the external storage device 2007.

  An I / F 2009 includes an analog video port for connecting the imaging apparatus 1010, a digital input / output port such as IEEE1394, and an Ethernet (registered trademark) port for outputting information related to the identified index to the outside. Has been. The input data is taken into the RAM 2002 via the I / F 2009. A part of the function of the image input unit 1020 is realized by the I / F 2009. 2010 is a bus that connects the components shown in FIG.

  FIG. 3 is a flowchart illustrating an example of a processing procedure for preventing erroneous detection of an index displayed on the display device 1090 in the present embodiment. Each process shown in FIG. 3 is performed by the CPU 2001 executing a software program of the erroneous detection prevention processing unit 1060. Note that it is assumed that the program code according to the flowchart of FIG. In FIG. 3, a processing procedure for superimposing a graphic or a small image on the index portion and the index candidate area in the display image displayed on the display device 1090 by the erroneous detection prevention processing unit 1060 will be described.

First, in step S3010, the erroneous detection prevention processing unit 1060 reads out information on the position and orientation of the imaging device 1010 calculated by the position and orientation calculation unit 1050 from the data storage unit 1030. Further, the data storage unit 1030 stores the information of the index Q _k and its world coordinate x _W ^Qk that are held in advance as known information, and the information of the index Q _kn and its image coordinate u ^Qkn detected by the index detection unit 1040. Read from.

Next, in step S3020, the erroneous detection prevention processing unit 1060 determines k = 1 as an initial setting. In step S3030, the erroneous detection prevention processing unit 1060 determines whether or not the index Q _k exists in the index Q _kn , that is, whether or not the index Q _k is detected. If the index Q _k is detected as a result of the determination, the process proceeds to step S3040. If the index Q _k is not detected, the process proceeds to step S3050.

In step S3040, the erroneous detection prevention processing unit 1060 locally superimposes a graphic or a small image on the position of the image coordinate u ^Qkn on the captured image on which the index detection process has been performed by the index detection unit 1040. Then, the process proceeds to step S3090.

For example, when each index is composed of markers having different colors, a color figure (for example, a circle) that is not registered as a marker color is superimposed on the position of the image coordinate u ^Qkn to cover the marker. . When each index is composed of feature points having different textures, a figure or a small image is superimposed so as to cover a part of the texture. In addition, when a quadrangle index is used, a graphic or a small image is superimposed so that it is not recognized as a quadrangle index.

  FIG. 4 is a diagram illustrating an example of a case where a rectangle 410 is superimposed and displayed so as to cover a part of the specific pattern in the index region 400 of the rectangle index. In the example shown in FIG. 4, the image feature for determining the up, down, left, and right provided inside the quadrangular index is obscured.

On the other hand, in step S3050, the erroneous detection prevention processing unit 1060 performs perspective projection conversion on the world coordinates x _W ^Qk of the index Q _k based on the position and orientation of the imaging device 1010 to determine the index search area. As the search area, for example, the size or shape (rectangle, circle, etc.) of a predetermined area centered on the estimated position of the index may be used as the search area. Further, the size of the search area is set to a size obtained by increasing a certain amount in the size of the index photographed from the imaging device 1010 estimated from the result of perspective projection transformation of the world coordinates x _W ^Qk of the index Q _k . The search area may be determined based on the estimated position information of the index. Further, the index search region may be obtained using any other method.

  Next, in step S3060, the erroneous detection prevention processing unit 1060 searches for an index candidate region within the search region determined in the processing of step S3050. For example, in the case of a quadrangular index, the index candidate region is searched by a method such as extracting an edge in the search region and setting a region surrounded by edges on three or more sides as the index candidate region.

  In step S3070, the erroneous detection prevention processing unit 1060 determines whether or not an index candidate region exists by the processing in step S3060. If the index candidate area exists as a result of this determination, the process proceeds to step S3080, and if the index candidate area does not exist, the process proceeds to step S3090.

  Next, in step S3080, the erroneous detection prevention processing unit 1060 locally superimposes a graphic or a small image on the index candidate region searched in step S3060.

FIG. 5 is a diagram illustrating an example in which a graphic is superimposed and displayed so as to obscure the specific pattern of the index candidate area of the rectangular index.
In FIG. 5A, a rectangular index candidate area 500 is treated as an index candidate area because a part of the index is hidden by the real object 510. Then, by superimposing the quadrangle 520, the image feature for discrimination between up, down, left, and right provided inside the quadrangle index is covered and corrected. Further, in FIG. 5B, the index candidate area 530 for the square index is treated as an index candidate area because only a part of the square index is shown in the captured image. Then, by superimposing the quadrangle 540, the image feature for discrimination between up, down, left and right provided inside the quadrangle index is covered and corrected.

Next, in step S3090, the false detection prevention processing unit 1060 increments the numerical value of k by one. Then, in step S3100, the erroneous-detection prevention processing section 1060, k determines whether less than K _Q. As a result of the determination, if k is equal to or less than K _Q , the process returns to step S3030. If k is greater than K _Q , the process proceeds to step S3110.

  In step S <b> 3110, the erroneous detection prevention processing unit 1060 outputs image data in which a figure or a small image is superimposed on the captured image to the data storage unit 1030. An image in which a figure or a small image is locally superimposed is displayed as data on the display device 1090 via the image generation unit 1070 and the image output unit 1080.

  With the above processing, an image in which a graphic or a small image is locally superimposed on the index region and the index candidate region is displayed on the display device 1090. Therefore, even if the display image is captured by the imaging device 1010, the display image The indicator displayed above is not detected as an indicator. In the present embodiment, the position and orientation of the imaging apparatus 1010 may be measured by attaching a position and orientation sensor to the imaging apparatus 1010. In this case, the position and orientation of the imaging device 1010 can be stably estimated as compared to the case where image information is used.

  In the present embodiment, an inertial sensor may be attached to the imaging apparatus 1010, and the position and orientation of the imaging apparatus 1010 may be calculated by a method disclosed in Non-Patent Document 6, for example. In this case, it is only necessary that at least two or more indices are always observed on the captured image acquired by the imaging device 1010. Also in this case, the position and orientation of the imaging device 1010 can be stably estimated as compared with the case where only image information is used.

  As described above, according to the present embodiment, the position and orientation of the imaging apparatus are calculated based on the world coordinates of the index and the image coordinates of the detected index on the image. Further, based on the calculated position and orientation of the imaging apparatus, an index that could not be detected is detected as an index candidate area, and a graphic or small image is superimposed and displayed. As a result, even when the index cannot be directly detected or when the position and orientation of the imaging device are deviated, it is possible to accurately prevent the index displayed on the display device from being detected as an index existing in the real space. Can do.

(Second Embodiment)
In the first embodiment, there is one imaging device and one display device. However, when the relationship between the relative positions and orientations of the imaging devices is known, the display device can be used even if there are a plurality of imaging devices. May have a plurality of units. Hereinafter, an information processing apparatus and an information processing method according to the present embodiment will be described.

FIG. 6 is a block diagram illustrating a configuration example of the information processing apparatus 6000 according to the present embodiment. The same components as those in FIG. 1 are given the same numbers and symbols, and the description thereof is omitted.
As shown in FIG. 6, the information processing apparatus 6000 in the present embodiment includes an image input unit 6020, a data storage unit 1030, an index detection unit 1040, a position / orientation calculation unit 6030, an erroneous detection prevention processing unit 6040, an image generation unit 6050, And an image output unit 6060. In addition, the information processing device 6000 is connected to the first imaging device 6010, the second imaging device 6011, the first display device 6070, and the second display device 6071. Note that the first imaging device 6010 is the same as the imaging device 1010 in FIG. 1, and the first display device 6070 is the same as the display device 1090 in FIG.

A plurality of indices Q _k (k = 1, 2,..., K _Q ) are arranged at a plurality of positions in the real space as indices for photographing with the first imaging device 6010. Note that the position information of the plurality of indices Q _k is known, and the position x _W ^Qk is defined by the world coordinate system.

In the present embodiment, it is desirable that the index Q _k is set so that at least three or more indices are always observed on the captured image acquired by the first imaging device 6010. In the example shown in FIG. 6, four indexes Q ₁ , Q ₂ , Q ₃ , and Q ₄ are set, and three of the indexes Q ₁ , Q ₃ , and Q ₄ are within the field of view of the imaging device 1010. Indicates an included situation.

Hereinafter, each configuration of the information processing apparatus 6000 in the present embodiment will be described.
When captured image data output from the first imaging device 6010 and the second imaging device 6011 is input to the image input unit 6020, the image input unit 6020 converts the captured image data into digital data, and stores the data. Stored in the unit 1030.

The position / orientation calculation unit 6030 reads from the data storage unit 1030 information on the image coordinates u ^Qkn of each index detected by the index detection unit 1040 and the world coordinates x _W ^Qkn previously stored as known information. Based on these pieces of information, the position and orientation of the first imaging device 6010 are calculated.

Further, information on the relative positions and orientations of the first imaging device 6010 and the second imaging device 6011 is read from the data storage unit 1030, and based on these information, the relative information of the second imaging device 6011 is read. Calculate the position and orientation. Calculated information of the position and orientation, for example, is output to the data storage unit 1030 as information of the combination of the 3 × 3 matrix R _WC representing a three-dimensional vector x _W ^C and attitude representing the position.

The erroneous detection prevention processing unit 6040 is detected by the index detection unit 1040 when the first display device 6070 and the second display device 6071 are photographed by the first imaging device 6010 and the second imaging device 6011. A graphic or small image is locally superimposed on the position of the index. This is because the index displayed on the first display device 6070 (for example, Q ₁ ′, Q ₃ ′, Q ₄ ′ in FIG. 6) and the index displayed on the second display device 6071 (for example, FIG. 6). Q ₁ ″, Q ₃ ″, Q ₄ ″) is performed in order to prevent detection by the index detection unit 1040.

  Further, the erroneous detection prevention processing unit 6040 further determines whether or not there is an index candidate region for the index that is not detected by the index detection unit 1040 for the captured image output from the first imaging device 6010. Perform area search. In addition, for the captured image output from the second imaging device 6011, an area search is performed to determine whether or not an index candidate area exists for all indices. If the index candidate area exists, a graphic or small image is locally superimposed on the index candidate area. Here, the small image refers to an image having a size smaller than the captured image. The superimposed image may be another figure such as a three-dimensional virtual object.

  The image generation unit 6050 generates display image data to be displayed on the first display device 6070 and the second display device 6071. The image output unit 6060 converts the display image data generated by the image generation unit 6050 into data to be output to the first display device 6070 or the second display device 6071, and the first display device 6070 or the second display device 6070. To the display device 6071.

  Note that each configuration of the image input unit 6020, the data storage unit 1030, the index detection unit 1040, the position / orientation calculation unit 6030, the erroneous detection prevention processing unit 6040, the image generation unit 6050, and the image output unit 6060 illustrated in FIG. May be treated as an independent device. Alternatively, the function may be realized by installing the software on one or a plurality of computers and executing the software by the CPU of each computer. In the present embodiment, the image input unit 6020, the data storage unit 1030, the index detection unit 1040, the position / orientation calculation unit 6030, the false detection prevention processing unit 6040, the image generation unit 6050, and the image output unit 6060 are included in one computer. It is treated as software to be executed in Further, the hardware configuration of this computer is the same as the configuration shown in FIG.

  FIG. 7 is a flowchart illustrating an example of a processing procedure for preventing erroneous detection of an index displayed on the first display device 6070 and the second display device 6071 in the present embodiment. Each process illustrated in FIG. 7 is performed by the CPU 2001 executing a software program of the erroneous detection prevention processing unit 6040. Note that it is assumed that the program code according to the flowchart of FIG.

First, in step S7010, the erroneous detection prevention processing unit 6040 stores information on the position and orientation of the first imaging device 6010 calculated by the position and orientation calculation unit 6030 and the position and orientation of the second imaging device 6011 as a data storage unit. Read from 1030. Further, the data storage unit 1030 stores the information of the index Q _k and its world coordinate x _W ^Qk that are held in advance as known information, and the information of the index Q _kn and its image coordinate u ^Qkn detected by the index detection unit 1040. Read from.

Next, in step S7020, the erroneous detection prevention processing unit 6040 determines k = 1 as an initial setting. Then, in step S7030, the detection prevention processing section 6040 erroneously, if there is an index Q _k in the index Q _kn, i.e. whether the index Q _k are detected. If the index Q _k is detected as a result of the determination, the process proceeds to step S7040, and if the index Q _k is not detected, the process proceeds to step S7050.

In step S7040, the erroneous detection prevention processing unit 6040 locally ^applies a graphic or small image to the position of the image coordinate u ^Qkn on the captured image of the first imaging device 6010 that has undergone the index detection processing by the index detection unit 1040. Superimpose on. Then, the process proceeds to step S7090.

On the other hand, in step S7050, the erroneous detection prevention processing unit 6040 performs perspective projection conversion of the world coordinates x _W ^Qk of the index Q _k based on the position and orientation of the first imaging device 6010, and the first imaging device 6010. The search area of the index in the captured image is determined. In step S7060, the erroneous detection prevention processing unit 6040 searches for the index candidate area within the search area determined in the process of step S7050.

  Next, in step S7070, the erroneous detection prevention processing unit 6040 determines whether or not an index candidate region exists by the processing in step S7060. If the index candidate area exists as a result of this determination, the process proceeds to step S7080, and if the index candidate area does not exist, the process proceeds to step S7090.

  Next, in step S7080, the erroneous detection prevention processing unit 6040 locally superimposes a graphic or a small image on the index candidate region searched in step S7060. In step S7090, the erroneous detection prevention processing unit 6040 increments the numerical value of k by 1.

Then, in step S7100, the erroneous-detection prevention processing section 6040, k is equal to or less than K _Q. As a result of the determination, if k is equal to or less than K _Q , the process returns to step S7030, and if k is greater than K _Q , the process proceeds to step S7110. In step S7110, the erroneous detection prevention processing unit 6040 sets k = 1 again as an initial setting.

Next, in step S7120, the erroneous detection prevention processing unit 6040 performs perspective projection conversion of the world coordinates x _W ^Qk of the index Q _k based on the position and orientation of the second imaging device 6011, and the second imaging device 6011. The search area of the index in the captured image is determined. In step S7130, the erroneous detection prevention processing unit 6040 searches for an index candidate area within the search area determined in the process of step S7120.

  Next, in step S7140, the misdetection prevention processing unit 6040 determines whether or not an index candidate region exists by the process of step S7130. If the index candidate area exists as a result of this determination, the process proceeds to step S7150, and if the index candidate area does not exist, the process proceeds to step S7160.

  Next, in step S7150, the erroneous detection prevention processing unit 6040 locally superimposes a graphic or a small image on the index candidate region searched in the process of step S7130. In step S7160, the erroneous detection prevention processing unit 6040 increments the numerical value of k by one.

Next, in step S7170, the erroneous-detection prevention processing section 6040, k is equal to or less than K _Q. As a result of this determination, if k is equal to or less than K _Q , the process returns to step S7120, and if k is greater than K _Q , the process proceeds to step S7180.

  In step S7180, the erroneous detection prevention processing unit 6040 superimposes a captured image output from the first imaging device 6010 and a captured image output from the second imaging device 6011 with image data or a superposed image data. Is output to the data storage unit 1030. An image in which a graphic or a small image is locally superimposed is displayed on a first display device 6070 and a second display device 6071 via an image generation unit 1070 and an image output unit 1080, respectively. At this time, the image data corresponding to the captured image captured by the first imaging device 6010 is displayed on the first display device 6070, and the image data corresponding to the captured image captured by the second imaging device 6011 is Are displayed on the second display device 6071.

  With the above processing, an image in which a graphic or a small image is locally superimposed on the index region and the index candidate region is displayed. Therefore, even if the display image is captured by the first imaging device 6010, the display image Are not detected as indices in the real space.

  As described above, according to the present embodiment, even when a plurality of imaging devices and display devices are connected, the indicators displayed on the display device exist in the real space, as in the first embodiment. It is possible to prevent the detection as an index to be performed with high accuracy. That is, even when the index cannot be detected or when the position and orientation of the imaging device are deviated, it is possible to accurately prevent the index displayed on the display device from being detected as an index existing in the real space. .

(Other embodiments according to the present invention)
The first and second embodiments described above can also be applied to a system that presents a mixed reality using a video see-through head mounted display. In this case, for example, the imaging unit of the head mounted display corresponds to, for example, the imaging device 1010 illustrated in FIG. Then, the position and orientation of the head mounted display are obtained using the index, a virtual image corresponding to the obtained position and orientation is generated, and the virtual image is combined with the captured image captured by the imaging unit. Thereby, mixed reality can be presented to the user.

  Furthermore, by displaying an image corresponding to the image displayed on the head mounted display on, for example, a display device 1090 as shown in FIG. 1, for audiences other than the user who is experiencing mixed reality, The situation that the user is experiencing can be presented. Furthermore, in such a system, it is possible to prevent the accuracy of mixed reality from being lowered by erroneously detecting the index displayed in the display image as an index existing in the real space.

  Each unit constituting the information processing apparatus and each step of the information processing method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or a ROM of a computer. This program and a computer-readable storage medium storing the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  In the present invention, a software program for realizing the functions of the above-described embodiments (in the embodiment, a program corresponding to the flowcharts shown in FIGS. 3 and 7) may be supplied directly or remotely to a system or apparatus. Including. This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the storage medium for supplying the program include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. It can also be supplied by downloading the computer program itself of the present invention or a compressed file including an automatic installation function from a homepage to a storage medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  As another method, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and encrypted from a homepage via the Internet to users who have cleared predetermined conditions. Download the key information to be solved. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  As another method, a program read from a storage medium is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

It is a block diagram which shows the function structural example of the information processing apparatus in the 1st Embodiment of this invention. It is a block diagram which shows the hardware structural example of the information processing apparatus in the 1st Embodiment of this invention. 5 is a flowchart illustrating an example of a processing procedure for preventing erroneous detection of an index displayed on a display device in the first embodiment of the present invention. FIG. 10 is a diagram illustrating an example of a case where a quadrangle is superimposed and displayed so as to cover a part of a specific pattern in the index region of the quadrangle index in the first embodiment of the present invention. In the 1st Embodiment of this invention, it is a figure which shows an example at the time of displaying a figure superimposed so that the specific pattern of the parameter | index candidate area | region of a square parameter | index may be covered. It is a block diagram which shows the function structural example of the information processing apparatus in the 2nd Embodiment of this invention. 10 is a flowchart illustrating an example of a processing procedure for preventing erroneous detection of an index displayed on the first display device and the second display device in the second embodiment of the present invention. It is a figure which shows the example of the concrete square marker which is the parameter | index with which symbol information etc. were embedded.

Explanation of symbols

1000 Information processing apparatus 1010 Imaging apparatus 1020 Image input unit 1030 Data storage unit 1040 Index detection unit 1050 Position and orientation calculation unit 1060 False detection prevention processing unit 1070 Image generation unit 1080 Image output unit 1090 Display device 2001 CPU
2002 RAM
2003 ROM
2004 keyboard 2005 mouse 2006 display unit 2007 external storage device 2008 storage medium drive 2009 I / F
2010 Bus

Claims (11)

Image input means for inputting image data acquired by an imaging means for photographing a scene;
First detection means for detecting an index included in the scene from image data input by the image input means;
Storage means for storing information of world coordinates of an index included in the scene;
A position for calculating the position and orientation of the imaging means based on the information on the world coordinates of the index stored in the storage means and the information on the image coordinates on the image of the index detected by the first detection means Attitude calculation means;
The index detected by the first detection means based on the information on the world coordinates of the index stored in the storage means and the information on the position and orientation of the imaging means calculated by the position and orientation calculation means; A second detecting means for detecting an index included in the different scenes from the image data;
Image generation means for generating an image in which the index detected by the first detection means and the second detection means included in the image data input by the image input means is corrected;
An information processing apparatus comprising: a display control unit configured to display an image generated by the image generation unit on a display unit. Image input means for inputting respective image data acquired by a plurality of imaging means for capturing a scene;
First detection means for detecting an index included in the scene from one of a plurality of image data input by the image input means;
Storage means for storing world coordinate information of an index included in the scene, and positional and orientation information between the plurality of imaging means;
Information on the world coordinates of the index included in the scene stored in the storage unit, information on the relationship between the position and orientation between the plurality of imaging units, and an image of the index detected by the first detection unit Position and orientation calculation means for calculating the positions and orientations of the plurality of imaging means based on the information on the image coordinates above;
Based on the world coordinate information of the index stored in the storage means and the position and orientation information of the plurality of imaging means calculated by the position / orientation calculation means, detected by the first detection means. Second detection means for detecting an index included in the scene different from the index from the plurality of image data;
Image generation means for generating an image obtained by correcting the index detected by the first detection means and the second detection means included in each image data input by the image input means;
An information processing apparatus comprising: a display control unit configured to display each image generated by the image generation unit on a plurality of display units corresponding to the images.   The second detection unit sets an index search region based on the position and orientation of the imaging unit calculated by the position / orientation calculation unit, and detects an index included in the scene in the set search region. The information processing apparatus according to claim 1 or 2.   4. The information processing according to claim 1, wherein, when the index is a quadratic index, the image generating unit generates an image in which a feature inside the quadrangular index is corrected. 5. apparatus. An image input step of inputting image data acquired by an imaging means for capturing a scene;
A first detection step of detecting an index included in the scene from the image data input in the image input step;
Position and orientation for calculating the position and orientation of the imaging means based on information on the world coordinates of the index stored in the storage means and information on image coordinates on the image of the index detected in the first detection step A calculation process;
Based on the world coordinate information of the index stored in the storage unit and the position and orientation information of the imaging unit calculated in the position and orientation calculation step, the index detected in the first detection step A second detection step of detecting an index included in the different scenes from the image data;
An image generation step for generating an image in which the index detected in the first detection step and the second detection step included in the image data input in the image input step is corrected;
A display control step of displaying on the display means the image generated in the image generation step. An image input step of inputting respective image data acquired by a plurality of imaging means for capturing a scene;
A first detection step of detecting an index included in the scene from one of a plurality of image data input in the image input step;
On the image of the index detected in the first detection step, information on the world coordinates of the index included in the scene stored in the storage unit, information on the relationship between the position and orientation between the plurality of imaging units A position and orientation calculation step of calculating the position and orientation of the plurality of imaging means based on the image coordinate information in
Detected in the first detection step based on the world coordinate information of the index stored in the storage unit and the position and orientation information of the plurality of imaging units calculated in the position and orientation calculation step. A second detection step of detecting an index included in the scene different from the index from the plurality of image data;
An image generation step for generating images obtained by correcting the indices detected in the first detection step and the second detection step included in each image data input in the image input step;
An information processing method comprising: a display control step of displaying each image generated in the image generation step on a plurality of display means corresponding to the images.   In the second detection step, an index search region is set based on the position and orientation of the imaging unit calculated in the position and orientation calculation step, and an index included in the scene is detected in the set search region. The information processing method according to claim 5 or 6, wherein:   The information according to any one of claims 5 to 7, wherein, in the image generation step, when the index is a square index, an image in which a feature inside the square index is corrected is generated. Processing method. An image input step of inputting image data acquired by an imaging means for capturing a scene;
A first detection step of detecting an index included in the scene from the image data input in the image input step;
Position and orientation for calculating the position and orientation of the imaging means based on information on the world coordinates of the index stored in the storage means and information on image coordinates on the image of the index detected in the first detection step A calculation process;
Based on the world coordinate information of the index stored in the storage unit and the position and orientation information of the imaging unit calculated in the position and orientation calculation step, the index detected in the first detection step A second detection step of detecting an index included in the different scenes from the image data;
An image generation step for generating an image in which the index detected in the first detection step and the second detection step included in the image data input in the image input step is corrected;
A program causing a computer to execute a display control step of displaying an image generated in the image generation step on a display unit. An image input step of inputting respective image data acquired by a plurality of imaging means for capturing a scene;
A first detection step of detecting an index included in the scene from one of a plurality of image data input in the image input step;
On the image of the index detected in the first detection step, information on the world coordinates of the index included in the scene stored in the storage unit, information on the relationship between the position and orientation between the plurality of imaging units A position and orientation calculation step of calculating the position and orientation of the plurality of imaging means based on the image coordinate information in
Detected in the first detection step based on the world coordinate information of the index stored in the storage unit and the position and orientation information of the plurality of imaging units calculated in the position and orientation calculation step. A second detection step of detecting an index included in the scene different from the index from the plurality of image data;
An image generation step for generating images obtained by correcting the indices detected in the first detection step and the second detection step included in each image data input in the image input step;
A program that causes a computer to execute a display control step of displaying each image generated in the image generation step on a plurality of display means corresponding to the images.   A computer-readable storage medium storing the program according to claim 9 or 10.

Images