JP2005234757A - Extended reality sense system, image composition display method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent that a marker appears to protrude from a display virtual object even when the display virtual object is small relatively to the marker. <P>SOLUTION: A marker concealing data storage part 70 stores three-dimensional information about a preset marker arrangement position as marker concealing data. When an image composition part 50 composes an virtual object image in a marker position inside a photography image, the image composition part 50 composes the virtual object image on an image wherein a marker concealing image stored in the marker concealing data storage part 70 is composed in the marker arrangement position of the photography image to generate a final composition image. Because composing the virtual object image 102 after concealing the marker 100 by the marker concealing image 103, it can be prevented that the marker 100 is viewed from the side of the display virtual object even when the virtual object image 102 is small as compared to the marker 100. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an augmented reality system and an image composition display method using augmented reality technology, which is a technology for superimposing a virtual object as if it exists in real space. An augmented reality system that shoots an image including a marker for the purpose and synthesizes a virtual object image corresponding to the type of marker in the captured image into the captured image based on the position and direction of the marker in the captured image And an image composition display method.

  In recent years, with the improvement of three-dimensional image technology, augmented reality systems that superimpose virtual objects as if they existed in real space have been used for product presentations and the like.

  In order to realize such an augmented reality system, a technique for recognizing the type of virtual object to be superimposed, the combined position on the image obtained by photographing the real space with the camera, and the display direction of the virtual object is required.

  As one method for solving these problems, an augmented reality system using a marker for measuring a three-dimensional position for displaying a virtual object has been proposed (for example, see Non-Patent Documents 1 and 2).

  An example of the configuration of a conventional augmented reality system using this marker is shown in FIG. As shown in FIG. 6, the conventional augmented reality system includes a photographing unit 10, a marker recognizing unit 20, a marker three-dimensional position / direction determining unit 30, a display virtual object data determining unit 40, and an image synthesizing unit. 150 and a virtual object data storage unit 60.

  FIG. 6 shows a case where a camera with a marker 100 placed on a table is photographed. The marker 100 is configured by a black frame having a preset size, and by identifying a pattern such as a character or a pattern in an internal white portion, the marker type is specified and a virtual object to be superimposed is determined. Further, the three-dimensional position and direction where the marker is placed are detected from the size and inclination of the pattern, and the combined position and display direction on the captured image are determined.

  The imaging unit 10 captures an image including the marker 100 via a connected video camera. The marker recognizing unit 20 performs processing for recognizing the marker from the captured image captured by the capturing unit 10. The marker 3D position / direction determination unit 30 detects the 3D position and direction of the marker recognized by the marker recognition unit 20.

  The virtual object data storage unit 60 stores virtual object image data corresponding to the type of marker. The display virtual object data determination unit 40 selects and displays virtual object data corresponding to the marker type recognized by the marker recognition unit 20 from the virtual object data stored in the virtual object data storage unit 60. It is determined as a virtual object image 102 for performing.

  The image synthesis unit 150 synthesizes the captured image 101 captured by the imaging unit 10 and the virtual object image 102 from the display virtual object data determination unit 40 and outputs the synthesized image.

  Next, the operation of this conventional augmented reality system will be described with reference to the flowchart of FIG.

  First, the image capturing unit 10 captures an image including the marker 100 (step 701). Then, the marker recognizing unit 20 recognizes the marker in the captured image 101 photographed by the photographing unit 10 (step 702). Based on the type of marker recognized by the marker recognizing unit 20 by the display virtual object data determining unit 40, the virtual object data corresponding to the virtual object image stored in the virtual object data storing unit 60 is displayed. Data is determined (step 703). Then, the three-dimensional position / direction determination unit 30 detects the three-dimensional position and direction of the marker in the captured image (steps 704 and 705).

  Finally, in the image composition unit 150, as shown in FIG. 8, based on the position information and direction information from the marker three-dimensional position / direction determination unit 30, a virtual image displayed in the captured image 101 from the imaging unit 10 is displayed. A process of synthesizing the virtual object image 102 from the object data determination unit 102 is performed (step 706).

  By using such an augmented reality system, when an image including the marker 100 is captured, a virtual object image can be synthesized and displayed in the captured image corresponding to the position and direction in which the marker 100 is disposed. It becomes possible.

However, in the conventional augmented reality system using such a marker, if the displayed virtual object is small relative to the marker, the marker cannot be completely erased by the virtual object image synthesized in the captured image. There was a problem that it seemed to protrude from the side of the display virtual object. For example, in the composite image shown in FIG. 8, the marker 100 protrudes from the bottom of the virtual object image 102 inserted in the composite image. It is possible to solve this problem by reducing the size of the marker itself, but if the size of the marker is reduced, another problem occurs that the recognition rate of the marker type, distance, direction, etc. deteriorates. To do. For this reason, the marker is required to be practically larger than a certain size, and there is a limit to reducing it.
Hirokazu Kato, Mark Billinghurst, Koichi Asano, Keihachiro Tachibana: Augmented Reality System Based on Marker Tracking and its Calibration, Transactions of the Virtual Reality Society of Japan, Vol.4, No.4, pp.607-616 (1999) [Search January 13, 2016] Internet <URL: http://www.sys.im.hiroshima-cu.ac.jp/people/kato/publications/c-00.his2000.1.pdf> Hirokazu Kato, Mark Billinghurst: Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System, Proc. Of 2nd Int. Workshop on Augmented Reality, pp.85-94 (1999), [Searched on January 13, 2016 ], Internet <URL: http://www.sys.im.hiroshima-cu.ac.jp/people/kato/publications/c-01.ismr2001.1.pdf>

  In the above-described conventional augmented reality system, if the display virtual object is small relative to the marker, the marker cannot be completely erased by the virtual object image synthesized in the captured image, so the marker protrudes from the side of the display virtual object. There was a problem that it looked like.

  An object of the present invention is to provide an augmented reality system and an image composition display method capable of preventing a marker from protruding from a display virtual object even when the display virtual object is small relative to the marker. is there.

In order to achieve the above object, the augmented reality system of the present invention captures an image including a marker for three-dimensional position recognition, and displays a virtual object image corresponding to the type of the marker in the captured image. In an augmented reality system that synthesizes in the captured image based on the position and direction of the marker in the captured image,
A marker concealment data storage unit in which the three-dimensional information of a preset marker arrangement position is stored as marker concealment data;
When the virtual object image is combined with the marker position in the captured image, the marker concealment image stored in the marker concealment data storage unit is combined with the marker arrangement position of the captured image, and then the virtual object image And an image synthesizing unit that generates a synthesized image by synthesizing the images.

  According to the present invention, the virtual object image is synthesized after concealing the marker for recognizing the three-dimensional position with the marker concealing image, so even if the virtual object image is smaller than the marker, the display is possible. It is possible to prevent the marker from being seen from the side of the virtual object.

  As described above, according to the present invention, the virtual object image is combined with the marker for concealing the marker for recognizing the three-dimensional position with the marker concealing image, so that the virtual object image is compared with the marker. Even when is small, it is possible to prevent the marker from being seen from the side of the display virtual object.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an augmented reality system according to an embodiment of the present invention. In FIG. 1, the same components as those in FIG. 6 are denoted by the same reference numerals, and description thereof is omitted.

  The augmented reality system of the present embodiment has a configuration in which a marker concealment data storage unit 70 is newly provided and the image composition unit 150 is replaced with an image composition unit 50, compared to the conventional augmented reality system shown in FIG. It has become.

  The marker concealment data storage unit 70 stores three-dimensional information of preset marker arrangement positions as marker concealment data. In the present embodiment, since the center portion of the table upper surface is preset as the marker placement position, as shown in FIG. 2, the image of the center portion of the table upper surface is acquired as marker concealment image data, and the marker concealment is performed. Stored in advance in the data storage unit 70. In the present embodiment, it is assumed that the marker placement position for placing the marker is set in advance.

  Further, when the image composition unit 50 in this embodiment synthesizes the virtual object image with the marker position in the captured image, the marker concealment image stored in the marker concealment data storage unit 70 is used as the marker of the captured image. A final synthesized image is generated by synthesizing the virtual object image after being synthesized at the arrangement position.

  Next, a specific method for generating the marker concealment image 103 will be described with reference to FIG.

  Various methods are conceivable as a method for acquiring the three-dimensional information of the position where the marker is to be arranged. For example, as shown in FIG. The marker concealment image 103 may be generated by zooming in on the range where the marker is arranged and shooting from a multi-viewpoint direction. Further, by making such an operation hierarchical, the influence of the marker can be further reduced.

  Further, if it is assumed that the marker arrangement plane is a plane, a small marker 200 is arranged, and only one image obtained by photographing the small marker 200 from the front is prepared. May be warped (distorted) according to the display direction.

  Furthermore, the marker concealment image 103 may be generated by performing three-dimensional measurement of the marker arrangement surface by an existing method such as a laser scanner and generating a three-dimensional model of the marker arrangement surface.

  Next, the operation of the augmented reality system of this embodiment will be described with reference to the flowchart of FIG.

  The processing from step 401 to step 405 in the flowchart shown in FIG. 4 is the same as the processing from step 701 to step 705 in the flowchart of FIG. 7 showing the operation of the conventional augmented reality system. Omitted.

  In the augmented reality system of the present embodiment, the image composition unit 50 reads the data of the marker concealment image 103 stored in the marker concealment data storage unit 70 (step 406), and then, as shown in FIG. Based on the position information and direction information from the marker 3D position / direction determination unit 30, the data of the read marker concealment image 103 is combined with the marker arrangement position in the captured image 101 from the imaging unit 10. Then, a process of synthesizing the virtual object image 102 from the display virtual object data determination unit 102 is performed (step 407).

  According to the augmented reality system of the present embodiment, since the virtual object image 102 is synthesized after the marker 100 is concealed by the marker concealment image 103, the virtual object image 102 is smaller than the marker 100. However, it is possible to prevent the marker 100 from being seen from the side of the display virtual object.

  Although not shown in the figure, the augmented reality system of the present embodiment includes a recording medium on which a program for executing the image composition display method described above is recorded. This recording medium may be a magnetic disk, a semiconductor memory, or another recording medium. This program is read from the recording medium into the augmented reality system and controls the operation of the augmented reality system. Specifically, the above processing is realized by the CPU in the augmented reality system instructing the hardware resource of the augmented reality system to perform a specific process under the control of this program.

It is a block diagram which shows the structure of the augmented reality system of one Embodiment of this invention. It is a figure for demonstrating the production | generation method of the marker concealment image. It is a figure for demonstrating the specific production | generation method of the image 103 for marker concealment. It is a flowchart which shows operation | movement of the augmented reality system of FIG. It is a figure which shows an example of the synthesized image produced | generated by the augmented reality system of one Embodiment of this invention. It is a block diagram which shows the structure of the conventional augmented reality system. It is a flowchart which shows operation | movement of the conventional augmented reality system. It is a figure which shows an example of the synthesized image produced | generated by the conventional augmented reality system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image pick-up part 20 Marker recognition part 30 Marker three-dimensional position / direction determination part 40 Display virtual object data determination part 50 Image composition part 60 Virtual object data storage part 70 Marker concealment data storage part 100 Marker 101 Photographed image 102 Virtual object image 103 Marker concealment image 150 Image composition unit 200 Small marker 401-407 Step 701-706 Step

Claims (3)

An image including a marker for three-dimensional position recognition is captured, and a virtual object image corresponding to the type of marker in the captured image is displayed in the captured image based on the position and direction of the marker in the captured image. In the augmented reality system to synthesize,
A marker concealment data storage unit in which the three-dimensional information of a preset marker arrangement position is stored as marker concealment data;
When the virtual object image is combined with the marker position in the captured image, the marker concealment image stored in the marker concealment data storage unit is combined with the marker arrangement position of the captured image, and then the virtual object image An augmented reality system comprising: an image synthesis unit that generates a synthesized image by synthesizing. An image including a marker for three-dimensional position recognition is captured, and a virtual object image corresponding to the type of marker in the captured image is displayed in the captured image based on the position and direction of the marker in the captured image. An image composition display method for composition,
Photographing an image including a marker for three-dimensional position recognition by the photographing unit, and recognizing the marker in the photographed image by the marker recognizing unit;
Based on the recognized type of the marker, the display virtual object data determination unit determines the corresponding virtual object image data from the stored virtual object data;
In the marker 3D position / direction determining unit, detecting the 3D position and direction of the marker in the captured image;
A step of generating a synthesized image by synthesizing the virtual object image after synthesizing three-dimensional information of a preset marker arrangement position as a marker concealment image at the marker arrangement position of the captured image in an image synthesis unit; An image composition display method comprising: An image including a marker for three-dimensional position recognition is captured, and a virtual object image corresponding to the type of marker in the captured image is displayed in the captured image based on the position and direction of the marker in the captured image. A program for causing a computer to execute an image composition display method to be synthesized,
Processing of capturing an image including a marker for three-dimensional position recognition by the capturing unit, and recognizing the marker in the captured image by the marker recognizing unit;
Based on the recognized type of the marker, the display virtual object data determination unit determines the corresponding virtual object image data from the stored virtual object data;
Processing for detecting a three-dimensional position and direction of a marker in the captured image in a marker three-dimensional position / direction determining unit;
A process of generating a synthesized image by synthesizing the virtual object image after synthesizing the three-dimensional information of a preset marker arrangement position as a marker concealment image at the marker arrangement position of the captured image in an image synthesis unit; A program that causes a computer to execute.

Images