JP2938845B1 - 3D CG live-action image fusion device - Google Patents

Abstract

Abstract: PROBLEM TO BE SOLVED: To fuse a three-dimensional CG and a video image in real time and use a video image instead of the three-dimensional CG. SOLUTION: An object installation position estimating unit 1100 which dynamically calculates a destination of the real image display object according to a change in a viewpoint position / line of sight of a user who views a real image display object on which a two-dimensional real image is mapped.
And a mapping image changing unit 120 that changes the three-dimensional CG model of the real image display object according to the change in the user's viewpoint position / viewing direction and changes the image to be mapped.
0, the object installation position calculation unit 1100, and the fusion unit 105 including the texture object unit 1000 that creates a live-action video display object modified by an instruction from the mapping video change unit 1200, so that a real object exists. Realism and image quality as if you were

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for fusing and displaying a real image taken by a video camera and an object made by three-dimensional CG (computer graphics) so as to look natural.

[0002]

2. Description of the Related Art A description will be given below of an invention in which a real image is fused with a conventional three-dimensional CG. JP-A-5-1741
In Japanese Patent No. 29, an output image of a three-dimensional scene is converted into a three-dimensional CG image.
By combining three types of data such as model data, background video data, and mid-view video data, a three-dimensional CG
There has been invented a modeling device for generating a model as a real-time photorealistic expression in a realistic environment. The background image is, as shown in FIG.
This is background video data mapped on the inner surface of a hemisphere that is set so as to surround the three-dimensional CG model 2 that is the object.
The video data defining the background is defined on the inner surface of the hemisphere. For example, the background is a full 3
It is obtained by projecting a still image forming a field of view of 60 degrees. The intermediate scene image is a scene image between the distant background and the three-dimensional CG model 2. The image defining this intermediate view is defined on a topographic surface above the hemisphere 6, as shown in FIG. FIG. 27 shows a basic functional block diagram of the modeling device. The modeling device combines the output from the computer graphics modeler 11 and the video images from the background video storage device 16A and the intermediate scene video storage device 16B in the control device 15 to generate an output image of a three-dimensional scene, 20 is displayed.

[0003] In addition, there is a type of composition in which the depth value of a subject and that of a three-dimensional CG object are compared to determine the anterior-posterior relationship, and which part of the object is determined and hidden by the actual background image is displayed. is there. JP-A-8-1
No. 53213 discloses an image combining and displaying method described below for the purpose of creating and displaying a three-dimensional and realistic moving image by combining a two-dimensional real moving image with a three-dimensional CG with motion. Are listed. According to the image synthesizing and displaying method described in the present invention, a command input unit is used to select a CG model to be used and to specify a three-dimensional position and a direction in which the selected CG model is to be drawn, so as to represent a viewpoint position of a photographed image. From the viewpoint information and the three-dimensional information of the object captured in the actual image, the display image is calculated by calculating which part of the CG model is visible and which part is hidden and combining only the visible part with the actual image. Is created. That is, the CG model can be embedded in the three-dimensional structure of the object photographed in the real image.

Further, an image is selected from a photographed video image, and the selected image is pasted as a texture on a static flat plate or the like having a simple shape and synthesized with CG. There is a type that generates a continuous synthesized image by repeating the process every time (199).
3.4.25 Groupware Study Group Materials, "Image Communication Environment Using Virtual Space," NTT Human Interface Laboratories, pp. 67-74).

[0005]

Generally, when a three-dimensional virtual world is constructed by three-dimensional CG, a user can freely walk through the virtual world. However, constructing a virtual world only with three-dimensional CG requires more development costs as the virtual world becomes more complex.
Experts can not create without using special tools,
There is a problem that the image quality of the content is lower than that of the actual video. Therefore, in order to solve these problems, it has been considered to capture a real image in a virtual world constructed by three-dimensional CG. However, according to the above-described conventional method, (1) the distant view and the intermediate view are still images and are images captured at a certain viewpoint position, that is, a certain 3D CG model. It can only move around the dimensional CG model. (2) Even if the viewpoint moves, the video data of the distant view and the intermediate view do not change, and at the same time, since the projection unit that maps the image does not move, the change in the appearance of the real image due to the movement of the viewpoint is reflected. I can't let it. (3) Walk-through can be performed only in the traveling direction of the camera at the time of shooting the actual video. (4) Immersion into the virtual world of 3DCG (Immersion is to walk into the virtual world by walking through the virtual world constructed by 3D CG) and immersion into the world of live-action video Can not. (5) The world created by live-action video cannot be shared by a plurality of users. Therefore, during the immersion in the live-action video, it is not possible to meet and communicate with other users immersing in the world of the same live-action video.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and has been developed to fuse a two-dimensional real image with a three-dimensional virtual world constructed by a three-dimensional CG model. It is an object of the present invention to provide a three-dimensional CG live-action image fusion device that provides a certain three-dimensional virtual world.

[0007]

A three-dimensional C according to the present invention is provided.
The G live-action image fusion device is a three-dimensional computer graphics (hereinafter, referred to as “hereafter”) that arranges a live-action image display object obtained by fusing a real object image with a plane object in a three-dimensional virtual world.
"Computer graphics" is called "CG")
The live-action video fusion device is provided with the following elements. (A) a virtual world construction unit that constructs a three-dimensional virtual world with a three-dimensional CG model, (b) a viewpoint management unit that detects a change in viewpoint with respect to the three-dimensional virtual world constructed with the three-dimensional CG model, c) a storage unit for storing information of a live-action video display object that maps and displays a video on a plane object, and detects a live-action video display object in a field of view based on a change in viewpoint detected by the viewpoint management unit. A fusion unit for arranging at least one of the shape and the position of the planar object and arranging the real image display object in the three-dimensional virtual world using the information of the real image display object stored in the storage unit.

[0008] The fusion unit obtains a field of view based on a change in the viewpoint detected by the viewpoint management unit, and a real image display object database unit that stores object information specifying the real image display object. A viewpoint position gaze direction detecting unit that searches for the real image display object by searching the real image display object database unit, and calculates an installation position of the plane object based on the object information detected by the viewpoint position gaze direction detection unit. An object setting position calculating unit, a mapping image changing unit that determines a shape of the plane object and an image fused with the plane object based on the object information detected by the viewpoint position gaze direction detecting unit, and the mapping image change The video determined by the Texture object unit that fuses with the planar object having the shape determined by the video image changing unit and installs the planar object that fuses the image at the installation position calculated by the object setting position calculation unit to generate a real image display object And characterized in that:

The mapping image changing section includes an image database section for storing an image and a shape database section for storing a shape of a plane object.
The image to be merged with the plane object is determined with reference to the image database section, and the shape of the plane object is determined with reference to the shape database section.

The live-action image display object database stores a plurality of pieces of object information for specifying the live-action image display object for one view.
A selection condition for selecting predetermined object information from the plurality of pieces of object information is stored, and the viewpoint position gaze direction detection unit acquires object information from the real image display object database unit based on the selection condition. The three-dimensional CG live-action image fusion device includes a plurality of object management units including the viewpoint position gaze direction detection unit, the real-life image display object database unit, the object setting position calculation unit, and the mapping image change unit. And one texture object section.

[0011] The three-dimensional CG live-action image fusion device is connected to a network for each user. And a sharing control unit that controls to share any of the shapes among predetermined users.

The real image display object is composed of a distant view object, an intermediate view object and a near view object, and the texture object portion is a three-dimensional CG model for displaying as a distant view object regardless of a change in a viewpoint position and a line of sight direction. A distant view texture object section that fixes the installation position of the distant view object and the image displayed on the distant view object, and an intermediate view that changes the installation position of the intermediate view object according to a change in one of the viewpoint position and the line of sight A texture object portion, a three-dimensional CG model displayed as a foreground object, a setting position of the foreground object, and a video displayed on the foreground object in response to a change in one of the viewpoint position and the line-of-sight direction. Parts It is characterized in.

The object management unit calculates a distance from the new viewpoint position detected by the viewpoint position gaze direction detection unit to the installation position of the displayed real image display object, and displays based on the calculated distance. A control mode change unit that determines whether to display the actual photographed video display object as an intermediate view object or a near view object, and notifies the texture object unit of the determined result. In accordance with the notification from the mode changing unit, a new real image display object is generated by one of the intermediate view texture object unit and the close view texture object unit.

A three-dimensional CG live-action image fusion device according to the present invention is a three-dimensional CG real-view image fusion device for arranging a real-life image display object obtained by fusing a real-life image with a plane object in a three-dimensional virtual world.
The G live-action image fusion device includes the following elements. (A) a viewpoint live-action database unit that stores a viewpoint position and a line-of-sight direction in which the live-action video was captured in association with the live-view video; (C) an input unit for changing at least one of the viewpoint position and the line-of-sight direction in the three-dimensional virtual world; (d) the viewpoint position and the line-of-sight direction changed by the input unit; It guides according to a shooting path determined by the viewpoint position and the line-of-sight direction stored in the viewpoint actual-photography database unit, obtains a predetermined area based on the viewpoint position and the line-of-sight direction input from the input unit, and displays the predetermined area. Obtain object information specifying the live-action video display object from the live-action video display object database unit A viewpoint control unit for determining a real image displayed on the real image display object based on the acquired object information; and (e) a plane object for fusing the real image displayed on the real image display object determined by the viewpoint control unit. Texture object movement destination calculation unit that calculates the installation position of the real image display object so that the image fusion plane and the line of sight are orthogonal to each other,
(F) a mapping image changing unit that determines the shape of the plane object and an image to be merged with the plane object based on the object information acquired from the real image display object database unit; and (g) the mapping image changing unit. The determined image is fused with the planar object having the shape determined by the mapping image changing unit to generate a real image display object, and the generated real image display is generated at the installation position calculated by the texture object moving destination calculation unit. Texture object section where objects are placed.

[0015] A three-dimensional CG live-action image fusion device according to the present invention is a three-dimensional CG real-view image fusion device in which a real-life image display object obtained by fusing a real-life image with a plane object is arranged in a three-dimensional virtual world.
The G live-action image fusion device includes the following elements. (A) an input unit that changes at least one of a viewpoint position and a line-of-sight direction in a three-dimensional virtual world; (b) a live-action video display object database that stores object information for specifying a live-action video display object to be displayed in a predetermined field of view (C) a cylindrical three-dimensional virtual world is installed by laminating a plurality of live-action video display objects in the horizontal direction, and a predetermined three-dimensional virtual world is set from the plurality of live-action video display objects according to the viewpoint position input from the input unit. (D) selecting one of the plurality of live-action video display objects and selecting a viewpoint position input from the input unit; That the camera is moving toward the live-action video image display object The user monitoring unit for monitoring the presence in a region of a predetermined range of the other photographed image display objects.

The three-dimensional CG live-action image fusion device is further monitored by the viewpoint live-action database unit for storing a viewpoint position and a line-of-sight direction in which the real-life image was taken in correspondence with the real-life image, and by the user monitoring unit. A viewpoint control management unit that acquires a photographing path from the viewpoint actual photograph database unit based on a photographed image display object including a region within a predetermined range in which the viewpoint position exists, and restricts movement of the viewpoint position according to the acquired photographing path; It is characterized by having.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 Embodiment 1
A live-action video display object in which a video is mapped to a plane object, which is a three-dimensional CG model, is arranged in a three-dimensional virtual world, and the real-life video display object is adjusted according to the viewpoint position and movement (gaze direction) of a user who views the three-dimensional virtual world. Move and change the live-action video to map. As a result, it is possible to provide a sense of reality as if there is a real object, and to improve the sense of presence. The live-action video display object is composed of a three-dimensional CG model and a video texture mapped and displayed on the three-dimensional CG model.

FIG. 1 is a functional block diagram showing the basic configuration of the fusion device. In FIG. 1, reference numeral 100 denotes an input unit such as a keyboard / mouse / joystick / bicycle-type device for processing input commands and received data / signals, and 101 denotes a three-dimensional virtual world by operating the input unit 100. This is a viewpoint management unit that detects a new viewpoint position and a line-of-sight direction when the viewpoint position / line-of-sight direction of the user who views the image is changed. 105 acquires object information that specifies the real-life image display object in the user's field of view based on the change in the viewpoint and the line-of-sight direction detected by the viewpoint management unit 101, and, based on the acquired object information, This is a fusion unit that changes the installation position, selects a video to be displayed on the plane object, and fuses the selected video with the plane object to generate a live-action video display object. The detailed configuration of the fusion unit 105 will be described later. Reference numeral 1600 denotes a three-dimensional CG model database unit that stores and stores a three-dimensional CG model. 1500 denotes a texture database unit that stores texture data.
3D CG model database unit 16
00, a three-dimensional CG model to be displayed, and texture data required by the three-dimensional CG model are extracted therefrom. A rendering unit for synthesizing the texture data and a display unit 106 displays the video synthesized by the rendering unit 102. The virtual world construction unit 99 includes an input unit 100, a texture database unit 1500, a three-dimensional CG model database unit 1600, a rendering unit 102, and a display unit 106.

In the following, in a system in which the input unit 100 is operated to walk through a three-dimensional virtual world, a real object model exists by arranging a texture object on which a real image is mapped in the three-dimensional virtual world. An operation for making it appear as if it is will be described.

First, the user operates the input unit 100 to change the viewpoint position and line-of-sight direction of the user (avatar) viewing the three-dimensional virtual world. When the viewpoint position or the line-of-sight direction is changed by the input unit 100, the change is recognized by the viewpoint management unit 1.
01 detects. Using the detected new viewpoint position and line-of-sight direction, the fusion unit 105 generates a real image display object that enters the user's field of view determined from the viewpoint position and the line-of-sight direction. Then, in the rendering unit 102, the three-dimensional C that enters the user's field of view from the new user's viewpoint position and line-of-sight direction detected by the viewpoint management unit 101.
The G model is extracted from the three-dimensional CG model database unit 1600, and the texture data required by the three-dimensional CG model is extracted from the texture database unit 1500. The dimensional CG model and the texture data are synthesized. The synthesized video is displayed on a display unit 106 such as a display.

The operation of the fusion unit 105 will be described in detail. FIG. 2 is a functional block diagram of the fusion unit of the fusion device according to the first embodiment. In FIG. 2, reference numeral 1000 denotes a texture object unit for mapping an image onto a plane object formed of a polygon (a polygon is a minimum unit forming a figure) formed by three-dimensional CG. 19
Reference numeral 00 denotes a viewpoint position / viewing direction detecting unit.
01, the user's field of view is determined from the user's viewpoint position and line-of-sight direction, and object information for specifying a real-field video display object entering the field of view is obtained from the real-field video object database unit 1700.

FIG. 3 shows an example of object information stored in the live-action video display object database unit 1700. In FIG. 3, reference numeral 17001 denotes an area definition table which defines a predetermined area determined by the minimum and maximum values of the X coordinate and the minimum and maximum values of the Y coordinate. Area I is defined by the X coordinate and the Y coordinate.
D, the information (17002, 17003) defining the distant view, the information (17004, 17005) defining the intermediate view, and the information (1) defining the near view for each area ID.
7006, 17007, 17008, 17909) are stored. Each definition table has an item indicating “image file name”, and information of the actually shot video is stored in the table indicated by the image file name. Furthermore, the viewpoint position /
The line-of-sight direction detection unit 1900 obtains the positional relationship between the user's viewpoint position and line-of-sight direction and the real image display object. An example is shown in FIG. FIG. 4A shows three-dimensional coordinates. Although the Y coordinate indicates the height, it is assumed that there is no scalar value for the height because the three-dimensional virtual world is represented on a two-dimensional plane. The base line 1 in FIG. 4B is parallel to the Z axis and is oriented in the positive direction of the Z axis. Then, it is assumed that the basic real image display object is installed so as to be perpendicular to the base line. The real image display object is generated by rotating the basic real image display object based on a change in the user's line of sight. For this reason, 1900 finds the viewing angle α between the user's viewing direction vector 2 and the base line 1. Further, a distance 4 from the viewpoint 3 to the real image display object 5 is obtained. The distance 4 is a distance from the user's viewpoint 3 to a point at which the line-of-sight direction vector 3 and the real image display object 5 intersect vertically. The size of the real image display object is increased or decreased depending on the distance. This makes the image more realistic. Reference numeral 1100 denotes an object setting position calculation unit that calculates a destination of a plane object that constitutes a real image display object according to the viewpoint position / line of sight of the user detected by the viewpoint management unit 101. 1200 selects a video to be displayed (mapped) in accordance with the user's viewpoint position / viewing direction detected by the viewpoint management unit 101 and the positional relationship obtained by the viewpoint position / viewing direction detection unit 1900, and sends it to the texture object unit 1000. This is a mapping video changing unit that issues a change instruction for a video to be displayed (mapped). The mapping video changing unit 1200 selects a video to be displayed from the video database unit 1300 based on the user's viewpoint position / sight direction detected by the viewpoint management unit 101 and the positional relationship obtained by the viewpoint position / view direction detection unit 1900. Also, the three-dimensional shape (three-dimensional CG model) of the planar object displaying the selected video is obtained from the shape database unit 1400. For example, the horizontal / vertical ratio of the plane object is changed according to the distance obtained by the viewpoint position / viewing direction detecting unit 1900. The texture object unit 1000 is notified of the three-dimensional shape of the plane object from the mapping image changing unit 1200, and is notified of the change destination / movement destination of the installation position of the plane object from the object setting position calculation unit 1100. Thereafter, the texture object unit 1000 deletes (non-displays) the displayed live-action video display object.
Then, a live-action video display object in which the specified video is mapped to the specified object shape is generated and installed at the specified location. Video database unit 1300
The video acquired by the mapping video changing unit 1200 may be a still image or a moving image. Further, the video image may be a video image input in real time from a designated camera or a user's viewpoint. When the image is a moving image, the reproduction speed is changed according to the moving speed of the viewpoint (camera) of the user, and the image frame to be displayed is changed according to the current position of the viewpoint / camera.

The object installation position calculation unit 1100 includes:
For example, as shown in FIG. 5, in the direction in which the horizontal outward line-of-sight vector a starting from the user's viewpoint position and the real-view image display object c are orthogonal to the input position of the user's viewpoint, Calculate the setting position of. When the line-of-sight vector moves from a to b, the set position of the real image display object is calculated in a direction orthogonal to the line-of-sight vector b. Texture object part 1
000 is a notification that changes the three-dimensional shape of the live-action video display object and the mapping video,
When the information is received from 200, the shape of the real image display object and the mapping image are changed. As a result, a 3D virtual world obtained by fusing actual shot images can be displayed only by calculating and rendering a video that can be viewed from the user's viewpoint with respect to the 3D virtual world in which the texture object unit 1000 is installed in a conventional browser. .

Next, the operation of the fusion unit 105 will be described for several types. First, a flat plate as shown in FIG.
A method of rotating an image in the direction of E around the axis and changing the image to make it appear as if there is a real object will be described. FIG. 7 shows the processing flow. In this type (tentatively referred to as “type A”), an image of a specific object (for example, a tree) is taken at 360 ° and one-turn video image, and is divided into n.
Then, 360 ° / n ° * i (where 1 ≦ i ≦ n−1,
An image viewed from the direction of n> 0) is extracted, a specific object is extracted from the image, and a portion other than the extracted object is transparently displayed. For example, if a tree, sky, and turf are photographed in the extracted video, the tree is extracted as a specific object, and a portion where the sky and turf are photographed is displayed in a transparent manner. The n videos are registered in the video database unit 1300.

In the processing flow of FIG. 7, first, the viewpoint management unit 101 sends the information to the viewpoint position / viewing direction detection unit 1900.
The user's viewpoint position / viewing direction is notified (S1). The viewpoint position / line-of-sight direction detection unit 1900 calculates the line-of-sight angle at which point the user's viewpoint is rotated from the base line C of the target object based on the user's viewpoint position / line-of-sight direction (S2). As shown in FIG. 6, assuming that the height direction is z, the depth is x, and the spread is y, the base line C is a line passing through the rotation axis E and parallel to y, as shown in FIG. Then, the real image display object is set so as to be perpendicular to the line-of-sight vector B. Therefore, the viewing angle with respect to the viewpoint A and the viewing vector B becomes “α” as shown in FIG. The viewpoint position / viewing direction detecting unit 1900 notifies the mapping view changing unit 1200 and the object setting position calculating unit 1100 of the detected viewing angle. Mapping video changing unit 120
0 selects an image captured from the visual line angle α from the image database unit 1300, and sets the three-dimensional shape of the image display object on which the image is mapped to the shape database unit 14;
Select from 00. However, in the case of FIG. 6, the three-dimensional shape is not changed. Next, the texture object unit 100
For 0, the mapping video changing unit 1200 instructs a video to be displayed on the live-action video display object (S3).
Object setting position calculating unit 110 that receives the user's viewpoint position (or may be the user's gaze angle calculated earlier)
0 (S4) calculates the position at which the real image display object is installed so that the received line of sight vector B at the viewpoint position / line of sight of the user is orthogonal to the plane direction of the real image display object. After the calculation, the object setting position calculation unit 1100 sets the texture object unit 100
0 is notified of the change destination of the real image display object (S5). The texture object unit 1000 maps the real image instructed by the mapping image changing unit 1200 to a plane object, and maps the plane object on which the real image is mapped to the object setting position calculating unit 1.
It is installed at a location designated by 100 (S6). By repeating such processing, when the user moves, the live-action video display object moves, and the display video is changed, so that it is possible to display as if a specific target is at that location.

In the case of a building such as a building in which it is only necessary to paste texture on a rectangular parallelepiped, images taken from each of the five directions are mapped onto a cubic polygon or five flat plate-shaped planar objects on each surface. Then build the building into a cubic shape. In this case, the live-action video display object does not need to be moved even when the viewpoint of the user changes, and the video to be mapped does not need to be changed.

Next, as shown in FIG. 8, the outside of the passage, the building, or the tunnel is displayed by a three-dimensional CG model, and the inside of the passage, the building, or the tunnel is replaced by a real image, and the inside of the passage, the building, or the tunnel is replaced. A process of displaying an image viewed from the outside will be described. This type is temporarily called "type B". The image of the target object is shot by setting the height at which the image is shot and the viewpoint during the walk-through at the same position. As shown in FIG. 9, the target object is photographed, for example, by drawing a circle having a radius of a predetermined length centering on the target object e, and dividing points A, B, and C on the circumference of the circle. Install a camera at the point and shoot. The points A, B and C are
It is equally divided by the angle β. A specific target is extracted from the video captured from each division point, the three-dimensional shape of the plane object on which the extracted video is mapped is determined, and the video portion other than the specific target is transparently displayed at the time of mapping.
FIG. 8 shows an image of the entrance of the tunnel viewed from the outside. For example, if an image of a person is displayed at the entrance of a tunnel, and if the captured image shows the sea behind the person, the background of the sea is transparently displayed and only the person is extracted from the image to create a planar object. To map. The processing flow is almost the same as in FIG. The difference from FIG. 7 is the calculation processing of the destination of the real image display object to which the image is mapped by the object setting position calculation unit 1100. Texture object section 10
In FIG. 6, in FIG. 6, the three-dimensional shape is not changed, and only the real image display object is rotated. However, in the case of FIG. 8, a three-dimensional shape corresponding to the user's viewpoint position is selected, and the object of the three-dimensional virtual world where the image is pasted, for example, the image of the building side model and the image of the real image display object match. The live-action video display object must be installed as shown. FIG. 8 shows an image of the tunnel entrance viewed from the outside. The three-dimensional shape of the tunnel entrance seen from the viewpoint A is a real image display object G
It is a shaded portion of. The image includes a checkered portion in addition to a hatched portion. Plaid portions must not be displayed. Further, the live-action video display object G shown in FIG. 8 is installed so as to cover the entire tunnel entrance so that there is no space. If the live-action video display object G is placed upside down or shifted downward, the live-action video display object G and the three-dimensional CG model of the tunnel entrance will not overlap. For this reason, the installation position of the live-action video display object is calculated and calculated in advance, and the video database 13
Register at 00. Therefore, at the time of the walk-through, the real image display object is set at the position indicated by this data. After installation, renderer (Renderer is a 3D C
It is a device that displays a three-dimensional virtual world composed of G and video images), and calculates and displays images that can be viewed from the user's viewpoint. By doing so, the windows of the identified building and the inside of the entrance of the passage can be seen from the outside as if they were looking at the real thing.

Embodiment 2 In the second embodiment, unlike FIGS. 6 and 8 described in the first embodiment, a live-action image is created from the entrance of the building, the entrance of the passage, and the entrance of the tunnel shown in FIG. A case of walking through a building, a passage, a tunnel, or the like will be described. This type is temporarily called "type C". In FIG.
An example of walking through the inside of the tunnel will be described. In FIG. 10, in the virtual world, the shape of the live-view image display object is the same size as the real path and mapped to the live-view image display object is obtained by pasting, as a texture, an actual photograph taken from the user's viewpoint. Consider the case of walking through. FIG. 11 is a functional block diagram of a main part of the three-dimensional CG live-action image fusion device according to the second embodiment. In FIG. 11, reference numeral 2000 denotes each frame of a video image taken by moving the camera at the viewpoint position of the user with the camera mounted on a car or a trolley, and the viewpoint position / This is a viewpoint actual photograph database unit that stores information on the line of sight in pairs. A viewpoint control unit 2100 guides the user's viewpoint along the movement path of the camera, and determines a new viewpoint / gaze direction of the user based on the forward movement distance / reverse movement distance of the path determined by the user. Texture object movement for calculating the installation position of the live-action video display object so that the direction of the user's line of sight and the image plane of the live-action video display object are orthogonal to each other in the same direction as the movement of the user's viewpoint and by the same amount of movement This is the calculation unit. The other elements are the same as the elements having the same reference numerals in FIGS. 1 and 2.

At the position of the viewpoint of the user who walks through the three-dimensional virtual world, the real world is photographed by a camera. Video data is created so that the video at the time of shooting and the shooting location (viewpoint position / line of sight of the camera) can be understood, and registered in the viewpoint real-photography database unit 2000. Unnecessary parts of the video data are edited and deleted. The texture object unit 1000 writes the background in the deleted portion so that it looks natural at the time of rendering. Also, the exit portion is displayed in a transparent manner so that the outside three-dimensional virtual world can be seen from the exit. FIG. 12 shows a processing procedure of the three-dimensional CG live-action image fusion device according to the second embodiment. Until approaching the entrance of the building, the passage, or the tunnel, the display of the three-dimensional virtual world is realized by the type B method described with reference to FIG. 8 in the first embodiment (S10). When the video viewed from the user's viewpoint becomes only the video in the entrance, the video is switched to the captured video (S11). Whether or not only the image in the entrance has been reached is determined by the texture object section 1000
Judge by Texture object section 1000
Determines the field of view from the user's viewpoint position and line-of-sight direction, obtains video data entering the field of view from the real image display object database unit 1700, and determines whether the obtained video is data inside the entrance. At the moment the video is switched, the live-action video display object that has been displayed so far is hidden, and the display is switched to another live-action video display object. The user inputs a forward or backward command from the input unit 100 to move the viewpoint position. At this time, the movement of the viewpoint is restricted only by the viewpoint control unit 2100, and can only move along the photographing route of the camera registered in the viewpoint actual photograph database unit 2000 (S12). When the movement of the viewpoint is completed, the texture object movement destination calculation unit 2200 calculates that the movement destination of the live-action video display object follows the movement of the user's viewpoint (S13). When the movement destination, that is, the installation position is determined, the texture object unit 100
0 is notified of the destination. Then, the mapping video changing unit 1200 converts the live-view video displayed from the viewpoint / gaze direction of the user displayed at the destination to the video database unit 1300.
And then notifies the texture object unit 1000 (S14). Texture object section 1000
Receives the video data and the installation position of the live-view video display object, immediately deletes the live-view video display object that has been displayed so far, installs the live-view video display object at the received destination, and maps the video (S
15). In accordance with the progress of the viewpoint, the real image viewed from the viewpoint is mapped to the real image display object.
The distance between the user's viewpoint and the live-action video display object is
The real image display object moves in the three-dimensional virtual world in accordance with the movement of the user's viewpoint so that the object is always kept constant. At this time, the user can only move forward or backward on the determined course. The viewpoint control unit 2100 monitors the viewpoint position of the user, and controls (constrains) the viewpoint of the user when the viewpoint position is within a certain range and is moving toward the live-action video display object. Is started, and the viewpoint is controlled (bound) until a certain course (path registered in the viewpoint actual photograph database unit 2000) is completed. If the user exits the route, leaves a certain distance from the video display object, and does not proceed toward the object, the viewpoint is not restricted. When the image to be mapped to the installation position of the real image display object is determined, the image is displayed on the display unit 106 by the rendering unit 102 in the same process as the other three-dimensional CG models.

In type C, if the user moves forward quickly,
The video will also switch quickly. Therefore, when the traveling speed of the user is increased, the texture object unit 1000 thins out and displays the video to be displayed.

When approaching a passage or an exit of a building, mapping an image obtained by fusing an image of a three-dimensional virtual world formed by a three-dimensional CG model that is visible to the outside with a real image of a passage to a real image display object. (S1
6). When passing through the passage, the image display object, which has been displaying the actual image, is not displayed. That time,
An image of the outside three-dimensional virtual world seen from the exit is displayed.
Further, when a live-action video display object exists in the other three-dimensional virtual world, the live-view video is mapped to the video display object. In addition, if the reverse running image is created in the same manner, it is possible to switch between forward and backward during the walk-through. For example, if you set a camera on a car, adjust the height to the user's viewpoint, and display what was shot according to the walkthrough of the car, you would walk through the live-action video seen from the car It is possible to build a system.

Embodiment 3 FIG. FIGS. 13 and 14 are functional block diagrams of a third embodiment in which a real image and a three-dimensional CG model are combined with a real image display object. Figures 13 and 1
4, reference numeral 3000 denotes a live-action video CG model fusion unit that overwrites a three-dimensional CG model on a real-life video mapped to a real-life video display object.
An object selection unit 100 selects a three-dimensional CG model to be displayed as needed from the three-dimensional CG model displayed in the image of the virtual world viewed from the viewpoint of the user. Elements of the other reference numerals are the same as those of the same reference numerals in FIGS.

For example, when the image to be mapped to the real image display object is an image viewed outward from the exit of the passage of the building, and when the outside scene is constituted by a three-dimensional CG model, the image is displayed. A three-dimensional CG model must be fused to a part corresponding to the outside scenery. The object selecting unit 3100 extracts a three-dimensional CG model included in the field of view from the three-dimensional CG model database unit 1600 based on the user's field of view obtained by the viewpoint position / line-of-sight direction detecting unit 1900. And selects a three-dimensional CG model to be rendered according to the execution mode. The “execution mode” here is, for example, a mode indicating a season, and changes a three-dimensional CG model selected from “spring mode”, “autumn mode”, “winter mode”, and “summer mode”. Parameters to perform.
The live-action video CG model fusion unit 3000 overwrites and synthesizes the three-dimensional CG model selected by the object selection unit 3100 on the rendering image of the real-life texture mapped to the real-life video display object. Just like the three-dimensional CG model is displayed on the mapping video of the real video display object.
In other words, the live-action video CG model fusion unit 3000 creates a virtual
A three-dimensional CG model that can be viewed from the user's viewpoint is selected from the three-dimensional CG model database unit 1600 according to the execution mode, and the image of the selected three-dimensional CG model is mapped and synthesized on the image mapped to the real image display object. . Therefore, the position (depth) relationship between the object in the video displayed on the real video display object and the three-dimensional CG model is calculated, and no processing such as which part is displayed or hidden is not performed. By doing so, the processing can be speeded up. By providing the object selection unit 3100 and the real image CG model fusion unit 3000, not only the real image but also the three-dimensional CG model that exists at the end (the blind spot) of the image display object, that is, appears ahead of the image display object is displayed. It is possible to do.

In the three-dimensional CG real image fusion device having the configuration shown in FIG. 13, the user's viewpoint position / view direction is guided by the photographing path, and the real image display object is moved following the view position / view direction. However, instead of moving the real image display object in accordance with the movement of the viewpoint position, the real image display object at the viewpoint position / gaze direction before and after movement is displayed as a still image, and the real image display is performed. When a three-dimensional CG model is fused to a mapping video of an object, a three-dimensional CG real image fusion device having a configuration as shown in FIG. 14 is used.

Further, it is not necessary to create all the models of the three-dimensional virtual world by the three-dimensional CG, but to create only the specific object by the three-dimensional CG and to set them at the coordinate values in the accurate virtual world. In such a case, it is possible to supplement the three-dimensional virtual world with a real image. Thus, three-dimensional CG
By providing the live-action video CG model fusion unit 3000 that fuses the model and the live-action video, it is possible to reduce the content development cost and improve the realism / realism.

Embodiment 4 FIG. In the live-action video display object created by the texture object unit 1000 of the second embodiment, the traveling direction is the viewpoint real-life database unit 200
The direction is limited to the direction indicated by the route registered as 0. Therefore, the user cannot freely change the traveling direction. Therefore, the following mechanism is provided so that the user can select the traveling direction to some extent. FIG.
FIG. 14 is a functional block diagram illustrating a configuration of a main part of a three-dimensional CG real-life image fusion device according to a fourth embodiment. FIG.
FIG. 6 is a diagram showing a live-action video display object according to the fourth embodiment. In FIG. 15, as shown in FIG. 16, reference numeral 4000 denotes a cylindrical shape in which a plurality of live-view image display objects are horizontally attached to each other to be installed in a three-dimensional virtual world, and the cylindrical shape is attached to the cylindrical shape according to the user's viewpoint position. It is a display management unit for displaying and hiding the real image display object. 4100, the user selects one of the live-action video display objects, for example, selects the live-action video display object 4310a, and moves the viewpoint in a direction perpendicular to the video display surface of the selected live-action video display object. A user monitoring unit that monitors whether or not the viewpoint position is within a certain range and is moving toward the video display object. Reference numeral 4200 denotes a viewpoint control management unit that controls the movement path of the viewpoint based on the viewpoint position / sight line direction of the user. In the example of FIG. 16, it is determined whether or not the user's viewpoint position is approaching the selected live-action video display object 4310a based on whether or not the user's viewpoint position exists between the boundary lines 4310b and 4310c. Furthermore, whether to control the movement path of the viewpoint depends on the viewpoint position of the user.
It manages whether or not it is inside 340 (the checkerboard portion in FIG. 16), and if the user's viewpoint is inside the branching unit 4340, it controls to guide the user's viewpoint.

The photographed image display object 431 shown in FIG.
A cylindrical object, such as 0, in which a plurality of live-action video display objects are horizontally attached is referred to as a branch object. When the user monitoring unit 4100 confirms that the user's viewpoint has entered the area where the branch object is located, the display management unit 4000 displays the branch object. For example, as shown in FIG. 17, when the user moves from the viewpoint 4320 in the direction of the arrow of the line-of-sight vector 4330 and enters the hatched portion 4350, the display management of the live-action image display object 4310a, which is the branch object displayed so far, is performed. Not displayed by the unit 4000. The display management unit 4000 displays a live-action video display object that enters the field of view according to the user's viewpoint position / gaze direction. For example, if the user's viewpoint is inside the hatching unit 4350 and the line-of-sight vector points toward the real image display object 4310c, the display management unit 4
000 displays a live-action video display object 4310c. When the viewpoint position is advanced toward the live-action video display object on which the video to be walked is reflected, the viewpoint is bound to the live-action video display object.
The viewpoint control management unit 4200 monitors the viewpoint position of the user, and places the image in a certain area set by the real image display object.
For example, it is checked whether or not the user's viewpoint enters the branching unit 4340 in FIG. When recognizing that the user's viewpoint has entered, the movement of the user's viewpoint is restricted. That is, the movement of the viewpoint is along the route registered in the selected real image display object 4310a. The user can move the viewpoint only to either forward or backward travel. Therefore, the user can freely select the traveling direction by the number of prepared real image display objects. However, the viewpoint control management unit 4200 can realize the separation that the user can proceed or not depending on the attribute of the user. Also,
The display management unit 4000 can perform control such as changing a branch object to be displayed or switching between non-display / display according to the attribute of the user. For example, if a user in a certain department has a real-life video display object to be hidden, control is performed so that the traveling direction is restricted so as not to approach the real-life video display object to be hidden.

The real image display object that the user can immerse, that is, can advance, is predetermined. The more branch points you have, the more refined and realistic you will be.

Embodiment 5 In the fifth embodiment, by providing the object management unit 5000, the processing of the texture object unit 1000 can be performed collectively, thereby improving the processing performance and the modularity. Also, the mapping video changing unit 120
0, collectively manages the viewpoint position / line-of-sight direction input to the object setting position calculation unit 1100, detects a live-action video display object entering the user's field of view, and limits the process to a live-action video display object necessary for rendering. Is to improve.

FIG. 18 is a functional block diagram of the fusion device according to the fifth embodiment. 5000 monitors the position of the user's viewpoint (the viewpoint of the camera) when a plurality of live-action video display objects are present in the user's field of view, and uses the viewpoint position information to display the live-action video display object in the user's field of view. It is an object management unit that controls the positions of all the detected and detected real image display objects and the real images to be displayed. The object management unit 5000 includes the mapping video changing unit 1200, the video database unit 1300, and the shape database unit 1400.
The object setting position calculating unit 1100, the viewpoint position / line-of-sight direction detecting unit 1900 that detects a real image display object entering the field of view from the field of view obtained from the user's viewpoint position and line-of-sight direction, and the real image display object It comprises a database unit 1700. The viewpoint position / line-of-sight direction detection unit 1900 detects object information that specifies a real image display object that enters the field of view obtained from the detected user's viewpoint position and line-of-sight direction from the real image display object database unit 1700. Since there are a plurality of real image display objects existing in a predetermined field of view as detectable object information, it is assumed that a plurality of object information can also be detected. For each of the detected object information, a real video image to be mapped by the mapping video changing unit 1200 is acquired from the video database unit 1300, and a three-dimensional CG model of the real video image display object for mapping the video is obtained from the shape database unit 1400.
Then, the installation location (movement destination) of the real image display object is determined by the object installation position calculation unit 1100, and the texture object unit 1000 moves to the display position where the real image display object is determined. One texture object unit 1000 manages a plurality of object management units 5000. Therefore, it is possible to control one texture object unit 1000 for each user's video displayed in the virtual environment of a plurality of users.

Next, in the virtual environment of a plurality of users connected to the network, the viewpoint positions /
A method of monitoring the line of sight, obtaining a field of view, and controlling the installation position of a real image display object that enters the field of view and the display image will be described. FIG. 9 shows a processing flow of the viewpoint position / viewing direction detecting unit 1900 in the fifth embodiment.
Each time the viewpoint position / viewing direction is changed by the input unit 100, change information is transmitted from the virtual environment of each user to the viewpoint position / viewing direction detection unit 1900 (S20). The viewpoint position / line-of-sight direction detecting unit 1900 obtains a new field of view based on the change information, and obtains object information for specifying the real image display object newly added to the field of view from the real image display object database unit 1700 ( S21). Further, a real image display object that has disappeared from the field of view is detected. For each live-action video display object, a video corresponding to the viewpoint position is acquired from the video database unit 1300 by the mapping video changing unit 1200, and the three-dimensional shape is converted to the shape database unit 1.
400, the object installation position calculation unit 110
At 0, the installation position of the real image display object corresponding to the viewpoint position is obtained, and the texture object unit 1000
The real image is mapped in (S22).

In the object management section 5000, one texture object section 1000 performs a video mapping process in the virtual environment of each user. For this reason, it is possible to improve processing efficiency as compared with the case where a texture object unit is provided for each virtual environment of each user. For example, it is possible to shorten the processing time by repeatedly using the data once loaded. This corresponds to a case where a plurality of identical video display objects are installed by changing the video display object displaying such video only at the installation location.

Embodiment 6 FIG. A case where it is necessary to share a displayed image and a set position among users by a video displayed on a live-action video display object installed in a three-dimensional virtual world shared in a virtual environment of a plurality of users connected to a network. May be changed by individual users. By providing a sharing control mechanism in the live-action video fusion device to realize such a method of using the live-action video display object, it is possible to switch between sharing or non-sharing the live-action video display object between users. It also makes it possible.

FIG. 20 is a functional block diagram of the fusion device according to the sixth embodiment. In FIG. 20, reference numeral 6000 denotes a live-action video display object installed in a three-dimensional virtual world shared in a virtual environment of a plurality of users connected to a network. This is a shared control unit that changes shared control, such as freely changing or synchronizing an installation position, a display image, and a three-dimensional shape between specific users in real time, according to a situation. The sharing control unit 6000 manages the sharing of the live-action video display object between specific users, and performs dynamic change management of sharing members. It is assumed that each of the live-action video display objects has information indicating access rights. For the texture object unit 1000 that controls the individual live-view video display objects in the field of view of the user, the live-view video display object is displayed using the multicast viewpoint / viewpoint position of the user. It is also possible to control the position of the real image and the real image to be displayed for each real image display object. For example, data for managing an access right to object information specifying a certain live-action video display object is provided as follows. Live-action video display object name: access permitted member list Live-action video display object 1: Saeki, Miyauchi Live-action video display object 2: Iwase, Shimoda

When the live-action video display object enters the field of view of the user, the sharing control unit 6000 checks whether or not the user has an access right. Transfer the data to the user's usage environment. If access is not allowed, do not transfer data. For example, when the management data described above is registered in the object information, the user “Iwase” cannot display the live-action video display object 1 because the user has no access right. In addition, control such as synchronous shared use or independent individual use is performed for each specific object. The sharing control unit allows a user who satisfies the conditions 1 and 2 to synchronously share the live-action video display object 1 and controls the others individually. Control for achieving exclusive use of the video object can also be realized. The sharing control unit has such information. Real image display object: Access permission condition Real image display object 1: Condition 1 and condition 2 Real image display object 2: Condition 3

By providing the sharing control unit 6000,
Since it is possible to easily realize the switching between the specific users in a synchronized manner and to use the live-action video display object uniquely for each personal environment, the controllability of the live-action video display object is improved. When the installation position, the three-dimensional shape, and the displayed live-action video of the shared live-action video display object are changed, the change information is multicast to all virtual environments of the shared members. Examples of shared live-action image display objects include a bulletin board and an entrance door. For example, if the entrance door is configured by a live-action video display object, when the shutter of the door is down, it is only necessary to paste the shutter video, so that all users sharing the actual state of this live-action video display object Also, the display video setting position and the like are shared. In addition, when the shutter is raised to make the inside of a glass door visible, the video to be displayed and the installation position can be determined for each user in accordance with the user's viewpoint position / sight line direction, so that it is not shared. I do. As described above, the sharing control unit 6000 switches between sharing and non-sharing according to the content to be displayed. When the number of persons sharing the live-action video display object exceeds a certain value, when the virtual world where the live-action video display object exists is shared, the video display object is not displayed to the user. Further, the information to be shared, that is, the image of the real image display object is changed according to the attribute of the user.

Embodiment 7 FIG. In the seventh embodiment, the real image display object is divided into a distant view, an intermediate view, and a near view. In the distant view type, the installation position of the real image display object and the mapping image are kept fixed. In the intermediate scene type, the real image display object is moved so that the image viewed from the user's viewpoint looks like a three-dimensional image while the image is fixed. In the foreground type, the video to be mapped is changed in accordance with the viewpoint position / gaze direction of the user, the three-dimensional shape of the real video display object is changed, and the position of the real video display object is moved. As described above, the processing method is changed in accordance with the use of the real image display object, thereby improving the processing efficiency and improving the content development efficiency.

FIG. 21 is a functional block diagram of the fusion device according to the seventh embodiment. In FIG. 21, reference numeral 9000 denotes a distant view texture object unit that maps a distant view to a plane object and does not change the image to be mapped, the position, or the shape in which the image is displayed even if the user's viewpoint position / viewing direction changes. Reference numeral 9100 denotes an intermediate scenery texture object unit that changes only the position at which an image is displayed when the viewpoint position / gaze direction of the user changes. Reference numeral 9200 denotes a foreground texture object unit that maps a foreground onto a plane object and changes the shape of a video to be displayed and the position and location of a three-dimensional CG model for mapping the video in accordance with a change in the viewpoint position / gaze direction of the user. The texture object section 1000 includes the distant view texture object section 9000, the intermediate view texture object section 9100, and the near view texture object section 9200. For example, as shown in FIG. 22, a real image display object 9000a displayed by a distant view texture object unit 9000 and an intermediate view texture object unit 91
00 is a live-action video display object 9100b,
The real image display object 9200c displayed by the foreground texture object unit 9200 is arranged. As shown in FIG. 22, the distant-view texture object unit 9000 has a live-view video display object 9000a attached in a cylindrical shape, and an intermediate-view texture object unit 9100.
Arranges the real image display object 9100b in the dashed hatched area in FIG. 22, and the foreground texture object unit 9200 arranges the real image display object 9200c in the solid hatched area. In this way, by separately arranging the video display objects, it is possible to realize the label of detail (level of detail) of the video viewed from the user.

It is also conceivable to implement the real-view video display object as an intermediate view or a close-up view. Accordingly, the control mode changing unit dynamically changes the control mode of the texture object unit 1000 for controlling the live-action video display object, thereby freely changing the degree of detail of the displayed image of the three-dimensional virtual world. it can.

FIG. 23 shows a fusion device according to the seventh embodiment having a control mode changing unit. In FIG. 23, 7
000 calculates the distance between the user's viewpoint position and the video display object in the object management unit 5000, and changes the viewpoint position / line-of-sight direction of the user according to the distance. A control mode changing unit that switches between a control mode that changes only the position and a control mode that changes the image to be displayed and the shape and position of the three-dimensional object displaying the image in accordance with the change in the viewpoint position / gaze direction of the user. It is. Even if the user moves around in the solid hatched area in FIG.
The distant view texture object section 9000 does not move, and the image does not change. When the user moves in the solid-line hatched area, the middle-view texture object unit 9100 moves the live-action video display object to a place where the user looks most natural according to the viewpoint position of the user.

The operation of the intermediate scene texture object will be described with reference to FIG. In FIG. 24, a live-action image display object 94 mapping one photo of a building is shown.
00, 9500 are installed in a certain three-dimensional virtual world. This shows the behavior of the live-action video display object when the user views the live-action video display object from point A and when viewed from the point B. As shown in FIG.
00 appears to be present. When the user views the live-action video display object from point A, the live-action video display object 9500 moves to the position indicated by J in FIG. The position J is determined by the ratio of the size of the portion e and the size of the portion d when the user views the building from the point A, and the e of the building 9300.
This is a location where the ratio of the size of the portion and the size of the d portion is the same. Next, it is assumed that the user has moved to the point B. When the user looks at the live-action video display object from point B,
The live-action video display object is K as shown in FIG.
Go to the location. At the destination K, the ratio of the size of the portion e to the portion d when the user views the building from the point B is the same as the ratio of the size of the portion e to the portion d of the building 9300. As described above, the intermediate view texture object unit 9100 determines the right / left ratio (for example, the ratio between the e portion and the d portion) of the video (for example, the building) displayed on the real video display object as viewed from the user, and The real image display object is moved so as to match the left / right ratio (for example, the ratio between the e portion and the d portion) when viewed from the destination of the user.

Next, the operation of the distant view object section, the intermediate view object section, and the close view object section when the user moves from the point a to the point b in FIG. 25 as shown in FIG. 25 will be described. When the user's viewpoint moves from point a to point b, the distant view texture object unit 900
0, as shown in FIG. 25, hides the real image display object A displaying the distant view and newly displays the real image display object B shown in FIG. Further, as the user moves the viewpoint, the live-action image display object 9200c installed in the solid hatched portion changes from a near view to an intermediate view. 9100b installed in the wavy hatched area changes from the middle view to the close view. First, the viewpoint mode / line-of-sight direction detecting unit 1900 sends the control mode changing unit 7
000 is notified of the movement of the viewpoint position. The control mode change unit 7000 determines whether the real image display object is a near view or an intermediate view, and instructs the texture object unit 1000 to change the processing. In addition, the displayed real image display object is changed, for example, from the intermediate view type to the near view type.

The three-dimensional C described in the first to seventh embodiments.
The G live-action image fusion device includes an input unit capable of inputting a command and processing received data / signal, and a viewpoint position / line-of-sight direction of a user who views the three-dimensional virtual world in the three-dimensional virtual world by operating the input unit. A viewpoint management unit for detecting the change when the image is changed, and a three-dimensional CG model in the field of view of the user based on the detected change in the viewpoint position / viewing direction, and a live-action image to be mapped and displayed on the three-dimensional CG model A fusion unit that detects a real image display object composed of a texture, changes a three-dimensional shape and an installation position of the detected real image display object, selects an image to be displayed on the real image display object, and displays the image. And a 3D C that registers and stores a 3D CG model
A G model database unit, a texture database unit for storing texture data, and 3
A rendering unit that extracts a three-dimensional CG model, extracts a three-dimensional CG model to be displayed from the three-dimensional CG model, texture data required by the three-dimensional CG model, and synthesizes an image, and a display that displays the synthesized image. And a part.

Further, the fusion unit receives the two-dimensional real image to be mapped and the installation position, maps the real image to the real image display object, and moves the real image display object to the designated installation location. A texture object section to be set; an object installation position calculation section for calculating an installation position of the live-action video display object according to a change in the viewpoint position / line-of-sight direction of a user moving in the three-dimensional virtual world; And a mapping video changing unit that changes the actual video to be mapped.

Further, the live-action video display object in the field of view of the user is detected, and a user attribute or a user attribute is detected from the detected live-action video display objects.
The live view video display object is selected according to the execution mode of the entire system, and the user's viewpoint position / the mapping video change section, which manages the live view video display object, is calculated. An object management unit that notifies the gaze direction and instructs to change the three-dimensional shape object that displays the actual image, the position, and the image displayed by the selected texture object unit according to the viewpoint position / gaze direction. It is provided.

Further, in the texture object section which manages the live-action video display object installed in a three-dimensional virtual world shared by a plurality of network-connected virtual environments of the users, the virtual object of each of the users It is possible to freely change an installation position, a display image, and a three-dimensional CG model for mapping an image independently for each environment, and to set an installation position, a display image, and the three-dimensional CG model between a plurality of virtual environments of the specific user. It is provided with a sharing control unit that changes sharing control such as real-time synchronization according to the situation.

Further, a distant view is mapped in the texture object section which manages the live-action video display object, and the video to be mapped, the installation position, and the video are displayed even if the viewpoint / gaze direction of the user changes. A near-view texture object portion that changes only the position of a three-dimensional CG model that displays an image when the viewpoint position / line-of-sight direction of the user changes and a near-view texture object portion that does not change the three-dimensional CG model are mapped. It is provided with a video to be displayed, a three-dimensional CG model for mapping the video, and a foreground texture object section for changing the installation position in accordance with a change in the viewpoint position / gaze direction.

Further, in the object management unit,
Calculating a distance between the user's viewpoint position and the real image display object, and mapping an image when the user's viewpoint position / gaze direction changes according to the distance; 3
Control that switches between a control mode that changes only the position of the three-dimensional CG model and a control mode that changes the image to be displayed, the three-dimensional CG model that displays the image, and the installation position in accordance with the change in the viewpoint position / gaze direction of the user. It has a mode changing unit.

Further, the camera is mounted on a car or a trolley and the like, and each frame of the video image shot while moving the camera and the viewpoint position / line-of-sight direction of the camera that shot the image of the frame are paired. And a viewpoint control for guiding a user's viewpoint along a movement path of a camera and determining a new viewpoint / gaze direction of a user based on a forward movement distance / reverse movement distance of a path determined by the user. And a texture for calculating an installation position of the real image display object so that the user's line of sight is orthogonal to the image plane of the real image display object with the same direction as the movement of the user's viewpoint and the same amount of movement. An object destination calculation unit is provided.

Further, a plurality of the live-action video display objects are attached in a horizontal direction in a cylindrical shape and installed in a three-dimensional virtual world, and the live-action video display objects stuck together in the cylindrical shape according to a user's viewpoint position. Display,
A display management unit to be hidden, and whether the user selects one of the live-action video display objects and moves the viewpoint in a direction perpendicular to the video display surface of the selected live-action video display object, A user monitoring unit that monitors whether a viewpoint position is within a certain range and is moving toward the video display object, and a viewpoint that manages a control method of changing a viewpoint / gaze direction of the user. It has a control management unit.

[0061]

The three-dimensional CG live-action image fusion device according to the present invention is configured as described above, and has the following effects.

Point of View of User Looking at 3D Virtual World /
Based on a viewpoint management unit that detects the change when the gaze direction is changed, and based on the detected viewpoint position / gaze direction change,
A real-image display object consisting of a three-dimensional CG model in the field of view of the user and a real-image video texture mapped and displayed on the three-dimensional CG model is detected, and the detected three-dimensional shape and installation position of the detected real-image video object are changed. In addition, since a fusion unit for selecting an image to be displayed on the actual image display object and displaying the image is provided, it is possible to use the actual image for constructing a three-dimensional virtual world, and the realism is improved. Since no special mechanism is required for displaying (rendering), there is an effect that versatility is high and portability is excellent.

The fusion unit receives the two-dimensional real image to be mapped and the installation position, maps the real image to the real image display object, and sets the real image display object at the designated installation location. A texture object portion to be set, an object installation position calculation unit for calculating an installation position of the live-action video display object according to a change in a viewpoint position / line-of-sight direction of a user moving in the three-dimensional virtual world, and Since there is a mapping video changing unit for changing a real video to be mapped, if the real video display object on which the still image is pasted is moved only in accordance with the user's viewpoint position / line of sight, is there a real object on the spot? There is an effect that can create a sense of realism as described above. Further, there is an effect of reducing the amount of required video data.

Further, the live-action video display object in the field of view of the user is detected, and the live-action video display object is selected from the live-action video display objects according to the attribute of the user or the execution mode of the entire system. And notifies the object installation position calculation unit and the mapping image change unit that manage the live-action video display object of the viewpoint / gaze direction of the user, and responds to the viewpoint position / gaze direction. And the object management unit that instructs to change the three-dimensional CG model that displays the actual image, the position, and the image displayed by the selected texture object unit, so that only the actual image display object that is in the field of view of the user Is processed as a substance, which has the effect of improving processing performance. In addition, since a plurality of real image display objects can be processed collectively, there is an effect that processing performance can be improved.

In the texture object section which manages the live-action video display objects installed in a three-dimensional virtual world shared by the virtual environments of a plurality of users connected to a network, It is possible to freely change an installation position, a display image, and a three-dimensional CG model for mapping an image independently for each environment, and to set an installation position, a display image, and the three-dimensional CG model between a plurality of virtual environments of the specific user. Since the sharing control such as real-time synchronization can be changed according to the situation, the controllability of the texture object portion is improved.

Further, a distant view is mapped in the texture object section which manages the live-action video display object, and the video to be mapped, the installation position, and the video are displayed even if the viewpoint / gaze direction of the user changes.
A distant view texture object section that does not change the three-dimensional CG model; an intermediate view texture object section that changes only the position of the three-dimensional CG model that displays an image when the viewpoint position / line of sight of the user changes; Since it has a video to be displayed, a three-dimensional CG model for mapping the video, and a foreground texture object section for changing the installation position in accordance with a change in the viewpoint position / line-of-sight direction, according to the required video quality of the live-action video display object The data and the processing method can be changed, the processing efficiency can be improved, and the development efficiency of the content can be improved.

In the object management section,
Calculating a distance between the user's viewpoint position and the real image display object, and mapping an image when the user's viewpoint position / gaze direction changes according to the distance; 3
Switching between a control mode in which only the position of the three-dimensional CG model is changed and a control mode in which the image to be displayed, the three-dimensional CG model for mapping the image, and the installation position are changed in accordance with the change in the viewpoint position / gaze direction of the user. , One real image display object can be used for the foreground and intermediate views, and there is no need to install a plurality of real image display objects in the virtual world, so that processing efficiency is improved and model data of the three-dimensional virtual world is improved. This has the effect that the data size can be reduced.

The camera is mounted on a car or a trolley, and each frame of the video image shot while moving the camera is paired with the viewpoint position / line-of-sight direction of the camera that shot the image of the frame. And a viewpoint control for guiding a user's viewpoint along a movement path of a camera and determining a new viewpoint / gaze direction of a user based on a forward movement distance / reverse movement distance of a path determined by the user. And a texture for calculating an installation position of the real image display object so that the user's line of sight is orthogonal to the image plane of the real image display object with the same direction as the movement of the user's viewpoint and the same amount of movement. Equipped with an object destination calculation unit, allowing you to walk through live-action images, as if walking in the real world Enough can be obtained feel as if the route, there is an effect capable of improving the realism.

Further, a plurality of the live-action video display objects are attached in a horizontal direction in a cylindrical shape, and are installed in a three-dimensional virtual world. Display,
A display management unit to be hidden, and whether the user is to select one of the live-action video display objects and move the viewpoint in a direction perpendicular to the video display surface of the selected live-action video display object, That is, a user monitoring unit that monitors whether the viewpoint position is within a certain range and is moving toward the video display object, and manages a control method of changing the viewpoint / sight direction of the user. Since the viewpoint control management unit is provided, it is possible to freely move in the actual photographed video, and there is an effect that the degree of freedom of the user's viewpoint movement is improved.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of a fusion device according to Embodiment 1 of the present invention.

FIG. 2 is a functional block diagram of a fusion unit of the fusion device according to the first embodiment of the present invention.

FIG. 3 is a diagram showing an example of information stored in a live-action video display object database unit 1500 according to Embodiment 1 of the present invention.

FIG. 4A is a diagram showing three-dimensional coordinates. FIG. 3B is a diagram illustrating a line-of-sight angle and a distance obtained by a viewpoint position / line-of-sight direction detection unit 1900 according to Embodiment 1 of the present invention.

FIG. 5 is a plan view showing a relationship between a user's viewpoint and a real image display object.

FIG. 6 is an explanatory diagram of rotation of a live-action video display object.

FIG. 7 is a diagram showing a processing flow of the fusion device of the first embodiment.

FIG. 8 is a diagram illustrating a display of an image when the inside of a building / passage / tunnel or the like is viewed from outside.

FIG. 9 is a diagram for explaining shooting of a target object from a division point according to the first embodiment of the present invention.

FIG. 10 is a diagram illustrating display of an image when the viewpoint position / sight line direction is moved from the inside of a building / passage / tunnel to the outside.

FIG. 11 is a functional block diagram illustrating a configuration of a fusion device according to a second embodiment of the present invention.

FIG. 12 is a diagram showing a processing flow of the fusion device of the second embodiment.

FIG. 13 is a functional block diagram illustrating a configuration of a device that fuses a real image and a three-dimensional CG model in the fusion device according to the third embodiment of the present invention.

FIG. 14 is a functional block diagram illustrating a configuration of a device that fuses a real image and a three-dimensional CG model in the fusion device according to the third embodiment of the present invention.

FIG. 15 is a functional block diagram illustrating a configuration of a fusion device according to a fourth embodiment of the present invention.

FIG. 16 is a diagram supplementarily describing the configuration of the fusion device according to the fourth embodiment of the present invention.

FIG. 17 is a diagram for describing display of a real image display object according to Embodiment 4 of the present invention.

FIG. 18 is a functional block diagram illustrating a configuration of a fusion device according to a fifth embodiment of the present invention.

FIG. 19 is a diagram showing a processing flow of the fusion device of the fifth embodiment.

FIG. 20 is a functional block diagram illustrating a configuration of a fusion device according to Embodiment 6 of the present invention.

FIG. 21 is a diagram showing a configuration of a texture object unit 1000 according to Embodiment 7 of the present invention.

FIG. 22 is a diagram illustrating the use of a real image display object for a distant view, an intermediate view, and a near view according to the seventh embodiment.

FIG. 23 is a functional block diagram illustrating a configuration of a fusion device according to a seventh embodiment of the present invention.

FIG. 24 is an image diagram for explaining the operation of the intermediate scene texture object unit according to the seventh embodiment.

FIG. 25 is an explanatory diagram of a distant view, an intermediate view, and a near view accompanying movement of a user according to the seventh embodiment.

FIG. 26 is a conceptual diagram of a conventional technology.

FIG. 27 is a functional block diagram of a conventional technique.

[Explanation of symbols]

99 virtual world construction unit, 100 input unit, 101 viewpoint management unit, 102 rendering unit, 105 fusion unit, 1
06 display unit, 1000 texture object unit,
1100 Object setting position calculation unit, 1200 mapping video change unit, 1300 video database unit,
1400 shape database section, 1500 texture database section, 1600 three-dimensional CG model database section, 1700 real image display object database section, 1900 viewpoint position / view direction detecting section, 200
0 viewpoint live-action database unit, 2100 viewpoint control unit,
2200 Texture object destination calculation unit, 30
00 CG model fusion unit, 3100 object selection unit, 4000 display management unit, 4100 user monitoring unit, 4200 viewpoint control management unit, 4300 three-dimensional virtual world, 4310 real-world video display object, 432
0 viewpoint, 4330 gaze vector, 4340 branch, 5000 object management unit, 6000 shared control unit, 7000 control mode change unit, 9000 distant view texture object unit, 9100 intermediate view texture object unit, 9200 close view texture object unit, 9300 building, 9400 , 9500 live-action video display object.

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁶ , DB name) G06T 1/00-17/50

Claims (3)

(57) [Claims] An object management unit and a sharing control unit described below.
Three-dimensional computer graph characterized by comprising:
("Computer Graphics"
(CG)) Live-action image fusion device (1) (a) A user who walks through a three-dimensional virtual world
A viewpoint management unit for detecting a change in the viewpoint of the object ; and (b) a planar object constituted by a three-dimensional CG model.
Live video display object that maps video to
Object information that identifies the object,
The viewpoint position and line-of-sight direction of the user viewing the three-dimensional virtual world
Store the positional relationship with the live-action video display object
Based on the change of the viewpoint detected by the live-action image display object database unit and the viewpoint management unit
Obtains a field of view, and a live-action image display object in the field of view
Object information of the above-mentioned live-action image display object
Viewpoint position gaze direction detection by searching database part
And the object information detected by the viewpoint position / gaze direction detection unit.
Based on the information, each part seen when looking at the real thing from the above viewpoint
Minute ratio and each part of live-action video display object
In the position where the size ratio of the
Object setting to calculate the installation position of the plane object
The position calculation unit and the object information detected by the viewpoint position gaze direction detection unit.
The shape of the plane object and the plane
Mapping video that determines the video that fuses with the object
An image changing unit and the image determined by the mapping image changing unit
Plane object of the shape determined by the mapping video changing unit
Into the object setting position calculation unit
Plane object that fuses the above image with the installation position calculated by
Object to create a live-action video display object
And a fusion part with a texture object part
The object provided for each user connected to the network
Project management unit, be shared among a plurality of users (2) Network Connection
A live-action video display object in the 3D virtual world
Lok Si, particular Stock Video display Oh Buje, which is the registered
For each extract, specific viewpoint position to each user, using the line-of-sight direction
The location of the live-action video display object
Video and shape to be individually determined for each user
Between the separate use mode and the predetermined users connected to the network,
The installation position of the image display object, the video to be mapped,
Select one of the modes that share one of the shapes and share and use it synchronously as appropriate according to the situation
A shared control unit that controls the display by displaying. 2. The three-dimensional CG live-action image fusion device comprises:
3D CG model data for registering a 3D CG model
Database and the above-mentioned live-action video display object in a 3D virtual world
Is present in the blind spot, and
User's point of view if no video display object is installed
The three-dimensional CG model seen from
An object selection unit selected from the database unit and a three-dimensional CG model selected by the object selection unit.
The three-dimensional CG model seen from the viewpoint of the user
And merge it into a live-action video display object
Live image CG model that overwrites the live image
2. The combination according to claim 1, further comprising:
3D CG live-action image fusion device. 3. 3, characterized in that it comprises the following elements
Dimensional CG live-action image fusion device (a) shooting path corresponding to predetermined viewpoint position and line-of-sight
And the viewpoint position where the above live-action video was shot
And a viewpoint real-photographing database unit storing a plurality of viewpoint directions, and (b) at least a viewpoint position and a viewpoint in a three-dimensional virtual world.
An input unit for changing any of the line directions; (c) a viewpoint position and the line of sight changed by the input unit
The direction and the viewpoint stored in the viewpoint live-action database
Follow the shooting path determined by the point position and gaze direction
So that you can only move forward or backward in the shooting path
And bind so that you cannot freely change the direction of travel
To move forward on the route determined by the user.
From the distance / reverse travel distance, the new viewpoint position controlled above
Position and line-of-sight direction, and
Objects within a certain range and
Above the bunch of user perspectives when moving towards
Viewpoint control unit to start a strapping, (d) the same direction and the same amount of movement of the point of view of the user
In such a way as to follow the movement of the user's viewpoint,
Visual direction of user's gaze direction and live-action video display object
A live-action projection that maps live-action footage so that
A technique for calculating the installation position of the destination of the image display object
Texturizing object destination calculation unit, (e) a new user to the viewpoint control unit has determined viewpoint position /
Database of live-action video images viewed from the line of sight
Mapping image changing section for more selective, by (f) the textured object destination calculator
Install a live-action video display object at the calculated installation position
The actual video selected by the mapping video changing unit
Text mapping to live-action video display objects
(G) Each of the real shot images shot in the above plurality of shooting paths
One from each live-action video display object corresponding to
Select the viewpoint position input from the input unit.
Moving toward the live-action image display object
And that the viewpoint position is within a predetermined area.
A user monitoring unit that monitors the presence of the camera;
Each live-action video display object corresponding to
Laminated in the horizontal direction, the specified
The above plurality of live-action video tables according to the viewpoint position in the area of
Predetermined live-action video display object among display objects
To select the live-action video display object.
Select the corresponding shooting route and select the shooting route
A display management unit that selects and displays live-action images corresponding to.