JP2003264740A - Observation scope - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an observation scope capable of quickly and surely displaying the image of things such as remains in an actual view. <P>SOLUTION: The observation scope for appropriately processing the video images of a variety of view objects such as a landscape and providing them for a user comprises: a photographing means for photographing the video images of the view object; a storage means for storing the virtual video image of the thing whose present position is specified in the view object in relation to the view object; and a video synthesis means for extracting the corresponding virtual image from the storage means and synthetically displaying it for the photographed video image of the view object. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a viewing mirror capable of synthesizing and displaying an actual view image and a virtual image.

[0002]

BACKGROUND OF THE INVENTION It is a pleasure not only for history lovers but also for the general public to enjoy the view while feeling the eternal history of the visited land. For example,
If it is a land pattern such as Asuka and Nara that is the stage of ancient history, we will use the materials related to that period and compare it with the Yamakawa plants of the existing structures and topography such as temples and shrines and temples. That is the situation you can imagine. People imagine and enjoy the historical events that occurred in the distant past with some sense of reality.

In response to such a situation, measures have been taken to inform people of the relationship between the present situation and the ruins more clearly and simply. For example, at archaeological museums and sightseeing spots, panels and pamphlets that display the location and shape of archaeological sites are displayed and distributed on aerial photographs of the location of the archaeological site, or a diorama for the area where the archeological site is located is created. There is a case where the restored ruins are exhibited.
Of course, these expression forms may be broadcast to the user in place of video images such as videos.

[0004]

[Problems to be Solved by the Invention] It may be possible for people who visit the archaeological museums, etc., to inflate a certain restoration image of the archaeological sites, etc. by viewing the exhibition panels, pamphlets, explanation videos, etc. unknown. However, in any case, it will force you to think of the actual local landscape and the image of the ruins alternately with your head, which is difficult and troublesome for ordinary people who are not accustomed to image work in their heads. It was also a task. Moreover, the image obtained as a result of holding one image and simultaneously memorizing and overlapping the other images tends to be vague, and the scale of the virtual image and the positional relationship on the terrain are ambiguous, making it more realistic. Poor, and tended to be fixed to the image of the line of sight from a certain direction. Even if the live-action scenery and the restored image of the ruins are combined and provided with the above-mentioned video image, it only cuts out a section of the landscape and the user can freely view the image from any angle. The function of being able to browse was not provided.

Therefore, the present invention has been made in view of such conventional problems, and an object thereof is to provide a viewing mirror capable of displaying images of objects such as ruins in an actual view quickly and surely. To do.

[0006]

A perspective mirror of the present invention that achieves the above object is a perspective mirror that appropriately processes images of various view objects such as landscapes and provides them to a user. And a means for storing a virtual image of an object whose existence position is designated in the view target, and a means for combining and displaying a virtual image corresponding to the captured image of the view target. Is characterized by.

According to a second aspect of the present invention, in the first aspect of the present invention, a displacement detecting means for detecting a displacement of the photographing direction from the original position in the photographing means, and a photographed image are combined based on a detection result of the displacement in the photographing direction. And a composite adjusting means for changing the display mode of the virtual image.

A third aspect of the present invention is the observation mirror according to the second aspect, wherein the installation position is fixed, and the original position is constant.

According to a fourth aspect of the present invention, in the second or third aspect of the invention, a sensor for detecting at least one of a horizontal angle and a vertical angle for detecting the displacement from the original position in the photographing direction as an angle change is provided. The combination adjusting unit changes the display angle of the virtual image to be combined and displayed on the captured image according to the angular displacement detected by the detection sensor.

According to a fifth aspect of the present invention, in any one of the first to fourth aspects of the present invention, when a zoom photographing instruction is received, zoom photographing means for photographing the image of the view object by zoom photographing, and a virtual image combined with the zoom image. And zoom adjusting means for performing zoom processing and composition according to the zoom magnification of the view target.

According to a sixth invention, in any one of the first to fifth inventions, an illuminance sensor for detecting the illuminance of the view object at the time of photographing, and a virtual image to be combined with the photographed image based on the detection result of the illuminance at the time of photographing. And an illuminance adjusting means for changing the display illuminance of the image.

In a seventh aspect of the present invention, in any one of the first to sixth aspects of the present invention, a table is provided in which a relationship between various conditions such as shooting date and weather and the sunshine conditions is set. It is characterized by comprising a sunshine recognition means for recognizing a condition and a shadow adjusting means for adjusting a display mode of a shadow in which a virtual image should appear in a captured image based on the recognized sunshine condition.

In an eighth invention according to any one of the first to seventh inventions, the existence position is specified in the view object, and the thing which is the origin of the virtual image exists in the view object in the past. It is characterized in that the restored image of the building or the group of buildings is a virtual image.

A ninth invention is a program for operating the view mirror according to any one of the first to eighth inventions, wherein a photographing step for photographing an image of a view target and a position where the view target exists is specified. A storage step of storing a virtual image of the object to be viewed in association with the view target, and a video combining step of extracting a virtual image corresponding to the captured image of the view target from the storage means and displaying the composite image. Invention is a computer-readable recording medium in which the perspective operation program of the ninth invention is recorded.

A tenth aspect of the present invention is a computer-readable recording medium in which the perspective operation program described in the ninth aspect is recorded.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 is an explanatory diagram showing the technical concept of the viewing mirror in this embodiment. The background of the present embodiment assumes a situation in which CG (Computer Graphics) of a restored structure (past palace, temple, etc.) estimated from the ruins is applied as a virtual image to be combined with a captured image of an actual view target. . Of course, the scope of application of the present invention is not limited to the present embodiment, and it does not exist at present regardless of the past and future, or even if it exists, a thing having a different form (a "object" such as a building or a device). The present invention can be applied to any situation in which a person, a natural phenomenon such as weather and a volcanic activity, etc. are combined with a photographed image of the current view and provided to the user. Needless to say.

=== Summary of technological ideas === "Virtual Reality" Mixed Reality, which is a new technology system developed from virtual reality (VR)
The present invention is configured based on a technique related to y (hereinafter referred to as MR). Fusion of the virtual space, which has been represented by real-time CG, etc., and the real world as it is,
The perspective mirror 10 of the present invention provides the user with more complete information by complementing both. It should be noted that in the figure, it is described as an MR scope in the sense that it is a perspective mirror using MR technology.

The MR scope 10 includes a live-action image 20 of an excavation site, ruins, etc. (view target) acquired by a video camera 11 included in the MR scope 10, for example, an internal computer 12 included in the MR scope 10 or a network-connected MR.
It is imported to the server that controls and manages the scope 10 group. At the same time, the image data of the CG image 30, which is a virtual image obtained by reconstructing the past things (palaces that are the origin of ruins, etc.), is read from the video RAM of the internal computer 12 or the like, and is combined with the above-mentioned live-action image for real time To MR
It is output to the display 13 in the scope 10. It should be noted that, as long as it is possible to combine the live-action video 20 and the virtual video 30, it is possible to appropriately employ configurations and procedures other than those described above.

In addition, biaxial directions of the video camera 11 (eg:
The motion of the yaw angle and pitch angle) is detected by the angle sensor 14 (displacement detection means), and the CG image is also moved at the same angle according to this angle, so that the VR in the live-action image is detected.
Build space in a more natural way. As a result, it is possible to suppress the discomfort felt by the user who views the composited image, and it is possible to more easily understand the relationship between the archeological space and the existing state of the building that may have existed there. Of course, as the CG image, as shown in FIG. 1, for example, an animation of human movements related to ancient historical events and life styles may be used, or a restored image of a building that does not exist (or a completion plan of a building, etc.) (Including video)
Still images are also included.

=== Perspective Mirror System === FIG. 2 is an overall system diagram including the perspective mirror in this embodiment. The system 200 is capable of displaying the real image 20 or the virtual image 30 and switching the display thereof. further,
When the user selects a specific point on the composite image displayed on the display 13 of the MR scope 10 or on the display screen with a pointer such as a cursor, the composite image includes the matter, for example, the origin of the building. It is also possible to display or output the multimedia information (audio, video information, etc.) related to the history of characters, historical facts, etc. on the screen. Hereinafter, the functions and the like of the system 200 will be described.

The MR image generating function 12a is a function provided in the internal computer 12 of the MR scope 10. This function is applied to the live-action image 20 obtained by the video camera 11.
And a virtual image 30 created and stored in advance in real time. Specifically, the video camera 1
1 and the virtual image 30 extracted from the multimedia information database 51 constituted by the storage resources such as the video RAM 12b or the hard disk.
And a function of performing position / angle adjustment based on sensor information from the angle sensor 14 and a function of adjusting the image quality of the real image 20 and the virtual image 30.
These functions will be described later.

The display function 53 is the MR image generation function 12a.
The composite video 100 obtained from the above and the multimedia information added to the composite video 100 (multimedia information database 51
(Origin) is used as input data, and the data is controlled and output to the display 13 according to the user's operation or the like via the GUI. This function is to display the composite image 100 output from the MR image generation function 12a on the display 13 based on the FLT file and the ADF file in which the three-dimensional shape of the virtual image 30 is described in advance, and the three-dimensional image processing is performed in real time. Will be done.

In addition, the multimedia information database 5
From 1, the function of reproducing and displaying the moving picture information selected by the user by the information selecting function 52 to which various user interfaces can be applied is also exhibited. Specifically, the corresponding AVI file is extracted, and the moving image is reproduced and output by the reproducing application. Similarly to this,
It also has a sound playback function that extracts and plays audio files such as WAV files, and a character / still image display function that extracts and outputs various characters such as SWF files and still image files.

As described above, the composite image 100 of the real image 20 and the virtual image 30 or the multimedia information added to this is generated, and based on the control information such as yaw and pitch angle of the sensor information, The video reproduction and the output form to the display 13 are controlled.

As the display 12, a lens, a half mirror, etc. are arranged immediately in front of the eyeball, and a very small display device such as a liquid crystal display is directly looked at through the display, thereby displaying a display without using a large display device. Head Mounted Displ
ay) may be used. Also, as other technologies for displaying images, anaglyph method (complementary color filter method), polarizing glasses method, time division glasses method, head mounted display (HMD), stereo pair image, parallax method, lenticular method It is also possible to adopt stereo graphics technology that can adopt various methods such as. In addition, a data globe can be adopted as an input device for the MR scope 10 to enhance the user's immersive feeling in the virtual space. === Main Flow === FIG. 3 is a flowchart showing a video processing flow of the perspective mirror in the present embodiment. First, the view object is photographed by the video camera 11, and the video camera image, that is, the live-action image 20 is captured (capture processing) in the internal computer 12 or the like. This capture is processed in a situation where the user of the MR scope 10 as a viewing mirror normally looks into the scope by looking into the scope. At this time, the illuminance information of the photographed image 20 is recognized, and the capturing process is performed together.

In parallel with the above-mentioned capturing process, in order to synchronize the real image 20 and the virtual image 30 of the video camera 11 in real time, the displacement such as the yaw angle and the pitch angle of the camera 11 can be detected by a detecting device such as a rotary encoder. Get exactly. Generally, the number of drawing frames is 30
If the real video 20 and the virtual video 30 are synchronized so as to satisfy the frame / second, the synchronization can be achieved without a feeling of strangeness. Therefore, for example, an angle sensor, an acceleration sensor,
Using various attitude sensors such as rotary encoder,
Alternatively, camera posture information (displacement from the original position) with high accuracy may be acquired in combination. Of course, the means for detecting the posture information of the camera, that is, the information of the displacement from the original position is not limited to this.

Then, the orientation of the camera in the composite image 100 is determined using the posture information obtained by the above-mentioned sensors, in this case, the angle information, and the extraction / extraction is performed based on this result.
The edited real-time CG image 30 and the captured live-action image 20 are combined. At this time,
The following various adjustment processes are performed.

1. Position / angle alignment at start-up and correction of accumulated error Even if the posture information is obtained by the sensor once, the error of the sensor is accumulated when the MR scope 10 is started or continuously used. It is expected that the positional relationship will shift. As a method for correcting such initial error and accumulated error, the video camera 1
The angle of 1 is automatically changed by a slight angle, and the initial position and angle of the camera are automatically determined from the deviation of the two or more photographed images 20 obtained at that time by the edge extraction technology, the perspective matching technology, or the like. Can adopt technology. Further, when the continuity of the photographed image 20 is impaired at regular time intervals or due to a sudden camera angle change, etc., the accumulated error is calculated using the camera position obtained by slightly changing the angle of the video camera 11. It is also possible to adopt a coping method to correct the.

2. Image quality adjustment position and angle of real image and virtual image Real image 20 and virtual image 30
If the two are overlapped as they are, the composition result is generally unnatural and gives a feeling of strangeness due to the difference in brightness of sunlight and lighting. Therefore, in order to obtain a more natural synthesis result, it is necessary to correct the difference in brightness and light source direction between the real image 20 and the virtual image 30. Therefore, the sun position is calculated from the position where the MR scope 10 is installed, the date, and the time, and the brightness of the virtual image is adjusted using an illuminometer or the like to reduce the sense of discomfort. Specifically, the brightness and light source direction of the photographed image 20 are approximated to the brightness and light source direction set for the virtual image 30. Further, when the MR scope 10 is installed indoors, a method of estimating the light source direction, the brightness, and the light source shape from the change in the brightness by using, for example, a marker can be adopted.

3. Adjusting the Front-and-Back Relationship between the Real Image and the Virtual Image The front-and-rear relationship between the real object existing in the real image 20 and the virtual object in the superimposed virtual image is determined to adjust the visibility of both objects. . The position of the real object or the distance from the MR scope 10 is recognized by using the geographic information regarding the existing area of the view object or the real-time distance measurement, and the recognition information is compared with the position information related to the virtual image. By doing so, it becomes possible to adjust mutual concealment.

The composite image 100, which has been appropriately adjusted in this way and then combined, is displayed on the display 12 of the MR scope 10. For example, as shown in FIG. 5, a live-action image 20 that is actually a rural scene, and a reconstructed image of ruins that may have existed here in the past (CG image of a temple), that is, a virtual image 30. Will be presented to the user looking into the MR scope 10. FIG. 6 is an enlarged image of the composite image 100 at that time. In the modern landscape with utility poles, modern houses, etc., the restored CG of the shrine that has existed here in the past, for example, the Asukaita lid shrine, is beautifully combined. That is,
The user of the MR scope 10 can easily view an image of an ancient relic or the like in accordance with the actual view, with an excellent sense of reality, with the convenience of using an ordinary viewing mirror. .

FIG. 7 is an explanatory diagram showing a screen for providing text information and a movie in this embodiment. The MR scope 10 not only provides the user with the composite image 100 in which the live-action image 20 and the virtual image 30 are combined, but also, for example, various kinds of text information describing the ruins, narration, and animation explaining the same. It can also provide multimedia information. A list of the provided materials is shown in FIG. In this example, materials such as video, VR, CG animation, CG still image, photograph, text screen, narration, and BGM are aggregated together with their playback time. These materials are appropriately selected in response to an instruction from the user of the MR scope 10 and provided for reproduction.

For example, assume that the restored image represented in the composite image 100 shown in FIG. 5 is Asukaji. When the user who is looking at it selects the image of Asukaji using an appropriate user interface, the selected event is sent to the internal computer 12 of the MR scope 10. The internal computer 12 displays the introduction screen 7 of the corresponding Asuka temple from the multimedia information database 51.
00 is extracted and output to the display 13. Therefore,
For example, if the selection button for starting the explanation is pressed, the text information related to Asukaji is extracted from the multimedia information database 51 and reproduced. It is also possible to play CG animations, video images, and narrations. When these reproductions are completed, or if an instruction from the user is received on the way, the screen returns to the normal output screen of the composite video 100.

As described above, according to the present invention, the scale of a building or the like and the positional relationship on the terrain are accurate without the work of envisioning the actual local landscape and the virtual image of the ruins, etc. alternately. With this, it is possible to provide users with highly realistic composite images. Moreover, the composite video can be taken from any angle of the user, and a service with a high degree of freedom can be provided.

All the users are computer-generated CG and VR.
Unlike the situation of just looking at, you can see the restored image of the palace, castle, etc. that existed in the past in the actual historical landscape such as ruins in the natural environment with a simple and convenient way like a telescope on the observatory. Is possible. This means that you can "experience" the past in a real-scale space.

In addition, since the base of the present invention is fixed and its position in the view object is fixed, the information elements for various controls such as the attitude information of the video camera are limited. It is possible to inexpensively construct a sensor and its system for acquiring the attitude information. This effect makes it easy to perform the degree of overlap between the real shot image and the virtual video, that is, the concealment process when the real shot image and the virtual video are combined, and a simple and efficient system configuration can be obtained.

=== Other Applicable Range of the Present Invention === The present device is a media device suitable for edutainment purposes for obtaining various information while enjoying a three-dimensional real-time image. Therefore, various application ranges are conceivable in addition to the above-described embodiment in which past buildings in the archeological area are restored by CG and provided. As an example, we present the four seasons of the scenery in a tourist city, express the image of the completion of the building, and use it for explaining the neighborhood and attracting tenants at the planning stage, and the city observation at the observation deck on the upper floors of the high-rise building. It can be used for various situations such as business. Listed below.

1) Sightseeing: providing restoration images of historical buildings such as ruins, and providing guidance information on tourism-related facilities. 2) Events: Expositions, provision of guidance information at various events, etc. 3) Museum: A three-dimensional exhibition of cultural assets at the Digital Museum. 4) Urban observation: View of the city on the upper floors of the skyscraper and provision of information on the present, past, and future cities. 5) Architecture / Urban Development: Architectural design study, neighborhood explanations at the planning stage, attracting tenants, etc. 6) Town planning: Providing information on local and regional information by local governments and local shopping districts.

[0039]

According to the present invention, it is possible to provide a viewing mirror capable of displaying images of things such as ruins in an actual view quickly and reliably.

[Brief description of drawings]

FIG. 1 is an explanatory diagram showing a technical concept of a viewing mirror in the present embodiment.

FIG. 2 is an overall system diagram including a perspective mirror in the present embodiment.

FIG. 3 is a flow chart showing a video processing flow of the perspective mirror in the present embodiment.

FIG. 4 is a diagram showing a list of materials to be used, which are supposed to be used in the scope according to the present embodiment.

FIG. 5 is an explanatory diagram showing a combination of a current landscape and a CG image in the present embodiment.

FIG. 6 is an explanatory diagram showing an example of a browsed image of a user according to the present embodiment.

FIG. 7 is an explanatory diagram showing a screen for providing text information and a movie according to the present embodiment.

[Explanation of symbols]

10 Observation mirror, MR scope 11 Photographing means, video camera 12 Internal computer 13 Display 14 Displacement detection means, angle sensor 20 Shooting video, live-action video 30 virtual images, CG 100 composite images

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B050 AA06 AA08 BA08 BA11 DA01                       EA19 EA24 EA27 FA02                 5C023 AA11 AA37 AA38 BA11 CA03                       DA04                 5C054 CA04 CD03 EA01 EA05 FE14                       HA00

Claims (10)

[Claims] 1. A view mirror for appropriately processing images of various view objects such as landscapes and providing the same to a user, wherein a photographing means for photographing the image of the view object and its existence position are specified in the view object. Storage means for storing a virtual image of the object to be viewed in association with the view target, and image combining means for extracting a virtual image corresponding to the captured image of the view object from the storage means and displaying the combined image. Is a viewing mirror. 2. A displacement detecting means for detecting a displacement of an image capturing direction from an original position in the image capturing means, and a composition for changing a display mode of a virtual image to be combined with a captured image based on a detection result of the displacement in the image capturing direction. The viewing mirror according to claim 1, further comprising: adjusting means. 3. A viewing mirror whose installation position is fixed,
The observation mirror according to claim 2, wherein the original position is constant. 4. An angle sensor which detects at least one of a horizontal angle and a vertical angle for detecting a displacement of the photographing direction from the original position as an angle change, and an angle detected by the detection sensor by the synthesis adjusting unit. The viewing mirror according to claim 2 or 3, wherein a display angle of the virtual image to be combined and displayed on the captured image is changed according to the displacement. 5. When a zoom shooting instruction is received, zoom shooting means for zoom shooting the image of the view target and a virtual image to be combined with the zoom image are zoomed and combined according to the zoom magnification of the view target. A zoom adjusting means for adjusting the viewing angle, and the viewing mirror according to any one of claims 1 to 4. 6. An illuminance sensor for detecting the illuminance of a view target at the time of shooting, and an illuminance adjusting means for changing the display illuminance of a virtual image to be combined with the taken image based on the detection result of the illuminance at the time of shooting. The observing mirror according to any one of claims 1 to 5, characterized in that: 7. A sunshine recognition means for recognizing a sunshine condition from the various conditions at the time of shooting, comprising a table in which a relationship between various conditions such as shooting date and weather and the sunshine condition is set, and based on the recognized sunshine condition The shadow adjusting means for adjusting a display mode of a shadow in which a virtual image should occur in a captured image, and the perspective mirror according to any one of claims 1 to 6. 8. The thing whose existence position is specified in the view target and which is the origin of the virtual image is a building or group of buildings that existed in the past in the view target, The perspective image according to any one of claims 1 to 7, wherein a restored image of the group of buildings is a virtual image. 9. A program for operating the observing mirror according to claim 1, wherein a photographing step for photographing an image of a view target, and an existence position thereof is designated in the view target. A storage step of storing the virtual image of the virtual image in association with the view target, and a video synthesis step of extracting a virtual image corresponding to the captured video of the view target from the storage means and displaying the synthesized virtual image. Speculum operation program. 10. A computer-readable recording medium in which the perspective operation program according to claim 9 is recorded.