JP2009301350A - Virtual image display system and virtual image display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a virtual image display system that creates augmented reality in a simple manner without use of a marker or an attitude sensor. <P>SOLUTION: A virtual image display system 1 includes: an image pickup unit 20; a screen 21 disposed in a prescribed positional relationship with the image pickup unit 20; an image object storing unit 30 storing an image object; a virtual image generator 31 that reads a prescribed image object from the image object storing unit 30, in association with an actual image taken by the image pickup unit 20, and generates a virtual image from an arbitrary viewpoint on the basis of a reference virtual image which the image object is viewed from a prescribed viewpoint; and a virtual image display unit 32 displaying the virtual image on the screen 21. In the virtual image display system 1, the virtual image generator 31 is configured so as to generate the virtual image corresponding to a displacement of the viewpoint of the image pickup unit 20 calculated from a feature point of the actual image on the basis of the reference virtual image from a viewpoint of the image pickup unit 20 at a prescribed time point. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a virtual video display system and a virtual video display method for generating a virtual video corresponding to a real video shot by an imaging unit and displaying the virtual video on a screen.

  A virtual video display system has been proposed in which a video captured by an imaging device such as a video camera is combined with a virtual video generated by a virtual video generation device such as a computer and displayed on a display device such as a head mounted display. ing. Such a technique is called mixed reality or augmented reality.

  In a virtual video display system, an actual object displayed on a video shot by an imaging device and a virtual video generated by a virtual video generation device often have a subordinate positional relationship. The virtual video is synthesized with a close positional relationship with the real object and displayed on the display device.

  In Patent Document 1, as such a virtual image display system, a work table having a first plurality of markers arranged at known positions and a worker's head posture are detected in order to detect the head posture of the worker. An orientation sensor, a camera set so that at least one of the first plurality of markers is in a field of view, and an image signal from the camera is processed to Based on the detection unit that tracks an arbitrary marker and detects the coordinate value of the tracking marker, the coordinate value of the tracking marker obtained by the detection unit, and the head position and orientation signal from the position and orientation sensor, Calculation means for calculating a viewpoint position and orientation signal representing the position and orientation of the worker's viewpoint, and generation means for generating a virtual image for presenting mixed reality at the viewpoint position corresponding to the viewpoint position and orientation signal And ingredients Mixed reality presentation apparatus is disclosed which is characterized in that.

In the mixed reality presentation apparatus disclosed in Patent Document 1, a virtual image is generated by a generation unit at a viewpoint position of an operator on a work table corresponding to a position and orientation signal obtained by a calculation unit.
JP-A-11-136706

  In order to use the mixed reality presentation device disclosed in Patent Document 1, it is necessary to provide a marker on a real object such as a work table photographed by a camera and to provide a posture sensor to the worker. However, when there is no subordinate positional relationship between the real object and the virtual video, in other words, when no strict constraint is required on the positional relationship between the object and the virtual video, the marker and the posture sensor are not necessary.

  When it is not necessary to synthesize and display a virtual image on a real object in a close positional relationship, and it is only necessary to display the virtual image at a position within at least a predetermined range of the object, an unnecessary marker or sensor is required. Such a mixed reality presentation device is not preferable because it is over-spec. Also, from the viewpoint of aesthetics, it is not preferable to provide an object with an unnecessary marker.

  In view of the above-described conventional problems, an object of the present invention is to provide a virtual video display system and a virtual video display method that easily create augmented reality without using a marker or a posture sensor.

  In order to achieve the above-mentioned object, the first characteristic configuration of the virtual image display system according to the present invention is as described in claim 1 of the document of the claims, and includes an imaging unit and a predetermined positional relationship with the imaging unit. Corresponding to the actual video shot by the imaging unit, a screen that displays virtual video, a video object storage unit that stores video objects defined by virtual video reference data, and A virtual video generation unit that reads a predetermined video object stored in the video object storage unit and generates a virtual video viewed from an arbitrary viewpoint based on a reference virtual video viewed from the predetermined viewpoint; A virtual video display system comprising a virtual video display unit for displaying the virtual video generated by the virtual video generation unit on the screen, The video generation unit generates a virtual video corresponding to the displacement of the viewpoint of the imaging unit calculated from the feature points of the real video based on the reference virtual video viewed from the viewpoint of the imaging unit at a predetermined time. In the point.

  The virtual image generator generates a reference from the viewpoint of the imaging unit at a predetermined time corresponding to the displacement of the viewpoint of the imaging unit calculated based on the displacement of the feature point detected in the actual image captured by the imaging unit. The virtual video viewed from the virtual video is changed and generated. Accordingly, the posture of the position of the virtual video is changed by the virtual video generation unit according to the displacement of the viewpoint of the imaging unit detected based on the actual video.

  In the second feature configuration, as described in claim 2, in addition to the first feature configuration described above, the virtual video generation unit detects the switch signal that activates the display of the virtual video when the switch signal is detected. Based on the reference virtual video viewed from the viewpoint of the imaging unit, a virtual video corresponding to the displacement of the viewpoint of the imaging unit calculated from the feature points of the actual video is generated.

  The point in time when the virtual video of the video object viewed from the predetermined viewpoint is displayed as the reference virtual video viewed from the viewpoint of the imaging unit at the predetermined time is set as the time when the switch signal for starting the virtual video display is detected. .

  The first characteristic configuration of the virtual video display method according to the present invention is as described in claim 3, corresponding to the shooting step of shooting a real video with the imaging unit, and the real video shot in the shooting step, A virtual video generation step of reading a predetermined video object stored in the video object storage unit, and generating a virtual video viewed from an arbitrary viewpoint based on a reference virtual video viewed from the predetermined viewpoint; A virtual video display method comprising a virtual video display step for displaying a virtual video generated in the virtual video generation step on a screen, wherein the virtual video generation step is viewed from a viewpoint of the imaging unit at a predetermined time in the virtual video generation step. Based on the reference virtual video, a virtual video corresponding to the displacement of the viewpoint of the imaging unit calculated from the feature points of the real video is generated.

  In the second feature configuration, in addition to the first feature configuration described above, at the time of detecting a switch signal for starting display of virtual video in the virtual video generation step, the second feature configuration is described above. Based on the reference virtual video viewed from the viewpoint of the imaging unit, a virtual video corresponding to the displacement of the viewpoint of the imaging unit calculated from the feature points of the actual video is generated.

  As described above, according to the present invention, it is possible to provide a virtual video display system and a virtual video display method that easily create augmented reality without using a marker or an attitude sensor.

The virtual video display system according to the present invention will be described below.
As shown in FIG. 1, the virtual video display system 1 includes a head mounted display (hereinafter referred to as “HMD”) 2 and a computer terminal 3.

  The HMD 2 includes an imaging device 20, a screen 21, and a projection device 22. The imaging device 20 is an imaging unit of the present invention. The imaging device 20 is composed of, for example, a small video camera, and inputs a captured image of a real space (hereinafter referred to as “real image”) to the computer terminal 3. The screen 21 is arranged in a predetermined positional relationship with the imaging device 20 and displays a virtual video (hereinafter referred to as “virtual video”). The projection device 22 displays the virtual image input from the computer terminal 3 on the screen 21.

  In the virtual image displayed on the screen 21 by the projection device 22, a video object composed of a two-dimensional or three-dimensional virtual object is projected. The virtual video reaches the eyes of the wearer of the HMD 2 as a video synthesized with the real video that has passed through the screen 21. Therefore, it appears to the wearer that the video object exists in the real space.

  The computer terminal 3 includes a video processing unit 34, a viewpoint displacement calculation unit 33, a video object storage unit 30, a virtual video generation unit 31, and a virtual video display unit 32.

  The video object storage unit 30 includes an area partitioned by a storage unit such as a hard disk provided in the computer terminal 3. The video object storage unit 30 stores video objects defined by virtual video reference data. Specifically, the video object is stored in the video object storage unit 30 in the form of vector data.

  The video processing unit 34, the viewpoint displacement calculation unit 33, the virtual video generation unit 31, and the virtual video display unit 32 are controlled by a CPU that executes an operation program such as an image processing application stored in the storage unit, and the CPU. It is a functional block related to the present invention that is embodied by a memory, a storage unit, and an image processing hardware device (for example, an image processing board on which an image processing circuit is mounted).

  The virtual video generation unit 31 reads a predetermined video object stored in the video object storage unit 30 corresponding to the actual video shot by the imaging device 20 and views the video object from a predetermined viewpoint. A virtual video viewed from an arbitrary viewpoint is generated based on the video.

  Specifically, when a switch (not shown) that activates display of virtual video provided in the HMD 2 is operated, the switch signal is input to the computer terminal 3. When the switch signal is detected, the virtual video generation unit 31 reads a predetermined video object corresponding to the actual video captured by the imaging device 20 from the video object storage unit 30, and renders the video object viewed from a predetermined viewpoint. Thus, a virtual video is generated, and the virtual video is set as a reference virtual video for the viewpoint of the imaging device 20. For example, a virtual video corresponding to a video object with a front-facing posture is generated, and the virtual video is set as a reference virtual video for the viewpoint of the imaging device 20 at the time when the switch signal is detected.

  The virtual video display unit 32 displays the generated reference virtual video at a preset position such as the center of the screen 21 or the lower right.

  When the viewpoint of the imaging device 20 is displaced, the virtual video generation unit 31 detects the feature of the real video based on the reference virtual video viewed from the viewpoint of the imaging device 20 at the time of detecting the switch signal that activates the display of the virtual video. A virtual image corresponding to the displacement of the viewpoint of the imaging device 20 calculated from the points is generated.

  In other words, the virtual video generation unit 31 corresponds to the displacement of the viewpoint of the imaging device 20 and the video object having the position and orientation displayed in the reference virtual video viewed from the viewpoint of the imaging device 20 at the time when the switch signal is detected. Is generated from the current viewpoint of the imaging device 20.

  The virtual video display unit 32 displays the virtual video generated by the virtual video generation unit 31 on the screen 21 via the projection device 22.

  Accordingly, when the switch for starting the display of the virtual video is operated, the virtual video display unit 32 displays the virtual video in which the video object is displayed with the posture facing the front in the center of the screen 21, and then the imaging device 20 When the viewpoint is displaced, a virtual image obtained by viewing the image object at the position and orientation viewed from the viewpoint of the imaging device 20 at the time when the switch signal is detected is displayed.

  The viewpoint displacement of the imaging device 20 is calculated by the viewpoint displacement calculator 33. Details will be described below.

  The video processing unit 34 generates video data that can be processed by the computer terminal 3 from the actual video input from the imaging device 20. Specifically, the video processing unit 34 generates a frame image composed of bitmap data based on the input actual video.

  The viewpoint displacement calculation unit 33 extracts a feature point from the frame image input from the video processing unit 34 to obtain a displacement of the feature point, and calculates a displacement of the viewpoint of the imaging device 20 based on the displacement. Specifically, the viewpoint displacement calculation unit 33 performs differential processing on the frame image to extract edges, extracts intersections of the edges as feature points, and stores position information of each feature point.

  For example, the viewpoint displacement calculation unit 33 acquires the position information of each feature point extracted from the frame image shown in FIG. 2A on the basis of the lower left pixel of the frame image and stores it in a temporary storage device such as a memory. . Similarly, the frame image shown in FIG. 2B (hereinafter referred to as “current frame image”) input next to the frame image (hereinafter referred to as “previous frame image”) is similarly applied. The position information of each feature point is acquired and stored. The previous frame image and the current frame image include the same four feature points (P0, P1, P2, P3).

  The viewpoint displacement detection unit 33 compares the position information of each feature point included in the current frame image and the previous frame image with respect to each feature point, and obtains the displacement of each feature point and the correlation change of all feature points. Then, based on the displacement and the change in the correlation, the displacement of the viewpoint of the imaging device 20, that is, the movement of the viewpoint of the imaging device 20 in the vertical and horizontal oblique directions, the movement in the front-rear direction, the rotational movement, and the movement distance are calculated.

  Based on the current frame image shown in FIG. 2B and the previous frame image shown in FIG. 2A, the displacement of each feature point obtained by the viewpoint displacement detector 33 is “+ Δx” in the horizontal direction and in the vertical direction. “+ Δy”. Moreover, the positional relationship and the distance between feature points are the same, and the correlation is maintained.

  Accordingly, the viewpoint displacement detection unit 33 determines that the viewpoint of the imaging apparatus 20 is “−Δx” in the horizontal direction and “−Δy” in the vertical direction when the current frame image is captured from when the previous frame image is captured. Calculated as moved.

  Hereinafter, a virtual video display operation by the virtual video display system 1 will be described with reference to a flowchart shown in FIG.

  When a switch for starting virtual video display is operated, shooting of a real video is started by the imaging device 20, and a predetermined video stored in the video object storage unit 30 corresponding to the real video by the virtual video generation unit 31 is started. The video object is read (SA1 to SA3).

  The video object is rendered by the virtual video generation unit 31 to generate a reference virtual video, and the reference virtual video is displayed on the screen 21 by the virtual video display unit 32 (SA4, SA5).

  A frame image composed of bitmap data is generated from the actual video captured by the imaging device 20 by the video processing unit 34 (SA6), and feature points are extracted from the frame image by the viewpoint displacement calculation unit 33. The position information is stored (SA7, SA8).

  The viewpoint displacement calculation unit 33 compares the position information of the feature points of the current frame image and the previous frame image to obtain the displacement and correlation of each feature point, and based on the displacement and the correlation, the imaging device 20 The viewpoint displacement is calculated (SA9, SA10).

  If there is a displacement of the viewpoint of the imaging device (SA11), the virtual video generation unit 31 generates a virtual video of the video object at the position and orientation viewed from the viewpoint before the displacement viewed from the viewpoint of the current imaging device 20. (SA4) The virtual video display unit 32 displays the virtual video on the screen 21 (SA5).

  In the following, an information guidance apparatus employing the virtual video display system will be described.

  The information guide device 4 is an HMD that guides work information, author information, and the like of works displayed in a museum. As shown in FIGS. 4A and 4B, the information guide device 4 includes an imaging device 40, a screen 41, a projection device 42, and an information guidance control device 43.

  The information guidance control device 43 includes a video processing unit 48, a viewpoint displacement calculation unit 47, a video object storage unit 44, a virtual video generation unit 45, a virtual video display unit 46, and a video object selection unit 49.

  The video object storage unit 44 includes an area partitioned by a storage unit such as a flash ROM provided in the information guidance control device 43. The video object storage unit 44 stores, for each work, a video object corresponding to the work information and work information of the work for which information guidance is performed.

  The video processing unit 48, the viewpoint displacement calculation unit 47, the virtual video generation unit 45, the virtual video display unit 46, and the video object selection unit 49 are controlled by the microcomputer provided in the information guidance control device 43 and the microcomputer. It is a functional block embodied by a storage unit, a receiving antenna, and peripheral circuits such as an image processing circuit and a receiving circuit.

  The video object selection unit 49 is a work selection signal (for example, a code number assigned to each work individually) that is a radio wave transmitted from a transmission unit (not shown) installed in the vicinity of each work that provides information guidance. Signal).

  Then, the video object selection unit 49 selects the video object corresponding to the work indicated by the work selection signal as the video object corresponding to the virtual video displayed on the screen 41.

  The work selection signal is a signal indicating a work for which information is to be guided, and is a switch signal that activates display of a virtual video corresponding to the video object.

  When there are a plurality of received work selection signals, the video object selection unit 49 selects the video object indicated by the work selection signal having the highest reception level.

  The video object selection operation by the video object selection unit 49 will be described below with reference to the flowchart shown in FIG.

  When a plurality of work selection signals are input, the video object selection unit 49 confirms each reception level, and the video object of the work corresponding to the work selection signal with the highest reception level corresponds to the virtual video displayed on the screen 41. Is selected as the video object to be played (SB1 to SB4).

  When one piece selection signal is input, the video object selection unit 49 selects the video object of the piece corresponding to the piece selection signal (SB1, SB2, SB5).

  When a new video object is selected by the video object selection unit 49, the virtual video generation unit 45 reads the video object from the video object storage unit 44 as a video object corresponding to the actual video shot by the imaging device 40. Thus, a virtual video in which the video object in a predetermined posture is displayed at a predetermined position on the screen 41 is generated as a reference virtual video.

  In addition, when the viewpoint of the imaging device 40 is displaced, the virtual video generation unit 45 performs the actual video based on the reference virtual video viewed from the viewpoint of the imaging device 40 when the video object selection unit 49 detects the work selection signal. A virtual video corresponding to the displacement of the viewpoint of the imaging device 40 calculated from the video feature points is generated.

  That is, the virtual video generation unit 45 corresponds to the virtual object corresponding to the video object whose position and orientation viewed from the viewpoint of the imaging device 40 have changed in response to the displacement of the viewpoint of the imaging device 40 calculated by the viewpoint displacement calculation unit 47. Generate video.

  Below, the virtual video display operation | movement used as the information guidance by the information guidance apparatus 4 is demonstrated using the flowchart shown in FIG.

  When the first video object is selected by the video object selection unit 49, shooting is started by the imaging device 40, and the video object corresponding to the actual video shot by the imaging device 40 by the virtual video generation unit 45. Are read from the video object storage unit 44 (SC1 to SC3).

  A virtual video generation unit 45 generates a reference virtual video in which the video object is displayed at a predetermined position on the screen 41 in a predetermined posture, and the virtual video display unit 46 transmits the reference virtual video via the projection device 42. It is displayed on the screen 41 (SC4, SC5).

  Based on the actual video captured by the imaging device 40, a frame image composed of bitmap data is generated by the video processing unit 48, and the viewpoint displacement calculation unit 47 calculates the displacement of the viewpoint of the imaging device 40 from the feature points of the frame image. (SC6, SC7).

  If there is no movement in the viewpoint of the imaging device 40, the virtual video displayed on the screen 41 by the virtual video display unit 46 does not change (SC8).

  When the viewpoint of the imaging device 40 is moved (SC8) and a new video object is selected by the video object selection unit 49 (SC9), the virtual video generation unit 45 converts the actual video captured by the imaging device 40 into an actual video. Correspondingly, the video object is read from the video object storage unit 44, a reference virtual video is generated, and the reference virtual video is displayed by the virtual video display unit 46 (SC3 to SC5).

  If there is a movement in the viewpoint of the imaging device 40 (SC8) and no new video object is selected by the video object selection unit 49 (SC9), the virtual video generation unit 45 may display the position viewed from the viewpoint of the imaging device 40. A virtual video corresponding to the video object whose posture has changed is generated, and the virtual video display unit 46 displays the virtual video (SC4, SC5).

  The information guide device 4 is not used only for guidance on work information and author information of works exhibited at museums and the like. For example, as an operation method of an operation panel of an image forming apparatus such as a copying machine, a guide to troubles occurring in the apparatus (paper jam, out of paper, out of printing member such as toner or ink), and a troubleshooting method Can be used.

  In this case, it corresponds to the hardware key arranged on the operation panel, the explanation corresponding to the software key on the operation screen displayed on the liquid crystal panel, the explanation of the contents of each trouble, the explanation of the countermeasure, etc. The video object to be stored is stored in the video object storage unit 44.

  The virtual video generation unit 45 takes a picture with the imaging device 40 based on a key operation signal based on an operation of a guidance start request key attached to the information guide device 4 or a trouble signal corresponding to a trouble occurring in the image forming apparatus. A video object corresponding to the actual video that has been read out is read from the video object storage unit 44 to generate a reference virtual video, and a virtual video corresponding to the displacement of the viewpoint of the imaging device 40 is generated.

  Hereinafter, a mobile phone employing the virtual video display system will be described. As shown in FIG. 7, the mobile phone 5 includes an imaging unit 50, a screen 51, a video processing unit 56, a viewpoint displacement calculation unit 55, a video object storage unit 52, a virtual video generation unit 53, a virtual video display unit 54, and a video object. A selection unit 57 is provided.

  The imaging unit 50 is configured by a small camera built in the mobile phone 5, the screen 51 is configured by a liquid crystal display panel, and the video object storage unit 52 is configured by an area partitioned by a storage unit such as a flash ROM.

  The video object storage unit 52 stores a video object corresponding to a virtual paper (hereinafter referred to as “virtual paper”) on which a document such as an electronic publication book is displayed for each virtual paper.

  The video processing unit 56, the viewpoint displacement calculation unit 55, the virtual video generation unit 53, the virtual video display unit 54, and the video object selection unit 57 are a microcomputer provided in the mobile phone 5, and a storage unit controlled by the microcomputer. And functional blocks embodied by peripheral circuits such as an image processing circuit.

  The video object selection unit 57 displays a virtual paper (video object) corresponding to a key operation signal input in response to an operation such as a numeric key provided in the mobile phone 5 on the screen 51. Choose as. The key operation signal is a paper selection signal for selecting a virtual paper, and is a switch signal for starting display of a virtual video corresponding to the video object.

  When the video object selection unit 57 detects the key operation signal and selects the video object, the virtual video generation unit 53 selects the video object from the video object storage unit 52 as a video object corresponding to the actual video shot by the imaging unit 50. The video object is read, and a reference virtual video is generated so that the virtual paper is displayed at a predetermined position on the screen 51 in a predetermined posture.

  In addition, the virtual video generation unit 53 calculates from the feature points of the real video based on the reference virtual video obtained by viewing the virtual paper from the viewpoint of the imaging unit 50 when the paper selection signal is detected by the video object selection unit 57. The virtual image corresponding to the displacement of the viewpoint of the imaging unit 50 is generated.

  In other words, the virtual video generation unit 53 corresponds to the virtual paper whose position or orientation viewed from the viewpoint of the imaging device 50 has changed in response to the displacement of the viewpoint of the imaging unit 50 calculated by the viewpoint displacement calculation unit 55. Generate video.

  Hereinafter, a virtual document display operation by the mobile phone 5 will be described with reference to a flowchart shown in FIG.

  When the first video object is selected by the video object selection unit 57, shooting is started by the imaging unit 50, and the video object corresponding to the actual video shot by the imaging unit 50 by the virtual video generation unit 53. Are read from the video object storage unit 52 (SD1 to SD3).

  The virtual video generation unit 53 generates a reference virtual video in which the video object is displayed at a predetermined position on the screen 51 in a predetermined posture, and the virtual video display unit 54 displays the reference virtual video on the screen 51 ( SD4, SD5).

  When a new video object is selected by the video object selection unit 57 (SD6), the video object is stored in the video object storage unit 52 corresponding to the actual video shot by the imaging unit 50 by the virtual video generation unit 53. The reference virtual image is generated, and the reference virtual image is generated by the virtual image display unit 54 (SD3 to SD5).

  When no new video object is selected by the video object selection unit 57 (SD6), a frame image composed of bitmap data is generated by the video processing unit 56 based on the actual video shot by the imaging unit 50, and viewpoint displacement calculation is performed. The displacement of the viewpoint of the imaging unit 50 is calculated by the unit 55 (SD7, SD8).

  If the viewpoint of the imaging unit 50 has not moved, the virtual video displayed on the screen 51 by the virtual video display unit 54 does not change (SD9).

  If the viewpoint of the imaging unit 50 has moved (SD9), the virtual video generation unit 53 generates a virtual video corresponding to the video object whose position and orientation viewed from the viewpoint of the imaging unit 50 have changed, and displays virtual video. The virtual video is displayed by the unit 54 (SD4, SD5).

  Hereinafter, another embodiment will be described.

  The first mode in which the virtual video display system 1 described above allows generation of a virtual video corresponding to the displacement of the viewpoint of the imaging device 20 or the generation of a virtual video corresponding to the displacement of the viewpoint of the imaging device 20 is not allowed. An embodiment when a mode switching unit that switches the virtual video generation mode of the virtual video generation unit 31 is provided in one of the two modes will be described.

  The mode switching unit includes a CPU that executes an operation program such as an image processing application stored in a storage unit such as a hard disk provided in the computer terminal 3, a memory controlled by the CPU, a storage unit, and an image processing hardware. Hardware block (for example, an image processing board on which an image processing circuit is mounted) and the like.

  When a switch for switching the virtual video generation mode provided in the HMD 2 is operated, a mode switching signal is input to the computer terminal 3 in accordance with the operation.

  When detecting the mode switching signal, the mode switching unit switches the virtual video generation mode from the first mode to the second mode, or from the second mode to the first mode.

  When the virtual video generation mode is the first mode, the mode switching unit outputs to the virtual video generation unit 31 without changing the viewpoint displacement value of the imaging device 20 calculated by the viewpoint displacement calculation unit 33. Accordingly, the virtual video generation unit 31 generates a virtual video corresponding to the displacement of the viewpoint of the imaging device 20.

  When the virtual video generation mode is the second mode, the mode switching unit changes the value of the viewpoint displacement of the imaging device 20 calculated by the viewpoint displacement calculation unit 33 to a value when there is no displacement, and the virtual video generation unit To 31. Therefore, the virtual video generation unit 31 determines that the viewpoint of the imaging device 20 has not moved, and no new virtual video is generated.

  Since the range of the space that can be displayed on the screen 21 is finite, the video object may not be displayed in the virtual video when the video object is viewed from the viewpoint of the imaging device 20 that has moved greatly.

  In such a case, the virtual image generation mode is switched from the first mode to the second mode by the mode switching unit before the viewpoint of the imaging device 20 is moved, and after the movement, the second mode is switched to the first mode. The virtual video generation unit 31 determines that the viewpoint of the imaging device after movement is the same as the viewpoint of the imaging device 20 before movement.

  Therefore, even when the viewpoint of the imaging device 20 moves greatly, the video object can be reliably displayed in the virtual video, and the operability of the virtual video display system 1 is improved.

  Such a configuration can be suitably employed in a mobile phone or the like that displays virtual paper. Specifically, the mode switching unit is configured to switch the virtual video display mode in accordance with an operation of a numeric key or the like provided in the mobile phone.

  When an operator who is browsing virtual paper using a mobile phone wants to change his / her posture, for example, when he / she wants to lie down on a sofa or the like from a state facing a desk, When a key is operated, the virtual image generation mode is switched from the first mode to the second mode by the mode switching unit according to the key operation.

  When the numeric key is newly operated by the operator and the virtual video generation mode is switched from the second mode to the first mode by the mode switching unit, the virtual video generation unit moves the viewpoint of the imaging unit after the movement before the movement. It is determined that the viewpoint is the same as the viewpoint of the imaging unit.

  In the above-described embodiment, the viewpoint of the imaging device 20 that views the video object of the reference virtual video that is used as a reference when the virtual video generation unit 31 generates the virtual video is when the switch signal that activates the display of the virtual video is detected. However, the present invention is not limited to this.

  For example, the viewpoint may be the viewpoint of the imaging device 20 at a predetermined time, such as the viewpoint of the imaging device 20 when the virtual video display system 1 is activated, or the viewpoint of the imaging device 20 when the virtual video display system 1 is terminated last time. It may be a viewpoint.

  The above-described embodiments are merely examples of the present invention, and the scope of the present invention is not limited by the description, and the specific configuration of each part is appropriately changed within the scope of the effects of the present invention. It goes without saying that it can be done.

Illustration of virtual video display system (A) is explanatory drawing of the last frame image, (b) is explanatory drawing of this frame image. Flowchart for explaining virtual video display operation by virtual video display system 1A is an external view of an information guide device adopting the present invention, and FIG. 2B is a functional block diagram of the information guide device adopting the present invention. Flowchart explaining video object selection operation by video object selection unit Flowchart for explaining virtual video display operation by information guide device adopting the present invention Functional block diagram of a mobile phone adopting the present invention Flowchart for explaining virtual video display operation by mobile phone adopting the present invention

Explanation of symbols

1: Virtual video display system 2: HMD (head mounted display)
3: Computer terminal 4: Virtual video display system (information guide device)
5: Virtual video display system (mobile phone)
20: Imaging unit (imaging device)
21: Screen 30: Video object storage unit 31: Virtual video generation unit 32: Virtual video display unit 40: Imaging unit (imaging device)
41: Screen 44: Video object storage unit 45: Virtual video generation unit 46: Virtual video display unit 50: Imaging unit 51: Screen 52: Video object storage unit 53: Virtual video generation unit 54: Virtual video display unit

Claims (4)

An imaging unit, a screen that is arranged in a predetermined positional relationship with the imaging unit, displays a virtual video, a video object storage unit that stores a video object defined by virtual video reference data, and A predetermined video object stored in the video object storage unit is read out corresponding to the actual video shot by the imaging unit, and the video object is viewed from an arbitrary viewpoint based on a reference virtual video viewed from the predetermined viewpoint. A virtual video display system including a virtual video generation unit that generates a viewed virtual video, and a virtual video display unit that displays the virtual video generated by the virtual video generation unit on the screen,
The virtual video generator generates a virtual video corresponding to a displacement of the viewpoint of the imaging unit calculated from a feature point of the real video based on the reference virtual video viewed from the viewpoint of the imaging unit at a predetermined time. Virtual video display system to generate.   The virtual image generation unit is calculated from the feature points of the real video based on the reference virtual video viewed from the viewpoint of the imaging unit at the time of detecting a switch signal that activates display of the virtual video The virtual image display system according to claim 1, wherein a virtual image corresponding to the displacement of the viewpoint is generated. A shooting step of shooting a real video by the imaging unit, and a predetermined video object stored in the video object storage unit corresponding to the real video shot in the shooting step is read out, and the video object is viewed from a predetermined viewpoint A virtual image generation step for generating a virtual image viewed from an arbitrary viewpoint based on the viewed reference virtual image, and a virtual image display step for displaying the virtual image generated in the virtual image generation step on a screen An image display method,
In the virtual video generation step, based on the reference virtual video viewed from the viewpoint of the imaging unit at a predetermined time point, a virtual video corresponding to the displacement of the viewpoint of the imaging unit calculated from the feature points of the real video Virtual video display method to be generated.   The imaging unit that is calculated from the feature points of the real video based on the reference virtual video viewed from the viewpoint of the imaging unit when a switch signal that activates display of the virtual video is detected in the virtual video generation step The virtual video display method according to claim 3, wherein a virtual video corresponding to the viewpoint displacement is generated.

Images