JP2005044102A - Image reproduction method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem with an image reproduction device which provides an observer with an experience of walking through a virtual space wherein annotations to objects within the virtual space are sometimes not easy to observe. <P>SOLUTION: Map information is displayed in a virtual space image which represents the virtual space. A mark indicating the observer's viewing position and direction and a mark indicating an area and a direction in which the annotations can be observed are displayed together in the map information. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image reproducing apparatus and method having a function of notifying an observer in advance of a viewpoint position and a line-of-sight direction where an annotation related to an object in a virtual space can be observed in a virtual space walk-through system.
[0002]
[Prior art]
In recent years, as one form of digital archiving, many attempts have been made to incorporate a cultural heritage or city into a computer and construct it as a virtual space. Also, with the development of CPUs and graphics hardware, it has become possible to walk through a virtual space with high reality on a personal computer in real time.
[0003]
One method for constructing a virtual space is modeling by general computer graphics (CG) using a geometric model and texture. In modeling by CG, a real geometric model is generated / restored by using a remaining drawing or an actual survey, or by directly measuring a geometric shape using a laser range finder or the like. Also, the reality is enhanced by texture mapping the actual image of the object onto the geometric model of the object.
[0004]
On the other hand, as another method for constructing a virtual space, there is a method based on Image-Based Rendering (IBR) that constructs a virtual space using only a photographed image. In modeling by CG, if the model is not accurate, the reality is lost. However, in IBR, since a real image is used as it is, a virtual space with high reality can be expressed. In addition, modeling does not take time, and it is suitable for construction of a wide range of virtual spaces. However, since IBR uses a large number of actual images, the amount of data becomes enormous. Therefore, research on modeling methods that combine modeling by CG and modeling by IBR is also widely performed.
[0005]
In a virtual space constructed using these methods, an annotation (annotation) is displayed on an object in the virtual space, so that an observer who walks through the virtual space can see which building is now. It is possible to provide additional information such as whether or not (see, for example, Patent Document 1). An annotation can appear to follow the object by being displayed on the object, or can be made to appear as a telop by being displayed at a fixed position on the screen.
By observing the annotated virtual space, the observer can walk through the virtual space as desired without a guide, and can obtain various information through annotation. . In addition, there is no restriction | limiting in particular in the information displayed as an annotation in this specification, Arbitrary information regarding a certain object in a virtual space image can be displayed.
[0006]
In addition, the virtual space map is presented to the observer, and the observer's viewpoint position / gaze direction and the position of the object on which the annotation is displayed are displayed on the map, so that the observer It is also possible to grasp the positional relationship with the object on which is displayed.
[0007]
[Patent Document 1]
JP 2002-163670 A
[0008]
[Problems to be solved by the invention]
However, even if the observer's viewpoint position / gaze direction and the position of the object on which the annotation is displayed are displayed on the map, if the object is far away from the observer, it may be hidden by another object. Actually, you may not be able to see the annotation. In addition, depending on the intention of the creator of the content (in this case, virtual space), the viewpoint position / line-of-sight direction where the annotation can be viewed may be limited.
[0009]
In such a case, the observer has to search for a viewpoint position / line-of-sight direction where he can observe the annotation he / she wants to observe by groping.
[0010]
The present invention has been made in view of the above problems, and an object of the present invention is to enable an observer to observe annotations on objects in a virtual space in an image reproduction device that provides the observer with a walk-through experience in a virtual space. This is to make it easy to recognize the viewpoint position and the line-of-sight direction.
[0011]
Furthermore, another object of the present invention is to further improve the convenience of the observer in the above-described image reproduction device.
[0012]
[Means for Solving the Problems]
That is, the image reproduction method according to the present invention detects the viewpoint position and the line-of-sight direction in the observer's virtual space, and presents the corresponding virtual space image to allow the observer to experience a walk-through in the virtual space. A virtual space image for generating a virtual space image to be presented to an observer based on a viewpoint position and a line-of-sight direction in the observer's virtual space and virtual space data stored in a virtual space data storage unit A map image is read out based on the viewpoint position in the observer's virtual space from the generation step and the map image storage means, and a map image representing a predetermined range of the virtual space including at least the viewpoint position in the observer's virtual space is obtained. A map image generation step for generating, and a presentation step for synthesizing the virtual space image and the map image and presenting the map image to the observer. The step observes the annotation image based on the information about the area where the annotation image relating to the object in the virtual space where the annotation information storage means can be observed and the viewpoint position and the line-of-sight direction in the observer's virtual space. The method includes a mark composition step of combining a mark representing a possible area and a mark representing an observer's viewpoint position and line-of-sight direction into a map image.
[0013]
In addition, the image reproducing apparatus according to the present invention detects the viewpoint position and the line-of-sight direction in the observer's virtual space, and presents the corresponding virtual space image to allow the observer to experience a walk-through in the virtual space. A virtual space image for generating a virtual space image to be presented to an observer based on a viewpoint position and a line-of-sight direction in the observer's virtual space and virtual space data stored in the virtual space data storage means A map image is read out based on the viewpoint position in the observer's virtual space from the generation means and the map image storage means, and a map image representing a predetermined range of the virtual space including at least the viewpoint position in the observer's virtual space is obtained. A map image generating means for generating, and a presenting means for synthesizing the virtual space image and the map image and presenting them to an observer. Area where the annotation image can be observed based on the information about the area where the annotation image related to the object in the virtual space is observable, and the viewpoint position and the line-of-sight direction in the virtual space of the observer And a mark synthesizing unit that synthesizes a mark representing the viewpoint position and line-of-sight direction of the observer into a map image.
[0014]
The object of the present invention is also achieved by a program for causing a computer to execute the image reproduction method of the present invention.
[0015]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.
[0016]
<First Embodiment>
FIG. 1 is a block diagram illustrating a functional configuration of the walk-through system according to the present embodiment. The image playback device 1 constituting the walk-through system includes an operation unit 10, a viewpoint position / gaze direction determination unit 20, a virtual space data storage unit 30, an annotation data storage unit 40, a map data storage unit 50, and an image playback control unit 60. The display unit 70 is included.
[0017]
FIG. 2 is a block diagram illustrating a hardware configuration example of the image reproduction device 1 according to the present embodiment. The hardware configuration shown in FIG. 2 can be realized by a general-purpose computer apparatus that is normally available as a personal computer in the market and its peripheral devices. In FIG. 2, a disk 105 is a mass storage device represented by a hard disk drive, for example, and constitutes the virtual space data storage unit 30, the annotation data storage unit 40, and the map data storage unit 50 in FIG.
[0018]
The CPU 101 functions as the viewpoint position / gaze direction determination unit 20 and the image reproduction control unit 60 by executing a program stored in the disk 105 or the ROM 106 or an external storage device (not shown).
[0019]
When the CPU 101 issues various display instructions to the display controller (CRTC) 102, a desired display is made on the display (CRT) 104 by the display controller 102 and the frame buffer 103. In the figure, a CRTC is shown as the display controller 102 and a CRT is shown as the display 104. However, the display is not limited to a cathode ray tube, and a liquid crystal display or the like may be used. Of course, the display device 104 may be a fixed type or a portable type such as a head-mounted display (HMD). The CRTC 102, the frame buffer 103, and the CRT 104 constitute the display unit 70 described above.
[0020]
The mouse 108, the keyboard 109, and the joystick 110 are used by an observer to perform an operation input to the image reproduction apparatus 1, and constitute the operation unit 10 described above.
[0021]
Next, an outline of the operation of the image reproduction apparatus 1 having the above configuration will be described.
The operation unit 10 includes a mouse, a keyboard, a joystick, and the like. The operation unit 10 generates a viewpoint position movement parameter and a line-of-sight rotation parameter. In this embodiment, the joystick is used for controlling the viewpoint position and the line-of-sight direction, but it may be performed by another input device such as a game controller, or an output from a magnetic sensor or an optical sensor may be used. The joystick can independently control the tilt angle and rotation angle of the stick. In the present embodiment, the operation of tilting the joystick corresponds to the movement of the viewpoint position in the virtual space, and the operation of rotating the joystick left and right corresponds to the rotation in the line of sight. In the present embodiment, the roll rotation is not considered for the rotation in the line-of-sight direction.
[0022]
The map data storage unit 50 stores two-dimensional map image data. The two-dimensional map image data is associated with virtual space data stored in the virtual space data storage unit 30.
[0023]
The annotation data storage unit 40 stores annotation data. The annotation data includes annotation image data, a viewpoint position (or a viewpoint position on a map) in the virtual space of the observer, and correspondence data (annotation correspondence data) with the annotation. FIG. 3 shows a configuration example of annotation correspondence data. Annotation correspondence data is defined for each object. In the example of FIG. 3, the position (x _A , Y _A For the object building A of), the annotation observable area has the same coordinates (x on the map) with respect to the viewpoint position (x, y) on the map of the observer. ₀ , Y ₀ ), (X ₁ , Y ₁ ). An annotation image to be displayed is defined by a file name (or file path) (building A.jpg).
[0024]
Note that the annotation observable region is defined as a position where no other object exists between the observable region and the target object, or the annotation is not hidden even if it exists. In the latter case, the display position in the image of the predetermined annotation, the viewpoint position (height) of the observer, and the height of other objects existing between the target object and the annotation observable area are taken into consideration. Can be determined.
[0025]
The shape of the annotation observable region is not limited to a rectangle, and may be another shape or may be defined by another method. For example, all vertex coordinates may be specified, or a circular area may be specified by specifying the center and radius.
[0026]
In the example of FIG. 3, the viewpoint position (x, y) on the map of the observer is (x ₀ , Y ₀ ), (X ₁ , Y ₁ ) Within a range defined by a rectangle having diagonal vertices as a diagonal vertex is a necessary condition for the annotation display for the building A.
[0027]
Whether the annotation is actually displayed, and the position on the observation screen when the annotation is displayed is the object whose viewpoint position on the observer's map is in the annotation observable area and that displays the annotation (In the above case, the location (x _A , Y _A ) And the viewpoint position / line-of-sight direction of the observer (FIG. 4).
[0028]
FIG. 4 is a diagram illustrating, as an example, annotation correspondence data for the building A in FIG. 3 with respect to an example of a method for determining whether to perform annotation display.
In FIG. 4, the viewpoint position (x, y) on the observer's map is (x ₀ , Y ₀ ), (X ₁ , Y ₁ ) In the rectangular annotation observable area defined for the building A, with the diagonal vertices. In this case, the viewing direction of the observer and the angle of view θ defined in the system ₀ If the building A is included in the visual field of the observer defined by (for example, ± α ° centered on the line-of-sight direction), the annotation display for the building A is performed.
[0029]
Specifically, the observer's viewpoint position (x, y) and the position of building A (x _A , Y _A ) And the angle (β °) between the line-of-sight direction (shown by the dotted line in FIG. 4),
β ≦ α
When the condition is satisfied, the annotation for the building A is displayed. In addition, you may perform determination of whether the building A is contained in an observer's visual field by another method.
[0030]
Note that the position of the building A (x _A , Y _A ) May be, for example, the center or barycentric position on the map image of the building A, or may be another position. Further, when the width of the building A as viewed from the annotation observable region is large, the coordinate value having the range may be used as the position of the object instead of the determination at one point such as the center position or the center of gravity position.
[0031]
In this case, the position data of the object in the annotation correspondence data is x _A And y _A The data has a range for at least one of the above. In the determination shown in FIG. 4, when a coordinate value having a range at the object position is used, the upper limit value and the lower limit value of the range (if both x and y have ranges, the upper limit of each range The combination of the value and the lower limit value) is determined, and whether or not display is possible is determined depending on whether or not one is included in the field of view.
[0032]
Needless to say, the annotation correspondence data is set in a coordinate system that is the same as the coordinate system representing the viewpoint position of the observer to be compared. That is, when determining whether or not annotation display is possible using the coordinate position of the observer's virtual space, the coordinates stored as annotation correspondence data are not coordinate values on the map but values in the coordinate system in the virtual space. Become.
[0033]
Although the display position of the annotation can be arbitrarily set, in the example of FIG. 4, the observer is placed on an image plane that is predetermined as a plane that is orthogonal to the observer's line-of-sight direction and is a predetermined distance away from the viewpoint position. Display is performed assuming that an annotation image exists centering on a point where a straight line connecting the viewpoint position of the target object and the position of the target object (building A) intersects. The display position in the vertical direction of the image plane (display position in the direction orthogonal to the paper plane of FIG. 4) can be, for example, the lowermost part or the uppermost part of the image plane. Of course, as long as the relationship with the target object can be clearly understood, an annotation can be displayed at an arbitrary position on the image plane.
[0034]
The virtual space data storage unit 30 stores data (virtual space data) for constructing a virtual space. When the virtual space is constructed with a geometric model and a texture image, the geometric model and the texture image are stored as virtual space data. In addition, when the virtual space is constructed using the IBR technology, the photographed image data is stored as virtual space data. The virtual space data also includes data (moving range data) for defining a range in which the observer's viewpoint can move in the virtual space as necessary.
[0035]
When the image playback device 1 has a network interface, the virtual space data storage unit 30 does not need to exist locally on the image playback device 1 and is installed on the network and reads the virtual space data via the network. May be.
[0036]
The viewpoint position / line-of-sight direction determination unit 20 uses the movement parameter and rotation parameter from the operation unit 10 and, if necessary, the movement range data of the virtual space data storage unit 30, the viewpoint position / observer of the observer in the virtual space. Determine the line-of-sight direction. This determination can be made based on any method conventionally used in walkthrough applications. The viewpoint position / line-of-sight direction determination unit 20 also uses the viewpoint of the observer in the two-dimensional map image data stored in the map data storage unit 50 based on the determined viewpoint position / line-of-sight direction in the virtual space. Determine the position (viewpoint position on the map) and line-of-sight direction.
[0037]
■ (Overall processing)
FIG. 5 is a flowchart showing the overall processing of the image reproduction apparatus 1 described above. This processing is realized by the CPU 101 constituting the image reproduction control unit 60 executing a program stored in the disk 105 or the like and controlling each component of the image reproduction apparatus.
[0038]
In step S <b> 10, the image reproduction control unit 60 receives the viewpoint position and line-of-sight direction in the observer's virtual space from the viewpoint position / line-of-sight direction determination unit 20. In step S <b> 20, an observation image (virtual space image) to be displayed is generated from the virtual space data stored in the virtual space data storage unit 30 based on the viewpoint position and line-of-sight direction of the observer.
[0039]
In step S30, the map image data to be displayed is read from the map data storage unit 50 based on the viewpoint position and line-of-sight direction on the map of the observer, the display size of the map image, the scale, etc. An annotation mark and an observer mark described later are synthesized. Then, the obtained map image is synthesized with the virtual space image.
[0040]
In step S40, an annotation image to be displayed is acquired based on the viewpoint position and line-of-sight direction on the map of the observer and the annotation observable area data in the annotation data storage unit 40, and is synthesized with the virtual space image.
[0041]
Then, the image reproduction control unit 60 passes the finally obtained image to the display unit 70. The display unit 70 displays the image generated by the image reproduction control unit 60.
[0042]
Note that the scale of the map image to be displayed may be fixed, or may be variable according to an instruction from the observer. In the case of a variable scale, data of a plurality of scales may be held as map image data, and map image data corresponding to a set scale may be read out, or common map data may be simply displayed according to the scale. You may enlarge / reduce the display. In addition, when a condition such as the observer's viewpoint position approaches an annotation observable region is satisfied, the scale can be changed to change to a detailed display.
[0043]
■ (Example of mark display)
Next, the map image display process of the above-described image reproduction control unit 60 will be described in more detail. The image reproduction control unit 60 has a map information display unit 62 and an image (observer mark) representing the observer's viewpoint position / gaze direction on the two-dimensional map image determined by the viewpoint position / gaze direction control unit 20. Is drawn (synthesized) on the map image. Further, the image reproduction control unit 60 can indicate the annotation in which position in the virtual space the observer can observe the annotation, the position of the target object of the observed annotation, and in which direction it is. It has an observable area display section 64 for drawing an image (annotation mark) representing related information on a map.
[0044]
FIG. 6 is a diagram illustrating an example of map display realized by the map information display unit 62 and the observable region display unit 64.
Here, the building A shown in the upper part of FIG. 3 will be described as an example. That is, as shown in FIG. 6A, the position of the object (building A) on the two-dimensional map is (x _A , Y _A ),building _A The region where the annotations on can be observed (x ₀ , Y ₀ ), (X ₁ , Y ₀ ), (X ₁ , Y ₁ ), (X ₀ , Y ₁ ) In the rectangle determined by
[0045]
In this case, as shown in FIG. 6B, the map information display unit 62 and the observable region display unit 64 display an image representing the annotation observable region and the position of the display target object on the map as an annotation mark. An image representing the observer's own viewpoint position and line-of-sight direction is shown on the map as an observer mark. As a result, the observer can know the area where the annotation can be observed and the direction of the target object of the annotation that can be observed there, and the relative position between the current observer's own viewpoint position and the annotation observable area. You can know the relationship.
[0046]
In the example of FIG. 6B, two images, the image showing the entire annotation observable region and the image showing the position of the target object, are shown as annotation marks. However, as shown in FIG. The center of the observable area ((x ₀ + X ₁ ) / 2, (y ₀ + Y ₁ ) / 2), an image representing the direction θ of the target object (see FIG. 6A) may be shown as an annotation mark.
[0047]
In this case, if the annotation mark has the same shape as the observer mark, the observer can observe the annotation by controlling the display of the observer mark on the map image so as to approach the same position and direction as the annotation mark. Intuitively understand. Further, in FIG. 6C, an image indicating the display target object is not displayed as an annotation mark, but may be displayed.
[0048]
When the annotation mark has the same shape as the observer mark, the annotation mark can be displayed in different colors and transparency in order to avoid confusion between the two. Also, in FIG. 6, each mark is represented by an isosceles triangle, and the position and direction can be recognized in the same manner as a general arrow. However, any shape may be used as long as the position and direction can be recognized.
[0049]
In FIG. 6, the region where the annotation can be observed is a rectangle, but may be an arbitrary polygonal region as described above. Further, a circle or an ellipse may be used. Further, if the display format of the map image is a format with a sense of perspective or a three-dimensional effect, it can be combined with a mark representing a three-dimensional space such as a cube or a rectangular parallelepiped. In this case, the format of the annotation correspondence data may be changed together.
[0050]
FIG. 7 is a flowchart showing such map image generation and update processing, and corresponds to the processing in step S30 of FIG.
In step S32, the map information display unit 62 determines the map image data to be displayed in consideration of the display size and scale of the map image, if necessary, in addition to the viewpoint position and line-of-sight direction on the map of the observer. Read from the map data storage unit 50. Next, in step S34, the observable area display unit 64 displays the map on the map image based on the viewpoint position of the observer on the map, the display size and scale of the map image, and the observable area of the annotation correspondence data. Search for annotation marks that should be used.
[0051]
In step S36, the observable area display unit 64 combines the annotation mark and the observer mark for the observable area to be displayed, obtained as a result of the search in step S34, on the map image.
[0052]
■ (Annotation display processing)
Next, annotation display processing (corresponding to step S40 in FIG. 5) in the image reproduction control unit 60 will be described using the flowchart shown in FIG.
Here, as described above, it is assumed that the annotation correspondence data is defined by the coordinates on the map image. However, when the annotation correspondence data is defined by the coordinates on the virtual space, the virtual position is virtually set at the observer's viewpoint position. A similar process may be performed using the coordinates in the space.
[0053]
First, in step S101, the viewpoint position / gaze direction on the map image of the observer is acquired from the viewpoint position / gaze direction determination unit 20. Next, the variable i is set to 1 in step S102. The variable i is the total number n of objects O that display annotations. _i Is a variable for sequentially processing, and takes an integer value from 1 to n.
[0054]
In step S103, it is determined whether or not i ≦ n, that is, whether or not processing has been performed for all objects (display target objects) that display annotations. If all the display target objects have not been processed (if i ≦ n), in step S104, the viewpoint position on the map of the observer acquired in step S101 is the object O. _i Annotation observable area defined in the annotation correspondence data corresponding to (R _i To determine whether it is included.
[0055]
Annotation observation area R _i If it is included, the process proceeds to step S105, where the observer's viewpoint position, viewpoint direction, and object O _i And the object angle O from the preset angle of view θ. _i Is determined to be in the observer's field of view. When entering the field of view, the flag F corresponding to the i-th annotation in step S106. _i Is set to 1. If it does not enter the field of view, or the viewpoint position of the observer is the annotation observable region R in step S104 _i Is not included in the flag F in step S107 _i Is set to 0. From steps S106 and S107, the process proceeds to step S108, i is incremented by 1, and the process returns to step S103.
[0056]
If i> n in step S103 and it is determined that the processing has been completed for all the display target objects, the process proceeds to step S109. In step 109, the flag F set to 1 _i An annotation image corresponding to (i = 1 to n) is acquired from the annotation data storage unit 40 and synthesized on the virtual space image. Thereafter, an end determination is made in step S110, and if it is continued, the process returns to step S101.
[0057]
■ (Annotation mark display control according to the relationship with the observer mark)
By displaying the observer mark and the annotation mark on the map image, the observer can easily visually understand the distance relationship with the annotation observable region and the relationship with the direction of the target object. However, by visually varying the mark display method according to the relationship between the observer's viewpoint position and the annotation observable area and the relationship between the observer's line-of-sight direction and the direction of the target object, Further, it becomes possible to provide a highly convenient application. Hereinafter, an example of mark display control that can be performed in the map image display process in the present embodiment will be described.
[0058]
(1) When the observer mark approaches the annotation mark (part 1)
When the observer's viewpoint position enters the annotation observable region and the display target object enters the observer's field of view, annotation display is performed in the virtual space image.
At that time, by changing the annotation mark display method on the map image, the observer can more clearly recognize the correspondence between the displayed annotation and the observable region.
[0059]
For example, in the state shown in the left of FIG. 9, when the annotation display condition that the observer's viewpoint position enters the annotation observable region and the display target object enters the observer's field of view is satisfied, FIG. In a), the display color of the mark indicating the observable area as the annotation mark is changed before the condition is satisfied. In FIG. 9B, when an annotation display condition is satisfied, a mark indicating an observable area is blinked. Further, in FIG. 9C, the transparency of the mark indicating the observable area is increased before the annotation display condition is satisfied, and the transparency is decreased (for example, opaque) when the condition is satisfied.
[0060]
In addition, as shown in FIG. 10, when the annotation mark includes both a mark indicating the observable area and a mark indicating the display target object, the display method of both marks may be changed when the annotation display condition is satisfied. Is possible.
[0061]
In FIG. 10A, the colors of both the observable region and the mark indicating the object are changed before and after the annotation display condition is satisfied. In FIG. 10B, when the annotation display condition is satisfied, the mark indicating the observable region and the object is blinked. Further, in FIG. 10C, the transparency of the mark indicating the observable region and the object is increased before the annotation display condition is satisfied, and the mark is rendered opaque when entering the observable region.
[0062]
In this way, the image reproduction control unit 60, the virtual space image corresponding to the viewpoint position / line-of-sight direction in the observer's virtual space, the map image in which the annotation mark and the observer mark are combined, and the observed annotation The image is synthesized and passed to the display unit 70. An example of an image generated by the image reproduction control unit 60 is shown in FIG.
[0063]
As described above, according to the image reproducing device according to the first embodiment of the present invention, a map image is synthesized and displayed on a virtual space image, and annotation information such as an annotation observable region and an annotation target object is displayed on the map image. An annotation mark to represent and an observer mark to represent the observer's viewpoint position and viewpoint direction, so that the observer can easily know the viewpoint position and line-of-sight direction in which the annotation can be observed in the virtual space, and to the annotation observable position You will be able to move. In addition, by changing the display method of the mark that indicates the annotation observable area between when the annotation is displayed and when it is not displayed, the correspondence between the observed annotation and the observable area is clarified to the observer. Can be recognized.
[0064]
■ <Second Embodiment>
In the first embodiment, by displaying the annotation observable area on the map image, the observer can easily know the area where the annotation can be observed in the virtual space. In the second embodiment, by changing the display method of the annotation observable area displayed on the map image according to the purpose, the observer can discriminate the annotation observed in the area. Note that the configuration of the image reproducing apparatus according to the present embodiment may be the same as that described with reference to FIGS.
[0065]
FIG. 12 is a diagram illustrating an example of a map image display that is synthesized and displayed on the virtual space image in the image reproduction device according to the second embodiment. FIG. 12A shows an example in which the display method of the mark representing the observable region is changed (meaning a visually different display) depending on the importance of the annotation (or the display target object). In the example of FIG. 12A, a mark corresponding to an important annotation has a pattern (black circle) different from other marks.
[0066]
For example, when realizing cultural heritage as a virtual space and providing its walk-through application, the importance of annotation for buildings with high historical value and famous buildings is increased, and other annotation marks are displayed on the map image. By presenting the viewer with a different display method, it becomes possible for the viewer to observe the annotation with high importance, and to learn the cultural heritage efficiently. Here, the importance is distinguished by the difference in the shape of the mark (the presence or absence of a black circle), but any method that can be visually distinguished from other marks, such as the color, size, and transparency of the mark, can be used. .
[0067]
FIG. 12B shows an example in which the mark display method, here the display color, is changed depending on the genre of the annotation (or display target object). The genre is divided into, for example, history, geography, shopping, meals, and public facilities. Since the mark is color-coded according to the genre, the observer can know where the annotation of the genre that he is interested in can be observed in the annotation observable area displayed on the map. Here, the genre is distinguished by the color of the mark in the annotation observable area, but the shape of the mark may be changed. Further, a legend image explaining the correspondence between the mark color and the genre, importance, etc. may be displayed in addition to the map image or as another image.
[0068]
In order to realize the display method shown in FIG. 12, items of importance and genre are set in the annotation correspondence data stored in the annotation data storage unit 40, and the display method is performed in consideration of the setting values of these items at the time of display. Should be selected. FIG. 13 shows an example of annotation correspondence data to which a genre and importance are added. In the example of FIG. 13, the importance and genre are described with meaningful words such as “high” and “low” for the sake of explanation. However, the importance rank and genre type such as numerical values and symbols are used. Corresponding arbitrary format data can be used.
[0069]
In addition, it is possible to configure the genre and the importance of the mark to be displayed among the marks representing the annotation observable area so that the observer can specify the importance. In this case, the observer can select a mark indicating the annotation observable area to be displayed on the map by setting a desired importance and genre using a user interface as shown in FIG. 14, for example. It is.
[0070]
FIG. 14A shows an example of an importance selection user interface screen, and FIG. 14B shows an example of a genre selection user interface screen. For such screen display, for example, an operation for calling a screen is assigned to a key or operation such as a keyboard or a mouse constituting the operation unit 10, and when the viewpoint position / line-of-sight direction determination unit 20 detects this call operation, This can be realized by notifying the image reproduction control unit 60 to that effect, and the image reproduction control unit 60 calling screen data from the virtual space data storage unit 50 and displaying it on the display unit 70, for example.
[0071]
In addition, while the user interface screen is displayed, the operation unit 10 and the image reproduction control unit 60 substantially exchange directly, and the image reproduction control unit 60 detects and processes a selection operation by the operation unit 10, and the setting result is displayed. For example, it is stored in the annotation data storage unit 40. When the setting process is completed, the operation content of the operation unit 10 returns so that the viewpoint position / line-of-sight direction determination unit 20 processes the movement in the virtual space.
[0072]
Further, in the map image display processing described with reference to FIG. 7, when determining the annotation mark to be displayed in step S34, only the annotation mark of importance or genre checked in the user interface shown in FIG. 14 is selected. Like that. In addition, in the annotation display process shown in FIG. 8, a process for determining whether or not the genre should be displayed is added during steps S105 to S106, and only the flag F is displayed for the genre object to be displayed. _i Is set to 1.
[0073]
Note that the display method shown in FIG. 12A can be realized by preparing an annotation mark image associated with the importance in advance and using the annotation mark image corresponding to the importance during display.
[0074]
By performing such mark display, the observer can observe on the map image the observable region of the annotation of only the genre he / she desires. In addition, it is possible to have a walk-through experience according to the needs of the observer, such as observing only important annotations.
[0075]
In the above description, the display method is changed according to the importance, and only the items that match the set genre are displayed. However, only the items that match the predetermined importance are displayed. You may do it. In short, the present embodiment is characterized in that the display method is changed (including non-display) according to attributes such as the importance level and genre set in the annotation or object. The type and the display method corresponding to it can be arbitrarily set.
[0076]
■ <Third Embodiment>
For example, in the walk-through application, if the same annotation is displayed every time the same place is passed, there is a possibility that the observer may feel uncomfortable. Therefore, the third and fourth embodiments are characterized in that a plurality of observable annotation displays are provided in the same region, and the contents of the displayed annotations are changed according to conditions. As a result, it is possible to provide a wealth of information by preventing the observer from feeling uncomfortable by observing the same annotation many times.
[0077]
FIG. 15 is a diagram for explaining an annotation presentation method to an observer in the third embodiment. In FIG. 15, the annotations presented for the same object are changed depending on whether the observer enters the annotation observable region from the boundary A and enters the annotation observable region from the boundary B. Thus, for example, when entering from the boundary A, the annotation of the target object from the viewpoint of architecture is displayed, and when entering from the boundary B, the annotation from the viewpoint of archeology is displayed. , Can give a variety of information to the observer.
[0078]
Also, an observer often does not need to see an annotation once viewed, and there is a possibility that it will be complicated if the same annotation is presented. In order to eliminate the troublesomeness felt by the observer, it is possible to reduce the number of times the same annotation is not displayed or displayed by changing the displayed annotation when entering the annotation observable area. .
[0079]
In FIG. 15, the annotation to be presented is changed depending on which “side” of the two-dimensional area on the map image enters the annotation observation area. For example, the viewpoint position of the observer can move in the three-dimensional space. An application that can move the viewpoint as if the observer is flying in the sky, etc., the annotation observation area may be a three-dimensional area, and from which “surface” of the polyhedron has entered the annotation observation area You may change the annotation presented.
[0080]
FIG. 16 shows an example of a storage format of annotation corresponding data when the annotation to be displayed is changed depending on where the annotation enters the viewable area. Each object has a correspondence table between an entry location (in this case, a side) and an annotation image to be displayed. In FIG. 16, regarding the building A, when entering from the boundary A, the annotation image of “building A1.jpg” is displayed, and when entering from the boundary B, the annotation image of “building A2.jpg” is displayed. indicate. Further, when entering from other than the boundaries A and B, a default image (building A.jpg) is displayed. Since only the default is set for the building B, an annotation image of “building B.jpg” is displayed when entering from any boundary. If there is no default description, no annotation image is displayed.
[0081]
In the example of FIG. 16, it is assumed that how each boundary corresponds to the annotation observable region is determined in advance, but coordinates that specify the annotation observable region as necessary. A data item that defines the boundary may be added by using, for example.
[0082]
FIG. 17 shows a flowchart of annotation image determination in the third embodiment. This flowchart corresponds to the annotation drawing process in step S109 in FIG. The process of FIG. 17 is the same as the flag F in the flowchart of FIG. _i For all objects with = 1.
[0083]
For each object, first, in step S301, it is determined whether or not an annotation image corresponding to the entered boundary exists. As for the boundary from which the observer enters the annotation observable area, for example, the movement history of the observer's viewpoint position is recorded, and the observer's viewpoint position is the annotation observable area in step S104 in FIG. It is possible to determine based on the movement history when it is detected to be included in. The decision result is flag F _i When the value is set to 1, it is possible to set information that can specify the boundary in the object.
[0084]
In step S301, F _i For the object of = 1, the presence or absence of information whose boundary can be specified or its content and the annotation correspondence data image table shown in FIG. The presence or absence can be determined.
[0085]
If there is an annotation image corresponding to the boundary where the observer has entered in step S301, the process proceeds to step S303, and the corresponding annotation image is drawn. If it does not exist, the process proceeds to step S302 to determine whether or not a default annotation image exists. If a default annotation image exists, the process advances to step S304 to draw a default annotation image. If there is no default annotation image, nothing is drawn.
[0086]
As described above, according to the third embodiment, by changing the annotation for the same object according to the approach path to the annotation observable region, the discomfort felt by the observer can be eliminated, or rich information can be provided. It becomes possible to do.
[0087]
■ <Fourth Embodiment>
In the third embodiment, by changing the annotation to be presented depending on where to enter the annotation observable region, it is possible to eliminate the complexity felt by the observer or provide rich information. In the fourth embodiment, the same effect is realized by giving history information to the annotation and changing the display method of the annotation and the annotation mark on the map based on the history.
[0088]
FIG. 18 is a diagram illustrating an example of an annotation presentation method performed by the image reproduction device according to the fourth embodiment. In the present embodiment, the image playback apparatus holds annotation display history information of each object on the disk 105 or the like. In addition, the display history information changes the observer, a certain amount of time has passed since the previous display, a specific command is input from the mouse or keyboard, or the observer's viewpoint position moves to an area that is a predetermined distance away Or the content to be presented to the viewer is switched according to various conditions.
[0089]
When history information is reset for a certain object, the number of observations of the annotation image corresponding to that object is set to zero. The change of the observer is detected by restarting the walk-through system, user authentication, or input through a user interface such as a keyboard or a mouse. When the viewer's viewpoint position and line-of-sight direction satisfy the annotation display conditions of the object A for the first time, the annotation image is combined with the virtual space image and displayed as described in the first embodiment, for example. On the other hand, if the display condition for the same annotation satisfies the predetermined number of times or more, for example, after the second time, the annotation image is not displayed. As a result, the observer does not have to see the annotation once viewed, and does not feel uncomfortable even when the same annotation observable area is passed many times during the walk-through.
[0090]
FIG. 19 is a diagram for explaining another annotation presentation method in the fourth embodiment. The walk-through system holds a plurality of annotation images for the same object, refers to history information, selects an annotation image corresponding to the number of times of entry into the observable region, and displays the combined image. As a result, for example, it is possible to display an annotation of rough information about an object at first, and to display an annotation about detailed information as the number of times of entry increases. It is also possible to display a randomly selected annotation image without referring to history information.
[0091]
The display method of the mark indicating the annotation observable area on the map may be changed according to the history information. FIG. 20A shows a change in the mark indicating the annotation observable area when the annotation is displayed only at the first approach shown in FIG. The mark indicating the annotation observable area is displayed on the map until the first entry, but once the annotation observable area is exited, it is not displayed until the history information is reset. This prevents the observer from feeling uncomfortable that the annotation is not displayed even if there is a mark on the map.
[0092]
In addition, the mark indicating the annotation observable area is always displayed, the annotation is automatically displayed when entering the first observable area, and the mouse and keyboard are used for the second and subsequent entry. It is also possible to display only when there is an input from the user interface such as (eg, pressing a button). An annotation is automatically displayed when entering the first observable area, and when entering for the second time and after, the annotation is displayed when the time remaining in the annotation observable area exceeds a certain value. It can also be displayed.
[0093]
FIG. 20B shows a change in the mark indicating the annotation observable region when different annotations are displayed according to the number of times of entry shown in FIG. Here, the transparency increases and disappears gradually as the number of times increases, but the color may change as the number of times increases. Thereby, the observer can recognize that different annotations are observed for the same object.
[0094]
FIG. 21 shows an example of history information to be saved. History information is provided for each object or annotation image, and is created in memory at the same time the system is started. The observer changes, a certain amount of time has passed, a mouse or keyboard button is pressed, or is far away. It is reset every time it moves to an area or when content is switched, and is erased when the system ends. However, it may be saved as a file with the end of the system and read as the initial value of the history information when the system is restarted. The saved history information file is created as profile data for each observer. When the system is restarted, if the observer's profile data exists, it is read and the initial value of the history information is read. To do.
[0095]
As shown in FIG. 21, the history information records the number of times an annotation is displayed for each object. Depending on the number of times, display / non-display is switched and the annotation to be displayed is changed. In addition, by leaving the boundary when entering the annotation observable area last time as history information, it is possible to display various annotations using the number of entries and the history information of the entry boundary.
[0096]
[Other Embodiments]
Needless to say, the annotation image and mark display methods described in the above embodiments can be implemented in combination of two or more thereof.
[0097]
Further, the virtual space reproduced and presented by the image reproducing device may be not only a two-dimensional image but also a stereoscopic display using a parallax image or a stereoscopic display using the principle of autostereoscopic viewing. Such an image display technique is well known and will not be described further here.
[0098]
In the above-described embodiment, only the walk-through system that moves the viewpoint of the virtual observer using the operation unit without moving the observer itself has been described. However, for example, when an observer wears a head-mounted display (HMD), actually walks on a treadmill or the like, moves in a virtual space, and detects an actual line-of-sight direction with a sensor or the like. The present invention can also be applied to a system that acquires the observer's viewpoint position and line-of-sight direction and displays an appropriate virtual space image on the HMD.
[0099]
In addition, a function equivalent to that of the above-described image reproduction device may be realized by a system including a plurality of devices.
A software program that realizes the functions of the above-described embodiments is supplied directly from a recording medium or to a system or apparatus having a computer that can execute the program using wired / wireless communication. The present invention includes a case where an equivalent function is achieved by a computer executing the supplied program.
[0100]
Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.
[0101]
In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.
[0102]
As a recording medium for supplying the program, for example, a magnetic recording medium such as a flexible disk, a hard disk, a magnetic tape, MO, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R, DVD- There are optical / magneto-optical storage media such as RW, and non-volatile semiconductor memory.
[0103]
As a program supply method using wired / wireless communication, a computer program itself that forms the present invention on a server on a computer network, or a computer that forms the present invention on a client computer such as a compressed file including an automatic installation function A method of storing a data file (program data file) that can be a program and downloading the program data file to a connected client computer can be used. In this case, the program data file can be divided into a plurality of segment files, and the segment files can be arranged on different servers.
[0104]
That is, the present invention includes a server device that allows a plurality of users to download a program data file for realizing the functional processing of the present invention on a computer.
[0105]
In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to the user, and key information for decrypting the encryption for a user who satisfies a predetermined condition is provided via a homepage via the Internet, for example. It is also possible to realize the program by downloading it from the computer and executing the encrypted program using the key information and installing it on the computer.
[0106]
In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.
[0107]
Furthermore, after the program read from the recording medium is written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU of the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.
[0108]
【The invention's effect】
As described above, according to the present invention, it is possible to easily know the region where the observer can observe the annotation by synthesizing and presenting the map image obtained by combining the annotation marks with the virtual space image.
[0109]
Further, according to the present invention, by changing the expression of the annotation mark according to various conditions, it becomes possible for the observer to determine the annotation to be observed according to his / her own interests and desires. It is possible to provide a wealth of information without feeling uncomfortable.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an example of a functional configuration of an image playback device that constitutes a walk-through system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an example of a hardware configuration that can realize the image reproduction apparatus in FIG. 1;
FIG. 3 is a diagram for explaining a configuration example of annotation correspondence data;
FIG. 4 is a diagram illustrating an annotation display availability determination method and a display position.
FIG. 5 is a flowchart illustrating an overall process of the image reproduction device.
FIG. 6 is a diagram illustrating a display example of an annotation mark and an observer mark.
FIG. 7 is a flowchart for explaining map image display processing;
FIG. 8 is a flowchart for explaining annotation display processing;
FIG. 9 is a diagram illustrating another example of an annotation mark display method.
FIG. 10 is a diagram for explaining yet another example of a method for displaying annotation marks.
FIG. 11 is a diagram showing an example of a display screen of the image playback device 1;
FIG. 12 is a diagram for explaining a mark display method performed by the image reproduction apparatus according to the second embodiment.
13 is a diagram for explaining a configuration example of annotation correspondence data for realizing the display method of FIG. 12;
FIG. 14 is a diagram illustrating an example of a user interface for designating attributes of marks displayed by an observer in the second embodiment.
FIG. 15 is a diagram for explaining a mark display method performed by the image reproduction apparatus according to the third embodiment;
16 is a diagram for explaining a configuration example of annotation correspondence data for realizing the display method of FIG. 15;
FIG. 17 is a flowchart for explaining processing different from the flowchart shown in FIG. 8 among annotation display processing performed by the image reproduction device according to the third embodiment.
FIG. 18 is a diagram illustrating a mark display method performed by the image reproduction device according to the fourth embodiment.
FIG. 19 is a diagram illustrating another mark display method performed by the image reproduction device according to the fourth embodiment.
FIG. 20 is a diagram illustrating still another mark display method performed by the image reproduction device according to the fourth embodiment.
FIG. 21 is a diagram illustrating an example of history information data used by the image reproduction device according to the fourth embodiment.

Claims (12)

An image reproduction method for causing the observer to experience a walk-through in the virtual space by detecting a viewpoint position and a gaze direction in the virtual space of the observer and presenting a corresponding virtual space image,
A virtual space image generation step for generating a virtual space image to be presented to the observer based on the viewpoint position and line-of-sight direction in the observer's virtual space and the virtual space data stored in the virtual space data storage means; ,
A map image is read from the map image storage means based on the viewpoint position in the observer's virtual space, and a map image representing a predetermined range of the virtual space including at least the viewpoint position in the observer's virtual space is generated. A map image generation step,
A presentation step of synthesizing the virtual space image and the map image to present to the observer;
The map image generation step includes
Observe the annotation image based on the information about the region where the annotation image related to the object in the virtual space is observable stored in the annotation information storage means and the viewpoint position and the line-of-sight direction of the observer in the virtual space. An image reproduction method comprising: a mark composition step of combining a mark representing a possible area and a mark representing the viewpoint position and line-of-sight direction of the observer into the map image. In the presenting step, when the viewpoint position of the observer is in an area where the annotation image can be observed and the viewing direction of the observer is in a specific range, the annotation image is further converted into the virtual space image. The image reproducing method according to claim 1, wherein the images are synthesized. When a plurality of annotation images that can be observed in the same region are registered, the presenting step includes the plurality of annotation images according to a route in which the observer's viewpoint position enters the region where the annotation image can be observed. 3. The image reproduction method according to claim 2, wherein one of the selected annotation images is selected, and the selected annotation image is combined with the virtual space image. When a plurality of annotation images that can be observed in the same region are registered, the presenting step depends on the number of times that the viewpoint position of the observer has entered the region where the annotation image can be observed, or at random 3. The image reproduction method according to claim 2, wherein one of the plurality of annotation images is selected, and the selected annotation image is combined with the virtual space image. When a plurality of annotation images that can be observed in the same region are registered, and when the number of times that the observer's viewpoint position has entered the region in which the annotation image can be observed is a predetermined number of times or more 3. The image reproducing method according to claim 2, wherein the annotation image is not synthesized with the virtual space image. The image reproducing method according to claim 2, wherein the mark that represents the region where the annotation image can be observed is a mark that also represents the direction of the specific range. The mark composing step further composes a mark representing an object in the virtual space corresponding to the annotation image into the map image. Image playback method. The mark synthesis step changes a visual representation of a mark representing an area where the annotation image can be observed according to a distance relationship between the observer's viewpoint position and the area where the annotation image can be observed. The image reproduction method according to claim 1, wherein the image reproduction method is any one of claims 1 to 7. The mark combining step changes a visual expression of a mark representing an area where the annotation image can be observed according to at least one of a predetermined importance level or type of the annotation image. The image reproduction method according to any one of claims 1 to 8. The mark combining step changes a visual expression of a mark representing an area where the annotation image can be observed according to the number of times the observer's viewpoint position has entered the area. The image reproduction method according to claim 9. An image reproduction apparatus that detects a viewpoint position and a line-of-sight direction in a virtual space of an observer and causes the observer to experience a walk-through in the virtual space by presenting a corresponding virtual space image,
Virtual space image generation means for generating a virtual space image to be presented to the observer based on the viewpoint position and line-of-sight direction in the observer's virtual space and the virtual space data stored in the virtual space data storage means; ,
A map image is read from the map image storage means based on the viewpoint position in the observer's virtual space, and a map image representing a predetermined range of the virtual space including at least the viewpoint position in the observer's virtual space is generated. Map image generating means for
Presenting means for synthesizing the virtual space image and the map image and presenting it to the observer;
The map image generating means
Observe the annotation image based on the information about the region where the annotation image related to the object in the virtual space is observable stored in the annotation information storage means and the viewpoint position and the line-of-sight direction of the observer in the virtual space. An image reproducing apparatus, comprising: a mark composition unit that composes a mark representing a possible area and a mark representing the viewpoint position and line-of-sight direction of the observer into the map image. A program for causing a computer to execute the image reproduction method according to claim 1.

Images