JP2005174024A - Image forming apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To change a method for deciding the display position of annotation according to a relative position between an annotation object and a person who has had an experience, and so always display the annotation at proper positions. <P>SOLUTION: A real photographic image is extracted on the basis of a visual point position in a virtual space from real photographic images at a plurality of spots, and an output image is prepared on the basis of the extracted real photographic image. It is decided in which region among a plurality of regions which is set with respect to the annotation the visual point position exists, and the display method of annotation is switched on the basis of the decision result (S103). In this case, those plurality of regions include a first region including at least a portion of the regions where the object of annotation exists and a second region in the periphery of the first region. When the visual point position is present in the first region, the display position of the annotation is fixed to the predetermined position of an output image, and when the visual point position is present in the second region, the display position of the annotation in the output image is decided on the basis of the visual point position and display coordinates set in the annotation. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for displaying annotations on an image in a virtual space that is constructed based on image data obtained by photographing a real space.

  There has been proposed an attempt to photograph a real space with a photographing device mounted on a moving object, and to represent the photographed real space as a virtual space using a computer based on the photographed real image data (for example, non-patent) Reference 1 or non-patent reference 2).

  As a method for expressing the captured real space as a virtual space based on the real image data, there is a method of reproducing the geometric shape model of the real space from the real image data and expressing it with the conventional CG technology. There are limitations in terms of accuracy and realism. On the other hand, in recent years, Image-Based Rendering (IBR) technology that represents a virtual space by using a real image without reproducing a geometric model has been attracting attention. Since the IBR technique generates a virtual space image based on the photographed image, it is possible to represent the photographed virtual space. Moreover, enormous amounts of time and labor are required to create a geometric model of a vast space such as a city or town, but such a geometric model is unnecessary in IBR technology. No time or effort is needed to create it.

  In order to construct a virtual space that can be walk-through using the IBR technology, it is necessary to generate and present an image according to the position of the experience person in the virtual space. Therefore, in this type of system, each image frame of the live-action image data is stored in association with the position in the virtual space, and the corresponding image frame is acquired based on the position in the virtual space of the experience person and the line-of-sight direction. And play this.

  In such a walk-through in the virtual space, the image frame corresponding to each viewpoint position covers a wider range than the angle of view at the time of playback so that the user can see the desired direction at each viewpoint position. Save as a panoramic image. At the time of reproduction, a panoramic image is read based on the viewpoint position in the virtual space of the experience person, and the partial image is cut out from the panoramic image and displayed based on the viewing direction. When the trajectory of the viewpoint position in the virtual space is the same as the trajectory of the vehicle equipped with the photographing device, the observer can feel as if he / she is running on the vehicle.

Furthermore, by combining and displaying an annotation (annotation) such as a building name on the building in the image being reproduced, richer information can be provided to the observer. It is also possible to display signs that are dark and difficult to see on a live-action image as annotations for easy viewing.
“Cyber imaging of urban spaces using mobile vehicles mounted cameras” (Endo, Katayama, Tamura, Hirose, Watanabe, Tanigawa: Shingaku Society, PA-3-4, pp.276-277, 1997) “Creation of a cyberspace urban space using a mobile vehicle camera (2) -Generation of a wide virtual space using live-action images-” (Hirose, Watanabe, Tanikawa, Endo, Katayama, Tamura: Japan Virtual Reality Society 2nd Conference papers, pp.67-70, 1997)

  In order to effectively display the annotation to the experience person who walks through the virtual space, the display position of the annotation in the image is determined according to the relative movement of the object (annotation target object) to be annotated and the experience person. For example, when an annotation is displayed on a building in the image, the display position of the building in the image changes when the user's line of sight changes, and the display position of the annotation also changes accordingly. Since the annotation always exists on the annotation object for the experiencer, the experiencer can easily grasp the relationship between the object in the image and the annotation.

  However, when the annotation object is too close to the object, the annotation display position may become inappropriate. For example, when an annotation is displayed on the upper side of a building, the annotation goes out of sight when approaching the building. Furthermore, when the experience person and the annotation object are relatively close to each other, the movement of the annotation according to the movement of the experience person becomes intense, and the movement may give the experience person a sense of incongruity. Furthermore, if the annotation target has an area, for example, if the location of the experiencer is within the annotation target area or building, the relative position with the experience is The position where the annotation is displayed in the image becomes indefinite.

The present invention has been made in view of the above problems, and its purpose is to change the method for determining the display position of the annotation depending on the relative position between the annotation object and the user, and always set the annotation at an appropriate position. Is to be displayed.
Another object of the present invention is to obtain the annotation display position even when the relative position between the annotation object and the user is not clear.

In order to achieve the above object, an image generation method according to an aspect of the present invention includes:
An image generation method for extracting a live-action image based on a viewpoint position in a virtual space from a live-action image at a plurality of points, and generating an output image based on the extracted live-action image,
A determination step of determining in which of the plurality of regions the viewpoint position exists in a virtual space in which a plurality of regions related to the annotation are set;
A switching step of switching the display method of the annotation based on the determination result of the determination step,
The plurality of regions include a first region including at least a part of a region where the object of the annotation exists, and a second region around the first region,
The switching step employs a first method in which the display position of the annotation is fixed to a predetermined position of the output image when the viewpoint position is in the first area, and when the viewpoint position is in the second area. The second method of determining the display position of the annotation in the output image based on the viewpoint position and the display coordinates set in the annotation is adopted.

  According to the present invention, the method for determining the display position of the annotation is changed depending on the relative position between the annotation object and the experience person, and the annotation can always be displayed at an appropriate position.

  Hereinafter, a preferred embodiment of the present invention will be described with reference to the accompanying drawings.

  First, the virtual space walk-through system according to the present embodiment will be described. In the present embodiment, for example, panoramic image data is generated from real image data obtained by photographing with a plurality of photographing devices mounted on a moving body such as an automobile, and this panoramic image data is mapped to a position in the real space. Hold it in association with the upper position. Then, a walk-through in the virtual space is realized by generating a display image from the stored panoramic image data according to the viewpoint position (position on the map) and the line-of-sight direction in the virtual space of the experience person.

  FIG. 1 is a block diagram illustrating a functional configuration of the walk-through system according to the present embodiment. The image reproduction device 1 constituting the walk-through system includes an operation unit 10, a viewpoint position / gaze direction determination unit 20, a map data storage unit 30, an image reproduction control unit 40, an annotation display position determination unit 50, and an annotation data storage unit 60. The image data storage unit 70 and the display unit 80 are included. The function of each part will become clear from the following description.

  FIG. 2 is a block diagram showing a hardware configuration of the image reproduction apparatus 1 in the present embodiment. The hardware configuration shown in FIG. 2 is equivalent to that of a normal personal computer. In FIG. 2, a disk 105 constitutes an image data storage unit 70. The disk 105 also constitutes a map data storage unit 30 and an annotation data storage unit 60 at the same time. Examples of the disk 105 include a hard disk, a CD, and a DVD.

  The CPU 101 executes a program stored in the disk 105 or the ROM 106 or an external storage device (not shown) and loaded in the RAM 107 to thereby execute the viewpoint position / gaze direction determination unit 20, the image reproduction control unit 40, the annotation display position. It functions as the determination unit 50.

  When the CPU 101 gives various display instructions to the display controller 102, a desired display is made on the display device 104 by the display controller 102 and the frame buffer 103. In FIG. 2, a CRTC is shown as the display controller 102 and a CRT is shown as the display 104. However, the display is not limited to a cathode ray tube, and a liquid crystal display or the like may be used. The CRTC 102, the frame buffer 103, and the CRT 104 constitute the display unit 80 described above.

  The mouse 108, the keyboard 109, and the joystick 110 are for performing user operation input to the image playback apparatus 1, and constitute the operation unit 10 described above.

  Next, an outline of the operation of the image reproduction device 1 in the walk-through system of the present embodiment having the above configuration will be described.

  The operation unit 10 generates a viewpoint position movement parameter and a line-of-sight rotation parameter. In this embodiment, the joystick 110 is used for control of the viewpoint position and the line-of-sight direction, but it is obvious that it may be performed by another input device such as a game controller. The joystick 110 can independently control the tilt angle and the rotation angle of the stick. In the present embodiment, the operation of tilting the stick of the joystick 110 corresponds to the movement of the viewpoint position in the virtual space, and the operation of rotating the stick left and right corresponds to the rotation in the line-of-sight direction in the virtual space.

  The map data storage unit 30 stores two-dimensional map data. A map image can be generated and displayed based on this map data. The viewpoint position / gaze direction determination unit 20 is based on the movement parameter and the rotation parameter input by the operation unit 10, and the observer's viewpoint position and gaze direction on the two-dimensional map stored in the map data storage unit 30. To decide. That is, the viewpoint position and the line-of-sight direction in the virtual space of the experience person experiencing the walk-through correspond to the position and direction on the two-dimensional map.

  The image data storage unit 70 stores panoramic image data corresponding to each position on the two-dimensional map. Note that the image data storage unit 70 does not have to exist locally in the image reproduction apparatus 1. For example, the image data storage unit 70 may be installed on the network, and the image data may be read out via the network.

  The image reproduction control unit 40 receives the viewpoint position and line-of-sight direction on the map of the observer from the viewpoint position / line-of-sight direction determination unit 20, and reads panoramic image data corresponding to the viewpoint position from the image data storage unit 70. In order to read panoramic image data corresponding to the viewpoint position from the image data storage unit 70, in this embodiment, association data that associates the viewpoint position on the map with the image data is used. Although this association data may be stored anywhere in the system, it is assumed that it is stored in the image data storage unit 70 in this embodiment. The association data has, for example, the following data storage form.

  That is, the movement of the observer is limited to the photographing route, the route is divided by a dividing point such as an intersection (branch point) or a turning corner, and expressed using a dividing point and a road sandwiched between two dividing points. The division points are set on the two-dimensional map, and the road is a line segment between the division points. An ID is assigned to each division point or road, and a panoramic image group in which a panoramic image photographed at a corresponding real space position is sandwiched between the panoramic images assigned to the division points on both ends on the road. Is assigned. FIG. 3 is a diagram for explaining this situation. In FIG. 3, an ID R <b> 1 is assigned to a line segment (road) sandwiched between a segment point with ID C <b> 1 and a segment point with C <b> 2. When the panorama image corresponding to the segment points C1 and C2 is specified based on the GPS data or the like, the panorama image with the frame number n is allocated to the segment point C1, and the panorama image with the frame number n + m is allocated to the segment point C2. . When a panoramic image is assigned to a segment point, a panoramic image group with frame numbers n + 1 to n + m−1 is automatically assigned to the road R1.

  Further, as shown in FIG. 4, each segment point and each panoramic image has a two-dimensional coordinate on the two-dimensional map as an attribute. The two-dimensional coordinates on the map are obtained from latitude / longitude data obtained from GPS data, but may be obtained from image information using a computer vision. Alternatively, only the coordinates of the segment points at both ends are obtained, and the two-dimensional coordinates on the two-dimensional map of each panoramic image included in the path between the segment points are interpolated (for example, the path between segment points C1 and C2 in FIG. 3). And m-1 panoramic images (n + 1 to n + m-1) are assigned to each delimiter position). By using the association data as described above, the image reproduction control unit 40 can acquire the image data corresponding to the viewpoint position on the two-dimensional map from the image data storage unit 70.

  In addition, the image reproduction control unit 40 gives the annotation display position determination unit 50 the viewpoint position on the map. The annotation display position determination unit 50 determines the display position of the annotation based on the received viewpoint position information, and gives the annotation display position to the image reproduction control unit 40. A method for determining the display position of the annotation will be described later. The image reproduction control unit 40 cuts out a partial image from the panorama image corresponding to the viewpoint position in accordance with the angle of view displayed on the display unit 80 and performs projective transformation, and follows the annotation display position received from the annotation display position determination unit 50. An annotation image is combined to generate an image to be displayed on the display unit 80. The display unit 80 displays the image generated by the image reproduction control unit 40.

  Next, the operation of the annotation display position determination unit 50 will be described in detail. As shown in FIG. 5, in the present embodiment, on the road sandwiched between the dividing points, the front direction of the panoramic image is assumed to be parallel to the road direction, that is, parallel to the traveling direction during photographing.

  In the present embodiment, a plurality of areas are set in the virtual space for the annotation object, and the method for determining the display position of the annotation is changed depending on which area (the viewpoint position) the experience person exists in. In the present embodiment, the virtual space is divided into four regions (annotation display mode regions) for each annotation object in order to determine a method for determining the display position of the annotation. FIG. 6 is a diagram showing an example for explaining the present embodiment, and shows a map 601 represented by map data. The map 601 includes a park 602. FIG. 7 is a diagram showing an example of setting the area for the annotation target object (park 602) on the map shown in FIG. In the example of FIG. 7, a park 602 is an annotation object, and an asterisk 701 is the position of the annotation on the two-dimensional map. Reference numeral 702 denotes a region having an object, and reference numeral 703 includes a region 702, which is a region wider than 702. Further, the region 704 includes a region 703 and is wider than the region 703. Furthermore, the area 705 is another area of the virtual space. The regions 702, 703, 704, and 705 may be referred to as display mode regions A, B, C, and D, respectively.

In addition, the determination method of each said area | region can be determined by, for example, how much an annotation target object occupies in an experience person's visual field. For example, it can be determined according to the following criteria. First, the region of the annotation target object 602 is a region 702. An area where the experience person is outside the area 702 and the annotation object occupies 25% or less of the horizontal field of view of the experience person is defined as an area 705. Further, an area where the annotation object occupies 25% to 50% of the horizontal vision of the experience person is defined as an area 704. Then, a region 703 is defined as the location of the user who the annotation target occupies 50% or more of the horizontal field of view of the user.
In addition, here, a square area is given as an example for explanation, but it is not limited to this shape. For example, as shown in FIG. 8, the area may be determined by the distance from the annotation object, Other areas may be determined.

  FIG. 9 is a flowchart for explaining the operation of the annotation display position determination unit 50. First, in step S101, new viewpoint information (viewpoint position, line-of-sight direction) is acquired. Next, in step S102, an object for which an annotation is to be displayed is searched based on the corresponding viewpoint position information from the viewpoint information acquired in step S101. It is assumed that the search condition for the annotation object is in the field of view of the experience person. That is, in step S102, an annotation object that is in the field of view of the experiencer is searched. In the present embodiment, annotations can be displayed for a plurality of objects, and a process when one or a plurality of annotation objects are searched will be described below.

  In step 102, the first annotation object is determined from the searched annotation objects, and the process proceeds to step S103. In step S103, the display position of the annotation in the panoramic image corresponding to the corresponding viewpoint position is calculated. Detailed processing in step S103 will be described later with reference to the flowchart in FIG. Next, end determination is performed in step S104. If there is an annotation target object that has not been processed in step S103 among the annotation target objects searched in step S102, the process proceeds to step S105. In step S105, the next annotation object is determined, and the process returns to step S103. If the display position of the annotation has been determined for all the objects for which the annotation is to be displayed in step S104, this processing ends.

  FIG. 10 is a flowchart for explaining detailed processing for calculating the annotation display position in step S103 of FIG. In steps S111, S112, and S113, it is determined in which display mode region the viewpoint position of the experience person obtained in step S101 is included in the annotation target object. In the present embodiment, the position determination method of the annotation display differs depending on which of the display mode areas A to D is the viewpoint position of the experience person.

  If it is determined in step S111 that the viewpoint position is in the display mode area A, the process proceeds to step S114. On the other hand, when it is determined that the viewpoint position is not in the display mode area A but in the display mode area B, the process proceeds from step S112 to step S115. If it is determined that the viewpoint position is in the display mode area C, the process proceeds from step S113 to step S116, and it is determined that the viewpoint position is not in any of the display mode areas A to C (that is, in the display mode area D). If so, the process proceeds from step S113 to step S117. If the annotation display position is determined in any of steps S114, S115, S116, and S117, the annotation display position calculation process in step S103 is completed, and the process returns to the flowchart of FIG.

  Hereinafter, the display position determination process for each area in steps S114 to S117 will be described in detail.

  First, an annotation display position determination process (step S117) when the viewpoint position is in the display mode area D will be described. In the calculation of the annotation display position in this case, the display position in the image is determined based on the relative positional relationship between the annotation object and the viewpoint position. According to this display position determination method, for example, while the experience person is looking around the virtual space, the display position of the annotation can be always displayed on the video of the annotation target in the image.

  FIG. 11 is a diagram for explaining the annotation display position determination method in step S117. In FIG. 11, 1202 is a segment point C1, 1201 is a panorama image corresponding to the segment point C1, 1209 is a segment point C2, 1204 is a road R1, 1206 is an annotation object, 1205 is a map on which an annotation of the object 1206 is to be displayed. 1207 is the viewpoint position on the road R1, 1208 is the panorama image corresponding to the viewpoint position 1207, 1203 is the annotation display position in the panorama image 1208 corresponding to the viewpoint position 1207 (relative to the front direction of the panorama image) Angle θ).

For simplicity, the segment point C1 is taken as the origin of the xy plane, and the segment point C2 is taken as a point 1209 on the x axis (that is, the road R1 becomes a part of the x axis). In FIG. 11, the coordinates on the map of the position 1205 where the annotations of the segment point C1, the segment point C2, and the object 1206 are to be displayed are (0, 0), (x ₂ , 0), and (x _o , _yo ), respectively. is there. The coordinates on the map of the point 1207 on the road R1 are (x, 0). In the panoramic image corresponding to the point 1207, the horizontal position where the annotation of the object 1206 is displayed is expressed by the following expression when expressed by a relative angle θ (rad) from the front direction of the panoramic image.

  That is, when the viewpoint position is in the display mode region D (step S117), the relative position of the annotation display position (1205) to the viewpoint position (1207) is calculated, and the panorama image (1208) corresponding to the viewpoint position is calculated. Annotation display position (relative angle from the front direction of the panoramic image) θ is obtained by the above equation (1).

  Next, an annotation display position determining method (step S114) when the viewpoint position is in the display mode area A will be described. In the display mode area A, the viewpoint position is assumed to be in the annotation object. For example, this is a case where the viewpoint position is in the site of a park to be annotated or in a building. When the viewpoint position of the experience person is in the display mode area A, the display position of the annotation of the target object is fixed to a predetermined position on the screen displaying the scene, and the viewpoint position and the line-of-sight direction change in the display mode area A. However, the display position of the annotation should not change. Various methods for determining the display position of the annotation in the display mode area A are conceivable. For example, it may be determined as one display position in the entire virtual space, or a fixed display position of annotation may be determined for each annotation target. Moreover, it is good also as a position predetermined by the system, and an experience person may be made to decide a favorite position at the time of experience.

  Next, a method for determining the annotation display position when the viewpoint position is in the display mode area B (step S115) will be described. The display mode area B is an area where the annotation object is adjacent. In this area, when the user's line of sight is directed toward the object, the object occupies the whole or most of the field of view. When the line of sight is directed in the opposite direction of the object, the object cannot be seen (goes out of the field of view).

  FIG. 12 is a diagram illustrating a case where the viewpoint position is in the display mode area B. 1301 is an annotation object (for example, a park, a building, etc.), 1302 is the position of the annotation on the two-dimensional map, 1303 is a display mode area B for the object 1301, and 1304 is a viewpoint position and a panorama corresponding to the position. Images 1304 to 1313 are panoramic images on the road 1314.

  FIG. 13 is an example of a scene when the viewpoint position is in the display mode area B. Reference numeral 1401 denotes an annotation object such as a building, and 1402 denotes a display position set on the annotation map for the annotation object 1401. 1403 is an annotation displayed on the object 1401. 1404 is a panoramic image corresponding to one viewpoint position in the display mode area B, and 1405 is a panoramic image corresponding to another viewpoint position in the display mode area B. Reference numerals 1406 and 1408 denote images when the object 1401 is viewed from the viewpoint 1404 in the line-of-sight direction 1410. Similarly, 1407 and 1409 are images when the object 1401 is viewed at the viewpoint position 1405 and the line-of-sight direction 1411. In the images 1406 and 1407, the annotation 1403 is displayed at the display position relative to the annotation setting position on the map. For this reason, only half of each annotation is displayed in each image. On the other hand, in the images 1408 and 1409, the display position of the annotation is fixed at a predetermined position on the image. As can be seen from FIG. 13, the display positions of the annotations in the images 1408 and 1409 are more effective.

  Therefore, in this embodiment, the determination method of the annotation display position when the viewpoint position is in the display mode area B is as follows. First, if the line-of-sight direction is directed to the annotation object, the same determination method as that for the display mode area A is adopted. That is, when the object is in the field of view, the display position of the annotation on the image does not change even if the viewpoint position and the relative position of the object change, and the object is fixed. When the object comes out from the visual field of the experience person to some extent (for example, when 50% or more of the object comes out of the visual field), processing of different display mode areas is performed. For example, when the line-of-sight direction does not face the object, the annotation is not displayed. In FIG. 13, for example, when the line-of-sight direction is directed to the opposite side of the building 1401, the annotation of the building 1401 is not displayed.

  Next, a method for determining the annotation display position when the viewpoint position is in the display mode area C (step S1106) will be described. As described above, when the relative value between the viewpoint position and the target object changes in the display mode area D, the annotation display position also changes, and in the display mode area B, the annotation display position is fixed. The display mode area C is between the display mode area D and the display mode area B, and a display method for realizing a smooth transition of the annotation display position from the display mode area D to the display mode area B (or vice versa). Is executed in the display mode area C.

  FIG. 14 is a diagram illustrating a case where the viewpoint position is in the display mode area C. Reference numeral 1501 denotes an annotation object (for example, a park or a building), and 1502 denotes the position of the annotation on the two-dimensional map. Reference numeral 1503 denotes a display mode area B for the object 1501, and 1504 denotes a display mode area C for the object 1501. Reference numeral 1508 denotes a viewpoint position in the display mode area C and a panoramic image corresponding to the position, and reference numerals 1505 to 1513 denote panoramic images on the road 1514.

  FIG. 15 shows display scenes and respective annotation displays when the viewpoint positions are in the display mode region D, the display mode region C, and the display mode region B. Reference numeral 1604 denotes an annotation to be displayed. Reference numeral 1601 denotes the display state of the annotation when the viewpoint position is in the display mode area D, for example, the position 1507 (display mode area D) in FIG. As described above, in the display mode region D, the display position of the annotation changes depending on the relative position between the viewpoint and the object. 1603 shows an example of an annotation display when the viewpoint position is in the display mode region B (1503), for example, at the point of position 1510, and when the line of sight is directed toward the object 1501. Annotations are displayed at fixed positions. Reference numeral 1602 denotes a display example of the annotation when the viewpoint position is in the display mode area C, for example, the position 1508. In the present embodiment, the display position of the annotation when the viewpoint is in the display mode area C is determined to be an intermediate position between the display position in the display mode area D and the display position of the display mode area B. The determination method may be a method using linear interpolation, for example, but may be another determination method.

Here, the case where the annotation display position is determined by linear interpolation will be described with reference to FIG. For explanation, it is assumed that the viewpoint position in the display mode area C is at the position 1508. First, the display position is calculated from the position 1508 by the display position calculation method for the display mode area D. That is, the annotation display position on the panoramic image corresponding to the relative position between the viewpoint position 1508 and the annotation position 1502 is calculated using, for example, the above-described equation (1). Next, a fixed display position in the display mode area B (a predetermined position in the image of the cutout range) is acquired. The annotation display position at the viewpoint position 1508 of the display mode area C is the position on the two-dimensional map of the viewpoint position 1507 immediately before entering the display mode area C from the display mode area D ((x ₁₅₀₇ , y ₁₅₀₇ )). And the display position Hd at that time, the position on the two-dimensional map ((x ₁₅₁₀ , y ₁₅₁₀ )) of the viewpoint position 1510 immediately after entering the display mode area B from the display mode area C, the display position Hb at that time, and From the viewpoint position ((x ₁₅₀₈ , y ₁₅₀₈ )) on the two-dimensional map of the viewpoint position 1508 of the display mode area C, it is calculated by the following equation (2).

  The position in the height direction of the annotation display is the horizontal position (Hc, expression (2)) using the height Vd of the annotation display position at the position 1507 and the height Vb of the annotation display position at the position 1510. Similarly, it can be obtained by the following equation (3). In FIG. 17, the panorama display position determined by Hc and Vc is shown at a position off the broken line connecting (Hd, Vd) and (Hb, Vb), but calculated using equations (2) and (3). The set position (Hc, Vc) is on the broken line. However, the inclination of the broken line shown in FIG. 17 changes depending on the line of sight because the position of Hb changes depending on the line of sight.

  In the above description, when the display position in the display mode area C is linearly calculated from the display positions in the display mode area D and the display mode area B, the panoramic image distance on the two-dimensional map is used. The display position may be calculated linearly according to the number of images. For example, the display position is calculated based on the number of panoramic images from position 1508 to position 1507 and the number of panoramic images from position 1508 to position 1510.

  FIG. 16 is a diagram for explaining annotation composition processing in the image reproduction control unit 40. Here, it is assumed that the viewpoint position is in the display mode region D. The annotation display position determination unit 50 determines the annotation display position θ. The image reproduction control unit 40 cuts out a panoramic image according to the line-of-sight direction and the angle of view α. Then, the annotation image read from the annotation data storage unit 60 is synthesized on the extracted panoramic image, and a display image is generated. The oblique conversion for converting the panorama image into the perspective projection image is performed only on the panorama image. Similarly, when the viewpoint position is in another area, the annotation image is synthesized on the cut-out panoramic image at a fixed position or a position calculated by linear interpolation.

  As described above, according to the above-described embodiment, the viewpoint of the observer is determined by switching the annotation display mode according to the relative position on the two-dimensional map between the position of the object on which the annotation is to be displayed and the viewpoint position of the observer. It is possible to display the annotation of the object in an easy-to-understand manner for the observer even if it is far from the object or in the object.

  Note that the present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), or a device (for example, a copier, a facsimile device, etc.) including a single device. You may apply to.

  Another object of the present invention is to supply a storage medium storing software program codes for implementing the functions of the above-described embodiments to a system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the storage medium. Needless to say, this can also be achieved by reading and executing the program code stored in.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a block diagram explaining the functional structure of the walk through system in this embodiment. It is a figure which shows the hardware constitutions of the image reproduction apparatus 1 in this embodiment. It is a figure explaining the representation method of virtual space in this embodiment. It is a figure which shows an example of the attribute of a division point and a road. It is a figure explaining a response | compatibility with a panoramic image and the direction of a road. It is a figure which shows an example of the two-dimensional map of a virtual environment. It is the figure which showed on the map the several area | region of the virtual environment centering on the annotation target object. It is a figure which shows an example in which the annotation display mode area | region was determined by the distance from a target object. 5 is a flowchart for explaining the operation of an annotation display position determination unit 50. It is a flowchart explaining the determination method of an annotation display position with an annotation display mode area. 10 is a diagram illustrating an annotation display position determination method when an observer's viewpoint position is in an annotation display mode area D. FIG. It is a figure explaining the annotation display position determination method in case an observer's viewpoint position exists in the annotation display mode area | region B. FIG. It is a figure explaining the annotation display position determination method in case an observer's viewpoint position exists in the annotation display mode area | region B. FIG. It is a figure explaining the advantage of an annotation display position in case an observer's viewpoint position exists in annotation display mode field C. It is a figure which shows a display scene when an observer's viewpoint position exists in annotation display mode area | region D, the area | region C, and the area | region B, and each annotation display. FIG. 10 is a diagram for explaining annotation synthesis processing when the viewpoint position of the observer is in the annotation display mode region D in the image reproduction control unit 40. It is a figure explaining the determination method of the annotation display position when there exists a viewpoint position in the area | region C. FIG.

Claims (9)

An image generation method for extracting a live-action image based on a viewpoint position in a virtual space from a live-action image at a plurality of points, and generating an output image based on the extracted live-action image,
A determination step of determining in which of the plurality of regions the viewpoint position exists in a virtual space in which a plurality of regions related to the annotation are set;
A switching step of switching the display method of the annotation based on the determination result of the determination step,
The plurality of regions include a first region including at least a part of a region where the object of the annotation exists, and a second region around the first region,
The switching step employs a first method in which the display position of the annotation is fixed to a predetermined position of the output image when the viewpoint position is in the first area, and when the viewpoint position is in the second area. An image output method that employs a second method of determining a display position of the annotation in the output image based on the viewpoint position and display coordinates set in the annotation. The plurality of regions include a third region adjacent to the periphery of the first region,
The switching step employs a third method of turning on / off the display of the annotation according to the first method based on a line-of-sight direction when the viewpoint position exists in the third region. The image output method according to claim 1. The plurality of regions include a fourth region between the second region and the third region,
In the switching step, when the viewpoint position exists in the fourth area, the display position of the annotation on the image determined using the second method at the boundary position between the fourth area and the second area; The image output method according to claim 1, wherein a fourth method for determining a display position of the annotation between the predetermined position on the image in the first method is adopted.   In the fourth method, when the viewpoint position moves on a path from the second area to the third area via the fourth area, the fourth method is in the vicinity where the path enters the second area. On a line segment connecting the display position of the annotation on the image determined using the two methods and the predetermined position on the image in the third method, and the route to the second region of the route 4. The image output method according to claim 3, wherein the display position of the annotation is determined based on an intrusion position, an intrusion position of the route into the third area, and a current viewpoint position on the route. The live-action image is a panoramic image,
Selecting a corresponding panoramic image based on the viewpoint position, and further comprising a clipping step of cutting out a partial image from the panoramic image based on the line-of-sight direction;
The image output method according to claim 1, wherein the second method determines a display position of the annotation in the paranorama image.   The image output method according to claim 1, wherein the plurality of regions are set on a two-dimensional map. An image generation device that extracts a live-action image based on a viewpoint position in a virtual space, and generates an output image based on the extracted live-action image, from a live-action image at a plurality of points,
Determination means for determining in which of the plurality of regions the viewpoint position exists in a virtual space in which a plurality of regions related to the annotation are set;
Switching means for switching the display method of the annotation based on the determination result of the determination means,
The plurality of regions include a first region including at least a part of a region where the object of the annotation exists, and a second region around the first region,
The switching means employs a first method for fixing the display position of the annotation to a predetermined position of the output image when the viewpoint position is in the first area, and when the viewpoint position is in the second area. An image output apparatus adopting a second method for determining a display position of the annotation in the output image based on the viewpoint position and display coordinates set in the annotation.   A control program for causing a computer to execute the image output method according to claim 1.   A storage medium storing a control program for causing a computer to execute the image output method according to claim 1.

Images