JP3983108B2 - How to display images - Google Patents

Description

ここでｈは記憶した全方位画像の実空間における被対象物までの距離である。この関係からズーム率ａ′／ａは、式２で表される。
【式２】

ここで、αの２倍が視野角、ｄは現在格子点からの移動距離、ａは各格子点で表示する部分画像の幅である。観察者が移動するとき、移動距離ｄは、移動速度ｖに現在格子点を出発又は通過してからの時間Δｔを乗じた値となるから、移動速度と経過時間から距離ｄを算出してやれば、ズーム率ａ′／ａが算出できる。従って、切出した画像をこのズーム率に応じて拡大してディスプレイ１に表示することにより、観察者がある方向に移動すると、その移動に応じて画像が変化する。観察者が移動方向に隣接する次の格子点に到達したとき、補完プログラムによって補完された部分画像は、到達した格子点の全方位画像の当該移動方向の部分画像によって置き換えられる。
【００１６】
実空間の広さに対して格子点間隔が粗すぎると、この補完画像から新しい部分画像への置き換えの際に、格子点から被対象物までの距離の大小による画像の飛躍が生ずる。この問題は、格子点間隔を小さくし、隣接する格子点へ移動方向を制限することによって解決することができる。図３に示すように、縦横の格子を等間隔とし、観察者の移動方向を格子線に沿う方向と斜め４５度の方向とに制限すれば、格子点間隔を０．５ｍとしたとき、補完によって処理しなければならない最大の移動距離は、０．７０７ｍとなる。視野角２α＝２０度として部分画像の切出し幅ａ＝６ｍとすると、最大格子点間隔０．７ｍとして観察に耐え得る連続画像が得られる。
【００１７】
次に図６に示すフローチャートを参照して画像表示方法を説明する。観察が開始されると、デフォルトの格子点座標とビュー方向とが与えられ、当該座標で検索した全方位画像がメモリに読み込まれ、デフォルト方向の部分画像が設定された視野角ないし画像幅で切出されてディスプレイ１に表示される。この状態で観察者が方向ジョイスティック２を操作すると、その操作方向によって指令されたビュー方向の部分画像が同様に切出されてディスプレイ１に表示される。観察者が移動ジョイスティック３を操作すると、その操作方向が８方向に近似して入力され、ビュー方向がその移動方向に設定されて表示画面が移動方向の部分画像となり、予め設定された移動速度又は移動ジョイスティックの操作量によって指定される移動速度ｖに直前の格子点を出発又は通過してからの経過時間Δｔを乗じた移動距離ｄが算出され、切出しプログラムが切出し画像のズーム率を計算して、移動量に応じて拡大された部分画像をディスプレイ１に表示する。移動距離ｄが当該方向の格子点間隔に達すると、到達した格子点座標で検索された全方位画像がメモリに読み込まれ、当該全方位画像の移動方向をビュー方向とする部分画像が切出されて、ズーミングにより補完された画像と置き換えられる。
【００１８】
観察者が移動ジョイスティック３の操作方向を変更しなければ、到達した格子点を通過点として上記と同じ動作が繰り返され、観察者はある方向の格子点を渡り歩くようにして進行方向の画像を観察してゆく。観察者が移動ジョイスティック３の操作方向を変換すると、次の格子点に達したときに、ビュー方向と移動方向とが切換えられ、新たな方向での画像表示が繰り返される。従って、格子点間隔を十分小さく取ることによって、観察者は観察エリア内を自由に移動することができるようになる。そして、観察者が移動ジョイスティックの操作を止めれば移動は停止し、その位置で方向ジョイスティック２を操作することにより、自由にビュー方向を変えて周囲を見回すことができる。
【００１９】
上記の例は、各格子点に当該地点に対応する全方位の静止画像を登録して、これを当該格子点の座標をキーにして呼び出すようにした例である。前述したように、実空間の映像は、格子線に沿う全方位の動画として撮影できる。そこで他の方法として、縦横の格子線に沿って撮影した動画を格子点間隔に対応する長さで切り刻んで動画片とし、各動画片をその両端の格子点の座標をキーにして画像データベースに登録する例を実施例として示す。
【００２０】
まずデフォルトの格子点座標とビュー方向とが読込まれる。この実施例では、前回終了時のものがデフォルトとなっている。この実施例では、各格子点が指定されたときに、その格子点から四方に延びる線分に対応する動画片がメモリに読み込まれる。この読み込みは、格子点座標で画像データベースを検索することによって実現できる。
【００２１】
現在格子点に表示可能なフレーム（静止画像）は、読み込まれた４個の動画片の現在格子点側の端にあるから、そのどれを表示するかを判定させるために、予め優先フレームを決めておかなければならない。どれを優先フレームとするかは任意であるが、例えばＸ方向の最後のフレーム（動画片は格子線に沿って一方向に撮影した動画を格子点位置で切断した切片であるから、Ｘ方向の２個の動画片のうち、一方の最初のフレームと他方の最後のフレームとが現在格子点側のフレームである）というように決めておく。格子点位置で表示される画像は、この優先フレームの全方位画像からビュー方向に切出された部分画像である。この状態で方向ジョイスティック２が操作されると、優先フレームの全方位画像が指令された方向で切出された表示される。
【００２２】
移動ジョイスティック３が操作されたとき、操作方向を８方向に近似して移動方向を取得することは図６で説明した例と同じであるが、移動方向が格子線方向であるときの表示が図６の例と大きく異なる。そのため取得した方向が格子線方向か斜めの方向かを判定する判定ステップが設けられ、格子線方向であるときは動画片の再生動作、斜め方向であるときは前述した補完ズーミング動作で移動中の画像を表示する。
【００２３】
すなわち、この実施例では、格子線に沿う方向に移動するときの格子点間の画像は、連続的な動画像として記憶されているので、その動画片を再生してやればよい。この場合、撮影方向と移動方向が同じであれば正方向、反対であれば逆方向に動画片を再生することになる。
【００２４】
メモリ容量を小さくしたいときは、画像データベースからメモリに動画片を読み込むときに、各線分に対応する全方位動画像を現在格子点から隣接する格子点に向う方向の部分画像に切出してメモリに読み込み、最後のフレームのみ（最後のフレームを優先フレームとしたとき）を全方位画像のまま読み込む。
【００２５】
この実施例の場合には、斜め４５度の方向に移動するとき、対応する動画像がないので、斜め方向移動の場合にのみ、前記補完プログラムによる画面表示を行う。すなわち、移動ジョイスティックで指定された移動方向を近似した方向が斜め方向であったとき、現在格子点の全方位画像から当該斜め方向の部分画像を切出して、移動距離に応じて補完方法でズーミングすることによって移動中の画像を表示し、対角線上にある隣接する格子点に達したときに、到達格子点における優先フレームの全方位画像から移動方向に切出した部分画像に置き換えることにより、斜め方向の移動を可能にするのである。
【００２６】
図７は、この実施例のフローチャートを示すもので、観察が開始されると、最初の格子点座標とビュー方向とが読み込まれ、その格子点から延びる４つの格子線の方向の動画片がメモリに読み込まれて、予め定めた優先フレームのビュー方向の部分画像が表示される。観察者が方向ジョイスティックを操作したときの処理は、前述した例と同じである。操作者が移動ジョイスティックを操作したとき、取得した移動方向（近似した移動方向）が格子線の方向であれば、メモリの読み取られた当該方向の動画片を再生する。この動画片の再生が終了すると、隣接する次の格子点に達したことになるから、到達格子点の移動してきた方向の動画片を除く３方向の動画片がメモリに読み込まれる。また、移動方向が斜め方向であると判断されたときは、前述した例と同様な方法で格子点間の部分画像がズーミングにより補完されて表示され、対角方向に隣接する格子点に達したときに、到達格子点の４方向の動画片がメモリに読み込まれて優先するフレームの移動方向の部分画像が表示される。
【図面の簡単な説明】
【図１】 実施例のハードウェア構成を示す図
【図２】 観察エリアに設定された格子点を示す説明図
【図３】 格子点間の観察者の移動を示す説明図
【図４】 各格子点又は格子線上で撮影された円形の画像から所定高さの円筒状の全方位画像への変換を示す説明図
【図５】 仮想空間内での目の移動を部分画像のズーミング処理により補完する方法を示した説明図
【図６】 画像表示手順の例を示すフローチャート
【図７】 実施例の画像表示手順を示すフローチャート
【符号の説明】
１ ディスプレイ
2,3 ジョイスティック
２α視野角
４ 本体装置
Ｍ 全方位画像
Lx,Ly 格子線
Ｐ 格子点[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method of displaying an image taken in a real space on a display as if the observer is moving around in the real space or looking around.
[0002]
[Prior art]
When a local image such as a map or a bird's-eye view is displayed on the display screen, a point symbol is overlaid at a position corresponding to a predetermined specific point on this regional image, and when the point symbol is pointed, An image display method for displaying an image showing the scenery of the point on the display screen is known.
[0003]
Japanese Laid-Open Patent Publication No. 2001-215951 discloses a point provided with a direction indicating element on a regional image displayed on a display screen as a method of displaying image information that enables an observer to control the position and direction in an integrated manner. The applicant of the present invention has proposed an image display method for displaying a symbol and cutting out and displaying an image portion in a specified direction from image information corresponding to the point when direction specifying information of the point is input. Yes.
[0004]
Further, in the above publication, JP 2001-4389 A, JP 2000-244812 A, etc., a trajectory symbol indicating a movement path is displayed on the area image, and the moving object is moved along the trajectory. A method of displaying moving image information indicating a change in the landscape has been proposed.
[0005]
[Problems to be solved by the invention]
The conventional image display means as described above has a feature that the observer can freely select and display the landscape information of a plurality of movement trajectories displayed on the regional image. However, according to the conventional method, the observer can only select a specific movement trajectory determined in advance on the area image, and cannot move around freely in the area and see the surrounding scenery.
[0006]
It is an object of the present invention to provide an image display means that enables image information captured in a real space to be displayed on a display screen so that the observer can freely move around in the real space and observe the surroundings. It is an object of the present invention to obtain the above method that can be realized with a smaller auxiliary storage area, memory capacity and CPU speed.
[0007]
[Means for Solving the Problems]
The image display method according to the invention of this application relates an omnidirectional image captured at a large number of photographing points (lattice points in the embodiment) dispersedly set on a two-dimensional plane occupying a predetermined area in real space to the photographing points. When the moving direction of a virtual moving body (eyes in the observer's virtual plane) that is stored in the storage device of the electronic computer and moves in the virtual plane set corresponding to the predetermined area is designated, A partial image cut out from the omnidirectional image associated with the photographing point corresponding to the movement start coordinate of the virtual moving body at a predetermined viewing angle in the moving direction specified is displayed on the display of the electronic computer, A change in the size of the partial image cut out at the viewing angle generated in relation to the amount of movement in the direction is complemented and displayed by continuous zoom-up of the partial image, and the virtual moving object is adjacent to the direction. To the coordinates corresponding to the shooting point The omnidirectional image stored in association with the shooting point corresponding to the arrival coordinate is read and a partial image corresponding to the viewing angle in the moving direction is cut out and displayed, and the virtual image is displayed at the coordinate point corresponding to each shooting point. This is an image display method in which the moving direction of the moving body is maintained or converted and the above operation is repeated.
[0008]
According to the method of setting the shooting point at the intersection of the vertical and horizontal grids, it is easy to take an omnidirectional image at a large number of shooting points, input it to a storage device, and arithmetic processing in an electronic computer. There is.
[0009]
According to the first aspect of the present invention, which solves the above-mentioned problem, a moving image taken while moving along the grid lines of a vertical and horizontal grid set in a two-dimensional plane occupying a predetermined area in real space is displayed for each grid point interval. When a moving direction of a virtual moving body that moves in a virtual plane set in correspondence with the predetermined area is specified as a separated moving image piece, stored in a storage device of an electronic computer in association with lattice points at both ends thereof In addition, when the moving direction of the virtual moving body coincides with the direction of the grid line, a moving picture piece between the grid points from the movement start point to the adjacent arrival point is reproduced and displayed on the display of the computer, When reproduction is finished, a partial image cut out at a predetermined viewing angle in the moving direction is cut out from the omnidirectional image at the arrival point and displayed, and when the designated moving direction is a direction deviating from the grid line, The virtual move A partial image cut out from the omnidirectional image of the movement start point of the body at the viewing angle specified in the moving direction is displayed on the display, and the viewing angle generated in relation to the amount of movement of the virtual moving body in the direction is displayed. The change in the size of the extracted partial image is complemented and displayed by continuous zoom-up of the partial image, and the omnidirectional image of the arrival point is displayed when the virtual moving object reaches an adjacent grid point in the direction. A method for displaying a moving object visual field, comprising: reading and displaying a partial image corresponding to the visual field in the moving direction, and maintaining or converting the moving direction of the virtual moving object at each lattice point and repeating the above operation. It is.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing an example of a hardware configuration for carrying out the method of the present invention, and includes a display 1, two joysticks 2 and 3, and a main body device 4. The main unit 4 has a CPU, an auxiliary storage device such as a memory and a hard disk, and an image database, an image table program, an image cutting program, an image complementing program, a moving image reproduction program, etc. are registered in the auxiliary storage device. Yes.
[0011]
In the method of the present invention, the omnidirectional image registered in the database is acquired as an omnidirectional image at each grid point P when the real space to be observed is partitioned by vertical and horizontal grid lines Lx and Ly at predetermined intervals. This omnidirectional image is taken, for example, by moving a video camera with a fisheye lens facing upward along a grid line Lx, Ly assumed in real space at a constant speed, and the frame at each grid point position is the coordinates of the grid point. Is stored as an index.
[0012]
The photographed omnidirectional image is a disk-shaped image as shown in FIG. 4 (a), which is cut out in a ring shape with a radial width corresponding to the height of the desired real space ( Region A) in FIG. 4A is corrected to an image obtained by developing a 360-degree cylinder as shown in FIG. 4B and registered in the image database.
[0013]
The registered image information is searched based on the coordinates of the lattice point when a certain lattice point is designated, and is read into the memory. The omnidirectional image read into the memory is cut into a rectangular partial image by the cutting program, and the partial image is displayed on the display 1 by the image display program. When the cutout program receives the information on the view direction, the cutout program cuts out the partial image in the direction based on the direction information and a pre-registered viewing angle. That is, as shown in FIG. 5, an image of a region having a viewing angle 2α is cut out from the cylindrical omnidirectional image M and sent to the image display program. The image display program displays the received partial image on the display 1.
[0014]
The view direction is commanded by the operation direction of the direction joystick 2 or the drag direction of the mouse. Therefore, if the observer operates the direction joystick 2 in a desired direction, an image in an arbitrary direction at the current lattice point can be displayed on the display 1.
[0015]
An image that continuously changes when the observer moves in the observation area is generated by a supplement program that applies a zoom technique. That is, in FIG. 5, when viewing the partial image of the predetermined viewing angle 2α at the current grid point position P and moving in that direction by the distance d, the viewing angle 2α at the moving point Q does not change, and therefore, before the movement. The width a ′ of the partial image at the moving point Q after moving by d with respect to the width a of the partial image at the lattice point P is given by Equation 1.
[Formula 1]

Here, h is the distance to the object in the real space of the stored omnidirectional image. From this relationship, the zoom rate a ′ / a is expressed by Equation 2.
[Formula 2]

Here, twice the α is the viewing angle, d is the moving distance from the current grid point, and a is the width of the partial image displayed at each grid point. When the observer moves, the moving distance d is a value obtained by multiplying the moving speed v by the time Δt after starting or passing through the current grid point. Therefore, if the distance d is calculated from the moving speed and the elapsed time, The zoom rate a ′ / a can be calculated. Therefore, when the observer moves in a certain direction by enlarging the cut image according to the zoom ratio and displaying it on the display 1, the image changes in accordance with the movement. When the observer reaches the next grid point adjacent in the moving direction, the partial image complemented by the complementing program is replaced with the partial image in the moving direction of the omnidirectional image of the reached grid point.
[0016]
If the lattice point interval is too coarse with respect to the real space, an image jumps due to the distance from the lattice point to the object when the complementary image is replaced with a new partial image. This problem can be solved by reducing the lattice point interval and limiting the moving direction to adjacent lattice points. As shown in FIG. 3, if the vertical and horizontal grids are equally spaced and the movement direction of the observer is limited to a direction along the grid line and a direction of 45 degrees obliquely, complementation is achieved when the grid point spacing is 0.5 m. The maximum travel distance that must be processed by is 0.707 m. Assuming that the viewing angle 2α = 20 degrees and the cut-out width a of the partial image is 6 m, a continuous image that can withstand observation can be obtained with a maximum lattice point interval of 0.7 m.
[0017]
Next will be described the images display method with reference to the flowchart shown in FIG. When observation is started, the default grid point coordinates and view direction are given, the omnidirectional image searched with the coordinates is read into the memory, and the partial image in the default direction is cut at the set viewing angle or image width. And displayed on the display 1. When the observer operates the direction joystick 2 in this state, a partial image in the view direction commanded by the operation direction is similarly cut out and displayed on the display 1. When the observer operates the movement joystick 3, the operation direction is approximated and input in eight directions, the view direction is set to the movement direction, and the display screen becomes a partial image in the movement direction. A moving distance d is calculated by multiplying the moving speed v specified by the operation amount of the moving joystick by the elapsed time Δt from starting or passing through the previous grid point, and the cutting program calculates the zoom rate of the cutting image. The partial image enlarged in accordance with the movement amount is displayed on the display 1. When the movement distance d reaches the lattice point interval in the direction, the omnidirectional image searched with the reached lattice point coordinates is read into the memory, and a partial image having the movement direction of the omnidirectional image as the view direction is cut out. Thus, it is replaced with an image complemented by zooming.
[0018]
If the observer does not change the operation direction of the moving joystick 3, the same operation as described above is repeated with the reached lattice point as the passing point, and the observer observes the image in the traveling direction as he walks around the lattice point in a certain direction. I will do it. When the observer changes the operation direction of the movement joystick 3, when the next lattice point is reached, the view direction and the movement direction are switched, and image display in a new direction is repeated. Therefore, by making the lattice point interval sufficiently small, the observer can move freely within the observation area. Then, when the observer stops operating the moving joystick, the movement stops, and by operating the direction joystick 2 at that position, it is possible to freely change the view direction and look around.
[0019]
The above example, registers the omnidirectional still images corresponding to the point on the grid points, which is an example of a call to the key coordinates of the grid point. As described above, an image in real space can be taken as an omnidirectional moving image along a grid line. Therefore, as another method, a moving image shot along vertical and horizontal grid lines is cut into pieces corresponding to the interval between the lattice points to form moving image pieces, and each moving image piece is stored in the image database using the coordinates of the lattice points at both ends as keys. an example of registering as a real施例.
[0020]
First, the default grid point coordinates and view direction are read. In this embodiment, the one at the end of the previous time is the default. In the real施例this, when each lattice point is designated, the moving piece is loaded in the memory corresponding to the line segment extending from the grid point in four directions. This reading can be realized by searching an image database with lattice point coordinates.
[0021]
Since the frame (still image) that can be displayed at the current grid point is at the end of the read four moving image pieces on the current grid point side, a priority frame is determined in advance in order to determine which one is displayed. I have to keep it. Which frame is the priority frame is arbitrary. For example, the last frame in the X direction (the moving image piece is a slice obtained by cutting a moving image taken in one direction along the lattice line at the lattice point position. Of the two moving image pieces, the first frame on one side and the last frame on the other side are frames on the grid point side). The image displayed at the lattice point position is a partial image cut out in the view direction from the omnidirectional image of the priority frame. When the direction the joystick 2 is operated in this state, that is displayed omnidirectional image priority frame is cut out in the commanded direction.
[0022]
When the movement joystick 3 is operated, the operation direction is approximated to eight directions to acquire the movement direction, which is the same as the example described in FIG. 6, but the display when the movement direction is the grid line direction is shown in the figure. This is very different from the example of 6 . That direction obtained for the determination step is provided whether grid line direction or an oblique direction, the reproducing operation of the moving pieces when a grid line direction, when a diagonal direction is moving at complementing the zooming operation described above Display an image.
[0023]
That is, in the real施例this, the image between the grid points when moving in a direction along the grid lines, because it is stored as a continuous moving image, may do it to play the video piece. In this case, the moving image piece is reproduced in the forward direction if the shooting direction and the moving direction are the same, and in the opposite direction if the moving direction is opposite.
[0024]
When you want to reduce the memory capacity, when you load a movie fragment from the image database to the memory, cut out the omnidirectional video corresponding to each line segment to a partial image in the direction from the current grid point to the adjacent grid point and load it into the memory. Only the last frame (when the last frame is a priority frame) is read as an omnidirectional image.
[0025]
In the case of real施例this, when moving in the direction of angle of 45 degrees, because no corresponding moving image, only in the case of an oblique movement, and screen display by the complementary program. That is, when the direction approximated to the moving direction specified by the moving joystick is an oblique direction, a partial image in the oblique direction is cut out from the omnidirectional image of the current lattice point, and zoomed by a complementing method according to the moving distance. The moving image is displayed, and when an adjacent grid point on the diagonal line is reached, it is replaced with a partial image cut out in the moving direction from the omnidirectional image of the priority frame at the reached grid point. It allows movement.
[0026]
Figure 7 shows a flowchart of the real施例this, the observation is started, and the first grid point coordinate and the view direction are read, the direction of the moving piece of the four grid lines extending from the grid point Are read into the memory, and a partial image in the view direction of a predetermined priority frame is displayed. The processing when the observer operates the direction joystick is the same as the example described above . When the operator operates the movement joystick, if the acquired movement direction (approximate movement direction) is the grid line direction, the moving image piece in the direction read from the memory is reproduced. When the reproduction of the moving image piece is completed, the next adjacent lattice point is reached, and therefore the moving image piece in three directions excluding the moving image piece in the direction in which the arrival lattice point has moved is read into the memory. When it is determined that the moving direction is an oblique direction, the partial image between the lattice points is complemented and displayed by zooming in the same manner as in the above example, and the lattice points adjacent to the diagonal direction are reached. Sometimes, the moving image pieces in the four directions of the reaching lattice points are read into the memory, and the partial images in the moving direction of the priority frame are displayed.
[Brief description of the drawings]
FIG. 1 is a diagram showing a hardware configuration of an embodiment. FIG. 2 is an explanatory diagram showing lattice points set in an observation area. FIG. 3 is an explanatory diagram showing movement of an observer between lattice points. Explanatory drawing showing conversion from a circular image photographed on a grid point or grid line to a cylindrical omnidirectional image of a predetermined height. FIG. 5 Complements eye movement in virtual space by zooming processing of a partial image. flowchart eXPLANATION oF REFERENCE nUMERALS showing an image display procedure in the flowchart FIG. 7 real施例showing an example of illustration Figure 6 images display procedure of how to
1 Display
2,3 Joystick 2α Viewing angle 4 Main unit M Omnidirectional image
Lx, Ly Grid line P Grid point

Claims (1)

An omnidirectional video shot while moving along the grid lines of a vertical and horizontal grid set in a two-dimensional plane that occupies a predetermined area in real space is related to the grid points at both ends as a video piece cut at each grid point interval Stored in advance in the storage device of the electronic computer ,
When a moving direction of a virtual moving body that moves in a virtual plane set corresponding to the predetermined area is designated,
In the electronic computer,
The commanded moving direction is obtained by approximating the direction of the grid line and the direction deviating from the grid line,
When the acquired moving direction matches the direction of the grid line , the moving image piece between the grid points from the moving start point to the adjacent reaching point is cut out into a partial image in the direction toward the reaching point and reproduced and displayed. is displayed, is displayed on the playback cut out partial image to be cut out in a predetermined view angle of the moving direction from the omnidirectional image in point reached when the finished said display,
When the acquired moving direction is a direction deviating from the grid line, a partial image cut out at the viewing angle of the specified moving direction from the omnidirectional image of the moving start point of the virtual moving body is displayed on the display, A change in the size of the partial image cut out at the viewing angle that occurs in relation to the amount of movement of the virtual moving body in the direction is complemented by continuous zoom-up of the partial image, and is displayed on the display. When the moving body reaches an adjacent grid point in the direction, the omnidirectional image of the arrival point is read and a partial image corresponding to the visual field in the moving direction is cut out and displayed on the display .
And it is maintaining or converting the moving direction of the mobile virtual characterized by causing repeat the operation at each grid point, the display method of the image.

Images