KR20020075965A - Realization method of virtual navigation using still photograph - Google Patents

Abstract

PURPOSE: A method of realizing virtual navigation using a still image is provided to realize virtual navigation with a small storage space and rapid processing speed, to reduce a system load and to maximize the sense for the real. CONSTITUTION: An object to be photographed is divided into lattice points and still pictures of all directions of each lattice point are obtained(ST100), to form a panorama picture and store the picture in a database. A still picture with respect to each direction of the line of vision is extracted from the panorama picture according to a reverse panorama transformation(ST140). Characteristic points are extracted from the extracted still pictures(ST150). Correlation information of still pictures having the same direction of the line of vision as that of lattice points adjacent to each other is extracted for all lattice points according to the extracted characteristic points, to store the correlation information in the database. When a movement direction and speed are given by an observer, the still picture with respect to the direction of the line of vision of the lattice point of a starting point and the still picture with respect to the direction of the line of vision of the lattice point of an object to be moved are read from the database, and a picture for the movement direction is generated according to the correlation information based on the read still pictures and displayed.

Description

Realization method of virtual navigation using still photograph}

The present invention relates to a method for implementing virtual navigation, and to implementing a virtual navigation using a still picture to enable a video to be implemented when a viewer moves in the virtual reality by the still picture of the observer gaze direction of the movement start position and the movement completion position It is about.

In general, the virtual reality is implemented by 3D computer graphics work.

That is, the virtual 3D space is rendered by the 3D graphics application and related hardware.

However, since the virtual reality by the three-dimensional graphics does not set the background of the real environment, there is a disadvantage that the reality is inferior.

In order to solve these drawbacks, a real navigation may be implemented using a real photo, and when a virtual navigation video is implemented using the real photo, only a pre-prepared video is displayed regardless of an observer's intention.

In other words, it is a virtual navigation given unilaterally rather than virtual navigation in which the observer participates.

In addition, in the case of the virtual navigation environment implemented according to the observer's intention, the video data according to the observer's movement path must be provided, so the amount of the virtual navigation is not only difficult, but also the storage capacity of the virtual navigation is very large. .

The present invention was devised to solve the above problems, and divides a real object to be implemented by virtual navigation into grid points, and then takes a panoramic picture by a still picture at each grid point, and positions of arbitrary grid points. When the observer moves to the position of the next grid point at, the operation is easy by synthesizing the virtual video by the various information of the still picture between the grid point of the starting position and the grid point of the moving completion position. The purpose of the present invention is to provide a virtual navigation method using still pictures to implement navigation.

1 is a block diagram showing an apparatus for implementing virtual navigation using the still picture of the present invention.

2 and 3 are views for explaining a process of forming a panoramic picture by a still picture.

4 is a conceptual diagram showing a panoramic picture generated at each grid point of a real environment to implement virtual navigation.

5 is a diagram for explaining projective transformation.

Fig. 6 is a diagram for explaining the relationship between the still photographs in the eye direction seen when moving from an arbitrary lattice point to a lattice point of a moving object.

7 is a view for explaining a feature point correlation between a feature point of a photograph viewed from an arbitrary grid point in a line of sight and a photograph in a line of sight viewed from a moving object grid point;

Fig. 8 is a view showing photographs in the eye direction seen at arbitrary grid points and grid points of moving objects and feature points extracted from these pictures.

FIG. 9 is a diagram for describing a process of obtaining triangular figure information from respective feature points of FIG. 8; FIG.

10 and 11 are diagrams for explaining a process of generating an intermediate triangle by being divided equally n according to the moving speed of an observer between figures of triangles having correlation with each other obtained in FIG.

12 is a view showing a process of generating a picture at the position of the triangle created in the intermediate step.

13 and 14 are flowcharts illustrating a method for implementing virtual navigation using the still picture of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: image input unit 110: coordinate input unit

120: central processing unit 130: database

131: panorama database 132: feature point database

133: tracking information database

134: triangular partitioning information database

140: display unit

Virtual navigation implementation method using a still picture of the present invention for achieving the above object,

A first process of dividing a photographing object into a grid to obtain still pictures of 360 degree directions at each grid point, forming a panoramic picture, and storing the photograph in a database;

A second process of extracting a still picture for each gaze direction by an inverse panorama transformation from the formed panorama picture;

A third process of extracting feature points from the extracted still pictures;

A fourth step of extracting correlation information of still pictures having the same gaze direction as neighboring grid points for all grid points by the extracted feature points and storing the correlation information in a database;

Given the direction and speed of movement from the observer, a still picture of the gaze direction of the grid point of the starting point and a still picture of the gaze direction of the moving object grid point are read from the database and based on the correlation information, And a fifth process of generating and displaying an intermediate picture.

The first process may include performing a projective transformation to correct an inclined state of the photographing object in the captured still picture.

The fourth process may include a first step of extracting a location of a feature point having the most approximate value among feature points of a still picture having the same line of sight as a neighboring grid point for all grid points, and storing the information thereof in a database;

The third feature is performed by collecting three nearest feature points in each picture to form triangles and storing the triangle information in a database.

In the fifth process, when a moving direction and speed are given from an observer, a still picture of a gaze direction of a starting grid point, a still picture of a gaze direction of a moving grid point, and triangular information according to a characteristic point thereof are read from a database. First step;

A second step of generating n triangles in the middle between triangles having a high correlation between the still picture of the starting point of the grid point and the still picture of the moving point of the grid point;

Performing a third step of applying a transformation matrix inside the generated n triangles to convert and position a pixel of the still picture for the gaze direction of the starting grid point or the still picture for the gaze direction of the moving grid point, and then display the pixel. It is characterized by.

A method for implementing virtual navigation using the still picture of the present invention made as described above will be described in detail with reference to the accompanying drawings.

First, a photographing object (actual distance, a model of an exhibition hall, a static object, etc.) is divided into grid points through a camera, and a still picture of 360 degree omnidirectional is taken at each grid point (ST100).

That is, as shown in FIG. 2, still pictures P1, P2, P3, P4,..., P10 of 360 degrees in the lattice point position O are taken.

At this time, each still picture (P1, P2, P3, P4, ..., P10) is taken so as to fully overlap with the neighboring still picture (hatched portion in the drawing).

Still pictures P1, P2, P3, P4,..., P10 at each grid point position are applied to the central processing unit 120 through the image input unit 100, and are shown in FIG. 3 by a stitching technique. A panoramic photograph P as shown is formed (ST120).

The image input unit 100 may be various digital input devices such as a digital camera or a scanner, and a panoramic photo stitching tool for viewing 360 degrees in all directions is known as Apple's QuickTime VR, IPIX, Live Picture, and Arut. You can use the company's quick squelch.

In this case, when the object is photographed through the camera, as shown in FIG. 5, the object may be photographed at an inclined angle according to the position of the camera, thereby performing image processing as if photographed from the front through a perspective transformation (perspective transformation). ST110).

This projective transformation may be performed by Equation 1 below.

Projective Conversion: (x, y) ~ → ~ (x ', y')

Here, the coefficients (a_11, a_12, a_13, a_21, a_22, a_32, a_33) of the projective transformation are predicted by a nonlinear regression method using standard data.

By the above method, as shown in FIG. 4, the panorama picture P formed by the still picture taken at the positions of the respective grid points O1, O2, O3,..., O10, O11, 012 is obtained from the data. It is stored in the panorama database 131 of the base 130 (ST130).

In this way, the amount of data stored will be much less than when still pictures are saved.

That is, if there are M grid points in the X-direction, and N still in the Y-direction and P still pictures at each grid point, the total number of still pictures is M * N * P.

However, when the panorama image data is generated using these data, the amount of data is much smaller than this.

Meanwhile, the central processing unit 120 reads a panoramic picture of each grid point stored in the panoramic database 131 and performs inverse panoramic conversion, which forms a still picture from the panoramic picture (ST140).

In this way, the central processing unit 120 extracts the feature points from the still pictures formed in the reverse from the panorama picture, this feature point extraction process will be described.

Contrast information is extracted from each still picture, and then the rate of change of contrast on the x- and y-axes of each pixel is calculated.

At this time, the change rate of the contrast is calculated by the following equations (2) and (3).

DI = (dI / dx, dI / dy)

Here, I is a black and white picture made from contrast information or a color picture.

Thereafter, an eigenvalue of the 2 * 2 matrix of Equation 3 is obtained, and the smaller one is taken as the feature point value at the pixel position.

Only enough of the feature point values obtained at each pixel (about 400 per picture) is selected to determine the point as the feature point.

As a result, the position where the contrast sharply changes is determined as the position of the feature point.

When the feature point is extracted from all still pictures formed from the panorama picture by the feature point extraction process, the feature point database 132 stores the feature point information thereof.

Thereafter, the central processing unit 120 extracts feature point tracking information from the still picture of the neighboring grid point by using the feature point extracted from the still picture of each grid point. For example, as illustrated in FIG. 6, the grid point O6 is extracted. A feature point correlation, that is, tracking information, of the still picture I2 of the grid point O2 having the same visual direction as the still picture I1 of FIG.

That is, as shown in FIG. 7, the scale appears differently according to the position where the observer sees the same object, and the object will appear small at the position of the grid point O6, and the same object at the position of the grid point O2. Will look larger.

Therefore, based on the specific feature point coordinates I1 (x ₁ , y ₁ ) in the picture I1, the corresponding coordinate I2 (x) that is the minimum in the square of the difference between the picture I1 and the contrast I1-I2 in the picture I2. ₂ , y ₂ )

This can be summarized by the following equation (4).

As a result, the position where the difference in contrast between the two pictures I1 and I2 is the smallest is found, which is the feature point tracking information, which is stored in the tracking database 133 (ST160).

In reality, however, the image noise is taken into account, so a 7 * 7 window around a pixel corresponding to the corresponding coordinates is captured, and then the image noise type is assumed to be Gaussian. Noise can be attenuated by using a Gaussian mask and convolution.

8A and 8B show that the feature points are found on the picture I2 by tracking the feature points based on the feature point information of the picture I1.

In this case, FIG. 8A is a picture of 'I1', FIG. 8B is a picture of 'I2', and the indexed numbers indicate matching feature points between the two pictures.

As shown in FIG. 9, the central processing unit 120 collects the three nearest feature points among the specific points in the two pictures, divides them into triangles, and stores the triangle information in the triangulation partition database 134. .

As a result, feature point information extracted from each still picture, tracking information between feature points, and triangle information of feature points are extracted and stored in the database 130 (ST170 and ST180).

The display then assumes that the observer moves along the path 'M' from any lattice point (eg lattice point of O6) to lattice point O2 in order to experience the virtual navigation. A process displayed through the unit 140 will be described.

When the observer looks left and right through the coordinate input unit 110 (eg, keyboard, mouse, joystick, etc.) at the position of the grid point O6, the central processing unit 120 inputs the coordinate input unit 110. According to the value, the corresponding panorama picture is read from the panorama database 131 and displayed on the display unit 140.

When the viewer moves to the grid point O2, the database 130 only has a still picture corresponding to the viewing direction at each grid point O6 or O2, and thus a video from the moving path M must be generated.

Therefore, the central processing unit 120 receives data about the moving direction and the moving speed input through the coordinate input unit 110, and shows a photograph (see the direction of the line of sight of the current grid point O6 and the grid point O2 in the moving direction). Triangular information corresponding to I1 and I2 still pictures as shown in FIG. 6 is read from the triangulation division information database 134 (ST200, ST210).

The read triangular information generates a moving picture shown when the grid point O6 moves from the grid point O6 to the grid point O2, which will be described in more detail.

As shown in FIG. 10, a part of the triangle information obtained from the still picture I1 of the grid point O6 and the still picture I2 of the grid point O2 will be described.

When the viewer finishes moving from grid point O6 to grid point O2, the triangle consisting of points 1,2 and 4 will change shape from I1 to I2.

Since the vertex coordinates of each triangle are known, when two triangles are drawn on the same plane, line segments connecting the same feature point can be drawn as shown in FIG. 11 (ST220).

If you want to create n pictures in the middle of the triangles I1 and I2, you can divide these line segments by n and create a triangle for each divided portion (ST230).

The number of parts to be divided into equal parts is determined by adjusting the moving speed of the observer. To slow down the moving speed, the number is divided into a large number, and to increase the moving speed, the number is divided into a relatively small number.

For example, dividing a line segment of a triangle into three can produce two intermediate stages, I 'and I ".

With just two pieces of information, we can get a 3 * 3 homography matrix H that transforms (x ₂ , y ₂ , 1) = H (x ₁ , y ₁ , 1) (affine transformation). If the transformation matrix H is obtained from the triangle of the I1 photo and the triangle (I ', I ") of the intermediate stage, the intermediate matrix (I', I") is applied by applying the transformation matrix H to the pixels inside the triangle of the I1 photo. The location of the pixels inside is determined.

That is, for any intermediate triangle (I ', I "), the pixels inside the 1, 2, and 4 feature points of the I1 picture will be transformed into the correct positions inside the triangle (I', I") of the intermediate picture. It becomes possible (ST240).

Applying the above process to all the triangles created inside the pictures I1 and I2, it is possible to draw the pixels of all areas of the intermediate pictures.

After all, if the feature point of the first picture I1 and the feature point information (coordinates) of the second picture I2 corresponding thereto are known, as shown in FIG. ...) can be created so that the video can be implemented naturally during the movement.

Such a virtual navigation implementation method using the still picture of the present invention has the following effects.

1) Since only the correlation information between the panorama picture at each grid point and the picture generated when the grid point is moved between the grid point and the grid point is stored, it is possible to implement a small storage space.

2) Since the panorama picture and the correlation information are pre-calculated and stored, the virtual video can be realized by simply synthesizing the pictures according to the information in the actual display, thereby reducing the load on the system and having a high processing speed.

3) The observer can freely choose the direction of movement in the virtual space consisting of due diligence, which has the effect of maximizing reality.

Claims (5)

A first process of dividing a photographing object into a grid to obtain still pictures of 360 degree directions at each grid point, forming a panoramic picture, and storing the photograph in a database; A second process of extracting a still picture for each gaze direction by an inverse panorama transformation from the formed panorama picture; A third process of extracting feature points from the extracted still pictures; A fourth step of extracting correlation information of still pictures having the same gaze direction as neighboring grid points for all grid points by the extracted feature points and storing the correlation information in a database; Given the direction and speed of movement from the observer, a still picture of the gaze direction of the grid point of the starting point and a still picture of the gaze direction of the moving object grid point are read from the database and based on the correlation information, And a fifth process of generating and displaying an intermediate picture. The method of claim 1, The first process may include performing projective transformation to correct an inclined state of a photographing object in the captured still picture. The method of claim 1, The fourth process may include a first step of extracting a location of a feature point having the most approximate value among feature points of a still picture having the same line of sight as a neighboring grid point for all grid points, and storing the information thereof in a database; And forming a triangle by collecting three nearest feature points in each picture and storing the triangle information in a database. The method of claim 1, In the fifth process, when a moving direction and speed are given from an observer, a still picture of a gaze direction of a starting grid point, a still picture of a gaze direction of a moving grid point, and triangular information according to a characteristic point thereof are read from a database. First step; A second step of generating n triangles in the middle between triangles having a high correlation between the still picture of the starting point of the grid point and the still picture of the moving point of the grid point; Performing a third step of applying a transformation matrix inside the generated n triangles to convert and position a pixel of the still picture for the gaze direction of the starting grid point or the still picture for the gaze direction of the moving grid point, and then display the pixel. Virtual navigation implementation method using a still picture, characterized in that. The method of claim 4, wherein The number of newly created triangles is determined by the input of the movement speed of the observer.