JP3392088B2 - Image storage method, image drawing method, image storage device, image processing device, image download method, computer, and storage medium - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of accumulating a virtual image from spatial data generated based on real image data, a method of drawing a virtual space based on the accumulated spatial data, and these devices. In particular, the present invention relates to progressive accumulation of the spatial data, and further to drawing.

[0002]

2. Description of the Related Art Many techniques have been proposed for describing and describing a virtual space based on a photographed image, rather than based on a three-dimensional geometric shape. These are Image Based Rendering
It is called (hereinafter abbreviated as IBR), and is characterized in that it can represent a highly realistic virtual space that cannot be obtained from a method based on a three-dimensional geometric shape because it is based on a real image. . An attempt to describe a virtual space based on the ray space theory, which is a method of IBR, has been proposed. For example, IEICE Transactions "Realization of Virtual Environment by Fusion of CG Model and Ray Space Data" (D-11, Vol. J80-D-11 No.
11, pp3048-3057, November 1997), or "Mutual conversion between hologram and ray space for three-dimensional integrated image communication" (3D Image Conference), and further Japanese Patent Laid- Open No. 10-9.
See No. 7642.

The ray space theory will be described. f As shown in Fig. 1, the coordinate system OXYZ is installed in the real space. A ray passing through the reference plane P (Z = z) perpendicular to the Z axis is
A position (x, y) where the ray crosses P and variables θ and φ indicating the direction of the ray are used. That is, one ray
It is uniquely defined by the five variables (x, y, z, θ, φ). If the function expressing the light intensity of this ray is defined as f, the ray group data in this space can be expressed by f (x, y, z, θ, φ). This five-dimensional space is called "ray space".

Here, if the reference plane P is set to z = 0 and the parallax information in the vertical direction of the light beam, that is, the degree of freedom in the φ direction is omitted, the degree of freedom of the light beam is reduced to two dimensions (x, θ). Can be made. This x-θ two-dimensional space is a subspace of the ray space. Then, if a ray (Fig. 2) passing through a point (X, Z) in the real space is set as u = tan θ, as shown in Fig. 3 in x-u space, [Equation 9] X = x + uZ

Is mapped on a straight line. Shooting with a camera corresponds to an operation of receiving a light beam that passes through the lens focal point of the camera on an imaging surface and imaging its brightness and color. In other words, a group of light rays passing through one point in the real space, which is the focus position, is acquired as an image for the number of pixels. Here, since the degree of freedom in the φ direction is omitted and the behavior of light rays is considered only in the XZ plane, only pixels on a line segment that intersects the plane orthogonal to the Y axis in the image are considered. Become. In this way, light rays passing through one point can be collected by photographing an image, and data on one line segment in the x-u space can be acquired by one photographing.

[0006] If this photographing is carried out a large number of times by changing the viewpoint position (in this specification, unless otherwise specified, the viewpoint position includes both the viewpoint position and the line-of-sight direction), a ray group passing through a large number of points is obtained. Can be earned. N as shown in FIG.
When the real space is photographed using two cameras, the nth (n =
1, 2, ..., N) corresponding to the focal positions (X _n , Z _n ) of the cameras C _n , as shown in FIG.

[Equation 10] x + Z_nu = X_n You can enter the data on the straight line. like this
In addition, by shooting from a sufficient number of viewpoints,
The xu space can be densely filled with data.

On the contrary, it is possible to generate an observation image from a new arbitrary viewpoint position from the data in xu space (FIG. 6) (FIG. 7). As shown in this figure, the observation image from the new viewpoint position E (X, Z) expressed in the form of eyes is
It can be generated by reading the data on the straight line of Expression 1 on the xu space from the xu space.

[0009]

However, in the above-mentioned conventional example, the calculation for converting all the pixels of the photographed image into the light ray group is performed. That is, if there are E actual images and the number of pixels is m × n, then E × m × n
Since the calculation is performed once and converted into the ray group, the amount of calculation becomes very large. In particular, when a ray group is projected onto a ray space so as to maintain the resolution of an input image and the ray space data is discretized, the amount of discretized data becomes enormous.

It is an object of the present invention to propose a method and apparatus for accumulating theoretical light space data so that the light space data of a target object can be progressively displayed.

Another object of the invention proposes a method and apparatus for displaying progressively accumulated spatial data.

[0012]

In order to achieve the above object, an image storage method for storing spatial data configured for a predetermined space different from this image space from image data of a real image is

A first hierarchical division step of dividing the actual image data in the spatial data into hierarchies according to the resolution;

A second hierarchical division step of hierarchically dividing a map representing the correspondence between the actual image space and the predetermined space according to the resolution, wherein the resolution in each hierarchical layer is the first division. A second layer division step set to correspond to the resolution of each layer in the step,

It is characterized by comprising the step of storing the obtained layered image data and layered map data in a storage device.

According to claim 2 which is a preferable aspect of the present invention, the spatial data is ray space data.

According to claim 3 which is a preferable aspect of the present invention, the hierarchy and the resolution are set in ascending order or descending order.

According to claim 4 which is a preferred aspect of the present invention, the layer dividing step performs data thinning processing.

According to claim 5 which is a preferable aspect of the present invention, in the second layer dividing step, a plurality of pieces of the map data are provided corresponding to blocks having a size set according to the number of layer divisions. Block division step of dividing into blocks.

The map of spatial data contains address information. This address information must be properly converted as the map data is layered. Therefore,
According to the method of claim 6, the second layer division step comprises:
If the number of layers is p, the layer to be processed is i, and the storage address of the map data of the i-th layer is n _j , the converted address n _I

[Equation 11]

Is used. According to claim 7 which is a preferable aspect of the present invention, in the block dividing step, when the number of layers is p, the block size is N × N,

[Equation 12]

It is characterized in that it is set to. By satisfying this condition, the layering process is simplified. Hierarchical space data accumulated by the method according to any one of claims 1 to 7 is essentially effective in progressive display at the time of drawing. Therefore, the image drawing method for drawing in the virtual space according to claim 8 which is a preferable aspect of the present invention includes a step of sequentially reading from the hierarchical space data of a layer having a lower resolution and a virtual image is drawn in the order of reading. A drawing process is provided.

According to claim 9 which is a preferable aspect of the present invention, there is a detection step of detecting the viewpoint position of the user, and the drawing step draws a virtual image based on the detected viewpoint position. .

In order to further speed up the drawing of the virtual image, the drawing target may be limited. Therefore, according to the method of claim 10, the reading step reads the hierarchical space data of the space close to the detected viewpoint position.

The above object can also be achieved by providing a storage medium for storing a program for realizing the above-mentioned storage method or drawing method on a computer.

According to claim 12 which is a preferred aspect of the present invention, the present invention is a computer having a storage medium.

Further, the above object can also be achieved by an image storage device or an image processing device.

The above-mentioned problems are
This is also achieved by an image downloading method of downloading spatial data configured for a predetermined space different from the image space. That is, this method

A first hierarchical dividing step of dividing the real image data in the spatial data into hierarchies according to the resolution;

Dividing hierarchized according the downloading process, the pre-Symbol map resolution for download [0032] The resulting layered image data wherein the real image space and a map representing the correspondence relationship between the predetermined space to the outside Second hierarchical division process
Was immediately Bei,

The download step is characterized in that the obtained layered image data and layered map data are downloaded.

[0034]

[0035]

[0036]

[0037]

[0038]

The above object can also be achieved by the following image processing method. That is, this method includes a step of converting input image data into spatial data relating to predetermined spatial coordinates different from the image spatial coordinates of the image data, and a first layer for hierarchically dividing the spatial data according to the resolution. A dividing step, a step of generating a map showing a correspondence relationship between the image data in the image space coordinates and the spatial data in the predetermined space, and a second hierarchical dividing step of hierarchically dividing the map according to the resolution And a step of storing the obtained layered spatial data and the layered map data in a storage device, and an image generation step of generating an image from the layered spatial data and the layered map data.

According to claim 16 which is a preferred aspect of the present invention, in the second layer dividing step, the resolution in each layer corresponds to the resolution of each layer in the first dividing step. It is set.

According to claim 17 which is a preferable aspect of the present invention, the hierarchy and the resolution are set in ascending order or descending order.

[0042]

According to claim 18 which is a preferred aspect of the present invention, in the second layer division step, a plurality of pieces of the map data are provided in correspondence with blocks having a size set according to the number of layer divisions. Block division step of dividing into blocks.

[0044]

[0045]

According to a nineteenth aspect of the present invention, which is a preferred aspect of the present invention, the method further comprises a step of sequentially reading from the hierarchical space data of a hierarchy having a lower resolution, and a drawing step of drawing a virtual image in the order of reading.

According to a twentieth aspect which is a preferred aspect of the present invention, there is provided a detection step of detecting the viewpoint position of the user,
The drawing step draws a virtual image based on the detected viewpoint position.

In order to further speed up the drawing of the virtual image, the drawing target may be limited. Therefore, according to claim 21 , which is a preferable aspect of the present invention, the reading step reads hierarchical space data of a space close to the detected viewpoint position.

The above object can also be achieved by providing a storage medium for storing a program for realizing the above processing method on a computer.

Further, the above object can be achieved by the following image processing apparatus. That is, this device comprises means for converting the input image data into spatial data relating to predetermined spatial coordinates different from the image spatial coordinates of the image data,
First layer dividing means for hierarchically dividing the spatial data according to resolution, and means for generating a map showing a correspondence relationship between the image data in the image space coordinates and the spatial data in the predetermined space; Second hierarchical division means for hierarchically dividing the map according to the resolution, means for storing the obtained hierarchical spatial data and hierarchical map data in a storage device, the hierarchical spatial data and the hierarchical Image generating means for generating an image from the map data,
It is equipped with.

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. This embodiment is applied to an application that provides a walk-through environment to a user by using real shot image group data picked up at a plurality of viewpoint positions. FIG. 8 shows a schematic diagram of an embodiment of an apparatus for generating and displaying an image at an arbitrary viewpoint position from a group of photographed images taken at a plurality of viewpoint positions.

In the figure, reference numeral 101 denotes an image input device for picking up a photographed image group. The image input device may capture a large number of images by shifting one camera, or may capture a large number of images by setting a plurality of cameras. Further, instead of the image input device 101, a database that stores a large number of images captured in advance may be used. A CPU 102 performs processing according to a program stored in a high speed memory 103 such as a RAM. 1
Reference numeral 03 denotes a storage device, which stores image data of the photographed image group, ray space data generated from multi-viewpoint images, a walk-through application program showing the processing procedure of the CPU, and a part of the work memory. To use as. Reference numeral 104 denotes a viewpoint / line-of-sight detection device for detecting the viewpoint position / line-of-sight direction of the observer. As the viewpoint / gaze detecting device, an input device such as a keyboard or a mouse can be used, or a device with a sensor such as an HMD (head mounted display) can be used.
Reference numeral 105 denotes an image output device for displaying an image generated in accordance with the viewpoint position / line-of-sight direction of the observer. As the image output device, a general two-dimensional display such as CRT or liquid crystal may be used, or a three-dimensional display such as lenticular or HMD may be used. The program is
It may be recorded in a storage medium such as an FD (floppy (registered trademark) disk), a CD-ROM, or a magnetic tape, read from the storage medium reading device 106, and stored in the storage device 103.

200 is a ray space (hereinafter, ray space is referred to as "R
"S" for short) is a data database system, and the host 10 configured by the CPU 102, as will be described later.
00 to function as a file server.

The feature of this embodiment is that, as shown in FIG. 9, when the application program for providing walkthrough requires the ray space data for drawing the virtual space, the ray space data is stored in the database 200.
Instead of directly downloading from the RAM 103 to the RAM 103, the ray space data is hierarchized (in this embodiment, it is divided into four hierarchies for convenience), and the hierarchized ray space data is set to 1
It downloads each layer. The application program synthesizes and restores the downloaded layered ray space data, draws a virtual image based on the restored data, and displays it on the display device 108.

The RAM 103 is provided with a management table for managing the progress of download, the details of which will be described with reference to FIG.

As will be described later, the ray space data (RS data) is composed of a pair of image data and map data (a map showing the correspondence between the xu space and the xy space).
Therefore, it is important how the ray space data composed of image data and map data is hierarchized in the database 200, and how the hierarchized ray space data is restored in the host 100. Is.

FIG. 10 shows the concept of the layering process applied to the embodiment. In FIG. 10, the layering process for the image data of the ray space data is shown for convenience of explanation. First about map data hierarchization
This will be explained with reference to FIGS. 7 to 20. In the example of FIG. 10, as an example, with respect to the original image I _{n of} 2 ⁿ × 2 ⁿ pixels (the image of the lowest hierarchy in the figure), the image of n layers is further added, and the number of pixels of each is Is 2 ^n-1 × 2 ^n-1 _, 2 ^n-2 × 2 ^n-2 … 2 ⁰ ×
2 images _{I n-1, I n-} 2 ~ I 0 is ⁰ is generated. In the example of FIG. 10, the image data of a certain layer i is the layer i + 1.
Given by, for example, an average value of the pixel values in the corresponding 2 × 2 pixel range.

Next, the control procedure for creating the map data showing the correspondence between the photographed multi-view images and the ray space data will be described with reference to the flowchart of FIG.
This control procedure corresponds to the uploading of an image from the image input device 101 to the database 200 in FIG. In addition, the control procedure of FIG.
The process may be performed while controlling the image input device 101, or the image data once read from the image input device 101 may be stored in a memory (not shown) and the map data creation process may be performed on the image data. Good. In any case, the 11th control procedure is suitable for off-line processing (batch processing) by its nature.

In step S 102 of FIG. 11, E images are picked up from a plurality of viewpoint positions by using the image input device 101 and stored in the storage device 103. Step S104
Then, the data for the first one line in each image data stored in the storage device 103 is decomposed into a ray group, and projected into the ray space (x, u) according to the above [Equation 9] or [Equation 10]. This projection will be described with reference to FIG.

In FIG. 12, α is the deviation angle of the optical axis with respect to the Z axis, and the camera is placed at the lens center position Q (x ₀ , z ₀ ) (this is the viewpoint position) to take an image. It shows the situation. In the figure, 301 is the viewpoint position Q.
(x ₀ , z ₀ ), 302 is the imaging plane, 303 is the j-th pixel of an arbitrary line in the imaging plane, α is the angle formed by the camera optical axis from the user viewpoint (that is, the virtual camera viewpoint) with the Z axis, θ is the angle formed by the ray passing through the viewpoint position 301 and the jth pixel 303 with the Z axis, ω is the angle of view of the camera, and 304 is the point where the ray passing through the jth pixel intersects the X axis. . If the number of pixels of each line on the imaging surface is m, the angle data θ indicating the light ray direction can be obtained by solving the following equation.

[Equation 17]

The operation of converting the image data into the RS space in step S104 is performed by using [Equation 17] as the data for the first one line in the data of the E images (that is, the data for E × m pixels). ) To E × m ray directions θ
It is nothing but the calculation of _i (i = 1 to Em). In step S106, E having the ray direction θ _i (i = 1 to Em)
The xm light ray groups are projected in the light ray space according to [Equation 9] and [Equation 10]. The data recorded in the ray space means that in the k-th image (the image captured at the k-th viewpoint position),
When light generated from h-th pixel in the main scanning direction is projected at the position of (x _1, u ₁₎ in the light space, in this position in the light space (x _1, u _1),

The data is expressed by recording the values (k, h) (where, k: image number, h: pixel position in the main scanning direction). Here, the reason for projecting only the data for the first one line of each image data in the ray space is as follows.
As can be seen from [Equation 10], these expressions do not include a term in the height (y) direction of the image. Therefore, the data of the second and subsequent lines are also projected at the same position (in the ray space) as the data of the first line. Therefore, if only the leading line in each image is calculated, it is possible to automatically determine to which position in the ray space the image is projected without calculating other data. By calculating only the first line in this way, it is possible to increase the speed.

Further, in step S108, the projected ray space data is quantized on the x-axis and u-axis so as to maintain the same resolution as the input image. By quantizing in this way, meaningless data generation can be suppressed.

FIG. 13 shows an example of quantized ray space data at the time when the processing up to step 108 is completed. For simplicity of explanation, in this example, the x-axis, u
As a result of quantized axes, there are 11 × 5 elements. And each element corresponds to each ray

[Image number, pixel number] = [k, h] Although the set elements in the actual image space and the set elements in the ray space correspond one-to-one, their values are undetermined due to quantization. There is an element (blank part in FIG. 13). Therefore, in step S108, an interpolation process is performed, that is, a value of an element whose value is undetermined is estimated in addition to the quantization. Here, the nearest neighbor method is adopted as the estimation method, but the estimation method is not limited to this, and any method may be used for estimation. The estimated value is recorded in the corresponding element as a set of [image number, pixel number]. By holding the ray space data in such a format, the arbitrary viewpoint image generation, which will be described later, can be performed by referring to a table (correspondence table), and thus high-speed processing becomes possible.

FIG. 14 shows an example of the correspondence map obtained as a result of performing the processing from step S102 to step S108. The example of FIG. 14 is an interpolation of the example of FIG. 13 using the nearest neighbor method.

In step S110, the correspondence map thus obtained and the original image data (image number k
And a pixel position h and a pixel value P) are stored in a predetermined large capacity memory in the DB system 200 as ray space data (hereinafter abbreviated as “non-hierarchical RS data”).

As described with reference to FIG. 9, in the present embodiment, the ray space data is not directly downloaded between the database 200 and the host 100, but the ray space data once hierarchized. Is downloaded. Moreover, since the ray space data is composed of map data and image data, to make the ray space data hierarchical means to make both the image data and the map data hierarchical. In the present embodiment, the well-known layered encoding is used for the layering of image data, and thus the layering of map data will be mainly described below.

As described above, the map data of this embodiment is generated so as to have a two-dimensional structure as shown in FIG. When the map data is hierarchized, the map data is divided into blocks of 8 × 8 size. In each block, for each 8 × 8 = 64 cells,
Discreteness values from "1" to "4" are assigned, for example, as shown in FIG. The position in the block for each discreteness value is as shown in FIG. That is, in each block

An element having a discreteness value of “1” is assigned only to the upper left element, and consequently, only one element has a discreteness value of “1” in one block. Layered data consisting of elements with discreteness value of "1" is the first layer map DD
_It is called _b (1) (Fig. 17). The first hierarchical map DD _b (1) is obtained by extracting only the element having the discreteness value “1” (that is, the element at the upper left of each block) from the block. Therefore,
The data amount of the first layer map data is 1/64 of the original map data.

It has been compressed to. That is, the first layer map data is 1/8 of the vertical and horizontal of the original map data.
It is the map data thinned out to. The elements having the discreteness value “2” are assigned to every fourth element in the vertical direction, and consequently, only three elements have a discreteness value of “1” in one block. The layered data including the elements of which the discreteness value is “2” for the object b is referred to as second layer map data DD _b (2) (FIG. 18). Second layer map data
DD _b (2) is obtained by extracting only the elements having the discreteness values “1” and “2” from the block.

The elements having the discreteness value “3” are set such that only 12 elements are discretely distributed within one block. Referred to as a hierarchical data consisting of elements of the discrete level value is "3" third hierarchical map data DD _b (3) (Figure 19). An element having a discreteness value of "4" is set so that only 48 elements are discretely distributed within one block. Referred to as a hierarchical data consisting of elements of the discrete level value is "4" fourth hierarchical map DD _b (4) (Figure 20). The distribution (position and number) of the discreteness value is not limited to the above example, and various modifications can be freely made as long as the total number of pixels is the same. Further, the reason why the size of one block is 8 × 8 is that the number of divisions for hierarchization is four, which will be described later.

As will be apparent from the control procedure described later,
Since the light space data of each discrete degree with the reduced data amount as described above is downloaded from the DB 200 to the host 1000, the time required for the download becomes short. Therefore, the application program on the host 1000 side can draw the virtual image faster than when downloading the raw ray space data.

In the drawing of the virtual image in the present embodiment, the above-described hierarchical map data is not used for drawing as it is, but the composite hierarchical map data that has been downloaded and saved until then is drawn. This is done based on the ray space data obtained by merging the downloaded layered map data. That is, the composite layered map data D _b (i) of the number i of layers of a certain object b is the same as the composite layered map data SD _b (i-1) of the number of layers i−1 up to that point.
Based on the hierarchical map data DD _b (i) of d _b (i),

[Equation 18] D _b (i) = SD _b (i-1) + DD _b (i)

Is calculated by The above [Equation 18]
And + represents a logical sum. 21 to 24 respectively show the combined first layer map data SD _b (1), the combined second layer map data SD _b (2), the combined third layer map data SD _b (3),
The synthesized fourth layer map data SD _b (4) is shown.

That is, as shown in FIG. 21, the combined first layer map data SD _b (1) is the first layer map data DD _b (1).
(Fig. 17). Composite 1st layer map data SD
_b (1) is the element of the discreteness value “1” (that is,
Only the upper left element of each block) is extracted. There is only one such element in each block. Therefore, the combined first layer map data SD _b (1) is compressed to 1/64 of the original map data. That is,
The vertical and horizontal directions of the original map data are thinned to 1/8.

The combined second layer map data SD _b (2) is composed of the combined first layer map data SD _b (1) and the second layer map data DD.
_Since it is the sum of _b (2) (Fig. 18), it becomes as shown in Fig. 22. That is, the composite second layer map data SD _b (2) is obtained by extracting only the elements having the discreteness values “1” and “2” from the block. There are only one element with the discreteness value "1" and only three elements with the discreteness value "2" in each block.
Therefore, the number of elements of the combined second layer map data SD _b (2) is 4, and therefore the data amount is compressed to 1/16 of the original map data. That is, the composite second layer map data SD _b (2) is map data in which the vertical and horizontal directions of the original map data are thinned to 1/4.

The combined third layer map data SD _b (3) is the combined second layer map data SD _b (2) and the third layer map data DD.
_Since it is the sum of _b (3) (FIG. 19), it becomes as shown in FIG. The combined third layer map data SD _b (3) is obtained by extracting only the elements having the discreteness values “1”, “2”, and “3” from the block. In one block, there are only one element having the discreteness value “1”, three elements having the discreteness value “2”, and only 12 elements having the discreteness value “3”. Synthetic third
The number of elements of the hierarchical map data SD _b (3) is 16, so the data amount is compressed to 1/4 of the original map data. That is, the composite third layer map data SD _b (3)
Is the map data obtained by thinning out the original map data in both vertical and horizontal directions.

The combined fourth layer map data SD _b (4) is composed of the combined third layer map data SD _b (3) and the fourth layer map data DD.
_Since it is the sum of _b (4) (Fig. 20), it becomes as shown in Fig. 24. The composite fourth layer map data SD _b (4) has discreteness values “1”, “2”, and “3” from the block. Only the elements of "4" are extracted. One block has one element with discreteness value “1” and three elements with discreteness value “2”.
On the other hand, there are only 12 elements having the discreteness value “3” and 48 elements having the discreteness value “4”. The number of elements of the fourth layer data generated by the process of step S314 is 64. Therefore, the combined fourth layer map data SD _b (4) is the original map data.

The correspondence map is hierarchized as described above. Layered image data and map data (hereinafter referred to as “layered RS data”,

These hierarchical data are converted into “hierarchical R
S-space data ". Here, the relationship between the number of divisions p of the hierarchy (4 divisions in the above embodiment) and the block size N (8 × 8 in the above embodiment) will be described.

In this embodiment, it is desirable that the resolution of the highest layer is the same as the resolution of the original ray space data. Therefore, [Equation 19] Is. However, there are cases where the number of layers and the block size cannot be set due to such a relationship.

Particularly, since the layered map data includes the address of the image data before layering, the address that does not exist in the image data of the first layer is included. For example, (5,23) is included in the composite map data of the first layer.
The 23rd column indicates the pixel of the image data before being layered, but when the image data is divided into 4 layers, it should be decompressed to the 2nd column. is there. Here, 2 in the second column is [Equation 20], where [] is a Gaussian symbol. I asked from.

Therefore, in the actual processing, it is necessary to consider the above conversion. Generally, in this conversion, the number of divisions of the layer is p, the layer to be processed is i, and the converted address is
n _i, when the address indicating whether stored in what column of composite map data of the i-th layer and n _j, [Equation 21] It is represented by.

The layered ray space data generated as described above is obtained for multi-viewpoint images obtained for individual objects. That is, if there are L virtual objects (or real objects) in a certain virtual space, L sets of layered RS space data will be obtained. FIG. 25 shows that there are four virtual objects (or real objects) in a certain space, and for each of these objects 1 to 4, E ₁ image, E ₂ images, E ₃ images, respectively. Image, and E ₄ images are obtained, it indicates that four sets of hierarchical RS spatial data are obtained for four objects.

Next, a method of generating an image at a desired viewpoint position using the hierarchical RS space data will be described with reference to the flowchart of FIG. When drawing a virtual image from an arbitrary viewpoint position (including the line-of-sight direction), the virtual image is drawn based on the hierarchical image data and map data. Although it is desirable to perform this processing on a real-time basis, in this embodiment, since the hierarchical RS data is used, at least a virtual image drawing is generated by the RS data of the first hierarchy, which has a low resolution but has a good visibility. Therefore, it is possible to grasp the virtual space in real time.

In the present embodiment, the number of layer divisions is not limited to four, and can be set to an arbitrary number of divisions if the predetermined conditions ([Equation 20], [Equation 21], etc.) are satisfied as described above. In the following description, the number of divisions is i.

FIG. 26 shows the processing procedure of the application program of this embodiment. In this walkthrough application, it is assumed that the object number b is assigned to the object of the virtual space pillar, and the ray space data for each object is stored in the DB 200. In other words, a plurality of objects are placed in the walk-through target space in advance.

In step S200, the position of the viewpoint of the user (for example, the position of the observer / the direction of the line of sight) is detected.
A joystick, a keyboard, a magnetic sensor, a line-of-sight detector, and the like may be used as means for inputting the position / line-of-sight direction of the user, but any device that achieves this purpose may be used. In step S202, an object within the visual field range that should be seen from this viewpoint position is searched. As described above, since the arrangement position of the virtual object (the virtual object formed by the actual image of the physical object) arranged in the space is known, if the viewpoint position of the user can be determined in step S200, it can be seen from that position. The object numbers b of all supposed virtual objects can be specified.

In step S204, the management table is searched to determine whether or not the data of the virtual object searched in step S22 has been downloaded from the DB 200. By this search, step S200
It is possible to know the object number of the virtual object that first appears when the user moves to the viewpoint position detected in.

Management table used in step S204
The structure of the package is shown in FIG. In Fig. 27, the management
Each record in the table consists of four fields.
Has been. Each record consists of object number b
Flag F_bPAnd the layer number i _bHave and. Management table
Is the walkthrough application of this embodiment.
Number of objects defined in the virtual space targeted by the program
Has a number of records equal to. Drawing target flag F_bDIs that
Is a flag indicating that the object b is a drawing target.

The data presence / absence flag F _bPP and the layer number i _b indicate that the layered data of the layer number i _b of the ray space data of the object b exists in the RAM 103 of the host.
The data presence / absence flag F _bPP and the layer number i _b are updated each time the host application program receives the ray space data.

The value of the layer number i _{b has} a value of 0 to 4.
The maximum value of the layer number i _b is 4 because the number of divisions of layering is 4 layers for convenience of explanation of this embodiment. In FIG. 27 showing an example at the time when the download has progressed to some extent, the object having the object number b of 1 is the ray space data having the layer number i _b of “3”, and the object having the object number b of 2 is the layer. Ray space data with the number i _b of “1”, light space data with the layer number i _b of “2” for the object with the object number b of 3, and layer number i for the object with the object number b of 4 _It shows that the ray space data in which _b is "4" is downloaded to the host 1000 side, respectively.

The layer number i _b corresponds to the resolution. In this embodiment, if the ray space data of the layer number i _{b having} any one of the values 1 to 4 exists on the host 1000 side (if downloaded), the data is displayed on the display device 108 of the host 1000 side. Since display data exists for an object with i _b ≠ 0, the value of the corresponding flag F _bD is set to “1”. 27th
In the example of the figure, since the display data of any resolution exists for the objects 1 to 4, the value of the corresponding flag F _bD is “1”, and the object of the object number 5 corresponds. The value of the flag F _bD is “0”.

Therefore, in step S204, the flag F
_An object with _bD = 0 becomes an object that appears "newly".
Therefore, in step S206, such an object b (=
x) is present.

If such an object x exists, step S2
Proceeding to 08, the flag F _xD = 1 is set to indicate that the object x is the drawing target. And step S2
At 10, the DB 200 is requested to download the ray space data of the object x. And step S3
Go to 00.

On the other hand, if it is determined in step S206 that there is no newly appearing object, the process proceeds to step S300.

In step S300, the management table (second
_This is a routine for drawing and displaying ray space data of an object whose F _{bP in} FIG. 7) is “1”.

Thus, if the viewpoint position detected in step S200 has moved from the previous viewpoint position enough to force a change in the space (or object) to be rendered, in step S208, It stores that the object is a drawing target, requests the spatial data of the virtual object from the database 200 in step S210, and in step S300, the spatial data of the virtual object is stored in accordance with the downloaded new spatial data (hierarchical spatial data). A virtual image of the new object at the viewpoint position detected in step S200 is drawn. When a new object is found, the process of drawing a virtual image in step S300 is performed on the existing object by the spatial data with the highest resolution downloaded up to that point, as will be described later. draw.

The details of the drawing routine in step S300 are shown in FIG. 29, the layered data reception routine on the host 1000 side is shown in FIG. 30, and the ray space data transmission routine on the database side is shown in FIG. ing.

When the host 1000 requests the ray space data of a new object, the request command shown in FIG. 28 is sent to the database 200 in step S210. This command is for object b that requires ray space data.
(Object number b) and requested resolution (hierarchical level i _b ) are included. Since the command sent in step S210 requests the layered ray space data of the lowest resolution (layer number 1), i _b = 1. Hierarchical ray space data of higher resolution (hierarchy numbers 2 to 4) is requested in step S416 of FIG. 30 described later.

After issuing the request command to the database 200 in step S210, the host side executes step S210.
The drawing display routine of 300 is executed. On the other hand, the request command issued in step S210 is the database 20.
Will be received by 0.

Details of step S300 are shown in FIG. That is, in step S302 of FIG. 29, the virtual camera in the input viewpoint position and line-of-sight direction is set. In step S304, each pixel on the top line at the image coordinates of the virtual camera is decomposed into a ray group. Next, in step S306, the position of each ray in the ray space to which the ray group obtained in step S304 is projected is obtained using [Equation 9] and [Equation 10]. At this time, quantization is performed in the same manner as the processing in step S108 (FIG. 11). In step S308, the management table (Fig. 27)
By referring to the flags F _bD and F _bP of the above, an object which is a drawing target and whose ray space data has already been downloaded is searched for.

In the example of FIG. 27, the objects 1, 2, 3, 4 and x are objects to be drawn (visible from the viewpoint position), but the ray space data of the object x has not been downloaded yet. There is. Therefore, in steps S310 and S312, the object (object 1, 2, 3, 4) whose flag F _bP is “1” is drawn. In step S314, a virtual image of the object is displayed.

The download of the ray space data of the object x will be described below. The database 200 is
In step S501 (FIG. 31), the host awaits a ray space data request for the object b from the host at the hierarchical level i _b . Therefore, the data request for the new object x is accepted by the database 200 in step S501 (FIG. 31). In step S502, it is analyzed that the data request is hierarchical data for the hierarchical level i _b (= i _x ) of the object b (= x). In step S504, it is determined whether the data request is for the first hierarchical level. This determination is because all blocks of the ray space data of the object b must be decomposed into 8 × 8 once to obtain the hierarchical data. The disassembling process is step S
It is performed from 506 to step S510. Step S51
At 2, the hierarchical level i _b is saved in the work register d _b .

In step S514, map elements having the discrete degree d _b are collected. In the example of FIG. 27, for the object x, d _b
Since = 1, DD _b (1) in FIG. 17 is obtained. Step S
At 516, the elements of the image data having the discrete degree d _b (see FIG. 10) are collected. In step S517, the layered map data and the image data having the discrete degree d _b are sent to the host 1000 side as layered ray space data. That is, the layered ray space data is downloaded toward the host 1000.

In step S518, the counter d _b is incremented by 1. In steps S520 and S522,
It is checked whether or not the ray space data up to the fourth hierarchical level has been downloaded, and if completed, the work data saved in step S510 is released from the work memory.

In step S402 (FIG. 30), the host 1000 receives the hierarchical data (object b, hierarchical degree i _b ) sent from the database 200 in step S517.
In step S404, it is confirmed that the layered data is for the object b and the layer degree i _b ,
In 06, the data presence flag F _bP is set to “1” in order to store that the layered data for the object b exists for at least the first layer degree.

In step S408, step S402
I received at_bHierarchical data has been accumulated so far
Number of layers i_b-1 Composite hierarchical map data SD_b(i-1)When,
Synthesis is performed according to [Equation 18]. In addition, the first layered data
When received, the data becomes the composite first layered data.
(See FIG. 21). In step S410, the composite i-th
Hierarchical data will be overwritten on the i-1st hierarchical data up to that point.
Will be killed. In step S412, the hierarchy level is increased by 1,
It is confirmed in step S414 that the hierarchical level does not exceed 4.
Then, in step S416, the ray space data of the next hierarchical level is obtained.
Request data.

Thus, the first-layer synthetic space data of the object x is formed on the memory 103. This state is a flag
Since it is reflected in F _bP = 1, in step S312,
The virtual image of the object x is drawn and displayed based on the combined first hierarchical space data SD ₁ (1).

In step S416, the database 200
The request for downloading the spatial data of the second layer of the object b issued to the user is accepted in step S501 of FIG. 31, and is received in step S501 → step S502 → step S504 → step S514 → ... step S517.
Then, the second layered ray space data (FIG. 18) is sent, and the second layered ray space data is obtained in step S42.
At 0, it is received by the host 1000. The received second layered ray space data is formed as combined second layered ray space data in steps S408 and S410, and the combined data is rendered in step S310.

The above-mentioned download is performed in step S414.
Thus, it is continued until the fourth hierarchical level is reached. By this iterative process, virtual images of respective resolutions are displayed in order from low resolution (low hierarchy) to high resolution, that is, progressively displayed.

At the time of this progressive download, since the ray space data divided into layers is sent to the host, there is an effect that the time required for the download is shortened. Therefore, the drawing display is made efficient.
In particular, since the data with high resolution is downloaded from the first layer with the smallest data amount (low resolution) for the first time, in other words, the resolution of the virtual image of the displayed object gradually increases. Since it is controlled, especially in a walk-through application, a low-resolution image is displayed at an early stage, so that it can be determined early whether or not the object in the image is the target object.

In the above embodiment, the low resolution ray space data sent from the database is discarded (overwritten in step S410) when the ray space data having a higher resolution than that is sent. , Large capacity memory is unnecessary.

Incidentally, when the vertical parallax is taken into consideration when projecting in the light space, the above-mentioned applicant of the present invention, Japanese Unexamined Patent Publication No.
The method of -97642 is applied.

The viewpoint position / line-of-sight direction detecting device 104
May be anything that can detect the viewpoint position and the line-of-sight direction. Further, the image output device 105 uses a stereoscopic display unit capable of binocular stereoscopic vision such as a lenticular system or a glasses system, and an image corresponding to each of the left and right eye positions of the observer at the time of capturing a multi-viewpoint image. Is generated, it becomes a binocular stereoscopic display device capable of coping with the viewpoint movement of the observer.

<First Modification> In the above embodiment, the virtual image of the object in the entire virtual space is progressively generated and drawn. For example, in a system that provides a walkthrough experience, a virtual space is divided into a plurality of sub virtual spaces, and ray space data is generated for each sub virtual space. Alternatively, ray space data is generated for each object in the ray space. In other words, the virtual space is divided into predetermined units and converted into light space data.

The image processing apparatus of the modified example progressively displays the hierarchical RS data in the sub virtual space close to the user's viewpoint position. Such drawing is also possible in the second embodiment in the sub virtual space (or for each object).
This is because the ray space data for each of them is hierarchized and stored in the storage device.

Specifically, in this modified example, the user's viewpoint position is grasped, and the light space data of the object or the sub virtual space close to this viewpoint position is read from the storage device.

Therefore, first, the layered RS data of the object (or the sub virtual space) is drawn and generated in order from the object (or the sub virtual space) close to the user's viewpoint position. This drawing generation, for all objects (or sub-virtual space)
First, the drawing is performed using the lowest-resolution layered RS data, and then the same is used for all objects (or the sub-virtual space) using the coarser-resolution layered RS data.

When the user moves, drawing is performed in order from the hierarchical RS data of the object (or the sub virtual space) in the moving direction.

<Second Modification> The following modification is further proposed. That is, in the virtual space (mixed reality space), the virtual object (or real object) specified by the user should be the object (or sub-virtual space) that the user particularly needs to grasp. Therefore, when the user grabs, moves, or rotates a virtual object or a real object, the progressive drawing is performed for drawing a virtual image of the virtual object (real object).

<Third Modification> Further, in the above embodiment,
Both the image data and the map data in the ray space were hierarchized. However, the amount of map data is much smaller than the amount of image data. Therefore, the reduction width of the download time obtained by layering the map data is smaller than that of the image data. Therefore, even if the layering of the map data is omitted, the degree of increase in the download time may not be a problem as compared with the layering of the map data. Therefore, as a modification of the above embodiment, it is proposed to omit the layering of map data.
If the layering of the map data is omitted, there is no need to perform the restoration process on the host side, and the side effect of shortening the time until the drawing starts is obtained.

<Fourth Modification> Furthermore, in the above embodiment,
The layering of ray space data was performed on the database side, and the layered data was downloaded to the host side. On the host side, the layered data is synthesized each time it is downloaded, and the synthesized layered data at the resolution at the time of synthesis is displayed. That is, in the embodiment shown in FIG. 9, the work of converting the ray space data into the layered data is performed at the time of the request from the host side, but this conversion work is performed in advance by batch processing. Good. Since the hierarchical processing is not affected by the viewpoint position of the user, it can be performed in advance. Performing the layering process in advance requires an area for storing layered data on the database 200, but since the layering need not be performed at the time of downloading as in the above-described embodiment, the download speed is increased. It

<Fifth Modification> The fifth modification is a modification of the fourth modification. That is, in the fifth modified example, the composite layered data is processed by the database 200 by batch processing.
It is something to be prepared by the side. Since the combined hierarchical data has a larger amount of data than the hierarchical data, the time required for the download is longer than the download of the hierarchical data, but the advantage is that the processing for combining is unnecessary on the host 1000 side. to be born.

<Sixth Modification> In the above embodiment, the ray space database 200 is connected to the host through the communication line, but the present invention is also applicable to a parallel bus connection database.

<Seventh Modification> In the above embodiment, the ray space data once downloaded to the fourth hierarchical level from the database is stored in the memory 103, but it is out of the field of view due to the movement of the viewpoint position. Spatial data for such objects is unnecessary, as seconds are not
Therefore, if it is out of the field of view, the ray space data of the object is erased from the RAM 103, and at the same time, the flag F _bP is _set to “0”.

Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of one device. Further, it goes without saying that the present invention is also applied to the case of being implemented by supplying a program to a system or an apparatus. In this case, the storage medium storing the program according to the present invention constitutes the present invention. Then, by reading the program from the storage medium to the system or device, the system or device operates in a predetermined method.

[0140]

As described in detail above, according to the present invention,
When generating a virtual image from spatial data generated based on a real image, a layered virtual image is generated and drawn especially at a low resolution, so that the user can quickly create a virtual space. The structure of can be understood.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a principle of generating ray space data.

FIG. 2 is a diagram illustrating data in a real space.

FIG. 3 is a diagram when the space of FIG. 2 is represented by ray space data.

FIG. 4 is a diagram illustrating a principle of generating real space data when there are a plurality of cameras.

FIG. 5 is a diagram for explaining the principle of generating ray space data when there are a plurality of cameras.

[FIG. 6] From ray space data when there are multiple cameras,
The figure explaining the principle which produces | generates ray space data (x + Zu = X) in arbitrary viewpoint positions.

FIG. 7 is a diagram illustrating the principle of reconstructing the real space from the arbitrary viewpoint of FIG.

FIG. 8 is a block diagram showing a configuration of an image processing apparatus according to an embodiment.

9 is a block diagram showing the flow of data in the embodiment of FIG.

FIG. 10 is a diagram illustrating a principle of layering image data, which is adopted in the image processing apparatus according to the embodiment.

FIG. 11 is a flowchart illustrating a process of converting image data acquired from multiple viewpoints into light space data and accumulating the light space data in the embodiment.

FIG. 12 is a diagram illustrating a process of decomposing an input image into a ray group.

FIG. 13 is a diagram illustrating an example of quantized ray space data at the time when the ray space interpolation processing is completed.

FIG. 14 is a diagram showing a result of interpolating the ray space data of FIG.

FIG. 15 is a diagram illustrating how a map is divided into 8 × 8 blocks.

FIG. 16 is a diagram illustrating the distribution of discreteness values in one block.

FIG. 17 is a diagram showing the structure of first layer map data.

FIG. 18 is a diagram showing the configuration of second layer map data.

FIG. 19 is a diagram showing the structure of third layer map data.

FIG. 20 is a diagram showing the structure of fourth layer map data.

FIG. 21 is a view showing the structure of combined first layer map data.

FIG. 22 is a diagram showing a structure of combined second layer map data.

FIG. 23 is a diagram showing a structure of combined third layer map data.

FIG. 24 is a diagram showing the structure of combined fourth layer map data.

FIG. 25 shows that a virtual space is divided into a plurality of sub spaces,
Alternatively, a diagram illustrating that a plurality of objects are included.

FIG. 26 is a flow chart showing a control procedure of an application program for drawing and displaying a virtual space from an arbitrary viewpoint position such as walkthrough.

FIG. 27 is a diagram showing the structure of a download management table.

FIG. 28 is a diagram showing a configuration of a command requesting a download and a command responding to the download.

FIG. 29 is a flowchart illustrating the details of the drawing display routine in step S300.

FIG. 30 is a flowchart illustrating a reception routine of ray space data on the host side.

FIG. 31 is a flowchart showing details of a download request command processing routine on the database 200 side.

Front page continuation (72) Inventor Daisuke Kotake 6-145, Hanasaki-cho, Nishi-ku, Yokohama Hanasaki Building, Yokohama, M Systems Co., Ltd. (56) Reference JP-A-10-111951 (JP, A) Kaihei 7-21350 (JP, A) JP 10-97642 (JP, A) Akio Ishikawa 3 others, "Interactive operation of 3D live-action space based on ray description", IEICE Transactions , Japan Society of Electronic and Information Communication, May 25, 1998, Vol. J81-D-II, No. 5, p. 850-860 Ken Naemura, 3 persons, "Ray space projection method for efficient description of 3D real space by ray information", Technical Report of The Institute of Television Engineers of Japan, Television Society of Japan, February 1, 1996. , Volume 20, Issue 4, p. 49-56 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 17/40 G06T 1/00 H04N 1/387 G01B 11/00 CSDB (Japan Patent Office)

Claims (23)

(57) [Claims] 1. An image storage method for storing spatial data composed of a predetermined space different from this image space from image data of a real shot image, wherein real image data in the spatial data is hierarchized according to resolution. A first hierarchical division step of dividing, and a second hierarchical division step of hierarchically dividing the map showing the correspondence relationship between the actual image space and the predetermined space according to the resolution, in each hierarchy Second hierarchical division step in which the resolution of each of the layers is set to correspond to the resolution of each layer in the first division step, and the step of storing the obtained hierarchical image data and hierarchical map data in a storage device. An image storage method comprising: 2. The image storage method according to claim 1, wherein the spatial data is ray space data. 3. The image storage method according to claim 1, wherein the hierarchy and the resolution are set in ascending order or descending order. 4. The image storage method according to claim 1, wherein in the layer dividing step, data thinning processing is performed. 5. The second layer division step includes a block division step of dividing the map data into a plurality of blocks corresponding to blocks having a size set according to the number of layer divisions. The image storage method according to any one of claims 1 to 4. 6. In the second layer division step, when the number of layer divisions is p, the layer to be processed is i, and the storage address of the map data of the i-th layer is n _j , the converted address n _I is obtained. , [Equation 1] 5. The image storage method according to claim 1, wherein the image storage method is used. 7. In the block dividing step, when the number of layers is p, the block size is N × N, and The image storage method according to claim 5, wherein the image storage method is set to. 8. An image drawing method for drawing the layered spatial data accumulated by the method according to claim 1 in a virtual space, the layering of a layer having a lower resolution. An image drawing method comprising: a step of sequentially reading from spatial data; and a drawing step of drawing a virtual image in the order of reading. 9. The image drawing method according to claim 8, further comprising a detection step of detecting a viewpoint position of the user, wherein the drawing step draws a virtual image based on the detected viewpoint position. . 10. The image drawing method according to claim 9, wherein the reading step reads hierarchical space data of a space close to the detected viewpoint position. 11. The method according to any one of claims 1 to 10.
A storage medium for storing a program for realizing the method according to the item on a computer. 12. A computer equipped with the storage medium according to claim 11. 13. An image storage device for storing, from image data of a real shot image, spatial data configured for a predetermined space different from this image space, wherein real image data in the spatial data is hierarchized according to resolution. First hierarchical dividing means for dividing, and second hierarchical dividing means for dividing the map showing the correspondence relationship between the actual image space and the predetermined space into hierarchies according to the resolution, in each hierarchy. A second layer division means whose resolution is set to correspond to the resolution of each layer in the first division means, and a storage means for storing the obtained layered image data and layered map data. An image storage device comprising. 14. This image is obtained from image data of a photographed image.
Spatial data constructed for a given space different from the space
In the method of downloading an image, the real image data in the spatial data is layered according to the resolution.
A first hierarchical dividing step of dividing into a plurality of layers, the obtained hierarchical image data, the real image space, and the predetermined
Map showing the correspondence with the space of
Comprising a sul downloading step, the second hierarchy division step of dividing hierarchized according to the map resolution, the download process, which downloads the resulting hierarchized image data of layered map data how to download images you wherein a is. 15. A step of converting input image data into spatial data relating to predetermined spatial coordinates different from image space coordinates of the image data, and a first hierarchical division for hierarchically dividing the spatial data according to a resolution. A step of generating a map showing a correspondence relationship between the image data in the image space coordinates and the spatial data in the predetermined space; and a second hierarchical division step of hierarchically dividing the map according to the resolution. And a step of storing the obtained layered spatial data and the layered map data in a storage device, and an image generation step of generating an image from the layered spatial data and the layered map data. Characterized image processing method. 16. The second hierarchy division process, according to claim 15, characterized in that it is set to correspond to the resolution of each hierarchy resolution in the hierarchy of each of the first division step Image processing method. 17. The method of claim 15 through claim 16 hierarchy and resolution, characterized in that it is set in ascending or descending order
The image processing method according to any one of 1. 18. The second layer division step includes a block division step of dividing the map data into a plurality of blocks corresponding to blocks having a size set according to the number of layer divisions. Claims 15 to 17
The image processing method according to any one of 1. 19. The image processing method according to claim 15 , further comprising a step of sequentially reading from the hierarchical space data of a hierarchy with a lower resolution, and a drawing step of drawing a virtual image in the order of reading. 20. The image processing method according to claim 19 , further comprising a detection step of detecting a viewpoint position of the user, wherein the drawing step draws a virtual image based on the detected viewpoint position. . 21. The image processing method according to claim 20 , wherein the reading step reads hierarchical space data of a space close to the detected viewpoint position. 22. A storage medium storing a program for realizing the method described on the computer in any one of claims 15 to 21. 23. A means for converting the input image data into spatial data relating to predetermined spatial coordinates different from the image spatial coordinates of the image data, and a first hierarchical division for hierarchically dividing the spatial data according to the resolution. Means, means for generating a map representing a correspondence relationship between image data in the image space coordinates and spatial data in the predetermined space, and second hierarchy dividing means for hierarchically dividing the map according to resolution. A storage unit for storing the obtained layered spatial data and the layered map data in a storage device; and an image generation unit for generating an image from the layered spatial data and the layered map data. A characteristic image processing device.