JPH10111951A - Method and device for image processing and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce work memory capacity without lowering a process speed by extracting pixels by tracing a pointer from a compressed multi-viewpoint image to a reference image according to a straight line in a light beam space derived from a viewpoint position, a sight direction, and a parameter and a light beam space correspondence table. SOLUTION: Multi-viewpoint images are picked up from viewpoint positions by using a multi-viewpoint image input device 101 to generate the light beam space correspondence table wherein the multi-viewpoint images are made to correspond to straight lines in the light beam space. Then some of the multi- viewpoint images are used as reference images, and the remaining images are represented with pointers indicating pixels in the reference images, compressed, and stored in a storage device 103. Then a viewpoint position and sight detecting device 104 are used to detect the viewpoint position and sight direction of an observer and the pointers are traced by using the compressed light-beam space data stored in the storage device 103 to read in corresponding pixel values from the reference images, so that an image is generated and displayed on an image output device 105.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method and apparatus for generating and displaying an image at an arbitrary viewpoint position in real time in accordance with the movement of an operator by pointer reference processing of compressed image data. And storage media.

[0002]

2. Description of the Related Art Conventionally, when an image at an arbitrary viewpoint position is generated and displayed using a group of real images captured at a plurality of viewpoint positions, data of the group of real images is read into a memory. Based on the concept of a ray space, where each image data is considered to be a set of rays that fly in a three-dimensional space, a method has been proposed in which the luminance value of each pixel is calculated to generate an image at an arbitrary viewpoint position. I have.

Here, the concept of the light beam space will be described. Light rays are emitted in the three-dimensional space by light sources and reflected light of an object. A ray crossing one point in the three-dimensional space is uniquely determined by five variables representing its position (x, y, z) and direction (θ, φ). If a function representing the light intensity of this light ray is defined as φ, the ray group data in the three-dimensional space is represented as φ (x, y, z, θ, φ). Further, considering the time change of the ray group data, φ (x,
y, z, θ, φ; t), and the ray group in the three-dimensional space is described as a six-dimensional space. This space is called a ray space.

Here, attention is paid to a group of rays that pass through a plane of Z = z at t = 0. This surface is called a reference surface.
Considering a horizontal plane (XZ plane) perpendicular to the Y axis and not considering the vertical parallax (y = 0, φ = 0), the real space is as shown in FIG. 3 (a). A group of rays emitted from the reference plane is described as φ (x, θ) by two variables of a position x and an angle θ 2. Therefore, a ray group passing through a certain point (X, Z) in the real space has a relation of X = x + Z tanθ (1). Here, if a variable u = tanθ is defined, equation (1) becomes X = x + uZ (2). Therefore, in the light ray space, one light ray in the real space is mapped as one point, and light intensity, that is, color information is recorded there. From equation (2), it can be seen that a ray group passing through a certain point in the real space is mapped to a straight line in the xu space. FIG. 3B shows a state in which the light beam observed at the (X, Z) viewpoint position in the real space is mapped to the xu space. Note that the xu space forms a subspace of the five-dimensional ray space described above.

In order to accurately reconstruct an image at an arbitrary viewpoint position from this ray space, the dimension in the y-axis direction, that is, the vertical direction, is originally required. But in that case,
The ray space data must form at least a four-dimensional space of xy-θ-φ, which has a very large data amount. So far, we have considered only the xu space, which is a subspace of the ray space. Furthermore, giving color information to the entire coordinates of the ray space is considered to be very redundant. Because, even if only xu space is considered, pixel information in the y-axis direction is needed to reconstruct an image, so the ray space becomes three-dimensional, and the light intensity of each ray must be recorded there. Because it does not become. Therefore, there has been a method of performing a ray space operation on all pixels of an image to be reconstructed, and obtaining a luminance value from a multi-viewpoint image (an image captured from a plurality of different viewpoint positions) read into a memory. In addition, the ray space calculation is to reconstruct an image at an arbitrary viewpoint position based on a multi-view image,
This is a calculation performed based on equation (2) in xu space.

[0006]

However, in the above conventional example, since the xu space considers only the parallax in the x-axis direction (lateral direction), the same light beam is applied to all scan lines in the y-axis direction. Spatial operations had to be repeated. Further, in order to generate and display an image at an arbitrary viewpoint position in real time according to the movement of the operator, a high-speed ray space operation must be performed. In order to perform the calculation, it is necessary to perform an operation of randomly accessing the multi-view image and reading the pixel data. That is, high-speed random access to a multi-view image is required. Therefore, in the above example, a technique of reading the xu space and the multi-viewpoint image into the memory before the calculation is adopted.

As described above, hitherto, when generating and displaying an image at an arbitrary viewpoint position, it is necessary to repeat the same ray space operation many times, and further, it is necessary to use a very large work memory capacity. was there. The large number of calculations for obtaining the pixel data causes a loss of real-time motion. In addition, if the ray space data describing an object has a huge data capacity and all of it must be read into memory, the number of objects that can be represented in a three-dimensional virtual environment using the ray space is limited. It will be what was done. In order to display images in real time, it is necessary to avoid repetition of calculations, and to occupy a lot of objects described by ray space data in a three-dimensional virtual environment, the work occupied by ray space data Memory capacity must be as small as possible.

The present invention generates an image at an arbitrary viewpoint position.
It is an object of the present invention to provide an image processing method and apparatus and a storage medium that reduce the work memory capacity to be used without lowering the processing speed for display.

[0009]

In order to achieve the above object, in an image processing method according to the present invention, a multi-viewpoint image inputting step of inputting images picked up at a plurality of viewpoint positions; A correspondence table creating step of creating a ray space correspondence table in which a ray space and the coordinates of the multi-viewpoint image are associated with each other based on the viewpoint image, and at least one of the input multi-viewpoint images is referred to as a reference image. The remaining images are described by pointers pointing to the pixels of the reference image, so that a multi-view image compression step of compressing the multi-view image and light space data are created based on the compressed multi-view image. A ray space creating step, a viewpoint / viewing line detecting step of detecting a viewpoint position and a line-of-sight direction of the observer, and the detected viewpoint position and line-of-sight direction of the observer and parameters for image output. Extracting a pixel by tracing a pointer from the compressed multi-viewpoint image to a reference image according to a straight line deriving step of deriving a straight line on a ray space, and the derived straight line and the created ray space correspondence table; And an image output step of outputting the generated image.

Preferably, in the correspondence table creation step, a correspondence table is created in which the xu space, which is a subspace of the ray space, and the x coordinate of the multi-viewpoint image are associated.

Preferably, the reference images are selected at equal intervals from an image sequence of the input multi-view image.

Preferably, in the multi-view image compression step, the multi-view image is described by describing the remaining images with pointers indicating pixels of two reference images before and after the reference image located at a position sandwiching the image. Compress.

Preferably, in the multi-view image compression step, the input multi-view image sequence is rearranged prior to selection of a reference image.

Preferably, in the image generating step, an image number and a pixel position are detected from the ray space correspondence table according to the derived straight line, and it is determined whether or not the image number is a reference image. Is determined, a reconstructed image is generated based on the luminance value of the image, and if it is determined that the image is not the reference image, the luminance of the image obtained based on the ray space correspondence table is determined. A reconstructed image is generated based on the value.

Further, in the image processing apparatus of the present invention,
A multi-viewpoint image input means for inputting images captured at a plurality of viewpoint positions, and a ray space correspondence table which associates a ray space with coordinates of the multi-viewpoint image based on the input multi-viewpoint image. A correspondence table creating unit, and a multi-viewpoint image compression unit that compresses the multi-viewpoint image by describing at least one of the input multi-viewpoint images as a reference image and describing the remaining images with pointers indicating pixels of the reference image. Viewpoint image compression means, ray space creation means for creating ray space data based on the compressed multi-viewpoint image, viewpoint / gaze detection means for detecting the viewpoint position and gaze direction of the observer, Straight line deriving means for deriving a straight line on a ray space from the viewpoint position and line of sight of the observer and parameters for image output, and a pair of the derived straight line and the created ray space Accordance with the table, and an image generating means for generating an image by extracting pixels by tracing the pointer from the compressed multi-view image to the reference image,
Image output means for outputting the generated image.

[0016] Preferably, the correspondence table creating means includes:
A correspondence table is created by associating the xu space, which is a subspace of the ray space, with the x coordinate of the multi-viewpoint image.

Preferably, the reference images are selected at equal intervals from an image sequence of the input multi-viewpoint images.

[0018] Preferably, the multi-viewpoint image compressing means is configured to store the remaining image in two positions before and after the image.
The multi-viewpoint image is compressed by describing it with a pointer that points to a pixel of the reference image.

[0019] Preferably, the multi-view image compressing means rearranges the input multi-view image sequence before selecting a reference image.

Preferably, the image generating means detects an image number and a pixel position from the ray space correspondence table according to the derived straight line, determines whether the image number is a reference image, and Is determined, a reconstructed image is generated based on the luminance value of the image, and if it is determined that the image is not the reference image, the luminance of the image obtained based on the ray space correspondence table is determined. A reconstructed image is generated based on the value.

Further, in the storage medium of the present invention, a multi-viewpoint image input step of inputting images captured at a plurality of viewpoint positions, based on the input multi-viewpoint image, a light beam space and the multi-viewpoint image A correspondence table creating step of creating a ray space correspondence table that associates the coordinates, and at least one of the input multi-viewpoint images as a reference image,
The remaining images are described by pointers that point to the pixels of the reference image, and based on the multi-view image compression step of compressing the multi-view image and the compressed multi-view image,
A ray space creation step of creating ray space data, a viewpoint / sight line detection step of detecting the observer's viewpoint position and gaze direction, and the detected observer's viewpoint position and gaze direction and parameters for image output. Extracting a pixel by tracing a pointer from the compressed multi-viewpoint image to a reference image according to a straight line deriving step of deriving a straight line on a ray space, and the derived straight line and the created ray space correspondence table; A computer program for realizing an image processing method, comprising: an image generating step of generating an image by performing the image processing, and an image output step of outputting the generated image. is there.

[0022]

BEST MODE FOR CARRYING OUT THE INVENTION

(First Embodiment) FIG. 1 shows an example in which an image at an arbitrary viewpoint position is generated from a group of photographed images taken at a plurality of viewpoint positions.
1 shows a schematic diagram of an embodiment of a display device. In the figure, reference numeral 101 denotes a multi-viewpoint image input device for capturing the real image group. The multi-viewpoint image input device may capture a large number of images by shifting one camera, or may capture a large number of images by setting a plurality of cameras. Alternatively, image data held in the multi-view image database may be read as input data. Reference numeral 102 denotes a CPU that performs processing according to the processing procedure stored in the storage device 103. Reference numeral 103 denotes a storage device, which stores image data captured from multiple viewpoints, a compressed light space generated from the multiple viewpoint images, and a program indicating a processing procedure of the CPU. Reference numeral 104 denotes a viewpoint / sight line detection device for detecting the viewpoint position / sight line direction of the observer. As the viewpoint / viewpoint detection device, an input device of a general-purpose computer such as a keyboard or a mouse may be used, or a wearable input device with a space sensor may be used. Reference numeral 105 denotes an image output device for displaying an image generated according to the observer's viewpoint position and line-of-sight direction. An image output device can be a general two-dimensional display such as a CRT or liquid crystal, or it can be a lenticular or HM
A three-dimensional display such as D may be used. The program may be recorded on a storage medium such as an FD (floppy disk), CD-ROM, or magnetic tape, read from the storage medium reading device 106, and stored in the storage device 103. Reference numeral 107 denotes a bus for connecting each unit.

FIG. 2 is a flowchart showing a processing method for creating a ray space by capturing a multi-viewpoint image using the apparatus shown in FIG. 1, and a processing method for generating and displaying an image at an arbitrary viewpoint position from the ray space. Shown in The CPU 102 executes processing according to a program stored in the storage device 103. Note that the frame A in FIG. 2 is processed off-line in a portion from capturing a multi-view image to creating a ray space. Further, a frame B in FIG. 2 is processed online, from generation to display of an image at an arbitrary viewpoint position from the ray space.

First, a process of capturing a multi-viewpoint image and creating a ray space will be described. In step S01, the multi-view image input device 101 is used to capture multi-view images from a plurality of viewpoint positions. In step S02, the multi-viewpoint image obtained in step S01 is made to correspond to a straight line in the ray space, and a ray space correspondence table is created. In step S03, several of the multi-viewpoint images are used as reference images, and the remaining images are represented by pointers that point to pixels of the reference image.
Compress multi-viewpoint images. In step S04, the storage device 103 stores the ray space data created in steps S02 and S03, that is, the ray space correspondence table and the compressed multi-viewpoint image. Steps S01 to S04 are the process of creating the ray space.
Done offline.

Next, a description will be given of a process of generating and displaying an image at an arbitrary viewpoint position from the ray space as the observer moves the viewpoint. In step S05, the viewpoint position and the line of sight of the observer are detected using the viewpoint and line of sight detection device 104. In step S06, an image to be observed at the viewpoint position and line-of-sight direction detected in step S05 is stored in the storage device 10.
Using the compressed ray space data stored in 3, the corresponding pixel value is read from the reference image by following the pointer to generate an image. In step S07, the image generated in step S06 is displayed on the image output device 105.

The above is the overall processing flow of the first embodiment. Next, detailed processing for each part will be described.

First, the process of creating the ray space correspondence table in step S02 will be described with reference to the flowchart of FIG. In step S201, each pixel value in each image is converted into a three-dimensional space in accordance with the imaging environment in step S01, that is, camera parameters such as a camera position (viewpoint position) / posture (viewing direction) and a camera angle of view / focal length. Is converted to ray group data. In step S202,
The ray group data converted in step S201 is projected onto an xu space, which is a subspace of the ray space, based on equation (2).
At this time, if the parallax in the vertical direction is not considered, the position of the ray x
And the direction u (tan θ) are the same, they are projected on the same point. However, here, since the camera positions in the imaging in step S201 are discrete, the light beam space may not be partially projected. In step S203, interpolation synthesis is performed for all portions not projected in step S202. Thereby, the ray group data can be projected on all parts of the ray space. In step S204, according to the result of step S203, a ray space correspondence table that records data of the light projected on the xu space, that is, the image number of the pixel and the x coordinate is created. The ray space correspondence table created in this way is stored in the storage device 103. FIG. 5 shows the relationship between the ray space correspondence table and the multi-viewpoint image at this time. In addition, as the interpolation synthesis method of the ray space data in step S203, the nearest neighbor (Nearest Neighbor) that uses the value of the nearest ray space data as the estimated value is used.
Neighbor) method, a linear interpolation method of linearly interpolating based on a distance from neighboring data, and the like can be applied.

Next, the concept of the multi-viewpoint image compression processing in step S03 will be described with reference to FIG. The multi-view image can be considered as an image when the target object is viewed from various positions and angles. Therefore, when these are considered as a series of image sequences in which the positions of the viewpoints are sequentially changed,
It can be seen that there is a strong correlation between the images. Therefore, several images in the image sequence of the multi-viewpoint image are regarded as representative images and used as reference images, and only the reference images are held as image data. The remaining images have data in such a manner as to refer to the reference image, and do not hold large-capacity image data itself. At this time, the reference format of the reference image is a representation method using pointers indicating two reference images located at positions sandwiching the remaining image groups. Here, each image was captured by uniformly rotating the target object on the turntable. Therefore, the multi-view image can be regarded as an image sequence in which the viewpoint is moved by uniform rotation and the entire circumference of the object is imaged. Then, reference images were selected at regular intervals from this uniform image sequence.

Referring to the flowchart of FIG. 7, a specific algorithm of the multi-viewpoint image compression processing in step S03 will be described. In step S301, m images are obtained from N multi-viewpoint images.
The reference images are selected at equal intervals. Step S
In step 302, the first image that is not the reference image is extracted, and its image number is set to n = 1. Next, the process proceeds to step S303, and the two preceding and succeeding reference images at the position sandwiching the image n are set as target images to be referenced from the image n by a pointer. Note that among the two reference images selected here, let L be the one where the viewpoint position is located on the left side with respect to the line of sight of the image n, and R be the one located on the right side. Proceeding to step S304, all pixels (x _n ,
y _n ) is represented by pointers to reference images L and R.
The process proceeds to step S305, and if n = Nm, the process ends. Otherwise, n is incremented in step S306, and the process from step S303 is repeated. Thus,
The processing is performed for all images that are not reference images.

FIG. 8 is a diagram showing the principle of pointer expression in step S304. First, attention is focused on a certain image A in the multi-viewpoint image. It is considered that there is a strong correlation between the image A and the reference images L and R. Therefore, for each pixel of the image A, a pixel having a similar color is searched for in either one of the reference images L and R. Pointer data is obtained by looking up a pointer table created in advance with respect to the movement amount of the coordinate value at that time, and that value is used as the pixel value of the image A. If the pointer data created in advance is described in one byte, the value of one pixel can be one byte data. Conventionally, image data has an RGB plane and an α plane for separating an object and a background,
It had a data amount of 4 bytes per pixel. If this can be described in 1 byte, the data amount can be reduced to 1/4.

FIG. 9 is a flowchart of the pointer representation process in step S304. Next, the flow of this processing will be described with reference to FIG.

First, in step S401, a pixel (x
_n , y _n ) = (0, 0). Proceeding to step S402, the pixel (x
_n, if y _n) is a background pixel, the value of the pixel is set to 0 at step S413, proceeds to step S414. If a background pixel proceeds to step S403, and the variable w = 1 for the threshold variable, set the threshold TH = w · TH _0. In step S404, the variables indicating the amount of pixel movement are set to i = 0, j
Set to = 0. In step S405, it is determined whether (i, j) does not exceed the maximum range (p, θ), and if not, the process proceeds to step S406, and if not, the process proceeds to step S410.
Move on to The maximum range of (i, j) is p = [− 17, 1
7], θ = [-1, 1]. In step S406, the pixel (x _n, y _n) and the image L of the pixel _{(x n + i, y n} + j) ( or the image R pixel _{(x n + i, y n} + j)) of the color difference between Calculate and determine if the color difference is below the threshold. If the color difference is equal to or smaller than the threshold, the process proceeds to step S407. If the color difference is equal to or larger than the threshold, i and j are sequentially changed in step S408.
It returns to step S405. The color difference is calculated using the L ^* a ^* b ^* color system, but the values of L ^* , a ^* , and b ^* for each pixel of each image are calculated first in advance, and the same value of the same reference image is used. Duplicate calculation when referring to a pixel is not performed. In step S407, "L, (i,
j) "(or" R, (i, j) " as the value of) the pixel 1-byte data corresponding to the (x _n, y _n). Note," L, (i, j ) "
A pointer table in which 1-byte pixels and pointers are assigned in advance to "R, (i, j)".
FIG. 10 shows the relationship between the pointer and the table at this time. In step S409, it is determined whether the above process has been completed for all pixels of the image n. If not completed, the pixel (x _n , y _n ) is moved to the next pixel in step S414, and the process is repeated from step S402. In step S410, it is determined whether w <w _max . If w <w _{max is} not satisfied, the process moves to step S411, and the pointer corresponding to the pixel (x _n , y _n ) is set to a pointer indicating the pixel (x _n-1 , y _n ) or (x _n , y _n-1 ) And If w <w _max , the process proceeds to step S412, w is incremented, a value obtained by multiplying the initial threshold value TH0 by w is set as a threshold value (TH = w · TH ₀ ), and the process is repeated from step S404. It is to be noted here is the _{TH 0 = 5, w ma x} = 5.

Next, the process of generating an image of an arbitrary viewpoint using the compressed ray space data in step S06 of FIG. 2 will be described. FIG. 11 shows the algorithm of this processing. First, in step S501, m reference images and pointer data are read into the memory together with the ray space correspondence table. Conventionally, all N images are read into the memory. However, since only m images and pointer data need be read into the memory, the amount of data to be read into the memory can be smaller than before. Step S50
Moving to 2, a straight line in the xu space is determined based on equation (2) in order to reconstruct an image to be observed at the observer's viewpoint position and line-of-sight direction detected in step S05 of FIG. In step S503, the image number p and the pixel position (x _p , y _p ) are detected from the ray space correspondence table according to the straight line in the xu space. In step S504, it is determined whether or not the image number p obtained in step S503 is a reference image. Image number
If p is the reference image, the process moves to step S505, where the pixel
Let the luminance value of (x _p , y _p ) be the luminance value of the reconstructed image. If the image number p is not the reference image, the process moves to step S506, where the reference image number p ′ to be referred to is obtained from the pointer data of (x _p , y _p ) using the pointer table.
Get the value of (i, j). In step S507, step S50
According to the pointer data obtained in step 6, the reference image p '
The luminance value of (x _p + i, y _p + j) is set as the luminance value of the reconstructed image.

(Second Embodiment) Next, an example in which the image processing apparatus of the present invention is applied to a more general multi-viewpoint image sequence will be described. In the first embodiment, the multi-viewpoint image is moved in viewpoint by a uniform rotation to form an image sequence in which the entire circumference of the object is captured, and reference images are selected from the uniform image sequence at equal intervals. In this embodiment, an example is shown in which a reference image is arbitrarily selected using a multi-view image sequence that is not uniformly changed, but is sparsely picked up at a certain place.

In this case, the same image processing apparatus as that shown in the first embodiment can be used, and only the ray space creation processing in the processing shown in FIG. 2 needs to be changed. In the multi-viewpoint image input device 101 of FIG. 1, when using one camera, the multi-viewpoint image may be taken at an arbitrarily shifted position. May be imaged in an arbitrary arrangement, not at equal intervals.

Next, the multi-viewpoint image compression processing in step S03 in FIG. 2 in the second embodiment will be described. FIG. 12 shows a specific algorithm of the multi-viewpoint image compression processing. In step S601, the N multi-view images captured at an arbitrary viewpoint position are regarded as a series of image sequences, and the multi-view image sequence is rearranged so that a change in viewpoint position between the image sequences is reduced. In step S602, m reference images are arbitrarily selected from the rearranged multi-view images. In step S603, the image number represented by the pointer to the reference image is set to n = 1. In step S604, the two reference images sandwiching the image n are set as target images to be referenced from the image n by the pointer. Note that, of the two reference images selected here, one is L and the other is R, and the two reference images referenced by the image n are distinguished. This is the first
Unlike the first embodiment, a case is considered in which the relationship between the viewpoint positions changes freely in the front, rear, left, and right directions. Therefore, the relationship between the viewpoint positions of the image n and the two reference images cannot always be simply divided into left and right. That's why. Move to step S605,
All pixels (x _n , y _n ) of image _n are referred to as reference image L,
Expressed as a pointer to R. Move to step S606, where n = N-
If it is m, the process ends; otherwise, step S6
In 07, n is incremented and the processing from step S604 is repeated.

With such a processing method, a pointer can be set to the reference image captured at the closest viewpoint position to each image, and a multi-view image captured at an arbitrary viewpoint position can be compressed.

The present invention may be applied to a system constituted by a plurality of devices or to an apparatus constituted by a single device. Needless to say, the present invention is also applicable to a case where the present invention is implemented by supplying a program to a system or an apparatus. In this case, the storage medium storing the program according to the present invention constitutes the present invention. Then, by reading the program from the storage medium to a system or an apparatus, the system or the apparatus operates in a predetermined method.

[0039]

As described above, according to the present invention,
The work memory capacity to be used can be reduced without lowering the processing speed when generating and displaying an image at an arbitrary viewpoint position.

Further, more objects described by the ray space data can be arranged in the three-dimensional virtual space, and an image with higher reality can be generated and displayed to the observer. .

[Brief description of the drawings]

FIG. 1 is a schematic configuration diagram of an arbitrary viewpoint image generation / display device according to a first embodiment.

FIG. 2 is a flowchart of processing of an arbitrary viewpoint image generation / display device according to the first embodiment.

FIG. 3 is an explanatory diagram of a light beam space.

FIG. 4 is a flowchart of a process of creating a ray space correspondence table.

FIG. 5 is a diagram showing an internal data structure of a ray space.

FIG. 6 is a conceptual diagram of a multi-viewpoint image compression process.

FIG. 7 is a flowchart of a multi-viewpoint image compression process according to the first embodiment.

FIG. 8 is a principle diagram of pointer expression.

FIG. 9 is a flowchart of a pointer expression process.

FIG. 10 is a relationship diagram between pointers and 1-byte data.

FIG. 11 is a flowchart of an image generation process.

FIG. 12 is a flowchart of a multi-viewpoint image compression process according to the second embodiment.

[Explanation of symbols]

 Reference Signs List 101 Multi-viewpoint image input device 102 CPU 103 Storage device 104 Viewpoint / viewpoint detection device 105 Image output device 106 Storage medium reading device 107 Bus

Claims (13)

[Claims] A multi-viewpoint image inputting step of inputting images captured at a plurality of viewpoint positions; and a light-ray space in which a light-ray space is associated with coordinates of the multi-viewpoint image based on the input multi-viewpoint image. A correspondence table creating step of creating a correspondence table, at least one of the input multi-viewpoint images is used as a reference image, and the remaining images are described by pointers pointing to pixels of the reference image, whereby the multi-viewpoint A multi-viewpoint image compression step of compressing an image, a ray space creation step of creating ray space data based on the compressed multi-viewpoint image, and a viewpoint / viewpoint detection step of detecting a viewpoint position and a view direction of an observer. A straight line deriving step of deriving a straight line on a ray space from the detected viewpoint position and line of sight of the observer and parameters for image output; and the derived straight line and the creation. An image generation step of extracting pixels to generate an image by tracing a pointer from the compressed multi-viewpoint image to a reference image in accordance with the obtained ray space correspondence table, and an image output for outputting the generated image And an image processing method. 2. The image processing according to claim 1, wherein in the correspondence table creation step, a correspondence table is created in which an xu space, which is a subspace of the ray space, and an x coordinate of a multi-viewpoint image are associated with each other. Method. 3. The image processing method according to claim 1, wherein the reference image is selected at equal intervals from an image sequence of the input multi-viewpoint image. 4. In the multi-view image compression step, the multi-view image is compressed by describing the remaining images with pointers indicating pixels of two reference images located before and after the reference image at a position sandwiching the image. 2. The method according to claim 1, wherein
The image processing method described in the above. 5. The image processing method according to claim 1, wherein in the multi-view image compression step, the input multi-view image sequence is rearranged before selecting a reference image. 6. In the image generation step, an image number and a pixel position are detected from the ray space correspondence table according to the derived straight line, and it is determined whether or not the image number is a reference image. Is determined, a reconstructed image is generated based on the luminance value of the image, and if it is determined that the image is not the reference image,
The image processing method according to claim 1, wherein a reconstructed image is generated based on a luminance value of the image obtained based on the ray space correspondence table. 7. A multi-view image input means for inputting images captured at a plurality of viewpoint positions, and a light space in which a light space is associated with coordinates of the multi-view image based on the input multi-view image. A correspondence table creating means for creating a correspondence table, wherein at least one of the input multi-view images is used as a reference image, and the remaining images are described by pointers pointing to pixels of the reference image. Multi-viewpoint image compression means for compressing an image; ray space creation means for creating ray space data based on the compressed multi-viewpoint image; viewpoint / viewpoint detection means for detecting the viewpoint position and view direction of the observer Straight line deriving means for deriving a straight line in a ray space from the detected viewpoint position and line of sight of the observer and parameters for image output; and the derived straight line and the creation. Image generation means for generating an image by extracting pixels by following a pointer from the compressed multi-viewpoint image to a reference image according to the obtained ray space correspondence table, and an image output for outputting the generated image And an image processing apparatus. 8. The image processing apparatus according to claim 7, wherein the correspondence table creating unit creates a correspondence table in which an xu space, which is a subspace of the light ray space, and an x coordinate of the multi-viewpoint image correspond to each other. apparatus. 9. The image processing apparatus according to claim 7, wherein the reference image is selected at equal intervals from an image sequence of the input multi-view image. 10. The multi-viewpoint image compressing means describes the remaining image by a pointer indicating pixels of two reference images before and after the reference image at a position sandwiching the image.
The image processing apparatus according to claim 7, wherein the multi-viewpoint image is compressed. 11. The image processing apparatus according to claim 7, wherein the multi-viewpoint image compressing unit rearranges the input multi-viewpoint image sequence before selecting a reference image. 12. The image generating means detects an image number and a pixel position from the ray space correspondence table according to the derived straight line, determines whether the image number is a reference image, and determines whether the image number is a reference image. Is determined, a reconstructed image is generated based on the luminance value of the image, and when it is determined that the image is not the reference image, the luminance value of the image obtained based on the ray space correspondence table is calculated. 8. A reconstructed image is generated based on the reconstructed image.
The image processing apparatus according to any one of the preceding claims. 13. A multi-viewpoint image inputting step of inputting images captured at a plurality of viewpoint positions, and a light-ray space in which a light-ray space is associated with coordinates of the multi-viewpoint image based on the input multi-viewpoint image. A correspondence table creating step of creating a correspondence table, at least one of the input multi-viewpoint images is used as a reference image, and the remaining images are described by pointers pointing to pixels of the reference image, whereby the multi-viewpoint A multi-viewpoint image compression step of compressing an image, a ray space creation step of creating ray space data based on the compressed multi-viewpoint image, and a viewpoint / viewpoint detection step of detecting a viewpoint position and a view direction of an observer. A straight line deriving step of deriving a straight line in a ray space from the detected viewpoint position and line of sight of the observer and parameters for image output; An image generating step of extracting pixels to generate an image by tracing a pointer from the compressed multi-viewpoint image to a reference image according to the generated ray space correspondence table; and an image outputting the generated image A computer-readable storage medium storing a computer program for realizing an image processing method comprising an output step.