JP3532823B2 - Image composition method and medium recording image composition program - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing method for modeling a three-dimensional object based on a photographed image and synthesizing an arbitrary viewpoint image, and a medium having an image synthesizing program recorded therein.

[0002]

2. Description of the Related Art Virtual reality, which is a computer utilization technology, CG (computer graphics) when used for moving images and games, and
In a virtual store on the Internet, an image-based rendering technique is important, which can obtain an image of an object viewed from the viewpoint direction by following the ever-changing viewpoint direction. In this case, the image needs to change smoothly and naturally as the direction in which the object is viewed changes.

That is, what is important here is an image-based rendering technique for a three-dimensional object, and it is necessary to be able to create an image with the highest possible resolution at a high speed.

The image-based rendering technique is for modeling a three-dimensional object based on a real image and synthesizing an arbitrary viewpoint image, and a conventional method therefor is as follows.
There are a method of synthesizing an arbitrary viewpoint image using a ray base as shown in FIG. 10 and a method of synthesizing an arbitrary viewpoint image using distance information as shown in FIG.

Here, of these, a method of synthesizing an arbitrary viewpoint image based on light rays will be described with reference to FIG.

FIG. 10 shows a three-dimensional object 701 at an image position 70 using a camera having a lens having a lens focus 703.
It is the figure which looked at the state of photography as an image seen from the direction of 2 surface. Obtaining an image viewed from the surface direction of the image position 702 by capturing an image of the three-dimensional object 701 with a camera corresponds to recording a light ray 704 passing through a focal point 703 of the camera.

FIG. 10 shows a state in which two rays passing through the focal point 703 are recorded with the image of the three-dimensional object 701 as the luminance value of two pixels on the image position 702. Similarly, the brightness values of all the pixels in this line, and further in the pixels on all the lines in the image, can be considered as a ray record in the real space.

If the change accompanying the propagation of the light beam is ignored, not only the focal point 703 of the camera but all the light beams in the recorded light beam direction are recorded. Therefore, if one image is taken, all rays within the viewing angle passing through the focal point,
And the ray on the extension can be recorded. Therefore, if the viewpoint is changed and the three-dimensional object 701 is photographed from a larger number of viewpoints, more ray information about the three-dimensional object 701 can be acquired.

For the synthesis of the arbitrary viewpoint image based on the light rays, the reverse of the acquisition of the light ray information described above may be considered.

Image position 702 and focus position 70 to be combined
When 3 is designated, the light ray 704 passing through the focal position 703 is collected from the light ray information and the image 702 is synthesized.

This light ray information is usually obtained by using a multi-viewpoint image taken by a fixed camera by placing a three-dimensional object on a turntable and rotating it. In order to increase the realism of the composite image, more viewpoint images are required. Therefore, in general, in order to compress the amount of information by using the correlation included in the images, the multi-view images are divided into blocks. The luminance information is compressed by dividing with.

Next, a method of synthesizing an arbitrary viewpoint image using distance information will be described with reference to FIG.

A certain three-dimensional object is modeled by brightness information 801 obtained by photographing with a camera from multiple viewpoints and distance information 802 obtained by a range finder or the like. Further, in order to compress the brightness information, the brightness information and the distance information are managed in block units.

Luminance information 8 managed in units of blocks
01, focusing on one line surrounded by a dotted line in the distance information 802, and projecting points in the three-dimensional space together with the distance information of all the other blocks, a cross section 803 of the three-dimensional object is formed. When the image position 804 and the focus position 805 to be combined are designated, the focus 8 from each point of this three-dimensional object 803.
The projection is performed toward 05, and the brightness information 801 is embedded at the intersection with the image 804. In this way, by performing similar projection on all lines of all blocks, it is possible to synthesize an arbitrary viewpoint image based on the distance information.

[0015]

However, in the above-described image synthesizing method of a three-dimensional object based on light rays and the image synthesizing method using distance information, the projection destination of each pixel in the block is always projected to the synthetic image frame memory. It had to be calculated, and it took a lot of computation time and a lot of synthesis time. Therefore, when the viewpoint position is changed by the observer's will, it is difficult to synthesize images at a sufficient frame rate, and the screen does not expand so that the viewpoint position changes smoothly. The image was lacking.

Therefore, the object of the present invention is to eliminate the above-mentioned drawbacks of the prior art and to increase the drawing speed when the viewpoint position changes and to secure a sufficient frame rate. Therefore, it is an object of the present invention to provide a medium on which an image synthesizing method and a program are recorded so that an image having a smooth viewpoint position change can be obtained.

[0017]

In order to achieve the above object, in the image synthesizing method of the present invention, a three-dimensional object is modeled by data including luminance information, and the luminance information is VQ (vector quantum) in block units. In the image synthesizing step of synthesizing an arbitrary viewpoint image from the coded model, a step of determining whether the viewpoint position is stationary or moving is provided, and when moving, for each block, VQ decoding is applied only to a part of the luminance information, and normal VQ decoding is applied when the image is not moving to perform image composition.

According to the present invention as described above, since the observer draws at a high frame rate while the viewpoint is moving and draws at a high resolution when gazing at a three-dimensional object, It is possible to synthesize images of three-dimensional objects that give the person a more realistic feeling.

[0019]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) An observer looks at a three-dimensional object by synthesizing images by changing the VQ (vector quantization) decoding method depending on whether the viewpoint state is stationary or moving. A method for synthesizing images with high resolution while the viewer is moving and synthesizing images so that the frame rate becomes higher while the observer is moving will be described.

FIG. 1 is a flow chart showing the flow of the image synthesizing method in the first embodiment.

In FIG. 1, reference numeral 101 denotes a viewpoint position / visual line direction input step in which the observer inputs a viewpoint position / visual line direction through an input interface using a (pointing device) such as a mouse or a joystick, and 102 indicates the observer's viewpoint state. A viewpoint state determination step of determining whether the object is stationary or moving, 103 is a high-resolution image of a three-dimensional object based on the viewpoint position / gaze direction information input in the viewpoint position / gaze direction input step 101. A high-resolution image synthesizing step for synthesizing, a low-resolution image synthesizing step for synthesizing an image of a three-dimensional object at a low resolution based on the viewpoint position / gaze direction input step 101, Reference numeral 105 denotes an image combined by these image combining steps 103 and 104 on a display device such as a display. An image display step of displaying the viewer by force.

Now, the viewpoint position / line-of-sight direction input step 101
At, when the viewpoint position / line-of-sight direction for a certain three-dimensional object is input, next, in the viewpoint state determination step 102, it is determined whether the observer's viewpoint position is stationary or moving. .

Here, when the viewpoint position of the observer is stationary, it can be determined that the observer is gazing at the three-dimensional object, and in this case, image combination with higher resolution is performed. There is a need.

When the observer's viewpoint position is moving, it can be considered that the observer is moving around the three-dimensional object, so that the images are continuously and smoothly changed. It is possible to give the observer a virtual reality by synthesizing images at high speed.

Therefore, according to the present invention, if the result of the above determination is that the viewpoint state is still, the processing is transferred to the high resolution image synthesizing step 103, the images are synthesized by this high resolution image synthesizing step 103, and the viewpoint image Is moving, the process is transferred to the low-resolution image synthesizing step 104, and the low-resolution image synthesizing step 104 synthesizes images at low resolution but high distance.

When the images are combined in this manner, the combined image is output to a display device such as a display in the image display step 105, and the observer again waits for the viewpoint position and line-of-sight direction input.

Therefore, when the observer inputs the viewpoint position and the line-of-sight direction, the images viewed from the viewpoint direction are combined and displayed on the display device. Moreover, when the viewpoint state is still, the image is high. The resolution images are combined, and when the viewpoint state is moving, the low resolution images are combined and displayed. Therefore, the optimum image according to the state can be combined and displayed in real time.

<Three-dimensional object image synthesizing method> Next, a high resolution image synthesizing step 103 and a low resolution image synthesizing step 104.
The three-dimensional object image synthesizing method in 1 will be described.

In the image synthesizing method adopted in the present invention, a three-dimensional object is modeled by data including brightness information, and the brightness information is VQ (vector quantization: V) in block units.
ector Quantization) Synthesizes an arbitrary viewpoint image from an encoded model.

A few words about VQ encoding.

The vector quantization (VQ) is a method of replacing the multidimensional space with a representative vector in a vector unit and encoding the number (index) of the representative vector, and means to quantize the multidimensional space. To do. In terms of image encoding, an image is divided into a plurality of blocks, pixel values included in the blocks are regarded as vector elements, and the entire block is collectively replaced with a representative block for quantization.

For example, the vector space in the vector quantization in the two-dimensional case is as shown in FIG. A region represented by each of the representative vectors y is determined, and the plane is a Voronoi-divided plane with the representative vector. For example,
When the vector x is input in the two-dimensional plane, it is quantized into the representative vector y having the shortest distance, and the representative vector y is output.

A general image coding method in vector quantization will be described with reference to FIG. Input image as K
The block is divided into [pieces] of pixels, and each is regarded as a K-dimensional vector P.

P = (p1, p2, ..., PK) where (p1, p2, ..., PK) are sample values.

Transmitting side (encoding side) and receiving side (decoding side)
Then, a codebook {Q1, Q2, ..., QN} composed of N [pieces] K-dimensional representative vectors is prepared. On the transmitting side, a representative vector having the shortest distance from the input vector P is obtained from this codebook by distance calculation, and if it is Qi, the index i of the representative vector is transmitted as an output code.

On the receiving side, the representative vector indicated by the index i that can be sent is read from the codebook, deblocked, and output as a decoded image.

In order to realize efficient vector quantization, codebook design becomes important. In general, a method is used in which a plurality of test images that represent input images are prepared and a set of representative vectors that minimizes coding distortion (quantization distortion) is sequentially obtained for these test images. A typical design method for codebooks is "k-mea"
n-method "(For more information about this" k-mean method ", see Mikio Takagi, Yohisa Shimoda," Image Analysis Handbook, "The University of Tokyo Press, pp.648-651, (1991)).
A brief description of the above is given below.

(1) Let Si be the region of the m × n dimensional vector space that is quantized into the representative vector Qi. Further, training sequences t1, ..., tk are prepared.

(2) Representative vector Qi (1 = 1 to N)
Give an appropriate initial value to.

(3) The distance between the representative vector Qi and the training sequence t1, ...
Find out which training sequence t belongs to i.

(4) For each Si, the center of gravity of t belonging to it is set as a new representative vector Qi.

The above (3) and (4) are repeated until the quantization error converges to a sufficiently small value.

Image coding is performed using the codebook designed in this way.

Next, the general image coding described above is applied to a multi-viewpoint image, and an attempt is made to code / decode the multi-viewpoint image.

When encoding a multi-viewpoint image using vector quantization (VQ), the point that is greatly different from general VQ image encoding is that a codebook is designed for each three-dimensional object and the designed code is used. The point is that the book and all index data are transmitted to the decoding processing side and held in the memory on the decoding processing side.

In the present invention, a codebook is designed for each three-dimensional object, and the multi-viewpoint images required for it are images of the same subject, and the same portion of the subject is often captured in a plurality of images. To fully utilize this correlation, all blocks of all multi-view images are input as a training sequence.

By grouping similar objects into typical objects, one object can be represented with a small amount of information. Similarly, this method is separately performed for each different subject.

As described above, even if a subject-specific codebook using all multi-viewpoint images of each three-dimensional object is created and given to the decoding processing side, each codebook is general. A smaller amount of information is required than a codebook with VQ, and the reproduced image can realize high image quality. It should be noted that all vector quantization data may be fixed length data. By using fixed length data, the decoding process does not take much time, and it is possible to play back only by referring to any single block. Decoding that maintains high speed and random accessibility of any block. This is also the optimum encoding method.

Specifically, consider multi-view image coding by vector quantization.

On the encoding side, as shown in FIG. 4, first, a block of all multi-viewpoint images is prepared as a training sequence, and "km
The ean method is applied to design the codebook. Based on the created codebook, distance calculation is performed for blocks in all multi-view images, and the index of the nearest representative vector is output.

Finally, the encoding processing side transmits a codebook designed from all multi-view images and index data for all multi-view images.

On the decoding side, the codebook and index data for all multi-viewpoint image blocks are received and held in the memory, and which block is decoded by the arbitrary viewpoint image synthesizing step according to the viewpoint information. Wait for instructions. Then, you can select a block in the codebook that has the index number corresponding to the requested block,
Decode blocks needed for composition.

Although the basic concept of the present invention is as described above, there is a conventional method of decoding using the codebook and index information. However, in the case of the conventional method, the same codebook is prepared on the encoding processing side and the decoding processing side from the beginning, and the encoding processing side has its own index corresponding to the image to be combined. The image is extracted from the codebook, only the index is sent, and the decoding processing side obtains the image corresponding to this index from the codebook and synthesizes the image.

In this case, the codebook must be a general-purpose one, and the image finally obtained must be a pseudo one.

In order to solve this problem, a codebook for an image is created according to the image and sent to the decoding side, and the decoding side uses this to reproduce the image from the received index. To In this way, a virtual reality and walk-through based on live-action with realism, and an optimal encoding / decoding method for video fields such as product exhibitions in virtual shops and interactive electronic museums are realized. It will be possible.

The data held in the memory is the designed codebook and the index data for each block in all multi-view images, both of which are fixed length data. Therefore, unlike the intra-frame DCT coding and the hybrid coding, it is not necessary to perform variable-length decoding or hierarchical decoding, and it is possible to perform arbitrary operation only by a simple operation of referring to the representative vector in the codebook according to the index. Blocks can be decrypted quickly.

In the present invention, the high resolution image synthesis is performed by using the normal V
It is realized by performing Q (vector quantization) decoding processing, and low resolution image composition is realized by performing VQ decoding processing only for some blocks in the block.

In FIG. 5, reference numeral 201 is information on the viewpoint position / visual line direction input in the viewpoint position / visual line direction input step S101, and 202 is data of a three-dimensional object modeled by luminance information in block units. Of these, a block selection step of selecting a block required for synthesis, 20
Reference numeral 3 is a V for performing VQ (Vector Quantization) decoding of luminance information based on the index corresponding to the block selected in the block selection step 202.
A Q decoding step, 204 is a codebook referred to in the VQ decoding step 203, 205 is a block from the viewpoint position / line-of-sight direction information 201 and the decoded luminance information, and a block is projected to a frame memory to synthesize an image. An image synthesizing step, 206 is an end determining step of determining whether all the luminance information in block units has passed through the block selecting step 202, and 207 is a synthetic image sent to the image displaying step 105.

In the present invention, when the information 201 of the viewpoint position and the line-of-sight direction for the three-dimensional object is input, in the block selecting step 202, the image combining method based on the light ray as described in the conventional example or the distance information is obtained. A block required for composition is selected from the three-dimensional object data (block unit) by the image composition method used.

Then, based on the index corresponding to the selected block, the codebook 204 is referred to, and VQ
The luminance information of the coded block is VQ-decoded by the VQ decoding step 203.

In the VQ decoding step 203, if the stationary state or the moving state is determined in the viewpoint state determining step S102, if the stationary state, the VQ decoding is normally performed and the moving state is in progress. If so, only some of the blocks are decoded.

Next, in the image synthesizing step 205, the luminance information is stored in the frame memory from the viewpoint position / line-of-sight direction 201 and the VQ-decoded luminance information by an image synthesizing method based on light rays or an image synthesizing method using distance information. Projected.

This projection calculation is performed only on the VQ-decoded luminance information. Therefore, when the operation is in progress, the projection is performed with less data than when the operation is stationary, and accordingly, the drawing can be performed at high speed.

Finally, in the end determination step 206, all blocks of the three-dimensional object data are selected in the block selection step 20.
If it is determined that all blocks have passed, and the composite image 207 is completed.

As a result of such processing, the image is synthesized by changing the VQ (Vector Quantization) decoding method depending on whether the viewpoint state is stationary or moving, whereby the observer is three-dimensional. It becomes possible to perform high-resolution image synthesis while gazing at an object, and to synthesize images so that the frame rate becomes higher while the observer is moving.

<Decoding Method According to Judgment Result of Viewpoint State Judging Step> Next, a decoding method according to the judgment of the viewpoint state judging step S102 in the VQ decoding step 203 will be described with reference to FIG.

The size of the representative vector in VQ (vector quantization) is a 4 × 4 pixel 16-dimensional vector. In FIG. 6, 301, 302, 303, and 304 are 4 × 4 pixels 1 stored as a codebook, respectively.
6-dimensional representative vector, 305 is a block of 8 × 8 pixels consisting of 301 to 304, 306 is a representative vector 301
A block in which 16 pixels are divided into 8 × 8 pixels and expanded to 8 × 8 pixels. Reference numeral 307 is an 8 × 8 block which is based on the representative block 301 and has a resolution “1/2” corresponding to a block 305 of 8 × 8 pixels. It is a block of pixels.

If it is determined in the viewpoint state determination step 102 that the observer's viewpoint is still,
The VQ decoding step 203 normally performs VQ decoding. That is, the representative vector corresponding to the index of the selected block is referred to from the codebook 204 to perform VQ decoding.

For example, the area 305 of 8 × 8 pixels in FIG.
VQ decoding of the representative vectors 301 to 304 is performed independently.

On the other hand, in the viewpoint state determining step 102,
If it is determined that the observer's viewpoint is moving, VQ
The decoding step 203 VQ-decodes only the representative vector 301, divides the representative vector 301 as in 306, and expands the same luminance value to 2 × 2 pixels 8 × 8 block 30.
An 8 × 8 pixel block 307 corresponding to 5 and having a resolution of ½ is used.

Next, the processing in the image synthesizing step 205 starts. In the image synthesizing step 205, an image is synthesized by performing block projection calculation and projecting pixels on the frame memory. This block projection calculation will be described with reference to FIG. 7.

First, a case where image composition is performed using distance information will be described.

When the viewpoint state determining step 102 determines that the viewpoint is moving and the 8 × 8 pixel block 307 having the resolution “1/2” is projected on the frame memory, the luminance information decoded in the block 306 is used. Projection is performed using the distance data of only the pixels in the portion where is entered. That is, as shown in FIG. 7, the block 306 is divided into 16 2 × 2 blocks.
With respect to the pixel block, the projection calculation is performed only on one pixel among the four pixels, and the remaining three pixels are similarly projected onto the frame memory.

In the case of performing image synthesis based on rays, only the decoded area of the block 301 is projected onto the frame memory based on the rays, and the remaining 3/4 area is projected onto the frame memory. Similar projection is performed as shown in FIG. Therefore, the amount of calculation that has to be performed when the viewpoint is moving is only 1/4 that when the viewpoint is stationary, and accordingly, the three-dimensional object can be synthesized at high speed.

As described above, by combining the images by changing the VQ decoding method depending on whether the viewpoint state is stationary or moving, when the observer is gazing at the three-dimensional object, a high resolution image is obtained. By synthesizing images, it becomes possible to synthesize images with a higher frame rate while the observer is moving.

In the present embodiment, there are two VQ decoding methods, one for high resolution image synthesis and one for low resolution image synthesis. However, the quality of the decoded image at high resolution is not affected. In addition, the codebook code amount has not increased. However, the image quality of a low-resolution decoded image is lower than that of a general image having a resolution of "1/2".

Therefore, an example in which the quality of a low-resolution decoded image is not deteriorated will be described next as a second embodiment.

(Second Embodiment) In the VQ decoding step 203 and the codebook 204 in the image synthesizing method of the first embodiment described above, when the codebook 204 is created, there is a space between each pixel in the vertical and horizontal directions. By designing the codebook in advance with the pattern of the resolution "1/2" which does not overlap with each other, it is possible to suppress the image quality deterioration of the decoded image at the low resolution. In the second embodiment, it is possible to suppress the image quality deterioration of the decoded image at low resolution by utilizing this fact.

Hereinafter, a method of synthesizing an image of a three-dimensional object which prevents deterioration of image quality at low resolution will be described.

VQ decoding step 2 in the second embodiment
A decoding method according to the determination in the viewpoint state determination step 102 in 03 will be described with reference to FIG.

In this method, as shown in FIG.
Instead of dividing the block into four 4x4 blocks,
A codebook is designed by a training sequence in which an 8 × 8 block 505 is divided into patterns such as 501, 502, 503, and 504 as patterns of alternating resolution “1/2” with one pixel left in each of the vertical and horizontal directions. .

When the viewpoint state determining step 102 determines that the observer's viewpoint is still, the VQ decoding step 203 decodes the representative vectors 501 to 504,
Decrypt the × 8 block 505.

In the viewpoint state determining step 102, when it is determined that the observer's viewpoint is moving, only the representative vector 501 is VQ-decoded and the resolution “1/2” corresponding to the 8 × 8 block 505 is obtained. The 8 × 8 pixel block 506 is created as the block of No.

Then, the projection calculation is performed by the same method as that described in the first embodiment. As a result, the projection calculation handled when the viewpoint of the observer is moving is
1/4 of the projection calculation when the observer's viewpoint is stationary
Therefore, it becomes possible to synthesize a three-dimensional object at a high distance, and it is possible to maintain the image quality of an image having a general resolution of "1/2", and suppress image quality deterioration.

As described above, in the present embodiment, the observer pays attention to the three-dimensional object by combining the images by changing the VQ decoding method depending on whether the viewpoint state is stationary or moving. When the observer is moving, it is possible to synthesize images with high resolution and to synthesize images with a higher frame rate.

Further, according to this embodiment, in a low-resolution decoded image that can be synthesized at high speed, a general resolution of "1 /
2 "image can be realized. However, codebook 204
Since the pattern is designed by the pattern of the resolution "1/2" which does not overlap with each other and has one pixel in the vertical and horizontal directions, the VQ encoding using the intra-block correlation is performed as in the first embodiment. Not done. Further, as in the first embodiment, the codebook code amount does not increase.

The above is the embodiment in which the resolution can be maintained at "1/2" when synthesizing the low resolution image.

Next, a codebook used in VQ decoding is selectively used depending on whether the viewpoint state is stationary or moving to synthesize an image, thereby obtaining a high resolution when the observer is gazing at a three-dimensional object. An example will be described as a third embodiment in which the images are combined so that the image can be combined with a higher frame rate while the observer is moving.

(Third Embodiment) VQ decoding step 203 and codebook 20 in the image combining method of the first embodiment.
4, the two types of codebooks 204 of high resolution and low resolution are prepared in advance, and the codebook 204 is used properly while the observer's viewpoint is stationary or moving, so that the image quality of the high resolution is not deteriorated and the low resolution natural It is possible to perform various samplings.

VQ decoding step 2 in the third embodiment
A decoding method according to the determination in the viewpoint state determination step 102 in 03 will be described with reference to FIG. In this method, the first
6 in the above embodiment, the 8 × 8 block 601 is replaced by four 4 × 4 blocks 603, 604, 605, 606.
Design a high-resolution codebook by dividing it into.

A low-resolution block 602 corresponding to the 8 × 8 block 601 is created in advance, and a low-resolution codebook is also designed based on the 4 × 4 block 607. The codebook code amount requires 1/4 of the high-resolution codebook. Viewpoint state determination process 102
If it is determined that the observer's viewpoint is still, the VQ decoding step 203 uses the codebook for high resolution and the 4 × 4 blocks 603, 604, 605, 6
Decode 06.

If it is determined in the viewpoint state determination step 102 that the observer's viewpoint is moving, VQ decoding is performed on the 4 × 4 block 607 using the codebook for low resolution to obtain an 8 × 8 block. Resolution corresponding to 601 "1 /
An 8 × 8 pixel block 602 is created as a 2 ″ block.

Then, the projection calculation is performed by the same method as the method described in the first embodiment. As a result, the projection calculation handled when the viewpoint of the observer is moving is
1/4 of the projection calculation when the observer's viewpoint is stationary
Therefore, it becomes possible to synthesize a three-dimensional object at a high distance, and it is possible to perform natural sampling even at a low resolution without deteriorating the image quality at a high resolution.

As described above, depending on whether the viewpoint state is stationary or moving, the codebooks used in VQ decoding are selectively used to synthesize an image, so that
It becomes possible to perform high-resolution image synthesis while gazing at a three-dimensional object, and to synthesize images so that the observer can increase the frame rate while moving.

In this embodiment, it is possible to achieve both the image quality of a high resolution composite image that can be realized in the first embodiment and the image quality of a low resolution composite image that can be realized in the second embodiment. However, the codebook code amount becomes 5/4 times.

The present invention is not limited to the above-mentioned embodiments, but can be carried out by appropriately modifying it within the scope of the invention. In addition, the method described in the embodiment is a program that can be executed by a computer.
Magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD
Etc.), and can be distributed by being stored in a recording medium such as a semiconductor memory.

[0098]

As described above, according to the image synthesizing method of the present invention, the resolution of the block to be decoded is changed depending on whether the viewpoint state is the still state or the moving state so that the observer can see the still state. Has the effect of enabling high-resolution image composition and image composition at a high frame rate during operation.

[Brief description of drawings]

FIG. 1 is a diagram for explaining the present invention and is a flowchart showing a flow of an image synthesizing method according to the first embodiment of the present invention.

FIG. 2 is a diagram for explaining a vector space in QV.

FIG. 3 is a diagram for explaining general image encoding by vector quantization.

FIG. 4 is a diagram for explaining other-viewpoint image coding by vector quantization.

FIG. 5 is a diagram for explaining the present invention and is a flowchart showing a flow of image synthesizing steps S103 and S104 in the image synthesizing method according to the first embodiment of the present invention.

FIG. 6 is a diagram for explaining the present invention, which is a diagram for explaining the VQ encoding step 203 of the first embodiment of the present invention.

FIG. 7 is a diagram for explaining the present invention and a diagram for explaining a projection method in the first embodiment of the present invention.

[Fig. 8] Fig. 8 is a diagram for explaining the present invention and a diagram for explaining a VQ encoding step 203 of the second embodiment of the present invention.

FIG. 9 is a diagram for explaining the present invention, which is a VQ encoding step 203 and a codebook 2 of the third embodiment of the present invention.
FIG.

FIG. 10 is a diagram illustrating a method of synthesizing an arbitrary viewpoint image using a ray base.

FIG. 11 is a diagram illustrating a method of synthesizing an arbitrary viewpoint image using distance information.

[Explanation of symbols]

101 ... Viewpoint position / direction input step 102 ... Viewpoint state determination step 103 ... High resolution image combining step 104 ... Low resolution image combining step 105 ... Image display step 201 ... Viewpoint position / viewpoint direction 202 ... Block selecting step 203 ... VQ decoding Step 204 ... Codebook 205 ... Image combining step 206 ... End determination step 207 ... Combined images 301 to 304 ... 4x4 block 305 ... 8x8 block 306 ... Blocks 307 ... 8 obtained by dividing pixels in 4x4 block 301. Resolution 8 corresponding to × 8 block 305
Blocks 501 to 504 ... 16-dimensional representative vectors 505 of non-overlapping patterns ... 8 × 8 block 506 ... Resolution 1/2 corresponding to 8 × 8 block 505
Block 601 ... 8 × 8 block 602 ... 8 × 8 block 601 with resolution 112 blocks 603 to 606 ... 8 × 8 block divided into 4 4 × 4 block 607 ... 8 × 8 block 601 with resolution 112 701 ... Three-dimensional object 702 ... Camera image 703 ... Lens focus 704 ... Ray information 801 ... Luminance information block 802 ... Distance information block 803 ... Three-dimensional object section 804 ... Camera image 805 ... Lens focus

─────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-9-289638 (JP, A) JP-A-5-35847 (JP, A) JP-A-8-186720 (JP, A) JP-A-11- 66346 (JP, A) JP-A-11-18091 (JP, A) JP-A-63-283273 (JP, A) KATAYAMA Akihiro 3 others, "Compression and implementation of ray space data considering real-time drawing" , Transactions of the Virtual Reality Society of Japan, Japan Virtual Reality Society, December 31, 1999, Volume 4, Issue 4, p. 631-638 Miyakoshi Masaaki, "Fuzzy Clustering and its Applications / Application of Fuzzy Clustering to Image Analysis", Journal of the Japan Society for Fuzzy Science, The Japan Society for Fuzzy Science, June 15, 1996, Vol. 8, No. 3, p. ． 440-447 (58) Fields surveyed (Int.Cl. ⁷ , DB name) H03M 7/30 G06T 3/00 G06T 17/40 H04N 5/262 H04N 1/41-1/419

Claims (5)

(57) [Claims] 1. An image synthesizing method for synthesizing an image at an arbitrarily designated viewpoint from a model in which a three-dimensional object is modeled by data including luminance information and the luminance information is vector-quantized and coded in block units. In the case of determining whether the specified viewpoint position is stationary or moving, and when the determination result is that the viewpoint position is moving,
Image synthesis method of the by subjecting only a portion of the luminance information in each block to be processed the vector quantization processing, and performs image synthesis. 2. The image synthesizing method according to claim 1, wherein in the vector quantization decoding processing of the luminance information, the image synthesizing is performed by representing the luminance information of 4 blocks by one block. 3. In the vector quantization coding processing of the luminance information, a codebook is designed for each target object, and the designed codebook and all index data are held on the decoding processing side, and image reproduction is performed. Is performed by using a representative vector obtained from the codebook corresponding to the given index data, and the codebook is designed in advance by a pattern with a non-overlapping staggered resolution of 1/2 for each pixel in the vertical and horizontal directions. The image synthesizing method according to claim 1, wherein the image synthesizing method is one. 4. A three-dimensional object is modeled by data including luminance information, and further, in image synthesizing processing for synthesizing an arbitrary viewpoint image from a model in which the luminance information is vector-quantized and coded in block units, vector quantization is performed. An image characterized by designing a standard codebook and a low-resolution codebook when creating a codebook, and performing image synthesis using this low-resolution codebook when the viewpoint is moving. Synthesis method. 5. A three-dimensional object is modeled by data including luminance information, and luminance information is vector-quantized and coded on a block-by-block basis to synthesize an image at an arbitrarily specified viewpoint , Read by computer
A medium in which a finely executable image synthesis program, and determining whether moving or viewpoint position said specified is stationary, this determination is base if is in the viewpoint position movement vector by subjecting only a portion of the luminance information in each block to be processed quantization processing, a medium recording the images synthesis program that includes a step of performing image synthesis.