JPH1013860A - Stereoscopic picture interpolating device and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a precisely interpolated picture by obtaining a movement vector between pictures from right and left pictures and the input of picture view point information, weighting it by a coefficient decided by interpolated picture view point information and outputting the interpolated picture to reduce erroneous detection. SOLUTION: The pictures supplied from right and left picture inputs 1 and 11 are stored in picture memories 2 and 12 and given to picture moving devices 3 and 13 at the same time. In addition, a movement vector detector obtains the movement vector between the pictures from the right and left pictures and an existent picture view point information 7 for each block. A moving quantity calculator 9 calculates the moving quantity of the right and left pictures from view point information and movement vector information. The devices 3 and 13 obtain the pictures from the memories 2 and 12 by each block, move the pictures by the portion of the moving quantity given from the part 9 and output outputs to multipliers 4 and 14. Then the outputs are weighted by a coefficient decided by interpolated picture view point information 1 and added by an adder 5 to output an interpolated picture output 6. Thereby, the searching range for the movement vector is minimized to reduce erroneous detection to obtain a precisely interpolated picture.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic image apparatus which makes it possible to stereoscopically view a stereoscopic image obtained by image pickup by a camera or processing on a computer by a display or printing, and more particularly to a stereoscopic image apparatus having more than two images for both eyes. The present invention relates to a three-dimensional image capturing, processing, and display device capable of reproducing a viewpoint video.

[0002]

[Prior art]

<Stereoscopic Image> In stereoscopic imaging and stereoscopic display of an image, a device that acquires and displays only two types of images, a right-eye image and a left-eye image, is generally used. In this case, only one type of stereoscopic image can be viewed. Only.

On the other hand, by capturing images from more viewpoints (angles) and displaying each image so as to be viewed from the corresponding viewpoint, it is possible to display a stereoscopic image corresponding to many viewpoints. In this case, when you look at the playback image, if you change the viewpoint, the image actually seen will change accordingly,
Very natural stereoscopic image display is achieved. It is possible to display many images according to angles by using high-precision printing and lenticular lenses.

In such an apparatus, in order to obtain images from many viewpoints, when the object is a still object, the image is taken while moving the camera. In this case, if the viewpoint is finely set to improve the quality of the stereoscopic image, a large number of images are taken in, so that it takes time, and during that time, it is necessary to maintain the photographing state of the still life.

If the object is a person or an animal, it is necessary to instantaneously obtain images from all viewpoints, so that a camera is placed for each viewpoint and images are taken simultaneously. In this case, a very large number of cameras are required, and the storage of the image information must be performed instantaneously. It is necessary to adjust the position and status (zoom, focus, iris) of each camera, and it is difficult to obtain images from many viewpoints.

Therefore, a method of thinning out the viewpoints to be imaged and interpolating and generating the images of other viewpoints by image processing using a computer or the like, instead of obtaining images of all viewpoints by imaging, has been studied. ing.

FIG. 4 shows an image pickup mode in this case. The object of the stereoscopic image is imaged while changing the viewpoint from two directions, left and right, and an interpolated image is obtained from the obtained image by electronic processing. These processing methods are described in "Electronically Interpolat
ed Integral Photography System "(USP-54556
89) and “Method and apparatus for constructing an intermediate image for a depth image from a stereo image” (JP-A-7-210686).
No.).

[0008]

In the conventional stereoscopic image interpolation apparatus, the detection of the movement vector is limited to one scanning line (within a row of water pixels) in order to speed up the operation. Therefore, it is not possible to cope with a minute movement in the vertical direction of the image or a shift in the imaging position caused by a change in the depth due to a change in the viewpoint.

Further, since the number of pixels used for the movement vector calculation is not sufficient, there is a high possibility that an erroneous movement vector is obtained due to noise, a minute shift of a pixel sample position, or the like. On the other hand, if the technique used for motion vector detection of a moving image is applied as it is, the amount of pattern matching calculation for obtaining a motion vector is extremely large, and the processing requires a lot of time and circuits.

The present invention has been made in view of the above points, and is used for obtaining a movement vector, which is a shift between images necessary to interpolate an image between the viewpoints, from obtained images at a plurality of viewpoints. It is another object of the present invention to provide a stereoscopic image interpolating apparatus that obtains an accurate motion vector and performs interpolation by appropriately setting a search range by block matching.

Further, in the conventional three-dimensional image interpolation device, the left image and the right image do not match (match) at the boundary between the object and the background, and a correct movement vector cannot be obtained. Therefore, an appropriate interpolated image cannot be obtained. Similarly, a correct movement vector cannot be obtained at the end of the image, and an appropriate interpolated image cannot be obtained. In particular, if the viewpoint distance of the existing image is increased, a portion where the left and right images do not match occurs over a wide range, and the image quality of the interpolated image deteriorates. The present invention has been made in view of the above points, and it is an object of the present invention to provide a stereoscopic image interpolation device that obtains an accurate movement vector and performs appropriate interpolation by correcting a movement vector different from that of the surroundings. .

[0012]

SUMMARY OF THE INVENTION The present invention relates to an apparatus for handling a three-dimensional image using images of many viewpoints. When an image corresponding to is created by interpolation, a search range of a movement vector is set according to the viewpoint distance of the existing image, and a movement vector representing a displacement of a pattern between the two left and right existing images is obtained. From the viewpoint positional relationship of the existing image and the movement vector, the movement amounts of the left and right existing images for interpolation are calculated in block units, and the left and right existing images are moved and added in block units by the respective movement amounts to perform interpolation. This is a stereoscopic image interpolation device for obtaining an image.

In the moving vector detecting means, the three-dimensional image interpolating device for obtaining the moving vector by setting the horizontal search range, which is the direction in which the viewpoint changes, at least twice the vertical search range orthogonal to the horizontal search range. It is. Further, in the movement vector detecting means, a three-dimensional image interpolation apparatus for setting the horizontal search range, which is the direction in which the viewpoint changes, by changing its absolute value between the left direction range and the right direction range, and obtaining the movement vector It is.

According to the present invention, when obtaining a movement vector, which is a shift between images required for interpolation, the number of reference pixels for the matching operation is increased by block matching, and erroneous detection is reduced. By searching the horizontal direction wider than the vertical direction based on the viewpoint distance of the existing image, it is suitable for the image movement of the stereoscopic image whose vertical movement is slight.

In addition, the search range on the left and right is changed, and if there is a background, the movement of the background is larger than the object, so that the wider direction is set as the moving direction of the background. Since the background does not move with the movement of, it is possible to adapt to the actual movement by reducing the moving direction of the background. By adapting the search range to the actual moving range as described above, it is possible to perform image interpolation with less erroneous detection and less calculation amount.

The present invention also relates to an apparatus for handling a stereoscopic image using images of many viewpoints, wherein the left and right images of different viewpoints are used.
When using two existing images and interpolating and creating an image corresponding to a viewpoint sandwiched between the viewpoints of the existing image, a movement vector representing a displacement of a pattern between the two left and right existing images is obtained,
If the motion vector is significantly different in the periphery, the motion vector is corrected to a peripheral motion vector, and the amount of movement of the left and right existing images for interpolation is calculated in block units from the viewpoint positional relationship between the interpolated image and the existing image and the motion vector. Then, the left and right existing images are moved by the amount of each movement in block units, and the left and right images are added for blocks that do not modify the moving block, and only one of the right and left images is used for the corrected block. This is a stereoscopic image interpolation device for obtaining an interpolation image.

In the above-described processing, a three-dimensional image interpolating apparatus that corrects a motion vector, corrects a block having a different motion block from any of its neighboring neighbors, and selects the right or left block having the larger absolute value of the motion vector. is there. Further, there is provided a stereoscopic image interpolating apparatus which uses only an image in a direction in which the motion vectors do not intersect by interpolation of a block in which the motion vectors have been corrected in the above processing. Further, there is provided a stereoscopic image interpolating apparatus for correcting a motion vector to an area in which a motion vector is correctly obtained at an end of an image for which a motion vector is not properly obtained in the above processing.

According to the present invention, a movement vector, which is a shift between images required for interpolation, is obtained, and when the movement vector is different from the surrounding blocks, the movement vector is corrected. Since the correlation is high, the moving vector different from the surroundings has no matching part in both the left and right existing images, and it is highly likely that it is the wrong moving vector, so it should be corrected to one of the surrounding moving vectors Is a more probable movement vector. In such a block, there is no matching part on both the left and right sides, but in the case of a stereoscopic image, since there is a matching part on either the left or right side, appropriate interpolation can be performed by interpolating from one side.

In the correction of the movement vector, the one having the larger absolute value of the movement vector is selected in the right or left block. Is high, which results in a likely motion vector. Furthermore, by using only the image in the direction in which the motion vectors do not intersect in the interpolation of the block in which the motion vector has been corrected, an image portion having a large moving area is used for interpolation, and the possibility that a compatible portion is used for interpolation is obtained. high.

Although an appropriate movement vector cannot be obtained at the end of the image, blocks inside a certain degree or more are not affected by the end, so that correct interpolation can be performed by replacing the block with the movement vector. By correcting the motion vector as described above and using only one of the images for interpolation, more reliable interpolation becomes possible and the quality of the interpolated image is improved.

[0021]

BEST MODE FOR CARRYING OUT THE INVENTION

<Stereoscopic Image Interpolating Apparatus 1> An embodiment of the stereoscopic image interpolating apparatus of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of the stereoscopic image interpolation apparatus according to the present invention. Existing images supplied from the right image input 1 and the left image input 11 are stored in image memories 2 and 12, respectively. The right and left existing images stored in the image memories 2 and 12 are supplied to the image movers 3 and 13 and used by the movement vector detector 8 to obtain a movement vector between the images.

The motion vector detector 8 calculates a motion vector for each block of 4 × 4 to 16 × 16 pixels from the left and right existing images and the viewpoint information (viewpoint distance) of the existing image given from the existing image viewpoint information input 7. Ask for. The moving vector detection processing method performed here is similar to the motion vector detection processing method used in motion compensated image interpolation for format conversion of a moving image.

Processing methods for format conversion are described in "Image Signal Scanning Line Converter" (JP-A-6-133280 and JP-A-6-141291) and "Video Signal Inter
polation Using Motion Estimation ”(USP-438
3272), in which images at different viewpoints are used instead of images at different times. When obtaining an interpolated image of a viewpoint sandwiched between existing images, a relative position (angle) between the existing image and the interpolated image is given from viewpoint information input.

The movement amount calculator 9 calculates the movement amount of the right image and the left image from the viewpoint information and the movement vector information. The movement vector information is one for the input existing image regardless of the viewpoint, but the movement amount changes depending on the viewpoint.

If the movement vector is (x, y), the distance between the viewpoint of the interpolation image and the viewpoint of the right image is a, and the distance of the viewpoint between the interpolation image and the viewpoint of the left image is b, movement of the right image (x ', y') X ′ = x × a / (a + b), y ′ = y × a / (a + b)
Becomes

The image movers 3 and 13 obtain the images from the image memories 2 and 12 for each block, move the images by the amount of movement given from the movement amount calculator 9, and supply the outputs to the multipliers 4 and 14. . The multipliers 4 and 14 add coefficients k and (1) to the image signal.
-K).

The coefficient k is determined by the viewpoint information, and the ratio of the image closer to the viewpoint is increased. One specific method is k = b / (a + b). Further, this may be simplified, and k may be approximated to 0, 1/2, and 1.

The images weighted by the multipliers 4 and 14 are
The sum is added by the adder 5 to become an interpolated image and an interpolated image output 6
Output from The viewpoint information is sequentially changed as necessary, and an interpolated image corresponding to the viewpoint information is obtained. If the viewpoint of the existing image is different by 10 degrees, if an interpolated image of the viewpoint obtained by dividing the viewpoint into 10 is obtained, finally, 11 images corresponding to the viewpoints of 1 degree are obtained.

<Search Range in Detecting Movement Vector> The feature of this embodiment is the setting of the search range in the detection of the movement vector shown in FIG. It is important to set the search range to a necessary and sufficient one when obtaining the motion vector in order to obtain a correct interpolation image.

If the search range is wider than necessary, the possibility of detecting an erroneous movement vector increases. This is significantly different from the fact that the search range for detecting a motion vector of a moving image is generally wider as it is better.

In the motion compensation, the generation state of the motion vector spreads slightly in the horizontal direction, but there is no large difference between the vertical direction and the horizontal direction. On the other hand, in the case of a three-dimensional image, it is greatly different, and is basically only in the horizontal direction. In the conventional example, processing limited to the horizontal direction is performed. However, unless the position of the target object is at infinity, if the position in the depth direction (distance from the viewpoint to the target) changes depending on the viewpoint, a slight shift occurs in the vertical direction in the upper and lower parts of the image.

On the other hand, the movement of the image always falls within a certain range due to the positional shift of the object and the viewpoint shift between the existing images. Therefore, the search range in the horizontal direction is also limited thereby. In the case of imaging in the form as shown in FIG. 3, since it is usually usual to image the object so as to be located at the center of the image, the movement vector of the background portion distant from the object in the depth direction becomes large.

If the background is a dark curtain or the like and the picture is lost, the movement vector can be ignored, so that only the object moves and the search range can be reduced.

The internal configuration of the motion vector detector 8 is shown below together with FIG. In FIG. 4, the image memories 2 and 12 correspond to the image memories 2 and 12 in FIG. The basic operation is performed by pattern matching in block units. In this process, the addresses of the image memories 2 and 12 where the existing images are recorded are set, the existing image data is obtained for each of the motion vectors within the search range, and the motion vector with the least matching error is selected.

The address generator 41 generates and outputs an image address when the movement vector is 0. There are addresses that change on a block basis and addresses that change on a pixel basis within the block.

On the other hand, the temporary vector generator 46 sequentially outputs the motion vectors within the search range. The temporary movement vector from the temporary vector generator 46 and the block address from the address generator 41 are added by the adders 42 and 47, and the address including the movement vector is given to the image memories 2 and 12.

The motion vector added by the adder 42 is
The inverted value is obtained by the vector inverter 45, so that the right image and the left image move in opposite directions as shown in FIG. Here, the search range is wide in the horizontal direction and narrow in the vertical direction as shown in FIG.

The image data at the designated address is output from the image memories 2 and 12, and the difference is calculated by the divider 43. The output of the divider 43 is squared by a square error detector 44, and the pixel differences are cumulatively added in block units to obtain a square error value for each temporary movement vector.

This value is compared by a minimum value selector 48 between all the provisional movement vectors within the search range, and the provisional movement vector giving the minimum value is used as the movement vector of the block as a movement vector output terminal 50. Output more.

<Stereoscopic Image Interpolating Apparatus 2> In the motion vector detection in the first embodiment, as shown in FIG. 5A, the horizontal search range is made equal to positive and negative (left and right) from -15 pixels to +15 pixels. I have. On the other hand, when the center of an image captured from each viewpoint in a stereoscopic image is fixed to the center of the object, the movement vector between the images is opposite to the point at the intersection of the centers of both images (the center of the object). become. The amount increases as the distance from the center of the object increases.

Therefore, the object to be imaged is almost before the center of the object, and the background is all deeper than the object and farther from the object. Therefore, the background motion vector has a larger value in the opposite direction to the object motion vector.

Therefore, the horizontal search range is as shown in FIG.
As shown in the figure, +5 pixels to -15 pixels, or +
It is assumed that the absolute value differs between positive and negative, such as 15 pixels to -5 pixels. Either of the two types is switched depending on the definition of the vector (which direction is positive), but the moving direction of the background portion is set to the wider search range.

When the background does not exist as a dark curtain or the like, the movement vector of the background can be ignored. Therefore, as shown in FIG. 5C, the larger of the vectors can be set to 0, and 0 to −5 pixels or +5 to 0 pixels. In this case, the moving direction of the object becomes wider,
The direction of widening is opposite to the case with the background.

<Stereoscopic Image Interpolating Apparatus 3> A motion vector detecting apparatus according to another embodiment of the present invention will be described below.
FIG. 6 is a configuration diagram of a stereoscopic image interpolation device that performs stereoscopic image interpolation.

Existing images input from the right image input 21 and the left image input 31 are stored in image memories 22 and 32, respectively. The left and right existing images stored in the image memories 22 and 32 are supplied to the image movers 23 and 33, and are used by the movement vector detector 28 to obtain a movement vector between the images.

The motion vector detector 28 calculates a motion vector of 4 × 4 from the left and right existing images and the viewpoint information (viewpoint distance) of the existing image given from the existing image viewpoint information input 27.
It is obtained for each block of 16 to 16 pixels. The motion vector detection processing method performed here is similar to the motion vector detection method used in motion compensation image interpolation for format conversion of a moving image.

Processing methods for format conversion are described in "Image Signal Scanning Line Converter" (JP-A-6-133280 and JP-A-6-141291) and "Video Signal Inter
polation Using Motion Estimation ”(USP-43
83272), in which images from different viewpoints are used instead of images at different times.

The movement vector obtained in this way is
The value greatly differs between the object area and the background area, and changes at different positions in the depth direction even in the object.
Basically, there is little change between blocks in the portion of the object or the background where the depth positions are continuous.

The problem in the detection of the movement vector is the changed portion, and there is no matching image between the left and right images, so that a correct movement vector cannot be obtained. However, in a stereoscopic image, it is rare that an image corresponding to an interpolated image does not exist on both the left and right sides, and usually, a corresponding image exists on at least one of them. Therefore, by correcting the movement vector and performing interpolation using only one of them, appropriate interpolation can be performed in most cases.

The motion vector obtained by the motion vector detector 28 is provided to a vector corrector 29. The vector corrector 29 holds the motion vector for one screen and compares the motion vector of the target block with the motion vector of an adjacent peripheral block. By referring to the information on the detection of the movement vector of the peripheral block, a movement vector obtained by correcting the movement vector detection result of only the target block is output.

In this comparison, a difference of about one pixel or less is ignored, and only a difference larger than one pixel is determined as a different vector. The vector is compared with four blocks in the upper, lower, left, and right or eight peripheral blocks including the diagonal direction. If all the vectors are different, it is determined to be an erroneous vector and corrected.

FIG. 7 shows the state of the movement vector. In FIG. 7, a section is a block, and an arrow indicates its movement vector. In the figure, the center is the block to be corrected, which is different from all the surrounding blocks, so that the block is corrected.

As a specific correction method, the absolute values of the motion vectors of the right block and the left block are compared, and the vector is changed to the vector having the larger absolute value. In FIG. 7, the right side is larger, so it is selected. FIG. 8 shows this state by cutting out one scanning line, and showing the right and left existing images, the interpolated images, and the movement vectors.

In FIG. 8, the blocks on the interpolated image indicate the respective blocks. In the existing image, those indicated by bold lines are the object portion and the others are the background portions. As shown in the figure,
In many cases, at the end of the portion (background image) where the movement amount is large, an image suitable for interpolation exists only on one of the left and right sides. Therefore, it is understood that it is appropriate to correct the movement vector to the vector on the right. The movement vector corrected in this way is output to the next movement amount calculator 30.

On the other hand, when the movement vector is corrected, only one of the right and left is used for interpolation, and the determination is also made by the vector corrector 29. This is determined by the direction of the block of the selected vector and the sign of the horizontal component of the vector, and selects an image in a direction in which the correction vector does not approach (intersect) the vector of the backward block.

That is, if the correction vector selected in FIG. 8 is defined as positive and the vector is defined, if the selected vector is rightward and positive, the image is a right image. In FIG. 8, the vector direction of the solid line is valid, and the direction indicated by the broken line is not used for interpolation.

Similarly, if it is negative in the right direction, it is a left image, if it is positive in the left direction, it is a left image, and if it is negative in the left direction, it is a right image. As a result, the weight coefficient k for controlling the multipliers 24 and 34 is determined and output. That is, when the motion vector is not corrected, k is set to 0.5. When the motion vector is corrected, k = 1 for the right image selection and k = 0 for the left image selection.
And

The moving amount calculator 30 calculates the moving amount of the right image and the left image from the interpolated image viewpoint information and the moving vector information. The relative position (angle) between the existing image and the interpolated image is given from the viewpoint information input 35. The movement vector information is one for the input existing image regardless of the viewpoint, but the movement amount changes depending on the viewpoint.

If the movement vector is (x, y) and the distance between the viewpoint of the interpolated image and the right image is a and the distance between the viewpoint of the interpolated image and the left image is b, the movement of the right image (x ', y') X ′ = xxa / (a + b), y ′ = y × a / (a + b)
Becomes

The image movers 23 and 33 obtain an image from the image memory for each block, move the image by the amount of movement given by the amount of movement calculator 30, and provide outputs to the multipliers 24 and 34. The multipliers 24 and 34 provide a coefficient k and a coefficient (1-k) given from the vector corrector 29 to the supplied image signal.
Multiply by

The respective images weighted by the multipliers 24 and 34 are added by the adder 25 and output as an interpolated image from the interpolated image output terminal 26. The viewpoint information is sequentially changed as necessary, and an interpolated image corresponding to the viewpoint information is obtained.
If the viewpoint of the existing image is different by 10 degrees, it is changed to 10
When the interpolated images of the divided viewpoints are obtained, finally, 11 images corresponding to the viewpoints for each degree are obtained.

<Stereoscopic Image Interpolation Apparatus 4> This embodiment relates to processing at the end of an image. In the interpolation of the three-dimensional image, since the image is shifted between the right existing image and the left existing image, there is an area at the end of the image that exists only in one of the images. In this case, the motion vector cannot be obtained correctly, but an image corresponding to the interpolated image exists in one of the images. Therefore, the motion vector is corrected in the same manner as in the third embodiment, and interpolation can be performed using one image. become.

The configuration of the stereoscopic image interpolation apparatus of this embodiment is the same as that of the third embodiment, except for the operation of the vector corrector 29. The vector corrector 29 determines the distance from the end of the image to a block (invalid block) in which the motion vector is determined to be erroneous, and determines the motion vector of the block among those determined to be correct (valid block). Replace with the vector value of the block closest to the end (the block adjacent to the invalid block).

The determination of a valid block or an invalid block is performed as follows. The inside of the image from the left and right ends to some extent is not affected by the image end,
The area is determined by the displacement of the image, that is, the value of the movement vector.

Specifically, a block inside the number of pixels of the horizontal maximum value of the movement vector which is a vector search range is:
Unaffected by the ends. Therefore, the movement vector of the block is used as a reference. Next, since the amount of movement of the background portion of the image is substantially determined by the value of the reference vector, blocks up to the number of pixels of the horizontal value become invalid blocks.

FIG. 9 shows the state of the movement vector.
If the block is 4 × 4 pixels and the horizontal search range is ± 10 pixels, those that are not affected by the end are shown as the reference in the figure.
It is up to the block. Assuming that the horizontal absolute value of the movement vector of the block is 6, six pixels from the end are an unusable area, and up to two blocks (eight pixels) including the boundary are invalid blocks. Since the third block next to the block is a valid block, the movement vector value of the third block is put into the invalid block.

In another block valid / invalid determination method, the difference between the motion vectors of blocks adjacent to the outside in order from the reference block is determined. If the difference is less than about one pixel, the block is determined to be a valid block. I do. All blocks outside the invalid block are invalid blocks.

In the block whose vector has been corrected in this manner, interpolation is not performed from both directions as in the third embodiment.
Switching to interpolation from one direction. Here, since one of the interpolated images protrudes from the end of the image, the inevitably existing image is used for interpolation.

Although this processing is basically performed independently for each block in the horizontal direction, the valid / invalid area and the corrected vector value itself do not change much in the vertical direction. Therefore, it is natural that addition averaging and thinning-out processing can be performed in the vertical direction.

In the third and fourth embodiments, the horizontal direction is strictly the moving direction of the viewpoint, and is usually the horizontal direction. However, when the viewpoint changes to another direction, the horizontal direction is used. Is processed.

[0071]

According to the present invention, when a movement vector, which is a shift between images required for interpolation, is obtained, block search is used to limit the search range of the movement vector to the minimum necessary. False detection of
An accurate interpolated image can be obtained.

As a result, a natural three-dimensional image can be obtained in displaying the three-dimensional image. Further, since the search range is narrow, the amount of calculation for detecting the movement vector can be reduced. In the detection of the motion vector, the erroneous interpolation tends to increase as the viewpoint of the existing image moves away. However, since the erroneous interpolation can be reduced, the viewpoint distance of the existing image can be increased, and the viewpoint (angle) of a wider range can be detected. An image is obtained, and it becomes a stereoscopic image that can be viewed from a wider range.

Further, within the same range, it is possible to reduce the number of existing images that need to be imaged, and it is possible to reduce the time required for the image pickup apparatus and the image pickup.

In the present invention, a motion vector, which is a shift between images required for interpolation, is obtained, and when the motion vector is different from the surrounding blocks, the motion vector is corrected to a neighboring block and interpolated from one of the images. Therefore, appropriate stereoscopic image interpolation can be performed even in a portion where the conventional image interpolation cannot be properly performed in a change portion of the movement vector.

In stereoscopic image interpolation, erroneous interpolation tends to increase as the viewpoint of the existing image moves away. However, since erroneous interpolation can be reduced, the viewpoint distance of the existing image can be increased, and a wider range of viewpoint (angle ) Is obtained, and it becomes a stereoscopic image that can be viewed from a wider range.

Further, within the same range, it is possible to reduce the number of existing images that need to be imaged, and to reduce the time required for the image pickup apparatus and the image pickup.

At the end of the image, there is always an area in which a motion vector cannot be obtained. However, by correcting the motion vector, interpolation can be performed there, and an image having the same pixel size as the existing image can be obtained even in the interpolated image.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an embodiment of a stereoscopic image interpolation device of the present invention.

FIG. 2 is a diagram showing how an interpolation image is created according to the present invention.

FIG. 3 is a diagram showing a correspondence relationship between stereoscopic image capturing and interpolation according to the present invention.

FIG. 4 is a diagram showing a configuration of a motion vector detector of the present invention.

FIG. 5 is a diagram showing a search range in detecting a movement vector according to the present invention.

FIG. 6 is a diagram showing a configuration of another embodiment of the stereoscopic image interpolation device of the present invention.

FIG. 7 is a diagram illustrating a state of a movement vector according to the present invention.

FIG. 8 is a diagram showing a state of correction of a movement vector and selection of an image according to the present invention.

FIG. 9 is a diagram illustrating a state of a movement vector at the end of an image according to the present invention.

[Description of Signs] 1,21 Right image input 2,12,22,32 Image memory 3,13,23,33 Image mover 4,14,24,34 Multiplier 5,25,42,47 Adder 6, 26 Interpolated image output 7,27 Existing image viewpoint information input 8,28 Movement vector detector 9,30 Movement amount calculator 10,35 Interpolated image viewpoint information input 11,31 Left image input 29 Vector corrector 41 Address generator 43 Division Unit 44 square error calculator 45 vector inverter 46 temporary vector generator 48 minimum value selector 50 moving vector output k, (1-k) weighting factor

Claims (9)

[Claims] 1. A stereoscopic image interpolating apparatus for handling a stereoscopic image using images of many viewpoints, wherein an image corresponding to a viewpoint sandwiched between viewpoints of the existing image is used by using two existing images on the left and right of different viewpoints. A moving vector detecting unit that sets a search range of a moving vector according to a viewpoint distance of the existing image when the interpolation image is formed as an interpolated image, and obtains a moving vector representing a pattern shift between the two existing images on the left and right. From the viewpoint position relationship between the interpolated image and the existing image and the motion vector, a moving amount calculating means for calculating a moving amount of the left and right existing images for interpolation in block units, and An image moving means for moving the block by the moving amount in blocks, and multiplying the moved left and right existing images by a coefficient corresponding to the positional relationship between the interpolated image and the existing image to obtain an interpolated image A three-dimensional image interpolating device. 2. A stereoscopic image interpolation method for handling a stereoscopic image using images of many viewpoints, wherein an image corresponding to a viewpoint sandwiched between viewpoints of the existing image is used by using two existing images on the left and right of different viewpoints. When creating an interpolated image by interpolation, a search range of a motion vector is set according to the viewpoint distance of the existing image, and a motion vector representing a displacement of a pattern between the two existing images on the left and right is obtained. From the viewpoint positional relationship of the existing image and the movement vector, the movement amounts of the left and right existing images for interpolation are calculated in block units, and the left and right existing images are moved in block units by the respective movement amounts. A stereoscopic image interpolation method, characterized in that the left and right existing images are multiplied by a coefficient corresponding to the positional relationship between the interpolated image and the existing image and added to obtain an interpolated image. 3. The stereoscopic image interpolation apparatus according to claim 1, wherein a search range in a horizontal direction, which is a direction in which a viewpoint changes, is set to be approximately twice or more than a search range in a vertical direction orthogonal to the direction.
A three-dimensional image interpolation device, wherein the movement vector is obtained. 4. The stereoscopic image interpolation apparatus according to claim 1, wherein a search range in a horizontal direction, which is a direction in which the viewpoint changes, is set by changing its absolute value between a leftward range and a rightward range.
A three-dimensional image interpolation device, wherein the movement vector is obtained. 5. A stereoscopic image interpolating apparatus for handling a stereoscopic image using images of many viewpoints, wherein an image corresponding to a viewpoint sandwiched between viewpoints of the existing image is used by using two existing images on the left and right of different viewpoints. When creating an interpolated image by interpolation, a motion vector representing a pattern shift between the two existing images on the left and right is obtained for each block obtained by finely dividing the image. Moving vector detecting means for outputting a moving vector obtained by correcting the moving vector detecting result of only the target block; and moving the moving vector obtained by the moving vector detecting means when the moving vector is significantly different from the peripheral block. Vector correction means for correcting a vector to a value of an adjacent block, and outputting the vector value and information of an image used for interpolation; From the viewpoint positional relationship between the interpolation image and the existing image and the movement vector, the movement amounts of the left and right existing images for interpolation are calculated in block units, and the left and right existing images are moved in block units by the respective movement amounts. For a block in which the difference between the motion vector of the target block and the motion vector of the peripheral block does not reach a predetermined value, the two existing images that have moved are added, and the difference is greater than or equal to the predetermined value. A stereoscopic image interpolating apparatus comprising an interpolated image generating means for obtaining an interpolated image using only one of the left and right blocks. 6. A stereoscopic image interpolation method for handling a stereoscopic image using images of many viewpoints, wherein an image corresponding to a viewpoint sandwiched between viewpoints of the existing image is used by using two existing images on the left and right of different viewpoints. When creating an interpolated image by interpolation, a motion vector representing a pattern shift between the two existing images on the left and right is obtained for each block obtained by finely dividing the image. Then, a motion vector obtained by adding a correction to the motion vector detection result of only the target block is output. If the motion vector is significantly different from the surrounding blocks, the motion vector is corrected to a value of an adjacent block, and the vector is corrected. Output values and information of an image to be used for interpolation, and determine the left and right for interpolation based on the viewpoint positional relationship between the interpolated image and the existing image and the movement vector. Is calculated in block units, the left and right existing images are moved in block units by the respective movement amounts, and the difference between the movement vector of the target block and the movement vector of the peripheral block reaches a predetermined value. For blocks that do not, the two existing left and right images that have moved are added, and for blocks where the difference is greater than or equal to the predetermined value, an interpolated image is obtained using only one of the left and right sides. Stereoscopic image interpolation method. 7. The stereoscopic image interpolating apparatus according to claim 5, wherein, when correcting the movement vector detection result of only the target block, moving the left or right adjacent block in the direction in which the viewpoint changes. A stereoscopic image interpolating apparatus comprising a vector correcting means for selecting a vector having a larger absolute value among vectors. 8. The stereoscopic image interpolating apparatus according to claim 5, wherein, when interpolating from only one of the left and right neighboring blocks in the direction in which the viewpoint changes, A stereoscopic image interpolating device comprising an interpolated image creating means for determining a direction of an existing image to be interpolated based on which block having a large absolute value is left or right. 9. The stereoscopic image interpolating apparatus according to claim 5, wherein a block within a range of a horizontal maximum value of a motion vector search range from a left or right end in an image is compared with an adjacent block motion vector. A stereoscopic image interpolating apparatus comprising a vector correcting means for correcting the moving vector to the value of the adjacent block when the moving vector differs from the moving vector of the adjacent block by a predetermined value or more.