JPH0749944A - Method and device for image processing - Google Patents

Abstract

PURPOSE:To provide the image processor by which an observer can move smoothly a visual point. CONSTITUTION:Images inputted by image input parts 1 and 2 are inputted to an image correcting part 3. In this part, a correction processing such as uneven sensitivity, etc., is executed. The images to which the correction processing is executed by the image correcting part 3 are inputted to an image interpolation processing part 4, and in this part, an image obtained from a visual point being different from two of inputted images is generated by an interpolation processing. The image outputted from the image interpolation processing part 4 is inputted to an image display part 5. In this part, suitable two in the inputted images are handled as a stereo pair, and these two of images are displayed, while switching them alternately at a high speed. In this case, an observer looks at a stereoscopic image by observing it by using shutter spectacles, etc., synchronized therewith.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method and apparatus for performing image processing for stereoscopically displaying images taken from a plurality of viewpoints by a plurality of image pickup means (cameras).

[0002]

2. Description of the Related Art Conventionally, as a device for stereoscopically displaying an image viewed from a plurality of viewpoints, there are a stereo display and a lenticular display. 2 stereo displays
The images from the two cameras are alternately switched at high speed and displayed, and the observer can stereoscopically observe the images by using shutter glasses or polarizing glasses synchronized with the images. Further, in the lenticular display, for example, when images from four cameras are denoted as A, B, C, and D, and the pixel position (1,1) of A is denoted as A (1,1), as illustrated in FIG. By arranging A, B, C, and D on a pixel-by-pixel basis and attaching a lenticular sheet to the front surface, it is possible to three-dimensionally express images from four viewpoints (Electronics Theory C, Vol. 112, No. 5). , 1992, p.
p. 281-282, JP-A-3-269680).

[0003]

However, in the above-mentioned conventional example, only the stereoscopic image in the direction aimed by the camera can be observed. That is, in a stereo display, when two cameras are fixed and an object is photographed, the image that can be seen is the same even if the observer moves his or her viewpoint. Also, in the lenticular display, the viewpoint of the observer can be moved, but this is to see the images seen from any of the multiple cameras in a sparse manner, and the viewpoint cannot be continuously moved. If you try to display them continuously, the number of cameras will increase dramatically, which is not practical.

The present invention has been made in view of the above problems, and an observer can smoothly see the viewpoint by generating images of N viewpoints (M <N) from images of M image pickup means (cameras). It is an object of the present invention to provide an image processing method and apparatus that can be moved to another location.

[0005]

An image processing method according to the present invention comprises an image inputting step of inputting images photographed from a plurality of viewpoints, and a corresponding point detecting step of detecting corresponding points between the input images. An image interpolation step of generating an image from a viewpoint different from the viewpoint of the input image by performing interpolation using the input image and the detected corresponding points.

The image input process is a process of inputting images from a plurality of cameras and a process of inputting images from an image database.

Corresponding points between the input images can be detected on the epipolar plane, and detection of the corresponding points can be replaced with straight line detection on the epipolar plane.

Image interpolation is preferably performed for each epipolar plane of the input image.

The image processing apparatus of the present invention includes image input means for inputting images photographed from a plurality of viewpoints, corresponding point detecting means for detecting corresponding points between these input images, the input image and detected images. Image interpolation means is provided for generating an image from a viewpoint different from the viewpoint of the input image by performing interpolation using the corresponding points.

The image input means can be one that can input images from a plurality of cameras or one that can input images from an image database.

Corresponding points between the input images can be obtained by detecting a straight line on the epipolar plane.

The image interpolation is preferably performed for each epipolar plane of the input image.

[0013]

According to the present invention, with the above construction, an image obtained from a plurality of viewpoints is interpolated, and an image obtained when viewed from a viewpoint different from the input image is generated.

[0014]

Embodiments of the present invention will now be described with reference to the drawings.

[First Embodiment] FIG. 1 is a block diagram of a first embodiment of the present invention.

In FIG. 1, 1 and 2 are an image input unit for inputting an image, 3 is an image correction unit for correcting the input image, and 4 is 1 and 2 by interpolation processing from two corrected images. An image interpolation processing unit 5 that generates an image from another viewpoint is an image display unit that displays the image input from the image input units 1 and 2 and the image generated by the image interpolation processing unit 4. The image input units 1 and 2 are configured by image input devices such as SV cameras and TV cameras. The images input by the image input units 1 and 2 are input to the image correction unit 3. Here, geometric correction processing such as chromatic aberration and distortion of the lens, deviation of the optical axis, camera attitude and position, and correction processing such as uneven sensitivity of the CCD sensor are performed. In this process, calibration is performed in advance and the correction data is RO
If it is recorded in M or RAM, correction processing can be performed at high speed in the form of table reference. Further, if this correction data is obtained each time before image input, more accurate correction becomes possible.
The image that has undergone correction processing such as geometric correction and sensitivity unevenness correction by the image correction unit 3 is input to the image interpolation processing unit 4 and is obtained from a different viewpoint from the two images input here. The image is generated by the interpolation process. This processing will be described later in detail. The image output from the image interpolation processing unit 4 is input to the image display unit 5. Here, treat two of the input images as a stereo pair,
The two images are displayed while alternately switching at high speed. In this case, the observer can see a stereoscopic image by observing using shutter glasses or the like synchronized with this. In the image display unit 5, when different polarized lights are alternately applied when switching at high speed alternately, a stereoscopic image can be observed by wearing polarizing glasses corresponding to the polarized light at the time of display to the left eye and the right eye.

FIG. 2 is a block diagram of the image interpolation processing section 4.

In FIG. 2, 11 and 12 are frame memories for storing the input images corrected by the image correction processing unit 3, and 13 is a motion vector for detecting a motion vector from the images stored in the frame memories 11 and 12. A detection unit, 14 is an interpolation image estimation unit that obtains an interpolation image using the result of the motion vector detection unit 13, and 15 is a frame memory that stores the estimated interpolation image. First, the input image corrected by the image correction processing unit 3 is stored in the frame memories 11 and 12. Next, the motion vector detection unit 13 detects a motion vector (corresponding point detection) from the images stored in the frame memories 11 and 12. The flow of this processing will be described with reference to the flowchart of FIG.

First, as an initial setting, x =
1, y = 1, Nx = the image size in the x direction and Ny = the image size in the y direction are set. Next, in step S2, image data of a block of 5 × 3 pixels centering on the pixel of interest (x, y) is read from the frame memory 11, the threshold value TH is calculated, and each variable EE = TH, k = 0. , M =
Set to 0. Image data of a 5 × 3 pixel block centered on this (x, y) is represented as Ax, y. As for the images stored in the frame memories 11 and 12, the image input unit on the right side when the object is viewed from the image input unit is 1 and the image input unit on the left side is 2 as shown in FIG. It is assumed that the image data from the unit 1 is corrected and stored in the frame memory 11, and the image data from the image input unit 2 is corrected and stored in the frame memory 12. The image stored in the frame memory 11 is the image A, and the image stored in the frame memory 12 is the image B.
(See FIG. 5). FIG. 5 is a diagram for explaining the relationship between the pixel of interest and the block in the input image. Threshold TH
Is calculated by 100 × the total number of pixels in the block. However, when a block of 5 × 3 pixels cannot be secured near the edge of the image, the following processing is performed. For example, image A
When the (1,1) pixel of is a target pixel, a block of 5 × 3 pixels centering on the (1,1) pixel cannot be secured, so (1,1), (2,1), (3,1) , (2,1),
The above processing is performed with 6 pixels of (2, 2) and (2, 3) as blocks. In this case, TH = 600. Also,
If the (2,1) pixel of the image A is the target pixel, (1,
1), (2,1), (3,1), (2,1), (2
2), (2,3), (3,1), (3,2), (3
The above processing is performed with 9 pixels of 3) as a block. In this case, TH = 900. The point where exception processing occurs when the block for corresponding point search is 5 × 3 pixels is
x = 1, 2, Nx-1, Nx and y = 1, Ny. This part may be processed according to the above example. In the present embodiment, the value of the threshold value TH is 100 × the total number of pixels in the block as described above, but the value is not limited to this and may be another appropriate value.

Next, in step S3, (x +
A block of 5 × 3 pixels centered on (k, y) pixels is read out, and the following calculation is performed with this as Bx + k, y (step S4). Ek is defined as the error between Ax, y and Bx + k, y.

[0021]

[Equation 1] However, Ax, y (i, j) and Bx + k (i, j)
Indicates the value of the pixel whose pixel position in Ax, y and Bx + k, y is represented by (i, j), k is an integer from 0 to N inclusive, where N is the distance between the camera and the camera and the object. It is a value determined by the distance. For example, when the distance between the cameras is fixed, N has a large value when the distance between the camera and the object is short, and conversely, has a small value when the distance is long. N here
The description will be made assuming that the camera interval and the distance between the object and the camera are adjusted so that = 20, that is, only the object that is away from the camera by a certain distance or more is imaged.

In step S5, variables Ek and E
E is compared, and if Ek is smaller, the process proceeds to step S6, and if not, the process proceeds to step S7. Step S6
Now set EE = Ek and m = k. Step 7
Then, it is determined whether the search range k = 0 to 20 has been completed or the search has been performed up to the right edge of the image. If not, the value of k is set to "1" in step S8.
Increase and return to step S3. If so, the process proceeds to step S9, and it is determined whether the value of EE matches TH. If EE = TH, it is determined that the point corresponding to (x, y) of image A does not exist on image B. EE again
= Ek, the point corresponding to (x, y) of image A is image B
It is determined to be (x + k, y). However, when a plurality of Ek's are applicable, the youngest one of the values of k is adopted. If the value of EE matches TH, the process proceeds to step S10, and the value of m is estimated from m of surrounding pixels. Regarding this estimation method, for example, there is a method of taking the average value of m of surrounding pixels or taking the value of m that occurs most frequently. If they do not match, the process proceeds to step S11 to store the values of x, y and m in a work memory (not shown). Then, the process proceeds to step S12, where it is determined whether or not the processing has been completed up to the final pixel in the x direction. If not, step S13
At x, the value of x is increased by "1" and the process returns to step S2. If the processing is completed up to the final pixel in the x direction, the process proceeds to step S14, and it is determined whether the processing is completed up to the final line in the y direction. Step S if not done yet
At 15, the value of y is increased by "1", the value of x is set at "1", and the process returns to step S2. When the processing is completed up to the final line in the y direction, the corresponding point search processing ends. However, with respect to the rightmost 19 pixels of the image A, the search range of the image B (the range of values that can be taken by k) cannot be 20 pixels. Therefore, in this case, the processing is performed within the range. If the corresponding point cannot be detected within the searchable range, the average value of the surrounding corresponding points (motion vector) (however, only the average value in the x-axis direction is used) is set as the corresponding point of this point. Alternatively, the value (m above) most frequently occurring in the surroundings is set as the corresponding point.

The interpolated image estimation unit 14 interpolates an image viewed from an arbitrary viewpoint according to the result of the motion vector detection unit 13. The method at this time will be described. 6 to 9 are diagrams for explaining the operation of the interpolation image estimation unit 14.
FIG. 6 shows a corrected image obtained from the image input units 1 and 2 (the brightness value of the white background in the image is 2 for the sake of simplicity).
55, and the luminance value of the shaded area is 100).
Table 1 shows the result of motion vector detection performed on this image.
Shown in.

[0024]

[Table 1] Here, in order to simplify the description, the 5 × 3 pixel block described in the motion vector detection unit 13 will be changed to a 3 × 1 pixel block, and the description will proceed. Pixels not shown in Table 1 have no motion. First, the area of the image A in the table (Table 1) is searched to extract a point having no motion and copied to the frame memory 15. This is shown in FIG.

The shaded portion in FIG. 7 is the point extracted as a stationary point. The luminance value of the shaded portion is the same as the luminance value of the point at the same coordinate in the image A. Next, interpolation is performed on the points recorded in the table (Table 1). Here, the images viewed from two viewpoints when the line of sight between the image A and the image B is divided into three equal parts are interpolated. That is, FIG. 8 shows this. A1 and B1 in FIG. 8 represent the positions (optical axis) of the cameras that photographed images A and B, respectively, and H1 and H2 represent the positions (optical axis) of the virtual cameras when it is assumed that the interpolated images are virtually photographed, respectively. ) Is represented. When interpolating the image seen from the position of H1,
The point of the image A is a and the corresponding point of the image B is b. The point that internally divides point a and point b in 1: 2 is c, and point c
The luminance value of is the same as the luminance value of the point a. Here, the luminance value of the point c is set to be the same as the luminance value of the point a, but when the luminance values of the points a and b are different, the luminance value of c may be determined according to the ratio of the internal division. For example, c ′ = (R1 · b ′ + R2 ·
a ') / (R1 + R2). However, a ', b',
c'represents the brightness values of a, b and c, respectively, and R
1, R2 indicates the ratio when point a and point b are internally divided into R1: R2.

This operation is performed for all points in the table (Table 1). This process is performed and the data is stored in the frame memory 15. This state is shown in FIG. An image 51, which is a shaded portion in FIG. 9, is a set of points that do not move.
The image 52 (luminance value 100), which is the shaded area, and the image 53 (luminance value 255), which is the second shaded area, are a set of points obtained by internal division. Next, the white background portion of FIG. 9 is interpolated. The white part is the part where the object is moved and the background is exposed. Therefore, of the points of the image A and the image B in FIG. 6, the point having the same coordinates as the point on the white background is examined, and the point having no motion is adopted as the interpolated point and stored in the frame memory 15. For example, the points (2, 3), (2, 3) on the white background in FIG.
Since points having the same coordinates as 4) and having no motion are points (2, 3) and (2, 4) in the image B, the brightness values of these points are respectively set to the points (2, 3) of the interpolated image. The luminance values of 3) and (2, 4) are used. In addition, points (7,3), (7,
4), (8, 3), and (8, 4) having the same coordinates as the non-moving points are (7, 3), (7, 4), (8,
3) and (8, 4), the point (7,
The luminance values of 3), (7,4), (8,3), and (8,4) are points (7,3), (7,4), and (8,4) in image A, respectively.
3) and luminance values of (8, 4). By the above processing, an interpolated image as viewed from H1 of FIG. 8 can be virtually obtained. The interpolation image viewed from H2 can also be obtained by similar processing.

At the time of display, a stereoscopic image viewed from a plurality of viewpoints can be displayed by an appropriate combination of the images obtained from the image input units 1 and 2 and the image obtained by interpolation.

By performing the processing with the above configuration, even when using a small number of image input devices, it is possible to obtain the same image as when using a larger number of image input devices.
For example, it becomes possible to observe a stereoscopic image by changing the viewpoint even for images from two cameras. Further, since the image input unit can be made compact, the entire structure can be made smaller accordingly.

[Second Embodiment] In the motion vector detection unit 13 of the first embodiment, by weighting the mask (5 × 3 pixel block) to be used, it is possible to search for corresponding points of locally changing portions. You will be able to respond. As a method of weighting, a method of changing the weight between the line in which the pixel of interest exists and another line, and a method of decreasing the weight as the pixel is away from the pixel of interest can be considered. In addition, the size of the mask to be used is small in a complicated image and large in a simple image according to the type of image, so that the accuracy of the corresponding point search can be improved.

Further, although the image interpolation processing section 14 has explained the processing method on the assumption of the frame memory, it is obvious that this processing can be successively performed pixel by pixel. Further, the motion vector search (corresponding point search) and the interpolation process can be performed in parallel, and the processing speed can be improved by adopting a hardware configuration that simultaneously processes a plurality of lines.

[Third Embodiment] When there are three or more cameras as the image input means, the accuracy of the corresponding point search (motion vector detection) can be improved. FIG. 10 shows an example of the case where four cameras are used for the image input unit according to the third embodiment of the present invention.
It is a block diagram of an Example. In FIG. 10, 20-2
3 is a camera as an image input unit, 24 is an input port, 25
Is a CPU, 26 is a RAM, 27 is a ROM, 28 is a disk I / O port, 29 is a hard disk, and 30 is a VRA.
M and 31 are video signal output I / Fs, and 32 is a lenticular display. The cameras 20 to 23 are the same as the image input units 1 and 2 in FIG. Although the four cameras 20 to 23 are used for description here, the number of cameras is not limited to this, and any number of cameras may be used.

FIG. 11 is a flow chart showing the flow of processing of the third embodiment. The original image input from an image input device such as a camera is first processed in step S16 in the same manner as the image correction processing unit 2 shown in the first embodiment, that is, the chromatic aberration and distortion of the lens, the shift of the optical axis, the camera Posture
Geometrical correction processing such as position and correction processing such as sensitivity unevenness of the CCD sensor are performed. In this process, calibration is performed in advance and the correction data is stored in the ROM or R
If it is recorded in the AM, the correction process can be performed at high speed in the form of table reference. Further, if this correction data is obtained each time before image input, more accurate correction becomes possible. When these correction processes are completed, the process proceeds to step S17, and a corresponding point search (motion vector detection) between each image is performed. When the corresponding point search is completed, the process proceeds to step S18 to perform image interpolation processing, and then, in step S19, the corrected input image and the interpolated image are displayed on the lenticular display.
FIG. 12 is a flowchart of the corresponding point search process in step S17.

In step S21, the raster of interest is set as the first raster of each image as an initial setting. Next, in step S22, the raster of interest of each image is read into the work memory to virtually form the first epipolar plane. The j-th epipolar plane here is
As shown in FIG. 13, each point EP _j (x, i) on the image plane
Is a set of points EP _j (x, i) satisfying EP _j (x, i) = Ni (x, j). However, Ni (x, j) is represented by the x-th pixel value in the j-th line of the i-th image (here, i = 1 to 4), that is, the coordinates in the i-th image are represented by (x, j). Represents the value of the pixel. When the input devices (cameras) are installed in parallel at equal intervals, all the corresponding points are lined up on the epipolar plane. Therefore, the image interpolation should be performed on this straight line. Therefore, in step S23, a straight line having a corresponding point is extracted, and in step S24, the corresponding point is calculated from the obtained straight line and stored. The specific algorithm is shown below.

Process A1. With EP _j (x, 1) as the pixel of interest, within the range of m = 0 to k1,

[0035]

[Equation 2] Find all m that satisfy However, TH2 is a threshold value for finding a corresponding point, and here, 1600 (= 4
It is set to × 20 ² ). Further, k1 is a value determined by the camera interval and the distance to the object, and is set to 20 here (that is, it is assumed that 20 pixels or more do not move).

Process A2. The above processing A1 is repeated for all x of x = 1 to nx, and all the values of m corresponding to the values of x are held. However, nx represents the number of pixels in the main scanning direction of the image. If EP _j ((x + m × −1), i) does not exist, it is determined that the corresponding point in m does not exist and the process is continued.

Process A3. The corresponding point of priority 1 is obtained from the straight line of the slope m obtained by the above processes A1 and A2,
Store in memory. When a plurality of corresponding points are obtained, all of them are stored as corresponding points of priority 1 for convenience. Pixels found as corresponding points are processed pixels.

Process A4. The above processing A1, A2, A3 is 1
As a cycle, the above cycle is repeated for pixels that have not been processed. In process A1, EP _j (x + m ×
If (i-1), i) have been processed, EP _j (x + mx)
(I-1), i) -EP _j (x, 1) = 0 and the processing is continued. If the corresponding point obtained from the straight line of the slope m in the process A3 has already been processed, this point is excluded from the corresponding points. The corresponding points obtained in the nth cycle are stored as the corresponding points of the priority order n.

Process A5. If the number of unprocessed pixels does not decrease even after performing the process A4, the same process as the processes A1 to A4 is performed with EP _j (x, 2) as the target pixel. However, x = 1 to nX.

Process A6. If the number of unprocessed pixels does not decrease even after performing the process A5, the same process as the processes A1 to A4 is performed with EP _j (x, 3) as the target pixel. However, x = 1 to nX.

Process A7. The value of j is incremented by "1" and the process returns to A1.

Process A8. When the processing is completed up to the final raster, the corresponding point search processing is ended.

By performing the processing as described above, it is possible to detect the corresponding points which cannot be found from the two images, and it is also possible to deal with occlusion and the like, so that the accuracy of the corresponding point search is improved.

Next, in step S18, an image interpolation process is performed. The image interpolation processing is performed using the corresponding points obtained in step S17. Figure 1 shows the concrete algorithm.
4 and FIG. 15 as an example.

FIG. 14 shows the j-th epipolar plane. a1 and b1 indicate corresponding points of priority 1, c
2 indicates the corresponding point of priority 2. Consider a case where n images are interpolated at equal intervals between input images. Here, n = 2 is set to simplify the description. When this is considered for the j-th epipolar plane, two lines are interpolated between the lines of the epipolar plane as shown in FIG. 15, and the interpolation is on the straight line connecting the corresponding points of the epipolar plane of the original image. The pixel value of the selected line may be set to the average value of corresponding points. That is, the following processing B1 to
Perform B5.

Process B1. Consider a straight line connecting corresponding points of priority 1, and set the pixel value of the interpolation line on this straight line to the average value of the pixel values of the original image on the straight line. Taking the corresponding points a1 and b1 in FIG. 15 as an example, the pixel values of points a and b on the straight line connecting the corresponding points are a1 and b, respectively.
The average value of the pixel values shown by 1 is taken.

Process B2. After the processing of the corresponding points of priority 1 is completed, the processing of the corresponding points of priority 2 is performed next. This process is basically the same as the process B1, but the process B1
In, no processing is performed on the pixels that have already been interpolated. This will be described with reference to FIG. Pixels (3,8) and (2,9) are normally interpolated by the corresponding point c2, but since they are already interpolated by the corresponding point a1 of priority 1, no processing is performed on this pixel. Not performed. Therefore, the pixel interpolated by the corresponding point c2 is (9, 2),
The four pixels are (8, 3), (6, 5), and (5, 6).
In the example of FIG. 15, occlusion occurs in this portion, but the problem of occlusion can be solved by performing the processing in this way.

Process B3. When the processing of the corresponding points of priority 2 is completed, the processing of the corresponding points of priority 3 is started next.
Similar to the process B2, nothing is performed on the pixels which have already undergone the interpolation process. In the same manner, processing is performed up to the corresponding point of the final priority order.

Process B4. For pixels that have not been interpolated even after the processes B1 to B3 have been completed, interpolation is performed from surrounding pixels. At this time, there are a method of using an average value of surrounding pixel values, a method of directly using the value of the nearest pixel, and the like.

Process B5. Process B for j = 1 to ny
1 to B4 are processed and lines j2, j3, j5, j
An interpolated image is obtained using 6, j8 and j9. However,
As shown in FIG. 5, the lines interpolated by the processes B1 to B4 are represented as j2, j3, j5, j6, j8, j9. For example, the interpolation image 2 has an interpolation line j2 (j = 1 to n).
It can be configured by arranging y) (see FIG. 16).
The same applies to the interpolated images 3, 5, 6, 8, and 9.

By generating an interpolated image from a large number of input images using the configuration and method described above, it is possible to detect corresponding points that could not be obtained from two images, and Accuracy is improved. Moreover, since corresponding points are obtained from a large number of images, problems such as occlusion can be solved as described above.

[Fourth Embodiment] As a three-dimensional display method,
There is holography. This is a method of displaying a stereoscopic image by irradiating the interference fringes generated by irradiating the object light of the object with the laser light (reference light) with the reference light again. At this time, if the laser beam cannot be applied or the object is large, a method called holographic stereogram can be used. In this method, a large number of photographs are taken while changing the viewpoint little by little, each photograph is projected on a transmission diffusion plate, etc., the projection surface is irradiated with reference light to generate interference fringes, and the generated interference fringes are generated. This is a method in which a three-dimensional image is obtained by cutting out and combining them into one interference fringe, and irradiating this interference fringe with reference light. For details, refer to pages 191 to 207 of “Holographic Display” (Sangyo Tosho) by Junpei Tsujiuchi.

In the method of holographic stereogram, a large number of cameras are required because a large number of images photographed by changing the viewpoint little by little are required. In the case of a still image, a method of moving one camera little by little can be used, but in the case of a moving image, many cameras have to be used. This problem can be solved by using the interpolation image of the present invention. That is, an image obtained from a viewpoint between the cameras is generated by the interpolation of the present invention using images obtained from several cameras to several tens of cameras as an input image. This method has the advantage that the number of cameras can be small. Considering a television using a holographic stereogram, there is an advantage that data transmission is achieved by transmitting several to several tens of images and generating an interpolated image on the receiving side. If an image compression method is used when transmitting the input image, the data can be further compressed.

[0054]

As described above, according to the present invention, the images obtained from a plurality of viewpoints are interpolated, and an image obtained from a viewpoint different from the input image is generated to obtain the input image. Images from more viewpoints can be obtained, so viewpoints can be moved smoothly when displaying in 3D,
By increasing the number of viewpoints, the reality of the stereoscopic image is improved. Moreover, since the number of image pickup means (cameras) for input can be reduced, the apparatus can be made compact.

By using the present invention to generate an interpolated image of multi-viewpoint images (images photographed by shifting viewpoints little by little) stored in a database or the like, a lenticular display or a holographic display having a larger number of viewpoints can be obtained. Therefore, it is possible to obtain a more realistic stereoscopic image, and at the same time, the number of viewpoints is increased, so that smooth viewpoint movement is possible.

[Brief description of drawings]

FIG. 1 is a block diagram of a first embodiment of the present invention.

FIG. 2 is a block diagram of an image interpolation processing unit 4.

FIG. 3 is a flowchart showing an operation of a motion vector detection unit 13.

FIG. 4 is a diagram showing a positional relationship between an object and a camera.

FIG. 5 is a diagram for explaining a relationship between a pixel of interest and a block in an input image.

FIG. 6 is a diagram for explaining the operation of the interpolated image estimation unit 14.

FIG. 7 is a diagram for explaining the operation of the interpolation image estimation unit 14.

FIG. 8 is a diagram for explaining the operation of the interpolated image estimation unit 14.

FIG. 9 is a diagram for explaining the operation of the interpolated image estimation unit 14.

FIG. 10 is a block diagram of a third embodiment of the present invention.

FIG. 11 is a flowchart showing a processing flow of the third embodiment.

FIG. 12 is a flowchart showing the flow of corresponding point search processing.

FIG. 13 is a diagram for explaining an epipolar plane.

FIG. 14 is a diagram for explaining an interpolation process on an epipolar plane.

FIG. 15 is a diagram for explaining an interpolation process on an epipolar plane.

FIG. 16 is a diagram for explaining the configuration of an interpolated image.

FIG. 17 is a diagram for explaining a lenticular display.

[Explanation of symbols]

 1, 2 image input unit 3 image correction processing unit 4 image interpolation processing unit 5 stereo display 11, 12, 15 frame memory 13 motion vector detection unit 14 interpolation image estimation unit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Office reference number FI technical display location H04N 13/04

Claims (10)

[Claims] 1. An image inputting step of inputting images photographed from a plurality of viewpoints, a corresponding point detecting step of detecting corresponding points between these input images, and an interpolation using the input image and the detected corresponding points. And an image interpolation step of generating an image from a viewpoint different from the viewpoint of the input image. 2. The image processing method according to claim 1, wherein the image input step inputs images from a plurality of cameras. 3. The image input step is to input an image from an image database.
The described image processing method. 4. The image processing method according to claim 1, wherein corresponding points between the input images are detected on the epipolar plane. 5. The image processing method according to claim 1, wherein the image interpolation is performed for each epipolar plane of the input image. 6. Image input means for inputting images photographed from a plurality of viewpoints, corresponding point detection means for detecting corresponding points between these input images, and interpolation using the input image and the detected corresponding points. And an image interpolating unit for generating an image from a viewpoint different from the viewpoint of the input image. 7. The image processing apparatus according to claim 6, wherein the image input means can input images from a plurality of cameras. 8. The image processing apparatus according to claim 6, wherein the image input means can input an image from an image database. 9. The image processing apparatus according to claim 6, wherein corresponding points between the input images are detected on the epipolar plane. 10. The image processing apparatus according to claim 6, wherein the image interpolation is performed for each epipolar plane of the input image.