JPH07296194A - Method and device for image processing - Google Patents

Abstract

PURPOSE:To geneate an interpolated image with high accuracy based on multiple viewpoint images. CONSTITUTION:An epipola plane image (EPI) is generated by arranging lines on the same position in sequence of a viewpoint based on images from plural viewpoints. The line to be interpolated is inserted to each line, and corresponding points a1, b1, and c1 are detected by detecting a straight line. At this time, a score is given to each picture element through which the straight line passes corresponding to the number of straight lines detected from remarked picture elements, and as for the picture element in which the correspondence of a picture element which a low score and one with a high score is overlapped, the correspondence with the low point is eliminated. The point on the interpolated line of the straight line consisting of those points is set as the point generated by interpolation. However, the interpolation is performed preferentially starting from the correspondence with the high score by performing interpolation according to the correspondence with the low score in advance and then the interpolated image can be generated.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method for generating an image of a viewpoint different from that of the image pickup means by interpolating images obtained from a plurality of image pickup means (cameras or the like). is there.

[0002]

2. Description of the Related Art Conventionally, a method of using an epipolar plane image (hereinafter abbreviated as EPI) has been used as a method of generating an image captured from a position different from the captured position based on images captured from a plurality of viewpoint positions. is there.

This method extracts lines at the same position in each image as shown in FIG.
(A)) One EPI is combined (Fig. 16 (b)), and a line corresponding to the point of interest (corresponding point) is obtained by performing straight line detection on this EPI (Fig. 16 (c)), and imaging is performed. An image obtained when the image is captured from a position different from the position (FIG. 16D) is generated.

[0004]

However, in the above-mentioned conventional example, an object of uniform color such as a background or a wall is
When straight line detection was performed on I, innumerable straight lines were detected and the corresponding points could not be specified. Therefore, when a small object is placed in front of such an object of uniform color, a straight line is detected on the EPI by overtaking the corresponding point of the small object (that is, the context of the object is reversed. The problem was that

The present invention has been made in view of the above-mentioned conventional example, and an object of the present invention is to provide an image processing method and apparatus for accurately generating an interpolated image regardless of the contents of the image.

[0006]

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and when detecting a corresponding point, the number of straight lines detected from a certain point of interest is used as a corresponding point. A straight line is detected by expressing certainty. That is, in the image processing method of the present invention, the multi-viewpoint image input step of inputting images shot from a plurality of positions and the corresponding points by dividing the input multi-viewpoint image data into EPIs and detecting straight lines on the EPIs. And a image interpolation step of generating an image that should be obtained when shooting from different positions using the detected straight line.
In the above configuration, the multi-viewpoint image input step inputs a large number of images obtained from one or more cameras or a multi-viewpoint image (images taken from a plurality of different viewpoints) accumulated in the database, and the straight line detection step While dividing the input multi-viewpoint image data into EPI, a straight line is detected in order to find a corresponding point on the EPI, and the image interpolation process is performed from different viewpoints based on the corresponding point obtained by the straight line detecting process. The image of the case is generated by the interpolation processing.

As another aspect, the image processing method of the present invention has the following configuration.

An image inputting step of inputting a plurality of images corresponding to viewpoints at a plurality of positions, a candidate detecting step of detecting a corresponding pixel candidate between the plurality of images input by the image inputting step, and the corresponding pixel A checking step of checking the likelihood of correspondence with respect to the candidate, and a corresponding detection step of detecting a corresponding pixel from the candidates of the corresponding pixel detected by the candidate detecting step, according to the certainty of the correspondence. And an image interpolation step of generating an image corresponding to a viewpoint different from the viewpoint of the image input in the image input step based on the relationship between the corresponding pixels detected in the correspondence detection step.

As another aspect, the image processing method of the present invention has the following configuration.

An image input step of inputting a plurality of images corresponding to viewpoints at a plurality of positions, and a straight line detection of corresponding pixel candidates between the plurality of images input by the image input step on the epipolar plane image. The candidate detection step of detecting by the above, and a score indicating the certainty of correspondence to the corresponding pixel candidate is given to each pixel according to the number of straight lines detected by the detection means passing through each pixel. From the corresponding pixel candidates detected by the applying step and the candidate detecting step, a pixel candidate having a corresponding pixel having a more corresponding relationship is excluded from corresponding points of pixels having a corresponding pixel having a more uncertain corresponding relationship. A corresponding detection step of detecting a corresponding pixel, and generating an interpolation point on a straight line on the epipolar plane image passing between the corresponding pixels detected by the corresponding detection step, By combining the interpolation point for all epipolar plane images, and an image interpolation step that generates an image corresponding to the different viewpoints from the viewpoint of the image input by the SL image input step.

Further, in the image processing apparatus of the present invention,
Multi-viewpoint image input means for inputting images photographed from a plurality of positions, straight-line detecting means for decomposing the input multi-viewpoint image data into EPI and detecting straight lines on the EPI, and detected straight lines And an image interpolating means for generating an image that should be obtained when images are taken from different positions by using. In the above structure, the multi-view image input means inputs a large number of images obtained from one or more cameras or multi-view images (images taken from a plurality of different viewpoints) accumulated in the database, and the straight line detection means While dividing the input multi-viewpoint image data into EPI, a straight line is detected in order to obtain a corresponding point on the EPI, and the image interpolating means is viewed from different viewpoints based on the corresponding point obtained by the straight line detecting means. The image of the case is generated by the interpolation processing.

As another aspect, the image processing apparatus of the present invention has the following configuration.

Image input means for inputting a plurality of images corresponding to viewpoints at a plurality of positions, candidate detection means for detecting corresponding pixel candidates among the plurality of images input by the image input means, and the corresponding pixels Check means for checking the probability of correspondence of the candidate, and correspondence detection means for detecting a corresponding pixel from the candidate of the corresponding pixel detected by the candidate detecting means, according to the probability of correspondence. And image interpolation means for generating an image corresponding to a viewpoint different from the viewpoint of the image input by the image input means, based on the relationship between the corresponding pixels detected by the correspondence detection means.

As another aspect, the image processing apparatus of the present invention has the following configuration.

Straight line detection on the epipolar plane image of image input means for inputting a plurality of images corresponding to viewpoints at a plurality of positions and pixel candidates corresponding between the plurality of images input by the image input means. The candidate detection means for detecting by the above and the score indicating the certainty of correspondence to the candidate of the corresponding pixel are given to each pixel according to the number of straight lines detected by the detection means passing through each pixel. From the assigning means and the corresponding pixel candidates detected by the candidate detecting means, a pixel candidate having a corresponding pixel having a more corresponding relationship is excluded from corresponding points of pixels having a corresponding pixel having a more uncertain corresponding relationship. A corresponding detection means for detecting the corresponding pixel and an interpolation point on a straight line on the epipolar plane image passing between the corresponding pixels detected by the corresponding detection means, By combining the interpolation point for all epipolar plane images, and an image interpolation unit that generates an image corresponding to the different viewpoints from the viewpoint of the image input by the SL image input means.

[0016]

With the above structure, the correspondence between the images of different viewpoints is detected while checking the certainty of the correspondence, and the interpolated image based on the correspondence is generated.

[0017]

First Embodiment Next, a first embodiment of the present invention will be described with reference to the drawings.

<Device Configuration> FIG. 1 shows an example of an image processing device in which four cameras are used as an image input unit. In the figure, the image input unit 33 includes four cameras 20 to 23,
The captured image as a digital image signal is input port 24
To the device via. The CPU 25 is a processor that controls the entire image processing apparatus, and is a ROM 27 or R
A predetermined control procedure is executed by executing the program stored in the AM 26. The RAM 26 and the ROM 27 store data as well as programs. RAM2
6 also stores digital image data input from the image input unit 33. The hard disk 29 is the disk I
CPU 25 or RAM 26 via the / O port 28
Exchange data etc. Image data is disk 29
Also stored in. The captured image data or the processed image data is expanded in the VRAM 30, and the stereo display 32 is output via the video signal output I / F 31.
Display output from 0. It should be noted that although the description is given using four cameras here, the number of cameras is not limited to this, and
Any number is acceptable.

<Image Interpolation Processing> FIG. 2 is a flow chart showing the flow of image interpolation processing by the image processing apparatus of the first embodiment.

In step S21, an original image input from an image input device such as a camera is first subjected to geometrical correction processing such as chromatic aberration and distortion of a lens, deviation of an optical axis, camera posture and position, and a CCD sensor. Correction processing for uneven sensitivity is performed. If this process is calibrated in advance and the correction data is recorded in the ROM or RAM, the correction process can be performed at high speed in the form of table reference. Further, if this correction data is obtained each time before image input, accurate correction can be performed. Further, when this correction process is completed, the process proceeds to step S22, and the range for performing the corresponding point search is determined from the shooting parameters. Using the result, the process proceeds to S23 and the corresponding points between the images are searched. When the corresponding point search is completed, the process proceeds to step S24 to perform image interpolation processing, and then, in step S25, the corrected input image and the interpolated image are displayed on the lenticular display.

Next, the processing of each unit will be described in detail.

<Calculation of Corresponding Point Search Range> First, the calculation processing of the corresponding point search range will be described with reference to FIG.

FIG. 3 is a diagram showing the principle of calculating the corresponding point search range. S101 is the rearmost surface of the space you want to shoot, S
102 is the photographing space Ω, S103 is the frontmost surface of the space to be photographed, S104 is the distance to the rearmost surface of the space to be photographed (Le), and S105 is the distance to the frontmost surface of the space to be photographed. (Lf). In S106, the vertical line from the shooting position P1 (S108) to the imaging surface and the shooting position P
Shooting position P2 that is a distance d (S109) from 1
An angle α with a straight line connecting (S110) to the shooting object a (S112) on the frontmost surface of the shooting space Ω, S1
Similarly, 07 is a straight line connecting the shooting object b (S113) on the rearmost surface S101 of the shooting space Ω and the point P2,
Angle β from the viewpoint position P1 to the perpendicular to the imaging surface, S111
Is the focal length f. a '(S114), b' (S11
5) is the object a (S112), b (S11) from the point P2.
The position of 3) on the imaging surface is shown. According to the geometry of this Figure 3,
The distance between a'and b'in the imaging plane, which is the corresponding point search range, is calculated.

First, the photographing positions P1 (S108) and P2
In (S110), the moving distance d on each imaging surface of a (S112), which is the frontmost object in the imaging space Ω.
Ask for 1. It can be calculated from the geometry of FIG.
The size in the main scanning direction on the image pickup surface is represented by nx.

D1 = nx / (2tanα) = nx / (2 (d / Lf)) (1) Similarly, the rearmost object b (S113) in the shooting space Ω is the viewpoint positions P1 (S108) and P2. In (S110), the moving distance d2 on each imaging surface is obtained.

D2 = nx / (2tanβ) = nx / (2 (d / Le)) (2) From these, the inclination of the straight line search for searching the corresponding point between the image of the viewpoint P1 and the image of the viewpoint P2 Range is d1 to d
You can change it by 2.

The search step s can be obtained from the number N of cameras by the following equation.

S = 1 / N As described above, by restricting the space to be photographed to the frontmost and rearmost surfaces, it is possible to prevent the corresponding point search range from being unintentionally expanded and to speed up the search processing. Is possible. Further, by obtaining the upper limit (fineness) of the search step from the number of cameras, it is possible to prevent the search processing time from increasing due to the careless reduction of the search step.

<Corresponding Point Searching Process> Next, the corresponding point searching process using the obtained corresponding point searching parameter will be described with reference to FIG. FIG. 4 is a flowchart of the corresponding point search process in step S23. In step S31, the raster of interest is set as the first raster of each image as an initial setting.

Next, in step S32, the target raster of each image is read into the work memory to virtually form the j-th epipolar plane image (EPI). The j-th EPI referred to here is a point EPj (x, i) at which each point EPj (x, i) on the image plane satisfies EPj (x, i) = Ni (x, j) as shown in FIG. i) is a set. However, Ni (x, j) is represented by the x-th pixel value in the j-th raster of the i-th image (here, i = 1 to 4), that is, the coordinates in the i-th image are represented by (x, j). Represents the value of the pixel. If the input devices (camera, etc.) are installed in parallel at equal intervals, on this EPI,
All the corresponding points are lined up on a straight line. Therefore, the corresponding points can be detected by detecting the straight line, and
The image interpolation may be performed on the detected straight line. Therefore, corresponding points are calculated from the straight line obtained in step S34 and stored.

A concrete algorithm will be described with reference to the flowchart of FIG.

First, in step S71, the priority order n =
1, the line of the target pixel is set to r = 1. Next, the process proceeds to step S72, where EPj (x, r) is set as a target pixel, and m
= K1 to k1 + k2, and x = 1 to nx,

[Equation 1] All m satisfying the above are obtained, and all the detected straight line numbers are stored as L. However, since m can be a real number, the value of x + m × (i-1) is rounded to the corresponding x.
Determine the coordinates. TH2 is a threshold for finding a corresponding point, and is set to 1200 here. This 120
It means 0, but since it is an EPI of 4 lines, the difference is calculated 3 times, and if each difference value is about 20 or less, 3 × 20 ² = It is set to 1200. If there is no specular component in the image with an ideal input system (the points appearing in each image have the same pixel value), TH2 = 0 is acceptable, but in the actual input system, even at the same point in each image, Since the pixel values vary, the difference value is set to 20 by providing a width. Therefore, the higher the accuracy of the input system, the smaller the difference value, and in the opposite case, the difference value needs to be set larger.

The above method is applied to each of the case where the pixel value is RGB, but not only that, but the YI
It is possible to deal with the case of once converting into various color systems such as Q or HSI, and it is also possible to set and use a threshold value suitable for each. Also, k1 and k2 are step S
The parameters d1 and d2 obtained in step 22 are used (that is, k1 = d2 and k2 = d1). Also, EPj (x
If + m × (i−1), i) does not exist, the corresponding point in m is not present and the process is continued. However, in step S72, EPj (x + m × (i−
When r) and i) have been processed, EPj (x + m * (i-
r), i) -EPj (x, r) = 0 and the processing is continued.

Then, the process proceeds to step S73, and step S7
The corresponding point of the priority order n is obtained from the straight line of the gradient m obtained by 2, and is stored in the memory as (n, f (L)) together with the value of the score f (L). Here, L is the number of detected straight lines, and f () is a function showing a small value when the number of straight lines is large and a large value when the number of straight lines is small.

When a plurality of corresponding points are obtained for the pixel of interest, all of them are stored as corresponding points of priority n for convenience. Pixels found as corresponding points are processed pixels.
Here, the corresponding point obtained from the straight line of the slope m has already been processed, and the corresponding point is stored (n,
When the value of the score f (L ′) of f (L ′) is larger than the score of the pixel of interest, this point is excluded from the corresponding points. That is, for a certain point, the number of corresponding points is shown as a score, and this score is also a measure showing the certainty of the correspondence relationship regarding the target point. When two points are attempted to be associated with each other, the certainty of correspondence is determined using this scale, and when a pixel is determined to have a correspondence relationship with another pixel, that pixel already has a more probable correspondence relationship. If so, no new correspondence is made. In this way, uncertain correspondence can be reduced, and more reliable correspondence can be performed. Further, in step S73, the number of unprocessed pixels is set to w.

FIG. 17 is an example of a flowchart showing step S73 in more detail.

First, the score f (L) is calculated based on the straight line obtained in step S72 (S171). Next, one corresponding point is obtained based on the straight line (S172). If there is no corresponding point, the number of unprocessed pixels is set to w and step S7
The process of 3 is terminated (S179). If a corresponding point is found, it is determined whether the point has been processed (S174), and if not processed, it is stored as a corresponding point (S178). If processed, the score f (L ') of the corresponding point and the score f (L) of the target point are compared (S175), and f (L')>
If it is f (L), that is, if the correspondence that has already been made for the corresponding points is more certain, then the corresponding points are excluded from the corresponding points without further matching (S177). ). If f (L ') <f (L), the correspondence between the corresponding point and the target point is more likely than the correspondence already given for the corresponding point, and is registered as the corresponding point (S178). The above is repeated while finding the corresponding points. In this way, the process of step S73 ends.

Next, in step S74, it is determined whether or not the number of unprocessed pixels w is 0. If it is 0, the process in the j-th epipolar plane is ended, and if it is not 0, the process proceeds to step S75 and the line of interest is selected. Determine if r is the last line.
If it is the last line, r = 1 (leading line) is set in step S77, and if not, the value of the target line r is incremented by 1 in step S76. N in FIG. 6 indicates the number of EPI constituent lines (here, N = 4).

Next, in step S78, it is determined whether or not n indicating the priority is equal to P. P represents the complexity of a phenomenon (occlusion) in which objects conceal each other. That is, when P is a large value, the objects are overlapped many times, and when P is small, the objects are less overlapped. The setting of the value of P depends on how much occlusion is reproduced. Here, P = (N-1) × 10 = 30 is set as an empirical value. If n ≠ P in step S78, step S
Moving to 80, 1 is added to the priority order n, W is set to the value of w, and the process returns to step S72. If n = P, step S7
9, it is determined whether or not the number of unprocessed pixels is smaller than that in the previous processing, and if it is decreased, the processing proceeds to step S80, and if not, the straight line detection processing in the jEPI, that is, FIG. The step S34 is ended, the process returns to the step S36, and the process for the next EPI is started. In step S36, it is determined whether or not all rasters of the input image have been processed. If not, the value of j is incremented by 1 and the process returns to step S32. If the processing is completed for all rasters, that is, in FIG. When step S23 of is completed, the process proceeds to step S24.

By performing the processing as described above, it is possible to detect the corresponding points which cannot be found from the two images and to deal with occlusion and the like, so that the accuracy of the corresponding point search is improved. In addition, the number of straight lines detected from a certain pixel of interest is expressed as a point of certainty as the corresponding point of the pixel of interest, and the size of the point is compared to obtain a more probable corresponding point. The corresponding points can be searched without reversing the context of the small object in front of the object.

<Image Interpolation Processing> Next, in step S24, image interpolation processing is performed. The image interpolation processing is performed using the corresponding points obtained in step S23. A specific algorithm will be described with reference to FIG.

FIG. 7 shows the jth EPI. a
1 indicates a corresponding point of priority 1 and score f (L) = 1, b
1 represents the corresponding points of priority 1 and score 20, and c2 represents the corresponding points of priority 2 and score 20. Consider a case where n images are interpolated at equal intervals between input images. Here, the number of interpolation lines is set to n = 2 to simplify the description. When this is considered for the j-th EPI, as shown in FIG. 9, two lines are interpolated between the lines of the EPI and the pixels of the interpolated line on the straight line connecting the corresponding points of the original image are inserted. The value may be set to the average value of corresponding points on the straight line. This will be described below with reference to the flowchart of FIG. FIG. 8 is a flowchart showing details of step S24.

First, in step S81, initial setting for image interpolation processing is performed. Set j = 1 and pay attention E
Set PI to jth EPI. In addition, the priority order n
= P. Here, P is the same as P (P = 30) used in step S78.

Next, in step S82, a straight line connecting the corresponding points of the priority n on the j-th EPI is considered, and the pixel values of the interpolation line on this straight line are averaged of the pixel values of the original image on the straight line. Set to the value. However, even if a value has already been assigned as a corresponding point in the interpolated line, if the score of another corresponding point is high, the pixel value of that interpolation line can be overwritten, and if the score is low. Cannot be overwritten. Taking the corresponding points a1 and b1 in FIG. 9 as an example, the pixel values of the points a and b on the straight line connecting the corresponding points substitute the average values of the pixel values indicated by a1 and b1. However, at the coordinate (11, 3), the average value b of the pixel values on the straight line is substituted as the corresponding point of b1 which is a corresponding point of both a1 and b1 but has a high score.

When this processing is completed, the routine proceeds to step S83, where it is judged whether or not the processing of step S82 has been completed for all the corresponding points of the priority order n on the noted EPI, and if not, it proceeds to step S82. When the process returns and ends, the process proceeds to step S84, and it is determined whether or not the priority order currently processed is 1.

If the result of determination in step S84 is that the priority is not 1, the priority is set to 1 in step S85.
Then, (n = n-1), the process proceeds to step S82, and if 1 is determined in step S86, it is determined whether the focused EPI is the final EPI. Here, ny is the total number of rasters of the input image. If the raster of interest is not the final EPI, the EP of interest
I is set to the next EPI (j = j + 1), and step S
Return to 85. If it is the final EPI, the image interpolation processing is terminated and the process returns to step S25.

As shown in FIG. 9, if the lines interpolated by the above process are represented as j2, j3, j5, j6, j8, j9, for example, the interpolated image 2 will have an interpolated line j2 (j = 1 to n).
It can be configured by arranging y) (see FIG. 10).
The same applies to the interpolated images 3, 5, 6, and 9.

As described above, the processing is performed in order from the corresponding point having the lowest priority (the value of n is large), and the corresponding points having a high priority or the points having the same priority but having a high score are overwritten. As shown in FIG. 9, it is possible to perform interpolation processing in consideration of occlusion. However,
Symbols a, b and c represent pixel values interpolated by the corresponding points a1, b1 and c1, respectively.

By generating an interpolated image from a large number of input images using the configuration and method described above, it is possible to detect corresponding points that could not be obtained from the two images. Accuracy is improved. Even in the case of searching for corresponding points, the search parameter can be automatically determined using the shooting parameter, so that the search process can be speeded up. Further, since the corresponding points are obtained from a large number of images, the problem of occlusion can be solved as described above, and the problem that the front and rear arrangement of the imaged object is reversed can be solved by the interpolation method using the detected number of straight lines.

Further, although the case where the vertical parallax is omitted has been described above, multi-view images taken from shooting positions of coarse viewpoint intervals on a plane are held and mixed, and these multi-view images are horizontally mixed. The viewpoint image is interpolated between viewpoints, and then interpolated between viewpoints in the vertical direction to generate an image in which vertical parallax is taken into consideration.

Next, a method of further increasing the speed in the corresponding point searching section in the image processing apparatus of this embodiment will be described.

In the corresponding point searching unit of this embodiment,
Instead of just searching for a straight line from the first line to the last line of the epipolar plane image,
It can also be realized by conducting a search from the last line to the first line. The search range, the search step, and the slope of the straight line, which are the search parameters at that time, can be directly searched for the line by inverting the sign of each parameter. In this way, by searching for the corresponding points from above and below one EPI, the corresponding point search process can be sped up.

[0053]

[Second Embodiment] Next, when the user moves the viewpoint up, down, front, rear, left and right by changing the position of the head forward and backward, the viewpoint position of the observer is detected and the image seen by the observer is reconstructed. An example of an image processing apparatus that can handle a case where an observer smoothly moves the viewpoint up, down, front, back, left, and right will be described. However, in order to make the description easier to understand, a case where the vertical parallax is omitted will be described below.

FIG. 11 is a diagram showing the configuration of an image display device according to the second embodiment of the present invention. In FIG. 11, 1 is a fixed display screen for image display, 2 is a viewpoint detector for detecting the position of the eyes of the user who views the display screen 1, and 3 is a coarse viewpoint interval arranged on a plane. It is a multi-view image database that holds multi-view images of an object to be displayed.

Reference numeral 4 is a display parameter holding unit for holding parameters relating to the display screen 1, and 5 is a photographing viewpoint coordinate which holds a coordinate system representing a plane of the viewpoint arrangement when photographing the multi-view images of the multi-view image database 4. System holding part, 6
Is a multi-view image parameter holding unit that holds the image parameters of the multi-view images of the multi-view images including the images interpolated between viewpoints of the multi-view image database 4.

Reference numeral 7 is a viewpoint parameter calculator for calculating viewpoint parameters based on the signal from the viewpoint detector 1, 8 is an image generator for generating an image corresponding to the movement of the viewpoint, and 9 is a pixel indicating a pixel of interest. The index signal, 10 is a line-of-sight parameter calculation unit that calculates the line-of-sight direction corresponding to the pixel indicated by the pixel index signal 9 from the viewpoint parameter and the display parameter, and 11 is the line-of-sight represented by the line-of-sight parameter and the line-of-sight viewpoint represented by the shooting-viewpoint coordinate system A virtual viewpoint parameter calculation unit that calculates an intersection (virtual viewpoint) with the plane of, and 12 is a pixel position corresponding to the viewpoint direction of the image in the virtual viewpoint from the viewpoint parameter, the photographing viewpoint coordinate system, the virtual viewpoint parameter, and the multi-viewpoint image parameter. A pixel position calculation unit for calculating
Is a pixel value calculation unit that calculates a pixel value that calculates a corresponding pixel value from a multi-view image that also includes images inter-view interpolated in the multi-view image database 3 based on the pixel position and the virtual viewpoint parameter. An image display unit that displays an image on the screen 1.

Reference numeral 15 is an update signal indicating that the viewpoint parameter has been updated, and 16 is a pixel value signal indicating a pixel value. Reference numeral 17 denotes an inter-viewpoint interpolation processing unit that generates a multi-viewpoint image having a sufficiently fine inter-viewpoint interval from a multi-viewpoint image captured from a coarse viewpoint interval used in the first embodiment, and 18 is collectively referred to as a pixel value generation unit.

Next, the outline of the operation of the image display device having the configuration of FIG. 11 will be described. When the user looking at the display screen 1 changes the position of his head to move the viewpoint, the viewpoint detector 2
Changes, the viewpoint parameter calculation unit 7 sends the update signal 15 to the image generation unit 8 in response to this change. Upon receiving the update signal 15, the image generator 8 starts generating a new image corresponding to the movement of the viewpoint. The image generation unit 8 obtains the pixel value signal 16 corresponding to the pixel index signal 9 from all the pixels from the pixel value generation unit 17. The pixel value generation unit 17 acquires each display parameter from the display parameter storage unit 4 and calculates the line-of-sight parameter corresponding to the pixel index signal 9. Next, the virtual viewpoint parameter calculation unit 11
Acquires a shooting viewpoint coordinate system from the shooting viewpoint coordinate system holding unit 5 and sets a virtual viewpoint parameter representing an intersection (virtual viewpoint) between the line of sight represented by the line-of-sight parameter and the plane of the array of shooting viewpoints represented by the shooting viewpoint coordinate system. calculate.

On the other hand, the pixel position calculation unit 12 obtains the multi-viewpoint image parameter from the multi-viewpoint image parameter holding unit 6, and in addition, the line-of-sight direction of the image at the virtual viewpoint position from the line-of-sight parameter, the photographing viewpoint coordinate system, and the virtual viewpoint parameter. The pixel position corresponding to is calculated. The pixel value calculation unit 13 uses the pixel position and the virtual viewpoint parameter to generate a corresponding pixel value signal 16 from the multi-viewpoint image database 3 and the inter-viewpoint interpolation image generated by the inter-viewpoint interpolation processing unit 17 from them.
To calculate. When the pixel value signal 16 is obtained from the pixel value calculation unit 13 for all the pixels, the image generation unit 8 sends it to the image display unit 14. The image display unit 14 displays the image thus generated on the display screen 1 corresponding to the new viewpoint.

As a result, the user displays the image of the object according to the movement of the viewpoint even if the viewpoint is moved forward, backward, upward, downward, leftward or rightward other than the viewpoint position at which the image stored in the multi-viewpoint image database 3 is photographed. It can be viewed through screen 1.

Next, the processing of each unit will be described in detail.

First, the processing of the line-of-sight parameter calculating section 10 will be described with reference to FIGS.

FIG. 12 is a diagram showing the calculation principle of the line-of-sight parameter calculation unit 10. 1 is a display screen, 21 is an end point of the display screen 1 (indicated by a position vector Xs), 22
Is the vector whose length is the pixel pitch of the display screen 1 and whose inclination matches the inclination of the display screen 1 (shown by the display screen vector p), and 23 is the pixel position of interest on the display screen 1 (shown by the position vector Xp). , 24
Is the user's viewpoint position (indicated by position vector Xv), 25
Is the line of sight corresponding to the pixel position 23 of interest, and 26 is the line of sight 2
It is a line-of-sight vector (indicated by a vector a) indicating the inclination of 5.

FIG. 13 is a flow chart showing the processing of the line-of-sight parameter calculation unit 10. First, in S31, the viewpoint parameter is acquired from the viewpoint parameter calculation unit 7. Here, the viewpoint parameter is the viewpoint position 24 of the user in FIG. Next, in S32, the display parameter is acquired from the display parameter holding unit 4. Here, the display parameters are the end point 21 (vector Xs) of the display screen 1 and the display screen vector 22 (vector p). The display screen vector 22 is determined from the tilt of the display screen, the actual size and the pixel size. Next, in S33, the pixel position 23 of interest on the display screen 1 corresponding to the pixel index signal 9 is calculated according to the geometry shown in FIG. However, the pixel index signal 9
Be i.

Xp = Xs + i · p (1) Next, in S34, the line-of-sight parameter corresponding to the direction of the pixel position 23 when viewed from the viewpoint position 24 is obtained. The line-of-sight parameter is a set (Xv, a) of the viewpoint position 24 and the line-of-sight vector 26. The line-of-sight 25 is a straight line passing through the two points of the pixel position 23 (vector Xp) and the viewpoint position 24 (vector Xv) according to the geometry shown in FIG. 12, and the line-of-sight vector 26 (vector a) is expressed by the following equation (2). calculate.

A = Xp-Xv (2) The line-of-sight parameter calculating unit 10 calculates the line-of-sight parameter in this way.

Next, the processing of the virtual viewpoint parameter calculator 11 and the pixel position calculator 12 will be described with reference to FIG.

FIG. 14 is a diagram showing the calculation principle of the virtual viewpoint parameter calculation unit 11 and the pixel position calculation unit 12. Two
5 is a line of sight, 41 is a line of sight lines which is a plane of the line of sight when a multi-viewpoint image of the multi-viewpoint image database 3 is photographed, 42 is an intersection of the line of sight 25 and the line of sight 41 (virtual viewpoint, position vector). X is shown), 43 is a vector representing the inclination of the line of sight line 41 (shown as the line of sight vector T), and 44 is an end point of the line of sight line (shown as the position vector X1). Reference numeral 45 is a field of view with an angle of view θ at the virtual viewpoint 42, 46 is a vector whose length is the focal length of the camera that captured the multi-viewpoint image, and whose inclination is the inclination of the camera (indicated by the focus vector f), 47 Is a virtual imaging plane at the virtual viewpoint 42, 48 is a virtual imaging plane 47 and the line of sight 2
5 is a pixel position (shown by a position vector Xp ′) which is an intersection with 5, and 49 is a pixel pitch of the imaginary imaging surface 47 having a length, and the inclination of the imaginary imaging surface 47 has an inclination (usually the focus vector 46 The vector (shown by the imaging plane vector p ′) that matches the (right angle).

Here, the viewpoint arrangement vector 43 and the end point 44 of the viewpoint arrangement line are values representing the photographing viewpoint coordinate system.
It is held in the photographing viewpoint coordinate system holding unit 5. The focus vector 46 and the imaging plane vector 49 are held in the multi-viewpoint image parameter holding unit 6 as multi-viewpoint image parameters. The focus vector 46 is determined from the focal length and the tilt of the camera that captured the multi-viewpoint image. The imaging plane vector 49 is a vector that is orthogonal to the focus vector 46 and has a size equal to the cell size (1 pixel size) of the imaging plane.

Next, the processing of the virtual viewpoint parameter calculation unit 11 will be described with reference to FIG. According to the geometry shown in FIG. 14, the virtual viewpoint 42 is expressed by the following equations (3) and (4).

X = Xl + t.multidot.T (3) X = Xv + .alpha..multidot.a (4) Here, t is a virtual viewpoint parameter that uniquely represents the virtual viewpoint. α is a coefficient in the line-of-sight direction.
By calculating equations (3) and (4), t is calculated and X is calculated.

Next, referring to FIG. 14, the pixel position calculation unit 12
The processing of will be described. According to the geometry shown in FIG. 14, the pixel position 48 is expressed by the following equations (5) and (6).

Xp ′ = X + f + i ′ · p ′ (5) Xp ′ = X + β · a (6) Here, i ′ is a pixel position parameter that uniquely represents the pixel position 48. β is a coefficient in the viewing direction. I ′ is calculated by solving the equations (5) and (6), and this is used as the output of the pixel position calculation unit 12.

Next, the processing of the pixel value calculation unit 13 will be concretely described.

In this embodiment, the multi-view image database 3
The multi-viewpoint image held in is an image captured at coarse viewpoint intervals, and the inter-viewpoint interpolation processing unit 17 can obtain a sufficiently fine inter-viewpoint interpolated image. This interpolation processing is performed as described in the first embodiment. Therefore, first, as an approximate image of the image from the virtual viewpoint 42 indicated by the virtual viewpoint parameter t calculated by the virtual viewpoint parameter calculation unit 11,
An image captured from a viewpoint closest to the virtual viewpoint is obtained from the multi-viewpoint image database 3 and an inter-viewpoint interpolation processing unit generates an interpolated image based on the image. Next, in this image, the value of the pixel at the position closest to the pixel position 48 calculated by the pixel position calculation unit 12 is acquired, and this is set as the output pixel value signal 16.

For the sake of clarity, the processing of each part when the vertical parallax is omitted has been described above. However, if a vertical multi-viewpoint image is prepared, the vertical parallax can be calculated in the same manner. This is a binocular stereoscopic display device that can be moved in the front, rear, up, down, left, and right directions, taking into consideration.

By generating an interpolated image from a large number of input images using the configuration and method described above, it is possible to detect corresponding points that could not be obtained from two images. The accuracy of is improved. Even in the case of searching for corresponding points, the search parameter can be automatically determined using the shooting parameter, so that the search process can be speeded up. Also, since the corresponding points are obtained from a large number of images, the problem of occlusion can be solved as described above,
Further, the interpolation method using the detected number of straight lines can solve the problem that the front and rear arrangement of the photographed object is reversed. In order to display the interpolated image obtained in this way according to the viewpoint of the observer, the image display apparatus of the present embodiment is capable of smoothly and accurately moving the image as the viewpoint of the observer moves. It is possible.

The display screen 1 and the image display unit 8 are a stereoscopic display screen and a stereoscopic image display unit capable of binocular stereoscopic vision such as a lenticular system or a glasses system, and the viewpoint parameter calculation unit 7 is left and right. By calculating the viewpoint parameter corresponding to the position of the eyes of the eyes and correspondingly generating the image to be presented to each of the left and right eyes, the image generation unit 8 can move the viewpoints in the front, rear, up, down, left and right directions. It becomes a display device.

[0079]

Third Embodiment Next, an example of a so-called head mounted display (HMD) type image display device in which the display screen 1 in the second embodiment is fixed to the user's head will be described.

The configuration of this embodiment is the same as that of the second embodiment except that only the processing contents of the line-of-sight parameter calculation unit 10 are replaced with the processing described below. In the description, the case where the vertical parallax is omitted will be described.

FIG. 15 is a diagram showing the calculation principle of the line-of-sight parameter calculation unit 10 in this embodiment. 1 is a display screen, 22 is a vector whose length is the pixel pitch of the display screen 1 and whose inclination matches the inclination of the display screen 1 (display screen vector p), and 23 is the pixel position (position) of interest on the display screen 1. Vector Xp), 24
Is the user's viewpoint position (position vector Xv), 111 is the vector from the viewpoint position 24 to the center point of the display screen 1 (front vector F), 25 is the line of sight corresponding to the pixel position 23 of interest, and 26 is the line of sight 25. It is a line-of-sight vector (vector a) that represents the inclination.

Next, the processing of the line-of-sight parameter calculation unit 10 will be described with reference to FIG. In the HMD type display device, the viewpoint detector 2 detects the tilt in the front direction, that is, the tilt of the front vector 111, in addition to the position of the viewpoint 24 of the user. The tilt of the display screen vector 22 is determined from the tilt of the front vector 111 (usually a right angle). On the other hand, the distance from the viewpoint position 24 to the display screen 1, that is, the length of the front vector 111, and the pixel pitch, that is, the length of the display screen vector 22, are fixed values determined by the shape of the HMD. Are stored in the display parameter storage unit 4. According to the geometry shown in FIG. 15, the pixel position 23 (vector Xp) of interest and the line-of-sight vector 26 (vector a) are calculated by the following formulas. However, the pixel index 9 is i.

Xp = Xv + F + i · pa = Xp-Xv As described above, the viewpoint parameter calculation unit 10 calculates the pixel position and the line of sight, and other configurations and processing are performed in the same manner as in the first embodiment. , An HMD type display device capable of moving and displaying a multi-viewpoint image at any viewpoint.

Further, even if the display screen 1 is not fixed to the head, a cockpit type display in which the relative positional relationship between the fixed display screen 1 and the user's viewpoint position 24 is fixed. Even if the apparatus is used, the same processing of the line-of-sight parameter calculation unit 10 as in the present embodiment provides an image display apparatus capable of arbitrary viewpoint movement display. At this time, instead of the viewpoint detector 2, a viewpoint position input device for operating the viewpoint position 24 in the reference coordinates with a handle or the like is used.

By generating an interpolated image from a large number of input images by using the configuration and method described above, it is possible to detect corresponding points that could not be obtained from two images. The accuracy of is improved. Even in the case of searching for corresponding points, the search parameter can be automatically determined using the shooting parameter, so that the search process can be speeded up. Also, since the corresponding points are obtained from a large number of images, the problem of occlusion can be solved as described above,
Further, the interpolation method using the detected number of straight lines can solve the problem that the front and rear arrangement of the photographed object is reversed. In order to display the interpolated image obtained in this manner according to the viewpoint of the observer, according to the HMD of the present embodiment, smooth and highly accurate image movement is possible as the observer's viewpoint moves. It is possible.

In each of the above-mentioned first to third embodiments, the multi-viewpoint image database 3 including pre-captured multi-viewpoint images and images interpolated between them is held. Is replaced with a multi-view television camera capable of capturing multi-viewpoint images in real time, thereby providing a real-time arbitrary-viewpoint image capturing / display system.

With the above structure, the apparatus of each of the above embodiments is
It is possible to generate an interpolated image with high accuracy even when the context of the object is reversed.
In addition, by automatically determining the corresponding point search parameter, the speed of the interpolation process can be increased. In addition, by reconstructing the image corresponding to the viewpoint movement in the front-back direction using a large number of images obtained by interpolation processing or a multi-viewpoint image captured by moving the viewpoint finely enough, It became possible to move the viewpoint to.

The present invention may be applied to a single image processing apparatus, a system equipment such as a multi-view television, a multi-view video telephone terminal, a multi-view video conference system, or a computer. Also, it can be applied to a composite device combined with another image processing device.

[0089]

As described above, the image display device according to the present invention can accurately generate an interpolated image regardless of the content of the image, and also uses the generated interpolated image to display with high accuracy. Can be realized.

[0090]

[Brief description of drawings]

FIG. 1 is a block diagram of an image display device according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a flow of image interpolation processing of the first embodiment.

FIG. 3 is a diagram showing a principle of calculating a corresponding point search range from the shooting parameter in step S22 of FIG. 2;

FIG. 4 is a flowchart showing a flow of corresponding point search processing in step S23 of FIG.

FIG. 5 is a diagram showing a j-th epipolar plane image.

FIG. 6 is a flowchart showing the flow of straight line detection in step S24 of FIG.

FIG. 7 is a diagram for explaining interpolation processing.

FIG. 8 is a flowchart showing a flow of interpolation processing.

FIG. 9 is a diagram for explaining interpolation processing.

FIG. 10 is a diagram for explaining interpolation processing.

FIG. 11 is a block diagram of an image display device according to a second embodiment.

FIG. 12 is a diagram showing the calculation principle of the line-of-sight parameter calculation unit 10 of the second embodiment.

FIG. 13 is a flowchart showing the processing of the line-of-sight parameter calculation unit 10.

FIG. 14 is a diagram showing calculation principles of a virtual viewpoint parameter calculation unit 11 and a pixel position calculation unit 12.

FIG. 15 is a line-of-sight parameter calculation unit 1 according to the third embodiment.
It is a figure which shows the calculation principle of 0.

FIG. 16 is a diagram illustrating a flow of inter-viewpoint interpolation processing.

FIG. 17 is a flowchart of a process for detecting corresponding points according to the first embodiment.

[Explanation of symbols]

 20, 21, 22, 23 camera, 25 CPU, 26 RAM, 27 ROM, 29 disk, 30 VRAM, 32 display.

Claims (28)

[Claims] 1. An image input step of inputting a plurality of images corresponding to viewpoints at a plurality of positions, and a candidate detection step of detecting corresponding pixel candidates between the plurality of images input by the image input step, A checking step of checking the likelihood of correspondence with respect to the candidate of the corresponding pixel, and a step of detecting a corresponding pixel from the candidate of the corresponding pixel detected by the candidate detecting step, according to the certainty of the correspondence. A detection step; and an image interpolation step of generating an image corresponding to a viewpoint different from the viewpoint of the image input by the image input step based on the relationship between the corresponding pixels detected by the correspondence detection step. An image processing method characterized by the above. 2. The image processing method according to claim 1, wherein the image input step inputs images from a plurality of cameras. 3. The image processing method according to claim 1, wherein the image input step inputs an image from an image database. 4. The image processing method according to claim 1, wherein the candidate detecting step detects the corresponding points between the input images by detecting straight lines on the epipolar plane image. 5. The image processing method according to claim 4, wherein the candidate detecting step obtains the inclination of a straight line detected on the epipolar plane with decimal precision. 6. The checking step assigns a low score to the pixels on each straight line if the number of straight lines is large and a high score to the pixels on each straight line according to the number of straight lines detected from each pixel of interest. 5. The image processing method according to claim 4, wherein the correspondence detection step excludes pixels having a high score from the corresponding pixels of pixels having a low score among the corresponding pixel candidates. 7. The candidate detecting step comprises:
The image processing method according to claim 4, wherein the detection parameter is automatically determined by using the parameter at the time of shooting. 8. The image processing method according to claim 4, wherein in the candidate detecting step, line detection is also performed in a reverse direction. 9. The image processing method according to claim 4, wherein the image interpolation step performs image interpolation for each epipolar plane image of the input image. 10. The correspondence detection step excludes a pixel having a corresponding pixel having a more reliable correspondence from a corresponding point of a pixel having a more uncertain correspondence. Image processing method. 11. The image interpolating step generates an interpolation raster by compensating pixels on a straight line detected on each epipolar plane image by the candidate detecting means, and connects the interpolation raster for each epipolar plane image. The image processing method according to claim 4, wherein the interpolated image is generated by the above method. 12. An image input step of inputting a plurality of images corresponding to viewpoints at a plurality of positions, and a straight line detection on the epipolar plane image of pixel candidates corresponding to the plurality of images input by the image input step. A candidate detection step of detecting the corresponding pixel, and a score indicating the likelihood of correspondence with respect to the candidate of the corresponding pixel, for each pixel according to the number of straight lines detected by the detecting means passing through each pixel. From the corresponding pixel candidates that are detected by the applying step and the candidate detection step, a pixel candidate having a corresponding pixel having a more corresponding relationship is selected from corresponding points of pixels having a corresponding pixel having a more uncertain corresponding relationship. Exclude the corresponding detection step of detecting the corresponding pixel and generate the interpolation point on the straight line on the epipolar plane image passing between the corresponding pixels detected by the corresponding detection step. And,
An image interpolation step of synthesizing the interpolation points for all epipolar plane images to generate an image corresponding to a viewpoint different from the viewpoint of the image input in the image input step. Image processing method. 13. A viewpoint detection step of detecting the position of the eyes of an observer, an image seen from the detected viewpoint position, an image input by the image input step, and an interpolation image generated by the image interpolation step. The image processing method according to claim 1, further comprising an image reconstructing step of reconstructing the reconstructed image and an image outputting step of outputting the reconstructed image. 14. The image reconstructing step calculates a parameter required for image reconstruction from the viewpoint position of the observer and the type of the image output device, and each pixel of the reconstructed image is input using the parameter. 14. The image processing method according to claim 13, wherein which pixel of the multi-view image corresponds to is calculated, and the corresponding pixel is extracted from the multi-view image to reconstruct the image. 15. Image input means for inputting a plurality of images corresponding to viewpoints at a plurality of positions, and candidate detection means for detecting a candidate of a corresponding pixel among the plurality of images input by the image input means, Checking means for checking the probability of correspondence with respect to the candidate of the corresponding pixel, and detecting the corresponding pixel from the candidate of the corresponding pixel detected by the candidate detecting means, according to the certainty of the correspondence. A detection means; and an image interpolation means for generating an image corresponding to a viewpoint different from the viewpoint of the image input by the image input means, based on the relationship between the corresponding pixels detected by the correspondence detection means. An image processing device characterized by the above. 16. The image processing apparatus according to claim 15, wherein the image input unit includes a plurality of cameras and inputs images from the cameras. 17. The image processing apparatus according to claim 15, wherein the image input unit includes an image database and inputs an image from the database. 18. The image processing apparatus according to claim 15, wherein the candidate detecting unit detects the corresponding points between the input images by performing straight line detection on the epipolar plane image. 19. The image processing apparatus according to claim 18, wherein the candidate detecting means obtains the inclination of a straight line detected on the epipolar plane with decimal precision. 20. According to the number of straight lines detected from each pixel of interest, the checking means gives a low score to each pixel on each straight line if the number of straight lines is large and a high score if the number of straight lines is small. 19. The image processing apparatus according to claim 18, wherein the correspondence detection unit excludes, from the corresponding pixel candidates, pixels having a high score from corresponding pixels of pixels having a low score. 21. The image processing apparatus according to claim 18, wherein the candidate detecting means automatically determines a detection parameter by using a parameter at the time of shooting when the straight line is detected. 22. The image processing apparatus according to claim 18, wherein the candidate detecting unit searches for a straight line in the reverse direction. 23. The image interpolation means performs image interpolation,
The image processing apparatus according to claim 18, wherein the processing is performed for each epipolar plane image of the input image. 24. The correspondence detection means excludes a pixel having a corresponding pixel having a more certain correspondence relationship from corresponding points of pixels having a corresponding point having a more uncertain correspondence relationship. Image processing device. 25. The image interpolating means supplements pixels on a straight line detected on each epipolar plane image by the candidate detecting means to generate an interpolating raster, and connects the interpolating raster for each epipolar plane image. 19. The image processing apparatus according to claim 18, wherein the interpolated image is generated by the interpolation. 26. An image input unit for inputting a plurality of images corresponding to viewpoints at a plurality of positions, and a pixel candidate corresponding to the plurality of images input by the image input unit are detected by a straight line on the epipolar plane image. And a score indicating the certainty of correspondence with respect to the candidate of the corresponding pixel, for each pixel according to the number of straight lines detected by the detecting unit passing through each pixel. From the corresponding pixel candidates having a more reliable correspondence pixel from the corresponding pixel candidates detected by the granting means and the candidate detection means, from corresponding points of pixels having a more uncertain correspondence pixel in the correspondence relationship. Exclude and generate the interpolation detection point on the straight line on the epipolar plane image passing between the corresponding pixels detected by the corresponding detection means and the corresponding detection means. And,
Image interpolation means for synthesizing the interpolation points for all epipolar plane images to generate an image corresponding to a viewpoint different from the viewpoint of the image input by the image input means. Image processing device. 27. A viewpoint detecting means for detecting the position of the eyes of an observer, an image seen from the detected viewpoint position, an image inputted by the image input means, and an interpolated image generated by the image interpolating means. 16. The image processing apparatus according to claim 15, further comprising an image reconstructing unit for reconstructing the reconstructed image and an image output unit for outputting the reconstructed image. 28. The image reconstructing means calculates a parameter necessary for reconstructing an image from the viewpoint position of the observer and the type of the image output device, and each pixel of the reconstructed image is input using the parameter. 28. The image processing apparatus according to claim 27, further comprising: calculating which pixel of the multi-viewpoint image corresponds, extracting the corresponding pixel from the multi-viewpoint image, and reconstructing the image.