JP3054312B2 - Image processing apparatus and method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method and an image processing apparatus for inputting a plurality of images having different viewpoints and outputting an image having a viewpoint position corresponding to the current eye position of an observer.

[0002]

2. Description of the Related Art Conventionally, as a device for stereoscopically displaying an image viewed from a plurality of viewpoints, there are a stereo display and a lenticular display. The stereo display alternately displays images obtained from two cameras at high speed. The observer can observe the image three-dimensionally by using shutter glasses or polarized glasses synchronized with the switching. The lenticular display is capable of three-dimensionally displaying images from four viewpoints by rearranging images from, for example, four cameras in pixel units and attaching a lenticular sheet to the front surface.

[0003]

However, in the above-described conventional stereo display, only a three-dimensional image in a shooting direction of a camera at the time of shooting an image can be observed. That is, since an object is generally photographed with two cameras fixed, even if the observer moves his / her viewpoint (eye position), the same image is seen, and the viewpoint movement on the observer side is not reflected. There is a problem of lack of realism.
In addition, the lenticular display can cope with the movement of the viewpoint of the observer in the left and right direction, but it can see the image viewed from any of the multiple cameras in a discrete manner and cannot cope with the continuous viewpoint movement. In addition, it was not possible to move the viewpoint in the front-back direction. The movement of the viewpoint in the front-back direction is performed when stereoscopic viewing is performed based on an image created by computer graphics, but this is simple as a computer graphics image and a point within the image is used. This is under a special situation where the coordinate values in the corresponding three-dimensional space are all clear. When stereoscopically viewing an image captured by a camera, moving the viewpoint in the front-back direction has hardly been considered so far.

The present invention has been made in view of the above problems, and when an observer's eye position moves in each direction including the front-back direction, an image viewed from the moved position can be given. An object of the present invention is to provide an image processing method and an image processing device.

[0005]

In order to achieve the above object, an image processing apparatus according to the present invention has the following arrangement.
That is, an image input unit that inputs a plurality of images having different viewpoints as viewpoints as multi-view image data, a viewpoint detection unit that detects a position of a viewpoint of an observer, and a viewpoint position detected by the viewpoint detection unit. Image reconstructing means for reconstructing an image having the viewpoint as the viewpoint from the multi-viewpoint image data, correcting means for correcting distortion of the reconstructed image, and outputting the corrected image to an image output device Image output means.

In order to achieve the above object, an image processing method according to the present invention has the following arrangement. That is,
An image input step of inputting a plurality of images having different viewpoints as viewpoints as multi-viewpoint image data, a viewpoint detection step of detecting the position of the observer's viewpoint, and a viewpoint position detected by the viewpoint detection step. An image reconstructing step of reconstructing an image having the viewpoint as the viewpoint from the multi-viewpoint image data, a correcting step of correcting a distortion of the reconstructed image, and an image outputting the corrected image to an image output method. And an output step.

[0007]

With the above arrangement, an image corresponding to the viewpoint position of the observer is differently constructed and output based on the multi-viewpoint image.

[0008]

Next, embodiments of the present invention will be described with reference to the drawings.

[0009]

First Embodiment <Structure of Apparatus> FIG. 1 is a diagram showing the structure of an image processing apparatus according to a first embodiment. In the figure, reference numeral 11 denotes a fixed display screen for displaying an image, 12 denotes a viewpoint detector for detecting the position of the eyes of a user who looks at the display screen 11, and 13 denotes a finely spaced viewpoint having a sufficiently long interval. 5 is a multi-view image database that stores multi-view images obtained by photographing an object to be displayed from. Reference numeral 14 denotes a display parameter storage unit that stores parameters related to the display screen 11, and reference numeral 15 denotes a capturing viewpoint coordinate system that retains a coordinate system that represents a straight line of viewpoints when capturing a multi-view image in the multi-view image database 13. Reference numeral 16 denotes a multi-view image parameter holding unit that holds image parameters of multi-view images in the multi-view image database 13.

Reference numeral 17 denotes a viewpoint parameter calculation unit for calculating viewpoint parameters based on a signal from the viewpoint detector 12, 18 an image generation unit for generating an image corresponding to movement of the viewpoint, 19
Is a pixel index signal indicating a pixel of interest, and 110 is
A gaze parameter calculation unit that calculates a gaze direction corresponding to the pixel indicated by the pixel index 19 from the viewpoint parameter and the display parameter, and a line-of-sight line represented by the line-of-sight parameter represented by the line-of-sight parameter represented by the line-of-sight parameter. A virtual viewpoint parameter calculation unit for calculating an intersection (virtual viewpoint),
Reference numeral 112 denotes a pixel position calculation unit that calculates a pixel position corresponding to a line-of-sight direction of an image at a virtual viewpoint from a line-of-sight parameter, a shooting viewpoint coordinate system, a virtual viewpoint parameter, and a multi-viewpoint image parameter. A pixel value calculation unit 11 for calculating a corresponding pixel value from the multi-view image in the multi-view image database 13 based on the parameters;
Reference numeral 4 denotes an image display unit that displays an image on the display screen 11. Reference numeral 115 denotes an update signal indicating that the viewpoint parameter has been updated, and reference numeral 116 denotes a pixel value signal. Reference numeral 117 denotes a pixel value generation unit. Reference numeral 118 denotes a distortion correction unit that corrects vertical distortion.

Here, the multi-view image database 13 is
From a number of viewpoints arranged on a straight line with sufficiently fine viewpoint intervals,
This is an image obtained by photographing an object to be displayed. Correspondingly, the data held in the shooting viewpoint coordinate system holding unit 15 is data of a coordinate system that represents a straight line of viewpoints at the time of capturing these images. The image generation unit 18 outputs the update signal 1
15 is received, the image generation unit 18 generates a pixel index 19 indicating the coordinates of the pixel of interest in the reconstructed image, that is, the image on the display screen 11. Has been output. When reconstructing an image, the pixel index 19 is sequentially output so as to make a cycle through all pixels of the reconstructed image.

<Operation of Apparatus> Next, the operation of this embodiment will be described. First, an outline of this operation will be described.

When the user looking at the display screen 11 changes his / her head position and moves his / her viewpoint, the signal of the viewpoint detector 12 changes, and the viewpoint parameter calculation unit 17 receives the change and generates an update signal 115. Send to the image generation unit 18. Upon receiving the update signal 115, the image generator 18 starts generating a new image corresponding to the viewpoint movement. To generate this new image, the image generation unit 18 sequentially outputs the pixel indexes 19 for all the pixels, and outputs the pixel value signal 11 for each pixel.
6 is sequentially obtained from the pixel value generation unit 117. Here, the operation of the pixel value generation unit 117 will be described.

[Pixel Value Generation Unit] In the pixel value generation unit 117, first, the line-of-sight parameter calculation unit 110 acquires viewpoint parameters from the viewpoint parameter calculation unit 17 and display parameters from the display parameter holding unit 14, and receives them. A gaze parameter corresponding to the pixel index 19 is calculated. Next, the virtual viewpoint parameter calculation unit 111 acquires the shooting viewpoint coordinate system from the shooting viewpoint coordinate system holding unit 15,
A virtual viewpoint parameter representing an intersection (virtual viewpoint) between a line of sight indicated by the line-of-sight parameter and a straight line of the arrangement of the imaging viewpoints represented by the imaging viewpoint coordinate system is calculated. On the other hand, the pixel position calculation unit 112
Acquires the multi-viewpoint image parameters from the multi-viewpoint image parameter holding unit 16, and calculates the pixel position corresponding to the line-of-sight direction of the image at the virtual viewpoint from the line-of-sight parameters, the shooting viewpoint coordinate system, and the virtual viewpoint parameters. Then, based on the pixel position and the virtual viewpoint parameters, the pixel value calculation unit 113 calculates, from the image in the multi-viewpoint image database 13,
A corresponding pixel value signal 116 is calculated. In this way, the pixel value generation unit 117 outputs the input pixel index 1
The pixel value signal 116 is calculated every 9 and output to the image generation unit 18.

[Processing of Generated Pixel Value] As described above, when the image generation unit 18 obtains the pixel value signal 116 for all the pixels from the pixel value calculation unit 113, it sends this to the distortion correction unit 118. . The distortion correction unit 118 performs vertical scaling of the image in order to correct vertical distortion of the generated image, and displays the corrected image on the image display unit 11.
Output to 4. The image display unit 114 displays the image corresponding to the new viewpoint thus generated on the display screen 11. As a result, a series of image generation operations accompanying the movement of the viewpoint of the user is completed. As will be apparent from the description below, when the user moves the viewpoint back and forth, left and right, even if that is a place other than the viewpoint where each image held in the multi-view image database 13 is captured, The image of the target object in accordance with the movement of the viewpoint can be viewed through the display screen 11.

Next, the processing of each section will be described in detail.

[Calculation of Line-of-Sight Parameters] First, the process of calculating the line-of-sight parameters in the line-of-sight parameter calculation unit 110 will be described.

FIG. 2 is a diagram showing the calculation principle of the visual axis parameter calculation unit 110 of the image processing apparatus according to the first embodiment. In the figure, 21 is an end point of the display screen 11, 22 is a display screen vector whose length is the pixel pitch of the display screen 11, and whose inclination matches the inclination of the display screen 11,
23 is a pixel position of interest on the display screen 11, 24
Is a viewpoint position of the user, 25 is a line of sight corresponding to the pixel position 23 of interest, and 26 is a line of sight vector representing the inclination of the line of sight 25.

Here, the end point 21 and the pixel position 2 of interest
3, user's viewpoint 24, display screen vector 2
2. It is assumed that the line-of-sight vector 26 is represented by a vector Xs, a vector Xp, a vector Xv, a vector p, and a vector a, respectively. In addition, the processing is performed in the plane of the left, right, front and rear, ignoring the vector components in the vertical direction.

FIG. 3 is a flow chart showing the processing of the line-of-sight parameter calculator 110 of the image processing apparatus according to the first embodiment.

First, in step S31, viewpoint parameters are obtained from the viewpoint parameter calculation unit 17. The viewpoint parameter is the viewpoint position 24 of the user in FIG. Also,
In step S32, display parameters are acquired from the display parameter holding unit 14. The display parameters are represented by an end point 21 of the display screen 11 and a display screen vector 22. The display screen vector 22 is determined from the tilt of the display screen 11, the actual size, and the pixel size. Next, in step S33, a pixel position 23 of interest on the display screen 11 corresponding to the pixel index 19 is calculated by the following equation (1) based on the arrangement shown in FIG. Here, the pixel index 19 is i.

Xp = Xs + ip (1) Next, in step S34, a line-of-sight parameter corresponding to the direction of the pixel position 23 of interest when viewed from the user's viewpoint position 24 is determined. The line-of-sight parameter is represented as a pair (Xv, a) of the user's viewpoint position 24 and the line-of-sight vector 26. Since the line of sight 25 is a straight line passing through two points of the pixel position 23 of interest and the user's viewpoint position 24, the line of sight vector 26 can be calculated by the following equation (2).

A = Xp−Xv (2) [Calculation of Pixel Position and Virtual Viewpoint] Next, the processing of the virtual viewpoint parameter calculation unit 111 and the pixel position calculation unit 112 will be described with reference to FIG.

FIG. 4 shows a virtual viewpoint parameter calculator 111 and a pixel position calculator 112 of the image processing apparatus according to the first embodiment.
It is a figure showing the calculation principle of. In the figure, reference numeral 41 denotes a line of viewpoint arranged at the time of photographing a multi-viewpoint image of the multi-viewpoint image database 13, 42 denotes a virtual viewpoint which is the intersection of the line of sight 25 and the line of viewpoints 41, and 43 denotes the inclination of the line of viewpoints 41. The viewpoint array vector 44 shown is an end point of the viewpoint array straight line 41.
45 is a visual field of the angle of view θ at the virtual viewpoint 42, 46 is a focal length of the focal length of the camera that has captured the multi-viewpoint image, and a tilt is the inclination of the camera. 47 is a virtual imaging surface at the virtual viewpoint 42. , 48 are virtual imaging planes 4
The pixel position 49, which is the intersection of the line of sight 7 and the line of sight 25, the length of which is the pixel pitch of the virtual imaging plane 47, the inclination of which coincides with the inclination of the virtual imaging plane 47 (usually perpendicular to the focus vector 46) Is the imaging plane vector.

Here, the virtual viewpoint 42, the viewpoint arrangement vector 43, the end point 44, the focus vector 46, the pixel position 48, and the imaging plane vector 49 are represented by a vector X, a vector T, a vector X1, a vector f, a vector Xp ', and a vector Xp', respectively. Let it be represented by p '. For the sake of explanation, the vector components in the vertical direction are ignored, and the processing is performed in the planes before, after, left, right, front and rear.

Here, the viewpoint array vector 43 and the end point 44 are held in the shooting viewpoint coordinate system holding unit 15 as values representing the shooting viewpoint coordinate system. The focus vector 46
The imaging plane vector 49 is held in the multi-view image parameter holding unit 16 as a multi-view image parameter. The size of the imaging plane vector 49 is equal to the actual cell size of the imaging plane (the size of one pixel). Further, the viewpoint arrangement straight line 41 is arranged in parallel with the display screen vector 22.

When each point and each vector are represented as described above, the virtual viewpoint 42 is represented by the following equations (3) and (4).

X = Xl + t · T (3) X = Xv + α · a (4) Here, t is a virtual viewpoint parameter as a parameter uniquely representing a virtual viewpoint. α is a coefficient in the line-of-sight direction. The virtual viewpoint parameter calculation unit 111 calculates a virtual viewpoint parameter t by solving equations (3) and (4), and calculates a virtual viewpoint position vector X.

The pixel position 48 is defined by the following (5),
It is expressed by equation (6).

Xp ′ = X + f + i ′ · p ′ (5) Xp ′ = X + β · a (6) Here, i ′ is a pixel position parameter as a parameter uniquely representing the pixel position 48. β is a coefficient of the line-of-sight direction. The pixel position calculation unit 112 calculates a pixel position parameter i ′ by solving equations (5) and (6), and outputs the calculated pixel position parameter i ′.

[Calculation of Pixel Value] Next, the pixel value calculation unit 11
The processing of No. 3 will be specifically described.

In this embodiment, the multi-view image database 1
The multi-viewpoint image held in 3 is an image captured at a sufficiently small viewpoint interval. Therefore, first, as an approximate image of the image from the virtual viewpoint 42 indicated by the virtual viewpoint parameter calculated by the virtual viewpoint parameter calculation unit 111, an image taken from the viewpoint closest to the virtual viewpoint 42 is stored in the multi-view image database. Find from 13. Next, in this image, the value of the pixel at the position closest to the target pixel position 23 calculated by the pixel position calculation unit 112 is obtained, and this is set as the output pixel value signal 116.

[Distortion Correction] Next, the processing of the distortion correction section 118 will be described in detail. Since the image in the multi-view image database 13 is a multi-view image only in the left-right direction, there is no parallax information in the up-down direction. For this reason, the image generation unit 18
Cannot completely completely reconstruct an image whose viewpoint has been moved in the front-back direction, and distortion occurs in the vertical direction. The distortion correction unit 118 enlarges / reduces the image in the vertical direction according to the viewpoint position 24 of the user, thereby obtaining the viewpoint alignment straight line 41.
To correct the distortion of the image of the subject at a specific distance from.

FIG. 5 is a diagram showing the principle of the distortion correction unit 118 of the image processing apparatus according to the first embodiment. In the figure, 51 is the height of a certain subject, 52 is the distance to the subject, 53 is the focal length of the camera, 54 is the height of the image of the photographed subject, 55 is the distance to the front and rear viewpoints, and 56 is the reconstructed image. This is the height of the image of the subject.

Here, the height 51 and the distance to the subject 5
2, focal length 53, image height 54, viewpoint moving distance 55,
The height 56 of the image is defined as T, Td, F, s, z,
Let it be denoted by s'. When each value is represented in this way, the following equations (7) and (8) hold.

S / F = T / Td (7) s ′ / F = T / (Td−z) (8) Equation (9) is derived by solving equations (7) and (8).

K = s ′ / s = Td / (Td−z) (9) Here, k is a distortion degree representing a distortion in the vertical direction. The distortion correction unit 118 calculates the coordinate values in the front-back direction of the user's viewpoint position 24 obtained from the viewpoint parameter calculation unit 17 and the coordinate values in the front-back direction of the viewpoint line 41 obtained from the shooting viewpoint coordinate system holding unit 15. The difference is defined as a viewpoint moving distance 55. In the multi-viewpoint image parameter holding unit 16, a rough distance value to a point of interest (or a point at which the image is focused) of the photographed object is stored in advance as a distance 52 to the object. The distortion correction unit 118 acquires this value from the multi-view image parameter holding unit 16. Then, the distortion correction unit 118 calculates the degree of distortion by solving the equation (9), and enlarges / reduces the image received from the image generation unit 18 in the vertical direction by the degree of distortion. The distortion correction unit 118 outputs the image corrected in this way to the image display unit 114.

As described above, the image processing apparatus according to the present embodiment displays an image in accordance with the viewpoint position even if the viewpoint position of the observer moves not only in the direction along the line of viewpoints but also in the front-back direction. be able to.

That is, by detecting the position of the observer's eyes and reconstructing the image seen by the observer from a plurality of images, an image corresponding to the movement of the observer's viewpoint is smoothly output. In addition, there is an effect that the viewpoint can be moved in the front-back direction, which cannot be handled conventionally.

It should be noted that a stereoscopic display screen and a stereoscopic image display unit capable of binocular stereoscopic viewing, such as a lenticular system or an eyeglasses system, are used for the display screen 11 and the image display unit 114, and the viewpoint parameter calculation unit 17 controls each of the left and right. A viewpoint parameter corresponding to the position of the eye is calculated, and the image generation unit 18 generates an image to be presented to each of the left and right eyes in response to the viewpoint parameter. Device.

[0041]

Second Embodiment Next, a description will be given of an image display apparatus capable of freely moving and displaying a viewpoint even when the viewpoint intervals of the multi-view images held in the multi-view image database 13 are not sufficiently small.

FIG. 12 is a block diagram showing the configuration of the image processing apparatus of this embodiment. Portions common to the first embodiment are denoted by the same reference numerals, and description thereof is omitted. The image processing apparatus includes an inter-view interpolation processing unit 1 between the multi-view image database 13 and the pixel value calculation unit 113 of the first embodiment.
20 is provided. The inter-viewpoint interpolation processing unit 120 generates a group of images having sufficiently fine inter-viewpoint intervals by performing interpolation processing from each image captured at the coarse viewpoint intervals in the multi-viewpoint image database 13. By using an image with a sufficiently fine inter-viewpoint interval, an image corresponding to a change in the user's viewpoint is generated as in the first embodiment. The multi-viewpoint image database 13 holds images from the photographing viewpoints arranged on one straight line in the left-right direction. Hereinafter, the inter-viewpoint interpolation processing unit 120 will be described in detail with reference to FIGS.

FIG. 6 shows an inter-view interpolation processing unit 12 according to this embodiment.
11 is a flowchart showing the flow of the process 0.

First, in step S61, a multi-viewpoint image taken at a coarse viewpoint interval is obtained from the multi-viewpoint image database 13. Next, in step S62, a corresponding point search (motion vector detection) between the viewpoint images is performed. When the corresponding point search is completed, the process proceeds to step S63, in which interpolation processing between the viewpoints of the images is performed to obtain a multi-viewpoint image having sufficiently fine inter-viewpoint intervals.

FIG. 7 is a flowchart of the corresponding point searching process in step S62.

In step S71, the target raster is set as the first raster of each image as an initial setting. Next, in step S72, the raster of interest of each image is read into the work memory, and a first epipolar plane is virtually configured. Here, the j-th epipolar plane refers to each point EPj (x, x,
A set of points EPj (x, i) such that i) satisfies EPj (x, i) = Ni (x, j). Where Ni (x, j) is the i-th image (here,
It represents the x-th pixel value in the j-th line (i = 1 to 4), that is, the value of the pixel whose coordinates are represented by (x, j) in the i-th image. When input devices (cameras) are installed in parallel at equal intervals, all the corresponding points exist on the epipolar plane in a straight line. Therefore, image interpolation may be performed on this straight line.
Therefore, in step S73, a straight line having a corresponding point is extracted, and in step S74, a corresponding point is calculated from the obtained straight line and stored. Steps S75 and S75
Repeat for all rasters according to 76. The specific algorithm is shown below.

(Step A1). Let EPj (x, r) be the pixel of interest, and within the range of m = 0 to k1,

(Equation 1) Find all m that satisfies. However, r = 1. Also,
TH2 is a threshold for finding a corresponding point, and is given according to an error amount allowed between corresponding pixels. Here 12
00 (= 3 × 20 × 20). Also, k1
Is a value determined by the camera interval and the distance to the object,
Here, it is set to 20 (that is, it is assumed that no more than 20 pixels are moved).

(Step A2). Step A1 is repeated for all x of x = 1 to nx, and all values of m corresponding to the value of x are held.

Here, nx represents the number of pixels of the image in the main scanning direction. If EPj (x + m × (ir), i) does not exist, the process is continued assuming that there is no corresponding point in m.

(Step A3). The corresponding point of priority 1 is obtained from the straight line having the slope m obtained in steps A1 and A2 and stored in the memory. When a plurality of corresponding points are obtained, all of them are stored as corresponding points of priority 1 for convenience. Pixels determined as corresponding points are processed pixels.

(Step A4). Steps A1, A2,
With A3 as one cycle, the above cycle is repeated for pixels that have not been processed. In step A1, EPj
If (x + m × (ir), i) has been processed, EPj
(X + m × (ir), i) -EPj (x, r) = 0 and the process is continued. If the corresponding point obtained from the straight line having the slope m in step A3 has already been processed, this point is excluded from the corresponding points. The corresponding points obtained in the n-th cycle are stored as corresponding points of the priority n.

(Step A5). If the number of unprocessed pixels does not decrease even after performing the processing in step A4, r = 2
And the same processing as in steps A1 to A4 is performed using EPj (x, r) as the pixel of interest. However, x = 1 to nx.

(Step A6). If the number of unprocessed pixels does not decrease even after the processing of step A5, r = 3
And set as EPj (x, r) target pixel in step A
The same processing as 1 to A4 is performed. However, x = 1 to nx.

(Step A7). The value of j is incremented by 1 and the process returns to processing 1.

(Step A8). When the processing has been completed up to the final raster, the corresponding point search processing ends.

By performing the processing as described above, it is possible to detect a corresponding point that could not be obtained from two images, and it is also possible to cope with occlusion and the like, so that the accuracy of the corresponding point search is improved.

When the process in step S62 is completed, the process proceeds to step S63, where an image interpolation process is performed. The image interpolation processing is performed using the corresponding points obtained from step S62. A specific algorithm will be described with reference to FIG.

FIG. 9 shows the j-th epipolar plane. a1 and b1 indicate the corresponding points of the priority order 1, and c
Reference numeral 2 indicates a corresponding point of priority order 2. Consider a case where n images are interpolated at equal intervals between input images. Here, for the sake of simplicity, n = 2. When this is considered in the j-th epipolar plane, as shown in FIG. 10, two lines are interpolated between the lines of the epipolar plane, and interpolation is performed on a straight line connecting corresponding points of the epipolar plane of the original image. The pixel value of the obtained line may be set to the average value of the corresponding points. That is, the algorithm is as follows.

(Step B1). Consider a straight line connecting the corresponding points of priority 1, and set the pixel value of the interpolation line on this straight line to the average value of the pixel values of the original image on the straight line.
Taking the corresponding points a1 and b1 in FIG. 10 as an example, the pixel values of points a and b on a straight line connecting the corresponding points are respectively a
The average value of the pixel values indicated by 1, 1 is taken.

(Step B2). After the processing of the corresponding point of the priority 1 is completed, the processing of the corresponding point of the priority 2 is performed next. This process is basically the same as the process in step B1, but does not perform the process on the pixels already interpolated in step B1. This will be described with reference to FIG. The pixels (3, 8) and (2, 9) are normally interpolated by the corresponding point c2, but are already interpolated by the corresponding point of a1 having the priority order 1, so that no processing is performed on this pixel. Not performed. Accordingly, the pixels interpolated by the corresponding point c2 are (9, 2), (8, 3), (6, 5), and (5, 6).
Pixel. (In the example of FIG. 10, occlusion occurs in this portion, but the occlusion problem can be solved by performing such processing.) (Step B3). When the processing of the corresponding point of the priority 2 is completed, the processing of the corresponding point of the priority 3 is started. As in step B2, nothing is performed on the pixels that have already been subjected to the interpolation processing. In the same manner, the processing is performed up to the corresponding point of the final priority.

(Step B4). For pixels that have not been interpolated even after the processing of steps B1 to B3, interpolation is performed from surrounding pixels. As a method at this time, there is a method using an average value of surrounding pixel values, a method using the value of the nearest pixel as it is, and the like.

(Step B5). The processing of steps B1 to B4 is performed for j = 1 to ny, and j2, j3, j
5, j6, j8 and j9 are used to obtain an interpolated image. However,
As shown in FIG. 10, the lines interpolated by the processing of steps B1 to B4 are represented by j2, j3, j5, j6, j8, j9.
Will be expressed as For example, the interpolation image 2 has an interpolation line j
2 (j = 1 to ny) (see FIG. 11). The same applies to the interpolated images 3, 5, 6, 8, and 9.

As described above, by generating the inter-viewpoint interpolated image from the image in the multi-viewpoint image database 13, an image from a viewpoint other than the photographing viewpoint can be obtained on the straight line of the photographing viewpoints. Thus, an image from an arbitrary viewpoint can be generated. Therefore, the multi-view image database 13 does not need to hold multi-view images with sufficiently fine viewpoint intervals, and the storage capacity of the multi-view image database 13 is greatly reduced.

[0064]

Third Embodiment In the first and second embodiments, in order to calculate the degree of distortion by the distortion correction unit 118, the point of interest (or the focus) of the photographed object is stored in the multi-viewpoint image parameter holding unit 16. It is necessary that the approximate distance value to the combined point) be held in advance as the distance 52 to the subject. In the present embodiment, an image processing apparatus that extends the image processing apparatus of the second embodiment and automatically determines the value of the degree of distortion of the distortion correction unit 118 will be described. The configuration of the image processing apparatus of this embodiment is substantially the same as that shown in FIG.

In the present embodiment, the distortion correction unit 118 automatically determines the degree of distortion in cooperation with the inter-viewpoint interpolation processing unit 120. The inter-viewpoint interpolation processing unit 120 notifies the distortion correction unit 118 of the gradient m of the extracted straight line in the detection of the straight line in the corresponding point search (FIG. 6, step S62) (FIG. 7, step S73). The distortion correction unit 118 calculates the distance Td from the camera to the corresponding point (bright point of the object), based on the inclination m of the straight line.
i is obtained by the following equation (10).

Tdi = W · A / (2 · m · tan (θ / 2)) (10) where W is the width (unit pixel) of the image, θ is the angle of view of the camera, and the multi-viewpoint image parameter It is held by the holding unit 16. A is the viewpoint interval between the multi-view images in the multi-view image database 13, and is a photographing viewpoint coordinate system holding unit 15.
Is held in. The expression (10) is clear from FIG. In FIG. 13, the distance A on the viewpoint line 132
The same object 131 from two distant viewpoints 133 and 134
Think about seeing The relationship among the angle of view θ, the focal length F, the pixel size S, and the width W of the image is given by S · W / 2 = Ftan (θ
/ 2). Further, the slope m of the detection straight line is a value indicating the amount of deviation of the corresponding point of interest between adjacent images by the number of pixels. Therefore, from FIG. 13, Tdi / A = F
/ (M · S) is also given. From these two relationships, it can be seen that equation (10) holds.

Next, the distortion correcting section 118 calculates the degree of distortion ki of each corresponding point by the following equation (11) according to the equation (9).

Ki = Tdi / (Tdi−z) (11) Then, instead of equation (9), ki of all corresponding points in the image is used.
Is estimated and calculated as the degree of distortion k.

According to the above-described processing, the present embodiment has an advantage unique to the embodiment that it is not necessary to know in advance the distance between the camera and the subject.

Other methods for calculating the estimated distortion degree k include a method in which the average value of the gradient m is calculated first, and a method in which weighting is performed for each image region to estimate a central subject. Conceivable.

[0071]

[Other Embodiments] In each of the embodiments described above, the multi-viewpoint image photographed in advance is held in the multi-viewpoint image database 13, but the multi-viewpoint image can be captured in real time. By replacing the camera with a multi-view television camera, a real-time arbitrary viewpoint image capturing and displaying system can be realized.

The present invention may be applied to a single image processing apparatus, may be applied to a system device such as a multi-view television, a multi-view video telephone terminal, or a multi-view video conference system, or may be applied to a computer. Or a multifunction device combined with another image processing device.

The image processing apparatus of each embodiment described above
Since the position of the observer's eyes is detected and the image seen by the observer is reconstructed from a plurality of images, when the observer's viewpoint moves, an image corresponding to the viewpoint is smoothly output.

As the multi-view image data composed of a plurality of images having different viewpoint positions, a large number of images obtained from one or more cameras or a large number of images stored in a database can be used. In order to perform image reconstruction, it is desired that the multi-view image data is an image in which the shooting positions are changed at sufficiently small intervals, but even if the shooting position intervals are coarse, the captured images are captured. An image is subjected to interpolation processing to generate an image having a position between adjacent photographing positions as a viewpoint position, and these photographed images and the generated image are used as multi-viewpoint image data, whereby image reconstruction is performed. It can be carried out.

Further, in the image reconstruction, parameters required for image reconstruction are calculated from the position of the eyes of the observer and the type of the image output device, and each pixel of the image to be reconstructed is converted into multi-view image data. This is performed by calculating which pixel of the image corresponds to the calculated parameter, and extracting the corresponding pixel from the multi-view image. In this case, even when the position of the observer's eyes does not match the viewpoint position of any image in the multi-view image data, the correspondence between the pixels is obtained, and therefore, the image reconstruction is favorably performed. .

As an image output device, a stereo display, a lenticular display, etc. can be used in addition to a normal display.

As described above, according to the image processing method and apparatus of the present invention, even if the position of the observer's eye moves in each direction including the front-back direction, an image corresponding to the moved position is obtained. This has the effect of being able to be displayed.

[0077]

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of an image processing apparatus according to a first embodiment.

FIG. 2 is a diagram illustrating a calculation principle of a gaze parameter calculation unit 110 of the image processing apparatus according to the first embodiment.

FIG. 3 is a flowchart illustrating a process of a gaze parameter calculator 110 of the image processing apparatus according to the first embodiment.

FIG. 4 is a diagram illustrating a calculation principle of a virtual viewpoint parameter calculation unit 111 and a pixel position calculation unit 112 of the image processing apparatus according to the first embodiment.

FIG. 5 is a distortion correction unit 118 of the image processing apparatus according to the first embodiment.
FIG.

FIG. 6 is a flowchart illustrating a flow of processing of an inter-viewpoint interpolation processing unit of the image processing apparatus according to the second embodiment.

FIG. 7 is a flowchart illustrating a corresponding point search process performed by the image processing apparatus according to the second embodiment.

FIG. 8 is a diagram illustrating a j-th epipolar plane by the image processing apparatus according to the second embodiment.

FIG. 9 is a diagram illustrating an explanation of an interpolation processing algorithm performed by the image processing apparatus according to the second embodiment.

FIG. 10 is a diagram illustrating an explanation of an interpolation processing algorithm performed by the image processing apparatus according to the second embodiment.

FIG. 11 is a diagram illustrating an explanation of an interpolation processing algorithm performed by the image processing apparatus according to the second embodiment.

FIG. 12 is a diagram illustrating a configuration of an image processing apparatus according to a second embodiment.

FIG. 13 is a diagram illustrating a principle of calculating a distance to an object by the image processing apparatus according to the third embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 11 ... Display screen 12 ... Viewpoint detector 13 ... Multi-viewpoint image database 14 ... Display parameter holding part 15 ... Shooting viewpoint coordinate system holding part 16 ... Multi-viewpoint image parameter holding part 17 ... Viewpoint parameter calculation part 18 ... Image generation part 19 ... Pixel index 110: line-of-sight parameter calculation unit 111: virtual viewpoint parameter calculation unit 112: pixel position calculation unit 113: pixel value calculation unit 114: image display unit 115: update signal 116: pixel value signal 117: pixel value generation unit 118: distortion Correction unit

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Hiroyuki Yamamoto 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Hideyuki Tamura 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon (56) References JP-A-5-37965 (JP, A) JP-A-60-198685 (JP, A) US Pat. No. 5,287,437 (US, A) (58) Fields investigated (Int. Cl. ⁷⁾ G06T 1/00 G06T 15/00 H04N 13/00-15/00

Claims (20)

(57) [Claims] An image input unit for inputting a plurality of images having different viewpoints as viewpoints as multi-viewpoint image data; a viewpoint detecting unit for detecting a position of a viewpoint of an observer; Image reconstructing means for reconstructing an image having the viewpoint as a viewpoint from the multi-view image data based on the viewpoint position, correcting means for correcting distortion of the reconstructed image, and outputting the corrected image to an image An image processing apparatus comprising: an image output unit that outputs an image to an apparatus. 2. The image processing apparatus according to claim 1, wherein the image input unit includes a database in which images previously captured from a plurality of directions are stored, and inputs the images from the database. 3. The image processing apparatus according to claim 1, wherein the image input unit includes one or more cameras and inputs an image from the cameras. 4. Based on the multi-view image data input by the image input means, the image data is subjected to interpolation processing to generate image data of a viewpoint not possessed by the image input means. The image processing apparatus according to claim 1, further comprising an image generating unit that adds the image data to the image data, wherein the image reconstructing unit reconstructs an image from the multi-view image data and the interpolated image data. . 5. The image reconstructing means calculates parameters necessary for reconstructing an image from the position of the viewpoint of the observer and the type of the image output device, and calculates the number of pixels of the reconstructed image using the parameters. 2. The method according to claim 1, further comprising calculating which pixel of the image in the viewpoint image data corresponds to the pixel, extracting the corresponding pixel from the multi-view image data, and reconstructing the image.
An image processing apparatus according to claim 1. 6. The viewpoint by the image input means is on a straight line of viewpoints equidistant from a subject, and the correcting means is configured to determine a distance between the straight line of viewpoints and the viewpoint of the observer detected by the viewpoint detecting means. 2. The image processing apparatus according to claim 1, wherein the image distortion caused by the correction is corrected. 7. The correction means includes means for calculating a distance between the photographed subject and the camera based on a movement amount of a corresponding point of the multi-view image data, and corrects an image distortion based on the distance. The image processing apparatus according to claim 6, wherein: 8. The image processing device according to claim 1, wherein the image output device is a display device. 9. The image processing device according to claim 1, wherein the image output device is a stereo display. 10. The image processing apparatus according to claim 1, wherein the image output device is a lenticular display. 11. An image inputting step of inputting a plurality of images having different viewpoints as viewpoints as multi-viewpoint image data; a viewpoint detecting step of detecting an observer's viewpoint position; Image reconstruction means for reconstructing an image having the viewpoint as a viewpoint from the multi-view image data based on the viewpoint position, a correction step of correcting distortion of the reconstructed image, and outputting the corrected image to an image An image processing method comprising: outputting an image to an apparatus. 12. The image processing method according to claim 11, wherein in the image input step, an image is input from a database that stores images captured in a number of directions in advance. 13. The image processing method according to claim 11, wherein the image input step inputs an image from one or more cameras. 14. Based on the multi-view image data input in the image input step, the image data is subjected to interpolation processing to generate image data of a viewpoint not included in the image input step, and The image processing method according to claim 11, further comprising an image generating step of adding the image data to the image data, wherein the image reconstructing step reconstructs an image from the multi-view image data and the interpolated image data. . 15. The image reconstructing step calculates parameters necessary for reconstructing an image from a position of a viewpoint of an observer and a type of an image output method, and calculates a number of pixels of the reconstructed image using the parameters. 12. The image processing method according to claim 11, wherein a pixel corresponding to an image in the viewpoint image data is calculated, and the corresponding pixel is extracted from the multi-view image data to reconstruct the image. 16. The viewpoint in the image input step is on a straight line of viewpoints equidistant from a subject, and the correcting step includes:
12. The image processing method according to claim 11, wherein an image distortion caused by a distance between the viewpoint arrangement straight line and an observer's viewpoint detected in the viewpoint detection step is corrected. 17. The correcting step includes a step of calculating a distance between the photographed subject and the camera based on a movement amount of a corresponding point of the multi-viewpoint image data, and correcting an image distortion based on the distance. 17. The image processing method according to claim 16, wherein: 18. The image processing method according to claim 11, wherein the image output step displays an image on a display device. 19. The image output step of displaying an image on a stereo display device.
2. The image processing method according to 1. 20. The image processing method according to claim 11, wherein the image output step displays an image on a lenticular display device.