JP3032414B2 - Image processing method and image processing apparatus - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method and an image processing apparatus for inputting a plurality of images having different viewpoints and outputting an image having a viewpoint position corresponding to the current eye position of an observer.

[0002]

2. Description of the Related Art Conventionally, as a device for stereoscopically displaying an image viewed from a plurality of viewpoints, there are a stereo display and a lenticular display. In a stereo display, images obtained from two cameras are alternately displayed at high speed. The observer can view the image three-dimensionally by using shutter glasses or polarized glasses synchronized with the switching. In addition, as shown in FIG. 13, the lenticular display expresses pixel positions (1, 1) of A as A (1, 1), where images from four cameras are A, B, C, and D, for example. By arranging images A to D on the liquid crystal display 91 in pixel units and attaching a lenticular sheet 92 to the front surface of the liquid crystal display 91, images from four viewpoints can be three-dimensionally expressed.

[0003]

However, in the above-described conventional stereo display, only a three-dimensional image in a shooting direction of a camera at the time of shooting an image can be observed. That is, since an object is generally photographed with two cameras fixed, even if the observer moves his or her viewpoint (eye position), the same image can be seen, and the viewpoint movement on the observer side is not reflected. There is a problem of lack of feeling. In addition, the lenticular display can cope with the movement of the observer's viewpoint position in the left-right direction, but it can see the image viewed from any of the multiple cameras in a discrete manner and cannot cope with continuous viewpoint movement. In addition, it was not possible to move the viewpoint in the front-back direction. The viewpoint movement in the front-back direction is performed in the case of stereoscopic viewing based on an image created by computer graphics, but this is simple as a computer graphics image and a point in the image. This is under a special situation where the coordinate values in the corresponding three-dimensional space are all clear. When stereoscopically viewing an image captured by a camera, the movement of the viewpoint in the front-back direction has hardly been considered so far.

An object of the present invention is to provide an image processing method and an image processing apparatus capable of providing an image viewed from the moved position in real time when the position of the observer's eyes moves in various directions including the front-back direction. Is to provide.

[0005]

An image processing method according to the present invention is a multi-view image input method for inputting, as multi-view image data, a plurality of images obtained by shooting a subject from different positions as viewpoints, together with parameters at the time of shooting the images. A viewpoint input step of inputting a viewpoint position and a line-of-sight direction for observing the subject; and an image obtained when the line-of-sight direction is viewed from the viewpoint position input in the viewpoint input step, in the multi-viewpoint image data. image data by selecting, the pixel position information corresponding to the viewing direction of the image, calculated in pixel units based on the parameters at the time of photographing of the image, and outputs in units of pixels as the corresponding pixel information of the display screen from the And an image output step of outputting the reconstructed image data to the image output device.

[0006] An image processing apparatus according to the present invention comprises a multi-view image input means for inputting a plurality of images of a subject taken from different positions as viewpoints as multi-view image data together with parameters at the time of capturing the images, A viewpoint input unit for inputting a viewpoint position and a line-of-sight direction to observe, and selecting an image when the line-of-sight direction is viewed from the viewpoint position input by the viewpoint input unit from the multi-viewpoint image data; the pixel positions <br/> information corresponding to the viewing direction, the pixel single based on the parameters at the time of photographing of the image of
Calculated in place, and an image reconstruction means for reconstructing image data by <br/> output in units of pixels as the corresponding pixel information of the display screen, an image output that outputs the image data reconstructed in the image output device Means.

[0007]

Since the position of the observer's eyes is detected and the image seen by the observer is reconstructed from a plurality of images, an image corresponding to the movement of the observer's viewpoint can be output smoothly. .

[0008] As multi-view image data composed of a plurality of images having different viewpoint positions, a large number of images obtained from one or more cameras or a large number of images stored in a database can be used. In order to reconstruct an image, it is desired that the multi-view image data is an image in which the shooting positions are changed at sufficiently small intervals. However, as described later, when the intervals between the shooting positions are coarse, Also, by performing interpolation processing on the captured image to generate an image with the position between adjacent imaging positions as the viewpoint position, and by using these captured image and the generated image as multi-view image data, Image reconstruction can be performed.

In the image reconstruction, parameters required for image reconstruction are calculated from the position of the observer's eyes and the type of image output device, and each pixel of the image to be reconstructed is converted to an image of multi-viewpoint image data. This is performed by calculating which pixel corresponds to the calculated parameter, and extracting the corresponding pixel from the multi-view image data. In this case, even when the position of the observer's eye does not match the viewpoint position of any image in the multi-view image data, the correspondence between pixels is obtained, and therefore, image reconstruction is favorably performed. .

As an image output device, a stereo display, a head mounted display, a lenticular display, etc. can be used in addition to a normal display.

[0011]

Next, embodiments of the present invention will be described with reference to the drawings.

<< First Embodiment >> FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment of the present invention. This image processing device is for displaying a reconstructed image to a user (observer) through a fixedly installed display screen 1 as described later. That is, the image processing apparatus includes a viewpoint detector 2 for detecting a position of an eye of a user looking at a display screen 1, and a multi-view image database 3 which is a database for holding multi-view image data.
A display parameter 4 for storing parameters relating to the display screen 1, a shooting viewpoint coordinate system holding unit 5 for storing, for each image in the multi-view image database 3, coordinate system data of a shooting position when the image is shot. A multi-viewpoint image parameter holding unit 6 for holding image parameters of images in the multi-viewpoint image database 3, and calculating a viewpoint parameter based on a signal from the viewpoint detector 2 when the viewpoint position of the user changes. A viewpoint parameter calculation unit 7 that outputs an update signal 15, an image generation unit 8 that generates an image corresponding to the user's viewpoint, an image display unit 14 that displays an image generated by the image generation unit 8 on a display screen, A pixel value generation unit 17 that calculates each pixel value for image reconstruction and outputs the pixel value as a pixel value signal 16 to the image generation unit 8.

Here, the images in the multi-view image database 3 are images obtained by photographing an object to be displayed from a number of viewpoints arranged on a plane at sufficiently fine intervals. Correspondingly, the data held in the shooting viewpoint coordinate system holding unit 5 is data of a coordinate system representing a plane in which the viewpoints are arranged when these images are shot. The image generation unit 8 is configured to generate an image when the update signal 15 is received, and from the image generation unit 8, attention is paid to the reconstructed image, that is, the image on the display screen 1. A pixel index signal 9 indicating the coordinates of the pixel to be output is output. The pixel index signal 9 is used when reconstructing an image.
The pixels are sequentially output so as to make one cycle through all the pixels of the reconstructed image.

Next, the configuration of the pixel value generator 17 will be described. The pixel value generation unit 17 includes a line-of-sight parameter calculation unit 10 that calculates a line-of-sight direction corresponding to the pixel indicated by the pixel index signal 9 from the viewpoint parameters and the display parameters.
A virtual viewpoint parameter calculation unit 11 for calculating a virtual viewpoint,
A pixel position calculating unit 12 that calculates a pixel position corresponding to a line-of-sight direction of an image at a virtual viewpoint, and a pixel that calculates a corresponding pixel value from an image in the multi-view image database 3 based on the pixel position and the virtual viewpoint parameters A value calculation unit 13 is provided. Here, the virtual viewpoint is an intersection point between the line of sight indicated by the line of sight parameter and the plane of the arrangement of the imaging viewpoints represented by the imaging viewpoint coordinate system. In addition, the pixel position calculation unit 12
The pixel position is calculated using a line-of-sight parameter, a shooting viewpoint coordinate system, a virtual viewpoint parameter, and a multi-viewpoint image parameter. The pixel value calculated by the pixel value calculation unit 13 becomes a pixel value signal 16.

Next, the operation of this embodiment will be described. First, an outline of this operation will be described.

When the user looking at the display screen 1 changes his or her head position and moves his / her viewpoint, the signal output from the viewpoint detector 2 changes, and the viewpoint parameter calculation unit 7 receives this change and updates the update signal 15. Send to the image generation unit 8. Upon receiving the update signal 15, the image generator 8 starts generating a new image corresponding to the viewpoint movement. The creation of this new image
The image generation unit 8 sequentially outputs the pixel index signals 9 for all the pixels, and sequentially obtains the pixel value signals 16 for each pixel from the pixel value generation unit 17. Here, the operation of the pixel value generation unit 17 will be described.

In the pixel value generation unit 17, first, the line-of-sight parameter calculation unit 10 obtains the viewpoint parameters from the viewpoint parameter calculation unit 7 and the display parameters from the display parameter holding unit 4, and corresponds to the input pixel index signal 9. Gaze parameters to be calculated. Next, the virtual viewpoint parameter calculation unit 11 obtains the shooting viewpoint coordinate system from the shooting viewpoint coordinate system holding unit 5 and intersects the intersection (virtual) of the line of sight indicated by the line-of-sight parameter and the plane of the shooting viewpoints indicated by the shooting viewpoint coordinate system. (Viewpoint) is calculated. on the other hand,
The pixel position calculation unit 12 is a multi-view image parameter holding unit 1
7, and obtains the multi-viewpoint image parameters from the line-of-sight parameters, the shooting viewpoint coordinate system, and the virtual viewpoint parameters.
A pixel position corresponding to the line-of-sight direction of the image at the virtual viewpoint is calculated. Then, the pixel value calculation unit 13 calculates the multi-view image database 3 based on the pixel position and the virtual viewpoint parameters.
Then, the corresponding pixel value signal 16 is calculated from the images in the above. In this way, the pixel value generation unit 17 calculates the pixel value signal 16 for each input pixel index signal 9 and outputs the pixel value signal 16 to the image generation unit 8.

When the pixel value signal 16 is obtained from the pixel value calculation unit 13 for all the pixels, the image generation unit 8 sends it to the image display unit 14. The image display unit 14 displays the image corresponding to the new viewpoint generated in this manner on the display screen 1. As a result, a series of image generation operations accompanying the movement of the viewpoint of the user is completed. As will be apparent from the following description, when the user moves his / her viewpoint from front to back, up, down, left and right, even if that is a place other than the viewpoint where each image held in the multi-view image database 3 is captured, The image of the object in accordance with the movement of the viewpoint can be viewed through the display screen 1.

Next, the processing of each section will be described in detail. However, for the sake of simplicity, the following description assumes no vertical parallax and considers only horizontal parallax.

First, the process of calculating the line-of-sight parameters in the line-of-sight parameter calculation unit 10 will be described. FIG. 2 is a diagram showing the calculation principle of the eye-gaze parameter calculation unit 10, and FIG. 3 is a flowchart showing the processing of the eye-gaze parameter calculation unit 10.

In FIG. 2, an end point 2 of the display screen 1 is shown.
1. The position vector of the pixel position of interest 23 on the display screen 1 and the position vector of the user's viewpoint position 24 are respectively

[0022]

[Outside 1] And The vector 22 is a vector whose length is the pixel pitch of the display screen 1 and whose inclination matches the inclination of the display screen 1.

[0023]

[Outside 2] And The line of sight 25 corresponds to the pixel position 23 of interest,
A vector 26 representing the inclination of the line of sight 25 is

[0024]

[Outside 3] And

First, a viewpoint parameter is obtained from the viewpoint parameter calculator 7 (step 31). The viewpoint parameter is the viewpoint position 24 of the user in FIG. In addition, display parameters are acquired from the display parameter storage 4 (step 32). The display parameters are represented by an end point 21 of the display screen 1 and a display screen vector 22. The display screen vector 22 is determined from the inclination, the actual size, and the pixel size of the display screen 1. Next, based on the arrangement shown in FIG. 2, the target pixel position 23 on the display screen 1 is calculated by the following equation (1) in correspondence with the pixel index signal 9 (step 3).
3). Here, the pixel index signal 9 is i.

[0026]

(Equation 1) After obtaining the pixel position 23, a line-of-sight parameter corresponding to the direction of the pixel position 23 when viewed from the viewpoint position 24 is obtained (step 34), and the processing for obtaining the line-of-sight parameter is ended. The line-of-sight parameter is a set of a viewpoint position 24 and a line-of-sight vector 26.

[0027]

[Outside 4] Is represented as Since the line of sight 25 is a straight line passing through the two points of the pixel position 23 and the viewpoint position 24, the line of sight vector 2
6 can be calculated by the following equation (2).

[0028]

(Equation 2) Next, processing in the virtual viewpoint parameter calculation unit 11 and the pixel position calculation unit 12 will be described. FIG. 4 is a diagram illustrating the principle of calculating virtual viewpoint parameters and pixel positions.

As described above, in the present embodiment, the viewpoints at the time of photographing each image in the multi-viewpoint image database 3 are arranged on the same plane. The cross section of the plane of this arrangement is shown as a viewpoint arrangement straight line 41 in the figure. Virtual viewpoint 42
Is represented as the intersection between the line of sight 25 and the line of viewpoints 41, and its position vector is

[0030]

[Outside 5] And In addition, a vector 43 representing the inclination of the viewpoint arrangement straight line 41 is represented by

[0031]

[Outside 6] And the position vector of the end point 44 of the line of viewpoints 41 is

[0032]

[Outside 7] And The field of view 45 represents the angle of view θ at the virtual viewpoint 42, the vector 46 is the length of the focal length of the camera that captured the image of the multi-viewpoint data, and the inclination of the camera is the inclination.

[0033]

[Outside 8] That. A virtual imaging plane 47 at the virtual viewpoint 42, and a pixel position 4 which is an intersection of the virtual imaging plane 47 and the line of sight 25
8 is determined. The position vector of pixel position 48 is

[0034]

[Outside 9] Indicated by Further, a vector 49 whose length is one pixel pitch of the virtual imaging surface 47 and whose inclination matches the inclination of the virtual imaging surface 47 (usually perpendicular to the focus vector 46).

[0035]

[Outside 10] And

Here, the viewpoint arrangement vector 43 and the end point 44 of the viewpoint arrangement straight line 41 are held in the imaging viewpoint coordinate system holding unit 5 as values representing the imaging viewpoint coordinate system. The focus vector 46 and the imaging plane vector 49 are held in the multi-viewpoint image parameter holding unit 6 as multi-viewpoint image parameters. The size of the imaging plane vector 49 is equal to the actual cell size of the imaging plane (the size of one pixel).

When each point and each vector are linked as described above, the virtual viewpoint 42 is expressed by the following equations (3) and (4).

[0038]

(Equation 3) Here, t is a virtual viewpoint parameter as a parameter that uniquely represents the virtual viewpoint. α is a coefficient in the line-of-sight direction.
The virtual viewpoint parameter calculation unit 11 calculates t by solving equations (3) and (4),

[0039]

[Outside 11] Is calculated.

The pixel position 48 is represented by the following equations (5) and (6).

[0041]

(Equation 4) Here, i ′ is a pixel position parameter as a parameter that uniquely represents the pixel position 48. β is a coefficient of the line-of-sight direction. The pixel position calculation unit 12 calculates a pixel position parameter i ′ by solving equations (5) and (6), and outputs the calculated pixel position parameter i ′.

Next, the processing of the pixel value calculating section 13 will be specifically described. In the present embodiment, the multi-view images stored in the multi-view image database 3 are images captured at sufficiently small viewpoint intervals. Therefore, first, as an approximate image of the image from the virtual viewpoint 42 indicated by the virtual viewpoint parameter calculated by the virtual viewpoint parameter calculation unit 11, an image taken from the viewpoint closest to the virtual viewpoint 42 is stored in the multi-view image database. Find among three. Next, in this image, the pixel position 48 calculated by the pixel position calculation unit 12 is used.
The pixel value at the position closest to is obtained, and this is set as the output pixel value signal 16.

In the above, for the sake of simplicity, the processing of each unit in the case where the vertical parallax is omitted has been described. However, if a vertical multi-viewpoint image is prepared, the vertical parallax is obtained in the same manner. In consideration of the above, a binocular stereoscopic display device capable of moving the viewpoint forward, backward, up, down, left and right is provided. The display screen 1 and the image display unit 8 use a stereoscopic display screen and a stereoscopic image display unit that can perform binocular stereoscopic viewing such as a lenticular system or an eyeglass system, and the viewpoint parameter calculation unit 7 uses the left and right eyes. A binocular stereoscopic display device capable of moving the viewpoint forward, backward, up, down, left, and right by calculating viewpoint parameters corresponding to the position, and generating an image to be presented to each of the left and right eyes by the image generation unit 8 in response to the calculation; Become.

<< Second Embodiment >> Next, an image processing apparatus capable of freely performing a viewpoint movement display even when the viewpoint interval of each image held in the multi-view image database 3 is not sufficiently small will be described. . This image processing device is a multi-view image database 3 of the image processing device of the first embodiment described above.
The configuration is such that an inter-viewpoint interpolation processing unit is provided between the image processing unit and the pixel value calculation unit 13. The inter-viewpoint interpolation processing unit generates a group of images with sufficiently fine inter-viewpoint intervals by performing interpolation processing from each image captured at the coarse viewpoint intervals in the multi-viewpoint image database 3. By using an image with a sufficiently fine inter-viewpoint interval, an image corresponding to a change in the user's viewpoint is generated as in the first embodiment.

Hereinafter, the inter-view interpolation processing unit will be described in detail. Here, in order to make the description easy to understand, a case will be described in which the vertical parallax is not considered. At this time, the multi-viewpoint image database 3 holds images from the respective shooting viewpoints arranged on one straight line in the left-right direction.

FIG. 5 is a flowchart showing the flow of the processing of the inter-viewpoint interpolation processing section. Multi viewpoint image database 3
An image photographed at a coarse viewpoint interval is obtained from among (step 51). Next, a corresponding point search (motion vector detection) is performed between the acquired images (step 52). When the corresponding point search is completed, an interpolation process based on the photographing viewpoint is performed to generate an image with sufficiently fine inter-viewpoint intervals (step 53), and the process ends.

First, the corresponding point search will be described with reference to the flowchart of FIG.

As an initial setting, the raster of interest is set to the first raster of each image (step 61). Next, the raster of interest of each image is read into a work memory (not shown) (step 62) to virtually configure the first epipolar plane. Here, the j-th epipolar plane refers to each point EP _j (x, i) on the image plane as shown in FIG.

[0049]

## EQU5 ## This is a set of points EP _j (x, i) satisfying EP _j (x, i) = N _i (x, j). Here, N _i (x, j) is the i-th image (here, i = 1 to
4) The x-th pixel value in the j-th line, that is, i
In the second image, the coordinates represent the value of the pixel represented by (x, j). When image input devices such as cameras are installed in parallel at equal intervals, all the corresponding points exist on the epipolar plane in a straight line. Therefore, image interpolation may be performed on this straight line.

Then, a straight line having a corresponding point is extracted (step 63), and a corresponding point is calculated from the obtained straight line.
It is stored (step 64). A specific corresponding point calculation algorithm is shown below.

Procedure A1: EP _j (x, 1) is set as a target pixel, and within a range of m = k _{1 to} k ₁ + k _2.

[0052]

(Equation 6) Find all m that satisfies. Here, TH2 is a threshold for finding a corresponding point, and here, 1600 (= 4 × 2
0 ² ). In addition, k ₁ is a value depending on a method of capturing an input image, and k ₁ = 0 when cameras are arranged at equal intervals and the optical axis is captured in parallel. k ₂ is a value determined by the camera interval and the distance to the object, and here 2
It is set to 0 (that is, it is assumed that 20 pixels or more do not move).

[0053] Procedure A2: for all x of x = 1 to n _x, repeat steps A1, holds all the values of m corresponding to the value of x. However, n _x represents the number of pixels in the main scanning direction of the image. If EP _j (x + m × (i−1), i) does not exist, the process is continued assuming that there is no corresponding point in m.

Procedure A3: A corresponding point of priority 1 is obtained from the straight line having the slope m obtained by procedures A1 and A2 and stored in the memory. When a plurality of corresponding points are obtained, all of them are stored as corresponding points of priority 1 for convenience. Pixels obtained as corresponding points are processed pixels.

Step A4: Steps A1 to A3 are defined as one cycle, and the above cycle is repeated for unprocessed pixels. In procedure A1, EP _j (x + m × (i−1), i)
Has been processed, EP _j (x + mx × (i−1), i) −E
The processing is continued with P _j (x, 1) = 0. If the corresponding point obtained from the straight line having the slope m in step A3 has already been processed, this point is excluded from the corresponding points. The corresponding points obtained in the n-th cycle are stored as the corresponding points of the priority n.

Procedure A5: If the number of unprocessed pixels does not decrease even after the processing of procedure A4, the procedures A1 to A4 are executed with EP _j (x, 2) as the pixel of interest. However,
It is _x = 1~n x.

Step A6: If the number of unprocessed pixels does not decrease even after the processing of step A5, steps A1 to A4 are executed with EP _j (x, 3) as the pixel of interest. However,
It is _x = 1~n x.

Step A7: Similarly, the value of j is increased by one, and steps A1 to A4 are repeated.

Step A8: When the above processing is performed up to the final raster, the corresponding point search processing is completed.

By performing the processing as described above, 2
The corresponding points that could not be obtained from the images can be detected,
Further, since it is possible to cope with occlusion and the like, the accuracy of the corresponding point search is improved.

Next, the inter-viewpoint interpolation processing of an image (step 53 in FIG. 5) will be described. This interpolation processing is performed on the corresponding points searched in the corresponding point search processing described above. Here, a specific algorithm of this interpolation processing will be described with reference to FIG.

FIG. 8 shows the j-th epipolar plane. a1 and b1 indicate the corresponding points of the first priority, and c2 indicates the corresponding points of the second priority. Between the input images,
Consider a case where n images are generated at equal intervals. Here, for the sake of simplicity, n = 2. When this is considered in the j-th epipolar plane, as shown in FIG. 9, two lines (interpolation lines j-2, j-3, j-5, j-6, j-8, j-9), and the pixel value of the interpolated line on the line connecting the corresponding points of the epipolar plane of the original image is calculated as the average of the pixel values of the corresponding points. Set. That is, the following procedure is performed.

Procedure B1: Consider a line segment connecting the corresponding points of priority 1, and set the pixel value of the interpolation line on this line segment to the average value of the pixel values of the original image on the line segment. Taking the corresponding points a1 and b1 in FIG. 9 as an example, the pixel values of points a and b on the line connecting the corresponding points are the average values of the pixel values indicated by a1 and b1.

Procedure B2: When the processing of the corresponding point of priority 1 is completed, the processing of the corresponding point of priority 2 is performed next.
This processing is basically the same as the procedure B1, but does not perform the processing on the pixels already interpolated in the procedure B1. This will be described with reference to FIG. Pixel (3,8)
Although (2, 9) is normally interpolated by the corresponding point c2, no processing is performed on this pixel because it is already interpolated by the corresponding point of a1 of priority 1. Therefore, the pixels interpolated by the corresponding point c2 are (9, 2), (8, 3),
There are four pixels (6,5) and (5,6). (In the example of FIG. 9, occlusion occurs in this portion, but the occlusion problem can be solved by performing such processing.) Procedure B3: When the processing of the corresponding point of priority 2 is completed, Next, the process for the corresponding point of priority 3 is started. As in the procedure B2, nothing is performed on the pixels that have already been subjected to the interpolation processing. In the same manner, processing is performed up to the corresponding point of the final priority.

Procedure B4: For pixels that have not been interpolated even after completing the processing of procedures B1 to B3, interpolation is performed from surrounding pixels. As a method at this time, there is a method using an average value of surrounding pixel values, a method using the value of the nearest pixel as it is, and the like.

[0066] Step B5: j = about 1 to n _y, performs the processing of steps B1 to B4, the interpolation line j-2, j-3, j-5, j-
Interpolated images are obtained using 6, j-8 and j-9. For example, the interpolation image # 2 shown in FIG. 10, an interpolation line _{j-2 (j = 1~n y} ) can be constituted by Namiberu. Interpolated images # 3, # 5, #
The same applies to 6, # 8, and # 9.

As described above, by generating an inter-viewpoint interpolated image from the image in the multi-viewpoint image database 3, an image from a viewpoint other than the photographing viewpoint can be obtained on a straight line of the photographing viewpoints. Thus, an image from an arbitrary viewpoint can be generated. Therefore, the multi-view image database 3 does not need to hold multi-view images with sufficiently fine viewpoint intervals, and the storage capacity of the multi-view image database 3 is greatly reduced.

In the above description, the parallax in the vertical direction is omitted.
By holding the images captured from each of the viewpoints with a coarse lattice spacing on the plane, first interpolate these images in the horizontal direction, and then interpolate them in the vertical direction. An image is generated in consideration of the parallax in the direction.

<< Third Embodiment >> Next, the display screen 1
An example in which a so-called head mounted display (HMD) type image display device fixed to a user's head will be described. The image processing apparatus according to the present embodiment is obtained by modifying the processing of the line-of-sight parameter calculation unit 10 of the image processing apparatus according to the first embodiment as follows so as to conform to the HMD. A case where the parallax in the vertical direction is not considered will be described.

FIG. 11 is a diagram showing the calculation principle of the gaze parameter calculator 10 in this embodiment. In the figure,
The vector 22 is a vector whose length is the pixel pitch of the display screen 1 and whose inclination matches the inclination of the display screen 1, that is,

[0071]

[Outside 12] It is. Further, the position vector of the pixel position of interest 23 on the display screen 1 and the position vector of the user's viewpoint position 24 are

[0072]

[Outside 13] And Further, the vector 111 is a vector from the viewpoint position 24 to the center point of the display screen 1, that is,

[0073]

[Outside 14] It is. A vector 26 is a line-of-sight vector representing the inclination of the line of sight 25 corresponding to the pixel position 23 of interest.

[0074]

[Outside 15] It is.

In the display device of the HMD type, the viewpoint detector 2 is integrated, and in addition to the position of the user's viewpoint 24, the inclination in the front direction, that is, the inclination of the front vector 111 can be detected. The inclination of the display screen vector 22 is determined from the inclination of the front vector 111 (usually a right angle). On the other hand, the distance from the viewpoint position 24 to the display screen 1, ie, the length of the front vector 111, and the pixel pitch, ie, the length of the display screen vector 22, are fixed values determined by the shape of the HMD. It is stored in the parameter storage unit 4. Therefore, the pixel position 23 of interest and the line-of-sight vector 2
6 can be calculated by the following equation. However, pixel index 9
Is i.

[0076]

(Equation 7) With the above configuration, in the HMD type display device,
An arbitrary viewpoint movement display can be performed from an image in the multi-view image database. Further, even if the display screen 1 is not fixed to the head, a cockpit type display device is used in which the relative positional relationship between the fixedly installed display screen 1 and the user's viewpoint position 24 is fixed. However, the image processing apparatus can perform an arbitrary viewpoint movement display by the same processing of the line-of-sight parameter calculation unit 10 as in the present embodiment.
In this case, instead of the viewpoint detector 2, a viewpoint position input device that operates the viewpoint position 24 in the reference coordinates with a handle or the like may be used.

Fourth Embodiment In each of the above embodiments, the viewpoints of the images held in the multi-view image database 3 are arranged on a plane. However, the present invention is not limited to this. Absent. For example, it is also possible to use an image in which the viewpoints are arranged on a cylindrical surface at sufficiently small intervals and an image of the inside of the cylindrical surface is taken (image of the entire circumference). The present embodiment is directed to this all-around image.

The configuration of this embodiment is such that the processing contents of the virtual viewpoint parameter calculation unit 11 and the pixel position calculation unit 12 of the first embodiment are replaced with the processing described below.
A case where the parallax in the vertical direction is omitted will be described. FIG.
FIG. 2 is a diagram illustrating a principle of calculating virtual viewpoint parameters and pixel positions.

The cross-section of the cylinder on which the viewpoints are arranged when the images constituting the multi-viewpoint image database 3 are photographed is defined as a viewpoint arrangement circle 121. The intersection between the line of sight 25 and the circle of viewpoints 121 is a virtual viewpoint 42, and its position vector is

[0080]

[Outside 16] And The position vector of the center 122 of the viewpoint line circle 121 is

[0081]

[Outside 17] And a vector 123 from the center 122 to the virtual viewpoint 42.
To

[0082]

[Outside 18] And Also, a visual field 45 of the angle of view θ at the virtual viewpoint 42
Is defined. Vector 46 is a vector whose length is the focal length of the camera that captured the image and whose tilt is the camera tilt, ie,

[0083]

[Outside 19] It is. Considering a virtual imaging plane 47 at the virtual viewpoint 42, a pixel position 48 which is an intersection of the virtual imaging plane 47 and the line of sight 25 is determined. The position vector of this pixel position 48 is

[0084]

[Outside 20] And In addition, the vector 49 has a length corresponding to one pixel pitch of the virtual imaging surface 47, and a vector whose inclination matches the inclination of the virtual imaging surface 47 (usually perpendicular to the focus vector 46).

[0085]

[Outside 21] It is. Since it is a full circumference type, there are two intersections between the circle 121 and the line of sight 25, and the intersection that is not the virtual viewpoint 42 is the fake virtual viewpoint 124.

Here, the center 122 of the viewpoint arrangement circle 121
And the radius R are held in the shooting viewpoint coordinate system holding unit 5 as values representing the shooting viewpoint coordinate system. The virtual viewpoint vector 123 is defined by the following equation, where t is a parameter that uniquely represents the virtual viewpoint 42.

[0087]

(Equation 8) The focus vector 46 and the imaging plane vector 49 are
The multi-viewpoint image parameters are held in the multi-viewpoint image parameter holding unit 6. The size of the imaging plane vector 49 is determined from the actual size and the pixel size of the imaging plane.

Next, the calculation processing of the virtual viewpoint parameters will be described. According to the geometry shown in FIG. 12, the virtual viewpoint 42 is expressed by the following two equations.

[0089]

(Equation 9) Here, α is a coefficient in the line-of-sight direction. T is calculated by solving the two equations. At this time, in order to distinguish the fake virtual viewpoint 124 from the correct virtual viewpoint 42, the solution is examined so that the following equation is satisfied.

[0090]

(Equation 10) After that, the pixel position may be calculated in the same manner as in the first embodiment, in consideration of the arrangement shown in FIG. With the above configuration, it is possible to obtain an image processing apparatus capable of moving an arbitrary viewpoint of an entire circumference using an image of an entire circumference. In this image processing apparatus, a photographed target object can be viewed from any direction and any distance at 360 degrees.

<< Other Embodiments >> In each of the above embodiments,
Although the image captured in advance is held in the multi-view image database 3, it is replaced with a multi-view TV camera capable of capturing images from multiple viewpoints in real time, thereby realizing arbitrary time in real time. It becomes a viewpoint image shooting and display system.

The present invention may be applied to a stand-alone image processing apparatus, may be applied to a system device such as a multi-view television, a multi-view video telephone terminal, or a multi-view video conference system. The present invention can also be applied to a multifunction device combined with a computer or another image processing device.

[0093]

As described above, the present invention detects the position of the observer's eye and reconstructs the image seen by the observer from a plurality of images, so that when the observer's viewpoint moves. An image corresponding to this can be output smoothly, and the viewpoint can be moved in the front-rear direction, which could not be handled conventionally.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a principle of calculating a line-of-sight parameter in the first embodiment.

FIG. 3 is a flowchart illustrating a process of a gaze parameter calculating unit.

FIG. 4 is a diagram illustrating a principle of calculating a virtual viewpoint parameter and a pixel position in the first embodiment.

FIG. 5 is a flowchart illustrating a flow of processing of an inter-viewpoint interpolation processing unit in the image processing apparatus according to the second embodiment of the present invention.

FIG. 6 is a flowchart illustrating a corresponding point search process.

FIG. 7 is a diagram showing a j-th epipolar plane.

FIG. 8 is a diagram for explaining an interpolation processing algorithm.

FIG. 9 is a diagram for explaining an interpolation processing algorithm.

FIG. 10 is a diagram for explaining an interpolation processing algorithm.

FIG. 11 is a diagram illustrating a principle of calculating a line-of-sight parameter in a third embodiment.

FIG. 12 is a diagram illustrating a principle of calculating a virtual viewpoint parameter and a pixel position in a fourth embodiment.

FIG. 13 is a schematic perspective view showing a conventional lenticular display.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 display screen 2 viewpoint detector 3 multi-viewpoint image database 4 display parameter storage unit 5 shooting viewpoint coordinate system storage unit 6 multi-viewpoint image parameter storage unit 7 viewpoint parameter calculation unit 8 image generation unit 9 pixel index signal 10 gaze parameter calculation unit 11 Virtual viewpoint parameter calculation unit 12 pixel position calculation unit 13 pixel value calculation unit 14 image display unit 15 update signal 16 pixel value signal 17 pixel value generation unit 23 pixel position 24 viewpoint position 25 line of sight 41 viewpoint line 42 virtual viewpoint 47 virtual Imaging surface

Claims (20)

(57) [Claims] 1. A multi-viewpoint image inputting step of inputting a plurality of images of a subject taken at different positions as viewpoints as multi-viewpoint image data together with parameters at the time of photographing the images, a viewpoint position and a line of sight for observing the subject. A viewpoint inputting step of inputting a direction, and selecting an image when the gaze direction is viewed from the viewpoint position input in the viewpoint inputting step from the multi-viewpoint image data, corresponding to the gaze direction of the image. Pixel
The position information is stored in the image based on the parameters at the time of shooting the image.
Calculated in element unit, pixel isolated as the corresponding pixel information of the display screen
The image processing method includes an image reconstruction step of reconstructing image data, and an image output step of outputting the image data reconstructed in the image output apparatus by outputting at positions. 2. The image processing method according to claim 1, wherein the multi-viewpoint image inputting step is a step of inputting an image from a database storing images previously photographed from a number of directions. 3. The image processing method according to claim 1, wherein the multi-viewpoint image inputting step is a step of inputting an image from one or more cameras. 4. A multi-viewpoint image input step performs an interpolation process on a plurality of images photographed by a camera to generate an image having a viewpoint position different from a photographing position of the camera, and generates the image. 2. The image processing method according to claim 1, further comprising the step of inputting a photographed image and a photographed image. 5. An inter-view interpolation processing step of generating a new image by performing an interpolation process on an image included in the multi-view image data and adding the new image to the multi-view image data. Item 2. The image processing method according to Item 1. 6. A view input step, the image processing method according to claim 1 together with the position of the observer's eye is a step of detecting a gaze direction of the observer. 7. An image reconstructing step calculates parameters necessary for reconstructing an image from the position of an eye of an observer and the type of an image output device, and uses the parameters to convert each pixel of the reconstructed image into a multi-viewpoint image. 2. The image processing method according to claim 1, further comprising calculating which pixel of the image in the image data corresponds to the pixel, extracting the corresponding pixel from the multi-view image data, and reconstructing the image data . 8. The image processing method according to claim 1, wherein the image output device is a stereo display. 9. The image processing method according to claim 1, wherein the image output device is a head mounted display. 10. The image processing method according to claim 1, wherein the image output device is a lenticular display. 11. Multi-view image input means for inputting, as multi-view image data, a plurality of images of a subject taken at different positions as viewpoints, together with parameters at the time of photographing the images, a viewpoint position and a line of sight for observing the subject. Viewpoint input means for inputting a direction, and selecting an image when the gaze direction is viewed from the viewpoint position input by the viewpoint input means from the multi-view image data, corresponding to the gaze direction of the image Pixel
The position information is stored in the image based on the parameters at the time of shooting the image.
Calculated in element unit, pixel isolated as the corresponding pixel information of the display screen
An image processing apparatus, comprising: an image reconstructing unit that reconstructs image data by outputting the image data in different places; and an image output unit that outputs the reconstructed image data to the image output device. 12. The image processing apparatus according to claim 11 , wherein the multi-viewpoint image input means inputs an image from a database storing images previously photographed from a number of directions. 13. The image processing apparatus according to claim 11 , wherein the multi-view image input means inputs an image from one or more cameras. 14. A multi-view image input unit performs interpolation processing on a plurality of images photographed by a camera to generate an image having a viewpoint position different from the photographing position of the camera, and generates the image. The image processing apparatus according to claim 11 , wherein the image processing apparatus inputs the captured image and the photographed image. 15. An inter-view interpolation processing means for performing an interpolation process on an image included in multi-view image data to generate a new image, and adding the new image to the multi-view image data. Item 12. The image processing device according to item 11 . 16. viewpoint input means, claims along with the position of the viewer's eye performs the observer's gaze direction detection 11
An image processing apparatus according to claim 1. 17. The image reconstructing step calculates parameters necessary for reconstructing an image from the position of an observer's eye and the type of an image output device, and uses the parameters to convert each pixel of the reconstructed image into a multi-viewpoint image. 12. The image processing apparatus according to claim 11, wherein the image processing apparatus calculates which pixel of the image in the image data corresponds to, extracts the corresponding pixel from the multi-view image data, and reconstructs the image data . 18. The image processing device according to claim 11 , wherein the image output device is a stereo display. 19. The image processing device according to claim 11 , wherein the image output device is a head mounted display. 20. The image processing device according to claim 11 , wherein the image output device is a lenticular display.