JPH09200803A - Image processing unit and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To observe an image stereoscopically from a moved position when a position of eyes of the observer is moved in each direction including forward/ backward directions. SOLUTION: A hierarchical data generating section 15 converts multi-visual point image data from an input multi-visual point image data storage section into multi-visual point image data with different resolution to generate hierarchical data. Then a corresponding point locus discrimination section 11 detects parametric data based on a locus coefficient outputted from a locus coefficient control section 12, a threshold level outputted from a discrimination dual threshold level control section 14 and depth information stored in a detected depth arrangement storage section 15 as already detected information. On the other hand, a hierarchical data selection section 17 selects hierarchical data based on a machine power of a CPU or the like, executes interpolation processing of the corresponding point locus based on the parametric data in response to the selected hierarchical data and the visual point position and a stereoscopic image is displayed on a display screen 8 via an image display section 21.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method for inputting a plurality of images having different viewpoints and outputting an image having a viewpoint position corresponding to the current eye position of an observer.

[0002]

2. Description of the Related Art Conventionally, a stereo display, a lenticular display or the like has been known as an image processing apparatus for stereoscopically displaying an image viewed from a plurality of viewpoints.

The stereo display alternately displays images obtained from two cameras at high speed.
The observer can stereoscopically observe the image by using the shutter glasses and the polarized glasses that are synchronized with this switching.

Further, in the lenticular display, for example, images from four cameras are rearranged in pixel units and a lenticular sheet is attached to the front surface so that four-viewpoint images can be stereoscopically observed.

[0005]

However, the conventional stereo display described above has a drawback in that only a stereoscopic image in the shooting direction of the camera can be observed when shooting an image. That is, in a conventional stereo display, two cameras are generally fixed and an object is photographed. Therefore, the image that can be seen is the same even if the observer moves the viewpoint (eye position). As a result, the observer side There was a problem that it was not possible to reflect the movement of the viewpoint in the image in the image, and the sense of realism was lacking.

Further, although the lenticular display is capable of coping with the movement of the observer's viewpoint in the left-right direction, it is intended to see one of the images of a plurality of cameras in an intermittent manner, and to continuously move the viewpoint. There was a problem that the viewpoint could not be moved in the front-back direction in addition to being unable to handle it.

That is, the viewpoint movement in the front-rear direction is conventionally performed when stereoscopic viewing is performed based on an image created by computer graphics.
Computer graphics is a simple image, and realizes stereoscopic vision in a special situation in which all coordinate values in a three-dimensional space corresponding to points in the image are known. The fact is that the viewpoint movement in the front-back direction in the case of stereoscopically viewing the image captured in 1. is not currently considered.

The present invention has been made in view of the above circumstances. Even when an image is taken by an image pickup device such as a camera, when the position of the observer's eyes moves in each direction including the front-back direction. An object of the present invention is to provide an image processing device and an image processing method capable of stereoscopically observing an image from the moved position.

[0009]

In order to achieve the above object, an image processing apparatus of the present invention is an image processing apparatus for generating an image according to a viewpoint, and is a multi-viewpoint having a plurality of different positions as viewpoints. A multi-view image data holding unit that holds image data, a hierarchical data generation unit that converts the multi-view image data held by the multi-view image data holding unit into multi-view image data of different resolutions, and the hierarchical data generation unit. In a plurality of generated multi-viewpoint image data, corresponding point locus detection means for sequentially detecting coefficients that define corresponding points that correspond to each other in each multi-viewpoint image, and image generation from the corresponding point locus coefficients. Hierarchical data selection means for selecting corresponding point locus coefficient data corresponding to hierarchical data corresponding to the arithmetic processing means used, and selected corresponding point locus coefficient data Data input means for inputting, viewpoint position input means for inputting the viewpoint position of the observer, corresponding point trajectory interpolation means for generating an image from the corresponding point trajectory coefficient data according to the observer viewpoint position, and It is characterized by having an image display means for displaying the displayed image.

Further, the image processing method of the present invention is an image processing method for generating an image according to a viewpoint, which is a multi-view image data holding step of holding multi-view image data having a plurality of different positions as viewpoints. A hierarchical data generation step of converting multi-viewpoint image data held by the multi-viewpoint image data holding step into multi-viewpoint image data of different resolutions, and a plurality of multi-viewpoint image data generated by the hierarchical data generation step. Corresponding point locus detection step of sequentially detecting coefficients that define corresponding points that correspond to each other in images in the multi-viewpoint image, and from the corresponding point locus coefficient to hierarchical data according to the arithmetic processing means used for image generation. Hierarchical data selection step of selecting corresponding corresponding point locus coefficient data, data input step of inputting selected corresponding point locus coefficient data, and observer A viewpoint position input step of inputting a viewpoint position, a corresponding point trajectory interpolation step of generating an image from the corresponding point trajectory coefficient data according to the viewpoint position of the observer, and an image display step of displaying the generated image. It is characterized by including.

[0011]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a perspective view schematically showing an embodiment of an image processing apparatus according to the present invention. The image processing apparatuses are arranged in parallel in a horizontal direction to photograph a subject 1. From the multi-lens cameras 2, ..., Corresponding point locus detection unit 3 that detects corresponding points that correspond to each other between pixels in the multi-viewpoint images output from the multi-lens cameras 2 ,. A storage medium 4 such as a disk for storing output parametric data (corresponding point trajectory coefficient), eyeglasses 6 provided with a viewpoint detector 5 for detecting the position of the user's eye, parametric data and a viewpoint detector 5 An on-demand interpolation display unit 7 for appropriately interpolating a locus of corresponding points based on the detection result of A to generate an image visible from the position of the eye, and a display screen for displaying the image created by the on-demand interpolation display unit 7. 8 and It is.

In the image processing apparatus having such a system configuration, the corresponding point locus detection unit 3 receives the input multi-viewpoint image and outputs the parametric data to the storage medium 4. On-demand interpolation display unit 7 is storage medium 4
The parametric data is received from and the image is displayed on the display screen 8 according to the change of the viewpoint position of the user.
Here, the parametric data includes information necessary and sufficient for arbitrary viewpoint display, and its data amount is extremely small compared to the case where all the images corresponding to possible viewpoint movements are held. In addition, the device is configured so that the amount of memory required for the device is reduced.

FIG. 2 is a block diagram showing the internal details of the image processing apparatus.

That is, the corresponding point locus detection section 3 holds an input multi-viewpoint image holding section 9 for holding an image of the subject 1 photographed by the multi-view camera 2, and the input multi-viewpoint image holding section 9.
A hierarchical data generation unit 10 that generates a multi-viewpoint image having different resolutions from the multi-viewpoint image stored in the multi-viewpoint image, and a linear shape of an image of a bright spot captured in the multi-viewpoint image generated by the hierarchical data generation unit 10. A corresponding point locus determination unit 11 that determines the locus of
A locus coefficient control unit 12 that controls the linear corresponding point locus coefficient detected by the corresponding point locus determination unit 11, and a detected locus that stores the corresponding point locus coefficient detected by the corresponding point locus determination unit 11 and the gray value of the locus. Storage unit 13 and corresponding point locus determination unit 11
Position information in the image of the determination dual threshold value control unit 14 that controls two variable thresholds (threshold A and threshold B) used for the determination of the corresponding point locus, and the corresponding point locus detected by the corresponding point locus determination unit 11. And the detected depth array holding unit 15 that holds the depth information
And a data output unit 16 for outputting the corresponding point locus coefficient stored in the detected locus storage unit 13.

The on-demand interpolation display unit 7 determines the hierarchical data to be input from the parametric data stored in the storage medium 4, and the hierarchical data selection unit 17. From the viewpoint (eye position) by interpolating the data input unit 18 for inputting desired hierarchical data from the storage medium 4 based on the selection result of No. 1 and the corresponding point locus of the data input unit 18 based on the viewpoint position. Corresponding point trajectory interpolation unit 19 for generating a visible image
And a viewpoint position input unit 20 for inputting the viewpoint position detected by the viewpoint detector 5, and an image display unit 21 for displaying an image on the display screen 8.

Next, the operation of the corresponding point locus detection unit 3 will be described.

First, the input multi-view image holding unit 9 holds images taken from a large number of viewpoints through the multi-view cameras 2 arranged side by side in the horizontal direction. A set of multiple multi-view images with different resolutions is generated based on the images held in the viewpoint image holding unit 9.

Multi-view images with different resolutions are shown in FIG.
As shown in FIG. 3, n pieces of image size N × M (n is 2
When there are input multi-viewpoint images of the above integers), these images are obtained by filtering these images and subsampling them to 1: S so that aliasing distortion does not occur during subsampling. That is, a set of a plurality of multi-view images with different resolutions is, as shown in FIG.
S is, for example, a set of multi-view images obtained when S = 2, 4, 8, 16, ..., At this time, the size of each image is (N / S) × (M / S). Then, the plurality of multi-view images become hierarchical data and the following processing is performed.

First, in the multi-view image with the lowest resolution, the trajectory coefficient control unit 12 sequentially assumes virtual bright points in the subject space, and the bright points are generated by the hierarchical data generation unit 10. The trajectory coefficient formed in the viewpoint image is calculated. The corresponding point locus determination unit 11 determines whether or not the locus indicated by the locus coefficient of the locus coefficient control unit 12 exists in the input image of the input multi-viewpoint image holding unit 9. At this time, the determination double-threshold control unit 14 causes the corresponding point trajectory determination unit 11 to operate as described later.
The judgment conditions of are sequentially controlled. Then, the corresponding point locus determination unit 11 stores the detected corresponding point locus coefficient in the detected locus storage unit 13, and at the same time, detects the position information, depth information, and the like of the detected corresponding point locus in the image. It is stored in and used at the time of locus determination described later. When the processing of the corresponding point locus determination unit 11 is completed, the data output unit 16 outputs the corresponding point locus coefficient stored in the detected locus storage unit 13 as parametric data and stores it in the storage medium 4.

Next, similar processing is performed on the multi-viewpoint image having the second lowest resolution, and the data output unit 16 outputs the parametric data corresponding to the hierarchical data. In this way, the same processing is performed on the multi-viewpoint image with a lower resolution to the multi-viewpoint image with a higher resolution, and the parametric data corresponding to this hierarchical data is output from the data output unit 16 again. That is, in the case of having m hierarchical data, the above process is repeated m times, whereby parametric data corresponding to each hierarchical data is generated and stored in the storage medium 4.

Next, the operation of the corresponding point locus determination section 11 will be described in detail. As shown in FIG. 4A, the corresponding point locus determination unit 11 pays attention to a certain horizontal line 22 ... In the image output from the hierarchical data generation unit 10,
2. Create epipolar plane image (EPI) of 2 ... That is, with respect to the images taken from the respective viewpoints of the plurality of multi-lens cameras 2, a certain horizontal line 22 is collected as a common line and arranged in the order of the viewpoints, and as shown in FIG. An EPI including the display position and the vertical axis representing the viewpoint position is created. Here, the multi-view camera 2 in which the input images of the input multi-view image holding unit 9 are arranged in parallel in the horizontal direction
Since it is an image taken from ..., The locus of the image of a certain bright spot becomes a straight line on this EPI, and the above-mentioned locus coefficient is composed of a set of the position and the inclination of the straight line. Further, the inclination of the straight line becomes closer to vertical as the depth distance from the camera to the bright spot becomes longer, and becomes horizontal as the depth distance becomes shorter. Further, in such a straight line locus detection process, a phenomenon may occur in which the straight line locus of the bright spot in the foreground obscures the straight line locus of the bright spot in the back. It is said.

From the above, the corresponding point locus determination processing to be performed by the corresponding point locus determination unit 11 is equivalent to determining whether or not it is a straight line locus on EPI, and in the present embodiment, the corresponding point locus is a straight line. It is performed by determining whether or not it is a locus.

FIG. 5 is a flowchart showing the determination procedure of the corresponding point locus determination unit 11. That is, the trajectory coefficient control unit 12 sequentially assumes and calculates all possible trajectory coefficients for the trajectory coefficient (the position of the straight line and the inclination of the straight line), and then the corresponding point trajectory determination unit 11 first calculates in step S1. On the EPI corresponding to the straight line, whether or not there is an unprocessed pixel is searched, and if there is no unprocessed pixel, the process proceeds to step S6 described later, while if there is an unprocessed pixel, , And proceeds to step S2 to perform straight line determination processing.

The straight line determination processing will be described in detail below.

FIG. 6 is a diagram showing the conditions for determining whether or not the trajectory is a straight line for each point on the straight line. For all the pixels on the straight line, the following three conditions are determined for each point, and the effective pixel is determined. If there are at least two or more, it is determined to be a linear locus.

Here, the three conditions are (1) a first condition as a color condition for determining whether or not a difference in gray value between pixels is less than or equal to a threshold value (2) the pixel is already another linear locus Condition for determining whether or not it is a detected pixel as a part of (3) If it is determined as a detected pixel according to the second condition, a third condition for determining whether or not it has an occlusion relationship I mean the condition.

That is, according to the first condition, when the difference value of the gray value is less than or equal to the threshold value, it is determined that the color of the pixel is the same color, it is determined as a straight line locus, and is counted as an effective pixel, while the gray value of the gray value is determined. When the difference value is equal to or more than the threshold value, it is considered that the pixel colors are different colors, so it is determined that the pixel is an inappropriate pixel, and in principle it is determined that the trajectory is not a straight line locus.

When it is determined that the pixel has been detected under the second condition, the depth information stored in the detected depth array holding unit 15, that is, the inclination of the straight line and the magnitude of the inclination of the straight line to be detected at present are determined. Compare. Then, since the straight line loci are sequentially detected from the front side, if the slope of the straight line to be detected now is larger than the slope of the straight line stored in the detected depth array holding unit 15, it is determined that there is an occlusion relationship, and conversely. If it is small, it is judged that there is no occlusion relationship. Then, if the occlusion relation is not found despite the detection, it is processed as an inappropriate pixel regardless of the determination result of the first condition, and the existence of the linear locus is denied. Also, in the case of an occlusion relationship, it is determined that the difference is not the same color when the difference value of the gray value is less than or equal to the threshold value, and even when the gray value is greater than or equal to the threshold value because the linear locus is hidden by occlusion. Also, the pixel concerned is judged to be on a straight line locus and is skipped to move to the processing of the next pixel.

Here, the threshold value is variably controlled and is controlled by the determination double threshold value controller 14. That is, as the threshold value, the threshold value A for an undetected pixel is used as the discriminant value for the difference value of the grayscale values, and the threshold value B is used as the discriminant value for the difference value of the grayscale values when the pixels are in the occlusion relationship. Specifically, as shown in FIG. 7, the threshold A and the threshold B are variable according to a repeating loop for determining whether or not each point is on a straight line locus. That is, in an iterative loop, first,
While maintaining the state where the threshold value B is fixed to the initial value, the threshold value A is gradually increased each time the loop is repeated, and the determination processing of the corresponding point trajectory determination unit 11 is repeated. And the threshold A
When the threshold value reaches the predetermined upper limit value, the threshold value A is fixed to the predetermined upper limit value, the threshold value B is gradually increased, and the determination process is repeated. When the threshold value B also reaches the predetermined upper limit value, the determination process is performed. finish. That is, when determining a straight line locus, initially, processing is performed for undetected pixels.
In that case, the first condition, that is, the color condition is discriminated by the threshold value A, and when the occlusion relationship occurs thereafter, the threshold value B is determined.
Therefore, the color condition is determined.

Thus, two thresholds (threshold A, threshold B)
The reason why the straight line locus is determined using is that the detection accuracy of the straight line determination deteriorates near the upper limit value when the straight line determination is performed using only the threshold value A. By using a number of thresholds (threshold A, threshold B),
Detection accuracy can be improved.

Then, in step S3, it is determined by the straight line determination process whether or not it is determined to be a straight line locus, and if the answer is negative (No), it is determined that it is not a straight line locus and the process proceeds to step S6. On the other hand, when the answer is affirmative (Yes), it is determined that the trajectory is a linear trajectory, the process proceeds to step S4, the inclination of the line (depth information) is recorded in the detected depth array holding unit 15, and the trajectory coefficient is determined in step S5. The (straight line position and inclination) and the grayscale value information of the corresponding points (for example, the average value of the grayscale values of the corresponding points) are stored in the detection locus storage unit 13 as parametric data. In addition, the detection depth array holding unit 1
It becomes a material for determining whether or not the current pixel which is the target of the straight line determination based on the inclination of the straight line stored in 5 has already been detected as another straight line trajectory. In a succeeding step S6, it is determined whether or not the above-described straight line determination processing and the like have been completed for all the locus coefficients (the position of the straight line and the inclination of the straight line) output from the locus coefficient control unit 12, and the answer is When the answer is affirmative (Yes), the process returns to step S1 to repeat the above-described processing, whereas when the answer is affirmative (Yes), the process proceeds to step S7, and all thresholds (variably controlled by the determination double threshold controller 14 ( For the threshold A or the threshold B), it is determined whether or not the above processing is completed. That is, the threshold value is appropriately variably controlled in accordance with the number of times of the iterative loop as described above (see FIG. 7), and the processing ends for all values of these variable threshold values (threshold value A or threshold value B). It is determined whether or not. Then, when the answer is negative (No), the process returns to step S1 and the above-described processing is repeated, while when the answer is affirmative (Yes), the processing is ended.

As a result, the parametric data output by the data output unit 16 includes the positions of straight lines corresponding to the number of linear loci detected for each horizontal line,
It becomes a set of inclination and gray value information.

Next, the operation of the on-demand interpolation display section 7 will be described. The hierarchical data selection unit 17 determines the hierarchical data to be input from the processing speed of the CPU and the capacity of RAM of the computer being used. The data input unit 18 inputs the storage medium 4 corresponding to the desired hierarchical data according to the determination of the hierarchical data selection unit 17 and sends it to the corresponding point locus interpolation unit 19. When the user observing the display screen 8 changes the position of the head and moves the viewpoint (eye position), the signal of the viewpoint detector 5 changes. In response to this change, the viewpoint position input unit 20 sends the viewpoint position to the corresponding point trajectory interpolation unit 19. Upon receiving the viewpoint position, the corresponding point trajectory interpolation unit 19 generates a new image corresponding to the movement of the viewpoint and sends the image to the image display unit 21.

The corresponding point locus interpolation unit 19 is specifically
The viewpoint position in the horizontal direction from the viewpoint detector 5 is obtained via the viewpoint position input unit 20. For all corresponding point trajectories of each horizontal line of parametric data, the horizontal display position in the image of the bright spot is calculated, and the gray value of the bright spot is drawn at the display position to create a single sheet corresponding to the viewpoint position. Create an image.

Here, the horizontal display position of a bright point in a horizontal line is x, the linear position of the corresponding point locus is x0,
When the straight line inclination is k and the viewpoint position is X, the display position is calculated by the mathematical expression (1).

X = x0 + k · X (1) In this way, the display position x corresponding to the viewpoint position X is calculated to generate a new image, and the image is sent to the image display unit 21. Displays an image on the display screen 8.

As described above, according to the first embodiment, according to the determination of the hierarchical data selection unit 17, parametric data of the image size corresponding to the machine power such as the processing speed of the CPU and the storage capacity of the RAM is input and the corresponding points are input. Since it is sent to the trajectory interpolator 19, it is possible to avoid a situation in which the image generation at the time of moving the viewpoint is not in time due to insufficient machine power, and the observer can comfortably move the viewpoint while moving the image. Can be observed. That is, in the first embodiment, since the data obtained by independently performing the interpolation processing on each of the sets of multi-viewpoint images having different resolutions is held as hierarchical data, when the machine power to be reproduced is low, By using the low-resolution hierarchical data and when using the high-resolution hierarchical data, it is possible to obtain the same operational feeling (tracking ability of the viewpoint, etc.) even if the machine power is different.

Further, when the image is reproduced, it is sufficient that there is sufficient parametric data of the subject for the arbitrary viewpoint display, and the amount of the data is extremely small as compared with the case where all the images corresponding to the possible viewpoint movement are held. The storage capacity is sufficient, and therefore the memory capacity installed in the device is small.

Further, in the above-mentioned first embodiment,
The display screen 8 and the image display unit 21 use a stereoscopic display screen and a stereoscopic image display unit capable of binocular stereoscopic vision such as a lenticular system and a glasses system, and the viewpoint position input unit 20 uses the left and right eye positions. By calculating a viewpoint parameter corresponding to the eye point and generating an image for the corresponding point locus interpolating unit 19 to be viewed by the left and right eyes corresponding to the viewpoint parameter, both eyes capable of moving the viewpoint forward, backward, leftward and rightward. It is also possible to obtain an image processing device as a stereoscopic display device.

FIG. 8 is a diagram for explaining the interpolation processing of the corresponding point locus interpolation unit 19 when the present invention is applied as the image processing apparatus capable of moving the viewpoint forward, backward, leftward and rightward (second embodiment). Form).

In the figure, the horizontal axis (X axis) represents the position in the horizontal direction from the viewpoint, and the vertical axis (Z axis) represents the position in the depth direction from the viewpoint.

Reference numeral 31 denotes a certain specified bright point (X1, Z
1) and 32 are bright points (X1 + X, Z1 + Z) after movement, 3
Reference numeral 3 denotes an image pickup surface located at a focal length f from the viewpoint. When it is assumed that the image of the bright spot 31 is projected on the image pickup surface whose horizontal distance from the viewpoint is x0, and the image after movement is projected on the image pickup surface whose horizontal distance from the viewpoint is x. , Moving the viewpoint in a certain direction is equivalent to moving the bright spot 31 in the opposite direction, and using this principle, when the bright spot 31 moves to the bright spot 32, The horizontal display position x is obtained.

That is, assuming that the pixel pitch is p, FIG.
As a result, the expressions (2) and (3) are established, and the display position x is calculated from the expressions (2) and (3) as shown in the expression (4).

X1 / Z1 = (p · x0) / f (2) (X1 + X) / (Z1 + Z) = (p · x) / f (3) x = (x0 + k · X) / (1 + a ·) k · Z) (4) Here, a is a constant defined by the mathematical expression (5), and the linear slope k is expressed by the mathematical expression (6).

A = f / p (5) k = f / (p · Z1) (6) In this way, the display position x after the movement corresponding to the viewpoint movement is calculated by the mathematical expression (4). Therefore, the position of the image after movement can be obtained.

That is, the viewpoint detector 5 sends the left, right, front and rear viewpoint positions to the corresponding point locus interpolating section 19, and the corresponding point locus interpolating section 19 formulas for all the corresponding point loci of each horizontal line of parametric data. By calculating the display position x according to 4), it is possible to generate an image corresponding to the front, rear, left, and right viewpoint movements.

In the second embodiment, since there is no vertical parallax in the input image, vertical distortion occurs in the image generated with respect to the forward and backward viewpoint movement, but the distance to the subject is increased. Accordingly, when the image is displayed, the distortion can be corrected by performing the scaling conversion in the vertical direction to correct the distortion.

Next, a case where the processing of the second embodiment is executed at high speed will be described as the third embodiment.

First, in the second embodiment, the equation (4) can be transformed into the equation (7).

X = ((x0 / k) + X) / ((1 / k) + (a · Z)) = (x ′ + X) / (k ′ + Z ′) (7) where x ′, k ′ and Z ′ are equations (8) to (10)
It is represented by

X ′ = x0 / k (8) k ′ = 1 / k (9) Z ′ = a · Z (10) Since x ′ and k ′ do not depend on the viewpoint position, they are calculated in advance. Can be set. Further, since Z ′ does not depend on the corresponding point locus, it suffices to calculate only once for changes in the viewpoint position, and the time and effort for calculating one by one as in Expression (4) can be omitted, and high-speed calculation processing can be performed. It becomes possible to do.

FIG. 9 is a flowchart of the high speed calculation processing in the corresponding point locus interpolation unit 19 of the image processing apparatus according to the third embodiment.

The corresponding point locus interpolation unit 19 first determines in step S
In step 11 and step S12, x'and k'are calculated for all corresponding point locus coefficients based on equations (8) and (9), respectively. Next, in step S13, the viewpoint position is acquired from the viewpoint position input unit 20, and in the subsequent step S14, Z'is calculated based on the formula (10) only once each time the viewpoint position changes. Then, with respect to all the corresponding point locus coefficients of each horizontal line, the display position of the bright spot is calculated in step 15 based on the mathematical expression (7), the gray value of the bright spot is drawn in step 16, and the horizontal line is drawn in step S6. It is determined whether or not the processing has been completed for all corresponding point trajectories. When the answer is negative (No), the process returns to step S15, while the answer is affirmative (Yes).
In the case of s), the process proceeds to step S18, and it is determined whether or not the processing is completed for all viewpoint positions when the viewpoint position, that is, the position of the eyes of the observer has changed, and when the answer is negative (No) Returns to step S14 and the viewpoint detector 5
While acquiring a new viewpoint position via and continuing the process,
If the answer is affirmative (Yes), the process ends.

With the above processing, in the third embodiment, the corresponding trajectory interpolation processing can be performed at high speed, and a smoother viewpoint movement is possible.

Next, as a fourth embodiment, an image processing apparatus capable of moving the viewpoint forward, backward, upward, downward, leftward and rightward will be described.

FIG. 10 is a perspective view schematically showing the image processing apparatus according to the fourth embodiment, in which a plurality of multi-lens cameras 2 are used.
Are arranged on a plane in the vertical and horizontal directions.

In the fourth embodiment, the input image of the input multi-view image holding unit 9 is an image taken from viewpoints arranged on the XY plane in the left, right, up and down directions. Therefore, the locus of the image of a certain bright spot becomes a plane in the four-dimensional image space formed by the multi-viewpoint images of the input multi-viewpoint image holding unit 9. Therefore, the corresponding point locus determination process of the corresponding point locus determination unit 11 is equal to the plane determination process in the input four-dimensional image space. This plane is expressed by equation (11), where the position coordinates of the viewpoint are (X, Y), the position coordinates of the bright points on the locus are (x, y), and the position coordinates of the locus are (x0, y0). It

(X, y) = ((x0 + k · X), (y0 + k · Y)) (11) The trajectory coefficient calculated by the trajectory coefficient control unit 12 indicates the position and the inclination of the plane (x0, y0, k). ) Is a set of three coefficients. The corresponding point trajectory determination unit 11 is sequentially controlled by the trajectory coefficient control unit 12 and the determination double threshold control unit 14, determines a planar trajectory in the image space, and stores the detected plane coefficient in the detection trajectory storage unit 13. To do. The parametric data output by the data output unit 16 is a set of coefficients corresponding to the number of detected planar trajectories.

In the fourth embodiment, the display position (x, y) on the XY plane is expressed by Expression (12) by modifying Expression (7).

(X, y) = ((x ′ + X) / (k ′ + Z ′), y ′ + Y) (12) Here, y ′ is represented by Formula (13).

Y '= y0 / k (13) FIG. 11 is a flowchart showing the processing procedure of the corresponding point locus interpolation unit 19 in the fourth embodiment.

The corresponding point locus interpolating section 19 first determines in step S
In step 21 and step S22, x ', y', and k'are calculated for all corresponding point locus coefficients based on equations (8), (13), and (9). Next, in step S23, the viewpoint position is input to the viewpoint position input unit 2
Then, Z ′ is calculated based on the equation (10) only once each time the viewpoint position changes in step S24. Then, in step 25, the display position of the bright spot is calculated based on the mathematical expression (7), in step 26, the gray value of the bright spot is drawn, and in step S27, it is determined whether or not the processing has been completed for all corresponding point loci. Determine. Here, the processing on all the corresponding point loci is performed by simultaneously analyzing each horizontal line and each vertical line. When the answer is negative (No), the process returns to step S25, while when the answer is affirmative (Yes), the process proceeds to step S28.
It is determined whether or not the processing has been completed for all changes in the viewpoint position, and if the answer is negative (No), the process returns to step S24 to acquire a new viewpoint position and continue the processing, while the answer is affirmative (No. In the case of Yes), the processing ends.

In the fourth embodiment, since the input image has a vertical parallax, the vertical image distortion that occurs in the second and third embodiments may occur. Also disappears.

In this embodiment, the hierarchical data is input according to the processing speed of the CPU of the computer and the storage capacity of the RAM. However, if this is performed as described below, a higher quality image is generated. become able to.

First, the parametric data of all layers are read from the storage medium 4. Then, the optimum hierarchical data used for image generation is selected from the parametric data in view of the balance between the observer's viewpoint moving speed and the computer power, and the image is generated and displayed. When the observer does not move the viewpoint, for example, when observing at the same viewpoint position for 1 second or longer, it is interpreted that the user wants to see that part in more detail, and image generation / display using higher resolution hierarchical data. To do. Furthermore, it is also possible to measure the time during which the observer does not change the viewpoint position and increase the resolution of the hierarchical data used at regular intervals.

Also, due to human visual characteristics, it is not possible to clearly identify an object when moving at high speed.
Therefore, when the observer starts moving the viewpoint, even if the low-resolution hierarchical data with a light image generation load is used for image generation according to the speed, the observer does not feel much deterioration in image quality. Low-resolution hierarchical data with a light load depending on speed is used for image generation.

As described above, even if the computer power is low, it is possible to present a higher quality image to the observer by generating an image using appropriate hierarchical data according to the viewpoint moving speed. Moreover, in each of the above-described embodiments, the multi-viewpoint image captured in advance is held in the input multi-viewpoint image holding unit 9. However, this is a multi-viewpoint that can capture the multi-viewpoint image in real time. By replacing it with a TV camera, it becomes a real-time arbitrary viewpoint image capturing / display system.

Further, when data is exchanged between remote terminals using parametric data as communication data, a real-time arbitrary viewpoint image communication system is used.

The present invention may be applied to a single image processing apparatus, to a system device such as a multi-view television, a multi-view video telephone terminal or a multi-view video conference system, or a computer. Also, it can be applied to a composite device combined with another image processing device.

[0071]

As described above, the present invention detects the position of the eyes of the observer and reconstructs the image seen by the observer from a plurality of images, so that the viewpoint of the observer moves. Images corresponding to it can be output smoothly, and the storage capacity required for the device is larger than that in the case where all images corresponding to possible viewpoint movement are held.
Very few.

Further, since the input multi-viewpoint images are converted into multi-viewpoint images having different resolutions and the images can be generated with respect to the multi-viewpoint image data of a plurality of resolutions, the corresponding point locus is determined according to the power of the computer used. By selecting and using the coefficient, the image generation time when the viewpoint is moved is shortened, the reaction becomes faster, and comfortable observation becomes possible.

Furthermore, according to the present invention, it is possible to move the viewpoint in the front-rear direction, which was not possible in the past.

[Brief description of drawings]

FIG. 1 is a perspective view schematically showing an embodiment of an image processing apparatus according to the present invention.

FIG. 2 is a block diagram showing internal details of the image processing apparatus.

FIG. 3 is a diagram showing a set of multi-view images having different resolutions.

FIG. 4 is a diagram illustrating an epipolar plane image, a corresponding point trajectory, and a trajectory coefficient.

FIG. 5 is a flowchart showing a processing procedure of a corresponding point locus determination unit.

FIG. 6 is a diagram showing a determination condition of a straight line determination process.

FIG. 7 is a diagram showing a double threshold control sequence of a determination double threshold control unit.

FIG. 8 is an explanatory diagram illustrating a processing procedure of a corresponding point trajectory interpolation unit according to the second embodiment.

FIG. 9 is a flowchart illustrating a processing procedure of a corresponding point trajectory interpolation unit according to the third embodiment.

FIG. 10 is a perspective view schematically showing a fourth embodiment.

FIG. 11 is a flowchart showing a processing procedure of a corresponding point locus interpolation unit according to the fourth embodiment.

[Explanation of symbols]

 9 Input multi-view image holding unit 10 Hierarchical data generation unit 11 Corresponding point locus determination unit 12 Locus coefficient control unit 13 Detection locus storage unit 14 Judgment double threshold value control unit (reference value control means) 15 Detection depth array holding unit 16 Data output Part 17 Hierarchical data selection part 18 Data input part 19 Corresponding point locus interpolation part 20 Viewpoint position input part 21 Image display part

Claims (12)

[Claims] 1. An image processing apparatus for generating an image according to a viewpoint, a multi-view image data holding means for holding multi-view image data having a plurality of different positions as viewpoints, and the multi-view image data holding. Hierarchical data generation means for converting the multi-viewpoint image data held by the means into multi-viewpoint image data of different resolutions, and between the images in the respective multi-viewpoint images in the plurality of multi-viewpoint image data generated by the hierarchical data generation means. Corresponding point locus detecting means for sequentially detecting coefficients defining corresponding points corresponding to each other, and corresponding point locus coefficient data corresponding to hierarchical data corresponding to the arithmetic processing means used for image generation from the corresponding point locus coefficients. Hierarchical data selection means, data input means for inputting selected corresponding point locus coefficient data, and viewpoint position input for inputting the observer's viewpoint position Image processing, including force means, corresponding point locus interpolation means for generating an image from the corresponding point locus coefficient data according to the viewpoint position of the observer, and image display means for displaying the generated image. apparatus. 2. The image processing apparatus according to claim 1, wherein the hierarchical data generation means generates a plurality of images having different resolutions for each of the multi-viewpoint image data. 3. The hierarchical data selection means selects the corresponding point locus data from the corresponding point locus data according to at least one of a processing speed of the arithmetic processing means and a storage capacity of a storage means built in the arithmetic processing means. The image processing apparatus according to claim 1, wherein hierarchical data is selected. 4. The corresponding point locus detection means detects a corresponding point locus coefficient that defines a straight line that is a corresponding point locus on the epipolar plane image of the multi-viewpoint image data. The image processing apparatus according to claim 3. 5. The corresponding point locus detecting means detects a corresponding point locus coefficient that defines a plane that is a corresponding point locus of the multi-viewpoint image data. The image processing device according to item 1. 6. The image processing apparatus according to claim 1, wherein the corresponding point locus detecting means variably controls a reference value serving as a reference for detecting the corresponding points. 7. An image processing method for generating an image according to a viewpoint, a multi-viewpoint image data holding step of holding multi-viewpoint image data having a plurality of different positions as viewpoints, and the multi-viewpoint image data holding Hierarchical data generation step of converting multi-viewpoint image data held by a step into multi-viewpoint image data of different resolutions, and between images in respective multi-viewpoint images in a plurality of multi-viewpoint image data generated by the hierarchical data generation step. Corresponding point locus detection step of sequentially detecting coefficients defining corresponding points corresponding to each other, and corresponding point locus coefficient data corresponding to hierarchical data corresponding to the arithmetic processing means used for image generation from the corresponding point locus coefficients. Hierarchical data selection process to select, data input process to input selected corresponding point locus coefficient data, viewpoint position input to input observer's viewpoint position An image characterized by including a force step, a corresponding point locus interpolation step of generating an image from the corresponding point locus coefficient data according to the viewpoint position of the observer, and an image display step of displaying the generated image. Processing method. 8. The image processing method according to claim 7, wherein the hierarchical data generating step generates a plurality of images having different resolutions for each of the multi-viewpoint image data. 9. The hierarchical data selecting step is performed in accordance with at least one of a processing speed of the arithmetic processing means and a storage capacity of a storage means incorporated in the arithmetic processing means, in the corresponding point locus data. 9. The image processing method according to claim 7, wherein hierarchical data is selected. 10. The corresponding point locus detecting step detects a corresponding point locus coefficient that defines a straight line that is a corresponding point locus on the epipolar plane image of the multi-viewpoint image data. The image processing method according to claim 9. 11. The corresponding point locus detecting step detects a corresponding point locus coefficient that defines a plane which is a corresponding point locus of the multi-viewpoint image data. The image processing method described in. 12. The image processing method according to claim 7, wherein the corresponding point locus detecting step variably controls a reference value serving as a reference for detecting the corresponding points.