JP2001052177A - Image processor and method for processing image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide communication of high presence by displaying the video of a transmitter at the free viewpoint position of a recipient at a distance place by composing a device of plural cameras, a computer and a display thereof without using a special dedicated image processor. SOLUTION: In this image processor, the three-dimensional information of an object is extracted from an input image photographed by a photographing part 101 where plural cameras are arranged on both the sides of a display 113 and the output image of the object is displayed from that three-dimensional information and the information of the free viewpoint position of the recipient by a reconstituting/display part 115. Thus, the visual line of a user at a distant place can be matched and the video of the transmitter at the free viewpoint position of the recipient can be generated so that the communication of high presence can be provided.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and a method for use in, for example, a video conference system or a video telephone system. The present invention extracts three-dimensional information of a subject from an input image captured by a camera, and uses the three-dimensional information of the subject and the viewpoint position information of the user to determine a subject corresponding to the viewpoint position of the user. And an image processing method for reconstructing and displaying an image.

In particular, an input image is obtained by using an image pickup unit arranged such that the lens centers of a plurality of cameras are on a straight line, and the correspondence of feature points between the input images is determined by corresponding points on an epipolar plane image. By reducing to straight line detection, for a subject with a small amount of feature such as a human face, the 3D information of the subject is automatically obtained without manual intervention, and the user uses the 3D information of the subject to The present invention relates to an image processing apparatus and an image processing method for reconstructing an image of a subject corresponding to a free viewpoint position and displaying the image on a display of a computer.

[0003]

2. Description of the Related Art Recent advances in image compression technology and the video coding standard H.261 developed for video conferencing and video telephony based on ISDN at ITU-T and the H.263 improved version of H.263 Video conferencing systems have been put into practical use due to the standardization of the Internet, and it is expected that the necessity will increase in the future as the performance of hardware such as the flow of computerization in society and computers increases.

However, in the current video conference system,
Since the camera that shoots the user is installed outside the display (for example, on the top), when the user looks at the display, the camera will shoot the face of the user facing down, and the A face image of the user facing down is transmitted to one user. For this reason, there is a problem that the eyes of the users located at remote locations do not match, and it is very inconvenient to communicate.

[0005] Conventional techniques for solving this problem of line-of-sight mismatch include a videophone using a half mirror (for example, Showa 63).
IEICE Autumn National Convention D-38 "Gaze-matched type:
Development of Small Telephone Device ", published by Kunihiro Suetake and Kenichi Yoshikawa).

The principle of the conventional image processing apparatus will be described below with reference to FIG.

[0007] The principle of the conventional image processing apparatus is that the user (A) 2
In the image processing device 2602 on the 601 side, the face of the user (A) 2601 is photographed by the camera 2604 through the half mirror 2603 inclined at 45 degrees, and the other user (B ) And sends it to the similar image processing device 2607 on the 2606 side. The camera 26 in the image processing device 2607 on the user (B) 2606 side
The video of the remote user (B) 2606 captured in 09 is received by the image processing device 2602 on the user (A) 2601 side and reproduced on the television monitor 2605.
And is provided to the user (A) 2601. As a result, the users (A) and (B) who are in remote locations can communicate with each other by matching their eyes.

[0008]

However, in the prior art,
Although it is possible to match the line of sight of users at remote locations, the image processing device is specialized because the image processing device is configured by hardware using a half mirror. For this reason, there was a problem that an existing display could not be used and lacked versatility.

[0009] In addition, the image of one remote user displayed on the display in front of the receiving user is transmitted through a half mirror through a camera fixed to the back of the display in the image processing apparatus. Since the image is assumed to be captured, it is not possible for the receiving user to display an image at a free viewpoint position. As a result, it is not possible to acquire a video image of a user at a remote location from a different angle, so that the presence of the user at a remote location is not high, as if it were in front of the user. was there.

Therefore, an object of the present invention is to solve the above-mentioned problems, and it is possible to use a display of an existing computer without using an image processing apparatus with dedicated hardware using a half mirror, and to use a plurality of displays. It consists of a camera, a computer and software. Another object of the present invention is to provide an image processing apparatus and an image processing method capable of displaying an image of a transmitting user at a free viewpoint position by a receiving user in order to provide highly realistic communication.

[0011]

In order to achieve the above object, the following means are taken.

In research on human face image processing, image recognition and analysis of a human face is generally difficult because a human face has a small amount of features. It is said that it is very difficult to automate the attachment.

Therefore, in the present invention, an epipolar plane image is used in order to realize automatic correlation of feature points.

When the camera optical axis is perpendicular to the camera baseline (a straight line connecting the lens centers of the moving camera) and the camera is moving at a constant speed, the obtained images are arranged in the time axis direction. Sometimes, a spatiotemporal image is generated. The cut plane obtained when the spatiotemporal image is cut in the horizontal direction is called an epipolar plane image. In the epipolar plane image, the feature points between the images appear as a linear trajectory, and the inclination of the linear trajectory has the property of having information on the distance between the camera baseline and the subject.

In the present invention, on the epipolar plane image,
The feature points draw a straight line trajectory, and the inclination of the straight line trajectory utilizes the fact that it has distance information, and performs a process of detecting a straight line trajectory (detection of a corresponding point straight line) on an epipolar plane image composed of a small number of input images, Thereafter, an interpolation process is performed. Then, the distance information between the camera baseline and the subject is extracted from the inclination of the corresponding point line on the interpolated epipolar plane image, and the acquisition of three-dimensional information is performed by fully automated processing. .

Now, a plurality of cameras whose lens centers are arranged on both sides of the display and whose centers are arranged in a straight line are synchronized, an object is photographed, and an input image is obtained.

For these input images, a background image previously photographed and stored in a memory of a computer is read, and the background is removed by taking a difference from the background image to generate a background-removed image.

Since it is desired to convert the camera base line (in the present invention, a straight line connecting the lens centers of a plurality of cameras) into an image in which the optical axes of the respective cameras are perpendicular to each other,
Perform camera calibration, calculate camera parameters, calculate a camera baseline based on the camera parameters, and convert each camera optical axis to be perpendicular to the camera baseline, and By unifying the focal length, the background-removed image is normalized.

Next, the background-removed image is stored in the memory of the computer in the spatiotemporal coordinate system at the same ratio as the camera interval in the real space.

By cutting in the horizontal direction, an epipolar plane image in which lines for the number of input images are arranged is obtained.

On the epipolar plane image, the slope of the gentlest corresponding point straight line is defined as “decideK _min”.
K _min is calculated using the frequency distribution table and the characteristics of the human face.

Next, using the calculated decideK _min and the number of lines corresponding to the number of input images on the epipolar plane image, a corresponding point straight line on the epipolar plane image is detected, and interpolation processing is performed.

Next, distance information between each feature point of the subject and the camera baseline is calculated from the interpolated epipolar plane image, and three-dimensional information of each feature point is extracted.

The processing for calculating the three-dimensional information is based on the input image 1
Until the frame is completed, the data is stored in the memory of the computer.

Next, one frame of the accumulated subject
It reads the three-dimensional information and the input free viewpoint position information of the receiving user, reconstructs the subject using software capable of expressing the three-dimensional space in the computer, and displays it on a display or the like.

Further, the obtained three-dimensional information of the subject is transmitted, and the receiving side reconstructs and displays an image from a free viewpoint of the user.

[0027]

Embodiments of the present invention will be described below with reference to the drawings.

(Configuration of Image Processing Apparatus) FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus according to the present invention. 1, the image processing apparatus of the present invention includes a transmission image processing unit 110 and a reception image processing unit 117. Transmission image processing unit 110
Is the image of subject 111 and background 112
(Hereinafter, simply referred to as an input image) an image capturing unit 101 including four CCD (Charge Coupled Device) cameras and the like in which lens centers are arranged in a straight line, and captures images so as to be within the angle of view of all cameras. To perform camera calibration and calculate camera parameters of each of the four cameras in the photographing unit 101; and a camera coordinate system (X _c , Y
_c , Z _c ), a calibration processing unit 102 for normalizing to a virtual camera coordinate system (X _v , Y _v , Z _v ) described later, and an image in which a background 112 has been photographed in advance (hereinafter, also referred to as a background image). And a background processing unit 103 that subtracts the background image from the input image by performing a difference from the input image, and a plurality of cameras in the imaging unit 101 in which the background removal image is arranged in a virtual camera coordinate system. Space-time coordinate system
(u, v, E) and uE of this spatiotemporal coordinate system (u, v, E)
An epipolar plane image generating unit 104 that generates an epipolar plane image that is a cut plane obtained by cutting in parallel to a plane (hereinafter, also simply referred to as EPI) by sequentially scanning in the v-axis direction of a spatiotemporal coordinate system. On each EPI, a straight line drawn on the EPI is a feature point at which the distance between a straight line connecting the lens centers of a plurality of cameras in the photographing unit 101 (hereinafter, also simply referred to as a camera baseline) and the subject 111 is the shortest. Trajectory slope
(Hereinafter, also simply referred to as decideK _min ) and the shortest distance information calculation unit 105, using the decideK _min , each feature point on a reference line to be set on each EPI (hereinafter, also simply referred to as a reference pixel) An interpolation processing unit 106 that detects a straight line corresponding to the object and performs an interpolation process while also responding to an occlusion part where an object in front of the object blocks an object in the back, and each feature point of the subject 111 and a camera base from the interpolated EPI. Distance information between lines (depth information from camera baseline)
And a three-dimensional information extraction unit 107 that extracts three-dimensional information of each feature point in the virtual camera coordinate system, and accumulates the calculated three-dimensional information in a computer until one frame of the input image is calculated. 3D information storage unit 108 and one of the subjects 111
And a transmission unit 109 for transmitting the three-dimensional information for the frame from the image transmission processing unit 110 to the image reception processing unit 117. The reception image processing unit 117 receives the three-dimensional information transmitted from the transmission image processing unit 110 to the reception image processing unit 117.
A viewpoint position input unit 116 for inputting viewpoint position information of the user on the side of the received image processing unit 117 in the virtual camera coordinate system.
From the transmission image processing unit 110 to the reception unit of the reception image processing unit 117
The three-dimensional information transmitted to 114 and the viewpoint position information of the user on the reception image processing unit 117 side obtained from the viewpoint position input unit 116 are read, and the subject 111 of the transmission image processing unit 110 is reconstructed and displayed. And a reconfiguration / display unit 115.
The number of cameras of the photographing unit 101 provided in the image processing apparatus of the present invention may be any number as long as two or more cameras are provided on both sides of the display 113.

In the following, the number of cameras will be discussed as the minimum number of four in the present invention, for ease of explanation.

(Description of Coordinate System) FIG. 2 is an explanation of each coordinate system.

The cameras 0 (206) and 3 (20
The center of the lens of the virtual camera 205 existing in the middle of 9) is the origin O
_v (0, 0, 0), and the coordinate system of three straight lines X _v , Y _v , Z _v that are spatially orthogonal to each other and pass through their origins is the virtual camera coordinate system
(X _v , Y _v , Z _v ) 201.

Here, among the camera parameters calculated by the camera calibration in the calibration processing unit 102, the lens center (X _vi , 0, 0) of each camera from the origin of the world coordinate system in the external parameters.
The camera baseline vector is calculated using the coordinate values of (i = 0, 1, 2, 3). Then, the virtual camera coordinate system (X _v
, Y _v , Z _v ) The virtual camera coordinate system (X _v , Y _v ) such that the X _v axis direction of 201 is parallel to its camera baseline vector.
_v , _Zv ) 201 are set.

Here, the unit system of the virtual camera coordinate system may be defined by an existing unit system such as a metric unit system using camera parameters calculated by camera calibration.

Next, four cameras (camera 0) in the photographing unit 101
(206), camera 1 (207), camera 2 (208), camera 3 (209)), the lens center of each camera (X _vi , 0, 0) (i = 0, 1,
2, 3), on each camera optical axis (see below, camera coordinate system (X _c
, Y _c , Z _c ) in the Zc-axis direction of 701) and the focal length f _i
(i = 0, 1, 2, 3)
The imaging plane coordinate system (x, y) 202 is used. Shooting plane coordinate system (x, y) 202
The origin and the camera optical axis Z _c, lies in its imaging surface intersects. The shooting plane coordinate system (x, y) 202 is a virtual camera coordinate system (X
_v , _Yv , _Zv ) 201, the unit system is the same as the unit system in the virtual camera coordinate system ( _Xv , _Yv , _Zv ) 201.

Next, in the photographing plane coordinate system (x, y) 202, when a photographed image is stored in the memory, the method of storing the image differs depending on the computer. Here, the origin is set to the lower left, and the new coordinate system is set to the digital image coordinate system (u, v).
203.

Here, the digital image coordinate system (u, v) 203
Although the origin is set at the lower left, the origin may be appropriately determined according to the specification of storing the photographed image in the memory of the computer. In addition, the size of the input image is vertical V _size and horizontal U _size .
Since this coordinate system is obtained by changing only the origin position in the photographing plane coordinate system (x, y) 202, the unit system is the photographing plane coordinate system (x, y)
The same unit system as 202, and therefore the virtual camera coordinate system
The unit system is the same as the unit system of (X _v , Y _v , Z _v ) 201.

Next, the digital image obtained from each camera in the photographing unit 101 is converted into a virtual camera coordinate system (X
_v , Y _v , Z _v ) Spatio-temporal coordinates of the three straight lines u, v, E that can be created when placed in the computer memory at the same ratio as the camera interval of the four cameras installed in 201 System (u, v,
E) 204. The origin of the digital coordinate system should be on the E-axis of the spatiotemporal coordinate system.

Since the unit system of the spatiotemporal coordinate system (u, v, E) 204 uses an image expressed by the unit system of the digital image coordinate system (u, v) 203, the digital coordinate system (u, v, E) 203 is used. , V) 203
And the same unit system. Therefore, the virtual camera coordinate system (X _y ,
Y _v , Z _v ) The unit system is the same as the unit system of 201.

(Overall Flowchart of Image Processing Apparatus) FIG.
FIG. 3 is a flowchart showing the operation of the entire image processing apparatus. Hereinafter, the operation of the image processing apparatus will be described with reference to FIGS.

Here, the background image 6
Generate 03 and place it in the spatiotemporal coordinate system (u, v, E) 204
(Steps 302 to 309) will be described.

(Camera Calibration) FIG. 4 shows camera calibration processing in the calibration processing unit 102.

First, four cameras (camera 0 (20
It is determined whether the camera calibration has already been completed for the camera 3 (209) from 6) (step 302).
I do. Since the calibration is not performed in the initial state, a calibration pattern 401 having a known length is prepared, and shooting is performed so as to be within the view angles of the four cameras in the shooting unit 101.

Next, the calibration processing section 102 performs camera calibration (step 302) for each of the four cameras.

A world coordinate system whose origin is a known point in the calibration pattern 401 is set, the lens center coordinates of each camera from the origin of the world coordinate system, and the lens optical axis Z _c of each camera with respect to the world coordinate system. Calculate the external parameter called the direction of
Internal parameters such as f _i (i = 0, 1, 2, 3), image center coordinates, pixel size, and lens distortion coefficient are calculated.

When setting the four cameras in the photographing unit 101, the background-removed image 603 described later is normalized (step 30).
7), the camera coordinate system (X _c , Y _c , Z
_c ) is normalized to the virtual camera coordinate system (X _v , Y _v , Z _v ) 201. Then, in order to unify the focal length f _i (i = 0, 1, 2, 3) to F, the camera optical axis ( _Zc axis) of each camera is as perpendicular to the camera baseline as possible, and the It is desirable that the focal lengths f _i (i = 0, 1, 2, 3) be as equal as possible.

In the future, the cameras must not be moved so that the positional relationship between the four cameras in the photographing unit 101 is maintained.

(Shooting of Subject / Background) FIG.
In FIG. 1, a state in which the background 112 is photographed (step 304) is shown.

The four cameras (camera 0 (206)
To the camera 3 (209)), confirm that the background 112 is within the angle of view, shoot the four cameras in synchronization, and save the background image 602 obtained from each camera to the computer memory. Store it. The background 112 is desirably a single color different from the texture of the subject in order to easily remove the background image 602 from the input image 601.

FIG. 5B shows a state where the subject 111 and the background 112 are photographed (step 305).

As in FIG. 5A, it is confirmed that the subject 111 and the background 112 are within the angle of view of the four cameras, the images are taken in synchronization, and the input images obtained from the four cameras are obtained. Get 601.

(Background Removal Processing) FIG. 6 shows how the background processing unit 103 removes the background image 602 from the input image 601 by calculating the difference between the input image 601 and the background image 602 (step
306). Thus, the background removal image 603 is generated.

Here, it is assumed that the image sizes of the input image 601 and the background image 602 are the same in the horizontal U _size and the vertical V _size . The background removal image 603 generated by taking the difference is also a horizontal U _size and a vertical V _size .

(Normalization of Each Camera Coordinate System) FIGS. 7A to 7C
Are three straight lines X in the calibration processing unit 102.
_c , Y_c , Z _c Is the spatial origin O_c Go straight at (0,0,0)
Camera coordinate system (X_c , Y_c, Z_c) Normalized 701
(Step 307) FIG.

[0054] FIG. 7 (A) is Y _c-axis camera baseline vector of the camera coordinate system 701 (the virtual camera coordinate system _{(X v, Y v, Z} v
2) shows a state in which the camera coordinate system is rotated by θ around the _Zc axis so as to be perpendicular to ( _Xv axis of 201). Also,
Figure 7 (B) is a camera coordinate system _{(X c, Y c, Z} c) as 701 Z _c axis is perpendicular to the camera baseline vector, the camera coordinate system about the Y _c-axis (X _c, Y _c , Z _c ) 701 is further rotated by φ.

At this time, the focal lengths f _i (i = 0, 1, 2, 3) of the four cameras in the photographing unit 101 are unified with F. Thus, the background removal image 603 for each camera
Is normalized.

(Arrangement of Background Removed Image in Spatiotemporal Coordinate System) FIG.
Is the epipolar plane image generating unit 104, the virtual camera coordinate system _{(X v, Y v, Z} v) at a rate of each camera interval four cameras in the imaging portion 101 disposed on the 201, the background removed image 6
03 in the spatiotemporal coordinate system (u, v, E) 204 (step 30
8) indicates that

The horizontal direction of the background-removed image 603 is represented by the u-axis,
The vertical direction is the v axis, and the direction in which the background-removed images 603 acquired in the order of camera 0 to camera 3 are arranged in the order of E = E ₀ , E ₁ , E ₂ , and E ₃ is the E axis. Note that the image size of the background-removed image 603 is a horizontal U _size and a vertical V _size .

The virtual camera coordinate system (X_v , Y_v, Z_v ) 201
And each of camera 0 to camera 3 in the imaging unit 101
The coordinate value is (X_v0, 0, 0), (X_v1, 0, 0), (X_v2, 0,
0), (X_v3 , 0, 0), and the camera interval is (Camera 0 and
(Interval of camera 1): (Interval of camera 1 and camera 2): (Interval of camera 2
Camera 3 interval) = (X_v1 -X_v0) :( X_v2 -X_v1) :( X_v3 -X
_{v Two}). Note that X_v0 ≤0, X_v1 ≤0, X_v2 ≧ 0, X_v3 
≧ 0.

In the spatiotemporal coordinate system (u, v, E) 204, the background
In order of the removal image 603, E = E₀, E₁ , E_Two, E_ThreeCalculated
To the machine's memory.

Here, the virtual camera coordinate system (X _v , Y _v ,
Z _v ) When the background-removed image 603 is arranged in the spatiotemporal coordinate system (u, v, E) 204 at the same ratio as the camera interval up to the four cameras in the imaging unit 101 arranged in the Virtual camera coordinate system
(X _v , Y _v , Z _v ) If it is arranged in the same unit as X _{v of} 201, it will use a lot of memory in the E-axis direction of the spatiotemporal coordinate system (u, v, E) 204, so ΔX _v = a × ΔE (a ≧ 1) is reduced and arranged.

(Epipolar Plane Image Generation) FIGS. 9A and 9B show the epipolar plane image generation unit 104 in the spatiotemporal coordinate system.
For the four background removal images 603 arranged at (u, v, E) 204
Generate an epipolar plane image (hereinafter, also simply referred to as EPI) 901 obtained when cutting in parallel to the uE plane (step
309).

Hereinafter, the properties of the EPI will be described.

The four cameras (camera 0 (206)
To the camera 3 (209)) (camera coordinate system 701
_{(X c, Y c, Z} c) is Z _c axis) is perpendicular to the camera baseline, as shown in FIG. 9 (A), the epipolar plane cut surface formed when cut parallel to uE plane Image (hereinafter,
It is simply referred to as an EPI. On the EPI, the same feature point between images has a property of drawing a straight line locus (hereinafter, also simply referred to as a corresponding point straight line). By detecting the trajectory of the corresponding point straight line on the EPI, even if occlusion occurs in which the object in front hides the object in the back, correspondence between the same feature points between images can be taken, and from the slope of the corresponding point straight line, Distance information between each feature point in the image and the camera baseline can be obtained. Further, the gentlest straight line on the EPI is a corresponding point straight line drawn by an object located at the shortest distance from the camera baseline, and its inclination has information on the shortest distance from the camera baseline.

However, in FIG. 9A, the background removal image
The EP of FIG. 9 (B) is discretely arranged on the axis and obtained.
I has only four lines, and it is generally considered difficult to detect the corresponding point straight line.

Therefore, in the present invention, when the camera optical axis is perpendicular to the camera baseline, the corresponding point trajectory (corresponding point straight line) on the EPI is used as a straight line. Point line with a steep slope
(A straight line whose slope has the shortest distance information)
It takes a method of moving the minute slope Δk from decideK _min to interpolate the corresponding point straight line.

After the EPI interpolation described later is completed, V = V _epi is moved by ΔV = 1 at a time in order to proceed to the next EPI interpolation.

(Simple Description of Reconstruction / Display Processing from Shortest Distance Information) Here, the flow from step 310 to step 316 in FIG. 3 will be briefly described.

First, in the shortest distance information calculation unit 105,
On the epipolar plane image 902 obtained by the epipolar plane image generation unit 104, a corresponding point straight line where the distance between the subject 111 and the camera baseline formed by the four cameras in the imaging unit 101 is the shortest is detected. The slope with the shortest distance
A process of calculating (Step 310) is performed with decideK _min .

Next, the interpolation processing unit 106 uses the slope decideK _min having the shortest distance information (step 310) obtained by the shortest distance information calculation unit 105 to generate the epipolar plane image 902.
The corresponding corresponding point straight line is detected, and interpolation processing (step 311) is performed.

Next, the three-dimensional information extraction unit 107 extracts each feature point of the subject 111 and depth information of the camera baseline from the interpolated epipolar plane image obtained by the interpolation processing unit 106, Corollary ( _Xv , _Yv , _Zv ) 201
The above process of calculating the three-dimensional information ( _Xv , _Yv , _Zv ) of the feature point is performed (step 312).

Next, the three-dimensional information storage unit 108 determines whether or not the processing has been completed up to V _{size with} respect to the vertical V axis of the background processed image 603, that is, whether or not one frame has been completed ( Step 313) is performed, and if completed, the process proceeds to step 314. If not completed, the process returns to step 309 to continue the process until one frame is completed.

Next, if one frame is completed in step 313, the three-dimensional information for one frame stored in the three-dimensional information storage unit 108 is transmitted to the transmission unit 10 in the image transmission processing unit 110.
At 9, transmission is made to the reception unit 114 in the reception image processing unit 117 (step 314). Next, in step 315, the data is received from the transmission unit 109 in the transmission image processing unit 110 by the reception unit 114 in the reception image processing unit 117.

Next, 3 of the subject 111 received by the receiving unit 114
The reconstruction / display unit 115 reads the dimensional information and the viewpoint position information of the user input from the viewpoint position input unit 116, and allows the user to freely move the virtual camera coordinate system ( _Xv , _Yv , _Zv ) 201. A process of displaying a three-dimensional model of the subject 111 based on the viewpoint position (step 316) is performed.

The shortest distance information calculation unit 105, the interpolation processing unit 106, and the three-dimensional information extraction units 107 and 3 in the transmission image processing unit 110 are described below.
Dimension information storage unit 108, transmission unit 109, and reception image processing unit 117
Receiving section 114, reconstruction / display section 115 and viewpoint position input section 11
The details of 6 will be described in order.

(Details of Shortest Distance Information Calculation Unit) (Overall Flow of Shortest Distance Information Calculation Processing) FIG. 10 is a flowchart of the shortest distance information calculation unit 105.
Hereinafter, description will be made with reference to FIGS.

First, an optimum straight line slope K _opt at each reference pixel Pr (U _pr , E _pr ) is obtained, and K _opt (U _pr , E _pr)
pr ) (step 1001). Figure 11 shows the details of this process.
Shown in the flowchart of FIG.
This will be described below.

(Setting) FIGS. 12A to 12D show the case where the distance between the subject 111 and the camera baseline is the shortest on the epipolar plane image 902 obtained by the epipolar plane image generation unit 104. FIG. 9 is a diagram illustrating a state of calculating a slope “decideK _min” of a straight line of a corresponding point of a pixel of a feature point of a subject 111.

First, FIG. 12A shows the EPI at v = V _epi in the spatiotemporal coordinate system 204. E = E ₀ , E
_1, of the E _2, E _3, setting a reference line E _pr (Step 11
01), and the pixel Pr (U _pr , E _pr ) on the reference line E _pr is set as the reference pixel. _{Pr (U pr, E p r} ) is referred to as E _pr = 0 (step 1102). Then, Pr (U _pr , E _pr ) _moves on the reference line E _pr .
Scan one pixel at a time. The reference line E _pr is
Among the E ₀ of E _3, may be set many lines, the object 11
When 1 is a human face, since the occlusion part is small, most of the corresponding point trajectories on the EPI pass through the center and we want to reduce the processing amount as much as possible.
The upper center two lines (E = E ₁ , E ₂ ) may be selected as reference lines _Epr .

FIG. 12B shows a straight line L: u passing through Pr (U _pr , E _pr ).
= k × (E-E _pr ) + U _pr (step 1104)
k slopes from K _min to K _{max in} the range of K _min ≤ k ≤ K _max
FIG. 9 is a diagram showing a state in which a corresponding point straight line is detected for each Pr (U _pr , E _pr ) by moving k by Δk. In addition,
In the uE plane, the slope k is always k ≦ 0.

A straight line L: u = k × (EE _pr ) + U _pr and the remaining three lines other than the reference line E _pr are set as comparison lines,
And comparing the pixel an intersection P _i and the comparative line and the straight line L.

(Calculation of K _opt ) In FIG. 12B, first, Pr (U
_pr , E _pr ) are fixed, and the difference E between the pixel value of Pr (U _pr , E _pr ) for each slope k and the pixel value of each intersection P _i of the straight line L and the comparison line is calculated (step 1105). , And calculate the calculated E
It is added to [k] (step 1106). All on the straight line L
Once completed for P _i (step 1107), stores the Sum [k] in the memory of the computer (step 1109). This is a straight line L
And move Δk in the range of K _min ≤ k ≤ K _max
(Step 1111) For each k, the processing from step 1104 to step 1111 is performed, and Sum [k] is stored in the memory of the computer. Then, when it is out of the range of K _min ≤ k ≤ K _max , the calculation for obtaining Sum [k] for each k in Pr (U _pr , E _pr ) is completed.

Here, K _min and K _max are determined for the following reasons.

With respect to K _min , the closest feature point of the subject 111 between the subject 111 and the camera baseline is:
It always appears on the EPI without being occluded (occlusion) by other feature points, and the linear trajectory is not interrupted. The closest possible feature point for such a feature point is
It is considered that the feature point is always present in the input image 601 obtained and acquired by each camera, and the straight line is the gentlest on the EPI. So, this time,
Since the straight line L on the EPI is a straight line passing from the lower right to the upper left,
_Let K _min be K _min =-(U _size / (E ₃ -E ₀ )). In addition, as for K _max , it is desirable to input in advance a value that allows the straight line L to be turned to a state close to a right angle in order to search for K _opt for each Pr (U _pr , E _pr ).

Here, a straight line L: u = k × (E−E _pr
) + U _pr , but the straight line L is E = k × (u-U _pr ) + E _pr
It may be. In addition, when arranging in the spatiotemporal coordinate system (u, v, E) 204 in the order of camera 3 to camera 0, the linear trajectory goes from the lower left to the upper right on the uE plane, but since k ≥ 0, K
_It is sufficient to set _min = U _size / (E ₀ -E ₃ ).

FIG. 12C shows Sum [k] stored in the memory.
And searches for a value that minimizes Sum [k] (step 111).
2) shows the figure. Thus, the straight line L when Sum [k] is the smallest, considers that Pr (U _pr, E _pr) and the corresponding point straight against, and the gradient k and _{K o pt, K opt (U} pr, E
_pr ) is stored in the memory of the computer.

[0086] K the processing from step 1101 to step 1113, all of _{Pr (U p r, E pr} ) on the reference line E _pr performed with respect to (step 1113), each of Pr (U _pr, E _pr) for _opt
Is calculated.

FIG. _12D shows a diagram in which the reference line _Epr has shifted to the next reference line (step 1616). Perform the processing from step 1101 to step 1113 described above,
Calculate K _opt for Pr (U _pr , E _pr ).

However, in the corresponding point straight line here,
Sum [k] may be minimized due to erroneous detection when determining K _opt . P _i which is compared with the reference pixel Pr is 3
This is considered to be due to the fact that there are only about dots and that there are very many similar pixels in the human face.

However, since K _opt obtained by such an erroneous detection is not large for all the pixels on the reference line _Epr , avoiding such a portion, a gentle straight line slope is calculated on the EPI. Therefore, the following processing is performed.

[0090] (K _opt -N Frequencies) reference line E each reference pixel Pr (U _pr, E _pr) on _pr K _opt for (U _pr, E _pr)
Is read from the computer memory, and the horizontal axis is | K _opt | (K _opt
, And the vertical axis is N (the number of appearances of the reference pixel Pr (U _pr , E _pr ) with respect to | K _opt |), and a frequency distribution table of | K _opt | -N is created (step 1002).

(Method of calculating shortest distance information from front face and side face)
Next, FIG. 13, with respect to X _v Z _v plane position of the nose 1301 of the subject 111 facing the front, when the horizontally cut with v = V _epi,
FIG. 7 is a diagram showing a distance relationship between a subject 111 facing front and a camera baseline (X _v- axis).

FIG. 13 shows that the portion of the cheek 1302 has a relatively flat surface, and the camera base line (X
It is considered that the number of pixels separated by Z _cheek from ( _v- axis) is the largest in the images on the cut plane.

Also, the camera baseline (X _v
The part that is the shortest distance to the (axis) is the nose 1301 and Z
Only _nose away. The difference between the distances is ΔZ = Z _cheek-
Z _nose .

FIG. 14 shows the nose 1301 of the subject 111 facing the front.
12 is a | K _opt | -N frequency distribution table obtained by cutting horizontally at v = V _epi with _respect to the X _v Z _v plane at the position of. The vicinity of the maximum value of the appearance frequency N in the frequency distribution table of FIG. 14 is the portion of the cheek 1302 in FIG. 13, and the shortest distance Z _nose is calculated in FIG. | K _opt | rather than _max | K _opt |
Is larger (here, since the coordinate system is k ≦ 0, | K _opt | becomes larger as the position is closer to the camera baseline).

However, based on the above, if it is assumed that + ΔZ is the shortest distance Z _nose from | K _opt | when N _max is reached, as shown in FIG. In this case, the shortest distance is Z = Z _cheek of the _cheek part, so if the shortest distance is set to + ΔZ from the maximum value N _max of the appearance frequency, the erroneous shortest that the part that does not actually exist is the shortest Distance information will be calculated. That is, when the subject 111 as shown in FIG. FIG. 16 is a | K _opt | -N frequency distribution table when the subject 111 faces sideways.

Therefore, in the present invention, the cheek 13
The ratio of the number of pixels between 02 and the nose 1301 is 1.0: rate (0.0
<Assuming a rate <1.0), | in -N frequency distribution table, each | | K _opt K _opt | the occurrence frequency N with _{N [| K o pt |]} , the maximum value N _max of the frequency of occurrence _{when, N [| K opt |]} / N in a range satisfying _max ≧ rate, as much as possible, | K _opt | calculating what is large, the shortest distance information DecideK _min the K _opt on EPI
And This is stored in the memory (step 1003).

In this manner, the shortest distance information decideK _min can be calculated as shown in FIGS. 14 and 16 for the case where the subject 111 faces front or side.

[0098] As described above, each reference pixel Pr (U _pr, E _pr) to avoid K _opt caused by the erroneous detection when K _opt calculated for,
A value close to the shortest distance between the subject 111 and the camera baseline on each EPI is calculated.

The calculated decideK _min is used in interpolation of a corresponding point straight line on the EPI described later.

(Details of Interpolation Processing Unit) FIG.
6 shows a flowchart of FIG. Hereinafter, a description will be given with reference to FIGS.

(Interpolation processing unit) In the interpolation processing unit 106, the epipolar plane image 902 and the shortest distance information decideK
_{Using min} , interpolation processing of a corresponding point straight line on the epipolar plane image 902 is performed.

(A) to (E) of FIG. 18 and FIG. 19 are diagrams showing an interpolation process of an EPI generated by cutting at v = V _epi .

(Setting) FIG. 18A shows the EPI at v = V _epi in the spatiotemporal coordinate system 204. E = E ₀ , E
_1, of the E _2, E _3, the reference line E _pr one line (step
1701), and the pixels Pr (U _pr , E _pr ) on the reference line E _pr
Is a reference pixel. Pr (U _pr , E _pr ) is on the reference line E _pr
Scan one pixel at a time. The reference line E
_pr is the shortest distance information calculation similar, among the E ₀ of E _3, it may be set many lines, but when the object 111 is a person face, that most of the corresponding point trace on EPI passes a central To reduce the processing volume as much as possible,
The upper center two lines (E = E ₁ , E ₂ ) may be selected as reference lines _Epr .

(Overall Flow of Interpolation Processing) Next, in FIG. 18B, the interpolation processing is started using k = decisionK _min obtained by the shortest distance information calculation unit 105.

[0105] First, initial setting of the inclination k to DecideK _min (step 1702), the reference pixel Pr (U _pr, the E _{p r} to U _pr = 0 (step 1703). The straight line L, u = k × ( E -E _pr ) + U _pr and set
(Step 1704).

For each Pr on E = _Epr , it is determined whether or not the straight line L is a corresponding point straight line (step 1705).
If L is a corresponding point straight line, a corresponding point straight line is drawn on the EPI with the pixel value of the reference pixel Pr (U _pr , E _pr ) on the epipolar plane image (step 1706).

Next, all of the reference pixel Pr (U on E = E _pr
It is determined whether or not EPI interpolation processing has been completed for _pr and _Epr ) (step 1707). If it has been completed, the process proceeds to the next step.

Next, the slope k of the straight line L is expressed as K _min ≦ k ≦ K
It is determined whether the range of _max has ended (step 170).
9). If it has been completed, the reference line is set to the next line, and the processing of steps 1701 to 1711 is performed.

(Details of Determination of Straight Line of Corresponding Points) FIG.
FIG. 18 is a diagram showing details of the determination of whether or not the straight line L is a corresponding point straight line in 1705. This will be described with reference to FIGS.

[0110] In FIG. 18 (B), the slope k of the straight line L is DecideK _min
And the reference pixel Pr (U _pr , E _pr ) and the comparison pixel P
It is determined whether or not _i is the same color (step 2001). Same color determination, when the difference between the pixel values of Pr and each P _i is equal to or less than Th _E, regarded as the same color, the process proceeds to step 2002, considers the unique in other cases, the process proceeds to step 2004. In addition,
Th _E is a value obtained experimentally.

If it is determined in step 2001 that the colors are the same, the process proceeds to step 2002. Here, L 'has been watching now before
For _Pi , a corresponding point straight line exists and means a straight line that has already been interpolated. The P _i on the straight line L, L 'determines whether exists (step 2002), the case of FIG. 18 (B), the still straight L' because there is no, the process proceeds to step 2007.

[0112] Next, P _i to be compared with Pr is determined whether or not it was completed all (step 2007). When all _Pi have been completed, the straight line L is determined to be the corresponding point straight line (step 2009).

FIG. 18C shows the state of EPI interpolation when the gradient k changes by Δk in the range of K _min ≦ k ≦ K _max .

On the other hand, FIG. 18D shows a case where it is determined in step 2001 that the color is different. First, the process proceeds to step 2004, and similarly to step 2002, it is determined whether or not L 'is present in the current Pr. For Figure 18 (D), the P _i on E = E _0,
Since the corresponding point straight line has already been detected, it has L '. Therefore, the process proceeds to step 2005. In step 2005, the slope k of the current straight line L is compared with the slope K (u, E) of L ′. In the case of FIG. 18D, since k> K (u, E), it is determined that the straight line L is not a corresponding point straight line (step 2006). And the straight line L
Is terminated (step 1705), the process skips step 1706, and proceeds to step 1707.

[0115] Figure 19 (E), the reference line E _pr is a diagram showing a case where it becomes the next reference line E _pr = E _2. Slope k is d
Start with ecideK _min . First, in step 2001, E = E ₀
Since the P _i is determined to be unique, the process proceeds to step 2004. Next, determine whether L 'exists for the P _i
(Step 2005) Then, in the case of FIG. 19E, since L ′ already exists, the process proceeds to Step 2005. Here, since k> K (u, E), the process proceeds to step 2007. Next, P _i of E = E _1, performs the same processing as that of the E = E _0.

Next, to determine whether the same color or not the P _i of E = E ₃ (Step 2001). Here, _Pi is a different color, and the process proceeds to step 2004. Next, it is determined whether or not L 'exists (step 2004). In FIG. 19 (D), the on P _i of E = E _3,
Since L ′ does not exist, the process proceeds to step 2006, where it is assumed that the straight line L is not the corresponding point straight line, and the process proceeds to step 1707.

FIG. 19E shows a state of the EPI interpolation when the inclination k changes by Δk in the range of K _min ≦ k ≦ K _max . Here, since the P _i of E = E ₀ a unique, steps 20
04, and then determine whether L 'exists (step 200
Perform 4). Since L ′ exists, the process proceeds to step 2005.
Here, since k> K (u, E), the process proceeds to step 2007.

[0118] Next, the P _i of E = E _1, performs the same processing as that of the E = E _1.

[0119] Next, the P _i of E = E _3, performs determination of whether the same color (the step 2001). Here, since _Pi is the same color, the process proceeds to step 2002, and it is next determined whether or not L ′ exists (step 2003). In this case, since L 'does not exist, the process proceeds to step 2007, for all of P _i, so complete, regarded as a straight line L and the corresponding point line (step 2009).

In the EPIs shown in FIGS. 18 and 19, occlusion occurs. Even in such a case, the locus of the feature point is tracked (FIG. 19 (F)).

(Drawing a corresponding point straight line) FIG.
6 is a flowchart illustrating details of a process of drawing a corresponding point straight line with pixel values of reference pixels Pr (U _pr , E _pr ) on an epipolar plane image. First, the following processing is performed for each pixel on a straight line L determined to be a corresponding point straight line.
In this process, if several corresponding point straight lines have already passed through the pixels on the straight line L that is currently set, the slope k of the straight line L and the already stored straight line stored at that pixel position The slope K (u, E) of the corresponding point line L '
When drawing on the EPI, the slope k of the straight line L at that pixel position
Is stored in K (u, E)), and the minimum slope is defined as the pixel value at that pixel position.

When the straight line L passes through the pixel on the epipolar plane image for the first time, the process proceeds to step 2103. On the other hand, if the corresponding point straight line has already passed, step 21
Proceed to 02.

In step 2102, the slope k of the straight line L is compared with the slope K (u, E) of the corresponding point straight line L 'that has already passed (step 2102). If k <K (u, E), step 2103
If not, go to step 2105. This is to give priority to those with a gentle gradient and prevent overwriting those with a steep gradient.

In step 2103, a pixel value of the reference pixel Pr (U _pr , E _pr ) is drawn at that pixel position. In step 2104,
Update K (u, E).

If the processing has been completed for all the pixels on the straight line L, the flow advances to step 1707. With the above processing, the EPI of four lines can be interpolated.

FIG. 25 shows a state in which the EPI interpolation has been completed for one frame, and cut in parallel with the uv plane at the central portion (E = E _center ) in the E-axis direction. Since E = E _center is the position of the virtual camera 205, the acquired image is an image captured by the virtual camera 205. Since the virtual camera 205 is in front of the subject 111, the spatiotemporal coordinate system (u, v, E) 20
The image cut from Step 4 is an image in which the subject 111 faces front.

(Details of Three-Dimensional Information Extraction Unit) FIG. 22 shows a three-dimensional information extraction process in the three-dimensional information extraction unit 107 (step 312).
4 is a flowchart regarding three-dimensional information storage processing (step 312) in the three-dimensional information storage unit 108.

Hereinafter, description will be made with reference to FIG.

FIG. 23 shows the detection of the corresponding point straight line on the EPI and the interpolation processing thereof. The distance information between each feature point of the subject 111 and the camera baseline is extracted from the slope of the interpolated corresponding point straight line on the EPI. Virtual camera coordinate system ( _Xv , _Yv , _Zv ) 201
FIG. 6 is a diagram showing a state of acquiring three-dimensional information in FIG.

In the EPI after interpolation, the position of E = E _{center is} a position where it is considered that there are many linear trajectories of each feature point, and therefore, the K (u, E) is stored at the position of P (u, E _center ). , E)
Is read from the memory of the computer (step 2201), and distance information is calculated using the following equation (step 2202). X _v = (a / k) x (u-u ₀ ) Y _v = (a / k) x (v-v ₀ ) Z _v = F x | a / k | . Virtual camera coordinate system (X
_{In v} , _Yv , _Zv ) 201, three-dimensional information of the subject 111 is calculated. x = F × (X _v / Z _v ) (1) y = F × (Y _v / Z _v ) (2) Also, the imaging plane coordinate system (x, y) 202 and the digital coordinate system (u,
v) 203, x = u-u ₀ (3) y = v-v ₀ (4) where (u ₀ , v ₀ ) is the digital image coordinate system (u, v) 203 The origin position of the shooting plane coordinate system (x, y) 202
Is the normalized focal length.

[0131] with respect to the subject 111, a plurality of cameras X _v
In the axial direction, shooting is performed at the position of E with the camera 0 (206) as the origin, but since the positional relationship between the subject 111 and the plurality of cameras is relative, it is necessary to move at a constant speed in the -X _v direction. This is equivalent to the case where one camera (the moving distance of the camera is D) is shooting the subject 111. Therefore, it can be expressed as x = F × ( _Xv −D) / _Zv (5). When rearranging the above equation (5), x = − (F / _Zv ) × D + F × _Xv / _Zv (6) Here, if the straight line L to the formula deformed, x = k × E -k × E pr + U pr - u 0 , and the addition, the relationship between D and E the D = a × E (a ≧ 1) (7 ) by substituting, x = (k / a) × D -k × E pr + U pr - obtain u ₀ (8). Therefore, comparing Equations (6) and (8), k / a =-(F / Z _v ), so that Z _v = -F × a / k (a ≥ 1) (9) Equation (9) From equations (3) and (4), equations (1) and (2) are as follows. X _v = (a / k) × (u-u ₀ ) (1) 'Y _v = (a / k) × (v-v ₀ ) (2)' Therefore, the virtual camera coordinate system (X _v , Y _v , Z _v ) 201, the three-dimensional position of the subject 111 is X _v = (a / k) × (u−u ₀ ) (10) Y _v = (a / k) × (v−v ₀ ) (11) Z _v = -F × (a / k) (12) However, the unit system is the virtual camera coordinate system ( _Xv , Y
_v , Z _v ) 201, so the metric unit system is used.

(Details of Three-Dimensional Information Storage) Three-dimensional information storage unit 10
At 8, the three-dimensional information of the subject 111 calculated from each EPI is stored in the memory of the computer (Step 2203) until one frame is completed for the input image 601 (Step 313).

Next, when three-dimensional information has been extracted for all pixels on E = E _center (step 2204), the flow advances to step 313.

(Details of three-dimensional information transmission) Transmission image processing unit 110
The three-dimensional information for one frame of the subject 111 is transmitted from the transmitting unit 109 to the receiving unit 114 of the received image processing unit 117 (step 314).

(Details of Reception of Three-Dimensional Information) Transmission Image Processing Unit 110
3 of one frame of the subject 111 transmitted from the transmission unit 109 of
The dimensional information is received by the receiving unit 114 of the received image processing unit 117
(Step 315).

(Details of Reconstruction / Display) FIG. 24 is a view showing a flowchart for reconstructing / displaying the subject 111 based on the three-dimensional information accumulated for one frame (step 316).

First, the three-dimensional information of one frame of the subject 111 received by the receiving unit 114 in the received image processing unit 117 is read out by the reconstruction / display unit 115 (step 2401).

Next, the viewpoint position input unit 116 acquires viewpoint position information in the virtual camera coordinate system ( _Xv , _Yv , _Zv ) 201 input by the user on the receiving side (step 2402). next,
Based on the read three-dimensional information and the input viewpoint position information, it is displayed on the reconstruction / display unit 115 (step 2403).

At the time of reconstruction and display, any software can be used as long as it can represent a three-dimensional space, such as OpenGL or VRML.

Next, it is determined whether or not the user changes the viewpoint position (step 2404). If there is a change, the process returns to step 2402; otherwise, the process proceeds to step 317.

Note that both users on the receiving / transmitting side
It is assumed that the same image processing apparatus including the transmission image processing unit 110 and the reception image processing unit 117 is provided.

(Other Applications) In the present invention,
By using the image processing device shown in FIG. 27, a three-dimensional shape of the subject 111 is acquired from a plurality of camera images without irradiating a laser or the like, and the three-dimensional shape data is recorded on a recording medium. By doing so, it is also possible to read out from the memory of the computer, reconstruct and display.

[0143]

As described above, when the image processing apparatus and the image processing method of the present invention are used, it is assumed that the virtual camera defined by the present invention is captured from input images captured by a plurality of cameras. The three-dimensionalized human face image can be acquired by fully automatic operation without human intervention.

For this reason, it is easy for users in remote locations to communicate with each other by using the existing computer display and computer without using dedicated hardware using a half mirror or the like. Became.

According to the image processing apparatus of the present invention, a three-dimensional human face image assumed to be captured by the virtual camera can be obtained. Person's face image can be obtained. This has made it possible to communicate with a sense of reality.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus.

FIG. 2 is an explanatory diagram of each coordinate system.

FIG. 3 is a flowchart illustrating an operation of the image processing apparatus.

FIG. 4 is a block diagram illustrating camera calibration.

FIG. 5 is a block diagram illustrating a state in which a background and a subject are imaged.

FIG. 6 is a block diagram illustrating processing for removing a background from an input image.

FIG. 7 is a block diagram illustrating processing for normalizing a camera coordinate system.

FIG. 8 is a block diagram showing processing for developing a background-removed image into a spatiotemporal coordinate system.

FIG. 9 is a block diagram illustrating a process of generating an epipolar plane image.

FIG. 10 is a flowchart of processing for calculating shortest distance information.

FIG. 11 is a flowchart of a process for calculating a slope of a corresponding point straight line with respect to each reference pixel.

FIG. 12 is a block diagram illustrating a process of calculating an inclination of a straight line of a corresponding point with respect to each reference pixel.

FIG. 13 is a block diagram illustrating a configuration in a case where a subject faces the front with respect to a photographing unit.

FIG. 14 is a block diagram illustrating a frequency distribution table when the subject faces the front of the imaging unit.

FIG. 15 is a block diagram illustrating a configuration in a case where a subject faces sideways with respect to a photographing unit.

FIG. 16 is a block diagram illustrating a frequency distribution table when a subject faces sideways with respect to a photographing unit.

FIG. 17 is a flowchart of an interpolation process of an epipolar plane image.

FIG. 18 is a block diagram illustrating interpolation processing of an epipolar plane image.

FIG. 19 is a block diagram illustrating interpolation processing of an epipolar plane image.

FIG. 20 is a flowchart of a process for determining whether or not the corresponding point is a straight line.

FIG. 21 is a flowchart of a process of drawing a corresponding point straight line on an epipolar plane image.

FIG. 22 is a flowchart of a process for extracting three-dimensional information.

FIG. 23 is a block diagram illustrating a process of extracting three-dimensional information.

FIG. 24 is a flowchart of a process of reconstructing and displaying using viewpoint position information and extracted three-dimensional information.

FIG. 25 is a block diagram illustrating a process of acquiring an output image when the interpolation process is completed for one frame and cut at the center.

FIG. 26 is a block diagram illustrating the operation of a conventional image processing apparatus.

FIG. 27 is a block diagram illustrating the operation of the three-dimensional shape acquisition image processing apparatus.

[Explanation of symbols]

 101 imaging unit 102 calibration processing unit 103 background processing unit 104 epipolar plane image generation unit 105 shortest distance information calculation unit 106 interpolation processing unit 107 three-dimensional information extraction unit 108 three-dimensional information storage unit 109 transmission unit 110 transmission image processing unit 111 subject 112 Background 113 Display 114 Receiving unit 115 Reconstruction / display unit 116 Viewpoint position input unit 117 Received image processing unit 201 Virtual camera coordinate system 202 Imaging plane coordinate system 203 Digital image coordinate system 204 Spatio-temporal coordinate system 205 Virtual camera 206 Camera 0 207 Camera 1 208 Camera 2 209 Camera 3 401 Calibration pattern 601 Input image 602 Background image 603 Background removal image 701 Camera coordinate system 901, 902 Epipolar plane image 1301 Nose 1302 Cheek 2501 Output image at virtual camera 205 position 2601 User (A 2602, 2607 Image processing unit 2603, 2608 Half mirror 2604, 2609 Camera 2605, 2610 TV monitor 2606 Remote user (B)

Continued on the front page F term (reference) 2F065 AA04 AA06 AA53 BB05 FF04 JJ03 JJ05 JJ26 QQ31 UU05 2F112 AC03 AC06 CA08 FA03 FA35 FA45 5B057 BA02 CA08 CA12 CA16 CB13 CC01 CD20 5C064 AA01 AA02 AC02 AC07 AD14 AD06

Claims (7)

[Claims] 1. The lens center is arranged substantially on a straight line, and at least four images each of which captures an object to be input.
Photographing means including two cameras, background processing means for performing a difference process between a previously captured background image and an input image of a subject to generate a background-removed image, and for each camera of the photographing means, Before photographing the subject, the camera parameters of each camera are calculated by performing camera calibration in advance, and each time an input image obtained by photographing the subject with each camera is obtained, the camera parameter of each camera is used. Calibration processing means for taking an origin on a baseline where the plurality of cameras are located in common and normalizing the coordinate system to a virtual camera coordinate system which is spatially orthogonal to each other through the origin; and A spatio-temporal coordinate system in which the background-removed image is based on a distance between the plurality of cameras and a baseline is a time axis. An epipolar plane image generating unit that expands on a time axis and generates an epipolar plane image that is an image of a cut plane cut in the horizontal direction of the spatiotemporal coordinate system, and, on the generated epipolar plane image, A shortest distance information calculating means for detecting the angle at which the inclination of the feature point is the gentlest and calculating distance information between the subject and a point that is the shortest distance from the baseline at which the plurality of cameras are located; Based on each epipolar plane image, a corresponding point straight line for a specific pixel set on the epipolar plane image is detected,
Interpolation processing means for interpolating an epipolar plane image; calculating, from the interpolated epipolar plane image, distance information between each feature point of the subject and a base line at which the camera is located, in the virtual camera coordinate system. An image processing apparatus comprising: three-dimensional information extracting means for outputting three-dimensional information of each feature point. 2. The shortest distance information calculating means calculates an optimal corresponding point line for the specific pixel on an epipolar plane image composed of images input from the cameras, and calculates a slope of the corresponding point line. Means for reading the slope of the corresponding point straight line from the memory, examining the frequency of appearance of the specific pixel with respect to the slope of the corresponding point straight line, and creating a frequency distribution table; Means for measuring the ratio of the texture of the part and the texture of the most convex part, and calculating the shortest distance information using the ratio in the frequency distribution table.
The image processing device according to 1. 3. The interpolation processing unit, from the shortest distance information and the number of lines of the input image appearing on the epipolar plane image, a straight line passing through a specific pixel passes on the remaining line and the A means for determining whether a straight line is a corresponding point straight line and performing interpolation if it is a corresponding point straight line, and a corresponding point where the detected corresponding point straight line does not pass through all lines but is already detected and interpolated in the middle. If there is an occlusion that intersects with the straight line, the overlapping straight line portion is already detected and
3. The image processing apparatus according to claim 1, further comprising means for giving priority to the interpolated straight line and interpolating a non-overlapping straight line portion as a corresponding point straight line of the specific pixel. 4. The three-dimensional information extracting means reads out a slope of a corresponding point straight line passing through the specific pixel stored in the memory from a memory, calculates a distance between the baseline and the specific pixel, 4. The image processing apparatus according to claim 1, further comprising a unit configured to extract three-dimensional coordinates of the specific pixel in a virtual camera coordinate system. 5. A transmitting unit, comprising: a transmitting unit; and a transmitting unit, the transmitting unit including: a unit configured to transmit, to a receiving unit, three-dimensional information of each specific pixel extracted by the unit that extracts the three-dimensional coordinates. 5. The image processing apparatus according to claim 1, wherein the unit includes means for reconstructing an image from a viewpoint of a camera at an arbitrary position based on the received three-dimensional information and outputting and displaying the image. 6. An epipolar plane image interpolated by the interpolation processing means for one frame is accumulated, and is cut perpendicularly to the time axis at an arbitrary position on the time axis of the spatiotemporal coordinate system.
5. The image processing apparatus according to claim 1, further comprising a unit configured to generate an image captured by a camera at a virtual position on the baseline. 7. When an object is photographed by photographing means including at least four cameras arranged on a substantially straight line, and the image of the photographed object is used as a time base with respect to a baseline at which the cameras are located in common. An epipolar plane image is generated by expanding the spatiotemporal coordinate system in the horizontal direction by expanding the space on the time axis of the spatial coordinate system in accordance with the distance between the cameras. In this epipolar plane image, the inclination of the feature point of the subject is calculated. Detecting the gentlest angle, calculating distance information between the subject and a point at the shortest distance from a baseline at which the camera is commonly located, and interpolating the epipolar plane image based on the distance information at the point at the shortest distance From the interpolated epipolar plane image, calculate the distance between the feature point of the subject and the baseline where the camera is located in common, and share the camera Output as three-dimensional information of each feature point in a virtual camera coordinate system having an origin at a base line to be placed; An image processing method comprising reconstructing an image from an image.