JP2010250452A - Arbitrary viewpoint image synthesizing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an arbitrary viewpoint image synthesizing device which generates an image in which an image capturing direction and an eye direction match by subjecting images captured by a plurality of cameras in different image capturing directions to synthesis processing. <P>SOLUTION: An image from each camera 2A, 2B and 2C is coordinate-transformed by coordinate transformation parts 21, 22 and 23 based on a parameter stored in a camera parameter storage part 26. The transformed image is transferred to a distance estimation part 24 and an image synthesis part 25. The distance estimation part 24 evaluates those three images by block matching for each two image as one set. The image synthesis part 25 composites the images of virtual camera viewpoints by extracting and attaching a corresponding pixel color from the coordinate transformed image to the created distance image of an object. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an arbitrary viewpoint image composition device applicable to, for example, a video conference system or a video phone system, and subject images from a plurality of cameras arranged around a display device according to a user's viewpoint position. The present invention relates to an arbitrary viewpoint image composition device that composes an image.

  Non-linguistic information obtained from a moving image in a video dialogue system such as a video conference system or a video phone system includes a variety of gestures and postures. Among them, the line of sight is information that greatly contributes to the transmission of each other's feelings and atmosphere, such as showing confidence in the speech and the presence or absence of trust in the partner, or prompting the partner to speak. For this reason, it is important in the television dialogue system that the users of the terminals can obtain the line of sight accurately as in the real space.

  However, it is not easy to realize this with the current television dialogue system. In order to match the line of sight when the front faces each other in a one-on-one dialogue, it is necessary to arrange a camera for photographing the user in the center of the display. In this case, an image facing the front direction can be obtained, but it cannot be said to be a practical arrangement because the camera blocks the image on the display. For this reason, the camera is arranged at the periphery of the display. In this case, an image that is not facing the front is obtained because of a deviation between the shooting direction of the camera and the viewing direction of the user. When an image in which the line of sight is shifted is used in the system, there arises a problem that the line of sight cannot be matched with the other party to talk with. This problem is particularly noticeable when using large displays.

  As described above, since the line of sight is important information for communication, the mismatch of the line of sight greatly impairs the advantage of performing dialogue using moving images. Up to now, attempts have been made to solve the problem that the line of sight does not match from the viewpoint of hardware and algorithm. Many systems using half-mirrors have been known in the past. However, there are problems such as the device being bulky, the feeling of looking into the box (the image can be seen in the back), and the camera must be in the back and in front of you. Yes, not mainstream.

  In order to solve such a problem, a method of realizing line-of-sight matching by synthesizing an image of a display center viewpoint from surrounding cameras is known (for example, see Patent Documents 1 and 2 described later).

  The network conference image processing apparatus disclosed in Patent Document 1 includes two video cameras, a display device, an image processing unit, an image composition unit, a viewpoint detection unit, an image display processing unit, an image creation unit, a display control unit, and a transmission path. Then, image information, viewpoint information, and image control information are transmitted to and from other network conference image processing apparatuses, and the shooting position and shooting range of the image to be transmitted corresponding to the change in the position of the viewpoint of the conference party are determined based on this information. The line of sight can be matched in accordance with the movement of the viewpoint of the conference person.

  Further, in the image processing apparatus and the image processing method disclosed in Patent Document 2, three-dimensional information of a subject is extracted from an input image photographed by a photographing unit in which a plurality of cameras are arranged on both sides of a display, and the three-dimensional information and The output image of the subject is displayed on the reconstruction / display unit based on the information on the free viewpoint position of the receiver. Thereby, the user's line of sight can be matched, and the image of the sender at a free viewpoint position of the receiver can be generated.

  A simple method is also known in which an image of eyes that have been photographed in advance and that matches the line of sight is pasted on the face of the conversation person (for example, see Patent Document 3 described later). In the mobile phone with an imaging function and the control method and control program of Patent Document 3, an image whose line of sight coincides with the camera is stored in advance in the storage unit, and the image of the caller during a call captured by the camera is stored. Then, it is possible to create an image in which the line of sight is corrected by pasting an image in which the line of sight stored in advance is matched, and to transmit the image in which the line of sight is corrected to the other party.

JP 11-355804 A JP 2001-52177 A JP 2005-117106 A

  However, in the image processing apparatuses of Patent Documents 1 and 2, since the three-dimensional position is estimated on the basis of feature points and mapped to a spatial model, the image quality as a two-dimensional image is deteriorated, which hinders natural communication. There was a problem. In particular, there has been a problem that deterioration in image quality or unnaturalness of the face or eyes adversely affects natural communication such as line-of-sight matching. In addition, there is a problem in that a virtual camera in which the shooting direction and the line-of-sight direction coincide with the position of the eyes displayed on the display cannot be accurately set, resulting in an unnatural line of sight.

  In addition, Patent Document 3 has a problem that, since the front eyes photographed in advance are pasted, there is a feeling that only the eyes are looking at the front, but when the whole face is viewed, an unnatural image is formed. . In particular, since the front eye is a still image taken in advance, there is a problem that when it is converted into a moving image, it becomes an unnatural image in which only the eye portion does not move and cannot be applied to a moving image.

  The present invention has been made in view of such circumstances, and an arbitrary viewpoint image synthesis device that generates an image in which a shooting direction and a line-of-sight direction are matched by combining images of a plurality of cameras having different shooting directions. The purpose is to provide.

The present invention provides a camera that captures an image of a subject that is installed at at least three locations around the display unit of the display device, the right side surface, and the left side surface, and the line of sight of the subject based on each image captured by the camera. An image composition processing unit for compositing images taken from a virtual camera in a direction,
The image composition processing unit converts a camera coordinate system from a camera coordinate system to a virtual camera coordinate system using camera parameters obtained by camera calibration of the camera, and sets a distance model of a subject with respect to the image converted by the coordinate conversion unit. An arbitrary viewpoint image synthesizing apparatus comprising: a distance estimating unit for estimating; and an image synthesizing unit for synthesizing a virtual camera viewpoint image from the distance model of the subject.

  Here, the distance estimation unit performs evaluation by block matching for each pair of images converted by the coordinate conversion unit, changes the virtual plane distance within a range in which distance estimation is performed, and evaluates at all distances. The distance model having the smallest evaluation value is obtained as the distance value of the target pixel, and the distance model of the subject is estimated.

  The distance estimation unit may perform the evaluation by the block matching using an irregular block.

  The image composition unit may extract and paste a color of a pixel corresponding to the distance model of the subject from the image converted by the coordinate conversion unit.

  In addition, the image composition unit obtains pixels of surrounding cameras corresponding to the pixels of the virtual camera from the distance value in a one-to-one relationship.

  According to the present invention, with the above-described configuration, it is possible to obtain an effect that the front image can be improved in image quality and the line-of-sight matching feeling is improved. In particular, by improving the image quality of the peripheral region of the eyes, it is possible to further improve the line-of-sight matching feeling, and it is possible to realize an arbitrary viewpoint image composition device applicable to a television conversation system.

  Further, according to the present invention, the distance estimation unit performs block matching on the image acquired from the camera with an irregular block, so that the influence of occlusion can be reduced and the synthesis result of the front image can be improved in image quality. Can do. That is, the effect of improving the line-of-sight match is obtained.

It is a block diagram which shows the structure of the arbitrary viewpoint image composition system by embodiment of this invention. It is a flowchart of the arbitrary viewpoint image composition system by embodiment shown in FIG. FIG. 2 is a block diagram of an image composition processing unit according to the embodiment shown in FIG. 1. FIG. 4 is a block diagram illustrating detailed processing contents of an image composition processing unit illustrated in FIG. 3. It is a flowchart of the image composition process part by embodiment shown in FIG. It is a detailed flowchart of a distance estimation part. It is explanatory drawing of a pinhole camera model. It is explanatory drawing of the pinhole camera model which took out the image plane ahead. It is explanatory drawing of the relationship between a world coordinate system and a camera coordinate system. It is explanatory drawing about the problem at the time of applying a pinhole camera model to an actual camera, (a) is an actual camera model, (b) is the image distortion which a non-orthogonal coordinate axis brings. It is explanatory drawing of a camera internal variable. It is explanatory drawing of the checker pattern example for a calibration. It is explanatory drawing of a CG checker pattern imaging environment. It is a table | surface which shows an example of CG imaging | photography environment parameter. It is explanatory drawing of the checker pattern example for a calibration. It is a table | surface which shows an example of the camera parameter for CG simulation. It is explanatory drawing of a trinocular camera system. It is explanatory drawing of the checker pattern example for a calibration. It is a table | surface which shows an example of the camera parameter of a trinocular camera system. It is explanatory drawing of the camera parameter in each camera coordinate system of a trinocular camera system. It is explanatory drawing of the camera parameter in the left-right camera coordinate system when a center camera coordinate system is made into a world coordinate system. It is a table | surface which shows an example of a camera parameter when a center camera coordinate system is made into a world coordinate system. It is explanatory drawing of the block shape used for matching, (a) is a case where a monocular calibration is applied, (b) is a case where an internal parameter constraint system is applied. It is a table | surface which shows an example of the camera parameter of the trinocular camera system estimated using the internal parameter constraint. It is a table | surface which shows an example of a camera parameter when a center camera coordinate system is made into a world coordinate system. It is explanatory drawing of the camera arrangement | positioning at the time of calibration and system utilization, (a) is the camera arrangement | positioning at the time of calibration, (b) is the camera arrangement | positioning at the time of system use. It is explanatory drawing of the camera arrangement | positioning for camera viewpoint conversion, (a) is the camera arrangement | positioning at the time of calibration, (b) is the camera arrangement | positioning at the time of system utilization. It is explanatory drawing of the arrangement position of a virtual camera and a virtual plane. It is explanatory drawing of the correspondence of a conversion matrix. It is explanatory drawing of a virtual plane position and a camera viewpoint conversion corresponding | compatible location relationship. It is explanatory drawing of TV dialogue system schematic. It is explanatory drawing of the block shape used for matching. It is explanatory drawing of a surrounding camera arrangement position, (a) is a camera arrangement position, (b) is a camera arrangement coordinate. It is a table | surface which shows an example of CG camera setting. It is a table | surface which shows an example of the camera parameter for CG simulation. It is a table | surface which shows an example of a center viewpoint image synthetic | combination experiment condition. It is explanatory drawing of the camera arrangement | positioning of a TV interactive system. It is a table | surface which shows an example of a TV dialog system camera parameter. It is a table | surface which shows an example of the synthetic | combination conditions of UserA, B. It is a block diagram which shows the structure of the arbitrary viewpoint image composition system by the other Example of this invention. It is a block diagram which shows the structure of the arbitrary viewpoint image composition system by the further another Example of this invention.

  Embodiments of the present invention will be described below in detail with reference to the drawings.

  FIG. 1 is a functional block diagram showing the overall configuration of an arbitrary viewpoint image synthesis system 1 according to the first embodiment of the present invention. An arbitrary viewpoint image composition system 1 shown in FIG. 1 includes a display device 5 that displays an image, and an arbitrary viewpoint image composition device that synthesizes an image in which the line-of-sight direction of the subject 6 matches the shooting direction based on the captured image of the subject 6. 10. The arbitrary viewpoint image composition device 10 includes cameras 2A, 2B, and 2C that photograph the subject 6, an image composition processing unit 3 that composes a virtual camera viewpoint image from the photographed images, and an image output unit 4 that outputs a composite image. And basically consists of

  As shown in FIG. 1, the camera 2 </ b> A is mounted on the upper center of the display device 5, the camera 2 </ b> B is mounted on the lower left side of the display device 5, and the camera 2 </ b> C is mounted on the lower right side of the display device 5. Thus, the subject 6 is simultaneously photographed by the cameras 2A, 2B, and 2C arranged at the three positions around the display portion of the display device 5.

FIG. 2 shows a flowchart of the processing procedure of the arbitrary viewpoint image synthesis system 1.
First, the subject 6 is photographed by the cameras 2A, 2B, and 2C to acquire an image (step S21: Steps are omitted hereinafter). Next, the image composition processing unit 3 synthesizes the viewpoint image of the virtual camera in which the photographing direction and the line-of-sight direction coincide with each other from the three photographed images (S22). The synthesized image is output from the image output unit 4 to the display device 5 (S23), and the virtual camera viewpoint image is displayed on the display device 5 (S24).

Next, a detailed configuration of the image composition processing unit 3 is shown in FIG.
The image composition processing unit 3 includes coordinate conversion units 21, 22, and 23 that convert images captured by the cameras 2 </ b> A, 2 </ b> B, and 2 </ b> C from a camera coordinate system to a virtual camera coordinate system, and an image converted to the virtual camera coordinate system. A distance estimation unit 24 that estimates a distance image (distance model) of the subject, an image synthesis unit 25 that synthesizes a virtual camera viewpoint image from the estimated distance model, and a camera parameter storage unit that stores camera parameters used for coordinate conversion 26 basically.

Furthermore, the detailed processing relationship of each part of the image composition processing unit 3 is shown in FIG.
Based on the parameters stored in the camera parameter storage unit 26, the coordinate conversion units 21, 22, and 23 convert the images I ₁ , I ₂ , and I ₃ from the cameras 2A, 2B, and 2C into an image I _1conv as a result of coordinate conversion. , I 2 _conv , I 3 _conv are _passed to the distance estimation unit 24 and the image composition unit 25. The distance estimation unit 24 performs evaluation by block matching for each of the three images I _1conv , I _2conv , and I _3conv for each set. Here, the virtual plane distance z is changed within the range in which distance estimation is performed, and evaluation is performed at all distances. At this time, the distance having the smallest evaluation value is set as the distance value d (i, j) at the target pixel (i, j). By performing this operation on all the pixels, the distance image of the subject can be estimated (a detailed description of this process will be described later).

Next, the image composition unit 25 extracts and pastes corresponding pixel colors from the coordinate conversion images I _1conv , I _2conv , and I _3conv to the created distance image of the subject. When the distance information of the subject 6 is obtained, the surrounding camera pixels corresponding to the virtual camera pixels can be obtained in a one-to-one relationship by geometric calculation. By performing this operation on all the pixels of the virtual camera, a virtual camera viewpoint image can be synthesized (a detailed description of this process will be described later).

FIG. 5 shows a flowchart of the image composition processing unit 3.
In FIG. 5, the coordinate conversion units 21, 22, and 23 obtain images I ₁ , I ₂ , and I ₃ from the cameras 2A, 2B, and 2C (S51), and obtain images from the cameras 2A, 2B, and 2C. The camera coordinate system images I ₁ , I ₂ , I ₃ are converted into virtual camera coordinate system images I 1 _conv , I 2 _conv , I 3 _conv and sent to the distance estimation unit 24 and the image composition unit 25 (S52).

Next, the distance estimation unit 24 _changes the virtual plane distance z in the range in which the distance estimation is performed on the images I _1conv , I _2conv , and I _3conv converted to the virtual camera coordinate system, and one set of two at all distances. Block matching is performed one by one, and the distance image d is estimated with the distance having the smallest evaluation value as the distance value of the target pixel (S53).

Finally, the image synthesis unit 25 extracts the color of the pixel corresponding to the created subject distance image d from the images I ₁ , I ₂ , and I ₃ and pastes it on the distance image d to synthesize the virtual camera viewpoint image. (S54).

FIG. 6 shows a more detailed flowchart of the distance estimation method performed by the distance estimation unit 24 in S53.
In the distance estimation method, the virtual plane distance z of the images I 1 _conv , I 2 _conv , and I 3 _conv is changed within the range in which distance estimation is performed (S61), and block matching is performed for each pair of two sheets (S62). Next, the evaluation value I _SUMALL (i, j) for distance image estimation is calculated (S63). Details of the evaluation value I _SUMALL (i, j) will be described later. It is confirmed whether or not the virtual plane distance z has been changed in all ranges in which distance estimation is performed (S64). If the change has not been made in the entire range, the process returns to S61. When the process is completed for all ranges, distance image estimation is performed and a distance image d is output (S65). Details of the distance image d will be described later.

  The details of the processing related to the arbitrary viewpoint image composition device 10 will be described below.

[Explanation of camera calibration]
First, camera calibration will be described. This relates to the coordinate conversion units 21, 22 and 23 and the camera parameter storage unit 26 of the image composition processing unit 3.
Even if the same scene is shot using cameras of the same model number, the acquired image is affected by individual differences of each camera such as lens distortion and image center shift, and therefore the same image cannot be acquired. Therefore, in order to always perform image processing under the same conditions, individual differences must be corrected. In addition, when processing is performed using images acquired from a plurality of cameras such as stereo matching based on the principle of triangulation, it is necessary to grasp not only correction of individual differences of cameras but also a spatial positional relationship. The camera calibration is a process for obtaining an internal variable indicating an individual difference inherent to the camera and an external variable indicating the spatial position of the camera. There are several methods, which can be classified into a method of placing a pattern with a known shape and observing it, and a method of observing a world with an unknown shape. As the former representative technique, the Zhang technique is used in the present invention. First, the camera model and camera parameters will be described, and then the Zhang method, which is a monocular camera calibration method, will be described. Next, application of the method to calibration in the multi-view imaging apparatus used in the arbitrary viewpoint image synthesis apparatus 10 will be described.

[Camera model and camera parameters: description of camera model and coordinate system]
We see the subject as the light reflected by the object passes through the eye lens and reaches the retina. The camera can be considered similarly except that the CCD or CMOS sensor performs the role of the retina. A simple model of this mechanism is the pinhole camera model shown in FIG. The pinhole camera is composed of an image plane 31 and a pinhole 32, and the pinhole 32 is a minute hole provided at the center of the pinhole plane 33. The pinhole plane 33 is a virtual plane that blocks all light except the light passing through the pinhole 32.

The image plane 31 corresponds to a retina or a CMOS sensor. Here, f is the focal length of the camera, z is the distance between the camera (pinhole 32) and the subject 34, y is the size of the subject 34, and _yscreen is the size of the projection image 35 of the subject 34 on the image plane 31. . In this pinhole camera model, all the light reaching the image plane 31 passes through the pinhole 32 and forms an image at a position intersecting the image plane 31. Such a projection is called a central projection. At this time, the following equation (1) is obtained from FIG.

  This indicates that the image is displayed upside down on the image plane 31. However, this makes it difficult to handle projection intuitively. Therefore, the model of FIG. 8 in which the image plane 31 and the pinhole plane 33 of FIG. 7 are interchanged is often used. A coordinate system defined with the camera at the center is called a camera coordinate system. At this time, the image coordinate system has the image center as the origin, and the x-axis and y-axis are aligned with the arrangement axes of the camera elements. The camera coordinate system in the three-dimensional space is arranged such that the point C is the origin, the optical axis is the Z axis, and the X axis and Y axis are the right hand system. At this time, the pinhole 32 becomes the projection center C, and all light rays from the subject 34 go to the projection center C. Therefore, in FIG. 8, it can be seen that the following equation (2) holds, and the image is displayed on the image plane without being inverted.

At this time, the relationship between the image coordinate system m (x _screen , y _screen ) and the camera coordinate system M (X, Y, Z) in the three-dimensional space is expressed as equation (3).

[Camera model and camera parameters: description of external parameters]
When the expression (3) is expressed linearly, it can be expressed as an expression (4).

  Here, s is a scalar. m and M are homogeneous coordinate expressions, and mean the following expressions (5) and (6), respectively.

  Here, P is a projection matrix for central projection, and is expressed by the following equation (7).

  7 and 8, the points in the three-dimensional space are described in the camera coordinate system, but in reality, the world coordinate system is often used. FIG. 9 shows the relationship between the world coordinate system and the camera coordinate system. The suffix w of the coordinate axis represents the world coordinate system, and c represents the camera coordinate system. As shown in FIG. 9, in order to convert the world coordinate system to the camera coordinate system, the world coordinate system is rotated and then translated. The rotation is expressed by the following equation (8) as a rotation matrix R of 3 rows and 3 columns, and the translation is expressed by the following equation (9) as a translation vector t of 3 rows and 1 column.

Here, θ _X , θ _Y , and θ _Z are as shown in the following equations (10), (11), and (12), and represent rotations about the respective coordinate axes. The direction of rotation is the right-handed system. Note that rotation about each coordinate axis can also be obtained from the rotation matrix R.

Where (8), the geometric relationship between the coordinates M _W in the world coordinate system and the corresponding point M _C in the camera coordinate system using equation (9) can be expressed by the following equation (13).

  R and t are collectively referred to as an external parameter of the camera.

[Camera model and camera parameters: description of internal parameters]
The above describes the definition of the ideal camera model and the camera coordinate system, but an actual camera cannot be represented by this simple model. When the lens 41 is actually attached to the camera 2 as shown in FIG. 10A, the optical axis Z1 passing through the center of the sensor 42 does not always coincide with the optical axis Z0 of the lens. This is because the sensor 42 may be tilted when attached by the adhesive 43. Further, when the pixels of the sensor 42 are not square, the scales of both coordinate axes of the image may be different. Further, as shown in FIG. 10B, both coordinate axes of the actual image are not always orthogonal. For this reason, distortion 43 may occur in the image.

Here, as shown in FIG. 11, as new parameters, u ₀ and v ₀ indicating the shift amount between the coordinates at which the optical axis and the sensor surface intersect with the image center coordinates, θ indicating the angle formed by the two coordinate axes of the actual image, Introducing k _u and k _v indicating the unit length of both coordinate axes. At this time, the coordinate system c-xy has the image center c as the origin, and the x axis and the y axis have the same scale. The coordinate system o-uv is an image coordinate system. Here, when the coordinates of the coordinate system o-uv and the coordinate system c-xy are m = [u, v] ^T and m _s = [x, y] ^T , the following equation (14) is established.

  Here, H is the following equation (15).

  Equation (16) is established from Equation (4) and Equation (14).

Here, _{P M} is the following equation (17).

Here, when fk _u and fk _v are replaced with α _u and α _v , respectively, equation (17) is expressed as the following equation (18).

Here, A and _PN are matrices represented by the following equations (19) and (20).

  This matrix A is composed of internal variables of the camera and is called an internal parameter of the camera.

[Monocular camera calibration: Explanation of calibration method using known pattern]
Calibration using a known pattern is a method of estimating internal and external parameters of a camera by observing an object having a known three-dimensional shape and comparing it with a projection image onto the two-dimensional shape. The principle will be described below.

  From the equations (4), (15), and (18), the geometrical correspondence between the point m in the image coordinate system of the camera and the point M in the world coordinate system is expressed by the following equation (21).

  Here, P is the following equation (22).

  Here, the projection matrix P is a 3 × 4 matrix, and is composed of 11 variables including internal variables and external variables. When the expression (21) is expanded, the following expression (23) is obtained from one world coordinate point and the projected point on the image coordinate.

  Here, when the equation (23) is modified, it is expressed as the following equation (24).

  Here, when n corresponding points are acquired from one image, the following equation (25) is obtained by simultaneous equations (24).

B is a 2n × 12 matrix. If n ≧ 6, the solution of 11 variables of the projection matrix P can be obtained as an eigenvector for the minimum eigenvalue of B ^T B. By substituting the equations (8), (9), and (19) into the equation (21), the following equation (26) is obtained.

  Here, since the object is always in front of the camera, the following equation (27) is obtained.

  The following equation (28) is obtained by calculating the above equation.

  Each camera parameter can be estimated by acquiring 6 or more known three-dimensional points as described above. These camera parameters are stored in the camera parameter storage unit 26.

[Monocular Camera Calibration: Explanation of Zhang's Camera Calibration Method]
Zhang's camera calibration is a method for estimating camera parameters based on feature points in an image obtained by photographing a planar pattern 51 as shown in FIG. 12 from multiple directions. Since only the camera and the planar pattern 51 can be used for estimation, calibration can be performed with a simpler apparatus as compared with a method using a three-dimensional shape. The principle will be described below.

  The relationship between the point M (X, Y, Z) on the three-dimensional space and the point m (x, y) on the image plane is expressed as the above-described equation (4). At this time, if the plane 51 with the checker pattern for calibration is set to z = 0, the following equation (29) is obtained.

  Accordingly, the correspondence between the points on the checker pattern and the points on the image can be represented by a 3 × 3 matrix expressed by the following equation (30). This matrix is called a homography matrix.

  Here, the expression (30) is expressed in the form of the following expression (31).

At this time, by utilizing the fact that r ₁ and r ₂ are unit vectors orthogonal to each other due to the nature of the rotation matrix R, the following constraint conditions of Equations (32) and (33) are obtained.

Here, A ^−T = (A ^T ) ⁻¹ .

Since two equations can be obtained from one homography matrix, six unknowns included in H = [h1 h2 h3] can be obtained by obtaining three or more homography matrices. The calculation method is shown below. First, A- ^{TA- 1} has symmetry and is represented by the following equation (34).

  At this time, a vector in which the elements of B are arranged is defined by the following equation (35).

  Thus, the following equation (36) is established.

  Thereby, the expression (32) and the expression (33) can be expressed as the following expression (37).

  Here, when n images are obtained, the equation (38) is obtained by vertically connecting the n equations described above.

V is a 2n × 6 matrix. Thus, b is obtained as an eigenvector corresponding to the minimum eigenvalue of V ^T V. If B is Motomare by obtaining b, internal parameters of the camera from B = .lambda.A ^-T A ^-1 can be obtained by the equation (39).

  If A is obtained from this, the external parameter is also obtained from the equation (30) as the following equation (40).

  An ideal image photographed in the environment of FIG. 13 was created by CG, and camera calibration was performed by the Zhang method. The distance from the camera 2-1 to the checker pattern plane 51 was 1700 mm. An example of each parameter of the photographing environment at this time is shown in FIG. In this environment, 13 images are acquired while changing the angle of the checker pattern. As an example, FIG. 15A shows a camera in which a checker pattern is directly opposed to the camera, and FIGS. 15B to 15F show that the angle is changed little by little. In FIG. 15, (x, y, z) indicates how many times the checker pattern is rotated around each axis.

Camera calibration by the Zhang method was performed using 13 images with the rotation direction changed as shown in FIG. The OpenGL library is used for parameter estimation, and the output calculation result is an external parameter of the camera with respect to world coordinates with the upper left of the checker pattern as the origin. FIG. 16 shows the internal parameters and external parameters estimated by this. The external parameters are calculated for the image shown in FIG. Since the image center (u ₀ , v ₀ ) is (511.5, 383.5) in an ideal camera with a resolution of 1024 × 768 pixels, it can be confirmed that the image center is well calculated in the internal parameters. It can also be confirmed that there is almost no rotation in the external parameters, and the distance between the checker pattern and the camera is approximately 1700 mm and they are facing each other.

[Explanation of multi-eye camera calibration]
When obtaining the positional relationship of two or more cameras, the global coordinate system must be made common to all the cameras, and therefore, external parameter estimation must be performed simultaneously. Here, consider a trinocular camera system as shown in FIG. 17 in which two cameras 2-2 and 2-3 are added to the configuration of FIG. In this system, a plurality of scenes in which the angle of the checker pattern 51 is changed are simultaneously photographed from three eyes (cameras 2-1, 2-2, 2-3), and camera calibration is performed using the image group. Examples of images taken from the left camera 2-3 are shown in FIGS. 18A, 18C, and 18E, and examples of images taken from the right camera 2-2 are FIGS. 18B, 18D, and 18F. Shown in (X, y, z) indicates how many times the checker pattern 51 has been rotated around each axis, and images of the same rotation angle are simultaneously acquired from the left and right cameras 2-2 and 2-3. When shooting is performed in this way, as shown in FIG. 18, it can be seen that the images of the checker pattern 51 are displayed at positions shifted from the image centers of the left and right cameras 2-2 and 2-3. . FIG. 19 shows camera parameters estimated by performing the above-described camera calibration on the captured image group in this way. From FIG. 19, it can be confirmed that the center of the image is shifted in both the left and right cameras as compared with the monocular calibration result of FIG.

Here, as shown in FIG. 20, the camera parameters of the central camera 2-1 and the left and right cameras 2-2 and 2-3 are obtained. This time by the world coordinate system as shown in FIG. 21 again set in the camera coordinate system of the center camera 2-1, the external parameters of the right and left cameras _{2-2,2-3 [R 'left | t'} left], [ R ′ _right | t ′ _right t] can be obtained as equations (41) and (42).

This external parameter represents the absolute positional relationship between the central camera 2-1 and the left and right cameras 2-2 and 2-3 in the world coordinate system, that is, real space. FIG. 22 shows the calculation result of this new external parameter. Theoretically, t _left = [− 600, 0, 0] in the case of the left camera and t _right = [600, 0, 0] in the case of the right camera, but the calculation results are greatly different. This is because, as described above, the camera parameters are calculated in the order of the internal parameters and the external parameters. Therefore, if there is an estimation error in the internal parameters, the influence of the error appears when the external parameters are estimated. Therefore, this method cannot correctly determine the positional relationship between a plurality of cameras.

[Explanation of camera calibration method constraining internal parameters]
In the camera calibration method applying the above monocular calibration as it is, it was shown that the estimation error of the internal parameter affects the estimation of the external parameter. Therefore, in this section, a method for reducing the estimation error of external parameters by performing calibration with constraints on internal parameters will be described. In a highly accurate camera and lens used in the present system, the optical axis center (u ₀ , v ₀ ) of the internal parameter is at the center position of the image, and it can be approximated that there is no image distortion. If the image size obtained by the camera at this time is WIDTH × HEIGHT, (u ₀ , v ₀ ) indicating the image center and θ indicating image distortion can be fixed as shown in equation (43).

  Thus, equation (19) can be rewritten as the following equation (44).

  In order to further improve the estimation accuracy of the internal parameters, an image group is individually acquired at a position where the checker pattern appears in the center for each camera. At this time, it is assumed that only one scene for obtaining an external parameter is photographed simultaneously from three eyes. FIG. 23 shows a difference between image groups when the above-described monocular camera calibration is applied as it is and when the internal parameter constraint method of this section is applied. FIG. 23A shows a case where the monocular camera calibration is applied, and FIG. 23B shows a case where the internal parameter constraint method is applied. A thick frame image is an external parameter calculation target image. In the internal parameter constraint method of FIG. 23B, in addition to the calibration image 61 for each camera, an image 62 of three-lens simultaneous photographing is added.

  Camera calibration was performed on the trinocular camera system using internal parameter constraints. The camera arrangement is the same as in FIG. The estimation result is shown in FIG. Furthermore, the camera parameters (external parameters [R | t]) of the left camera 2-2 and the right camera 2-3 when the camera coordinate system of the central camera is set to the world coordinate system using the expressions (41) and (42). Is shown in FIG. Accordingly, the camera position can be estimated with an error of about 2 mm, and this method is effective for camera calibration of the multi-camera system.

  The coordinate system and the camera parameters have been described above, and the monocular camera calibration method, particularly the Zhang calibration method, has been described in detail. Furthermore, the application of the Zhang method to a multi-lens camera system has been described. In addition, by performing a camera calibration simulation using a CG image, the proposed method was able to estimate the camera position with an error of about 2 mm.

[Explanation of central viewpoint image synthesis using multi-view camera]
Here, a method of synthesizing a central viewpoint image using a multi-lens camera for synthesizing a face area image capable of matching the line of sight will be described. First, the prototype system is described, and then subject distance estimation using projective transformation is described. Furthermore, a method for synthesizing the viewpoint image from the center position of the display based on the estimated distance information will be described.

[Composition of Central Viewpoint Image: Camera Viewpoint Conversion Using Projective Transformation]
This system has four eyes (cameras 2A, 2B, 2C, 2D) as shown in FIG. 26A, and three eyes (cameras 2A, 2B, 2C) when the system is used as shown in FIG. ). In this section, in order to explain the conversion of the camera viewpoint, first, a system for arranging two cameras (cameras 2C and 2D) at the time of calibration as shown in FIG. 27A and a camera viewpoint conversion as shown in FIG. Consider a system with a single camera (camera 2-1) placement. First, multi-eye camera calibration is performed on the system shown in FIG. The camera parameters obtained at this time are parameters when the camera 2D is set as the origin of the world coordinate system as in the equation (41), and the rotation matrix R ′ ₀ and the translation vector t ′ ₀ of the camera 2D are expressed in the equation (45), Expression (46), the rotation matrix R ′ ₁ and translation vector t ′ ₁ of the camera 2C are expressed as Expressions (47) and (48).

  At this time, the equations (47) and (48) are values indicating how much the camera 2C is rotating and translating with respect to the camera 2D. In FIG. 27B, one-camera viewpoint conversion can be considered with the camera 2D as the virtual camera 2-0 and the camera 2C as the camera 2-1.

Consider a case where a virtual camera is arranged at the position of the camera 2-0 as shown in FIG. 28 and a virtual plane parallel to the virtual camera (camera 2-0) is arranged at a position separated by a distance z. When the origin of the world coordinate system is changed to a point where the optical axis (Z) of the virtual camera 2-0 intersects the virtual plane, the translation vectors t _im and t ₁ of the virtual camera coordinate system and the camera 1 coordinate system with respect to the new world coordinate system. Can be expressed as the following equations (49) and (50), respectively.

At this time, the rotation matrices R _im and R ₁ of the virtual camera coordinate system and the camera 1 coordinate system with respect to the world coordinate system can be expressed as the following equations (51) and (52), respectively.

Further, since the virtual camera is represented by replacing the camera 2-0 as it is, the internal parameter A _im of the virtual camera is represented by the following equation (53).

  Next, projective transformation for a virtual plane is considered. FIG. 29 shows the correspondence of the transformation matrix in this projective transformation.

The relationship between the image coordinate systems m _im and m _{1 of} each camera and the world coordinate system M is expressed as the following equations (54) and (55), respectively.

  At this time, since the virtual plane is represented by Z = 0 in the world coordinates, the equation (55) can be transformed into the following equation (56) with respect to the point M on the virtual plane.

Here, H ₁ (z) is a 3 × 3 homography matrix indicating the correspondence between the plane of the world coordinate system and the image coordinate system of the camera 1, and is a function of z. Similarly, the following equation (57) is obtained by transforming the equation (54).

  Next, the equation (56) is modified to obtain the following equation (58).

  By substituting this into the equation (57), the following equation (59) is obtained.

Equation (59) shows the correspondence between the point m _im on the image coordinates of the virtual camera and the point m ₁ on the image coordinates of the camera 2-1. Therefore, by using the equation (59), an image captured by the camera 2-1 can be converted into an image captured from the virtual camera viewpoint. However, at this time, only the portion where the subject is present at the distance z of the assumed plane assumed earlier is correctly converted, and the other portions are erroneously converted. For example, when a sphere is a subject as shown in FIG. 30 and z is changed so as to pass through the sphere on the virtual plane, only the thick line portion shown in FIG. 30 is correctly converted. 30A shows a case where the virtual plane is at the position z, FIG. 30B shows a case where the virtual plane is at z + α, FIG. 30C shows a case where the virtual plane is at z + 2α, Thick lines indicate which part of the sphere that is the subject is correctly converted.

[Description of subject model creation]
Although the correspondence relationship between the virtual camera installed in the center of the display and the camera 2-1 installed in the periphery is shown by the equation (56), this applies to all other cameras installed around the display 5 as well. Applicable. Consider the present TV dialogue system shown in FIG. As in FIG. 1, in the system of FIG. 31, the camera 2-1 (camera 2A in FIG. 1) is at the upper center of the display device 5, and the camera 2-2 (camera 2B in FIG. 1) is at the lower right side of the side of the display device 5. In addition, the camera 2-3 (camera 2C in FIG. 1) is mounted on the lower left side of the side surface of the display device 5. In FIG. 31, since it is the figure seen from the back side of the display apparatus 5, the camera has changed right and left. When the image coordinate system of the camera 2-2 and camera 2-3 is m ₂ and m ₃ and the projective transformation matrix is H ₂ (z) and H ₃ (z), the following equations (60) and (61) are obtained. .

The coordinate conversion units 21, 22, and 23 obtain images I ₁ , I ₂ , and (1) obtained from the expressions (59), (60), and (61) from the camera 2-1, the camera 2-2, and the camera 2-3, respectively. by applying the I _3, obtained image _I 1Conv obtained by converting the respective images to the viewpoint of the camera 2-0 is a virtual _camera, I _2conv, the _{I 3conv.}

Next, the distance estimation unit 24 performs evaluation by block matching for each of the three images I _1conv , I _2conv , and I _3conv obtained by the conversion. As shown in FIG. 32, various patterns are conceivable for the block shape used for matching. In this section, rectangular matching will be described as an example. When the pixel of interest is (i, j), the absolute value difference sum _ISUMαβ of the pixel values of the converted image shown in the following equation (62) is used as the evaluation value.

Here, α and β are corresponding camera numbers, m is a horizontal block size, n is a vertical block size, and I _αconv (i ′, j ′) and I _βconv (i ′, j ′) are two images used for evaluation, respectively. The pixel value at the position (i ′, j ′) of the converted image is shown. The result obtained by performing this equation (62) for all three combinations of three sheets is taken as an evaluation value I _SUMALL, and is shown in the following equation (63).

Here, W _αβ represents a weight for each set of absolute value difference sums, and is manually adjusted so as to be optimal in the range of 0 to 1. Next, the virtual plane distance z is changed within the range in which the distance is estimated, and the evaluation is performed using the equation (63) at all distances. The evaluation value at each distance is expressed as _ISUMALL (i, j; z), and the distance with the smallest _ISUMALL value is the distance value d (i, j) of the pixel of interest. be able to. By performing this operation on all the pixels, the distance image of the subject can be estimated.

[Description of determination of pixel value]
The image composition unit 25 pastes the corresponding pixel color from the surrounding camera to the created distance image of the subject. When the distance information of the subject is obtained, the surrounding camera pixels corresponding to the virtual camera pixels are calculated one-to-one in the geometric calculation as shown in the equations (56), (57), and (58). It can be found in a relationship. The pixel value of the _target pixel of the virtual camera calculated with respect to the distance value z is set as I _im (i, j, z), and the pixel values of the surrounding cameras corresponding to the _target pixel are set as I _αconv (i, j, z) ( In the case of α = 1, 2, 3), the calculation method of the color to be pasted is shown in equation (65).

Here, U _α (α = 1, 2, 3) represents a weight for each pixel value, and is manually adjusted so as to be optimal in the range of U ₁ + U ₂ + U ₃ = 1. By performing this operation on all the pixels of the virtual camera, it is possible to synthesize the virtual camera viewpoint image.

  The viewpoint position conversion method using the acquired camera calibration data and the subject distance model creation method using the conversion result have been described above. Furthermore, the method of combining images from the central viewpoint of the display by pasting the color of the surrounding camera to the distance model was explained. Here, an example in which there are three cameras has been described, but this method can be applied even when three or more cameras are used. For example, when there are four cameras, this method is applied assuming that there are two sets of three cameras, and two output composite images may be integrated. That is, when the cameras 2A, 2B, 2C, and 2D are arranged, for example, two sets of (2A, 2B, 2C) and (2B, 2C, 2D) are considered, and the combined image of each set is displayed. Two sheets are generated, and these two sheets are further integrated. A simple average, a weighted average, or the like can be used for image integration. That is, if there are at least three cameras, an arbitrary viewpoint image can be synthesized using this method, and the front image can be further improved in image quality by using three or more cameras.

[Explanation of optimal parameters for face area image composition]
Next, create an image of the surrounding camera using CG, perform the central viewpoint image synthesis simulation based on it, and optimize the parameters when the number of cameras, camera arrangement, and block shape at the time of matching are changed explain.

[Influence on composition accuracy by camera placement]
As shown in FIG. 33, five locations 2A, 2B, 2C, 2D, and 2E are considered as camera placement positions around the display device 5. FIG. 33A shows the camera arrangement position, and FIG. 33B shows the camera arrangement coordinates. The subject 6 is located on the other side of the display device 5, and FIG. 33 is a view seen from the back side of the display device 5. The cameras 2A, 2B, and 2D are disposed above the display unit center (virtual camera position), and the cameras 2C and 2E are disposed below the display unit center (virtual camera position). The cameras 2B and 2C are arranged on the right side with respect to the center of the display unit, and the cameras 2D and 2E are arranged on the left side with respect to the center of the display unit. At this time, all the cameras are arranged so that the optical axis is directed to the point (0, 0, 1700). The CG face model of the subject 6 is arranged so that the center is located at the point (0, 0, 1700). Six camera viewpoint images including the central virtual camera are created by CG. FIG. 34 shows the camera settings at this time. Among these, a plurality of camera arrangements are selected, and a virtual camera viewpoint image (hereinafter referred to as a central viewpoint image) is synthesized.

  In addition, each camera parameter acquired by camera calibration is used for calculation. Calibration using a CG checker pattern is performed, and camera parameters at this time are shown in FIG. In FIG. 35, the virtual camera is set to “0”. Further, other synthesis conditions in the simulation are shown in FIG.

  This composite image was generated satisfactorily to such an extent that the distance estimation of the face surface shape was able to show the roundness of the face regardless of which two cameras were used. In addition, the central viewpoint image could be synthesized using any two cameras. However, many composition errors occurred near the eyes. This is considered to be because the distance estimation is wrong in the occlusion area which is an area invisible from one camera or both cameras. In addition, when focusing on the nose region, many combinations of nose regions occurred in all combinations. This is also an influence of the occlusion area, and the influence of this occlusion cannot be avoided even if any two are selected.

  Therefore, based on the cameras 2C and 2E having the least influence of occlusion, the central viewpoint image was synthesized using three cameras including the camera 2A camera or the camera 2B. As a result, it was possible to estimate the distance near the eyes more accurately by using three cameras. Moreover, although both obtained better results than using two cameras, occlusion could be avoided more by using cameras 2B, 2C, and 2E. However, the nose region was synthesized better using the cameras 2A, 2C, and 2E. This means that at least one camera is arranged at the top and bottom with respect to the center of the display unit (virtual camera position), and at least one camera is arranged at the center of the display unit.

[Influence on composition accuracy by block shape]
Next, when three cameras were used, the central viewpoint image was synthesized by changing the block shape. The combinations of the three units are (2A, 2C, 2E) and (2B, 2C, 2E), the block shape is the rectangle shown in FIG. 32 (a), the rhombus shown in (b), and the irregularity shown in (c). Three types of shapes were evaluated.

As a result, although the processing amount was halved compared with the rectangular matching, the same synthesis result was obtained.
Next, as a result of distance estimation by irregular shape matching, it was possible to synthesize eye region images in which the influence of occlusion was hardly seen. As for the nose region, the synthesized result was better when the ACE camera was used.

  From the above results, in this system, three locations of 2A, 2C, and 2E are used for camera arrangement, and irregular block matching is used for matching. That is, as a result of investigating the effects of the camera arrangement, the number of cameras, the block shape at the time of matching, etc., if the cameras are arranged in three places, the upper part of the display, the lower right part, and the lower left part, the occlusion influence at the time of eye region synthesis can be reduced. I can do it. Also, the synthesis result was better when the block matching was performed with the irregular block than when the matching was performed with the rectangular block.

[Description of face area image composition using real images]
The subject 6 was photographed with the camera arrangement shown in FIG. The subject 6 sits at a position 1700 mm from the display 5, and each camera is arranged to capture the upper body of the subject. First, camera calibration was performed for all cameras. The result is shown in FIG. The origin of the world coordinate system is the camera 2-0. As a result of the calibration, it can be seen that the distance in the x direction between the camera 2-2 and the camera 2-3 is about 1300 mm, and the distance in the y direction between the camera 2-1 and the camera 2-0 is about 400 mm. This estimation result agrees with an approximate actual measurement value. After the calibration, a subject with a line of sight directed toward the center of the display (hereinafter referred to as a front line of sight) was photographed with four cameras including a comparative camera 2-0.

  Two subjects were User A and User B, and a central viewpoint image was synthesized using three images excluding the camera 2-0. FIG. 39 shows the synthesis conditions for each subject. FIG. 39 shows parameters adjusted for each subject, only the distance estimation range is different, and other synthesis conditions are common. As a result, it was confirmed that the distance estimation of the face area was performed well. In addition, the central viewpoint image of the frontal line of sight for the two subjects could be satisfactorily synthesized for the eye area, the nose area, and the mouth area.

[Description of gaze direction change image]
In reality, during the conversation, people's gaze constantly changes, so it must be able to cope with it. Therefore, in order to confirm the effect on the composite image due to the change in the line-of-sight direction, the central viewpoint image was synthesized using an image in which the line-of-sight direction was changed vertically and horizontally.

  The synthesis conditions at this time are the same as in FIG. As a result, the central viewpoint images in each line-of-sight direction were successfully synthesized. In addition, even when the line-of-sight direction was changed, the distance could be estimated satisfactorily without making a mistake in matching during distance estimation. In addition, changes in the position of the skin and eyebrows due to eye movement were also successfully synthesized.

[Virtual camera position change image]
In an actual usage environment, the user is not always in the center front position of the display. For example, when the body is tilted, if the head position changes, the appearance of the conversation partner should change. Therefore, in order to obtain a visual effect close to real space, it is desirable that a virtual camera can be installed at an arbitrary position on the display in accordance with the position of the user's head. Therefore, image synthesis was performed at a position other than the central viewpoint while changing the virtual camera position. Here, let us consider a situation in which the subject remains in the same position as before and only the virtual camera position is moved. The virtual camera position can be obtained at an arbitrary position by calculating based on the parameters of the camera 2-0.

  As the image, a front view direction image was used, and the virtual camera position was changed from (−600, 0, 0) to (600, 0, 0) in 300 mm increments in the x direction. Note that the optical axis of the camera is always directed to the point (0, 0, 1450). The synthesis conditions at this time are the same as in FIG. As a result, the visual effect by the virtual camera position change was obtained. In particular, it was confirmed that the line-of-sight direction changed due to the change in the camera position, compared with the case where the image was synthesized from the center position of the display under the same conditions. Moreover, although the area | region like a level | step difference was seen in the nose upper area | region, it is thought that this can be reduced by making an estimated width | variety small from 3 mm of present conditions.

[Face region image composition using moving images]
A blinking video was taken at 30 fps by synchronizing the three eyes of the camera 2-1, camera 2-2, and camera 2-3 of the constructed system, and a central viewpoint image was synthesized for each frame.

  As a result, it was confirmed that the synthesis was performed well even during the blinking operation. Since this method uses arbitrary viewpoint image composition based on distance estimation without processing such as eye extraction, so long as the subject is within the distance estimation range regardless of whether the eyes are open or closed In principle, it should not affect the synthesis results. From this, it can be said that this method is robust against the movement of the subject.

  As described above, the display central viewpoint image is synthesized using the TV dialogue system. First, the calibration data of the camera system is shown, and the processing result for the acquired image in the front line-of-sight direction is shown. As a result, we confirmed that this method can synthesize the central viewpoint image. In addition, processing was performed on an image whose line-of-sight direction was changed vertically and horizontally, and it was confirmed that a central viewpoint image can be synthesized regardless of the line-of-sight direction. It was also confirmed that the visual effect of moving the camera position can be obtained by rewriting the parameters of the virtual camera. Finally, we performed processing on the continuous images that captured the blinking action, and confirmed that it was possible to cope with the action of closing the eyes like blinking.

  As described above, according to the arbitrary viewpoint image composition device of the present invention, the display device, the imaging devices installed in at least three locations around the display unit of the display device, the lower right portion, and the lower left portion, and the image Since the image processing apparatus includes the synthesis processing unit, the front image can be improved in image quality, and the effect of improving the line-of-sight match can be obtained.

  According to the invention, in the image composition processing unit, a coordinate conversion unit that converts a viewpoint position using camera calibration data of the imaging apparatus, and a subject distance model converted by the coordinate conversion unit are created. Since it has a distance estimation unit and an image synthesis unit that synthesizes an image by pasting the color acquired by the imaging device to the subject distance model, the front image can be improved in image quality, and the line-of-sight matching feeling is improved. Is obtained. In particular, by improving the image quality of the peripheral region of the eyes, it is possible to further improve the line-of-sight matching feeling, and it is possible to realize an arbitrary viewpoint image composition device applicable to a television conversation system.

  In addition, according to the present invention, the distance estimation unit includes a block matching unit that performs block matching on the image acquired from the imaging device with an irregular block, thereby reducing the influence of occlusion and combining front images. The image quality can be improved. That is, the effect of improving the line-of-sight match is obtained.

  The present invention is not limited to the configuration of the above embodiment. Another embodiment of the present invention will be described in detail with reference to the drawings. FIG. 40 is a block diagram showing the overall configuration of this embodiment. The same parts as those in FIG. 1 are denoted by common reference numerals. The configuration of the system 101 is the same as that of the system 1 in FIG. 1, but the arrangement of the cameras 2B and 2C of the arbitrary viewpoint image composition device 110 is different. The camera 2 </ b> A is installed on the center of the display device 5, the camera 2 </ b> B is installed on the lower left side of the display device 5, and the camera 2 </ b> C is installed on the lower right side of the display device 5. Thus, the area of the triangle connecting the three cameras 2A, 2B, and 2C is maximized, and there is an advantage that the central viewpoint image can be easily obtained in a wider range than the configuration of FIG.

  Still another embodiment of the present invention will be described in detail with reference to the drawings. FIG. 41 is a functional block diagram showing a configuration example of this embodiment. The configuration of the arbitrary viewpoint image composition device 210 of the system 201 of this embodiment is different from the arbitrary viewpoint image composition device 10 of FIG. 1, and includes an image transmission unit 4500 and an image reception unit 4502 instead of the image output unit 4. . In the arbitrary viewpoint image composition device 210, the image photographed by the imaging device is composed by the image composition processing unit 3, and the synthesized image is connected to the external network 4501 via the image transmission unit 4500, and another arbitrary viewpoint image composition device. 210 and the like, and an image transmitted from another arbitrary viewpoint image composition device 210 or the like via the image receiving unit 4502 is displayed on the display device 5. In this way, this system 201 is suitable for conducting a video conference because users in remote places can communicate with each other by matching the line of sight via the network 4501.

DESCRIPTION OF SYMBOLS 1 Arbitrary viewpoint image system 2A, 2B, 2C, 2D, 2E Camera 3 Image composition process part 4 Image output part 5 Display apparatus 6 Subject 10 Arbitrary viewpoint image composition apparatus 21, 22, 23 Coordinate conversion part 24 Distance estimation part 25 Image composition Unit 26 Camera parameter storage unit 31 Image plane 32 Pinhole 33 Pinhole plane 34 Subject 35 Projected image 41 Lens 42 Sensor 43 Adhesive material 51 Checker pattern plane 101 System 110 Arbitrary viewpoint image composition apparatus 201 System 210 Arbitrary viewpoint image composition apparatus 4500 Image Transmission unit 4501 Network 4502 Image reception unit I ₁ , I ₂ , I _{3 Captured} image I 1 _conv , I 2 _conv , I 3 _conv Coordinate conversion image

Claims (5)

A camera that shoots an image of a subject, installed at at least three locations on an upper part, a right side part, and a left side part around the display unit of the display device;
An image composition processing unit configured to synthesize an image photographed from a virtual camera in the line-of-sight direction of the subject based on each image photographed by the camera;
With
The image composition processing unit
A coordinate conversion unit for converting from a camera coordinate system to a virtual camera coordinate system using camera parameters obtained by camera calibration of the camera;
A distance estimation unit that estimates a distance model of a subject with respect to the image converted by the coordinate conversion unit;
An arbitrary viewpoint image composition apparatus comprising: an image composition unit that composes an image of a virtual camera viewpoint from the distance model of the subject. The distance estimation unit
The image converted by the coordinate conversion unit is evaluated by block matching for each set of two images, the virtual plane distance is changed within the range of distance estimation, the evaluation is performed at all distances, and the evaluation value is the smallest distance The arbitrary viewpoint image composition apparatus according to claim 1, wherein the distance model of the subject is estimated by obtaining the distance value of the target pixel. The distance estimation unit
The arbitrary viewpoint image synthesis apparatus according to claim 2, wherein the evaluation by the block matching is performed by an irregular block. The image composition unit
4. The arbitrary viewpoint image composition apparatus according to claim 1, wherein a color of a pixel corresponding to the distance model of the subject is extracted and pasted from the image converted by the coordinate conversion unit. The image composition unit
5. The arbitrary viewpoint image composition apparatus according to claim 4, wherein pixels of surrounding cameras corresponding to the pixels of the virtual camera are obtained in a one-to-one relationship from the distance value.

Images