JP3328478B2 - Camera system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a camera technology for generating an image viewed from an arbitrary virtual viewpoint based on image data captured from a plurality of different positions.

[0002]

2. Description of the Related Art In a conventional imaging apparatus, light from the outside world is simply
An image was formed on an image sensor such as a CCD sensor by a lens or the like, and an image was taken. However, in such an imaging apparatus, if the imaging apparatus is fixed, it is natural that only an image at a fixed viewpoint can be physically captured, and an image from another arbitrary viewpoint cannot be obtained.

In order to solve the above problem, a method of generating an image of an arbitrary viewpoint using a plurality of imaging devices having different viewpoints without moving the physical viewpoint of each imaging device has been studied. .

[0004] For example, for a method of generating an image of an arbitrary viewpoint other than these two image pickup devices from two images picked up by two image pickup devices, see the reference document “Multi-camera image for holographic stereogram”. Intermediate Image Synthesis with Input "(Susumu Takahashi, Atsuo Honda, Masahiro Yamaguchi, Nagaaki Oyama, Toro Iwata, Technical Report of the Institute of Television Engineers of Japan, Vo
l. 17, No. 82, p. 19-23, 1993)
It is described in. The method described here is a method of obtaining a three-dimensional shape of a subject from a small number of parallax images obtained by imaging the subject from various directions, and reconstructing a new parallax image.

However, in this method, there is a restriction that the camera must be moved left and right without moving forward or rotating during imaging, and the image plane must always be kept parallel to the moving direction.

In addition to the above, from a plurality of images obtained by arranging two or more imaging devices on a straight line, an image is taken by installing a camera on the straight line and at an arbitrary position between the imaging devices. For a method of generating an image equivalent to “Construction of Inter,
medium Pictures for a Mu
ltview 3D System "(Jin Li
u and Robert Skerjanc, SPI
E Vol. 1669 Stereoscopic D
displays and Applications
III, 1992, pp. 10-19, "A Thr
e Cameras Approachfor Cal
curating Disparity and Sy
nthesizing Intermediate P
ictures ", R. Skerjanc and
J. Liu, Signal Processing: I
image Comm. 4,1991, pp. 55-6
4). In this example, 3
The main purpose is to compress the information of two-dimensional images.
We are studying a method to generate an intermediate viewpoint image from two images.

However, in this method, the subject is
The camera can be set only at a position in the vertical plane including the imaging device, and it is impossible to take an image at a position near or away from the subject.

[0008]

As described above, in the conventional means for obtaining an image from a virtual viewpoint (hereinafter referred to as a virtual camera), the degree of freedom of the position of the viewpoint which can be virtually generated is limited. . That is, the conventional virtual camera cannot generate an image when the viewpoint is moved to a position closer to the subject side or a position farther from the subject than the position where the plurality of imaging devices are set, and simply generates the image for the subject. Only the angle of the viewpoint could change.

According to the present invention, an image picked up when an image pickup device is installed at an arbitrary position where a viewpoint is moved to a subject side from each device or to a position far away from the subject by using a plurality of image pickup devices. An object is to provide a camera system that generates an equivalent virtual captured image.

[0010]

Means for Solving the Problems] The camera system according to the present invention for achieving the above object, a three or more imaging means for imaging from different positions of an object, two obtained from the three or more imaging means Means for detecting the three-dimensional position of the subject in the above captured image, virtual captured image calculation means for generating a virtual captured image viewed from an arbitrary viewpoint in a three-dimensional space based on the detected three-dimensional position, Comprising the above
The virtual picked-up image calculation means is at least 2
Generates a virtual captured image viewed from an arbitrary viewpoint
In doing so, the image from the imaging means sandwiching the arbitrary viewpoint
Select multiple sets of captured images and shoot
Generating virtual captured images from the image images,
The brightness of the same pixel in the created virtual captured image or the corresponding image
Calculates the average of the elements, and finally the virtual
It is characterized in that a captured image is generated . In addition,
In the camera system, in addition to the imaging unit and the virtual captured image calculation unit, a background image generation / supplementation unit generates a background image that supplements the small captured virtual captured image and combines the background image with the virtual captured image generated by the virtual captured image calculation unit. It is characterized by comprising a combining means.

According to the present invention, a plurality of images taken by a plurality of image pickup systems are used to convert a plurality of images from different viewpoints into images.
By obtaining the three-dimensional position coordinates of the subject in the image and reconstructing the image viewed from an arbitrary viewpoint from the coordinates,
It is possible to generate a virtual captured image equivalent to a captured image at an arbitrary position where the viewpoint is moved to a position closer to the subject than each imaging device or far away from the subject. Also,
By providing the background image generation / synthesis means, the background image is supplemented with an image generated small when the viewpoint is moved to a position far away, so that a wide-range virtual captured image can be obtained.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below in detail with reference to the drawings.

Reference Example 1 A first reference example of the present invention will be described with reference to FIG. FIG.
1 is a block diagram illustrating a schematic configuration of a virtual camera system according to the present reference example. As shown in FIG. 1, this virtual camera system includes imaging devices 11 and 12 and a position coordinate detection device 2.
1 and 22, a viewpoint position input device 30, a virtual captured image calculation device 40, and an image output device 50.

[0014] Figure 2 is an imaging apparatus of the present embodiment, the virtual camera,
It is an explanatory view of a subject position. Further, FIG. 3 is an explanatory view of a method of generating the imaging plane image of an image pickup apparatus of the present embodiment 1. FIG.
As shown in the figure, the imaging devices 61 and 62, the virtual camera 63,
When the subjects 64 and 65 are arranged, the imaging device 61,
Images 66 and 67 of the equivalent image plane corresponding to the image sensor 62 are obtained as shown in FIG. From these two images, a virtual camera system composed of the above devices
An image 68 is obtained on an equivalent imaging plane corresponding to the image sensor of the virtual camera 63.

Hereinafter, a method of generating an image of a virtual camera will be described with reference to FIG. FIG. 4 is an explanatory diagram for obtaining three-dimensional coordinates of a subject to be formed on the image sensor of the virtual camera according to the first reference example.

The imaging devices 11 and 12 in FIG.
As shown in FIG. 6, the distance between the imaging devices is, for example, d, and the objects are positioned in parallel, and images the subjects 64 and 65 in FIG. The subject image passing through the lenses of the imaging devices 11 and 12 is
An image is formed on an image pickup device such as a CCD and converted into an electric signal as image data. The position coordinate detecting devices 21 and 22 detect subject position coordinates from a plurality of captured images captured by the imaging devices 11 and 12 as follows.

The position coordinate detecting devices 21 and 22 will be described with reference to FIG. In FIG. 4, reference numeral 71 denotes an equivalent image forming plane corresponding to the image pickup device of the first image pickup device 11 or 61;
Is an equivalent imaging surface corresponding to the imaging device of the second imaging device 12 or 62, and 73 is an equivalent imaging surface corresponding to the imaging device of the virtual camera 63. The point P ₀ (x ₀ , y ₀ , z ₀ ) is an arbitrary position of the subject, and points O ₁ (−d / ₂ , ₀ , ₀ ) and O ₂ (d
/ 2,0,0) the imaging plane 71, respectively, the center position of the imaging surface 72, the point _{P 1 (x c1, y c1} , z c1) and the point P
₂ (x _c2 , y _c2 , z _c2 ) is the imaging coordinate position of the subject whose origin is the center position of each imaging surface 71, 72, and point P ₃ (x ₃ ,
y ₃ , z ₃ ) are imaging coordinate positions of a subject whose origin is the center position of the imaging surface 73 of any virtual camera to be obtained. Here, f indicates the focal length of the imaging devices 11 and 12.

The position coordinate detecting device 21 scans the image on the image forming plane 71 in the horizontal and vertical directions, and determines a point P ₁ (x _c1 , y _c1 , z _c1 ) corresponding to each pixel and a point P ₁ corresponding to the subject. ₀ (x
₀ , y ₀ , z ₀ ). Similarly, the position coordinate detecting device 22 scans the pixels on the imaging plane 72 in the horizontal and vertical directions, and determines a point P ₂ (x _c) corresponding to each pixel.
₂ , y _c2 , z _c2 ) and a point P ₀ (x ₀ , y ₀ ,
z ₀ ) is obtained.

[0019] In this reference example, to determine the position of the point P of the object by the relational expression given by the following equation _{0 (x 0, y 0,} z 0),
A point P ₃ (x ₃ , y ₃ , z ₃ ) at which the point P is formed on the image sensor, which is the image plane of the virtual camera, is obtained.

FIG. 2 shows the point P ₁ .

[0021]

(Equation 1)

A homogeneous coordinate system (x _c1h , y _c1h ,
z _c1h , w _c1h ), the perspective transformation from the point P ₀ shown in FIG. 4 to the camera C ₁ can be expressed by the following equation.

[0023]

(Equation 2)

Here,

[0025]

(Equation 3)

Is a perspective transformation matrix,

[0027]

(Equation 4)

Is a coordinate matrix representing the position and orientation of the camera.

From this equation, the point P ₁ (x _c1 , y _c1 , z _c1 ) on the camera C _{1 with} the focal length f is:

[0030]

(Equation 5)

Similarly, for the point P ₂ ,

[0032]

(Equation 6)

## EQU1 ##

Conversely, from the coordinates (x _c1 , y _c1 , 0) and (x _c2 , y _c2 , 0) of two independent cameras, a point (x
₀ , y ₀ , z ₀ ) are obtained as follows.

[0035]

(Equation 7)

The viewpoint position input device 30 inputs line-of-sight position information for obtaining an image from an arbitrary viewpoint in a three-dimensional space position. Here, the line-of-sight position information refers to information on the position of the virtual camera and its line-of-sight direction.

The virtual picked-up image calculation device 40 calculates the position of the viewpoint based on the viewpoint position information input by the viewpoint position input device 30.
An image equivalent to an image captured from an arbitrary viewpoint is generated from the two images input by the imaging devices 11 and 12. The generation method is as follows, for example.

P ₃ (x ₃ , y ₃ , z ₃ ) is obtained as follows.

[0039]

(Equation 8)

Further, when a virtual camera C ₃ at a certain position is assumed, (x ₀ , y ₀ , z ₀ ) obtained by the above calculation is obtained.
Of the virtual camera (x _c3 ,
y _c3 , z _c3 ).

[0041]

(Equation 9)

In the above equation,

[0043]

(Equation 10)

Are parameters representing the attitude of the camera, such as the line of sight, rotation, and translation of the camera. Each value is externally input by the line-of-sight position input device 30.

F ′ is the focal length of the lens of the virtual camera at an arbitrary viewpoint.

Here,

[0047]

[Equation 11]

Can be expressed as follows.

The imaging plane in the virtual camera coordinate system is z _c3h = 0
So a 3 × 4 C matrix called camera parameters

[0050]

(Equation 12)

The following can be simplified by using.

[0052]

(Equation 13)

The camera parameters include all data relating to the camera, such as the position, posture, and surface angle.

As described above, the point P ₃ (x ₃ , y ₃ , z ₃ ) located on the image sensor at the image forming position of the virtual camera can be obtained. The point P ₃ is calculated for all the pixels of the image obtained by the imaging devices, on the imaging device at the imaging position of the virtual camera to obtain a captured image in a virtual location.

The image output device 50 outputs the image data synthesized by the virtual captured image calculation device 40.

According to the above calculation, an image viewed from an arbitrary viewpoint can be reconstructed by calculating with the virtual imaging calculation means. For example, the viewpoint is moved to a position closer to the subject side than each imaging device. Such an equivalent image at an arbitrary position can be generated.

[0057] [Reference Example 2] The above has described one exemplary embodiment of the present invention, it is needless to say possible to change the seed s. For example, in the above reference example, the number of imaging devices is limited to two, but a plurality of imaging devices more than two are installed, and between the plurality of imaging devices or on the subject side of the plurality of imaging devices. If an arbitrary viewpoint is set at a closer position, a virtual camera can be realized in a wider area.
A second reference example of the present invention will be described with reference to FIG.

[0058] Figure 5 is an imaging device in the case of using the plurality of imaging elements of the present embodiment, the virtual camera, an illustration of the subject position. This virtual camera system includes a plurality of imaging devices 80-
, 80-n, subjects 81, 82,
And a virtual camera 83.

When a large number of imaging devices are installed as described above,
Even if the virtual camera 83 is set at an arbitrary position, there are always two imaging devices sandwiching the virtual camera. For example, in FIG. 5, the virtual camera 83 is sandwiched between the imaging devices 80-2 and 80-3. Therefore, the imaging devices 80-2 and 80
-3 is regarded as the two imaging devices in FIG.
An image captured by the virtual camera 83 can be generated from an image captured by one imaging device by the method described in Reference Example 1.

[0060] FIG. 6 is an explanatory view of generation method of the image pickup device and the virtual camera image plane image of the present embodiment. As shown in this figure, an image 91 of the equivalent imaging plane of the virtual camera 83 is obtained from the images 90-2 and 90-3 of the equivalent imaging plane corresponding to the imaging devices of the imaging devices 80-2 and 80-3. .

As described above, if a plurality of imaging devices 80-1,..., 80-n are set as shown in FIG. 5, a virtual camera is set at an arbitrary viewpoint over a wider range to generate an image. be able to.

According to a second reference example of [0062] Although the present invention, setting an arbitrary viewpoint position for installing the virtual camera more extensively freely, it is possible to obtain an image captured by the virtual camera.

Reference Example 3 In the two reference examples described above, the camera system that obtains a virtual captured image at an arbitrary viewpoint closer to the subject than the imaging apparatus has been described. In addition, it is possible to realize a virtual camera system when an arbitrary viewpoint is set at a position farther from the subject than the imaging device. A third reference example of the present invention will be described with reference to FIG.

[0064] Figure 7 is a block diagram showing the schematic configuration of the virtual camera system of this embodiment. As shown in FIG.
This virtual camera system has the configuration shown in FIG.
Imaging devices 11 and 12 and position coordinate detecting devices 21 and 22
And the viewpoint position input device 30 and the virtual captured image calculation device 40
In addition to the image output device 50, the background image generation /
It is configured to include the synthesizing device 41.

FIG. 8 is an explanatory diagram of the image pickup device, the virtual camera, and the position of the subject according to the present reference example. This virtual camera system includes imaging devices 101 and 102 and a virtual camera 103
And subjects 104 and 105.

[0066] Also FIG. 9 is an explanatory view of a generation method of the image pickup apparatus and the virtual camera image plane image of the present embodiment. 9, reference numeral 106 denotes an equivalent imaging plane image corresponding to the imaging device of the imaging device 101, 107 denotes an equivalent imaging surface image corresponding to the imaging device of the imaging device 102, and 108 denotes an equivalent corresponding to the imaging device of the virtual camera 103. 3 shows an imaging plane image.

As shown in FIG. 8, when the virtual camera 103 is set at an arbitrary position on the rear side opposite to the subject with respect to the imaging devices 101 and 102, as shown in FIG. The image 108 becomes small, and the background indicated by oblique lines including the periphery of the subject is required as the imaging range. In the image, for example, when the background pattern is uniform with respect to the subject of interest 104 and 105, the range of the background image can be easily extended. Therefore, the background image generation / synthesis device 41 in FIG. 7 generates a background image from the background portions of the images captured by the imaging devices 101 and 102, and combines and outputs the background image with the subject image generated by the virtual captured image calculation device 40. I do.

According to a third reference example of [0068] Although the present invention, a new background image generation and synthesizing device as shown in FIG. 7 of the 4
By setting 1 and setting an arbitrary viewpoint at a position farther than the subject, it is possible to realize a virtual camera capable of capturing images over a wider range. This reference example will be described later.
It can be applied to the embodiment of the present invention.

[ Reference Example 4] The three reference examples have been described above. By combining these three reference examples, the following imaging can be performed.

That is, in the camera system using a plurality of image pickup devices described in the second reference example, by installing the background image generation / synthesis device described in the third reference example, the camera system can be located farther away from the subject. Even if a virtual camera is installed at an arbitrary viewpoint over a wide range behind the imaging device, an image of the virtual camera can be generated.

According to the fourth reference example [0071] Although the present invention uses a new plurality of imaging devices as described above, when installing the background image generation and synthesis apparatus, setting an arbitrary viewpoint position farther than the object Thus, it is possible to realize a virtual camera capable of capturing images over a wider range.

[0072] Embodiment of the present invention] The above is a description of four exemplary embodiment of the present invention, ing to the embodiment of the present invention by adding the following changes. For example, in the second reference example, the imaging range is widened with a plurality of imaging devices, but a plurality of camera images are generated using images of a plurality of different imaging devices sandwiching the camera to be set, and the images are synthesized. Thereby, a more accurate camera image can be obtained.

[0073] The embodiment of the implementation of the present invention, FIG. 1
Explanation will be made using 0. FIG. 10 is an explanatory diagram of an imaging device, a virtual camera, and a subject position when a plurality of imaging elements according to the present embodiment are used.

This virtual camera system includes a plurality of imaging devices 110-1, 110-2,..., 110-n, subjects 111 and 112, and a virtual camera 113.

Even if the virtual camera 113 is set at an arbitrary position, there are always two imaging devices sandwiching the virtual camera. For example, in FIG. 10, the virtual camera 113 includes the imaging devices 110-1, 110-2, 110-3, and 110-
4 and so on. Therefore, for example, the imaging device 11
If 0-1 and 110-3 are regarded as the two imaging devices in FIG. 2, the virtual camera 1 is obtained from the image captured by the second imaging device by the method described in the first reference example.
An image captured at 13 can be generated.

FIG. 11 is an explanatory diagram of a method for generating an image plane image of the imaging device and the virtual camera according to the present embodiment. As shown in this figure, the imaging devices 110-1, 110-2,
From the images 120-1, 120-2, 120-3, and 120-4 of the equivalent imaging planes corresponding to the image sensors 110-3 and 110-4, an image 130 is obtained using the images 120-1 and 120-3. -1 and images 120-2 and 120-4 of image 130-2 using images 120-2 and 120-3.
Is used to obtain three camera images of the image 130-3 at the virtual camera 113. These images 130-1 and 130
The final image 131 of the equivalent imaging plane of the virtual camera 113 is obtained by synthesizing -2 and 130-3. As a method of synthesizing the three images, for example, a method of simply taking the average of the luminance of the same pixel, or extracting the contour of the subject in each image and fixing the center, and then calculating the average of the luminance of the corresponding pixel Various methods such as a taking method can be used.

As described above, if a plurality of imaging devices 110-1,..., 110-n are installed as shown in FIG.
A virtual camera can be set at an arbitrary viewpoint over a wider range to generate an image with high accuracy. Here, the case where the virtual camera is set at a position closer to the subject than the imaging device is described. However, the third reference example is similarly applied, and the virtual camera is set at a position farther from the subject than the imaging device. In this case, it is needless to say that a highly accurate virtual captured image can be obtained.

According to the implementation of embodiments of [0078] Although the present invention, it is possible to set an arbitrary viewpoint position for installing the virtual camera more extensively freely, obtain accurate image captured by the virtual camera.

[0079]

As described above, the present invention uses a plurality of images picked up by a plurality of image pickup systems to obtain a viewpoint closer to a subject than each device or at a position far away from the subject. It is possible to generate a virtual three-dimensional space image equivalent to an image captured when an imaging device is installed at an arbitrary position where the image capturing apparatus has moved. With this new feature,
The effect of being able to provide a new video service that could not exist conventionally can be obtained.

[0080] Further, to generate two or more virtual captured image viewed from an arbitrary viewpoint using the captured image from three or more imaging means, finally the by combining the two or more virtual captured image It was to obtain a virtual captured image viewed from an arbitrary viewpoint
In, it is possible to obtain a high accuracy virtual captured image.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of a camera system according to a first reference example of the present invention.

FIG. 2 is an explanatory diagram of an imaging device, a virtual camera, and a subject position in the first reference example.

FIG. 3 is an explanatory diagram of a method of generating an imaging plane image of the imaging device in the first reference example.

FIG. 4 is an explanatory diagram for obtaining three-dimensional coordinates of a subject formed on an image sensor of a virtual camera in the first reference example.

5 is an explanatory diagram of a virtual camera in the case of using a plurality of imaging elements of a second reference example of the present invention.

FIG. 6 is an explanatory diagram of a method of generating an image plane image of an imaging device and a virtual camera in the second reference example.

FIG. 7 is a block diagram illustrating a schematic configuration of a camera system according to a third reference example of the present invention.

FIG. 8 is an explanatory diagram of an imaging device, a virtual camera, and a subject position in the third reference example.

FIG. 9 is an explanatory diagram of a method of generating an image plane image of an imaging device and a virtual camera in the third reference example.

[10] an imaging device in the case of using the plurality of imaging elements in the implementation of embodiments of the present invention, the virtual camera, an illustration of the subject position.

11 is an explanatory view of a generation method of the image pickup apparatus and the virtual camera image plane image in the form above example you facilities.

[Explanation of symbols]

11, 12 ... Imaging device 21, 22 ... Position coordinate detecting device 30 ... Viewpoint position input device 40 ... Virtual captured image computing device 41 ... Background image generation / combining device 50 ... Image output device 61, 62 ... Imaging device 63 ... Virtual camera 64, 65: subject 66: equivalent imaging plane image corresponding to the imaging element of the first imaging apparatus 67: equivalent imaging plane image corresponding to the imaging element of the second imaging apparatus 68: corresponding to the imaging element of the virtual camera Virtual imaging plane image 71 ... Equivalent imaging plane corresponding to the imaging element of the first imaging apparatus 72 ... Equivalent imaging plane corresponding to the imaging element of the second imaging apparatus 73 ... Corresponding to the imaging element of the virtual camera Virtual imaging planes 80-1 to 80-n ... Imaging apparatuses 81 and 82 ... Subjects 83 ... Virtual cameras 90-1 to n ... Equivalent imaging plane images 91 corresponding to the imaging devices of the imaging apparatuses 80-1 to 80-n 91 ... Virtual cameras 83 Image sensor .. Equivalent imaging plane images 101, 102... Imaging apparatus 103... Virtual cameras 104, 105... Subject 106... Equivalent imaging plane images 107 corresponding to the imaging elements of the first imaging apparatus 107. Equivalent imaging plane image 108 corresponding to the element 108 Virtual imaging plane image 110-1 to n corresponding to the imaging element of the virtual camera Imaging apparatus 111, 112 Subject 113 Virtual camera 120-1 to n Imaging apparatus 110 Equivalent imaging plane images 130-1 to n corresponding to the imaging elements -1 to n 130-1 to n... Equivalent imaging plane images 131 corresponding to the imaging elements of the virtual camera 113 131. Image plane image

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-6-266827 (JP, A) JP-A-7-184115 (JP, A) JP-A-6-34343 (JP, A) (58) Field (Int.Cl. ⁷ , DB name) G06T 1/00 G06T 17/40 H04N 13/00-13/04

Claims (2)

(57) [Claims] 1. A three or more imaging means for imaging a subject from different positions, means for detecting a three-dimensional position of the object in two or more captured images obtained from the three or more imaging means A virtual captured image calculating means for generating a virtual captured image viewed from an arbitrary viewpoint in a three-dimensional space based on the detected three-dimensional position, wherein the virtual captured image calculating means includes at least the imaging means
From the two above, the virtual captured image viewed from any
In generating, the image capturing means sandwiching the arbitrary viewpoint
Select multiple sets of captured images, and select each set
Virtual captured images are respectively generated from the captured images of
The brightness of the same pixel of the generated virtual captured image
Calculate the average of the corresponding pixels and finally see from the arbitrary viewpoint
A camera system for generating a virtual captured image . 2. A background image generating unit that generates a background image that supplements a small generated virtual captured image in addition to the imaging unit and the virtual captured image calculating unit, and combines the background image with the virtual captured image generated by the virtual captured image calculating unit. 2. The camera system according to claim 1, further comprising a combining unit.