JP6223226B2 - Camera parameter calculation apparatus and program thereof - Google Patents

Description

  The present invention relates to a camera parameter calculation apparatus that calculates camera parameters of an IP stereoscopic camera from camera parameters of a virtual imaging camera, and a program thereof.

  In the conventional integral photography (IP) system, a subject is imaged through a lens array in which a plurality of microlenses are arranged with one camera. At this time, since the camera captures an image of the focal plane of the lens array, each micro lens constituting the lens array functions in the same manner as the micro camera. As a result, an image obtained by imaging the subject through the lens array becomes an element image group in which minute images (element images) corresponding to the positions of the minute lenses are arranged. This elemental image group is an image in which light information from a subject is recorded, and the number of light rays that can be recorded depends on the resolution of the camera. For this reason, a high-resolution camera is required for imaging by the IP method.

  Here, in the IP system, it is possible to realize an increase in the number of pixels of the imaging unit by using an imaging device in which cameras are replaced with a densely arranged camera array instead of the imaging lens array. According to this method, it is necessary to control the positions of a plurality of cameras with high accuracy.

  Conventionally, a technique for generating an IP element image group using computer graphics (CG) has been studied. In this conventional technology, a virtual three-dimensional space is generated in a computer, a three-dimensional subject and a background (set) are reproduced in the same manner as in the real world, and light rays from the subject are tracked in the virtual three-dimensional space. Generate element images. As an element image group generation method, the following method has been proposed in order to generate a video that pops out before the display surface of the display device.

  Non-Patent Document 1 describes a method of tracking a light beam from a subject through a depth control lens that controls the light beam from the subject. In Patent Documents 1 and 2 and Non-Patent Document 2, the pixel for generating the element image group and the center point of the lens corresponding to each pixel are connected, and when viewed from the observer side, the first is the CG object. A method is described in which information of corresponding points is acquired and the information is calculated as pixel values. According to these methods, the number of times of imaging with a virtual camera may be the number of times of obtaining points corresponding to the CG object viewed from the observer side by the number of pixels of the element image group.

  Here, in the methods described in Patent Documents 1 and 2 and Non-Patent Document 2, it is necessary to acquire pixel values from the observer side with a camera or the like as many times as the number of pixels in the element image group. In order to further increase the speed, Patent Document 3 and Non-Patent Document 3 describe a technique of capturing a CG object with a camera by orthographic projection and converting this image into an element image group. According to these methods, the number of times of imaging with a virtual camera may be the same as the number of pixels of the element image.

Japanese Patent No. 4567422 No. 00/059235 JP 2011-234142 A

Spyros S. Athineos et al., “Physical modeling of a microlens array setup for use in computer generated IP,” Proc. Of SPIE-IS & T Electronic Imaging, Vol.5664 pp.472-479,2005 Nakajima, et al., “High-speed image creation method for 3D displays using the principle of Integral Photography”, The Institute of Image Information and Television Engineers Vol. 54, No. 3, pp.420-425 (2000) M. Katayama et al., “A method for converting three-dimensional models into auto-stereoscopic images based on integral photography”, Proc of SPIE-IS & T Vol. 6805 68050Z-1 68050Z-8

  In the IP method, the resolution decreases as the distance from the display screen on which the element image group is displayed. Therefore, in the IP method, a stereoscopic image located in front of or behind the display screen is blurred. At this time, the degree of blur (resolution characteristics) of the stereoscopic image is not known unless actual imaging is performed. However, in the IP system, imaging cannot be performed unless a lens array and a depth control lens are arranged at appropriate positions. As a result, in the IP system, in order to obtain a desired resolution characteristic with a stereoscopic image, imaging is repeatedly performed, which takes a very long time.

Further, according to the method described in Non-Patent Document 1, since light ray information from a subject is obtained by ray tracing with a computer, it is difficult to calculate at high speed.
Further, in the methods described in Patent Document 3 and Non-Patent Document 3, since each pixel of the element image is approximately obtained, the quality of the reproduced image is deteriorated. In addition to this, the methods described in Patent Document 3 and Non-Patent Document 3 cannot accurately record the direction of the light beam reflected from the subject, so that the accuracy of the reproduced light beam also decreases.

  As described above, in the conventional IP system, it takes time to produce an IP stereoscopic video regardless of whether it is a live action or a CG. On the other hand, in the IP system that is an image reproduction type, if camera parameters such as an image sensor and an imaging lens array can be calculated, resolution characteristics of a stereoscopic image are determined, and IP stereoscopic video can be produced by CG. As a result, it is possible to reduce the production work in the IP system and shorten the production time of the IP stereoscopic video.

  Accordingly, it is an object of the present invention to provide a camera parameter calculation device and a program thereof that can shorten the production time of IP stereoscopic video.

  In view of the above-described problems, a camera parameter calculation apparatus according to the present invention is an IP stereoscopic apparatus including a lens array, a diffusion plate, and a stereoscopic imaging device based on camera parameters of a virtual imaging camera disposed in a virtual three-dimensional space. A camera parameter calculation device for calculating camera parameters of a camera, comprising: input means, position calculation means, direction calculation means, angle of view calculation means, lens interval calculation means, lens array focal length calculation means, and output And means.

According to such a configuration, the camera parameter calculation device has, as the camera parameter of the imaging camera, the position and size of the zero parallax plane in the imaging camera, the imaging direction of the imaging camera, the focal length of the imaging camera, and the imaging The size of the image sensor provided in the camera and the shift amount of the image sensor with respect to the optical axis of the image camera are input.
The imaging camera is a virtual camera other than the IP stereoscopic camera, for example, a virtual binocular stereoscopic camera or a monocular camera.

  In addition, the camera parameter calculation device calculates the position of the lens array at the same position as the position of the zero parallax plane by the position calculation means. Then, the camera parameter calculation device calculates the direction of the lens array in the same direction as the imaging direction by the direction calculation means. Furthermore, the camera parameter calculation device calculates the angle of view of the lens array by an angle of view calculation unit using an angle of view calculation formula including the size of the image sensor, the amount of deviation, and the focal length of the image pickup camera.

The camera parameter calculation device calculates the lens interval of the lens array based on the size of the zero parallax surface and the preset number of lenses of the lens array by the lens interval calculation means. Then, the camera parameter calculation device calculates the focal length of the lens array based on the lens interval and the angle of view of the lens array by the lens array focal length calculation means.
As described above, the camera parameter calculation device can calculate the camera parameters of the IP stereoscopic camera from the camera parameters of the imaging camera.

  After that, the camera parameter calculation device outputs the lens array position, the direction of the lens array, the angle of view of the lens array, the lens interval of the lens array, and the focal point of the lens array as camera parameters of the IP stereoscopic camera. Output the distance.

According to the present invention, the following excellent effects can be obtained.
Since the camera parameter calculation apparatus according to the present invention can calculate the camera parameters of the IP stereoscopic camera from the camera parameters of the imaging camera, it is not necessary to determine the camera parameters of the IP stereoscopic camera from the beginning. As a result, the camera parameter calculation apparatus can reduce the production work in the IP system and shorten the production time of the IP stereoscopic video.

It is a schematic block diagram of the conventional binocular stereoscopic image production system. 1 is a schematic configuration diagram of an IP stereoscopic video production system according to a first embodiment. In 1st Embodiment, it is explanatory drawing explaining a zero parallax surface. FIG. 4 is an enlarged view of the image sensor of FIG. 3. It is the schematic of the IP stereoscopic camera of FIG. It is a block diagram which shows the structure of the IP stereoscopic camera control apparatus of FIG. It is explanatory drawing explaining the maximum resolution in 1st Embodiment. In 1st Embodiment, it is explanatory drawing explaining viewing resolution. In 1st Embodiment, it is explanatory drawing explaining the resolution characteristic. In 1st Embodiment, it is explanatory drawing explaining calculation of a diffusion plate position. It is a flowchart which shows the whole operation | movement of the IP stereoscopic video production system of FIG. It is a flowchart which shows operation | movement of the IP stereoscopic camera control apparatus of FIG. It is a schematic block diagram of the IP stereoscopic video production system which concerns on 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

[Outline of binocular stereoscopic video production system]
The IP stereoscopic video production system 1 (FIG. 2) according to the first embodiment of the present invention is an extension of the conventional binocular stereoscopic video production system 2. Therefore, after describing the outline of the binocular stereoscopic video production system 2, the IP stereoscopic video production system 1 will be described.

  As shown in FIG. 1, the binocular stereoscopic video production system 2 generates a binocular stereoscopic video image in which a subject 10 arranged in a virtual three-dimensional space is captured by a binocular stereoscopic camera (imaging camera) 20. Yes, a binocular stereoscopic camera 20, a recording / reproducing apparatus 30, and a binocular stereoscopic television 40 are provided.

  The subject 10 is a subject arranged in a virtual three-dimensional space. That is, the subject 10 is a three-dimensional model (for example, a flower) represented by CG.

  The binocular stereoscopic camera 20 is virtual stereo cameras 20L and 20R that capture an image of the subject 10 in a binocular stereoscopic manner. Then, the binocular stereoscopic camera 20 outputs the captured stereo image to the recording / playback device 30.

  In other words, the binocular stereoscopic camera 20 is a virtual camera of binocular stereoscopic video production software used when producing a binocular stereoscopic video with CG. For example, in the binocular stereoscopic video production software, the camera parameters of the binocular stereoscopic camera 20 are set as production data for the binocular stereoscopic video. Then, the binocular stereoscopic video production software generates a stereo image as if the subject 10 in the virtual three-dimensional space was captured using the binocular stereoscopic camera 20 in which the camera parameters are set.

  The recording / reproducing apparatus 30 records a stereo image captured by the binocular stereoscopic camera 20 and outputs the recorded stereo image to the binocular stereoscopic television 40. For example, the recording / reproducing apparatus 30 includes a general recording apparatus.

The binocular stereoscopic television 40 is a binocular stereoscopic display device that displays a stereo image input from the recording / playback device 30. For example, the twin-lens stereoscopic television 40 may be a lenticular stereoscopic television. In addition, the twin-lens stereoscopic television 40 may be a stereoscopic television that alternately displays two images included in a stereo image in a time division manner.
In this way, the binocular stereoscopic video production system 2 can display the binocular stereoscopic video.

(First embodiment)
[Outline of IP stereoscopic video production system]
An outline of the IP stereoscopic video production system 1 according to the first embodiment of the present invention will be described with reference to FIG. 2 (see FIG. 1 as appropriate).

  As shown in FIG. 2, the IP stereoscopic video production system 1 generates an IP stereoscopic video, and includes a binocular stereoscopic camera 20, a recording / playback device 30, a binocular stereoscopic television 40, a camera parameter acquisition device 100, and the like. , An IP stereoscopic camera 110, an IP stereoscopic camera control device (camera parameter calculation device) 120, a recording / playback device 130, and an IP stereoscopic television 140.

  The camera parameter acquisition device 100 acquires camera parameters of the binocular stereoscopic camera 20. For example, the camera parameter acquisition apparatus 100 acquires camera parameters of the binocular stereoscopic camera 20 set in the binocular stereoscopic video production software via an API (Application Programming Interface). Then, the camera parameter acquisition device 100 outputs the acquired camera parameters of the binocular stereoscopic camera 20 to the IP stereoscopic camera control device 120.

  The camera parameters of the binocular stereoscopic camera 20 include the position of the zero parallax plane in the binocular stereoscopic camera 20 and the size of the zero parallax plane. The camera parameters of the binocular stereoscopic camera 20 include the imaging direction of the binocular stereoscopic camera 20, the focal length of the binocular stereoscopic camera 20, the size of the image sensor included in the binocular stereoscopic camera 20, and the binocular stereoscopic camera. The amount of deviation of the image sensor with respect to the 20 optical axes is included.

<Camera parameters of binocular stereo camera>
The camera parameters of the binocular stereoscopic camera 20 will be described with reference to FIGS.
In FIG. 3, the imaging elements 22L and 22R of the stereo cameras 20L and 20R are shown.
Further, in FIG. 3, an area that is closer to the subject 10 than the zero parallax plane ZDP and that combines the imaging areas of the stereo cameras 20L and 20R is illustrated by hatching.
FIG. 4 shows an enlarged view of the image sensor 22R of FIG.

The zero parallax plane ZDP is a plane where binocular parallax is zero. That is, the zero parallax plane ZDP is a plane where stereo images captured by the stereo cameras 20L and 20R overlap as shown in FIG.
The upper end position PU of the zero parallax plane ZDP is represented by the intersection of the upper limit line UR of the imaging region in the stereo camera 20R and the upper limit line UL of the imaging region in the stereo camera 20L. Further, the lower end position PD of the zero parallax plane ZDP is represented by the intersection of the lower limit line DR of the imaging area in the stereo camera 20R and the lower limit line DL of the imaging area in the stereo camera 20L. In this case, the position of the zero parallax plane ZDP is specified by the upper end position PU and the lower end position PD, and the size is represented by the distance from the upper end position PU to the lower end position PD.

  The imaging direction of the binocular stereoscopic camera 20 is the direction in which the binocular stereoscopic camera 20 images the subject 10, and represents the optical axis direction of the stereo cameras 20L and 20R. In the case of the parallel method, as shown in FIG. 3, the optical axes AL and AR of the stereo cameras 20L and 20R are parallel.

  In the crossing method, the imaging direction of the binocular stereoscopic camera 20 is a direction perpendicular to the line connecting the stereo cameras 20L and 20R (not shown in FIG. 3). For this reason, the intersection method is preferably not applied to imaging with the virtual binocular stereoscopic camera 20 because distortion occurs in the depth direction of the stereoscopic image.

The focal length of the binocular stereoscopic camera 20 is a focal length when the binocular stereoscopic camera 20 images the subject 10, and is a focal length of camera lenses (not shown) of the stereo cameras 20L and 20R.
The size w of the image sensor 22R represents the size of the image sensor 22 as shown in FIG.
The shift amount Δd of the image sensor 22R represents the shift amount between the optical axis AR of the stereo camera 20R and the center position CR of the image sensor 22R.
Note that the image sensors 22L and 22R have the same size and shift amount.

Returning to FIG. 2, the outline of the IP stereoscopic video production system 1 will be described.
The IP stereoscopic camera 110 is a virtual camera that captures an image of the subject 10 by the IP stereoscopic method in accordance with a command from the IP stereoscopic camera control device 120. Then, the IP stereoscopic camera 110 outputs the captured IP stereoscopic video to the recording / playback device 130.

  In other words, the IP stereoscopic camera 110 is a virtual camera of IP stereoscopic video production software used when generating an IP stereoscopic video with CG. As shown in FIG. 5, the IP stereoscopic camera 110 includes an imaging lens array (lens array) 112, a diffusion plate (light condensing member) 114, and a high-definition camera (stereoscopic imaging device) 116.

The imaging lens array 112 is a lens in which microlenses (element lenses) 112a are two-dimensionally arranged.
The diffusion plate 114 diffuses the light emitted from the imaging lens array 112 and focuses it on the high-definition camera 116.
The high-definition camera 116 images the subject 10 via the imaging lens array 112 and the diffusion plate 114 to generate an element image group (IP stereoscopic video).

  Note that since the IP stereoscopic camera 110 is a virtual camera, an aperture (not shown) can be provided on the subject side of the imaging lens array 112 (the side opposite to the diffusion plate 114). This diaphragm adjusts the amount of light passing through each element lens 112a of the imaging lens array 112.

  The IP stereoscopic camera control device 120 calculates the camera parameters of the IP stereoscopic camera 110 from the camera parameters of the binocular stereoscopic camera 20 acquired by the camera parameter acquisition device 100. Then, the IP stereoscopic camera control device 120 controls the IP stereoscopic camera 110 based on the calculated camera parameters of the IP stereoscopic camera 110. Specifically, the IP stereoscopic camera control device 120 instructs the IP stereoscopic camera 110 to image the subject 10 according to the calculated camera parameters, and records the recording of the IP stereoscopic video in synchronization with the imaging by the IP stereoscopic camera 110. Commands the playback device 130 (synchronization signal).

Here, the camera parameters of the IP stereoscopic camera 110 include the position of the imaging lens array 112, the direction of the imaging lens array 112, the angle of view of the imaging lens array 112, and the lens interval of the imaging lens array 112. The focal length of the imaging lens array 112, the position of the diffusion plate 114, and the focal length of the high-definition camera 116 are included.
Details of the IP stereoscopic camera control device 120 will be described later.

  The recording / reproducing apparatus 130 records the IP stereoscopic video input from the IP stereoscopic camera 110 in accordance with a command from the IP stereoscopic camera control apparatus 120. Then, the recording / reproducing apparatus 130 outputs the recorded IP stereoscopic video to the IP stereoscopic television 140. For example, the recording / playback device 130 may be a general recording device.

  The IP stereoscopic television 140 displays the IP stereoscopic video input from the recording / playback device 130. For example, the IP stereoscopic television 140 is a general IP stereoscopic display device that includes a display lens array 142 and a display element 144 (see FIG. 8).

[Configuration of IP Stereo Camera Control Device]
The configuration of the IP stereoscopic camera control device 120 will be described with reference to FIG. 6 (see FIG. 2 as appropriate).
As shown in FIG. 6, the IP stereoscopic camera control device 120 includes an input unit 210, an IP stereoscopic camera parameter calculation unit 220, an output unit 230, and an IP stereoscopic camera control unit 240.
Note that the IP stereoscopic television control unit 250 will be described in Modification 1 described later.

  The input unit 210 receives camera parameters of the binocular stereoscopic camera 20 from the camera parameter acquisition device 100. Then, the input unit 210 outputs the input camera parameters of the binocular stereoscopic camera 20 to the IP stereoscopic camera parameter calculation unit 220.

  The IP stereoscopic camera parameter calculation means 220 calculates camera parameters of the IP stereoscopic camera 110 from the camera parameters of the binocular stereoscopic camera 20. Then, the IP stereoscopic camera parameter calculation unit 220 outputs the calculated camera parameters of the IP stereoscopic camera 110 to the output unit 230.

  As shown in FIG. 6, the IP stereoscopic camera parameter calculation means 220 includes a lens array parameter calculation means 221, an aperture amount calculation means 227, a diffuser plate position calculation means (light condensing member position calculation means) 228, and a high-definition camera focus. Distance calculation means (stereoscopic imaging apparatus focal length calculation means) 229.

  The IP stereoscopic camera parameter calculation means 220 receives an initial setting value for calculating camera parameters of the IP stereoscopic camera 110. This initial set value includes the number of lenses of the imaging lens array 112, the depth of field of the imaging lens array 112, the imaging distance, and the pixel interval of the high-definition camera 116.

The number of lenses of the imaging lens array 112 represents the number of element lenses 112 a included in the imaging lens array 112. Usually, the number of lenses of the imaging lens array 112 is set to the same number as the element lenses 142a of the display lens array 142 provided in the IP stereoscopic television 140 (see FIG. 8).
The depth of field of the imaging lens array 112 represents the depth of field of the element lens 112 a included in the imaging lens array 112.

The pixel interval of the high-definition camera 116 represents the pixel interval of an image sensor (not shown) included in the high-definition camera 116. Usually, the pixel interval of the high-definition camera 116 is set to be equal to or less than the pixel interval of the display element 144 included in the IP stereoscopic television 140 (see FIG. 8).
The imaging distance represents the distance from the imaging lens array 112 to the subject 10.

  The lens array parameter calculation unit 221 calculates camera parameters related to the imaging lens array 112, and includes a position calculation unit 222, a direction calculation unit 223, an angle of view calculation unit 224, a lens interval calculation unit 225, and a lens. Array focal length calculation means 226.

The position calculation unit 222 calculates the position of the imaging lens array 112 at the same position as the position of the zero parallax plane ZDP input from the input unit 210.
The direction calculation unit 223 calculates the direction of the imaging lens array 112 in the same direction as the imaging direction input from the input unit 210. That is, the direction calculation unit 223 calculates the optical axis of the element lens 112 a included in the imaging lens array 112 in the same direction as the optical axis of the binocular stereoscopic camera 20.

  In this way, by arranging the position and direction of the imaging lens array 112 in the same manner as the position and the imaging direction of the zero parallax surface of the binocular stereoscopic camera 20, the IP stereoscopic video imaged by the IP stereoscopic camera 110 is 2 The image can be approximated to a binocular stereoscopic image captured by the eye stereoscopic camera 20.

  The angle-of-view calculation means 224 calculates the angle of view of the imaging lens array 112 using a preset angle-of-view calculation formula (1) below. Expression (1) includes the angle of view α of the imaging lens array 112, the shift amount Δd, the size w of the imaging device, and the focal length f of the binocular stereoscopic camera 20. An angle of view α of the imaging lens array 112 represents an angle of view at which the entire hatched area in FIG. 3 can be photographed.

The lens interval calculation unit 225 calculates the lens interval of the imaging lens array 112 based on the size of the zero parallax plane ZDP input from the input unit 210 and the number of lenses of the initial setting value.
Here, the imaging lens array 112 is configured by arranging the element lenses 112a without a gap. For this reason, the lens interval of the imaging lens array 112 represents the diameter of the element lens 112 a included in the imaging lens array 112. Further, the imaging lens array 112 has the same size as the zero parallax plane ZDP.
Therefore, the lens interval calculation unit 225 can calculate the lens interval of the imaging lens array 112 by dividing the size of the zero parallax plane ZDP by the number of lenses of the imaging lens array 112.

The lens array focal length calculation unit 226 calculates the focal length of the imaging lens array 112 based on the lens interval calculated by the lens interval calculation unit 225 and the field angle calculated by the field angle calculation unit 224. . For example, the lens array focal length calculation unit 226 can obtain the focal length from the lens interval and the angle of view of the imaging lens array 112 using a general lens formula.
The focal length of the imaging lens array 112 represents the focal length of the element lens 112a included in the imaging lens array 112.

  The aperture amount calculating unit 227 performs imaging based on the initial depth of field and imaging distance, the focal length calculated by the lens array focal length calculating unit 226, and the lens interval calculated by the lens interval calculating unit 225. The aperture amount of the lens array 112 is calculated.

For example, the aperture amount calculation unit 227 can calculate the aperture amount of the imaging lens array 112 using the depth-of-field calculation formula described in Reference Document 1 below.
Reference 1: [online], Formula for Depth of Field, [February 11, 2014 Search], Internet <http://www.asahi-net.or.jp/~kv5y-aon/DOF/ dof.html>

<Resolution characteristics in IP 3D system>
With reference to FIG. 7 to FIG. 9, the resolution characteristics in the IP stereoscopic method will be described because it relates to the calculation of camera parameters.
In FIG. 7, the horizontal axis Z represents the depth value with reference to the display lens array 142, and “0” is the position of the display lens array 142.

As shown in FIG. 7, the maximum resolution (maximum spatial frequency) is calculated based on the lens interval P of the display lens array 142 and the viewing distance L between the display lens array 142 and the viewpoint 300.
The maximum resolution is a resolution determined depending on the interval P of the display lens array 142, and is the resolution at the position of the display lens array 142 (depth value z in FIG. 8).

As shown in FIG. 7, the angle φ represents an angle with respect to the interval P of the display lens array 142 as in the following formula (2). In other words, the angle φ is an angle formed by a line connecting the upper end of the display lens array 142 and the viewpoint 300 and a line connecting the lower end of the display lens array 142 and the viewpoint 300. Therefore, the maximum resolution β _n can be approximated by the following equation (3).

As shown in FIG. 8, the viewing resolution (viewing spatial frequency) is based on the distance g between the display lens array 142 and the display element 144 and the pixel interval p of the element image 420 constituting the element image group 410. Can be calculated.
The viewing resolution is the resolution when the stereoscopic image 400 is viewed from the viewpoint 300.

In FIG. 8, the distance g is negative because it is on the far side of the display lens array 142.
Further, in FIG. 8, the pixels in the stereoscopic image 400 and the pixels in the element image 420 are alternately shown in black and white for easy viewing of the drawing.

  The pixel interval t represents the pixel interval in the stereoscopic image 400 as in the following formula (4). Here, the stereoscopic image 400 is formed at a position indicated by the depth value z.

  The angle θ represents an angle with respect to the pixel interval t as in the following formula (5). In other words, the angle θ is an approximate value obtained by dividing the pixel interval t by the value obtained by subtracting the depth value z from the viewing distance L. Accordingly, the viewing resolution β can be expressed by the following equation (6).

As shown in FIG. 9, the resolution characteristic γ is the smaller of the maximum resolution β _n or the viewing resolution β as shown in the following equation (7). In Expression (7), “min” is a function that selects the smallest argument among a plurality of arguments.

In the example of FIG. 9, the resolution characteristic γ represents the maximum resolution β _n in the vicinity of the display lens array 142, and represents the viewing resolution β when separated from the display lens array 142 to some extent. The resolution characteristic γ represents that the resolution decreases as the distance from the display lens array 142 increases.
As described above, the resolution characteristics and the viewing area in the IP stereoscopic method are determined by the lens interval and focal length of the display lens array 142 and the pixel interval of the display element 144. That is, there is a trade-off relationship between the resolution characteristics and the viewing zone. This viewing area is an area in which IP stereoscopic video can be correctly displayed.

  Here, the resolution characteristics in the IP stereoscopic method have been described on the display side (IP stereoscopic television 140 side), but the same applies to the imaging side (IP stereoscopic camera control device 120). Therefore, the resolution characteristic in the IP stereoscopic method is limited by the lens interval of the imaging lens array 112. As described above, if the viewing area in the IP stereoscopic system is equal to or larger than the angle of view of the binocular stereoscopic camera 20, the subject 10 can be imaged in the imaging area of the binocular stereoscopic camera 20 or beyond.

Since the resolution characteristics are described in Reference Document 2 below, further explanation is omitted.
Reference 2: H. Hoshino et al., “Analysis of Resolution Limitation of Integral Photography”, J. Opt. Soc. Am. A, Vol. 15, No. 8, pp. 2059-2065, 1998

Returning to FIG. 6, the description of the configuration of the IP stereoscopic camera control device 120 will be continued.
The diffusing plate position calculating unit 228 calculates the position of the diffusing plate 114 with respect to the imaging lens array 112 based on the focal length calculated by the lens array focal length calculating unit 226 (see FIG. 5).
The high-definition camera focal length calculation unit 229 calculates the focal length of the high-definition camera 116 so as to focus on the position of the diffusion plate 114 obtained by the diffusion plate position calculation unit 228.

<Diffusion plate position and high-definition camera focal length>
The calculation of the position of the diffusion plate 114 and the focal length of the high-definition camera 116 will be described with reference to FIG. 10 (see FIG. 6 as appropriate).
In FIG. 10, the imaging lens array 112 and the diffusion plate 114 are separated by a distance L1, and the diffusion plate 114 and the high-definition camera 116 are separated by a distance L2. In FIG. 10, the imaging lens array 112 is located on the zero parallax plane ZDP, and the imaging lens array 112 and the end face of the diffusion plate 114 are parallel to each other.

  Here, when the imaging lens array 112 focuses at infinity, the end surfaces of the imaging lens array 112 and the diffusion plate 114 are parallel, and the distance L1 and the focal length of the imaging lens array 112 are equal. .

  Therefore, by setting the distance L1 to a value other than the focal length of the imaging lens array 112, the imaging lens array 112 is focused on an arbitrary position. Here, when the distance L1 is shorter than the focal length of the imaging lens array 112, the focus is on the high-definition camera 116 side (back side) rather than the diffusion plate 114. Further, when the distance L1 is longer than the focal length of the imaging lens array 112, the object 10 side (near side) is focused with respect to the diffusion plate 114.

  Therefore, the diffuser plate position calculating means 228 calculates the position of the diffuser plate 114 so as to focus on a desired position by increasing or decreasing the distance L1 with reference to the focal length of the imaging lens array 112. Further, the high-definition camera focal length calculation unit 229 calculates the focal length of the high-definition camera 116 so that the high-definition camera 116 is focused on the diffusion plate 114.

As a result, the element lens 112a of the imaging lens array 112 focuses on a finite position rather than infinity, and causes the diffusion plate 122 to display the element image group. That is, the IP stereoscopic camera 110 can capture an image at the maximum resolution at the in-focus position of the binocular stereoscopic camera 20. For example, in the production of a two-dimensional video, the subject 10 is emphasized by focusing on the subject 10 and intentionally blurring the background. Similarly, the diffusing plate position calculating unit 228 can improve the resolution of the stereoscopic video by focusing on the subject 10 as compared with the case of focusing on infinity.
Note that when the distance L1 is other than the focal length of the imaging lens array 112, the resolution is drastically reduced as compared to when focusing at infinity.

Returning to FIG. 6, the description of the configuration of the IP stereoscopic camera control device 120 will be continued.
The output unit 230 outputs the camera parameters of the IP stereoscopic camera 110 input from the IP stereoscopic camera parameter calculation unit 220 to the IP stereoscopic camera control unit 240.

The IP stereoscopic camera control unit 240 controls the IP stereoscopic camera 110 based on the camera parameters of the IP stereoscopic camera 110 input from the output unit 230.
As described above, the IP stereoscopic camera 110 is a virtual camera of IP stereoscopic video production software. Therefore, the IP stereoscopic camera control unit 240 sets the camera parameters of the IP stereoscopic camera 110 in the IP stereoscopic video production software via the API, and the IP stereoscopic video captured by the IP stereoscopic camera 110 is set as the IP stereoscopic video production software. Control to be generated.
Thereafter, the IP stereoscopic camera control means 240 commands the recording / playback apparatus 130 to record the captured IP stereoscopic video.

[Overall operation of IP stereoscopic video production system]
With reference to FIG. 11, the overall operation of the IP stereoscopic video production system 1 in FIG. 2 will be described (see FIG. 2 as appropriate).
As shown in FIG. 11, the camera parameter acquisition device 100 acquires camera parameters of the binocular stereoscopic camera 20 (step S1).

The IP stereoscopic camera control device 120 calculates camera parameters of the IP stereoscopic camera 110 from the camera parameters of the binocular stereoscopic camera 20 acquired in step S1. Then, the IP stereoscopic camera control device 120 controls the IP stereoscopic camera 110 based on the calculated camera parameters of the IP stereoscopic camera 110 (step S2: IP stereoscopic camera control process).
Details of step S2 will be described later.

The IP stereoscopic camera 110 images the subject 10. Then, the recording / reproducing device 130 records the IP stereoscopic video captured by the IP stereoscopic camera 110 (step S3).
The IP stereoscopic television 140 displays the IP stereoscopic video recorded in step S3 (step S4).

[Operation of IP Stereo Camera Control Device]
The IP stereoscopic camera control process of FIG. 11 will be described with reference to FIG. 12 (see FIG. 6 as appropriate).
The input unit 210 receives camera parameters of the binocular stereoscopic camera 20 from the camera parameter acquisition device 100 (step S20).

The position calculation unit 222 calculates the position of the imaging lens array 112 at the same position as the position of the zero parallax surface (step S21).
The direction calculation means 223 calculates the direction of the imaging lens array 112 in the same direction as the imaging direction (step S22).

The angle-of-view calculation means 224 calculates the angle of view of the imaging lens array 112 using the angle-of-view calculation formula of the formula (1) (step S23).
The lens interval calculation unit 225 calculates the lens interval of the imaging lens array 112 based on the size of the zero parallax surface and the number of lenses of the initial setting value (step S24).

The lens array focal length calculation unit 226 calculates the focal length of the imaging lens array 112 based on the lens interval calculated in step S24 and the angle of view calculated in step S23 (step S25).
The aperture amount calculation means 227 calculates the aperture amount of the imaging lens array 112 based on the depth of field and the imaging distance, the focal length calculated in step S25, and the lens interval calculated in step S24 (step S26).

The diffusing plate position calculating means 228 calculates the position of the diffusing plate 114 with reference to the focal length calculated in step S25 (step S27).
The high-definition camera focal length calculation unit 229 calculates the focal length of the high-definition camera 116 so as to focus on the position of the diffusion plate 114 obtained in step S27 (step S28).

  The IP stereoscopic camera control unit 240 controls the IP stereoscopic camera 110 based on the camera parameters of the IP stereoscopic camera 110 obtained in steps S21 to S28. For example, the IP stereoscopic camera control unit 240 sets the camera parameters of the IP stereoscopic camera 110 in the IP stereoscopic video production software (step S29).

[Action / Effect]
Since the IP stereoscopic camera control device 120 can calculate the camera parameters of the IP stereoscopic camera 110 from the camera parameters of the binocular stereoscopic camera 20, it is not necessary to determine the camera parameters of the IP stereoscopic camera 110 from the beginning. As a result, the IP stereoscopic camera control device 120 can reduce the production work of the IP system and shorten the production time of the IP stereoscopic video.

  Furthermore, since the IP stereoscopic camera control device 120 can use production data (camera parameters of the binocular stereoscopic camera 20) when the binocular stereoscopic video is produced, the depth expression of the binocular stereoscopic video is reflected in the IP stereoscopic video. Can produce high-quality IP stereoscopic video intended by the creator.

(Second Embodiment)
[Outline of IP stereoscopic video production system]
With reference to FIG. 13, an IP stereoscopic video production system 1B according to the second embodiment of the present invention will be described while referring to differences from the first embodiment.
The IP stereoscopic video production system 1B is different from the first embodiment in that the camera parameters of the IP stereoscopic camera 110 are calculated using the camera parameters of the monocular camera (imaging camera) 150 instead of the binocular stereoscopic camera 20.

As shown in FIG. 13, the IP stereoscopic video production system 1B includes a recording / playback device 30, a camera parameter acquisition device 100B, an IP stereoscopic camera 110, an IP stereoscopic camera control device (camera parameter calculation device) 120B, and a recording / playback device. 130, an IP stereoscopic television 140, a monocular camera 150, and a two-dimensional television 160.
Hereinafter, after describing the monocular camera 150 and the two-dimensional television 160 first, the camera parameter acquisition device 100B and the IP stereoscopic camera control device 120B will be described.

The monocular camera 150 is a virtual camera that captures an image of the subject 10 arranged in a virtual three-dimensional space. Then, the monocular camera 150 outputs the captured CG image to the recording / playback apparatus 30.
In other words, the monocular camera 150 is a virtual camera of CG image production software used in CG production. For example, the CG image production software sets the camera parameters of the monocular camera 150 as CG image production data. Then, the CG image production software generates a CG image as if the subject 10 was captured using the monocular camera 150 in which the camera parameters are set.

  The two-dimensional television 160 displays the CG image input from the recording / reproducing apparatus 30 in two dimensions. For example, the two-dimensional television 160 is a general display device using a flat panel display such as a liquid crystal panel display.

  The camera parameter acquisition device 100B acquires camera parameters of the monocular camera 150. That is, the camera parameter acquisition device 100B acquires camera parameters set in the CG image production software. Then, the camera parameter acquisition apparatus 100B outputs the acquired camera parameters of the monocular camera 150 to the IP stereoscopic camera control apparatus 120B.

  The camera parameters of the monocular camera 150 include the imaging position of the monocular camera 150, the imaging direction of the monocular camera 150, the focal length of the monocular camera 150, the size of the image sensor included in the monocular camera 150, and the light of the monocular camera 150. The amount of displacement of the image sensor with respect to the axis is included.

  Here, among the camera parameters of the monocular camera 150, the imaging direction, the focal length, the size of the imaging device, and the shift amount can be handled in the same manner as the camera parameters of the binocular stereoscopic camera 20.

  In addition, since the monocular camera 150 does not handle the zero parallax plane, the camera parameters of the monocular camera 150 do not include the position and size of the zero parallax plane. Therefore, the camera parameter acquisition apparatus 100B supplements the position of the zero parallax plane using the imaging position and focal length of the monocular camera 150.

  From the imaging position and focal length of the monocular camera 150, the focal position z2 on which the monocular camera 150 is focused is known (hereinafter, “focal position information”). In general two-dimensional images, the focal position of the monocular camera 150 is often the position that the viewer wants to watch. As described above, in the IP stereoscopic system, the maximum resolution is obtained in the vicinity of the imaging lens array 112, and the resolution decreases as the distance from the imaging lens array 112 increases. Further, the imaging lens array 112 is disposed on the zero parallax plane. Thus, the camera parameter acquisition device 100B handles the focal position of the monocular camera 21 as the position of the zero parallax plane.

Further, the size w2 of the zero parallax plane can be obtained from the size w of the image sensor, the distance f2 between the optical principal point of the monocular camera 150 and the image sensor, and the focal position z2 as in the following equation (8). it can. Therefore, the camera parameter acquisition device 100B can acquire the size w2 of the zero parallax plane using Expression (9) obtained by modifying Expression (8).
Note that the optical principal point of the monocular camera 150 is the principal point position of the lens provided in the monocular camera 150.

The IP stereoscopic camera control device 120B calculates camera parameters of the IP stereoscopic camera 110 from the camera parameters of the monocular camera 150 acquired by the camera parameter acquisition device 100B.
Since the IP stereoscopic camera control device 120B can calculate the camera parameters of the IP stereoscopic camera 110 as in the first embodiment, further description thereof is omitted.

[Action / Effect]
Since the IP stereoscopic camera control device 120B can calculate the camera parameters of the IP stereoscopic camera 110 from the camera parameters of the monocular camera 150, the IP stereoscopic video production time can be shortened as in the first embodiment.

  Furthermore, the IP stereoscopic camera control device 120B can produce various IP stereoscopic images using production data (camera parameters of the monocular camera 150) when producing innumerable CG images, thus improving versatility. Can be made.

(Modification 1)
The IP stereoscopic camera control device according to the present invention is not limited to the above-described embodiments, and modifications can be made without departing from the spirit thereof.

Returning to FIG. 6, the IP stereoscopic camera control device 120C according to the first modification of the present invention will be described (see FIGS. 2 and 8 as appropriate).
The IP stereoscopic camera control device 120 </ b> C is a device in which the IP stereoscopic camera control device 120 further includes IP stereoscopic television control means 250.

  Originally, the resolution characteristic in the IP system is determined by the product of the resolution characteristic on the imaging side and the resolution characteristic on the display side. However, in the first embodiment described above, the resolution on the display side is assumed to be constant, and only the resolution characteristics on the imaging side are considered. Therefore, in the first modification, in addition to the resolution characteristic on the imaging side, the resolution characteristic on the display side is also taken into consideration.

  Specifically, the IP stereoscopic television control unit 250 controls the distance between the display lens array 142 of the IP stereoscopic television 140 and the display element 144 according to the position of the subject 10. Here, the IP stereoscopic television control means 250 obtains this distance by the same method as the diffusion plate position calculation means 228 and commands the IP stereoscopic television 140 to drive it.

  Here, the IP stereoscopic television 140 includes a distance adjustment mechanism that adjusts the distance between the display lens array 142 and the display element 144. That is, the IP stereoscopic television 140 includes a distance adjustment mechanism that adjusts the distance g in FIG. For example, the IP stereoscopic television 140 includes a rail extending in the normal direction of the display surface of the display element 144 and a pedestal on which the display lens array 142 is mounted as a distance adjustment mechanism. The pedestal includes a motor for driving on the rail. Then, in the IP 3D television 140, a pedestal mounted on the rail is driven according to a command from the IP 3D television control means 250.

As a result, the IP stereoscopic camera control device 120C can also adjust the resolution characteristics on the display side, so that the resolution of the IP stereoscopic video can be further improved.
Note that the IP stereoscopic camera control device 120C may include a subject information storage unit (not shown) that receives the position of the subject 10 as an initial setting value and stores the input position of the subject 10.

(Modification 2)
In each of the above-described embodiments, the IP stereoscopic camera 110 is a virtual camera, but the IP stereoscopic camera 110 may be a live-action camera. That is, the IP stereoscopic camera control device 120 may control the existing IP stereoscopic camera 110 according to the calculated camera parameters of the IP stereoscopic camera 110.

  For example, the positions of the imaging lens array 112 and the diffusion plate 114 can be controlled when the IP stereoscopic camera 110 includes the same distance adjustment mechanism as that of the first modification. The direction of the imaging lens array 112 can be controlled by mounting the IP stereoscopic camera 110 on an autonomously moving pan head. Further, the focal length of the high definition camera 116 can be controlled by driving the focus ring of the high definition camera 116.

Further, the angle of view and the focal length of the imaging lens array 112 can be controlled by making the imaging lens array 112 a camera array composed of a micro camera. At this time, the lens interval of the imaging lens array 112 can be controlled by performing lens number conversion processing on the element image group captured by the camera array.
As described above, the IP stereoscopic camera control device 120C can be controlled even when the IP stereoscopic camera 110 is a live-action camera.

(Other variations)
Although the IP stereoscopic camera control devices 120, 120B, and 120C have been described as controlling the IP stereoscopic camera 110, the camera parameters of the IP stereoscopic camera 110 may only be calculated. That is, the IP stereoscopic camera control devices 120, 120 </ b> B, and 120 </ b> C may not include the IP stereoscopic camera control unit 240.
Even in this case, if the production work is performed according to the camera parameters calculated by the IP stereoscopic camera control devices 120, 120B, and 120C, the production time of the IP stereoscopic video can be shortened.

  Although the IP stereoscopic camera control devices 120, 120B, and 120C have been described as including the aperture amount calculation unit 227, the aperture amount calculation unit 227 may not be included. That is, the IP stereoscopic camera control devices 120, 120 </ b> B, and 120 </ b> C do not have to calculate the aperture amount as the camera parameter of the IP stereoscopic camera 110.

  The IP stereoscopic camera control devices 120, 120B, and 120C have been described as independent hardware, but the present invention is not limited to this. For example, the IP stereoscopic camera control devices 120, 120B, and 120C can also be realized by an IP stereoscopic camera control program that causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to operate cooperatively as the above-described units. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

  Although the IP stereoscopic camera 110 includes the diffusion plate 114, the diffusion plate 114 may not be included. Furthermore, the IP stereoscopic camera 110 may include a condensing lens (not shown) instead of the diffusion plate 114 as a condensing member.

1,1B IP stereoscopic video production system 2 Binocular stereoscopic video production system 10 Subject 20 Binocular stereoscopic camera (imaging camera)
20L, 20R Stereo camera 22L, 22R Image sensor 30 Recording / playback device 40 Binocular stereoscopic television 100, 100B Camera parameter acquisition device 110 IP stereoscopic camera 112 Imaging lens array (lens array)
114 Diffuser (Condensing member)
116 High-definition camera (stereoscopic imaging device)
120, 120B, 120C IP stereoscopic camera control device (camera parameter calculation device)
130 Recording / Playback Device 140 IP Stereo Television 150 Monocular Camera (Imaging Camera)
160 Two-dimensional television 210 Input means 220 IP stereoscopic camera parameter calculation means 221 Lens array parameter calculation means 222 Position calculation means 223 Direction calculation means 224 Angle of view calculation means 225 Lens interval calculation means 226 Lens array focal length calculation means 227 Aperture amount calculation means 228 Diffuser position calculation means (light collecting member position calculation means)
229 High-definition camera focal length calculation means (stereoscopic imaging apparatus focal length calculation means)
230 Output means 240 IP stereoscopic camera control means 250 IP stereoscopic television control means

Claims (7)

A camera parameter calculation device that calculates camera parameters of an IP stereoscopic camera including a lens array and a stereoscopic imaging device from camera parameters of a virtual imaging camera arranged in a virtual three-dimensional space,
As camera parameters of the imaging camera, the position and size of the zero parallax plane in the imaging camera, the imaging direction of the imaging camera, the focal length of the imaging camera, the size of the imaging device included in the imaging camera, and the imaging An input means for inputting the shift amount of the image sensor with respect to the optical axis of the camera;
Position calculating means for calculating the position of the lens array at the same position as the position of the zero parallax plane;
Direction calculating means for calculating the direction of the lens array in the same direction as the imaging direction;
An angle-of-view calculating means for calculating an angle of view of the lens array by an angle-of-view calculating formula including the size of the image sensor, the shift amount, and the focal length of the imaging camera;
A lens interval calculating means for calculating a lens interval of the lens array based on the size of the zero parallax surface and a preset number of lenses of the lens array;
Lens array focal length calculation means for calculating the focal length of the lens array based on the lens interval and the angle of view of the lens array;
An apparatus for calculating a camera parameter, comprising: output means for outputting the position, direction, angle of view, lens interval, and focal length of the lens array as camera parameters of the IP stereoscopic camera. Condensing member position for calculating the position of the condensing member with reference to the focal length of the lens array for an IP stereoscopic camera provided with a condensing member that condenses the light emitted from the lens array on the stereoscopic imaging device A calculation means;
Stereoscopic imaging device focal length calculating means for calculating a focal length of the stereoscopic imaging device so that the stereoscopic imaging device is focused on the position of the light collecting member;
The camera parameter calculation apparatus according to claim 1, wherein the output unit further outputs a position of the light collecting member and a focal length of the stereoscopic imaging device as camera parameters of the IP stereoscopic camera. The output means outputs camera parameters of a virtual IP stereoscopic camera arranged in a virtual three-dimensional space as camera parameters of the IP stereoscopic camera,
The camera parameter calculation apparatus according to claim 1, further comprising: an IP stereoscopic camera control unit that controls the virtual IP stereoscopic camera based on a camera parameter of the virtual IP stereoscopic camera. The input means further inputs the depth of field of the imaging camera as a camera parameter of the imaging camera,
Aperture amount calculation means for calculating the aperture amount of the lens array based on the depth of field, the focal length of the lens array, the lens interval, and a preset distance from the lens array to the subject. Further comprising
4. The camera parameter calculation apparatus according to claim 1, wherein the output unit further outputs an aperture amount of the lens array as a camera parameter of the IP stereoscopic camera. 5. The output means outputs camera parameters of a live-action IP stereoscopic camera arranged in real space as camera parameters of the IP stereoscopic camera,
The camera parameter calculation apparatus according to claim 1, further comprising: an IP stereoscopic camera control unit that controls the actual shooting IP stereoscopic camera based on a camera parameter of the actual shooting IP stereoscopic camera.   The camera parameter calculation according to any one of claims 1 to 5, wherein the input unit receives camera parameters of the imaging camera which is either a binocular stereoscopic camera or a monocular camera. apparatus.   The camera parameter calculation program for functioning a computer as a camera parameter calculation apparatus as described in any one of Claims 1-6.

Images