JP2012227700A - Information processor and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor which, when composing an image that appears to be an image taken with a virtual camera composed of an optical system having an aperture of a fixed size using images taken by plural cameras, optimally controls the direction and the angle of field of each camera, thereby improving an image quality more than that when the control is not performed.SOLUTION: Respective cameras are controlled such that light beams pass through an imaging device conjugate surface 602 and an entrance pupil 606, thereby enabling efficient acquisition of light beam information. Actual multi-eye cameras are arrayed and orientations of the cameras can be changed in a three-dimensional manner. Then, control information is obtained by: determining a common portion of the shapes that can be obtained through perspective projection of the entrance pupil 606 and the imaging device conjugate surface 602 on a camera control information is desired to be obtained; calculating as an evaluation value a minimum angle of field capable of covering the common portion; and optimizing the rotation amount of the camera so as to reduce the calculated evaluation value.

Description

  The present invention relates to an information processing apparatus and a program for synthesizing a virtual viewpoint image using images captured by a plurality of cameras.

  Conventionally, the range of image resolution, noise, blur, angle of view, viewpoint, etc. that can be acquired by shooting with a digital camera is limited by the characteristics and arrangement of the digital camera's optical system and image sensor. It was. On the other hand, using multiple images captured by multiple cameras, it combines images as if they were captured by a single virtual camera, and acquires images that exceed the aforementioned limitations of individual cameras. There are technologies that enable this (see, for example, Patent Document 1 and Non-Patent Document 1).

  In the method described in Patent Document 1, images are captured by a plurality of cameras, and an image at a virtual camera viewpoint is synthesized based on the correspondence between acquired images. At this time, the virtual camera field of view is divided based on the number of cameras, and the angle of view and orientation are controlled so that the two cameras always cover each divided field of view. It is possible to widen the angle of view. Further, according to the method described in Non-Patent Document 1, images are captured by a virtual camera including an optical system having a certain size (blur) by combining images captured by a plurality of cameras close to pan focus. It is possible to acquire a blurred image as if it were.

JP-A-2005-80015

A. Isaksen et al. "Dynamically Reparameterized Light Fields", ACM SIGGRAPH, pp. 297-306 (2000) Z. Zhang “A flexible new technique for camera calibration” IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000 Sato Jun (1999). Computer Vision-Visual Geometry-Corona p80-96, 149-156

  However, with the method described in Patent Document 1, it is possible to widen the angle of view among images that exceed the above-mentioned limitations, but only an image that is focused on the entire screen can be obtained, and a finite size pupil is obtained. The blur cannot be reproduced as if it were taken with a camera. Further, since the images of the virtual camera are synthesized based on the relationship between the corresponding points between the parallax images, there is a possibility that the synthesis may fail when shooting a subject blur, a transparent subject, or a subject with low contrast.

  On the other hand, the method described in Non-Patent Document 1 can reproduce an image as if it was picked up by a virtual camera composed of an optical system having an aperture of a certain size, and subject blur or transparent subject or low contrast. There is no composition failure when shooting a subject. However, as the size of the aperture of the virtual camera is changed, there is a problem in that image quality characteristics such as resolution, noise, and naturalness of blur are deteriorated because the range captured by each camera becomes excessive and insufficient.

  The object of the present invention is to use images from a plurality of cameras as in Non-Patent Document 1, and to synthesize images as if they were captured by a virtual camera consisting of an optical system having an aperture of a certain size, An object of the present invention is to provide a means for obtaining higher image quality by optimally controlling the camera direction and angle of view.

  In order to solve such a problem, the information processing apparatus of the present invention is an information processing apparatus that processes an image of an imaging apparatus having virtual imaging system characteristics defined by an entrance pupil shape and an imaging element conjugate plane shape. And a plurality of control means for controlling a range captured by each of the plurality of imaging means, wherein the control means defines a range captured by each of the plurality of imaging means of a virtual imaging system having an entrance pupil shape. The amount of light passing through the entrance pupil and the plane conjugate to the imaging element of the virtual imaging system having the imaging element conjugate plane shape is controlled to be larger.

  The present invention optimizes the orientation and angle of view of each camera when combining images as if they were captured by a virtual camera consisting of an optical system having an aperture of a certain size using images from a plurality of cameras. By controlling, it is possible to improve the image quality compared to the case where the control is not performed.

1 is a schematic diagram illustrating an example of an imaging apparatus including a plurality of cameras that is Embodiment 1. FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment. It is a block diagram showing the example of a structure of an imaging part. It is a block diagram showing the structural example of a digital signal processing part. 3 is a flowchart illustrating a procedure at the time of shooting by the imaging apparatus according to the first embodiment. It is a figure explaining the light ray which contributes to an image in a camera, and the light ray which does not contribute. FIG. 3 is a diagram illustrating an optimum direction and an angle of view of each camera in the imaging apparatus according to the first embodiment. FIG. 6 is a diagram illustrating a positional relationship between each coordinate axis, an imaging element conjugate plane, and an entrance pupil, accompanying the description of the information calculation method in the imaging apparatus according to the first embodiment. FIG. 3 is a block diagram illustrating a configuration of a control information calculation unit in the imaging apparatus according to the first embodiment. 3 is a flowchart illustrating a control information calculation procedure in the imaging apparatus according to the first embodiment. It is the figure which showed the relationship between the coordinate axis of an image, image size, and an optical center. It is the figure which showed the positional relationship of each coordinate axis and an image pick-up element conjugate surface in the image composition method of a present Example. It is a figure which shows the relationship between each camera coordinate axis and a world coordinate axis of a present Example. FIG. 10 is a diagram illustrating an optimum direction and angle of view of each camera in the imaging apparatus according to the second embodiment. FIG. 10 is a diagram illustrating a positional relationship between each coordinate axis, an imaging element conjugate plane, and an entrance pupil, which accompanies the description of the information calculation method in the imaging apparatus according to the second embodiment. It is a block diagram which shows the structure of the control information calculation part of Example 2. 10 is a flowchart illustrating a control information calculation procedure in the imaging apparatus according to the second embodiment.

[Example 1]
<Overall configuration of imaging device>
FIG. 1 is a schematic diagram of an imaging apparatus including a plurality of cameras that is Embodiment 1 of the present invention. (A) of the camera body 101 is a front view, and (b) is a rear view. The cameras 105 to 129 are arranged in a lattice pattern on the front surface (a). On the camera body 101, a shooting button 102, a display 103, and an operation button 104 are arranged. Various settings relating to shooting are performed using the operation button 104 and the display 103, and when the shooting button 102 is pressed, images are captured by the cameras 105 to 129. The result of image composition using the plurality of acquired images is displayed as one image on the display 103. Hereinafter, such an imaging apparatus including a plurality of cameras is referred to as a multi-lens camera.

  FIG. 2 shows each processing unit of the multi-lens camera of FIG. The imaging units 201 to 225 correspond to the imaging units of the cameras 105 to 129 in FIG. 1, respectively, and perform imaging based on the control information and the imaging instruction received via the bus 226, and the acquired image data is a bus. Output to 226.

  The imaging units 201 to 225 will be described in detail with reference to FIG. Each of the imaging units 201 to 225 includes a control unit 301, an A / D conversion unit 302, a buffer 303, an imaging element 304, a shutter 305, a lens 306, an aperture stop 307, a lens 308, a camera direction driving unit 309, a lens driving unit 310, and The lens driving unit 311 is configured. The lens 306, the diaphragm 307, and the lens 308 constitute an optical system and form an image of the subject to be photographed. The formed image is converted into an electrical signal by the image sensor 304 and converted into digital image data by the A / D converter 302. The converted digital image data is stored in the buffer 303. The stored data is read appropriately through the bus 226. Upon receiving an imaging instruction via the bus 226, the control unit 301 opens and closes the shutter 305 to control the imaging element 304 and execute imaging.

  The camera direction drive unit 309 is a mechanical device that changes the orientation of the camera, and directs the optical system in the direction of the optimum optical axis as will be described later. The lens driving units 310 and 311 are mechanical devices that change the position of the lens in the optical axis direction, and control the focus position and the focal length from the positional relationship between the two lenses and the image sensor. Each of the driving units 309 to 311 is driven by a motor or the like (not shown), and the driving amount is controlled by the control unit 301 based on the information on the focus position, the angle of view, and the camera orientation received via the bus 226.

  The CPU 229 is a processor that controls the entire processing of each component, and sequentially reads and interprets instructions stored in the ROM 227 and RAM 228, and executes processing according to the result. The ROM 227 and the RAM 228 provide the CPU 229 with programs, data, work areas, and the like necessary for the processing.

  Generally, a liquid crystal display is widely used as the display unit 231. For example, a rear display 103 of the camera body 101 as shown in FIG. 1 can be used. The display unit 231 is controlled by the display control unit 230 and displays characters and images received from the CG generation unit 235 and an image composition unit 240 described later.

  The operation unit 232 can use buttons, dials, and the like, such as the operation button 104 and the shooting button 102 in FIG. The CPU 229 receives a user instruction via these operation units 232. Parameter setting for composition and imaging, which will be described later, is performed via a user interface realized by cooperation of the ROM 227, RAM 228, CPU 229, display control unit 230, display unit 231, operation unit 232, CG generation unit 235, and the like. . The external memory control unit 233 is an interface for inputting / outputting data by connecting to a PC or other media 234 (for example, hard disk, memory card, CF card, SD card, USB memory). Upon receiving an output instruction from the CPU 229, the external memory control unit 233 outputs data stored in the RAM 228 to an external PC or other media.

  The CG generation unit 235 generates characters and graphics for realizing a user interface. The digital signal processing unit 236 performs noise reduction processing, white balance processing, color conversion processing, and gamma processing on the digital image data obtained from each imaging unit. The configuration of the digital signal processing unit 236 is shown in FIG. For digital image data that has been captured and acquired by the imaging units 201 to 225, the pixel value is output in proportion to the luminance, and the noise that is dependent on the imaging element is reduced, so that the data is physically faithful. Apply conversion. Further, the image after the image synthesis is adjusted so as to have a desirable image quality.

  The compression / decompression unit 237 performs processing for converting an image that has been captured and subjected to signal processing, or an image generated by image synthesis into a file format such as Jpeg or Mpeg. The control information calculation unit 238 calculates the control information of the angle of view and the orientation of the imaging units 201 to 225 based on the parameters set by the user. The calculation method and parameters of the angle of view and orientation control information will be described in detail later.

  The calibration unit 239 is configured based on the control information calculated by the control information calculation unit 238 and the images captured by the imaging units 201 to 225 and subjected to signal processing by the digital signal processing unit 236. Calculate internal parameters. Similarly, the calibration unit 239 calculates individual external parameters of each camera based on the control information and the signal-processed image. The calibration unit 239 also corrects image distortion. The image compositing unit 240 synthesizes the images captured by the image capturing units 201 to 225 and subjected to the signal processing by the digital signal processing unit 236 based on the parameters set by the user, and as if captured by a virtual camera. Generate an image. Although there are other components of the actual apparatus other than the above, the description thereof will be omitted for the purpose of clearly explaining the present invention. The imaging apparatus of the present invention is characterized by the imaging units 201 to 225 and the control information calculation unit 238.

<Flow of processing during shooting>
FIG. 5 is a flowchart showing an overall processing flow at the time of photographing with a multi-lens camera. Of these processes, the calculation of control information in step S502 and the calibration in step S507, which are processes characteristic of the present invention, will be described in detail later. First, in step S501, parameters are set by the user. The parameter is a numerical value that serves as a basis for controlling the behavior during composition and imaging, and will be described in detail later. The parameter setting is performed via a user interface realized by cooperation of the ROM 227, RAM 228, CPU 229, display control unit 230, display unit 231, operation unit 232, CG generation unit 235, and the like.

  The set parameters are stored in the RAM 228 via the bus 226. In step S502, control information of each camera is calculated based on the set parameters. That is, when the parameter setting is completed, the control information calculation unit 238 reads the parameter from the RAM 228, calculates a value for controlling each camera, and stores it in the RAM 228 (calculation of control information described later). In step S503, each camera is controlled according to the calculated control information. That is, control information is passed to the control unit 301 of the imaging units 201 to 225 via the bus 226, and the camera direction driving unit 309 and the lens driving units 310 and 311 are controlled based on the received values. When control is completed based on the control information, the camera enters a standby state in which imaging can be performed.

  In step S504, an imaging instruction from the user is received from the user via the operation unit 232. When the imaging instruction is received, the imaging instruction is transmitted to the control unit 301 of the imaging units 201 to 225 via the bus 226. When the imaging instruction is transmitted, the imaging is executed in S505. That is, the control unit 301 performs control in the order of reset of the image sensor 304, opening of the shutter 305, closing of the shutter 305, and reading of the image signal from the image sensor 304. The read signal is digitized by the A / D converter 302, stored in the buffer 303, and the imaging process ends.

  In step S506, the digital signal processing unit 236 reads the image data from each of the buffers 303 of the imaging units 201 to 225, performs digital signal processing, and stores it in the RAM 228. In S507, the calibration unit 239 performs calibration processing for each camera based on the control information stored in the RAM 228 and the image data, and obtains internal parameters and external parameters for each camera. The image whose distortion has been corrected by the result of calibration and the internal parameters and external parameters of each camera obtained by the calibration are stored in the RAM 228.

  In step S508, the image composition unit 240 performs composition of the virtual camera image based on the parameters stored in the RAM 228, the distortion-corrected image, the internal parameters of each camera, and the external parameters. The synthesized image is stored in the RAM 228. In step S509, the digital signal processing unit 236 performs image processing on the composite image stored in the RAM 228. In step S510, the compression / decompression unit 237 compresses the composite image that has undergone image processing in the RAM 228, and the compressed image is stored in the RAM 228. In step S511, the external memory control unit 233 stores the compressed image in the external memory. In step S512, the composite image is displayed on the display unit 231 via the display control unit 230.

<Outline of control information calculation>
Information for controlling the angle of view and the orientation of each camera constituting the multi-view camera in the control information calculation unit 238 according to the parameters set by the user, that is, calculation of control information will be described in detail below. Here, in order to simplify the description, processing actually performed in three dimensions will be described in two dimensions.

  In the present invention, a virtual imaging system is assumed, and each camera is controlled so as to constitute a multi-lens camera with the best image quality when combining images as if they were captured by the imaging system. An actual camera is composed of various combinations of lenses, diaphragms, shutters, and image sensors having various characteristics. As a result, there are a wide variety of types. Identify the imaging meter. That is, for example, even if there are a plurality of cameras in which various lenses are combined, if the attributes set as the imaging system are the same, even if different lens combinations are used, the optical systems are considered to be the same. The attribute set in this way is a parameter, and is information representing a virtual imaging system. Therefore, in order to obtain the best image quality, it is necessary to have parameters for controlling each camera so that light rays contributing to an image when captured by a virtual imaging system are acquired without excess or deficiency.

  With reference to FIG. 6, a description will be given of light rays that contribute to an image when captured by a virtual imaging system. This imaging system includes an optical system 612 corresponding to a camera lens and an imaging element surface 601 corresponding to an imaging element. That is, the optical system 612 includes components such as lenses 603 and 604 and an aperture stop 605 in an actual camera. The properties of the optical system can be represented by a front focal point 609, a rear focal point 608, a front principal point 611, a rear principal point 612, an entrance pupil 606, and an exit pupil 607. It is these points and pupils that affect the characteristics of the optical system. If it is determined what point and movement to select, the actual optical system structure may be a structure other than this example. It does not matter. Further, the imaging element conjugate plane 602 can be obtained as a plane conjugate to the imaging element plane 601 from the imaging element plane 601, the focal points 608 and 609, and the principal points 611 and 612. The condition of light rays contributing to the image when imaged by this imaging system is that the light passes through the aperture stop 605 while being refracted and reaches the image sensor surface 601. In other words, a straight line passing through the imaging element conjugate plane 602 and the entrance pupil 606 is a light ray that contributes to an image when the imaging system captures an image.

  In FIG. 6, the light ray A 613 passes through the entrance pupil 606, but does not pass through the image sensor conjugate plane 602, and therefore does not reach the image sensor surface 601. The light beam B 614 passes through the imaging element conjugate plane 602 but does not pass through the entrance pupil 606, and is thus blocked by the aperture stop 605. Since the light ray C 615 passes through both the image sensor conjugate surface 602 and the entrance pupil 606, it reaches the image sensor surface 601. Therefore, passing through two plane figures, that is, the entrance pupil and the imaging element conjugate plane, is a light ray condition that contributes to creating a composite image. That is, the parameters set by the user are essentially information on the shape and position of the virtual camera entrance pupil and the shape and position of the virtual camera imaging element conjugate plane. In an actual system, such information is difficult to understand intuitively, so items that are easy to understand for the user are selected. For example, the shape of the image sensor, the focus position, the focal length, the F value, the type of virtual lens used for the virtual camera, etc. may be shown to the user, and the parameters may be calculated from such information.

The camera control method will be conceptually described with reference to FIG. By controlling each camera so as to pass through the imaging element conjugate plane 602 and the entrance pupil 606 as shown in FIG. 7 based on the condition of the light rays contributing to the image to be captured, light information can be acquired efficiently. That is, the position of a certain camera is r, the positions of the ends of the image sensor conjugate plane 602 are f ₁ and f ₂ , the positions of the ends of the entrance pupil 606 are p ₁ and p ₂ , the camera angle of view is α, When the angle from the x-axis is β, the relationship is as shown in equations (1) and (2). Here, the unit vector in the x-axis direction is e ₁ , and the unit vector in the y-axis direction is e ₂ .

  Therefore, in order to efficiently acquire the light ray information with each camera, it is only necessary to control each camera so as to have the angle of view α and the direction β obtained by the following expressions (3) and (4).

Since actual multi-lens cameras are arranged on the array and change their orientation three-dimensionally, it is not possible to simply determine the optimum values for the angle of view and orientation of the camera as in equations (3) and (4). . In this case, first, a common portion of the shape obtained by perspective projection of the entrance pupil 606 and the imaging element conjugate plane 602 on the camera for which control information is desired is obtained. Then, the minimum angle of view that can cover this is calculated as an evaluation value, and the control information may be obtained by optimizing the rotation amount of the camera so that the calculated evaluation value becomes small. This point will be described with reference to FIG. Assume that the position of the optical center of the camera for which control information is to be obtained is t, and the coordinate axes on the rectangular image sensor 601 are u-axis and v-axis along the sides. Assume that the world coordinate axes of the multi-lens camera are the x-axis, y-axis, and z-axis, and the u-axis and x-axis, and the v-axis and y-axis are parallel in the initial state. Here, if the target camera direction drive unit is a mechanism that can change the orientation of the camera by the rotation θ around the y-axis and the rotation φ around the z-axis while maintaining the optical center position, it is on a plane of distance 1 The perspective camera matrix M _p is as shown in Equation (5).

入射瞳形状、撮像素子共役面形状を多角形で表すとし、それぞれの３次元空間上でのN個の頂点の斉次座標をp_n、f_nと置く。このとき、距離1の平面上に透視投影したときの２次元座標 (p_{u n} ， p_{v n})、(f_{u n} ， f_{v n})は、式（８）のようになる。

Assume that the entrance pupil shape and the imaging element conjugate plane shape are represented by polygons, and the homogeneous coordinates of N vertices in each three-dimensional space are set as p _n and f _n . At this time, the two-dimensional coordinates (p _{u n} , p _{v n} ) and (f _{u n} , f _{v n} ) when perspective-projected on the plane of distance 1 are as shown in Expression (8).

N ′ vertices when the common part is obtained and expressed as a polygon are (c _{u n} , c _{v n} ), and the u-axis direction and the v-axis direction size of the image sensor are _su , s _v Then, the evaluation value E to be minimized is as shown in Equation (9).

Θ and φ that minimize E are the optimum values of the rotation angle, and when E is E _min at this time, the optimum values of the angle of view α _{u in the u-} axis direction and the angle of view α _{v in the} v-axis direction are given by the formula ( 10)

The method of minimization is to change the rotation angle at random, repeat the projective transformation of (c _{u n} , c _{v n} ) placed on the plane of distance 1, and search for the rotation angle that minimizes the evaluation value, etc. Can be used.

  It should be noted that the way of movement of the camera shown in this embodiment is an example, and the method of taking parameters may be set in accordance with the mechanical mechanism at the time of implementation, and this embodiment does not limit it. The direction and angle of view of each camera need only be set so as to be able to capture light rays that pass through the entrance pupil and the imaging element conjugate plane and acquire light ray information that contributes to imaging. The method is not limited. Further, instead of changing the orientation of each camera, the optical center may be shifted by changing the relative position of the imaging element and the lens in the direction perpendicular to the optical axis to change the imaging range.

<Configuration of control information calculation unit>
The configuration of the control information calculation unit 238 will be described with reference to FIG. The control information calculation unit 238 is connected to the bus 226, and acquires information stored in the RAM 228 via the bus 226. The entrance pupil shape setting unit 901 sets the three-dimensional shape of the entrance pupil shape based on the parameters representing the virtual imaging system acquired from the RAM 228. As an example of a method for setting the entrance pupil by parameters, there is a method in which a virtual imaging system whose entrance pupil shape is known in advance for each parameter is stored in the ROM 227 in association with the ID. There is a method of reading data that matches the parameters from the entrance pupil shape stored in the ROM 227 in such a manner. In addition, there is a method in which the position of the virtual imaging system with respect to the multi-view camera, the position and size of the entrance pupil in the virtual imaging system, etc. are used as parameters and converted into the entrance pupil shape in the coordinates of the multi-view camera.

  The imaging element conjugate plane shape setting unit 902 acquires parameters defining a virtual imaging system from the RAM 228, and sets the three-dimensional shape of the imaging element conjugate plane. The imaging element conjugate plane may be calculated using an ID for identifying a virtual imaging system and a focusing distance as parameters. Further, the imaging element conjugate plane shape may be calculated using the position of the virtual imaging system with respect to the multi-lens camera, the imaging element size of the virtual imaging system, the focal length, the principal point position, the focusing distance, and the like as parameters. This embodiment does not limit the expression of the parameters that define the virtual imaging system, and any expression may be used as long as there is information that can acquire the entrance pupil shape and the imaging element conjugate plane shape.

  First, the entrance pupil shape projection unit 903 reads, from the RAM 228, external parameters (position and orientation) of a camera for which control information is calculated, which has been obtained in advance by calibration. Then, the entrance pupil shape set by the entrance pupil shape setting unit 901 is projected using the read position and orientation of the target camera. First, the imaging element conjugate plane projection unit 904 reads, from the RAM 228, external parameters of a camera whose control information is to be calculated, which has been obtained in advance by calibration. Then, the image sensor conjugate plane shape set by the entrance pupil shape setting unit 902 is projected using the read position and orientation of the target camera.

  The common part shape calculation unit 905 calculates a two-dimensional common part shape obtained by the entrance pupil shape projection unit 903 and the imaging element conjugate plane shape projection unit 904. The optimal control value calculation unit 906 calculates the optimal field angle and orientation of the camera using the common part shape obtained by the common part shape calculation unit 905 and the external parameters of the camera read from the RAM 228. The flow of the control information calculation process performed with the above configuration will be described below.

<Flow of control information calculation processing>
The flow of the control center method calculation process will be described with reference to FIG. In step S1001, parameters that define the virtual imaging system are acquired. Parameters that define the virtual imaging system set in advance by the user are read from the RAM 228 by the entrance pupil shape setting unit 901 and the imaging element conjugate plane shape setting unit 902.

  In step S1002, external parameters of the camera for which control information is calculated are acquired. External parameters of the camera are read from the RAM 228 by the entrance pupil shape projection unit 903, the imaging element conjugate plane shape projection unit 904, and the optimum control value calculation unit 906. Here, the external parameter is calculated in advance by the calibration unit 239 and stored in the RAM 228, but is not limited thereto. In step S1003, the entrance pupil shape setting unit 903 sets the entrance pupil shape based on the parameter that defines the virtual imaging system acquired in step 1001. In step S1004, the imaging element conjugate plane shape setting unit 904 sets the imaging element conjugate plane shape based on the parameters that define the virtual imaging system acquired in step 1001.

  In step S1005, the entrance pupil shape set in step S1003 is projected according to the external parameters of the camera acquired in step S1002. The projection is performed by the entrance pupil shape projection unit 903 by the method described by the above formulas (5) and (8).

  In step S1006, the imaging device conjugate plane shape set in step S1004 is projected according to the external parameter acquired in step S1002. The projection is performed by the imaging element conjugate plane shape projection unit 904 by the method described using the above formulas (5) and (8). In step S1007, the common partial shape calculation unit 905 calculates a common partial shape between the projected entrance pupil shape obtained in step S1005 and the projected image sensor conjugate plane shape obtained in step S1005.

  In step S1008, the camera direction and the angle of view as the optimum control values are calculated using the common part shape calculated in step S1007 and the external parameters acquired in step S1002, using equations (9) and (10). Do as described. The optimum control value is calculated by the optimum control value calculation unit 906.

<Calibration process>
A calibration process executed by the calibration unit 239 will be described. Here, the calibration means obtaining an internal parameter, an external parameter, a distortion coefficient, and the like of the camera. If each camera is installed at a predetermined position with high accuracy, the direction and angle of view are controlled with high accuracy, and the optical system can also correct aberrations without aberration or from a previously determined table, calibration is not necessary. When such high-precision installation and control cannot be performed, it is necessary to perform calibration. For such a technique, a technique known in this technical field can be used.

  That is, for example, by using the technique disclosed in Non-Patent Document 2, after the camera orientation / angle of view is controlled, a chart on which a known pattern is printed is shot a plurality of times, so that the camera's internal parameters, external parameters, distortion A coefficient of aberration can be obtained. When distortion is negligible, the internal parameters can be obtained without using a chart by capturing images of three or more viewpoints by the technique disclosed in Non-Patent Document 3. Furthermore, the camera orientation and relative position can also be obtained from the basic matrix between the cameras. In this case, since the position of the camera has indefiniteness of the scale, the indefiniteness is eliminated by using the interval between any two cameras as the reference length.

<Image composition processing>
A process of combining a plurality of captured images captured by the image capturing units 201 to 224 and processed by the digital signal processing unit 236 as if they were captured by a virtual camera defined by predetermined parameters by the image combining unit 240 Will be described. Here, composition is performed from light ray information obtained by a camera (camera array) arranged in a lattice shape as shown in FIG. 1 so as to be an image captured by a virtual imaging system having an attribute of an ideal optical system.

  As shown in FIG. 11, in the virtual imaging system when the lateral magnification is M, all the light rays incident on the pixel at the position (x, y) on the imaging device surface 601 are on the imaging device conjugate surface 602 ( Pass through the position x / M, y / M). Therefore, an image captured by the virtual imaging system is obtained by superimposing the images captured by the respective cameras and projecting them on the imaging element conjugate plane 602 and sampling according to the number of pixels of the virtual camera.

First, a method for projecting an image captured by each camera onto the image sensor conjugate plane 602 will be described. Here, the coordinates are such that the x-axis and the y-axis are parallel to the horizontal and vertical directions of the imaging element conjugate plane 602 in the camera array plane, and the z-axis is along the optical axis of the virtual imaging system from the camera array to the imaging element. It is taken toward the conjugate plane 602. The horizontal angle of view of the nth camera is θ _{x n} , the vertical angle of view is θ _{y n} , the number of horizontal pixels is s _{x n} , the number of vertical pixels is s _{y n} , and the optical center position on the image is (c _{x n} , c _{y n} ), the internal parameter matrix A _n is _expressed by Equation (11). At this time, when the image coordinates xy are used, each parameter is shown as shown in FIG.

Each camera coordinate is in the left direction of the screen, the y axis is the top direction of the screen, and the z axis is in the direction of the optical axis. Let the vectors be v _{x n} , v _{y n} , and v _{z n} , respectively. At this time, _assuming that the position of each camera is t _n , the coordinate transformation matrix M _n from each camera coordinate to the world coordinate is _expressed by Equation (12). FIG. 13 shows the relationship between each camera coordinate axis and the world coordinate axis.

When a certain image coordinate in each camera is (x, y), the position of a point corresponding to (x, y) on the image sensor conjugate plane is obtained. H _n (x, y) is the equation (13) for the homogeneous coordinates of the desired point.

Here, k _{(x, y)} represents the depth when the image coordinates (x, y) in the n-th camera are projected onto the image sensor conjugate plane. λ is a coefficient representing indefiniteness of a constant multiple. When the distance from the origin of the world coordinates to the imaging element conjugate plane 602 is d, Expression (14) is established.

This equation (14) is solved for 1 / k _{(x, y)} , and _expressed as equation (15) using B _n .

  When Expression (13) and Expression (15) are used, the coordinates (u, v) of the points projected on the imaging element conjugate plane are expressed by Expression (16) when described in two-dimensional homogeneous coordinates.

  That is, by performing projective transformation in accordance with Expression (16), the image of each camera can be projected on the imaging element conjugate plane. In general, image data that is a conversion countermeasure is an image sampled at discrete positions, and therefore, when projective conversion is performed, a portion having no data is filled by bilinear interpolation or cubic interpolation.

Next, when considering the entrance pupil, the part that is out of focus is naturally blurred in an optical system with an actual finite size pupil, so when multiple images are combined to acquire an image from a virtual imaging system It is necessary to reproduce natural blur. In order to reproduce such a blur, in the present invention, only the light rays passing through the entrance pupil are superimposed on the imaging element conjugate plane. Here, as shown in FIG. 8, the shape of the entrance pupil 606 is considered as a collection of M points P = (p ₁ , p ₂ ... P _m ... P _M ) connected by a straight line (in FIG. 8, M is 8). Applying equation (16) to the perspective projection of the homogeneous coordinates P ′ _n when each point belonging to P is projected from the n-th camera onto the image sensor conjugate plane 602 to the n-th camera of P Thus, equation (17) is obtained.

From the obtained P _'n, the n-th camera P projected on the imaging element conjugate plane by a 1 in position surrounded, and in the other position considering the mask D _n with 0. When an image made on the basis of the equation (16) and I _n, the image I obtained by synthesis can be calculated by the equation (18).

  The processing for synthesizing an image as if captured by a virtual camera has been described above. However, the method shown in the present embodiment is an example, and the present embodiment does not limit the image synthesizing method. Therefore, according to the first embodiment, it is possible to generate an image as if it was captured by a single virtual camera having a pupil of a finite size by combining a plurality of images from a multi-view camera. In other words, by optimizing the orientation and angle of view of each camera constituting the multi-view camera, it becomes possible to obtain a higher-quality virtual camera image.

[Example 2]
In the first embodiment, it is assumed that the in-focus position in the virtual camera image is designated at the time of imaging, and the orientation and angle of view of each camera on the multi-lens camera are optimally controlled. However, there is a demand for specifying the focus position of the virtual camera after imaging. In this embodiment, processing is performed so as to optimally control the orientation and angle of view of each camera on the multi-lens camera within a range that can accommodate the change in the focus position of the virtual camera after imaging.

  The second embodiment is different from the first embodiment in the control information calculation unit 238 in FIG. In the present embodiment, the difference between the first embodiment and the second embodiment in the control information calculation method will be mainly described.

  In order to specify the focus position of the virtual camera after imaging, each camera can acquire light rays without any shortage, regardless of the position of the imaging element conjugate plane from the front focus to infinity, as shown in FIG. It is necessary to set the direction and angle of view. Considering that the imaging element conjugate plane changes from the front focal point 609 to the infinity focal point, the locus of the imaging element conjugate plane forms a quadrangular pyramid 1401 composed of the infinity plane and the front focal point 609. Control information for controlling the orientation and angle of view of each camera on the multi-lens camera is calculated by the control information calculation unit 238 of the present embodiment so that the ray passing through both the quadrangular pyramid 1401 and the entrance pupil 606 can be captured. To do.

Here, in Example 1, the imaging device conjugate plane shape is represented by a polygon 602, and the homogeneous coordinates of the apex in the three-dimensional space are represented by f _n . In the second embodiment, instead of this, the homogeneous coordinates f _{inf n} of the apex of the imaging element conjugate plane shape 1501 when focused at infinity and the homogeneous coordinates f _point of the front focal point 609 are used. FIG. 15 shows the positional relationship between the coordinate axes, the imaging element conjugate plane, and the vertex of the entrance pupil 606 at this time. When f _{inf n} and f _point are perspective projected using the perspective projection matrix of Equation (5), the resulting two-dimensional coordinates (f _{inf u n} , f _{inf v n} ), (f _{f u} , f _{f v} ) Is obtained as equation (19).

If the convex hull of (f _{inf u n} , f _{inf v n} ) and (f _{f u} , f _{f v} ) is obtained, the cone 1401 of the imaging element conjugate plane trajectory through which the light rays contributing to the image pass is expressed as Z = It is a figure when it is perspective-projected on the plane of 1. If this figure and the common part of the entrance pupil shape obtained by the equation (8) are obtained, and the vertex is set as (c′u n, c ′ _{v n} ), the evaluation value to be minimized is the equation (20). It becomes like this.

  Similar to the calculation of the control information in the first embodiment based on the equation (20), if the evaluation value is minimized and the minimum camera orientation and angle of view are obtained, the optimum camera orientation and angle of view in the second embodiment are obtained. Can be requested.

  FIG. 16 shows the configuration of the control information calculation unit in the second embodiment. Compared with the control information calculation unit 238 in the first embodiment, the imaging element conjugate plane shape setting unit 902 is set to the imaging element conjugate plane shape setting unit 1601, and the imaging element conjugate plane locus shape setting unit 903 is set to the imaging element conjugate plane locus setting. Part 1602. In other words, the imaging element conjugate plane locus shape setting unit 1601 represents the solid 1401 of the imaging element conjugate plane locus formed by changing the focusing distance of the virtual imaging system from the focal point to infinity, representing the virtual imaging system. Set from parameters. The imaging element conjugate plane locus shape projection unit 1602 projects the locus solid 1401 set by the imaging element conjugate plane locus shape setting unit 1601.

  FIG. 17 shows the flow of the control information calculation process in the second embodiment. Compared with the control information calculation process in the first embodiment, step S1004 and step S1006 are step S1701 and step S1702, respectively. In step S1701, the imaging element conjugate plane locus shape setting unit 1601 calculates a solid 1401 of the imaging element conjugate plane locus formed when the in-focus distance of the virtual imaging system changes from the focal point to infinity. In step S1702, the imaging element conjugate plane locus shape projection unit 1602 projects the solid 1401 set in step S1701 based on the method described in Expression (19).

  As described above, according to the second embodiment, the focus position of the virtual camera can be changed after imaging. In other words, it is possible to obtain a higher-quality virtual camera image by optimizing the orientation and angle of view of each camera constituting the multi-lens camera on the imaging element conjugate plane in the range from the front focus to infinity. become.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed. The present invention can also be realized by a plurality of processors cooperating to perform processing.

Claims (11)

An information processing apparatus for processing an image of an imaging apparatus having a virtual imaging system characteristic defined by an entrance pupil shape and an imaging element conjugate plane shape,
A plurality of control means for controlling a range in which each of the plurality of imaging means images;
The control means includes a range of images picked up by each of the plurality of imaging means, the entrance pupil of the virtual imaging system having the entrance pupil shape, and the virtual imaging system having the imaging element conjugate plane shape. An information processing apparatus that controls an amount of light that passes through a surface conjugate with an imaging element to be larger.   The information processing apparatus according to claim 1, wherein the control unit controls an angle of view and a direction with respect to a subject of each imaging unit.   The information processing apparatus according to claim 1, further comprising receiving means for receiving parameters for setting an entrance pupil shape and an imaging element conjugate plane shape from a user.   The control means is configured to input the virtual imaging system having the entrance pupil shape with respect to each imaging element conjugate plane shape when the focal distance of the virtual imaging system is changed from the front focus to the infinity focus. The amount of light passing through the pupil and the plane conjugate with the imaging element of the virtual imaging system having the imaging element conjugate plane shape is controlled to be larger. 4. The information processing apparatus according to any one of 3.   Further provided is an image sensor conjugate plane trajectory shape setting means for setting a shape of a trajectory drawn by a plane conjugate to the image sensor of the virtual image pickup system when the focal distance of the virtual image pickup system is changed. The information processing apparatus according to claim 4. An information processing apparatus for processing an image of an imaging apparatus having a virtual imaging system characteristic defined by an entrance pupil shape and an imaging element conjugate plane shape,
A plurality of control means for controlling the range of images captured by each of the plurality of imaging means,
The control means includes a range of images picked up by each of the plurality of imaging means, the entrance pupil of the virtual imaging system having the entrance pupil shape, and the virtual imaging system having the imaging element conjugate plane shape. An information processing apparatus that controls such that light rays that pass through a plane conjugate with an image pickup element are not included in an area captured by each of the plurality of image pickup means.   The information processing apparatus according to claim 6, wherein the control unit controls an angle of view and a direction with respect to a subject of each imaging unit.   8. The information processing apparatus according to claim 6, further comprising receiving means for receiving parameters for setting an entrance pupil shape and an imaging element conjugate plane shape from a user.   The control means includes the entrance pupil of the virtual imaging system having the entrance pupil shape with respect to each imaging element conjugate plane shape when the focal distance of the virtual imaging system is changed from the front focus to the infinity focus. And light rays that pass through both of the surfaces conjugate to the imaging element of the virtual imaging system having the shape of the imaging element conjugate surface and are not included in an area captured by each of the plurality of imaging units. The information processing apparatus according to claim 6, wherein the information processing apparatus is controlled to be less.   Further provided is an image sensor conjugate plane trajectory shape setting means for setting a shape of a trajectory drawn by a plane conjugate to the image sensor of the virtual image pickup system when the focal distance of the virtual image pickup system is changed. The information processing apparatus according to claim 9.   The program for functioning a computer as a control means of the information processing apparatus of any one of Claim 1 to 10.

Images