JP2014110442A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus having a plurality of imaging parts suitably arranged.SOLUTION: A plurality of imaging parts arranged in an imaging apparatus include: determination means having an imaging part group arranged in concentric shape and an imaging part group arranged in lattice shape; determination means for determining an image processing mode for image data photographed by the imaging apparatus; and selection means for selecting an imaging part used for imaging depending on the mode. The determination means determines one or more modes which contain at least one of: a re-focus processing, a field-angle change processing, viewpoint position change processing, noise reduction processing, resolution conversion processing, HDR processing, distant estimation, and parallax estimation.

Description

  The present invention relates to an imaging apparatus. More specifically, the present invention relates to an imaging apparatus that captures multi-viewpoint image data.

  In the past, if the camera focus was mistakenly imaged, it was necessary to re-focus and re-image. In addition, when it is desired to obtain images that are in focus with respect to a plurality of subjects having different depths, it is necessary to perform imaging with the subject in focus several times.

  In recent years, a technology called light field photography has been developed that can acquire images from multiple viewpoints by adding new optical elements to the optical system, and later adjust the focus position (refocus) by image processing. ing.

  If this technique is used, since focus adjustment can be performed after imaging, there is an advantage that failure of focus adjustment during imaging can be compensated by image processing. Furthermore, by changing the image processing method, it is possible to obtain a plurality of images focused on an arbitrary subject in the image from one captured image, and there is an advantage that the number of times of imaging can be reduced.

  In light field photography, the direction and intensity of light passing through each position (light field, hereinafter referred to as “LF”) is calculated for a plurality of positions in space from image data from a plurality of viewpoints. Then, using the obtained LF information, an image when passing through the virtual optical system and forming an image on the virtual sensor is calculated. Refocusing is also possible by appropriately setting such virtual optical systems and sensors. R. NG et al. Discuss mathematical properties and mathematical foundations related to LF (Non-patent Document 1).

  As an imaging technique for acquiring LF, a camera array in which small cameras (single-eye cameras) are arranged is known. In the camera array, image data on the virtual sensor can be synthesized from the acquired LF after imaging.

  As a method for synthesizing an image on a virtual sensor from LF, a method is known in which a plurality of acquired image data are subjected to projective transformation on a virtual sensor, added and averaged (Patent Document 1).

  In addition to the refocus processing described above, the image data of a plurality of viewpoints can also be used for HDR processing for expanding the dynamic range and processing for estimating distance and parallax.

International Publication No. 2008/050904

R.NG, M.Levoy, M.Bredif, G.Duval, M. Horowitz, P. Hanrahan, “Light Field Photography with a Hand-held Plenoptic Camera” (Stanford Tech Report CTSR 2005-02, 2005)

  Although the above-described various processing functions can be realized using image data of a plurality of viewpoints using a camera array, the single-eye camera may not always be arranged appropriately for each function.

  The imaging device according to the present invention is an imaging device having a plurality of imaging units, and the plurality of imaging units includes an imaging unit group arranged concentrically and an imaging unit group arranged in a grid pattern. It is characterized by being.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the imaging device suitable for a some function by arrange | positioning a single eye camera in the position suitable for various functions.

1 is a diagram illustrating an example of an appearance of an imaging apparatus according to Embodiment 1. FIG. 1 is a diagram illustrating an example of a configuration of an imaging apparatus according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating an example of an imaging unit of the imaging apparatus according to the first embodiment. It is a figure for demonstrating a synthetic | combination process. It is a figure for demonstrating a synthetic | combination process. 6 is a flowchart illustrating an example of processing of the imaging apparatus according to the first embodiment. It is a figure which shows the modification of an imaging device. It is a figure which shows the other modification of an imaging device. It is a figure which shows an example of the external appearance of an imaging device.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of an appearance of the imaging apparatus 100 according to the present embodiment. FIG. 1 is a perspective view of the imaging apparatus 100 as viewed from the front side (surface on which an imaging unit is disposed).

  As shown in FIG. 1, a plurality of image pickup units 101-112 are concentrically arranged on the front surface of the image pickup apparatus 100, and a plurality of image pickup units 113-116 are arranged in a lattice shape inside the concentric circles. A camera array is formed. That is, the imaging apparatus 100 in FIG. 1 shows an example in which the first imaging unit group is arranged in the first shape and the second imaging unit group is arranged in a second shape different from the first shape. ing.

  In the example of FIG. 1, the first imaging unit group has 12 imaging units and the second imaging unit group has 4 imaging units. It is only necessary to include an imaging unit having a shape as described above, and the present invention is not limited to the example described with reference to FIG. Although not shown, the imaging apparatus 100 can have other functions such as various interfaces such as buttons, a touch panel display, and a jog dial.

  FIG. 2 is a diagram illustrating an example of the configuration of the imaging apparatus 100 illustrated in FIG. The CPU 201 executes the OS and various programs stored in the ROM 203 using the RAM 202 as a work memory. Further, the CPU 201 controls each component of the imaging device 100 via the system bus 200. The RAM 202 stores imaging parameters, which are information indicating the state of the imaging units 101-116, such as focus settings and aperture settings that indicate control results of the imaging optical system. The ROM 203 stores the relative positional relationship of the imaging units 101 to 116, the pixel pitch of the imaging element of each imaging unit, the light receiving efficiency of light energy, the camera design parameters indicating the focal length (view angle) of the imaging unit, and the like. Store. Although not shown, the camera design parameters of the image capturing unit may be stored in the ROM of each image capturing unit 101-116.

  The CPU 201 controls the computer graphics (CG) generation unit 207 and the display control unit 204 to display a user interface (UI) on the monitor 213. Further, the CPU 201 receives a user instruction via the shooting button 163 and the operation unit 164. Then, the CPU 201 can set shooting conditions such as a subject distance, a focal length, an aperture, an exposure time, and a flash emission according to a user instruction. Further, the CPU 201 can perform an imaging instruction and display setting of the captured image in accordance with a user instruction. The CG generation unit 207 generates data such as characters and graphics for realizing the UI.

  When photographing is instructed by the user, the CPU 201 acquires an optical system control method corresponding to the user's instruction from the optical system control method generation unit 209. Next, the CPU 201 instructs the optical system control unit 210 to perform imaging based on the acquired optical system control method. Upon receiving this imaging instruction, the optical system control unit 210 controls the imaging optical system such as adjusting the focus, adjusting the aperture, or opening and closing the shutter. In addition, the optical system control unit 210 stores, in the RAM 202, imaging parameters that are information indicating the state of the imaging units 101-116, such as focus settings and aperture settings that indicate the control results of the imaging optical system. Instead of controlling the imaging optical system of each imaging unit 101-116 with one optical system control unit 210, each imaging unit 101-116 may be provided with an optical system control unit that can communicate with the CPU 201.

  Each of the imaging units 101 to 116 receives light from a subject in an imaging sensor 307 such as a CCD or a CMOS. Details will be described later with reference to FIG. The imaging unit 101-116 temporarily holds imaging data (hereinafter, RAW data) obtained by analog-digital (A / D) conversion of an analog signal output from the imaging sensor 307 in a buffer memory in the imaging unit 101-116. . The RAW data held in the buffer memory is sequentially stored in a predetermined area of the RAM 202 under the control of the CPU 201.

  The digital signal processing unit 208 performs development processing for generating image data from a plurality of RAW data (hereinafter, RAW data group) stored in a predetermined area of the RAM 202. The digital signal processing unit 208 stores the RAW data group and the generated image data in a predetermined area of the RAM 202. The development process includes a composition process for combining a plurality of RAW data, a demosaicing process, a white balance process, a gamma process, a noise reduction process, and the like. Further, the digital signal processing unit 208 can change the zoom magnification with respect to the image data after photographing, and can perform processing for generating the changed image data. Parameters for development processing (hereinafter referred to as image generation parameters) indicating the focus distance, zoom magnification, depth of field, and the like are added to the generated image data. The image generation parameter is generated based on a value designated by the user, for example. In addition, for example, at the time of initial development, an initial setting value can be used as an image generation parameter. In addition, although at least imaging parameters are added to the RAW data group, camera design parameters may be added in consideration of development processing by an external image processing apparatus.

  The CPU 201 controls the display control unit 204 to display image data stored in a predetermined area of the RAM 202 on the monitor 213. The compression / decompression unit 212 performs an encoding process for converting image data stored in a predetermined area of the RAM 202 into a format such as JPEG or MPEG. Further, the compression / decompression unit 212 performs a process for lossless compression of the RAW data group, if necessary.

  The CPU 201 controls the display control unit 204 to display image data stored in a predetermined area of the RAM 202 on the monitor 213. The compression / decompression unit 212 performs an encoding process for converting image data stored in a predetermined area of the RAM 202 into a format such as JPEG or MPEG, and if necessary, a process for lossless compression of a RAW data group.

  An interface (I / F) 205 has a function of reading and writing a recording medium 206 such as a memory card or a USB memory, and a function of connecting to a wired or wireless network. In accordance with an instruction from the CPU 201, the I / F 205 outputs, for example, JPEG or MPEG format image data and RAW data stored in the RAM 202 to an external medium or server device, or outputs various data from an external recording medium or server device. input.

  The image generation parameter generation unit 211 generates image generation parameters necessary for development processing in the digital signal processing unit 208.

<Configuration example of each imaging unit>
The block diagram of FIG. 3 is a diagram illustrating a configuration example of the imaging unit 101. FIG. 3 shows a configuration example of the imaging unit 101, but the other imaging units 102-116 have substantially the same configuration. However, the settings such as the angle of view, the focus, and the aperture of the imaging units 101 to 116 do not necessarily have the same configuration.

  Light from the subject passes through a focus lens group 301a, a diaphragm 302a, a fixed lens group 303a, a shutter 304a, an infrared cut filter 305, and a color filter 306, and forms an image on an image sensor 307 such as a CMOS sensor or a CCD. The A / D conversion unit 308 performs A / D conversion on the analog signal output from the imaging sensor element. The buffer 309 temporarily stores the RAW data output from the A / D conversion unit 308 and transfers the RAW data to the RAM 202 via the system bus 200 in response to a request from the CPU 201.

  The arrangement of the lens group and the diaphragm shown in FIG. 3A is an example and may be different. For example, some or all of the imaging units may not have the fixed lens group 303a for improving lens performance such as telecentricity.

(Image composition processing)
The concept of image synthesis by the digital signal processing unit 208 will be described with reference to FIGS. 4 and 5. FIG. 4A illustrates a state where subjects at different distances are captured by the imaging units 401 to 403.

  In FIG. 4A, imaging units 401 to 403 illustrate three typical imaging units among the imaging units 101 to 116. Dotted lines 404 to 406 illustrate three typical virtual focus positions among the virtual focus positions (positions to be focused). As shown in FIG. 4A, the subjects 407 to 409 are placed at different distances.

  FIG. 4B shows an image 410 acquired by the imaging unit 401. Images acquired by the imaging units 402 and 403 are images in which the subject is shifted by the amount of parallax corresponding to the distance between the subjects 404 to 406 in the image 410.

  FIG. 5 is a conceptual diagram of an image reconstructed (combined) by the digital signal processing unit 208. An image 501 is an image after reconstruction when the virtual focus position is set to the dotted line 406. In the image 501, the subject 407 is in focus, and the subjects 408 and 409 are blurred.

  An image 502 is a reconstructed image when the virtual focus position is a dotted line 404 when the virtual focus position is a dotted line 405. In the images 502 and 503, the subjects in focus are 408 and 409, respectively. By moving the virtual focus in this way, it is possible to obtain an image that focuses on a favorite subject.

  In this example of composition processing, by controlling the virtual focus position, it is possible to focus on a predetermined subject and simultaneously blur other subjects. Examples of the synthesis process are not limited to this. For example, a process for changing the angle of view, a process for changing the viewpoint position, a noise reduction process for reducing noise, an HDR process for expanding the dynamic range, and a high resolution process for increasing the resolution. There may be.

<Processing flow of imaging device>
FIG. 6 is a flowchart illustrating an example of processing of the imaging apparatus.

  In step S601, the optical system control method generation unit 209 determines a mode to be used among a plurality of preset modes (functions). Examples of the plurality of modes include the following modes. That is, a mode for performing a refocus process, a mode for performing a view angle changing process, a mode for performing a viewpoint position changing process, a mode for performing noise reduction processing, a mode for performing resolution conversion processing by converting the resolution of an image, and a mode for performing HDR processing In this mode, distance / disparity estimation is performed. Note that modes other than the modes described above may be included. The mode to be used can be determined by the optical system control method generation unit 209 based on information input from the user via the operation unit 164, for example. For example, the user can input a mode to be used to the imaging apparatus 100 via a jog dial (not shown) or a touch panel on a switch display.

  In step S602, the optical system control method generation unit 209 selects an imaging unit to be used for imaging according to the mode determined in step S601. That is, the optical system control method generation unit 209 determines an imaging unit to be controlled via the optical system control unit 210.

  When performing refocus processing to change the focus position after imaging, processing using the imaging unit group arranged concentrically rather than using the imaging unit group arranged in a grid is a process for processing or editing an image Suitable for For example, when controlling the depth of field at the time of refocusing or adding blur, the image pickup unit group arranged concentrically can reproduce natural blur.

  On the other hand, the imaging unit group arranged in a lattice shape has a smaller calculation error than the imaging unit group arranged in a concentric circle, and thus is suitable for distance / disparity estimation.

  Thus, a preferable imaging unit group differs depending on the mode to be used. Therefore, in step S602, the imaging unit used for imaging is selected according to the mode. In step S602, it is not necessary to select all the imaging units of the imaging unit group as imaging units used for imaging. The optical system control method generation unit 209 may determine an imaging unit group to be used for imaging according to the mode, and may select an imaging unit to be used for imaging from the determined imaging unit group. For example, more imaging units may be selected when the determined mode processing is performed more precisely, and fewer imaging units may be selected when the determined mode processing is performed at higher speed.

  In addition, although the case where one mode is determined is described as an example in step S601, a plurality of modes may be determined in step S702. For example, a refocus process and a distance estimation process may be determined. In this case, in step S602, the optical system control method generation unit 209 uses both the imaging units arranged concentrically and the imaging units arranged in a grid shape for imaging according to the determined mode. You may choose as

  In step S603, the optical system control unit 210 performs imaging using the imaging unit selected in step S602. The captured image data is recorded in the RAM 202.

  In step S603, the CPU 201 controls the digital signal processing unit 208 to perform image processing on the image data captured in step S603 according to the mode determined in step S601.

  As described above, in this embodiment, a camera array is formed in which the first imaging unit group is concentrically arranged and the second imaging unit group is arranged in a lattice shape inside the concentric circle. Yes. With such a configuration, an imaging unit to be used can be selected according to a mode used in image processing after imaging. Then, by using the image data obtained through the imaging unit selected in this way, suitable image processing can be performed according to each mode.

  In the first embodiment, an imaging apparatus configured to form a camera array in which the first imaging unit group is arranged concentrically and the second imaging unit group is arranged in a concentric circle inside the grid. Explained the example. In the second embodiment, various modifications will be described.

  FIG. 7 shows an example of an imaging apparatus 700 in which concentric imaging units 705-712 are arranged inside a grid-like imaging unit 701-704, contrary to FIG. 1 described in the first embodiment. In the example of FIG. 7, an example in which the imaging unit 713 is further arranged inside the concentric imaging units 705 to 712 is shown, but the imaging unit 713 may or may not be arranged.

  FIG. 8 shows an imaging apparatus 800 that is a modified example of the imaging apparatus 700 shown in FIG. That is, an example is shown in which the imaging units 805-812 arranged concentrically and the imaging units 801-804 arranged in a grid are arranged side by side in parallel. Note that, as described with reference to FIG. 7, the imaging unit 813 inside the imaging units 805-812 arranged concentrically may or may not be arranged.

  As described above, the present invention is applicable to any case where the first imaging unit group is arranged in the first shape and the second imaging unit group is arranged in the second shape different from the first shape. It is possible to apply

<Appearance of imaging device>
FIG. 9 is a diagram illustrating an example of the appearance of the imaging apparatus. FIG. 9A is a front view. FIG. 9B is a plan view. FIG. 9C is a bottom view. FIG. 9D is a left side view. FIG. 9E is a right side view. FIG. 9F is a rear view.

(Other embodiments)
In the example described above, a plurality of imaging units are described as being concentrically arranged. However, they are not necessarily concentric circles and may be arranged in an elliptical shape. In the above example, the case where the first image pickup unit group and the second image pickup unit group form two shapes has been described as an example, but there are three or more such as providing a third image pickup unit group. The shape may be formed.

  The object of the present invention can also be achieved by executing the following processing. That is, a storage medium that records a program code of software that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus is stored in the storage medium. This is the process of reading the code. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

Claims (7)

An imaging apparatus having a plurality of imaging units,
The plurality of imaging units are configured by an imaging unit group arranged concentrically and an imaging unit group arranged in a lattice shape. Determining means for determining an image processing mode of image data captured by the imaging device;
Selection means for selecting an imaging unit to be used for imaging according to the mode;
The imaging apparatus according to claim 1, further comprising:   The imaging apparatus according to claim 2, wherein the determining unit determines one or more modes.   3. The mode according to claim 2, wherein the mode includes at least one of refocus processing, angle of view change processing, viewpoint position change processing, noise reduction processing, resolution conversion processing, HDR processing, distance estimation, and parallax estimation. 3. The imaging device according to 3.   The imaging device according to any one of claims 1 to 4, wherein the imaging unit group arranged in a lattice shape is arranged inside the imaging unit group arranged in a concentric shape.   The imaging device according to claim 1, wherein the imaging unit group arranged concentrically is arranged inside the imaging unit group arranged in the lattice shape.   5. The imaging device according to claim 1, wherein the imaging unit group arranged in a lattice shape is arranged in parallel with the imaging unit group arranged in a concentric shape. 6.

Images