JP2012244583A - Imaging device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain highly accurate three-dimensional data with a simple configuration.SOLUTION: A multiple lens system image pickup device comprises: imaging units (101-117) which pick up a color image; a plurality of imaging units (118-125) which pick up a monochrome image; a three-dimensional data calculation unit which calculates three-dimensional data of a subject on the basis of the monochrome image obtained by the plurality of imaging units which pick up the monochrome image; and a synthesizing unit which generates color three-dimensional image data which associates the three-dimensional data with the color image obtained by the imaging units which pick up the color image.

Description

  The present invention relates to an image acquisition method for generating subject color three-dimensional data by using a multi-lens imaging device having imaging units capable of imaging from a plurality of different viewpoint positions, an imaging device using the image acquisition method, and a program It is.

  In recent years, there has been an increasing demand to display a subject in a three-dimensionally visible manner by measuring three-dimensional data of a subject and using a color three-dimensional image obtained by combining the color image of the subject and a measurement value. One method of measuring three-dimensional data of a subject is a method of performing stereo matching processing based on correlation between images using a plurality of images with parallax captured by a monochrome (monochrome) stereo camera. . In this method, in order to improve the measurement accuracy of the three-dimensional data of the subject, a multi-view method for measuring the three-dimensional data of the subject using three or more images having different parallax is known.

  In addition to the subject's three-dimensional data, a method for generating the subject's color three-dimensional image data by synthesizing the subject's color image captured by the color camera and the subject's three-dimensional data obtained by stereo matching. Has been proposed (see, for example, Patent Document 1).

  According to Patent Document 1, an imaging device that obtains a parallax image is a stereo camera including one monochrome camera and one color camera. After the color image C acquired by the color camera is converted into a monochrome image GA, the three-dimensional data of the subject is measured by stereo matching processing using the monochrome image GB and the monochrome image GA acquired by the monochrome camera. By associating the measured three-dimensional data of the subject with the color image C, color three-dimensional image data of the subject is generated.

  Furthermore, a technique has been proposed in which a color imaging area and a monochrome imaging area are combined on an imaging element of one imaging apparatus to generate color three-dimensional image data of a subject (for example, see Patent Document 2). According to Patent Document 2, a lens array is arranged in front of an image sensor on the optical axis of an image pickup apparatus, and an image with different parallax is generated by one image pickup apparatus. Three-dimensional data of a subject is calculated from image data obtained from a plurality of monochrome imaging regions set on an imaging element of the imaging device of the imaging device. A color image of a subject is acquired from a color imaging area set on the same imaging device. Then, the color three-dimensional image data of the subject is generated by combining the three-dimensional data of the subject and the color image.

Japanese Patent No. 4193292 JP 2009-284188 A

  In the method proposed in Patent Document 1, when a monochrome image is generated from a color image acquired by a color camera, interpolation is performed from a plurality of pixels having different color information in the vicinity of the target pixel of the color image. The luminance value (monochrome value) is obtained by calculation. Therefore, if an attempt is made to match the resolution with a monochrome image obtained by a monochrome camera, an error due to an interpolation process inevitably occurs, so that highly accurate three-dimensional data cannot be obtained. Furthermore, since it is necessary to dispose a color filter on the image sensor in order to acquire a color image, noise generated in the image is more conspicuous than an image obtained with a monochrome camera. Since this noise lowers the accuracy of stereo matching, there is a problem that it affects the calculation accuracy of the three-dimensional data desired to be acquired. Therefore, the technique described in Patent Document 1 has a problem that a highly accurate parallax image cannot be obtained.

  Further, in the method proposed in Patent Document 2, in order to obtain a parallax image with one imaging device, there is a problem that the size of the parallax depends on the size of the main lens, the lens array, and the imaging element of the imaging device. is there. That is, if a large amount of parallax is ensured, the estimation accuracy of the three-dimensional data is improved, but there is a problem that the cost of the imaging device is increased. If the conventional imaging apparatus is used, a sufficient amount of parallax cannot be secured, and highly accurate three-dimensional data data cannot be obtained.

  In order to solve the above problems, the present invention is based on an imaging unit that captures a color image, a plurality of imaging units that capture a monochrome image, and a monochrome image obtained by the plurality of imaging units that capture the monochrome image. A three-dimensional data calculation unit that calculates three-dimensional data of a subject; a synthesis unit that generates color three-dimensional image data in which a color image obtained by an imaging unit that captures the color image and the three-dimensional data are associated with each other And a multi-eye imaging device.

  According to the present invention, highly accurate three-dimensional data can be obtained with a simple configuration.

1 is a diagram illustrating an example of a multi-lens imaging device including a plurality of imaging units according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of the multi-lens imaging device of FIG. It is a figure explaining the detail of an imaging part. It is a block diagram which shows the structure of the image process part 212 which concerns on embodiment of this invention. It is a flowchart which shows the flow of an image process. It is the figure which showed typically the process of calculating the three-dimensional data of the to-be-photographed object in the pixel position on a sensor. It is the figure which showed typically the process of calculating the three-dimensional data of a to-be-photographed object. It is a flowchart which shows the flow of a three-dimensional data calculation process. It is a figure showing the other example of the imaging device of a multi-view system provided with the some imaging part. It is a figure showing an example of the imaging device of a multi-view system from which arrangement of an imaging part differs. It is a figure showing an example at the time of making the number of sensor pixels of a monochrome image pick-up part relatively smaller than the number of sensor pixels of a color image pick-up part.

  FIG. 1 is a multi-eye imaging apparatus including 25 imaging units according to the first embodiment of the present invention. Reference numerals 101 to 125 in FIG. Among them, 101 to 117 are color imaging units that acquire color images, and 118 to 125 are monochrome imaging units that acquire monochrome images. Details of the color imaging unit and the monochrome imaging unit will be described later. Reference numeral 126 denotes a photographing button, and 127 denotes a housing of the imaging device. Since a monochrome image is used when generating three-dimensional data, which will be described later, it is preferable to set the interval between the monochrome imaging units wider than the interval between the color imaging units. Further, as shown in FIG. 1, a color imaging unit is used as an imaging unit that is arranged near the center of the imaging unit, that is, when the imaging units 101 to 125 are arranged at predetermined intervals as shown in FIG. Is placed. For example, the color imaging units 104 to 114. Furthermore, a configuration in which the monochrome imaging units 118 to 125 are arranged outside so as to surround the color imaging units 104 to 114 is more preferable. With such a configuration, it is possible to obtain images obtained by imaging the subject from a plurality of different viewpoint positions.

  FIG. 2 shows each processing unit constituting the multi-eye imaging apparatus of FIG. The color image pickup units 101 to 117 and the monochrome image pickup units 118 to 125 receive optical information of a subject by a sensor (image pickup device), perform A / D conversion, and then output digital data to a bus 213 that is a data transfer path. To do.

  The central processing unit (CPU) 203 is involved in all the processes of each configuration, reads the instructions stored in the ROM 201 and the RAM 202 in order, interprets them, and executes the processes according to the results. The ROM 201 and the RAM 202 provide the CPU 203 with programs, data, work areas, and the like necessary for the processing.

  The operation unit 204 includes buttons, a mode dial, and the like, and receives input user instructions. The imaging unit control unit 207 controls the imaging system instructed by the CPU 203 such as focusing, opening a shutter, and adjusting an aperture.

  The digital signal processing unit 134 performs white balance processing, gamma processing, noise reduction processing, and the like on the digital data supplied from the bus 213 to generate a digital image. The encoder unit 209 performs processing for converting the digital data into a file format such as Jpeg or Mpeg.

  The external memory control unit 210 is an interface for connecting to a PC and other media 211 (for example, hard disk, memory card, CF card, SD card, USB memory).

  Generally, a liquid crystal display is widely used as the display unit 206, and displays a captured image and characters received from an image processing unit 212 described later. In addition, the display unit 206 may have a touch screen function, and in that case, a user instruction can be handled as an input of the operation unit 204. A display control unit 205 performs display control of a captured image and characters displayed on the display unit 206.

  The image processing unit 212 performs image processing using a digital image obtained from the imaging units 101 to 125 or a digital image group output from the digital signal processing unit 208, and outputs the result to the bus 213. It should be noted that the constituent elements of the apparatus can be configured other than the above by combining the constituent elements so as to have equivalent functions. The imaging apparatus of the present invention is characterized by the imaging units 101 to 125 and the image processing unit 212.

  Next, details of the imaging units 101 to 125 will be described with reference to FIG. A color imaging unit 311 illustrated in FIG. 3A illustrates a specific configuration of the color imaging units 101 to 117.

  The color imaging unit 311 includes a zoom lens 301, a focus lens 302, a shake correction lens 303, a diaphragm 304, a shutter 305, an optical low-pass filter 306, an iR cut filter 307, a color filter 308, a sensor 309, and an A / D conversion unit 310. .

  The color imaging unit 311 detects the amount of light of the subject using the components indicated by 301 to 309. Then, the A / D conversion unit 310 converts the detected light amount of the subject into a digital value. Note that the monochrome imaging unit 312 illustrated in FIG. 3B has a configuration in which the color filter 308 is removed from the color imaging unit 312.

<Configuration of image processing unit>
FIG. 4 is a block diagram illustrating a configuration of the image processing unit 212 illustrated in FIG. The image processing unit 212 includes a color image acquisition unit 401, a monochrome image acquisition unit 402, a three-dimensional data calculation unit 403, a camera parameter DB 404, and a synthesis unit 405.

  The color image acquisition unit 401 acquires a color image group supplied from the color imaging units 101 to 117 via the bus 213. In the first embodiment, one color image is supplied from each of the 17 color imaging units 101 to 117. That is, the color image acquisition unit 401 acquires 17 color images.

  The monochrome image acquisition unit 402 acquires monochrome images supplied from the monochrome imaging units 118 to 125 via the bus 213. In the present embodiment, eight monochrome imaging units are provided, and one monochrome image is supplied from each monochrome imaging unit. That is, the monochrome image acquisition unit 402 acquires eight monochrome images.

  The three-dimensional data calculation unit 403 calculates the three-dimensional data of the subject using the monochrome image group acquired from the monochrome image acquisition unit 402 and the camera parameters related to the imaging apparatus shown in FIG. The camera parameters are stored in the camera parameter database (DB) prior to the calculation of the three-dimensional data. The camera parameters used in the first embodiment are the relative position of each imaging unit, sensor size, number of sensor pixels, sensor pitch, angle of view, and the like.

  The synthesizing unit 405 associates the three-dimensional data of the subject calculated by the three-dimensional data calculating unit 403 with the color image of the subject acquired from the color image acquisition unit 401, and then combines and outputs the image data. Each processing unit is controlled by the CPU 203.

  Next, an image processing method performed by the image processing unit 212 will be described with reference to the flowchart of FIG. First, the monochrome image group image | photographed with the monochrome imaging parts 118-125 is input (step S501). In the first embodiment, one monochrome image captured by each monochrome imaging unit is input, and a total of eight monochrome images are input.

  Next, a reference camera serving as a reference when calculating the three-dimensional data is set (step S502). The reference camera is one color imaging unit selected from among the color imaging units 101 to 117. In this embodiment, first, the color imaging unit 101 is set as a reference camera.

  Next, a color image captured by the reference camera is input (step S503).

  In step S504, three-dimensional data corresponding to the pixel position of the reference camera is calculated based on the plurality of monochrome images input in step S501. Details of the calculation method of the three-dimensional data will be described later.

  When the calculation of the three-dimensional data is completed, the color image of the reference camera input in step S503 and the three-dimensional data calculated in step S505 are combined to generate color three-dimensional image data (step S505).

  Next, it is confirmed whether or not the three-dimensional data calculation process has been completed for the color images acquired by all the color imaging units (step S506).

  If there is an unprocessed color image (S506; NO), an unselected color imaging unit is selected from the color imaging units 101 to 117, and the reference camera is changed (step S507). The processes from step S504 to step S506 are repeatedly performed on the color images acquired by all the color imaging units until the calculation process of the three-dimensional data is completed. Note that the change of the reference camera in S507 is performed, for example, when the process for the color imaging unit 102 is completed, the color imaging unit 103 is set as the reference camera.

  If it is determined in step S506 that the calculation process of three-dimensional data has been completed for all color images (S506; YES), the combined data (color three-dimensional image data) for each color imaging unit is output (step S506). (S508) All image processing is completed.

<Calculation processing of 3D data>
In the three-dimensional data calculation process of S504, the three-dimensional data at each pixel position of the color imaging unit selected as the reference camera is calculated using the image group acquired by the monochrome imaging unit. The calculation process of the three-dimensional data used in the first embodiment is described in Vincent Nozick, Francois de Sorbier and Hideo Saito, “Plane-Sweep Algorithm: Various Tools for Computer Vision”, IEICE Technical Report, PRMU2007-259, 2008. It can be implemented based on the Plane Sweep method disclosed in March.

  6 and 7 are diagrams schematically showing a process of calculating three-dimensional data by the Plane Sweep method.

FIG. 6 is a diagram schematically showing how the three-dimensional data of the subject corresponding to the pixel position _Pout on the virtual sensor of the reference camera is calculated. In FIG. 6, X _out represents the physical position of the reference camera. The physical position is a coordinate position when observed from the world coordinate system. _Xc represents the physical position of the monochrome imaging unit. X _{d, out} is a vector indicating the direction of the reference camera on the screen. Y _{d, out} is a vector pointing in the right direction of the screen of the reference camera. Z _{d, out} is a vector indicating the direction of the reference camera. Since X _{d, out} , Y _{d, out} and Z _{d, out} are vectors that only represent directions, the length is set to 1. As a coordinate system representing these vectors, a coordinate system having a plane (reference plane) serving as a reference at the time of calculation as an xy plane and a height direction as az axis is used.

  FIG. 7 schematically shows a process of generating three-dimensional data of a subject observed from the reference camera using a monochrome image obtained from the monochrome imaging unit while changing the position of the reference plane in the z direction. The Plane Sweep method calculates three-dimensional data by changing the reference plane from the origin position of z = 0 in the z direction and considering the pixel value of the monochrome image corresponding to the pixel position of the sensor of the reference camera. is there.

  FIG. 8 is a flowchart showing a calculation procedure of the three-dimensional data.

  First, camera parameters such as the relative position of each imaging unit, sensor size, number of sensor pixels, sensor pitch, and angle of view are input (step S801).

Next, the pixel position _Pout on the virtual sensor of the reference camera is set (step S802).

Then, in calculating the three-dimensional data corresponding to the pixel position P _out, set the initial height of the reference plane (step S803). Here, the reference plane is set at a position where z = 0. FIG. 6 shows a case where the reference plane is at the position of z = Z _base .

Next, in step S804, the is an extension of the pixel P _out, to calculate a physical position X _p of a point corresponding to the reference plane. The position _Xp is obtained by the following formulas 1 to 4.

In Formula 1, X _pixel is the physical location of the pixel locations P _out. In Equation 2, θ _{v, out} is the vertical half field angle of the reference camera. In Expression 3, P _out is the coordinates of the pixel P _out for which the pixel value is to be determined. P _{out, center} is the center coordinates of the virtual sensor of the reference camera. h _out is the number of pixels in the vertical direction of the sensor provided in the reference camera. In Equation 4, X _{p, z} and X _{pixel, z} are z components of X _p and X _pixel , respectively.

Next, the c-th monochrome imaging unit (hereinafter referred to as monochrome camera c) to be referred to is selected (step S805), and the corresponding pixel position on the sensor of the selected monochrome imaging unit is calculated (step S806). Step S806 is a step of obtaining the pixel position P _{p, c} on the virtual sensor where the position X _p is captured in the monochrome camera c. The pixel position P _{p, c} is obtained by the following equations 5-8.

Here, X _{p, c, x} , X _{p, c, y} and X _{p, c, z} in Equation 5 are the x, y, and z components of X _{p, c} , respectively. θ _{h, c} and θ _{v, c} are the horizontal and vertical half angles of view of the monochrome camera c. _{Xc in} Expression 7 is the position of the monochrome camera c. X _{d, c} , Y _{d, c} , and Z _{d, c} in Expression 8 are direction vectors representing the upward direction, the right direction, and the direction of the monochrome camera c, respectively. The length of these vectors is 1. These series of operations are operations in which three conversion operations generally called view conversion, projective conversion, and screen conversion are combined.

Next, a pixel value I _{p, c} corresponding to the pixel position P _{p , c} is calculated (step S807). Since the pixel position P _{p, c} may have a non-integer value, the pixel value I _{p, c} is obtained by interpolation processing or the like using the pixel values of neighboring pixels. As the interpolation processing, linear interpolation, cubic interpolation, or the like can be used.

Then, for all monochrome camera determines whether the calculation of the pixel values corresponding to the position X _p has been completed (step S808). If there is an unprocessed monochrome camera, the process advances to step S809 to change the monochrome camera c. Thereafter, the processes in steps S806 and S807 are repeated. When the calculation of the pixel value I _{p, c} is completed for all the monochrome cameras, the process proceeds to step S810, and the evaluation value E _{p, z} for determining the three-dimensional data is calculated. The evaluation value E _{p, z} is obtained by Equation 9.

Here, std () is a function for obtaining a standard deviation value. In Expression 9, the standard deviation values of the pixel values I _{p, c} corresponding to all the monochrome cameras included in the imaging device are calculated as the evaluation value of the reference plane at the height z.

Next, it is determined whether or not the evaluation value E _{p, z} has been calculated for all heights z defined in advance (step S811). If there is an unprocessed height, the height of the reference surface is set. Update (step S812) and repeat the processing from step S804 to step S811.

Step 813 is a step of determining the height at the pixel _Pout using the evaluation value E _{p, z} . Of the evaluation values E _{p, z} at each height z, the height z indicating the minimum value is defined as the height at the pixel _Pout . That is, the corresponding pixel value is calculated on the sensor of each monochrome camera, and the height of the reference plane that minimizes the standard deviation of these pixel values is defined as the height of the pixel _Pout .

  Next, it is determined whether or not the height has been determined for all the pixels on the virtual sensor of the reference camera (step S814). If there is an unprocessed pixel, the pixel position is updated (step S815), and step S803 is performed. To step S814 are repeated.

  When the height is determined for all the pixels of the reference camera, it is output as three-dimensional data together with the pixel position (step S816). This completes the calculation process of the three-dimensional data.

Note that the evaluation value E _{p, z} derived in the process of calculating the three-dimensional data shown in FIG. 8 is a standard deviation value of the pixel value I _{p, c} , but other evaluation values may be used. As an example, the pixel frequency I _{p, c} may be the maximum frequency when the histogram is used. In the process of calculating the three-dimensional data, all the monochrome imaging units included in the imaging apparatus are referred to, but a configuration that refers to a part of the monochrome imaging units included may also be used.

  In the first embodiment, the multi-eye imaging device in which the color imaging unit and the monochrome imaging unit are arranged as shown in FIG. 1 is mentioned. However, the arrangement and the number of the color imaging unit and the monochrome imaging unit are not limited to this. Absent.

  For example, as indicated by reference numeral 901 in FIG. 9, the number of monochrome imaging units used for calculating the three-dimensional data may be further increased as compared with the configuration in FIG. 1, and the number of color imaging units may be reduced. Thus, when the number of monochrome imaging units is increased, more accurate calculation of three-dimensional data becomes possible. Also, as indicated by reference numeral 902, the number of monochrome imaging units may be reduced. Furthermore, as indicated by reference numeral 903, there may be a configuration in which the monochrome imaging unit is disposed only in the peripheral part of the left and right end portions of the imaging unit.

  FIG. 10 shows an example of an imaging apparatus having a configuration in which the number of imaging units is reduced. As long as the color imaging unit and the monochrome imaging unit are mixed, the number and arrangement of the imaging units are not limited to the configuration shown in FIG. For example, as indicated by reference numerals 1002 and 1003, the number of color imaging units may be reduced to one. As described above, since the monochrome image is used when generating the three-dimensional data, it is preferable to set the interval between the monochrome imaging units wider than the interval between the color imaging units. By setting a wide interval between the monochrome imaging units, it is possible to obtain more accurate three-dimensional data.

  In the present embodiment, as shown in FIG. 3, the number of sensor pixels (resolution) provided in the color imaging unit is the same as the number of sensor pixels provided in the monochrome imaging unit. The number of sensor pixels in the part may not match the number of sensor pixels in the monochrome imaging part. The three-dimensional data calculation process according to the present invention is configured such that the three-dimensional data calculation process is established even in such a case. A configuration in which the number of sensor pixels of the monochrome imaging unit is relatively reduced may be used. By relatively reducing the number of sensor pixels of the monochrome imaging unit, the housing size of the imaging device can be reduced.

  FIG. 11 shows an example in which the number of sensor pixels of the monochrome imaging unit is set to ¼ of the number of sensor pixels of the color imaging unit. Reference numeral 1101 denotes a sensor of the color imaging unit and a color filter arranged on the sensor surface. R represents a filter that transmits the red region, G represents a filter that transmits the green region, and B represents a filter that transmits the blue region. Reference numeral 1102 denotes a sensor of the monochrome imaging unit, and Y denotes each pixel. In FIG. 11, one pixel of the monochrome imaging unit corresponds to four pixels of the color imaging unit. In addition, one pixel of the monochrome imaging unit may correspond to 9 pixels of the color imaging unit.

  As described above, according to the present invention, high-resolution color tertiary is obtained by combining three-dimensional data generated from a monochrome image captured by a monochrome imaging unit and a color image captured by a color imaging unit. Original data can be obtained.

(Other embodiments)
This can also be realized by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments (for example, functions for executing the procedures shown in the flowcharts of FIGS. 5 and 8) to the system or apparatus. In this case, the above-described functions of the present invention can be realized by reading and executing the program code stored in the storage medium so that the computer of the system or apparatus can read it.

Claims (7)

An imaging unit for imaging a color image;
A plurality of imaging units that capture monochrome images;
A three-dimensional data calculation unit that calculates three-dimensional data of a subject based on monochrome images obtained by a plurality of imaging units that capture the monochrome images;
A synthesis unit that generates color three-dimensional image data in which a color image obtained by an imaging unit that captures the color image is associated with the three-dimensional data;
A multi-lens imaging device.   The imaging apparatus according to claim 1, wherein the plurality of imaging units that capture the monochrome image are disposed outside an imaging unit that captures the color image.   The number of pixels of an image sensor included in the plurality of image capturing units that capture the monochrome image is smaller than the number of pixels of the image sensor included in the image capturing unit that captures the color image. 2. The imaging device according to 2. An image processing method for generating color three-dimensional image data from an image obtained by a multi-view imaging device having an imaging unit that captures a color image and a plurality of imaging units that capture a monochrome image,
3D data calculation step for calculating 3D data of the subject based on monochrome images obtained by a plurality of imaging units that capture the monochrome images;
A color image obtained by an imaging unit that images the color image and the 3D data obtained in the 3D data calculation step are associated with each other to generate color 3D image data,
An image processing method comprising:   The image processing method according to claim 4, wherein the three-dimensional data is determined based on camera parameters of the multi-view imaging apparatus.   6. The camera parameter according to claim 5, wherein the camera parameter is at least one of a relative position, a sensor size, a number of sensor pixels, a sensor pitch, and a field angle of each imaging unit of the multi-lens imaging device. Image processing method.   A program for causing a computer to execute the image processing method according to any one of claims 4 to 6.

Images