JP2009211254A - Image processing unit, compound eye imaging device, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing unit generating a highly precise reconfigured image by suppressing deterioration in resolution of a reconfigured image due to defocusing included in a pickup image. <P>SOLUTION: An image processing part 4 detects a parallax between a plurality of eye images of an object whose image has been picked up by an image pickup part 3, and detects a surface shape of the object by using the parallax between the detected eye images. Also, the image processing part 4 divides the surface of the object into small regions based on the detected surface shape of the object, and detects the color of each of the divided small regions of the surface of the object, and generates an intermediate image. Further, the image processing part 4 estimates the degree of the influence of each small region of the surface of the object on each pixel of an image sensor 23, that is, the degree of the influence on each pixel due to defocusing by light beam tracking based on the previously detected surface shape of the object, and generates a reconfigured image based on the estimation result. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing unit that generates a high-definition reconstructed image of a captured image of a subject using three-dimensional information of the subject, and a compound eye imaging device to which the image processing unit is applied. The present invention also relates to an image processing method and an image processing program for generating a reconstructed image in which a captured image of a subject is refined using three-dimensional information of the subject.

  Conventionally, there has been proposed an image enhancement method for generating a high-definition reconstructed image using a captured image of a subject and three-dimensional information (three-dimensional shape data) of the subject (Patent Documents 1 and 2). Etc.). The three-dimensional information of the subject can be acquired from, for example, parallax information of a plurality of captured images obtained by capturing the subject from different directions. Examples of image processing methods for increasing the definition of a captured image of a subject include a pixel rearrangement method and an IBP (Iterative Back Projection) method.

Further, as shown in Patent Document 2, in a compound eye imaging device using a lens array in which a plurality of lenses are arranged, the captured image for each lens has a parallax. Three-dimensional information can be obtained. In addition, since the thickness of each lens of the lens array can be suppressed to several mm, the imaging optical system can be reduced in size. For this reason, the use for the visual device of a micro robot, a stereoscopic endoscope, etc. is considered.
JP-A-5-344422 JP 2007-74079 A

  However, the imaging optical system of a general compound-eye imaging apparatus is a pan focus that can be focused from a short distance to a long distance, but defocusing (focal blur) occurs when an image is taken close to the subject. Further, the above-described pixel rearrangement method, IBP method, and the like are image processing methods that do not consider the influence of defocus that occurs in the captured image. For this reason, when defocusing occurs in the captured image, a reconstructed image with sufficient resolution cannot be generated.

  In particular, when used for a visual device of a micro robot, a stereoscopic endoscope, or the like, even when a captured image taken close to a subject is processed frequently and defocused in the captured image, Development of a technique capable of generating a reconstructed image with sufficient resolution is desired.

  An object of the present invention is to provide an image processing unit, an image processing method, and an image processing program capable of generating a reconstructed image having a sufficient resolution even when a defocused image occurs. It is in.

  It is another object of the present invention to provide a compound eye imaging apparatus that can obtain a high-definition reconstructed image for a subject regardless of the distance to the subject by applying the image processing unit.

  The image processing unit of the present invention is configured as follows in order to solve the above problems and achieve the object.

  This image processing unit generates a reconstructed image from a captured image of a subject projected on an imaging element in which a plurality of pixels are arranged in a matrix, using the three-dimensional information of the subject. Specifically, the degree of influence of each small region obtained by dividing the surface of the subject for each pixel of the image sensor by the ray tracing using the three-dimensional information of the subject projected on the image sensor by the estimation unit Is estimated. For example, for each pixel of the image sensor, the degree of influence of each small region obtained by dividing the surface of the subject is estimated based on the number of light rays incident from the small region. Thereby, an optical system in which defocus occurs can be modeled. In addition, the calculation unit calculates a color for each small region obtained by dividing the surface of the subject based on the pixel value of each pixel of the image sensor and the estimation result of the estimation unit. Then, the reconstructed image generating means generates a reconstructed image based on the color of each small area calculated by the calculating means.

  In this way, the optical system in which defocusing occurs is modeled by ray tracing using the three-dimensional information of the subject, and a reconstructed image is generated based on the model. Therefore, it is possible to generate a reconstructed image in which the influence of defocus included in the captured image is suppressed. For this reason, it is possible to generate a high-definition reconstructed image for a captured image of a subject having a continuous distance change or occlusion.

  The calculation means may calculate the color of each small region in addition to the amount of change in color between adjacent small regions. In this way, the edge of the reconstructed image can be suppressed.

  Note that the edge referred to here is an abrupt change that occurs between surrounding pixels due to noise or the like.

  Further, the image processing unit is applied to a compound eye imaging apparatus having a lens array in which a plurality of lenses are arranged and provided with imaging means for imaging a single eye image of a subject projected by each lens on an imaging device. Thus, it can be suitably used for a visual device of a micro robot, a stereoscopic endoscope, or the like.

  According to the present invention, it is possible to generate a high-definition reconstructed image in which the influence of defocus included in the captured image is suppressed.

  Embodiments of the present invention will be described below.

  FIG. 1 is a block diagram illustrating a configuration of a main part of the compound eye imaging apparatus. The compound eye imaging apparatus 1 includes a control unit 2, an imaging unit 3, an image processing unit 4, and an output unit 5. The compound eye imaging apparatus 1 generates and outputs a high-definition reconstructed image from a compound eye image obtained by imaging a subject. A compound eye image consists of a plurality of single-eye images with low resolution. The control unit 2 controls the operation of each part of the main body of the compound eye imaging device 1.

  FIG. 2 is an exploded perspective view illustrating a configuration of the imaging unit. The imaging unit 3 includes a microlens array 21, an optical partition wall 22, and an image sensor 23. The microlens array 21 has a plurality of microlenses 21a. FIG. 2 shows a microlens array 21 in which a total of nine microlenses 21a are arranged in a 3 × 3 matrix. The image sensor 23 is disposed to face the microlens array 21. An optical partition 22 is disposed between the microlens array 21 and the image sensor 23. The optical partition 22 divides the area of the image sensor 23 onto which the captured image of the subject is projected for each micro lens 21 a of the micro lens array 21. Specifically, the optical partition 22 divides the entire region of the image sensor 23 into nine single-eye image regions onto which the captured image of the subject is projected by each microlens 21a. For example, when the image sensor 23 is a 1280 × 1024 pixel CMOS image sensor, the optical partition 22 divides the image sensor 23 into nine single-eye image regions of 340 × 260 pixels. Further, the optical partition 22 shields the projection light of the subject from the microlens 21a that does not correspond to the single-eye image region for each single-eye image region. Therefore, the influence of the micro lens 21a which does not correspond is suppressed in each single eye image area.

  The image sensor 23 is provided with a Bayer color filter (not shown). A captured image (single-eye image) of a subject projected by the corresponding microlens is projected on each single-eye image region. Here, a plurality of single-eye images projected on the image sensor 23 are collectively referred to as a compound eye image.

  In addition, since each microlens 21a of the microlens array 21 is physically different in position, there is a parallax between the nine single-eye images constituting the compound eye image.

  The image processing unit 4 detects parallax between a plurality of single-eye images of the subject imaged by the imaging unit 3, and uses the detected parallax between the single-eye images to detect the surface shape of the subject (three-dimensional information of the subject). ) Is detected. Further, the image processing unit 4 divides the subject surface into small regions based on the detected surface shape of the subject, detects the color of the small region for each small region of the subject surface divided here, and generates an intermediate image. Generate. Furthermore, the degree of influence of each small area on the subject surface is estimated for each pixel of the image sensor 23 by ray tracing based on the surface shape of the subject detected earlier. That is, for each pixel of the image sensor 23, the extraction of the small area on the subject surface that affects and the degree of influence of the extracted small area are estimated. Then, a reconstructed image is generated based on the estimation result. The image processing unit 4 corresponds to the image processing unit referred to in the present invention. The image processing unit 4 is composed of, for example, an image processing chip.

  The output unit 5 outputs the reconstructed image generated by the image processing unit 4. For example, the output unit 5 has an interface connected to an external device, and outputs the reconstructed image generated by the image processing unit 4 to a connected external device (display device, printing device, or the like). In addition, the display device may be configured to display the reconstructed image generated by the image processing unit 4 on the display device.

  Next, the operation of the compound eye imaging apparatus 1 will be described in detail.

  FIG. 3 is a flowchart showing the operation of the compound-eye image generation apparatus. When a shutter (not shown) is operated, the compound-eye imaging device 1 captures a compound-eye image projected on the image sensor 23 of the imaging unit 3 (s1). In s1, a plurality of single-eye images projected on the image sensor 23 are captured as compound eye images by a known rolling shutter system.

  The image processing unit 4 detects the parallax between the single-eye images included in the compound eye image captured in s1 (s2). The image processing unit 4 measures the distance to the subject based on the parallax between the single-eye images detected in s2 (s3). In s3, for example, the distance to the subject may be measured by a known multi-baseline stereo method. The image processing unit 4 generates three-dimensional information of the subject surface based on the measurement result of s3 (s4). In s4, based on the distance measured in s3, surface shape data in which the surface of the subject is constituted by a set of small triangular regions is generated.

  The image processing unit 4 detects a color for each small triangular region constituting the surface of the subject in the surface shape data generated in s4 (s5). In s5, the color of each triangular small area constituting the surface of the subject is detected by a known pixel rearrangement method using the compound eye image captured in s1 (see FIG. 4). The image processing unit 4 gives the color detected in s5 to each triangular small area constituting the surface of the subject, and generates an intermediate image obtained by projective transformation of the color (s6).

  In the above description, the color is detected for each small triangular area constituting the surface of the subject in the surface shape data generated in s4. However, for each quadrangular area obtained by combining two adjacent small triangular areas. Alternatively, the color may be detected. In this way, the processing amount for s5 can be suppressed.

  The intermediate image generated in s6 is a reconstructed image generated by a conventional general compound eye imaging device. As is clear from the above description, this intermediate image is an image that does not take into account defocusing that has occurred in the single-eye image constituting the compound-eye image captured in s1.

If defocusing occurs, the image processing unit 4 is affected by the defocusing. Further, the image processing unit 4 performs a defocusing correction process for correcting the influence of defocusing on the intermediate image generated in s6 (s7). In the defocus correction processing according to s7, the relationship between the light from the subject passing through the microlens 21a and reaching each pixel of the image sensor 23 is modeled by sufficient ray tracing, and based on the model obtained here, This is processing for correcting the intermediate image generated in s6.

  Specifically, for each pixel of the image sensor 23, each incident light ray is back-tracked, and it is detected from which small region of the triangle that constitutes the surface of the subject the light ray. This ray tracing may be performed using a three-dimensional rendering library such as OpenGL.

Here, the pixel value output by each pixel of the image sensor 23 is the sum of the light intensities of the light rays incident on the pixel. The light intensity of the light beam is determined by the color (luminance) of a small area on the surface of the subject that emits the light beam. Therefore, the position of each small area on the surface of the subject is represented by (x, y) as shown in FIG. 5A, and the position of each pixel of the image sensor 23 is shown in FIG. 5B. In terms of (a, b), the output pixel value I _{(a, b)} of the pixel i (a, b) of the image sensor 23 is the luminance R _{(x, y} ) of the small area (x, y) on the surface of the subject. ₎

Can be expressed as In [Equation 1], [k _{(a, b) (x, y)} ] indicates that the luminance R _{(x, y)} of the small region (x, y) is the output pixel value of the pixel i (a, b). It is a coefficient indicating the effect on I _{(a, b)} . Here, the number of light rays incident on the pixel i (a, b) is N, and the number of light rays incident on the pixel i (a, b) from the small region (x, y) is [n _{(a, b) ( x, y)} ],
[K _{(a, b) (x, y)} ] = [n _{(a, b) (x, y)} ] / N
Defined. This [k _{(a, b) (x, y)} ] is obtained by the modeling by ray tracing described above.

Here, for all the pixels of the image sensor 23, a set of R _{(x, y)} that satisfies the above-described [Equation 1] is an ideal value. However, because of the influence of noise or the like, in most cases, there is rarely a set of R _{(x, y)} that satisfies [Equation 1] for all the pixels of the image sensor 23.

  Therefore, the image processing unit 4 subtracts the right side from the left side in [Equation 1] and squares the sum of squared errors for all pixels, e (R)

A pair of R _{(x, y)} that minimizes is calculated. Then, the image processing unit 4 generates an image based on the set of R _{(x, y)} obtained here as a reconstructed image subjected to defocus correction (s8).

  Further, in [Equation 2], there is no limit on the amount of change in luminance between pixels adjacent vertically and horizontally, and therefore an edge due to noise or the like may appear in the reconstructed image. The edge referred to here is an abrupt change that occurs between neighboring pixels due to noise or the like.

Therefore, the above [Equation 2] is

Alternatively, the amount of change in luminance between adjacent pixels in the vertical and horizontal directions may be limited. C in [Formula 3] acts as a coefficient indicating the degree of smoothness of the reconstructed image. As C is reduced, the reconstructed image becomes sharper, but an edge due to noise or the like tends to appear.

  Furthermore, the above [Equation 2] or [Equation 3] is set so that the amount of change in luminance is limited not only between pixels adjacent vertically and horizontally but also between pixels adjacent obliquely.

May be replaced.

In addition, what is necessary is just to obtain | require the group of R _{(x, y) by} a well-known conjugate gradient method etc., even if it is any of the above-mentioned [Formula 2], [Formula 3], or [Formula 4]. Further, the intermediate image generated in s6 may be used as the initial value at this time.

  In this way, the optical system in which defocusing occurs is modeled by ray tracing using the three-dimensional information of the subject, and a reconstructed image is generated based on the model. Therefore, it is possible to generate a reconstructed image in which the influence of defocus included in the captured image is suppressed. For this reason, it is possible to generate a high-definition reconstructed image for a captured image of a subject having a continuous distance change or occlusion.

  The compound-eye imaging device 1 outputs the reconstructed image generated by performing the defocus correction process in s7 (s8), and ends this process.

  Next, a process of generating a reconstructed image from a compound eye image captured by the compound eye imaging apparatus 1 was performed using the above [Equation 4], and the result will be described. FIG. 6 is a diagram illustrating an imaging environment of a subject. The subject is a painting shown in FIG. As shown in FIG. 6 (B), this painting was arranged in a state inclined by 60 degrees with respect to the lens array 21. The size of the subject is vertical × horizontal = 10.24 mm × 10.24 mm.

  FIG. 7 is a compound eye image captured in the imaging environment shown in FIG. FIG. 8 shows the intermediate image generated in s6. 9 and 10 are reconstructed images that have been subjected to the defocus correction process in s7. FIG. 9 shows a case where C in [Equation 4] is 0.01, and FIG. 10 shows a case where C in [Equation 4] is 0.1.

  As is apparent from FIGS. 8 to 10, it is understood that a high-definition reconstructed image can be obtained by performing the defocus correction processing according to s7. Also, by increasing C, the edge of the reconstructed image can be suppressed.

  In the above embodiment, the case where the present invention is applied to a compound-eye imaging device has been described as an example. However, even if a monocular imaging device is used, if the three-dimensional information of the captured subject is obtained, the captured image is displayed. High definition can be achieved.

It is a block diagram which shows the structure of the principal part of a compound eye imaging device. It is a disassembled perspective view which shows the structure of an imaging part. It is a flowchart which shows operation | movement of a compound eye image generation apparatus. It is a figure explaining a pixel rearrangement method. It is a figure explaining the position of the small area on the surface of the subject and the position of each pixel of the image sensor. It is a figure which shows the imaging environment of a to-be-photographed object. It is a figure which shows the imaged compound eye image. It is a figure which shows the intermediate image produced | generated by s6. It is a figure which shows the reconstruction image produced | generated by s8. It is a figure which shows the reconstruction image produced | generated by s8.

Explanation of symbols

1-compound eye imaging device 2-control unit 3-imaging unit 4-image processing unit 5-output unit 21-microlens array 21a-microlens 22-optical partition wall 23-image sensor

Claims (6)

In an image processing unit that generates a reconstructed image from a captured image of a subject projected on an imaging element in which a plurality of pixels are arranged in a matrix, using the three-dimensional information of the subject.
Estimating means for estimating the degree of influence of each small area obtained by dividing the surface of the subject for each pixel of the image sensor by ray tracing using the three-dimensional information of the subject projected on the image sensor;
For each small region obtained by dividing the surface of the subject, a calculation unit that calculates a color based on the pixel value of each pixel of the image sensor and the estimation result of the estimation unit;
An image processing unit comprising: a reconstructed image generating unit that generates a reconstructed image based on the color of each small region calculated by the calculating unit.   The estimation means is means for estimating, for each pixel of the image sensor, the degree of influence of each small area obtained by dividing the surface of the subject based on the number of light rays incident from the small area. 2. The image processing unit according to 1.   The image processing unit according to claim 1, wherein the calculation unit is a unit that calculates the color of each small region in addition to the amount of change in color between adjacent small regions. In a compound eye imaging device having a lens array in which a plurality of lenses are arranged, and including an imaging unit that images a single-eye image of a subject projected by each lens on an imaging device,
The image processing unit according to claim 1,
The image processing unit includes a three-dimensional information acquisition unit that acquires three-dimensional information of a subject using a plurality of single-eye images captured by the imaging unit.
The estimation unit estimates a degree of influence of each small region obtained by dividing the surface of the subject for each pixel of the image sensor by ray tracing using the three-dimensional information of the subject acquired by the three-dimensional information acquisition unit. A compound eye imaging device. In an image processing method for generating a reconstructed image from a captured image of a subject projected on an imaging device in which a plurality of pixels are arranged in a matrix, using the three-dimensional information of the subject,

An estimation step for estimating the degree of influence of each small area obtained by dividing the surface of the subject for each pixel of the image sensor by ray tracing using the three-dimensional information of the subject projected on the image sensor;
For each small area obtained by dividing the surface of the subject, a calculation step for calculating a color based on the pixel value of each pixel of the image sensor and the estimation result of the estimation step;
An image processing method comprising: a reconstructed image generating step for generating a reconstructed image based on the color of each small region calculated by the calculating step. In an image processing program for causing a computer to execute a process of generating a reconstructed image from a captured image of a subject projected on an imaging device in which a plurality of pixels are arranged in a matrix, using the three-dimensional information of the subject.
Computer
Estimating means for estimating the degree of influence of each small area obtained by dividing the surface of the subject for each pixel of the image sensor by ray tracing using the three-dimensional information of the subject projected on the image sensor;
For each small region obtained by dividing the surface of the subject, a calculation unit that calculates a color based on the pixel value of each pixel of the image sensor and the estimation result of the estimation unit;
An image processing program that functions as a reconstructed image generating unit that generates a reconstructed image based on the color of each small region calculated by the calculating unit.

Images