JP2013026844A - Image generation method and device, program, recording medium, and electronic camera - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a blurred image in such a way that a region which is out of focus will be prettily blurred.SOLUTION: An image processing unit 25 comprises a blurring processing part 36, a focus stack image calculation part 37, a PSF kernel calculation part 38, and an image sharpening part 39. The blurring processing part 36 executes convolution processing on a plurality of multifocus images by using a circular opening of blurring radius proportional from a desired focus position or a Gaussian or any other blurring kernel. The focus stack image calculation part 37 calculates from the plurality of images processed by the blurring processing part 36 an intensity averaged image or a focus stack image as the sum of intensity averages. The image sharpening part 39 applies deconvolution processing to the focus stack image on the basis of a predetermined focus stack PSF kernel to generate a blurred image.

Description

  The present invention relates to an image generation method and apparatus, a program, a recording medium, and an electronic camera for generating an image including an intended blur based on a plurality of image groups having different viewpoints.

  Conventionally, an image obtained by imaging an object is arbitrarily adjusted by adjusting sharpness for each depth (focus adjustment) by substituting multiple images with different depths of focus (multi-focus images) into an identity equation based on a superposition model. A technique for generating a focus image or the like has been proposed (Patent Document 1). In the invention described in Patent Document 1, in order to convert the blur state, the blur state is converted for each multifocal image, and then the intensity sum of the blur image obtained by converting the plurality of blur states is obtained.

US Pat. No. 6,571,023

  However, in this method, when the blur state is changed for each image in the multifocal image group, a large conversion noise is generated. Therefore, an out-of-focus area becomes dirty and a good image cannot be obtained. In order to improve it, it has also been proposed to perform iterative calculation, but it was insufficient.

The present invention has been made in view of the above circumstances, and an image generation method and apparatus, a program, a recording medium, and an electronic camera that can generate an arbitrary blurred image so that an out-of-focus area is beautifully blurred. The purpose is to provide.
For the purpose.

  According to one aspect of the image generation apparatus illustrating the present invention, a plurality of in-focus images that have been input are each subjected to blur processing set in advance at an arbitrary focus position, and the plurality of images subjected to the blur processing are performed. A focus stack image which is an intensity sum image is calculated, and an image sharpening process (deconvolution) is performed on the calculated focus stack image to generate an arbitrary blurred image.

  According to the present invention, each of a plurality of in-focus images is subjected to blur processing set in advance at an arbitrary focus position, and a focus stack image that is an intensity sum image of the plurality of images subjected to the blur processing is obtained. Since the calculated focus stack image is calculated and an image sharpening process is performed to generate an arbitrary blurred image, it is possible to obtain a good blurred image in which the out-of-focus area is beautiful while suppressing the generation of noise.

It is a front side perspective view which shows the multi-lens camera which employ | adopted this invention. It is explanatory drawing which shows the numerical sequence of a pseudorandom series of M series. It is explanatory drawing which shows the two-dimensional arrangement | sequence of the imaging opening based on M series pseudo-random number sequence "110100" by a x, y direction point sequence, and has shown the position of the imaging opening by "(circle)". It is a back side perspective view of a multi-view camera. It is a block diagram explaining the electrical structure of a multi-view camera. It is explanatory drawing explaining the state which convolved image point p1, p2 with PSF, respectively. When the focal length is 5.34 mm, F1.4, and the image height × 2 = 3.17 mm, the image is formed in the optical axis direction z of the image space x, y, z when the image is formed from a short distance of 530 mm to infinity. It is explanatory drawing explaining a position. It is explanatory drawing explaining the outline of focusing by the synthetic aperture method using a some camera (cam). It is explanatory drawing explaining the amount of parallax which generate | occur | produces by arrangement | positioning of cam2 and cam4 located in a line in the x or y direction demonstrated in FIG. 8 with two straight lines. It is explanatory drawing explaining the state by which the bokeh kernel is convolved with the object which has a focus with parallax z = 4 with respect to 5x5 eye camera arrangement | positioning. It is explanatory drawing explaining the state in which the blur kernel is convolved with the object which is in focus by the blur kernel z = 4 of arbitrary blur amounts. It is explanatory drawing which represented the blur kernel demonstrated in FIG. 10 in x, y plane. FIG. 11 is an explanatory diagram illustrating the blur kernel (parallax z = 0 to z = 4) described in FIG. 10 in the x and y planes in the case of an M-series camera arrangement. FIG. 11 is an explanatory diagram illustrating the blur kernel (parallax z = 4 to z = 8) described in FIG. 10 in the x and y planes in the case of an M-series camera arrangement. It is explanatory drawing which shows the focus stack kernel which integrated the blur kernel in the parallax direction (z direction) in the focus position. It is explanatory drawing explaining the flow of an image process from the image capture of a multi-lens camera. It is explanatory drawing explaining the arbitrary blur algorithm which an image process performs. It is explanatory drawing which shows the several image taken in changing focus. It is an explanatory view showing a plurality of images captured by changing the focus and explaining that a convolution result between an object and an optical system can be obtained (explaining the equation described in [Equation 15]). It is explanatory drawing explaining the formula as described in [Formula 16]. FIG. 10 is an explanatory diagram in which an arbitrary defocus kernel is convolved in each x, y plane with a plurality of images (images obtained by a plurality of focuses) captured by changing the focus. It is a block diagram which shows the structure of the image process part which performs an arbitrary blurring algorithm. It is explanatory drawing which shows the focus stack kernel in the case of the multi-lens camera which arranged the imaging lens with the predetermined | prescribed periodic pattern according to the M series which is a pseudorandom series. It is a flowchart explaining the operation | movement procedure of a multiview camera. FIG. 10 is a diagram illustrating a part of the arbitrary blur algorithm, and is an explanatory diagram illustrating the integration range of the expressions described in [Expression 12] and [Expression 19] with respect to the expression described in [Expression 16]. It is explanatory drawing explaining the integration range of the formula as described in [Equation 19]. It is explanatory drawing which shows the outline of the cone of the light of the third order distribution (PSF) of point image intensity distribution. It is a graph which shows the relationship between the radius of the blur kernel of a Gaussian distribution, and a standard deviation. It is a graph which shows the relationship between a synthetic | combination blur radius and a standard deviation. An image taken with a general camera is shown. The blur image produced | generated using the conventional synthetic | combination synthetic aperture method is shown. A blur image obtained by designating a blur position corresponding to a focal position of 50 mm and a diaphragm F1 is shown. A blurred image obtained when a blur position corresponding to a focal position of 50 mm and a diaphragm F1.4 is designated. A blur image obtained when the blur position corresponding to the focal position of 50 mm and the stop F2 is designated is shown. A blurred image obtained when a blur position corresponding to a focal position of 50 mm and a diaphragm F4 is designated is shown. A blur image obtained when a blur position corresponding to a focal position of 50 mm and a diaphragm F42 is designated is shown. The blur image generated using the camera arrangement described in [Formula 3] is shown, and is an image obtained by designating the blur position corresponding to the focal position 50 mm and the aperture F1. 37 is an image obtained by designating a blur size corresponding to a focal position of 50 mm and a diaphragm F1.4 under the same conditions as described in FIG. 37 is an image obtained by designating a blur position corresponding to a focal position of 50 mm and a diaphragm F2 under the same conditions as described in FIG. 37 is an image obtained by designating a blur position corresponding to a focal position of 50 mm and a diaphragm F4 under the same conditions as described in FIG. 37 is an image obtained by designating a blur size corresponding to a focal position of 50 mm and a diaphragm F42 under the same conditions as described in FIG. An arbitrary blurred image obtained when the refocus distance is designated as 0.71 m is shown. An arbitrary blurred image obtained when the refocus distance is designated as 0.82 m is shown. An arbitrary blurred image obtained when the refocus distance is specified as 0.97 m is shown. An indoor arbitrary blurred image obtained when the refocus distance is specified as 1.2 m is shown. An indoor arbitrary blurred image obtained when the refocus distance is designated as 1.5 m is shown. An indoor arbitrary blurred image obtained when the refocus distance is specified as 2.1 m is shown. An indoor blur image obtained when the refocus distance is designated as 3.6 m is shown. An indoor arbitrary blurred image obtained when the refocus distance is designated as 10.7 m is shown. An outdoor blurred image obtained when the refocus distance is specified as 0.97 m is shown. An arbitrary outdoor blur image obtained when the refocus distance is specified as 1.2 m is shown. An outdoor blur image obtained when the refocus distance is designated as 1.5 m is shown. An arbitrary outdoor blur image obtained when the refocus distance is specified as 2.1 m is shown. An arbitrary outdoor blurred image obtained when the refocus distance is designated as 3.6 m is shown. An outdoor blur image obtained when the refocus distance is designated as 10.7 m is shown. An intensity average image (focus stack) image obtained by calculating an intensity average for a plurality of multifocal image groups acquired by the synthetic aperture imaging method is shown. 56 shows an all-focus image obtained by deconvolution of the focus stack image shown in FIG. 56 with a predetermined focus stack PSF kernel. An arbitrary viewpoint blurred image obtained by deconvolution with a focus PSF tack kernel with the viewpoint biased obliquely from the center to the lower left with respect to the multifocal image group is shown. An arbitrary viewpoint blurred image obtained by deconvolution with a focus stack PSF kernel with the viewpoint biased from the center to the left with respect to the multifocal image group is shown. An image in which an arbitrary viewpoint blurred image with parallax is generated with a shooting distance of 5.3 m and an inter-camera parallax of 0 pixel is shown. An image in which an arbitrary viewpoint blurred image with parallax is generated by specifying a shooting distance of 3.6 m and an inter-camera parallax of 2 pixels is shown. An image is shown in which an arbitrary viewpoint blurred image with parallax is generated by specifying a shooting distance of 2.1 m and an inter-camera parallax of 3 pixels. An image obtained by generating an arbitrary viewpoint blurred image with parallax with a shooting distance of 1.5 m and an inter-camera parallax of 5 pixels is shown. An image obtained by generating an arbitrary viewpoint blurred image with parallax with a shooting distance of 1.2 m and an inter-camera parallax of 7 pixels is shown. An image obtained by generating an arbitrary viewpoint blurred image with parallax with a shooting distance of 0.97 m and an inter-camera parallax of 9 pixels is shown. An image in which an arbitrary viewpoint blurred image with parallax is generated with a shooting distance of 0.89 m and an inter-camera parallax of 10 pixels is shown. An image obtained by generating an arbitrary viewpoint blurred image with parallax with a shooting distance of 0.82 m and an inter-camera parallax of 11 pixels is shown. An image in which an arbitrary viewpoint blurred image with parallax is generated with a shooting distance of 0.71 m and an inter-camera parallax of 13 pixels is shown. An image obtained by generating an arbitrary viewpoint blurred image with parallax with a shooting distance of 0.63 m and an inter-camera parallax of 15 pixels is shown. It is explanatory drawing explaining the synthesis | combination of an image by a synthetic aperture method. It is explanatory drawing which shows the image acquired by focusing on a building by the synthetic aperture method. FIG. 72 is an explanatory diagram in which a distant view region of the image described in FIG. 71 is disassembled. FIG. 72 is an explanatory diagram in which a region of a middle-distance subject in the image described in FIG. 71 is disassembled. FIG. 72 is an explanatory diagram in which a region of a short-distance subject in the image described in FIG. 71 is disassembled. It is a block diagram which shows other embodiment which produces | generates a focus image. It is a block diagram which shows other embodiment which produces | generates a blurred image. FIG. 10 is a block diagram illustrating another embodiment in which an image is divided into regions based on depth map information and blur processing is performed. FIG. 78 is a flowchart showing an operation procedure of the apparatus described in FIG. 77.

  As shown in FIG. 1, the multi-lens camera 10 has 16 shooting openings 11 provided on the front surface of the camera body 12. A photographing lens and an imaging element are arranged in the back of each photographing opening 11, and the photographing opening 11, the photographing lens, the imaging element, and the like constitute a single-eye imaging unit serving as a camera unit.

  As each image sensor, for example, an image sensor having low pixels and low power consumption is used. Further, the optical axes of the photographing lenses are substantially parallel. Each imaging element is arranged so that the imaging surface is perpendicular to the optical axis of the imaging lens.

  The multi-lens camera is provided with a shutter button 13 and a power switch 14 on the upper surface of the camera body 12, and simultaneously acquires 16 pieces of image data with different viewpoints by one shutter release, and these image data are parameterized. Depending on the image processing, sharpness adjustment (focus adjustment) for each depth, that is, an arbitrary focus image generated by intentionally blurring a specific subject in the screen, the intensity and arrangement of the blur (Bokeh generation image generated by adjusting (size, shape, weighting of blur), and an arbitrary focal point of an arbitrary viewpoint for stereoscopic viewing with a viewpoint generated by combining the processing of the blurred generation image and the arbitrary focal point image, It is also possible to generate an arbitrary blur generation image. That is, it is possible to configure a camera that can freely select three parameters: (1) “focal position in the optical axis direction”, (2) “blur size”, and (3) “viewpoint”. Further, by using a plurality of cameras, it is possible to increase the number of pixels, and it is also possible to generate high-resolution image data, that is, image data with multiple pixels.

  The imaging openings 11 are arranged in a two-dimensional manner. The intervals in the x direction and the y direction of the imaging aperture 11 are set to a predetermined periodic pattern according to the M sequence that is a pseudo-random sequence. In this case, the autocorrelation function of the M sequence is close to the delta (δ) function, and has a feature that the correlation function value is constant except for the peak. Since the autocorrelation function is a delta function, the power spectrum is flat (constant without depending on frequency). This is a desirable property because a specific frequency component is not emphasized when generating blur. According to the experiment, when a circular aperture was used for the blur kernel (blurring function), the blurry rectangular arrangement and the grid arrangement became blurred. However, it was found that when a Gaussian function is used for the blur kernel, a beautiful blur can be obtained without depending much on the camera arrangement. When a Gaussian function is used for the blur kernel, beautiful blur is obtained without much depending on the camera arrangement, so the smaller the distance between adjacent cameras, the smaller the distance between the blurs that appears discretely, and the more beautiful the blur is. If the cameras are arranged on the circumference at equal intervals or at random intervals, there is a possibility that the optimum blur will be obtained. A grid-arranged camera, for example, expressed as a formula described in [Equation 1], calculates 16 patterns by rotating 90 eyes by 90 degrees as an array of 5 × 5 arrays (described in [Equation 1]). The left term of the equation of), take these intensity average images. In terms of weighting, this is a weighting of “0.018 to 1” as shown in the right term of the equation described in [Formula 1]. The camera arrangement represented by the equation described in [Equation 1] is equivalent to weighting for each camera, but it has been found that even if the weight is set to Gaussian, the blur is beautiful. Since it is desirable that the weighting of the cameras be equal in the processing when the number of pixels is increased, for example, the number of pixels is increased by a four-pattern camera arrangement as shown in [Equation 1], and finally the number of pixels is increased once again. By making the number of pixels five times for convenience, the weight of the camera can be a Gaussian weight with a high center. Here, the “camera” includes individual imaging units that are built into a multi-lens camera and that include a photographing lens and an imaging element.

If the number of pixels is not considered, it is easy to set the weight directly to Gaussian. In the case of Gaussian with camera interval = 1σ (standard deviation), it is expressed by the equation shown in [Expression 2].

The M-sequence pseudorandom number sequence of the sequence Xi (i = 1 to m, where m is a cycle) is generally “0” having a cycle m of “2n−1” for an integer n of “1” or more. And “1” in a binary number sequence, the number of the number sequences in one period m is an odd number, and the difference between the number of “0” and the number of “1” is “1”. Therefore, combinations (n, m) of the integer n and the period m include (3, 7), (4, 15), (5, 31), (6, 63), and the like.

  As shown in FIG. 2, for example, in a sequence where the integer n is “2” and the period m is “3”, it is “1, 0, 1”, and for example, the integer n is “3” and the period m is “7”. In the numerical sequence of “1, 1, 1, 0, 1, 0, 0”, for example, in the numerical sequence in which the integer n is “4” and the period m is “15”, “1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0 ". In the present embodiment, for example, a numerical sequence “1, 1, 1, 0, 1, 0, 0” having an integer n of “3” and a period of “7” is employed. In this case, a two-dimensional array pattern as shown in FIG. 3 is obtained. In this pattern, the position indicated by “◯” is the position where the photographing aperture 11 is arranged.

  In addition, the method of making this number sequence is, for example, “Correlation Function and Spectrum-Its Measurement and Application” published by Takashi Isobe, published by the University of Tokyo in February 1968 and the second edition in January 1971. 170-175.

  By the way, the interval of the two-dimensional array of the photographing apertures 11 may be set to a predetermined periodic pattern which is a binary pseudo-random sequence such as a Q sequence (square residue sequence), a Gold sequence, a Walsh code, etc. in addition to the M sequence. it can. Further, the intervals in the two-dimensional direction of the imaging openings 11 may be equal. However, since it is desirable that there is no zero point in the pupil sampling points before the focus stack, which will be described in detail later, it is desirable to set the interval in the two-dimensional direction to a predetermined periodic pattern of M series. The number of imaging apertures 11 is not limited to 16 (16 eyes).

  As an application example of the M series described with reference to FIG. 3, in the camera arrangement represented by the equation described in [Equation 3], for example, as an array of 5 × 5 arrays, 16 eyes are rotated by 90 degrees in four patterns. When calculated (the left term of the equation described in [Equation 3]) and taking these intensity average images, it was found that the blur becomes more straightforward. When this is considered in terms of weighting, as shown in the right term of the equation described in [Equation 3], the weighting is from “0.25 to 1”. In this case, it is desirable to increase the number of pixels by separately performing the multi-pixel processing at 90 degrees and finally obtaining the average intensity.

An LCD 15 is arranged behind the camera body 12 as shown in FIG. The LCD 15 forms a touch panel together with the transparent tablet 34. The touch panel displays images such as menu characters and operation buttons overlaid on the live view image. By touch operation, zooming operation, focal plane designation, blur parameter (virtual aperture size, shape, weighting) selection, etc. Enter the parameters.

  As shown in FIG. 5, an AFE 21 and a frame memory 22 are connected to each image sensor 20, and a subject image formed by the photographing lens 23 is captured and an image signal is output to the AFE 21. Here, reference numeral 40 indicates the single-eye imaging unit 40.

  The AFE 21 includes a well-known CDS (correlated double sampling) / AGC (gain control amplifier circuit), A / D, and a signal processing circuit, and operates in synchronization with a pulse supplied from the TG together with the CPU 24. (Not shown). The signal processing circuit captures digital image data and performs corrections such as pixel defect correction, white balance correction, and gamma correction.

  The CPU 24 controls the charge accumulation time (electronic shutter) of the image sensor 20 and measures the luminance of the subject based on the image data obtained from the specific image sensor 20, and all the image sensors based on the measurement result. The exposure is controlled by changing the value of 20 electronic shutters to be constant.

  The AF unit 26 fetches at least two of the plurality of image data in response to the half-pressing operation of the shutter button, and obtains the relative shift amount between the two images based on the two image data. Subject distances for a plurality of subjects are calculated from the shift amounts, and one focusing driver 27 is controlled to move the photographing lens 23 to a focusing position corresponding to the subject distance, and the remaining focusing drivers 27 are controlled. Then, with the in-focus position corresponding to the subject distance as a reference, the in-focus positions of the remaining imaging lenses 23 are sequentially shifted so as to be shifted with a uniform blur amount. Note that the image pickup device 20 may be moved, or both the photographing lens 23 and the image pickup device 20 may be moved to shift the focus.

  As a result of estimating the necessity of AF, it is desirable to use a sensor (imaging device) having an optical size of 1/8 inch or less in order to realize a thin camera having a thickness of 3 mm. This is because if the lens diagonal length is “Dg”, the total optical path length is “Lp”, and a lens with sufficient aberration characteristics is designed, then Lp> 0.9 Dg.

  In the case of a multi-lens camera, the number of pixels can be increased up to several times as many as the camera, so each camera module may have fewer output pixels than the final camera. Although depending on the algorithm for increasing the number of pixels, when an algorithm such as using the pixel data of the nearest sensor is used, the generated image will not fail if the number of pixels is reduced to 0.56 times or less the number of cameras. For example, in the case of a camera with 6M (mega, 1 million) pixel output, it is easy to increase the number of pixels by about 9 times when 16 eyes are used. In this case, the camera module may have any number of pixels that is 6M × 1/9 = 0.67M pixels from 1 / (16 × 0.6) = 1/9 of 6M.

  An optical system for resolving 6M with a sensor size of 1/8 inch image height is required. Specifically, the MTF at the Nyquist wavelength must be at least 10%. In this case, the Nyquist wavelength is 1.169 μm. For this reason, the F number of a lens needs to be 1.4 or less. When the thickness of the 6M output camera is 5 mm, the nominal size of the sensor can be 1/4 inch and the F number of the lens can be 2.8 or less.

  The presence or absence of AF was examined for the above-mentioned 3 mm and 5 mm thick card cameras including lens aberrations. If the camera focus position was set to about 2 m to the subject, the distance to the subject was 1 m. It was found that the change in MTF from infinity to infinity is a change of 50% or less at the Nyquist frequency and can be regarded as pan focus. It is also possible to perform deconvolution so as to compensate for the change in MTF (PSF also changes at the same time) according to the focus position. When refocusing (focusing on an arbitrary subject distance after shooting), it is possible to compensate for the degradation of MTF according to the distance. That is, as shown in FIG. 75, the storage unit 40 that stores PSF or MTF for each focus position in advance, and image processing so as to cancel the change in PSF or MTF due to the focus position when generating a focus image by the synthetic aperture method. And an image processing unit 41 that performs the above. The storage unit 40 and the image processing unit 41 are connected together with the CPU 42 by a bus 43 and are controlled by the CPU 42 in an integrated manner. Reference numeral 44 denotes an image acquisition unit for capturing a plurality of images, such as a multi-lens camera, a scanner, and an external input unit. Even with this configuration, the out-of-focus image is blurred according to the present invention, so that the out-of-focus image is overcorrected and the image does not become unnatural.

  The AF actuator of the camera module may be used when setting the focus at a distance of 2 m to the subject. This is because the focus adjustment on the image side of the 1/8 inch sensor size is severe, and adjustment and stability on the order of several microns are required.

  By the way, the focus of one photographing lens 23 is adjusted to the reference subject distance measured, and the remaining photographing lenses 23 are sequentially shifted in focus so that the amount of blur is uniform with respect to the in-focus position. Going out of focus is difficult to control. Therefore, the AF unit 26 divides the shooting distance range from the shortest shooting distance to infinity by the number of the single-eye imaging units 40, and each focusing driver so that the shooting distance corresponding to each division position is in focus. It is preferable to move each photographic lens 23 individually by controlling 27. In addition, when the focus is not performed after the photographing, the pan focus may be configured by the diaphragm and the photographing lens 23. In this case, the AF unit 26 and the like can be omitted.

  The ROM 28 stores various programs and setting values necessary for executing the programs in advance. The RAM 29 is used as a work memory for the CPU 24 and as a temporary memory for each unit. The CPU 24, ROM 28, RAM 29, LCD driver 33, I / F 30, and image processing unit 25 are connected by a bus 32.

  The photographing lens 23 may be a zoom lens or a focal length switching type lens. In this case, all the photographing lenses 23 may be configured to be zoomed in the same manner in synchronization with the zooming operation.

Various operation signals are input to the CPU 24 from the tablet 34 (not shown). The CPU 24 simultaneously loads a plurality of image data (multifocal image data) having different focal points into each frame memory 22 in response to one release operation. The multifocal image data is individually taken into each frame memory 22 and then output to the image processing unit 25.
<Three-dimensional shift invariant imaging>
Considering the blur due to defocusing in the optical axis direction of the point image PSF formed by the optical system, it is necessary to consider a three-dimensional PSF. An object is also three-dimensional, and can be regarded as shift-invariant imaging within an isoplanatic range, and imaging can be expressed with the same PSF anywhere on the object surface. For example, when the image points p1 and p2 exist in the image space x, y, and z, the state in which the PSF is convolved with respect to these points is the image formation state. In general, an object is uneven, and in image formation, image points corresponding to object points on the surface visible from the camera are convoluted with PSF (see FIG. 6). The three-dimensional PSF shown in FIG. 6 may be called a three-dimensional blur kernel. When obtaining such a three-dimensional imaging state, in the case of normal lens imaging, for example, as shown in FIG. 7, the focal length is 5.34 mm, F1.4, the diagonal imaging range = image height × 2 = 3. In the case of .17 mm, even if an image is formed from a short distance of 530 mm to infinity, the imaging position in the optical axis direction z of the image space x, y, z is 5.34 mm and 5.39 mm, and only a change of 50 μm occurs at most. However, this range can be regarded as an isoplanatic range in the z direction. Since the PSF does not change significantly with respect to x and y as well, a three-dimensional image is obtained by moving and positioning a two-dimensional image sensor in parallel with the image plane I in the normal lens imaging state. It is possible to acquire image state data. It is desirable in terms of signal processing that the data acquisition intervals in the optical axis z direction of the two-dimensional imaging device are equal intervals. Note that z min corresponds to the infinity position. Although it is desirable that z max sufficiently include the latest position from the lens of the subject, there is no problem in a range where refocusing may be practically performed. When z min is focused only in the near view, the data of the two-dimensional image sensor up to infinity is not necessarily required, and the point farthest in the depth direction of the refocus range may be set as z min. Particularly in macro photography, since data in the z direction tends to increase, it is necessary to set appropriate ranges to z min and z max. The imaging magnification is obtained from the formula of the lens imaging formula “1 / f = 1 / Z + 1 / b” where “Z” is the distance from the lens to the subject and “b” is the distance from the lens to the image point. .

<Focusing by the synthetic aperture method>
As shown in FIG. 8, in the focusing by the synthetic aperture method, when focusing on the distance Z, if the distance from the center camera is “s”, the amount of parallax = camera interval × imaging magnification = s × b The image is shifted by / Z so that the center camera image and the image point at the focus position overlap.

When a pinhole is used instead of a lens, the imaging magnification is obtained from the formula “f / Z”. When a pan focus lens is used, the focus position of the lens calculates the change in MTF, and the fixed focus setting distance is determined so that the degradation of the shortest shooting distance and the MTF at infinity is comparable. For example, when the shortest shooting distance is 1 m, the focus position is set to 2 m. The value of b is obtained from the lens imaging formula. ([Formula 7])
In the following description, the unit of parallax is often expressed in pixels. However, in the present invention, the focus image is synthesized by the synthetic aperture method at a subject distance in which the parallax amount is an integral multiple of the pixel pitch of the image sensor, and image processing is performed. This is because it is efficient to perform the image processing, and the image quality is also good as a result of processing. As the pixel pitch, the original value of the image sensor may be used, or the pitch of the image sensor after the number of pixels is increased may be used. In addition, when there are too many focus points, such as during macro shooting, it is not necessary to generate a synthetic aperture image with the parallax amount corresponding to each pixel, and thinning can be performed as appropriate. It is necessary to judge the thinning ratio while viewing the image.

  Consider acquiring shift-invariant three-dimensional imaging state data using the synthetic aperture method. For that purpose, it is desirable to express the optical axis direction z of the image space by the amount of parallax. The reason is as follows.

  Consider the coordinate system of an object on an X, Y, Z linear (equally spaced) scale. The optical axis is taken in the Z-axis direction. A coordinate system (x, y, z) of an image space for focusing by a synthetic aperture method of a multi-lens camera is used. As in the case of a pinhole camera, each camera of a multi-lens camera usually uses a camera with a deep focal depth and can be refocused. Consider a pinhole camera as shown in FIG.

  Each pinhole camera does not have information in the Z direction, and an object at any distance is imaged on an imaging plane located at a focal length f from the pinhole. However, when focusing is performed by the synthetic aperture method, parallax = blurring due to camera arrangement occurs. Focusing with a multi-lens camera will be described with reference to FIG.

  The camera has a 5 × 5 arrangement in the X and Y directions. Since a 25-eye image is taken with a pinhole camera and focusing is performed by the synthetic aperture method, the imaging magnification of the subject is determined at the time of shooting and does not depend on the focusing. The imaging magnification is the magnification of the pinhole camera: f / Z, where Z is the distance to the subject.

  FIG. 8 represents a multi-view camera as a plurality of pinhole cameras. The viewpoint of each camera, that is, the position of the pinhole, is located on the camera plane (pupil conjugate plane) (s-t plane). The stage pupil position (equivalent to the viewpoint position and pinhole position) is indicated by D (s, t). The X, s, and x axes are parallel. The Y axis, t axis, and y axis are parallel. The object plane and the camera plane are parallel and represent a state in which the object plane is in focus at an arbitrary distance Z from the camera plane by the synthetic aperture method. In the figure, the pinhole of the pinhole camera is represented by a triangular apex.

The coordinates of the sensor surface of each camera are assumed to be x _{D (s, t)} , y _{D (s, t)} . The coordinates x _{D (s, t)} and y _{D (s, t)} are values without error after calibration. A point P on the object plane (X, Y) is at a distance Z from the camera plane and forms an image on x ' _{D (s, t)} and y' _{D (s, t)} of each camera D (s, t). To do. This is shown by the equations described in [Equation 4] and [Equation 5]. If cam3 is on the Z axis, x = X × f / Z and y = Y × f / Z.

Here, [Expression 4] according to Equation _{5] f D (s, t} ) is the measured focal _{length, cx D (s, t)} , cy D (s, t) is a constant for calibration error thrust is there.

Let C (x ′ _{D (s, t)} , y ′ _{D (s, t)} ) be the output of the image pixel on the sensor surface. The synthetic aperture output I (x, y, Z) focused on the distance Z can be calculated from the equation shown in [Equation 6].

However, the synthetic aperture output is the synthetic output of any selected camera. For example, 16 of 25 units. Schematically, the state of image synthesis by the synthetic aperture method is as shown in FIG. 70, in which the image is relatively moved in accordance with the focus position to perform focusing (focus adjustment).

  [Expression 4], [Expression 5], and [Expression 6] are the expressions described in the example of the pinhole camera. However, a multi-lens camera with a camera lens is actually used. In this case, the parallax amount is expressed by the equation described in [Equation 7] or [Equation 8].

  It is efficient to acquire an image at an object distance Z at which the parallax amount z is equally spaced, and if the parallax amount z is determined, the distance Z to the subject is expressed by the equation described in [Equation 7] or [Equation 8]. Is uniquely determined by The parallax amount z in the case of a pinhole camera is proportional to the product of camera interval (distance between any two viewpoints on the pupil plane): s0 and imaging magnification: b / Z, and is expressed by [Equation 7]. Therefore, the larger the camera interval, the greater the amount of change in parallax with respect to the subject distance. In a multi-lens arrangement camera, the amount of change in parallax with respect to the subject distance is maximized at the interval of two viewpoints: s max where the distance between the cameras is the longest. Therefore, in the case where the sampling pitch of the parallax amount is acquired most finely, the parallax amount z in which s0 = smax in [Equation 7] is equal to an integer multiple of the pixel pitch of the image sensor, and a plurality of focus points are obtained. You can sample or combine images.

[Equation 7]
Parallax amount (z) = minimum camera interval (s0) × imaging magnification = s0 × b / Z
<About the amount of parallax>
As seen above, the amount of parallax is expressed by the equation described in [Formula 7] in the case of a pinhole camera or a pan focus lens. On the other hand, consider the case of focusing with a monocular lens as shown in FIG. In this case, the entrance pupil of the lens has a certain area, and the distance between any two points in the pupil plane (sp) of the lens is s0. The maximum distance between any two points: s0 is the pupil diameter Ap. The parallax amount z2 is given by the equation shown in [Equation 8] where Z is the distance to the object point and f is the focal length. Also, when focusing on each objective lens in the configuration of the multi-lens camera, it is expressed by the equation described in [Equation 8].

[Equation 8]
Parallax amount (z2) = imaging magnification (m) × distance between any two viewpoints on the pupil plane (s0) = f × s0 / (Z−f)
Note that the pupil plane in FIG. 7 and the camera plane in FIG. 8 are both entrance pupils that define the incident light flux, and play an equivalent role. In addition, when a lens of a multi-lens camera with a synthetic aperture method is extended for each focus position of a subject to perform focusing, the distance between the image plane and the lens: b is a function of Z, so that the parallax is [ It becomes a formula as described in Formula 8]. In general, a synthetic aperture method using a multi-lens camera uses fixed focus and pan focus lenses, and thus the parallax amount z is derived from the equation described in [Equation 7].

  Since the parallax amount z between the two cameras is the amount of image movement when the lens is relatively shifted in parallel, the increase in the parallax amount from the in-focus state is equal to the blur amount. For example, in a state where the focus is set to infinity, a subject at a subject distance corresponding to one pixel of parallax is photographed with a blur radius of one pixel, and a subject at a distance of one pixel of parallax is focused. In total, an object at infinity is blurred with a blur radius of 1 pixel. When calculating the parallax amount z when focusing is performed by changing the position of the imaging surface with a single lens without performing the focusing by the synthetic aperture method, it is particularly closer to use the formula described in [Equation 8]. Although the error is reduced in the calculation of the distance parallax amount z, the equation described in [Expression 7] may be substituted.

  The act of setting the parallax amount z at an arbitrary distance to zero can be called focusing. The action of focusing is to make the image plane coincide with the focus position in a normal lens. In the synthetic aperture method, a plurality of images are shifted by the amount of parallax so that the amount of parallax between the images of a plurality of cameras becomes zero.

In the present invention, since the depth direction of the optical axis is replaced with parallax, in the function representing blur, the magnitude of blur is proportional to the amount of parallax. For example, a circular opening has a conical shape (cone shape).
<About the data acquisition interval>
An image is acquired from a camera in which x and y directions are arranged at equal intervals. This is achieved by sampling with sensors arranged in two dimensions at equal intervals. On the other hand, the parallax amount z is obtained by being divided so that the pixel count (parallax amount / pixel pitch (P)) of parallax generated between the two cameras is linear (equal intervals). In other words, the focus position is set so that the parallax amount z is an integer multiple of the pixel pitch, and images in a plurality of focus states are synthesized by the synthetic aperture method. Thereby, a calculation error can be reduced. The pixel pitch may be a pixel pitch of a sensor or a pixel pitch of an output image output from each camera (imaging unit). In the latter case, it is “1” corresponding to an integer of the pixel pitch of the original sensor in the multi-pixel processing.

<About the distance s0 between two points on the pupil plane>
Here, the number of times of focusing by the synthetic aperture method, that is, the resolution in the focus direction is determined. How to determine the distance s0 between two points on the pupil plane will be described.

  In order to increase the number of data in the focus direction (the number of focus images to be synthesized by the synthetic aperture method), the parallax amount is calculated by using [Equation 7] and [Equation 8] with s0 = smax or s0 = AP. It is only necessary to synthesize a focus image obtained at a distance Z to an object that is an integer multiple of the pixel pitch of the image sensor by a synthetic aperture method. This method is a method of generating a focus image at the most different subject distances Z. However, practically, it has been found that there is no problem even if thinning is performed to about a fraction. In particular, when multi-lens cameras are arranged at equal intervals (including M-sequence arrangement), the parallax amount z is set to [Equation 7] and s0 = interval between adjacent cameras. In this case, it is not necessary to horizontally shift the image of the sub-pixel (a shift amount of one pixel or less), and a quantization error due to the pixel pitch does not occur, thereby preventing blurring due to the pixel pitch. be able to. As a result, the number of data is thinned out. For example, if the number of cameras is about s × t = 5 × 5, the number of data is thinned out to about 1/4 of the maximum number of data, but it has been found that the number is practically sufficient. In macro photography, it was found that there is no problem even if the number of data is reduced to about half.

<About the amount of parallax in a monocular lens>
When using a monocular lens to determine the parallax according to the equation described in [Equation 8] and acquiring image data with multiple focus, the distance between two points on any pupil: s0 is set to about Ap / 4, and the data is thinned out. There is no practical problem even if it is performed.

  Since the value of the parallax amount z is distance: Z = infinity and parallax: z = 0, it is desirable to take the parallax coordinates as shown in FIG. x and y are two-dimensional coordinates in the imaging surface, and z is a parallax amount. In FIG. 9, a multi-view camera arranged in a two-dimensional grid at equal intervals is assumed, and the parallax amount z is set as [Equation 7], where s0 = interval between adjacent cameras. The two diagonal straight lines shown in FIG. 9 are one row in the st plane described in FIG. 8 arranged at equal intervals in a 5 × 5 grid arrangement, and cam2 above and below cam3 at the center of the cameras cam1 to cam5. The parallax generated in the x, y plane or the size of blur (the size of the x, y cross section of the three-dimensional blur kernel) is shown by the arrangement of cam4. It shows that a parallax of ± 8 times the pixel (P) is generated in x or y when the parallax amount z = 8. In this way, by dividing the depth direction (optical axis direction) so that the parallax amount z is equally spaced, the size of the blur kernel (see FIG. 10) or an arbitrary blur kernel (see FIG. 11) depending on the camera arrangement. The change in the length linearly changes in the amount of parallax, and there is no inconvenience that the shape of the three-dimensional blur kernel changes depending on the position of the parallax z-axis along the optical axis. Thereby, shift invariant, that is, data of a three-dimensional imaging state that can be expressed by a three-dimensional convolution can be acquired (formula described in [Equation 17]).

  The maximum distance between any two points in the s and t directions on the pupil plane (camera plane) is the distance of cam1 and cam5 as shown in FIG. 8, and the amount of parallax shown in FIG. 10 and FIG. z is four times the distance between adjacent cameras. That is, since the data in the three-dimensional imaging state has s0 set between adjacent cameras, the number of data is reduced to 1/4 from the maximum number of data. Since this number is sufficient for practical use, there is no problem.

  FIG. 10 shows the three-dimensional shape of the blur kernel and the size of the blur kernel in the x and y cross sections, which are determined by the camera layout (camera plane, pupil center layout in the s-t plane). x and y are two-dimensional coordinates in the imaging surface, and z is a parallax amount. In FIG. 10, a multi-lens camera arranged in a two-dimensional grid at equal intervals is assumed, and the parallax amount z is [Equation 7] and s0 = interval between adjacent cameras. The size of the blur kernel changes linearly according to the amount of parallax. This figure shows a state in which the blur kernel Bc is convoluted with an object in focus with parallax z = 4 (pixels). The x, y cross section of the blur kernel is a two-dimensional distribution function representing blur in the x, y plane (imaging plane), and is also a function of parallax z in the present invention. If this is set as f (x, y, z) as described in [Equation 9], for example, the function is normalized by 1 as shown in [Equation 10]. To obtain a function that does not blur at the focus position, a delta function, that is, a function of area 1 with a size of zero is used. A two-dimensional image can be blurred by convolving an arbitrary blur kernel with an image on the imaging plane (xy plane).

A distribution function whose spread changes in accordance with the parallax z in this case (proportionally in many cases) is hereinafter referred to as an x, y cross section of a three-dimensional blur kernel or simply a blur kernel, a blur function, or the like.

  FIG. 11 shows a state in which the blur kernel B is convoluted with an object in focus with an arbitrary blur amount of blur kernel z = 4 (pixels). x and y are two-dimensional coordinates in the imaging surface, and z is a parallax amount. When the radius of the blur kernel is set to zero, an omnifocal image is generated. A detailed description of the blur generation algorithm will be described later.

  Assuming multi-lens cameras arranged in a two-dimensional grid at equal intervals in the camera plane, the parallax amount z is [Equation 7], s0 = interval between adjacent cameras, adjacent camera interval 12 mm, focal length 5.34 mm. Table 1 shows the parallax amount z, the distance Z to the subject, and the distance b between the image plane and the lens when the pixel pitch P = 6 μm. The distance b between the lens and the image plane was determined by the lens imaging formula 1 / f = 1 / Z + 1 / b. In this way, when the cameras are equally spaced, the parallax is obtained from the adjacent camera interval, and when the parallax value used for image processing is determined at an interval such that the parallax is an integer multiple of the pixels, In any camera, the image shift amount can be an integral multiple of the pixel pitch, and the occurrence of quantization errors can be prevented. “b” indicates an image forming position by the image forming action of the lens, and is a distance from the lens to the image. db represents the amount of change of b. Strictly speaking, db is not equally spaced. Therefore, when the blur processing of the present invention is performed on an image of a monocular camera, the amount of parallax is evenly spaced compared to the conventional method of acquiring data by moving the sensor at regular intervals in the optical axis direction in the image space. It is desirable to obtain the subject distance from the equation [Equation 7] or [Equation 8].

  On the other hand, imaging of a lens by a synthetic aperture is generally photography using a pan focus lens, and is irrelevant to an imaging formula as described above. Strictly speaking, when a lens is used instead of a pinhole, the position of the image plane also changes according to the position of the subject. The focal length of the camera lens with a focal length of 5.34 mm and F2.8 is adjusted so that the distance to the subject is about 2 m and focused on the subject distance. The change of is not large (change of about 10 to 20%). There is no problem considering this state as pan focus. Since the change in MTF is known, it is possible to store MTF data for each shooting distance in the camera in advance and correct image blur from the MTF data for each shooting distance.

  The focus image at the distance Z corresponding to the parallax (z) is focused by the synthetic aperture method. For example, synthetic aperture images at 20 locations (parallax z = 1 to 20 (pixels)) shown in [Table 1] are generated. In FIG. 9, the parallax amount z is [Equation 7], s0 = interval between adjacent cameras, and an image in which the subject distance corresponding to nine parallaxes of parallax z = 0 to 8 pixels is focused is synthesized by the synthetic aperture method. Represents what to do. Dividing the parallax with a resolution of less than a pixel is meaningless from the viewpoint of generating a blurred image. Further, when thinning out, it is important to thin out so that the parallax intervals are equal. When the number of pixels is increased, the pixel pitch is used.

  When focusing with the synthetic aperture, the image of the surrounding camera is shifted in the xy plane by the amount corresponding to the amount of parallax with respect to the image of the camera at the center (center of the st plane). The amount and the amount of parallax in [Table 1] match. When the parallax is zero, it is at infinity, and when it is 1 pixel, the focus is in front of 10.68 m. When there are 25 cameras, once 25 images are captured, an arbitrary focus image can be obtained by selecting an arbitrary parallax from [Table 1]. A plurality of vertical lines shown in FIG. 9 of the image thus obtained are schematically shown.

FIG. 11 is an explanatory diagram of the blur due to the circular pupil of the normal lens. First, assume a multi-lens camera arranged in a two-dimensional grid at equal intervals in the camera plane. The parallax amount z is [Equation 7] and s0 = interval between adjacent cameras. The blur generated by the three cameras cam2 to cam4 arranged with the adjacent lens interval of 12 mm is, for example, when the focus is made by the synthetic aperture at z = 4 (pixel), and the parallax amount 1 in the optical axis direction intersecting at z = 4 (pixel) Is represented by two straight lines that generate a lateral shift of ± 1 pixel in a direction perpendicular to the increase / decrease. This is the parallax due to the three camera arrangements = the amount of blur. If it is assumed that the cameras are distributed innumerably in a circle having a diameter of cam2 to cam4, it is possible to handle a circular ordinary lens blur similarly. That is, a cone having a straight line intersecting at z = 4 (pixels) as a generating line corresponds to a blur of a lens having a lens pupil diameter of 24 mm. If the camera interval is replaced with the lens pupil radius, the lens blur of the circular pupil can be considered. Larger lens diameter blur and smaller lens diameter blur can also be given by a cone with the focus position at the apex. The blur blur kernel is given by a circle of diameter at each parallax of this cone. As will be described in detail later, the weight in the circular aperture can be arbitrarily changed, such as a uniform distribution or a Gaussian distribution.
<Bokeh shape of camera placement>
FIG. 10 assumes a multi-lens camera arranged in a two-dimensional grid at equal intervals, and the amount of parallax z is [Expression 7] and s0 = interval between adjacent cameras. Represents the shape. FIG. 9 illustrates a blurred shape with respect to parallax in the case of a 5 × 5 arrangement at equal intervals as illustrated in FIG. 8. Since the focus direction needs only to have data so that the parallax is equally spaced, the black circle in the figure represents a blurred shape or a blur kernel. When this is expressed in the xy plane, it is as shown in FIG. The M series is as shown in FIG. 13 and FIG. Arrows represent delta function weights. At the focal position, the strength of the arrow increases by a factor of several. When these blur kernels are integrated in the parallax direction (the direction is the same as the Z (large jet) direction in the depth direction of the object), the result is as shown in FIG.

FIG. 16 shows the flow of processing from image capture. A plurality of focus images obtained by the synthetic aperture method are passed to an arbitrary blur algorithm. The contents of the arbitrary blur algorithm are described in FIG. 17, and a blur kernel having a radius proportional to the parallax is convolved with respect to a plurality of images. These are added together to integrate the parallax of the optical system PSF (camera placement blur shape). Deconvolution is performed based on the result of integrating the parallax of the bokeh kernel.

  The relationship between FIG. 17 and the equation is shown in FIG. 18 and 19 show images obtained by changing the focus (parallax). 20 and 21 are explanatory diagrams for deriving the equation described in [Equation 18]. FIG. 20 illustrates that the calculation time can be shortened by approximating the equation [Equation 18]. In the present invention, the description based on the approximate expression of [Equation 18] is referred to as a “blur generation algorithm”.

  As shown in FIG. 22, the blur processing unit 36 constituting an arbitrary blur generation algorithm, for a plurality of multifocal image data, a circular aperture with a blur radius proportional to the parallax from a desired focus position, or Gaussian or the like Convolution processing is performed using a two-dimensional section (two-dimensional distribution function representing blur) of an arbitrary blur kernel. Here, “proportional to the parallax from the focus position” means, for example, that the convolution process is performed by increasing the radius of the two-dimensional section of the blur kernel in proportion to the parallax from the desired focus position. This is apparent with reference to FIG.

  The focus stack image calculation unit 37 obtains a focus stack image that is an intensity sum image of a plurality of image data processed by the blur processing unit 36. In addition, you may obtain | require an intensity | strength average. The PSF kernel calculation unit 38 calculates a focus stack PSF kernel corresponding to the focus stack of the pupil sampling point of the synthetic aperture (see FIGS. 12 to 15).

  The pupil sampling point represents the arrangement of the photographing lens 23 of the multiview camera 10. Here, since the arrangement of the photographing lens 23 is determined in advance, it is not necessary to calculate the focus stack PSF kernel. The focus stack PSF kernel is stored in advance in the RAM 29 and is read and used. FIGS. 15 and 23 show the focus stack PSF kernel when the pseudo-random arrangement of the 16 photographing lenses 23 is a predetermined periodic pattern according to the M series.

  The image sharpening processing unit 39 performs deconvolution processing using a predetermined kernel on the focus stack image. As a deconvolution process using a predetermined kernel, a deconvolution process using the focus stack PSF kernel read from the RAM 29 is performed on the focus stack image calculated by the focus stack image calculation unit 37 to obtain an arbitrary blurred image. Generate data.

  The deconvolution process using the focus stack PSF kernel is a focus that is a two-dimensional distribution function that represents an integration result in the optical axis direction of the three-dimensional PSF that is a three-dimensional blur function of the single-eye imaging unit. This is a deconvolution process using two kernels: a stack PSF kernel and a focus stack blur kernel corresponding to a focus stack of an arbitrary blur kernel used when the blur processing unit 36 performs the convolution process.

  The CPU 24 records arbitrary blurred image data in the recording unit 31 via the I / F 30. Note that a compression unit may be provided and arbitrary blurred image data may be recorded in a compression format such as the JPEG method. A plurality of multifocal image data obtained from each frame memory 22 may be stored in the recording unit 31. In this case, it is preferable to store a plurality of multifocal image data in association with the blurred image data.

  Next, the operation of the above configuration will be briefly described with reference to FIG. When the power switch 14 is turned on, a through image is generated based on the image data obtained from any one single-eye imaging unit 40 and displayed on the LCD 15. The composition is determined while viewing the through image, and the shutter button 13 is pressed halfway. In response to this half-press operation, the CPU 24 controls AE / AF. The AE control is performed so that each single-eye imaging unit 40 has the same exposure. AF control is focus shift control. In this control, the CPU 24 controls the AF unit 26 to move the photographic lens 23 to the in-focus position so that each photographic lens 23 is completely out of focus.

  If the shutter button 13 is fully pressed as it is, the CPU 24 controls each single-eye imaging unit 40 to capture a multifocal image data group into each frame memory 22 and perform image processing on the captured multiple multifocal image data. Send to part 25.

  The blur processing unit 36 of the image processing unit 25 performs convolution processing on a plurality of multi-focus image data using a circular aperture having a blur radius proportional to a desired focus position or an arbitrary blur kernel such as Gaussian. . The focus stack image calculation unit 37 calculates a focus stack image, which is a pixel intensity sum image, for each of a plurality of image data subjected to the blur processing by the blur processing unit 36.

  The PSF kernel calculation unit 38 reads a predetermined focus stack PSF kernel from the RAM 29. Based on the focus stack PSF kernel read out from the RAM 29, the image sharpening processing unit 39 performs deconvolution processing on the focus stack image calculated by the PSF kernel calculation unit 38 to generate arbitrary blurred image data. The generated blurred image data is recorded by the recording unit 31.

Here, the features of the image processing of the present invention will be briefly described. First, an arbitrary blur process is performed on a plurality of focused images, and then the intensity sum (average) of these images is obtained to obtain an intensity sum (average) image. There is no noise at this stage. Next, an image sharpening process is performed on the obtained intensity sum (average) image to generate an arbitrary blurred image. The image sharpening corresponds to the focus stack of the pupil sampling point of the synthetic aperture. ”Deconvolution. As this deconvolution kernel, a Fourier transform having no zero point is used to suppress the occurrence of restoration noise. This makes it possible to form a blurred image with good photographic quality. The focus stack may be performed while changing the focus state by the synthetic aperture method or the movement of the image sensor in the optical axis direction.
<Bokeh algorithm>
A specific blur generation method will be described using an example of a synthetic aperture method using a multi-lens camera. Specifically, the three-dimensional intensity distribution of the object to be imaged is Io (X, Y, Z). The Z axis is the optical axis direction of the photographing lens 23, and the x and y directions are the horizontal and vertical directions of the image sensor 20. The z-axis is the amount of parallax (parallax = camera interval × focal length / Z, Z is the distance from the lens pupil position to the subject). The in-plane direction xy of the sensor is photographed with a magnification, as can be seen from the equation “x = X × f / Z, y = Y × f / Z” as described above. A camera tertiary distribution of a point image intensity distribution representing the blurriness of the lens is represented by PSFc (x, y, z) , it was taken to focus the parallax z _r along the focus position on the optical axis of the camera The image Ic (x, y, z _r ) is a projection of Ic (x, y, z, z _r ) shown in the equation described in [Equation 11] onto the x, y plane. It is shown by the described formula. PSFc (x, y, z) in the equation of [Equation 11] is represented by a function that has a conical function along the optical axis in the case of a circular aperture, for example, so that the focal position of the apex of the cone is equivalent to the parallax zr. be able to. An image obtained by integrating the focus position in the depth direction of the scene to be photographed is expressed by the equation [Equation 6]. The integration range (zmin to zmax) is approximately matched with the region where the object exists. Io (x, y) is equivalent to a picture (omnifocal image) in which the scene is projected onto the image plane by an ideal pinhole camera without blur.

For example, in the case of a circular aperture, PSFc (x, y, z) in the equation described in [Equation 11] is represented by a function that has a conical function along the optical axis so that the apex of the cone is at the focus position zr. be able to. An image obtained by integrating the focus position in the depth direction of the scene to be photographed is expressed by the equation described in [Equation 13]. The integration range (z min to z max) is generally matched with the region where the object exists. Io (x, y) is equivalent to a picture (omnifocal image) projected on the image plane by an ideal pinhole camera without blur.

Since Ic_pn (x, y) is a convolution of PSFc_pn and Io (x, y) from the equation described in [Equation 13], Io (x, y) is obtained from the equation described in [Equation 15] (Io (x, y) is not used for blur generation according to the present invention but is calculated for reference). Here, PSFc_pn is expressed by the equation described in [Equation 14], and is an integral of PSFc (x, y, z) in the parallax z direction. This is called the focus stack kernel. Note that the subscript c indicates the state of the camera. pn indicates pan focus.

  Consider the integration range described in the equation [14]. In PSFc, the position where the focus is connected is the origin of the z axis (see, for example, FIG. 25). Since the range of the object is infinity = zmin to close distance = z max, the convolved PSFc is PSFc (x, y, z) to PSFc (x, y, z−z max) shown in FIG. Can exist in the range of The integration range of the focus stack is “−z max˜0” for an object at infinity from the figure, and “0−z max” for an object at a close distance. The result of these integrations is preferably the average of PSF shapes when the front and rear pins are asymmetric, for example in the case of FIG. 13 and FIG. It was confirmed by a live-action video that the integration range described in the formula [14] is sufficient. This integration range is optimally about “−z max / 2 to + z max / 2” for the intermediate portion between the close distance and infinity, so the integration range may be variable according to the distance to the subject. As described above, in short-distance shooting, z max may be the farthest position from the camera within the range where refocusing is possible in the subject.

In the same way as the equation described in [Equation 11], the blur function PSFc (x, y, z) is changed to PSFa (x, y, z) and the parallax z1 corresponding to an arbitrary focus position is focused and photographed. The image is shown by the equation described in [Equation 16]. The equation described in [Equation 17] is a three-dimensional image obtained by convolving PSF three-dimensionally, and indicates a luminance distribution that can be acquired while shifting the focus position under the optical condition of PSFc. This is three-dimensional data.

The equation of [Equation 18] is obtained from the equation of [Equation 17]. The part deconvoluted by PSFc_pn (x, y) in the equation [18], that is, “A” in the equation [19] is obtained by applying PSFa (x, It shows that y, zz _r ) is convolved and further integrated with respect to the parallax z. This corresponds to the processing of the blur processing unit 36 and the focus stack image calculation unit 37 of the above-described embodiment.

It should be noted that the equation described in [Equation 20] can be obtained from the equation described in [Equation 18]. For this reason, images Ic_p (x, y, z) at a plurality of focus positions are picked up by a synthetic aperture method or the like, and this is subjected to three-dimensional deconvolution by PSFc (x, y, z) to obtain Io (x, y , z) Obtain the intensity distribution of the object, and perform two-dimensional convolution of PSFa (x, y, z-zr) (arbitrary blur kernel) set to the parallax zr corresponding to the arbitrary focus position. An arbitrary blurred image and an image refocused at an arbitrary focus position may be obtained by integrating with the parallax z and projecting on the x, y plane.

  In this case, the blurred image generation unit may be configured by a three-dimensional deconvolution processing unit 45, a two-dimensional convolution processing unit 46, and an image generation unit 47 as shown in FIG. The three-dimensional deconvolution processing unit 45 regards a plurality of multifocal images (images obtained by focusing at a plurality of positions) acquired from the image acquisition unit 48 and having parallaxes at equal intervals as three-dimensional data, A three-dimensional deconvolution process is performed to determine the intensity distribution of the object. The two-dimensional convolution processing unit 46 performs two-dimensional convolution processing using an arbitrary blur kernel on the image that has been subjected to three-dimensional deconvolution processing. The image generation unit 47 projects an image that has been subjected to the two-dimensional convolution process onto a plane to generate an arbitrary blurred image.

  Note that the right side of the equation described in [Equation 18] indicates the desired blur kernel when the measured value Ic_p (x, y, z) and the parallax zr corresponding to the device-specific focus setting value are in focus: PSFa ( x, y, z-zr), the blur kernel of the focus stack PSF: PSFc_pn (x, y). The subscript a indicates the state of the camera. In the following, constants are omitted.

  PSFa_stk (x, y) in the equation [Equation 18] is expressed by the equation [Equation 21] (see FIG. 25). Similarly to PSFc, the integration range of the equation described in [Equation 21] has the relationship shown in FIG. 26, and the integration range of the focus stack is “−z max to z max” from FIG. This may also be changed to an optimum range such as “−z max / 2 to z max / 2” when focusing on an object at an intermediate distance between the close distance and infinity. There is no problem with “−zmax to zmax”.

The equation of [Equation 18] can be confirmed experimentally as follows. When PSF is expressed by a Gaussian distribution, PSFa (x, y, zz _r ) for newly generating a blur with respect to PSFc (x, y, z) of the camera is represented by [Equation 22] and [Equation 23], respectively. It is shown by the formula. C ₁ (z) and c ₂ (z) are standard deviations of the Gaussian distribution and are expressed by the equations [Equation 24] and [Equation 25]. The radius of a Gaussian blur kernel is proportional to the standard deviation. z _r indicates a set focus position when an arbitrary blurred image is generated, and indicates a position where a light cone of an arbitrary blurred kernel converges. That is, as shown in FIG. 27, the blur kernel is a point where the cone of light (conical blur shape or blur kernel) is most narrowed (converged) at a distance z _r to be focused.

The trace 2 shown in FIG. 28 shows the standard deviation c ₁ (z) of PSFc (x, y, z), and the trace 3 shows the standard deviation c ₂ (zz _r ) of PSFa (x, y, zz _r ). Here, the parallax corresponding to the defocus is 5 pixels (for example, z _r = 5 (pixels)). The horizontal axis shown in FIG. 28 indicates the parallax amount z taken in the direction of the optical axis. The vertical axis shows the standard deviation. Trace 1 shows the standard deviation c _ca (z, z _r ) of PSFca (x, y, z, z _r ) in the equation [Equation 26]. The standard deviation c _ca (z, z _r ) is given by the equation [Equation 27]. In this case, the object is at z = 0. Although the object takes an arbitrary value of z according to the unevenness, the light of PSFc (x, y, z) is observed in Ic_p (x, y, z) observed as a result of the convolution of the equation [17] The center of the cone is always located on the object surface (see FIG. 27). That is, the blur distribution acting on the object surface is PSFca (x, y, z, z _r ) by the convolution of the equation [Equation 19].

On the other hand, assuming that the blur radius at the desired defocus position is the blur function PSF ₃ (x, y, z) in the formula [29] and c ₃ (z) = k ₃ = constant, the synthesized blur radius PSFca3 ( The standard deviation of x, y, z) is shown by trace 4 in FIG. The same figure shows the standard deviation c ₁ (z) of PSFc (x, y, z) and the standard deviation c ₂ (z) of PSFa (x, y, z). The synthetic blur radius PSFca3 (x, y, z) is expressed by the equation described in [Equation 28], and the standard deviation is expressed by the equation shown in [Equation 30]. Further, the blur radius c ₃ (z) = k ₃ is shown as a trace 6.

For example, the coefficients k1 = 2, k2 = 1, and k3 = 5. As shown in FIG. 29, the value of trace 1 is 5 when z = 0. Trace 3 in FIG. 28 has k2 = 1 and focus 5, so the standard deviation is 5 at z = 0. Under this condition, the minimum value of the trace 1 is about 4.5, which is about 10% smaller than 5, but the trace 1 can be regarded as a translation of the trace 2 in the z direction. Since the information in the z direction is lost after the integration in the z direction of the formula [19] is performed, the integration results in the z direction of the trace 1 and the trace 2 can be regarded as the same. This means an approximation of the equation of [Equation 18].

  As described above, the function representing the PSF is a Gaussian function. As a result of simulating the function representing the circular aperture and the function representing the pupil sampling point of the synthetic aperture method, it was found that substantially the same effect can be obtained. In other words, this method can give images of various blur shapes to hardware with various blur functions.

  Thus, in the present invention, a desired image can be obtained directly from an acquired image. In the present invention, the three-dimensional imaging state is represented by a three-dimensional convolution model based on the three-dimensional luminance distribution of the object and the three-dimensional blur function, as shown by the equation described in [Equation 17]. Therefore, a desired two-dimensional image can be obtained by deconvolution calculation of the two-dimensional image of the formula [18].

  The multi-lens camera 10 performs camera calibration by a six-point method or the like. Measurement values are used for constants such as the position (s, t) of the lens center in the imaging plane and the focal length of each photographing lens 23. Note that an error amount such as a tilt of the optical axis of the camera is removed by camera calibration. The tilt of the optical axis is corrected as a distortion component.

  In FIG. 8, the center of the photographic lens is located on the image plane (s, t), and 25 (5 rows × 5 columns) single-eye imaging units are denoted by D (s, t). The object plane and the imaging plane are parallel, and the object plane is focused at a distance Z corresponding to an arbitrary parallax z from the imaging plane by the synthetic aperture method.

  The depth of focus of the photographing lens 23 of the multi-lens camera 10 of the present embodiment is deep, and the photographing lens 23 may be considered as pan focus from a short distance to infinity. When the photographing lens 23 is bright and the focal depth is shallow, and AF for each photographing lens 23 (ordinary focusing before and after a normal lens) is required, AF may be applied to the subject to be focused in advance for each photographing lens 23. . In this case, when refocusing, it is desirable to compensate for the degradation of MTF from known MTF and PSF data.

Regardless of the synthetic aperture method, the image of a plurality of focus positions can be obtained by adjusting the focus of the normal monocular photographing lens 23 (adjustment by converting the aerial image and the relative position of the image sensor 20 in the optical axis direction in the image space). In the case of obtaining the parallax amount, the parallax amount is obtained from the equation described in [Equation 7] or [Equation 8] from the pupil radius instead of the lens interval used when obtaining the parallax amount in the case of a multi-lens camera. Determine the focus distance so that For example, in the case of a 35 mm full-size camera, when f = 50 mm, for example, the F number is 2, and the pupil radius is 50 / (2) = 25 mm, but as described above, this value “25” is ¼. It should be about. If an image at each focus position is acquired and an image group with an equal interval of parallax is obtained, the subsequent blur algorithm processing is the same as when the multi-eye camera synthetic aperture method is used. (Focus adjustment includes methods such as changing the position of the sensor in the optical axis direction, changing the aberration of the lens, and changing the image plane to acquire an image at each focus position. is there.)
<Arbitrary blur generation>
The point that processing is performed using a multifocal image group adjusted to a plurality of focus positions by the focusing of the synthetic aperture method will be described below.

[Bokeh generation kernel: circular aperture]
When defocus blur of a circular aperture is generated as the blur generation kernel, the blur kernel PSFa (x, y, z) is expressed by the equation described in [Equation 31]. The radius r _a (z) of the blur kernel that is arbitrarily set is given by the equation described in [Equation 32]. A desired blur size can be obtained by arbitrarily setting the constant Apa. Further, by giving an offset, for example, zr in the z direction (replace “z” with “z-zr”), it is possible to focus on an arbitrary focus position zr.

[Bokeh generation kernel: Gaussian aperture]
In the case of Gaussian blur, the blur kernel PSFgss (x, y, z) represented by [Expression 33] is used. C is a constant. The magnitude of the Gaussian blur can be adjusted by arbitrarily setting σ _{a according} to the equation described in [Equation 34].

[Arbitrary viewpoint]
If an arbitrary viewpoint blur kernel PSFshift (x, y, z) is used, an all-focus image at an arbitrary viewpoint can be obtained as shown in FIG. Here, the viewpoints are arbitrarily set by arbitrarily setting the constants a _xs and a _ys .

Further, an image Ic_shift_p (x, y, z) measured by focus scanning is expressed by an equation described in [Equation 34] instead of the equation [Equation 17].

An image refocused to the parallax z1 of the viewpoint of the formula [Expression 35] including the blur of PSF and PSFa (x, y, z) indicating an arbitrary blur is expressed by the expression described in [Expression 37]. . The focus stack of the PSF with the viewpoint shifted is expressed by the equation described in [Equation 38].

The lens arrangement of the multi-eye camera 10 represents a pupil sampling point. The pupil sampling point acts as a focus stack blur kernel when focus stacking is performed. The focus stack blur kernel is represented by PSFc_pn (x, y) in the equation described in [Equation 14]. The pupil sampling point before the focus stack is PSFc (x, y, z). Since the desired image is obtained by the equation described in [Equation 18], there is no zero point in the amplitude of the Fourier transform of PSFc_pn (x, y) in order to accurately perform the deconvolution of the focus stack kernel PSFc_pn (xy). desirable. This is because deconvolution can be performed by applying the inverse of the Fourier transform of PSFc_pn (x, y) and then performing the inverse Fourier transform. The deconvolution of the equation described in [Equation 18] is represented by the equation described in [Equation 39].

By the way, it is desirable that the amplitude of the Fourier transform of PSFa_stk (x, y) and PSFc_pn (x, y) has no zero point. PSFa (x, y) is preferably a circular aperture or a Gaussian function. When a circular aperture is used for PSFa (x, y), PSFa_stk (x, y) is expressed by [Equation 41] and its Fourier transform is expressed by [Equation 42]. U and v are expressed by [Equation 43] and [Equation 44].

  The amplitude of the Fourier transform of PSFc_pn (x, y) needs to have no zero point, and the arrangement of the taking lens 23 is a circumferential arrangement with many arrangements including a predetermined pattern using vertical and horizontal M-sequence pseudo-random number sequences, The rectangular layout and grid layout also satisfy this condition. The focus stack kernel of M-sequence random arrangement is, for example, as shown in FIG. There is no zero point in the amplitude of the Fourier transform of this kernel, and there is no inconvenience such as divergence due to zero division in the equation [39]. Also, since the change in amplitude is about 20 times, noise of a specific frequency is not picked up.

  For the calculation of the equation shown in [Equation 19], the equation shown in [Equation 40] is actually used using Fourier transform. The Fourier transform of a Gaussian function is represented by a Gaussian function, and the Fourier transform of a circular aperture is represented by a Bessel function.

<Evaluation of blurred image>
FIG. 30 shows an image taken with a general camera. FIG. 31 shows a blurred image obtained by using the conventional raw synthetic aperture method with the same composition as the image in FIG. 30 (focus position 1.34 m, (background blur: equivalent to 8 pixels between adjacent cameras: 25 eyes). ).

  On the other hand, in the M-series 16-eye multi-lens camera 10 of this embodiment (background blur: equivalent to 8 pixels of parallax between adjacent cameras), by specifying the size of the blur corresponding to the focal position 50 mm and the aperture F1, A blurred image as shown in FIG. 32 is obtained. Further, FIG. 33 shows a blurred image obtained when a blur position corresponding to a focal position of 50 mm and a diaphragm F1.4 is designated. Further, FIG. 34 shows a blurred image obtained when the blur position corresponding to the focal position 50 mm and the stop F2 is designated. In the blur process of the present embodiment, a circular opening having a uniform weight is used for the blur kernel.

  Similarly, FIG. 35 shows a blurred image obtained when a circular aperture having a uniform weight is used for the blur kernel and the focal position is 50 mm and the blur size corresponding to the stop F4 is designated. FIG. 36 shows a blurred image obtained when the blur position corresponding to the focal position 50 mm and the aperture F42 is designated. In this way, by specifying the blur radius to the blur processing unit 36 and adjusting the blur amount, it is possible to generate a blur image that is more beautiful than the blur described in the related art in which the background blur is changed. Furthermore, by adjusting the parameters, for example, the background can be intentionally blurred by a circular opening or Gaussian.

<Evaluation of image subjected to blurring process by changing aperture value with 25 eyes in M series>
When the camera arrangement described in [Equation 3] is adopted, the blur position corresponding to the focal position of 50 mm and the aperture stop F1 is obtained in the M series 25 eyes with the focus position of 1.34 m (background blur: equivalent to 8 pixels between adjacent cameras). By specifying the size, a blurred image as shown in FIG. 37 is obtained. In addition, FIG. 38 shows a blurred image obtained when the blur position corresponding to the focal position 50 mm and the stop F1.4 is designated. Further, FIG. 39 shows a blurred image obtained when the blur position corresponding to the focal position 50 mm and the stop F2 is designated. FIG. 40 shows a blurred image obtained when the blur position corresponding to the focal position 50 mm and the stop F4 is designated. FIG. 41 shows a blurred image obtained when the blur position corresponding to the focal position 50 mm and the stop F42 is designated. According to this, it can be seen that the blur is more straightforward than the images of FIGS. In these blur processes, a circular aperture with uniform weight is used for the blur kernel.

[When convolution is performed in three dimensions before approximation of [Formula 18]]
As another application example using the image data structured based on the equidistant parallax of the present invention, the three-dimensional deconvolution of the equation described in [Equation 18] is performed, and the luminance distribution of the object (I0 (x, y, z)), an arbitrary two-dimensional blur kernel is convoluted in two dimensions, and the resulting three-dimensional intensity distribution is projected onto the x, y plane (integrated with respect to parallax z) It is. Three-dimensional deconvolution requires a three-dimensional Fourier transform and requires a long calculation time, which is promising in the future.

  Note that the invention described in Japanese Patent No. 4437228 (conventional invention) uses a Fourier transform image obtained by Fourier transforming an image using a three-dimensional or more blur filter corresponding to an arbitrary viewpoint, aperture, and focusing. Processing is performed, this is subjected to inverse Fourier transform, and clipping processing is performed. For this reason, a three-dimensional or higher filtering process is required, and the frequency characteristic in the optical axis direction is involved in the filtering process, which is complicated and difficult to understand intuitively. On the other hand, in the present invention, such a problem does not occur because blur generation is performed by a two-dimensional blur kernel.

  Further, in the above-described conventional invention, in the orthogonal coordinate system representing the imaging space, a plurality of images having different focal points at equal intervals in the optical axis direction are used. Because the z coordinate in the optical axis direction in z was converted to z = 1-f / Z, the shift invariant relationship was established, but there was no idea of using "parallax" as the scale in the depth direction. The relationship between the amount of blur and the direction of the optical axis could not be considered, and efficient processing could not be performed. On the other hand, in the present invention, since the shooting data is obtained by representing the depth direction of the object as “parallax” having the unit of length as in the x and y directions, for example, the pixel pitch (output pixel pitch or The depth direction (optical axis direction) can be sampled efficiently and with the minimum number of images with a parallax amount proportional to the pixel pitch of the sensor. In addition, when the focus adjustment of the objective lens is performed and images in a plurality of focus states are acquired, the parallax is calculated based on the equation [8], so that a shift / invariant three-dimensional imaging state is obtained. Therefore, no calculation error occurs, and no conversion noise occurs in the arbitrarily blurred image.

<Image evaluation of arbitrary blur image generation>
Using an M-sequence 25-eye camera arrangement, 20 focus positions and a circular aperture for the blur kernel were used to generate a blur image with a specified distance blurred (see FIGS. 41 to 48). In this case, the focal length is 50 mm, the aperture is equivalent to F1, and the image is taken indoors.
FIG. 42 shows a refocus distance of 0.71 m.
FIG. 43 shows a refocus distance of 0.82 m.
FIG. 44 shows a refocus distance of 0.97 m.
FIG. 45 shows a refocus distance of 1.20 m.
FIG. 46 shows a refocus distance of 1.50 m.
FIG. 47 shows a refocus distance of 2.10 m.
FIG. 46 shows a refocus distance of 3.60 m.
In FIG. 49, the refocus distance is set to 10.7 m and generated.

Moreover, the image image | photographed outdoors on the same imaging conditions is shown in FIGS.
FIG. 50 shows a refocus distance of 0.97 m.
FIG. 51 shows a refocus distance of 1.20 m.
FIG. 52 shows a refocus distance of 1.50 m.
FIG. 53 shows a refocus distance of 2.10 m.
54 shows a refocus distance of 3.60 m.
In FIG. 55, the refocus distance is set to 10.7 m and generated.

[Extended function for arbitrary blur image generation]
If this embodiment is used, an arbitrary blurred image can be generated. Therefore, if data of a plurality of focus positions of PSFs of a plurality of types of existing photographing lenses is stored in the camera in advance, the lens need not be replaced. Thus, it is possible to realize a camera (having a lens emulation function) that can obtain an image having the same result as that obtained by using a desired lens. In addition, the amount of blur can be changed during shooting after shooting regardless of the aperture. By inputting parameters from the camera, it is possible to arbitrarily change the blur amount, viewpoint, post-shooting focus target change, and refocus in addition to the existing change of the focus target, shooting magnification, and trimming.

[Arbitrary viewpoint image generation by M-sequence pupil sampling]
Arbitrary viewpoint image generation is well-known, but deconvolution can be performed at high speed by FFT calculation by arranging pupil sampling in an M-sequence arrangement. An endoscope, an actual microscope, a microscope, and the like that can generate an arbitrary viewpoint image or an arbitrary viewpoint stereo image are easily adopted. In that case, the light field type (single objective lens + microfield lens array pupil division) is more suitable than the plenoptic type (multi-lens, multiple objective lens type). In order to achieve the number of VGA pixels with 25 pupil divisions (use of 16 eyes in the M series), a total number of pixels of 768,000 is sufficient.

[All-focus (multi-pixel) image generation]
An all-focus image can be generated using this embodiment. In this case, a plurality of multifocal image groups are acquired by a synthetic aperture photographing method in which photographing is performed using a multi-eye camera in which a two-dimensional array of photographing lenses is set to a predetermined pattern which is an M-sequence pseudo-random sequence. Then, the intensity average is calculated to obtain an intensity average image (focus stack) image as shown in FIG. Note that the focus stack image shown in FIG. 56 is the center viewpoint. Then, an all-focus image can be generated as shown in FIG. 57 by performing a process of deconvolving the focus stack image shown in FIG. 56 with a predetermined focus stack PSF kernel.

[Bokeh generation + Arbitrary viewpoint image generation]
By combining arbitrary viewpoint image generation processing and blur generation processing, for example, an image other than the focus position of a stereo image can be intentionally blurred. In this way, it is possible to reduce discomfort caused by the absence of the focal position (display position) of the eye and the amount of parallax, which occurs when a large parallax image other than the focus position is viewed.

  For example, a focus stack obtained by acquiring a plurality of multifocal image groups by the synthetic aperture photographing method, calculating an average intensity by adding parallax of an arbitrary viewpoint, and adding an arbitrary viewpoint of parallax to the calculated intensity average image Deconvolution is performed using a PSF kernel (focus stack PSF kernel with parallax). The images shown in FIGS. 58 and 59 are images that have been subjected to deconvolution processing using a focus stack PSF kernel (focus stack kernel with viewpoint shown in the figure) obtained by adding a parallax of an arbitrary viewpoint to a multifocal image group. .

Here, using an imaging device with an M-series 25-eye camera arrangement and a camera interval of 12 mm pitch, a wall clock is imaged indoors to generate a blurred image of an arbitrary viewpoint with parallax, and attention is paid to the bokeh of the wall clock. I tried to evaluate how much parallax can be tolerated.
FIG. 60 shows an image corresponding to 0 pixels of parallax between adjacent variable cameras at the shooting distance = 5.3 m and the wall clock position.
FIG. 61 is an image corresponding to 2 pixels of parallax between adjacent variable cameras at a shooting distance = 3.6 m and a wall clock position.
FIG. 62 is an image corresponding to 3 pixels of parallax between adjacent variable cameras at the shooting distance = 2.1 m and the wall clock position.
FIG. 63 is an image corresponding to 5 pixels of parallax between adjacent variable cameras at the shooting distance = 1.5 m and the wall clock position.
FIG. 64 is an image corresponding to 7 pixels of parallax between adjacent variable cameras at the shooting distance = 1.2 m and the wall clock position.
FIG. 65 is an image corresponding to a parallax of 9 pixels between adjacent variable cameras at the shooting distance = 0.97 m and the wall clock position.
FIG. 66 shows an image corresponding to 10 pixels of parallax between adjacent variable cameras at the shooting distance = 0.89 m and the wall clock position.
FIG. 67 is an image corresponding to 11 pixels of parallax between adjacent variable cameras at the shooting distance = 0.82 m and the wall clock position.
FIG. 68 shows an image corresponding to a parallax of 13 pixels between adjacent variable cameras at the shooting distance = 0.71 m and the wall clock position.
FIG. 69 shows an image corresponding to a parallax of 15 pixels between adjacent variable cameras at the shooting distance = 0.63 m and the wall clock position.

  Referring to FIG. 60 to FIG. 69, it was found that there was no major failure in the blur of the wall clock up to 10 pixels of parallax. From this, in the case of a 12 mm pitch camera arrangement, it is safer to avoid the blur of the wall clock when the parallax is 11 pixels or more, because it appears discretely. Therefore, since it is too blurry and dirty, the parallax of blur is preferably about 10 pixels at maximum (in the case of VGA). This is about 640 × 10 = 17% in the horizontal direction in the case of VGA (640 × 480 dots). As a result, the shortest shooting distance is about 1 m.

  The amount of parallax allowed for this evaluation depends on the camera arrangement. Therefore, for example, as an array of 5 × 5 arrays, nine eyes are rotated 90 degrees at a time and calculated in four patterns (formula described in [Equation 45]), and these intensity average images are taken, the blur is more straightforward. I found out that This is thought to be because the weighting is medium-high and has a Gaussian distribution. In this case, it is desirable to increase the number of pixels once with nine eyes, and then further increase the number of pixels for the second time from the four sets of multi-pixel images, and finally obtain the average intensity.

[How to change the focus setting distance for each camera]
A camera with 16 or more eyes is used as the synthetic aperture camera. Further, in order to make the camera low in height, it is necessary to reduce the preliminary size of the image sensor. For this reason, a pixel becomes small and the number of pixels is insufficient in some cases. For example, when trying to obtain 4 million pixel output using 16 VGA 1/6 inch cameras (effective diagonal 3.5 mm), the resolution of each lens is 4 million pixel resolution (diagonal approximately 3000 pixels). Therefore, a resolution of about 1 to 2 μm is required. At this time, a lens brighter than F number 2 is required. If the highest resolution is always required, a focus adjustment mechanism is required. In general, a voice coil motor or a capacitance type micro electro mechanical system (MEMS) is used. Focus measurement uses the parallax of a multi-lens camera. However, so-called refocusing that focuses on an arbitrary place after shooting cannot be performed.

  Therefore, in the present embodiment, as described above, the range from the shortest shooting distance of the subject to infinity is divided by the number of the single-eye imaging units 40 and each is set in a state of being focused. Then, after performing the focusing at the synthetic aperture, a plurality of (for example, 16) focused images of the single-eye imaging unit 40 are effective in a portion without parallax. By deconvolution with an IPSF (Integral PSF) that represents a composite PSF of 16 shooting lenses (integrated PSF), it is always infinity (substantially 8000 mm inside and outside) from the shortest shooting distance (substantially inside and outside 50 mm). ) Was successfully made the refocusable range by the synthetic aperture. The focus offset amounts (n) of the multiple (N) cameras are set so that the parallax is equally spaced so that the amount of blur in each camera's coverage is uniform. .

  In the above-described embodiment, a plurality of multi-focus images are simultaneously acquired using the multi-lens camera 10, but a general single-lens camera is used to perform a plurality of shootings in time series while shifting the focus. A plurality of multifocal images may be acquired.

  In each of the above embodiments, the electronic camera is described. However, the present invention is not limited to this, and the imaging lens 23, the imaging device 20, the AFE 21, the frame memory 22, and the like are omitted, and an image processing unit is provided. An image processing apparatus or a method thereof may be used. In this case, a storage unit may be provided for capturing a plurality of multifocal image data that has been captured in advance.

  The program to be executed by the image processing unit constitutes a program according to the present invention. As a recording medium for recording the program, a semiconductor storage unit, an optical or magnetic storage unit, or the like can be used. By using such a program and a recording medium in a system having a configuration different from that of each of the above-described embodiments and causing the CPU to execute the program, substantially the same effect as the present invention can be obtained. Therefore, the present invention may be a program for causing a computer to function as the above-described image processing apparatus. Moreover, it is good also as a recording medium which recorded the program.

[Depth map method]
There is known a method of obtaining a depth map by obtaining a distance to an object by a method such as stereo matching using a publicly known stereo camera. In the present invention, as shown in FIG. 70, the degree of coincidence of pixels at each focus position of the refocusing of the synthetic aperture method is obtained from an evaluation amount such as a known variance or a value of max-min, and distance data for each pixel is obtained. get. Since this is two-dimensional data, it is generally called a depth map. The depth map is decomposed according to the parallax (distance), for example, as shown in FIGS.

  According to the knowledge newly found in the present invention, it has been found that it is possible to generate a blurred image superior to the conventional one from an image using a depth map and a synthetic aperture method.

  As shown in FIG. 77, an image acquisition unit 49, a depth map information acquisition / conversion unit 50, a region division unit 51, a convolution unit 52, and a blurred image generation unit 53 that acquire a plurality of images with shifted parallax or focus positions. These are connected by a bus 54 and are controlled centrally by a CPU 55. The acquired plurality of images are stored in the memory 56.

  The operation of the above configuration will be briefly described with reference to FIG. The image acquisition unit 49 acquires a plurality of images in which the parallax or the focus position is shifted. The CPU 55 stores a plurality of images acquired from the image acquisition unit 49 in the memory 56. The CPU 55 controls the depth map information acquisition / conversion unit 50, and the depth map information acquisition / conversion unit 50 acquires depth map information by a stereo matching method. At this time, the scale in the optical axis direction of the depth map information is replaced with parallax. The depth map information is converted into data so as to be equidistant parallax.

  The CPU 55 controls the area dividing unit 51, and the area dividing unit 51 generates one piece of an image focused on an arbitrary focus by the synthetic aperture method, and for each area in which the image is in focus according to the depth map information. Divide into

  The CPU 55 controls the convolution unit 52. The convolution unit 52 convolves a different blur kernel (blur function) for each divided region, and blurs the image of each region. The magnitude of the blur function is proportional to the amount of parallax from the best focus point.

  The CPU 55 controls the blurred image generation unit 53, and the blurred image generation unit 53 integrates the images of the regions after the convolution.

  According to such a configuration, since the multi-pixel image after the synthetic aperture refocusing of the multi-lens camera is divided and blurred for each depth, the multi-pixel can be easily formed. Conventionally, a depth map has been used, but since a blur process is performed on the output of a single-lens camera, the number of pixels cannot be increased. In the present invention, a depth map is created every time the parallax is an integral multiple of the pixel pitch, so that blur processing that looks natural efficiently can be performed. In addition, with such a process, blurring caused by camera placement by the synthetic aperture method can be completely removed, and a more prominent blur can be created.

DESCRIPTION OF SYMBOLS 10 Multi-lens camera 11 Imaging aperture 20 Imaging element 23 Shooting lens 25 Image processing part 36 Blur treatment part 37 Focus stack image calculation part 39 Image sharpening process part 40 Single eye imaging part

Claims (37)

  A plurality of in-focus images are subjected to blur processing set in advance at an arbitrary focus position, a focus stack image that is an intensity sum image of the plurality of images subjected to the blur processing is calculated, and the focus stack An image generation method, wherein an image is sharpened to generate an arbitrary blurred image. The image generation method according to claim 1,
The blur generation process includes a process of convolution using an arbitrary blur kernel. The image generation method according to claim 1 or 2,
The image sharpening process includes a deconvolution process using a predetermined kernel for the focus stack image. The image generation method according to claim 2,
The arbitrary blur kernel is a circular aperture having a blur radius proportional to a desired focus position, or a function representing a Gaussian. The image generation method according to claim 3.
The deconvolution processing using the kernel is a focus stack PSF (point spread function) that is a two-dimensional distribution function that represents an integration result of the three-dimensional PSF that is a three-dimensional blur function of the photographing camera in the optical axis direction of the photographing camera. ) Including deconvolution processing using two kernels, a kernel and a focus stack blur kernel corresponding to the focus stack of the arbitrary blur kernel used when the blur processing unit performs the convolution processing. Image generation method. The image generation method according to claim 1,
The plurality of in-focus images are images generated by a multi-lens camera composed of a plurality of cameras (a plurality of imaging units having an image sensor), and are focused by focusing using a synthetic aperture method. An image generation method characterized by being a plurality of in-focus images obtained as a result. The image generation method according to claim 5,
An image generation method, wherein a three-dimensional PSF that is a three-dimensional blur function of the photographing camera represents a camera arrangement of a multi-view camera. The image generation method according to claim 1,
The plurality of in-focus images are images generated by a multi-lens camera including a plurality of cameras (a plurality of imaging units having an image sensor), and parallax (parallax = between two points on the pupil plane ( Alternatively, an image generation method using an image obtained at a distance to a subject having a uniform distance (distance between optical axes of a plurality of cameras) × image forming magnification). The image generation method according to claim 1,
The plurality of in-focus images are images generated by a multi-lens camera including a plurality of cameras (a plurality of imaging units having an image sensor), and parallax (parallax = between two points on the pupil plane ( Alternatively, an image generation method using an image obtained at a distance to a subject whose distance between the optical axes of a plurality of cameras) × image formation magnification) is an integral multiple of the pixel pitch of the image sensor. The image generation method according to claim 1,
The plurality of in-focus images are images generated by a multi-lens camera including a plurality of cameras (a plurality of imaging units having an image sensor), and parallax (parallax = between two points on the pupil plane ( Alternatively, an image obtained at a distance to a subject in which the distance between the optical axes of a plurality of cameras) × image forming magnification) is an integral multiple of the pixel pitch of the output image output from the camera (imaging device) is used. A featured image generation method. The image generation method according to claim 2,
The arbitrary blur kernel is an image generation method characterized in that a pupil shape is the most narrowed at a distance to a subject to be focused. A blur processing unit that performs blur processing that is set in advance at an arbitrary focus position on a plurality of in-focus images that are input;
A focus stack image calculation unit that calculates a focus stack image that is an intensity sum image of a plurality of images subjected to the blur processing in the blur processing unit;
An image sharpening processing unit that sharpens the focus stack image to generate an arbitrary blurred image; and
An image generation apparatus comprising: The image generation apparatus according to claim 12, wherein
The image generation apparatus, wherein the blur processing unit performs a convolution process using an arbitrary blur kernel. The image generation apparatus according to claim 12 or 13,
The image sharpening processing unit performs a deconvolution process using a predetermined kernel on the focus stack image. The image generation apparatus according to claim 14.
The arbitrary blur kernel is a circular opening having a blur radius proportional to a desired focus position, or a function representing a Gaussian. The image generation apparatus according to claim 14.
The deconvolution process using the kernel includes a focus stack PSF kernel, which is a two-dimensional distribution function representing an integration result of the three-dimensional PSF, which is a three-dimensional blur function of the photographing camera, in the optical axis direction of the photographing camera; An image generation apparatus characterized by deconvolution processing using two kernels, a focus stack blur kernel corresponding to a focus stack of the arbitrary blur kernel used when performing a convolution process in a blur processing unit. The image generation device according to claim 14,
The plurality of in-focus images are images generated by a multi-lens camera composed of a plurality of cameras (a plurality of imaging units having an image sensor), and are focused by focusing using a synthetic aperture method. An image generating apparatus characterized by being a plurality of in-focus images obtained as a result. The image generation device according to claim 14,
An image generating apparatus characterized in that a three-dimensional PSF, which is a three-dimensional blur function of the photographing camera, represents an arrangement of an imaging unit of a multi-view camera. The image generation apparatus according to claim 12, wherein
The plurality of in-focus images are images generated by a multi-lens camera including a plurality of cameras (a plurality of imaging units having an image sensor), and parallax (parallax = between two points on the pupil plane ( Alternatively, an image generation apparatus using an image obtained at a distance to a subject having an equal interval (distance between optical axes of a plurality of cameras) × image forming magnification). The image generation device according to claim 13.
The arbitrary blur kernel is an image generating apparatus characterized in that a pupil shape is most narrowed by a distance to a subject to be focused.   A program for causing a computer to function as the image generation apparatus according to any one of claims 12 to 20.   A recording medium on which the program according to claim 21 is recorded. An imaging unit that captures a plurality of in-focus images, and a circular aperture with a blur radius proportional to the amount of parallax from a parallax value corresponding to a desired focus position with respect to an image with a parallax amount corresponding to the plurality of in-focus states, Or a blur processing unit that performs convolution processing using an arbitrary blur kernel such as Gaussian;
A focus stack image calculation unit that calculates a focus stack image that is an intensity sum image of a plurality of images subjected to the blur processing in the blur processing unit;
Based on a focus stack PSF kernel that is a two-dimensional distribution function that represents a result of integration of a three-dimensional PSF that is a predetermined three-dimensional blur function of the electronic camera in the optical axis direction of the electronic camera, the focus stack image is An image sharpening processing unit that generates an arbitrary blurred image by deconvolution processing;
An electronic camera characterized by comprising: The electronic camera according to claim 23, wherein
The imaging unit that captures the plurality of in-focus images is a multi-lens camera composed of a plurality of cameras, and can be focused by a synthetic aperture method, and can be placed at an arbitrary focus position by a synthetic aperture method. An electronic camera capable of outputting a plurality of in-focus images in focus. The electronic camera according to claim 23, wherein
An electronic camera characterized in that a three-dimensional PSF, which is a three-dimensional blur function of the electronic camera, represents an arrangement of imaging units provided in a multi-view camera. 24. The electronic camera according to claim 23.
The imaging unit includes a plurality of imaging apertures arranged in a two-dimensional manner, a plurality of imaging lenses arranged behind the imaging aperture, and a plurality of imagings for imaging a subject image formed by the imaging lens. An element,
The interval between the photographing apertures in the x and y directions is set to a periodic pattern in which binary pseudo-random sequences form M sequences, respectively. 24. The electronic camera according to claim 23.
The imaging unit includes a plurality of imaging apertures arranged in a two-dimensional manner, a plurality of imaging lenses arranged behind the imaging aperture, and a plurality of imagings for imaging a subject image formed by the imaging lens. An element,
An electronic camera according to claim 1, wherein each of the photographing openings is arranged on a circumference. The electronic camera according to claim 23 or 24,
Either one or both of the photographic lenses and the imaging elements is photographic optical axes so that the focus of the subject image formed by the plurality of photographic lenses is shifted with a uniform amount of blur for each of the imaging elements. An electronic camera comprising focus shifting means for moving in a direction. The electronic camera according to any one of claims 23 to 28,
The focus stack image generation unit calculates a focus stack image that is an intensity sum image of a plurality of images obtained by adding a parallax of an arbitrary viewpoint to a plurality of images subjected to the blur processing by the blur processing unit,
The image sharpening processing unit generates a blurred image of an arbitrary viewpoint by performing a deconvolution process on the calculated focus stack image based on a focus stack PSF kernel obtained by adding a parallax of an arbitrary viewpoint. .   Acquire in-focus images corresponding to a plurality of equally spaced parallax amounts, consider these images as three-dimensional data, perform three-dimensional deconvolution processing, obtain the intensity distribution of the object, and arbitrarily set the obtained intensity distribution An image generation method characterized by performing a two-dimensional convolution process using a blur kernel and projecting the image subjected to the two-dimensional convolution process onto a plane to generate an arbitrary blurred image.   An acquisition unit that acquires a plurality of images in focus corresponding to a plurality of equidistant parallax amounts, and a plurality of images acquired by the acquisition unit are regarded as three-dimensional data, and a three-dimensional deconvolution process is performed to perform an object intensity distribution A three-dimensional deconvolution processing means for obtaining a two-dimensional convolution processing means for performing a two-dimensional convolution process with an arbitrary blur kernel on the three-dimensional deconvolution processed image, and the two-dimensional convolution processed An image generating apparatus comprising: generating means for projecting an image onto a plane to generate an arbitrary blurred image.   An image generation method for generating a focus image by a synthetic aperture method, in which PSF or MTF for each focus position is stored in advance, and when a focus image is generated by the synthetic aperture method, a change in PSF or MTF according to the focus position is detected. An image generation method, wherein image processing is performed so as to cancel.   An image generation apparatus for generating a focus image by a synthetic aperture method, a storage unit that stores PSF or MTF for each focus position in advance, and a change in PSF or MTF according to the focus position when generating a focus image by the synthetic aperture method And an image processing means for performing image processing so as to cancel out the image.   Depth map information (image segmentation information) is acquired based on a plurality of images with the parallax or the focus position shifted, and the scale in the optical axis direction of the depth map information is replaced with parallax so that the parallax amounts are evenly spaced. The depth map information is converted into data, and an image focused to an arbitrary focus is divided into regions by the synthetic aperture method, and a blur kernel (blur) corresponding to the amount of parallax from the focus position is divided into the images of the divided regions. A convolution function), and the image after convolution is integrated to generate a blurred image.   Depth map information is acquired based on a plurality of images whose parallax or focus positions are shifted, and the scale in the optical axis direction of the depth map information is replaced with parallax, and the depth map information is set so that the amount of parallax is evenly spaced. Depth map information acquisition conversion means for converting to data, area dividing means for dividing an image that has been arbitrarily focused by the synthetic aperture method, and the amount of parallax from the focus position with respect to the image for each of the divided areas An image generating apparatus comprising: convolution means for convolving a corresponding blur kernel (blurring function); and a blurred image generating means for generating a blurred image by integrating the images after the convolution. The image generation apparatus according to claim 12, wherein
The plurality of in-focus images are images generated by a multi-lens camera including a plurality of cameras (a plurality of imaging units having an image sensor), and parallax (parallax = between two points on the pupil plane ( An image generation apparatus using an image obtained at a distance to a subject whose distance between the optical axes of a plurality of cameras x imaging magnification) is an integral multiple of the pixel pitch of the image sensor. The image generation apparatus according to claim 12, wherein
The plurality of in-focus images are images generated by a multi-lens camera including a plurality of cameras (a plurality of imaging units having an image sensor), and parallax (parallax = between two points on the pupil plane ( Alternatively, an image obtained at a distance to a subject in which the distance between the optical axes of a plurality of cameras) × image forming magnification) is an integral multiple of the pixel pitch of the output image output from the camera (imaging device) is used. A featured image generation apparatus.