JP4437228B2 - Imaging apparatus and imaging method using defocus structure - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method. Specifically, the present invention relates to an imaging apparatus and an imaging method for generating a visual image corresponding to an arbitrary viewpoint, aperture, and focusing from an image group having a defocused structure related to an imaging target.

In the basic method of X-ray tomography using a computer, parallel X-rays are imaged by rotating 360 degrees around the human body. As a result, information viewed from all angles is accumulated in a series of images, and a multi-layer human tomographic photograph viewed from an arbitrary direction can be obtained based on the information. In this way, by photographing from around the object using parallel transmitted light, an in-focus image of an arbitrary tomogram viewed from an arbitrary direction can be generated.
On the other hand, what about a photographed image by a camera that does not transmit an object and uses non-parallel visible light?

  As for a photographed image by a camera, an image generation method by light field rendering is known. For example, the imaging target is imaged from different viewpoints with cameras arranged in a two-dimensional plane, and 2D × 2D captured image data is acquired. Based on these data, a computer calculation can be performed to generate a fluoroscopic image viewed from an arbitrary viewpoint and line-of-sight direction on the camera side.

  Further, the inventor uses multi-focus imaging (MFI) that moves the imaging surface of the camera to acquire a group of captured images with different focusing, and generates an image by computer calculation based on the group. An omnifocal image is generated. (See Non-Patent Document 1)

  In addition, the inventors relate a plurality of captured images with different focusing and generated images by an identity equation including a constant representing convolution by a blur function, and generate an arbitrary focused image from captured images with different focusing Has proposed. In addition, Aizawa et al. Proposed a method of generating arbitrary focused images from captured images with different focus by associating multiple captured images with different focus and generated images using a filter function including blur parameters. ing. (See Patent Documents 1 and 2)

FIG. 17 shows a plurality of images (five in FIG. 17) with different focuss taken by a plenoptic camera proposed by Stanford University.
FIG. 18 is a diagram for explaining the plenoptic camera. A two-dimensional space is divided into, for example, 4 × 4 areas and a lens array or the like is arranged to function like a multi-view camera (equivalent to viewing an object from 4 × 4 different viewpoints). In other words, it can be said that a four-dimensional space is realized on a computer by further forming a two-dimensional space in the two-dimensional space. Therefore, it is possible to create a three-dimensional MFI image from this by computer calculation. That is, it is possible to generate images with different focusing as shown in FIG. (See Non-Patent Document 2)

JP 1999-295826 (paragraphs 0018 to 0033, FIG. 1) JP 2001-223874 A (paragraphs 0023 to 0097, FIGS. 1 to 13) Examination of image generation based on integrated conversion processing of a large number of focused images, Kazuya Kodama, 8th Symposium on Video Media Processing Symposium, p73-74, November 12-14, 2003 Light Field Photography with a Hand-Held Plenoptic Camera Ren Ng (Stanford Univ.) Et al. [Online], April 2005 [August 24, 2005 search], Internet <URL: http://graphics.paperford.edu. / lfcamera />

  However, the MFI method has not yet been able to generate a visual image corresponding to an arbitrary viewpoint, aperture, and focusing, and a plurality of captured images with different focusing and generated images can be related using an identity or a filter function. In both methods, the photographed image and the generated image are directly associated as two-dimensional information to find a solution of simultaneous equations, but an identity or a filter function that completely reflects the various three-dimensional information of the target is concrete. Is not defined, nor is a method for obtaining a complete solution in that case. In addition, a plurality of visual image images with different focusing are obtained using a plenoptic camera, but visual images are obtained from four-dimensional information, and the amount of information is excessive, so that the processing amount is large. There was a problem. In addition, there is a problem in that the resolution is poor because the photographing surface is divided into a plurality of parts with one camera.

An object of the present invention is to provide a novel imaging apparatus and imaging method capable of generating a visual image corresponding to an arbitrary viewpoint, aperture, and focusing from a group of captured images having a defocused structure related to an imaging target.
Another object of the present invention is to obtain a visual image from three-dimensional information to reduce the processing amount and improve the resolution.

  The present invention generates a visual image corresponding to an arbitrary viewpoint, line-of-sight direction, aperture, and focusing from an image group having a defocus structure. This can be realized by combining light field rendering (LFR) and multifocus imaging (MFI) described above. That is, the line-of-sight direction is changed using the LFR from the line-of-sight direction of the multifocal image group in the MFI, and converted to an image from an arbitrary viewpoint + line-of-sight direction.

In order to solve the above problems, an imaging apparatus 1 using a defocusing structure according to a first aspect of the present invention, for example, as shown in FIG. 10, captured image group g _n with defocusing structure for capturing object Q ( X and Y) are integrated and Fourier transformed to form an integrated Fourier transform photographed image G (u, v, w), and an integrated Fourier transform photographed image G (u, v, w). Extraction of frequency components corresponding to an arbitrary viewpoint, aperture, and focusing by using a three-dimensional or higher-dimensional filter processing related to defocusing to form a selective Fourier transform image A (u, v, w) a means 5, selective Fourier transform image a (u, v, w) the inverse Fourier transform and cut any viewpoint, iris, visual images on imaging target Q corresponding to focusing a _{n (X,} Y) Raw Inverse Fourier transform / cutout means 6 is provided.

  Here, the photographed image group has a defocused structure. The photographed image group is systematically photographed by changing the focus, changing the viewpoint, etc., and by integrating the photographed images, the blur state is the integrated image. It means that there is almost no change at each position. Further, the change of the photographed image group is not limited to focusing only and only the viewpoint, but both may be changed. The filter related to defocusing refers to a filter representing the degree of defocusing obtained on a plurality of imaging planes when shooting is performed so that a plurality of real space planes are in focus. Further, the filter processing is specifically shown in the specification as (Equation 2) and (Equation 3), but is not limited to such a function based on the Gaussian distribution. For example, when changing the viewpoint, a fisheye lens When a lenticular lens is used, it can be a different function such as a cylindrical shape, a stepped cylindrical shape, or a deformation thereof. Further, the visual image image means a visually perceivable image generated by computer calculation, and includes not only a still image but also a moving image and a color image.

  If comprised in this way, the novel imaging device which produces | generates the visual image image equivalent to arbitrary viewpoints, aperture | diaphragm | squeeze, and focusing from the picked-up image group which has a defocus structure regarding a target object can be provided. Further, it is possible to obtain a visual image from the three-dimensional information, reduce the processing amount, and improve the resolution.

Further, according to the second aspect of the present invention, in the imaging apparatus using the defocus structure according to the first aspect, the photographed image group g _n (X, Y) having the defocus structure is photographed with different focus. It is composed of an image group or an image group taken by changing the viewpoint. If comprised in this way, the image group which has a defocus structure can be obtained reliably and easily.

In order to solve the above problem, the imaging method using the defocus structure according to the third aspect of the present invention is, for example, as shown in FIG. 11, a captured image group g _n (X , Y) to obtain an integrated Fourier transform photographed image G (u, v) by integrating and Fourier transforming the image group obtaining step (S001) for obtaining and the photographed image group g _n (X, Y) obtained in the image group obtaining step. , W), a Fourier transform / integration step (S003), and an integrated Fourier transform photographed image G (u, v, w) using a three-dimensional or more filter processing related to defocusing, an arbitrary viewpoint, aperture, A frequency component extraction step (S004) for extracting a frequency component corresponding to focusing and forming a selective Fourier transform image A (u, v, w) and a selective Fourier transform image A (u, v, w) are performed. Inverse Fourier transform Conversion and cut comprising, an arbitrary viewpoint, iris, visual images a _{n (X,} Y) on imaging target Q corresponding to the focusing and an inverse Fourier transform-cutting step (S005) of generating.

  If comprised in this way, the novel imaging method which produces | generates the visual image image corresponded to arbitrary viewpoints, aperture | diaphragm | squeeze, and focusing from the picked-up image group which has a defocus structure regarding a to-be-photographed object can be provided. Further, it is possible to obtain a visual image from the three-dimensional information, reduce the processing amount, and improve the resolution.

According to a fourth aspect of the present invention, in the imaging method using the defocus structure according to the third aspect, a photographed image group g _n (X, Y) having a defocus structure is photographed at a different focus. It is composed of an image group or an image group taken by changing the viewpoint. If comprised in this way, the image group which has a defocus structure can be obtained reliably and easily.

The fifth aspect of the present invention is an invention of a program for causing a computer to execute the imaging method using the defocus structure according to the third aspect or the fourth aspect .

ADVANTAGE OF THE INVENTION According to this invention, the novel imaging device and imaging method which can produce | generate the visual image image corresponded to arbitrary viewpoints, aperture_diaphragm | restriction, and focusing from the picked-up image group which has a defocus structure regarding imaging | photography object can be provided.
Further, it is possible to obtain a visual image from the three-dimensional information, reduce the processing amount, and improve the resolution.

[First Embodiment]
In this embodiment, an example of creating an image corresponding to an arbitrary viewpoint, aperture, and focusing by rendering using a three-dimensional filter process related to defocusing from a group of images taken with different focusing. Will be explained.
First, the principle of the invention will be described.

  FIG. 1 is a diagram for explaining the relationship between perspective coordinates and orthogonal coordinates. The real space is originally expressed in a perspective coordinate system, but most of the processing of the present invention is expressed in an orthogonal coordinate system to which transformation is applied. FIG. 1 (a) represents real space, where the aperture of the camera is assumed to be a pinhole, and the line of sight from the camera in real space is indicated by an arrow. FIG. 1B shows the space after coordinate conversion. In the real space, the camera pinhole is set as the origin O (0, 0, 0), and in the space after coordinate conversion, the position of the focal length on the optical axis of the camera is set as the origin O (0, 0, 0) (this specification). Except here, O indicates the origin of the perspective coordinate system, and in the drawing indicates the origin of the perspective coordinate system and the orthogonal coordinate system). In perspective coordinates, the left-right direction is x, the up-down direction is y, the front-rear direction is z (forward is +), the left-right direction is X, the up-down direction is Y, and the front-rear direction is Z (forward is +) in orthogonal coordinates. The coordinates of the perspective coordinate system are represented by (x, y, z), and the coordinates of the orthogonal coordinate system are represented by (X, Y, Z).

  The line of sight when viewing coordinates (p, q, f) (f is the focal length) from the camera is a straight line (x, y) = (z / f) * (p, q) (x = (z / f) in real space. ) * P, y = (z / f) * q and the following two expressions are shown together, and the following expression is also used below (* is a multiplication operator)), and after coordinate conversion In this space, (X, Y) = (p, q), corresponding to a straight line extending in the ± Z direction, is taken as one point (p, q). That is, in the Cartesian coordinate system, the line of sight at infinity is a parallel line extending in the ± Z direction. Q1 and Q2 are objects to be photographed, and even in the real space, the one closer to the camera is greatly represented even if the dimensions are the same. With respect to the optical axis direction of the camera, the imaging target on the xy plane at distances f, Nf / (N−1),..., Nf / i,. In the orthogonal coordinate system, 0, 1 / N, 2 / N,..., (N−i) / N,..., (N−2) / N, (N−1) / N, 1 and the Z direction from the origin Conversion is performed on an XY plane arranged at equal intervals. In the perspective coordinate system and the orthogonal coordinate system, coordinate conversion is performed such that (X, Y) = (f / z) * (x, y), Z = (1−f / z).

  FIG. 2 shows the relationship between perspective coordinates and orthogonal coordinates when the pinhole of the camera of FIG. 1 is replaced with a lens. 2A shows a real space (perspective coordinate system), and FIG. 2B shows a space (orthogonal coordinate system) after coordinate conversion. In addition to the line of sight viewed from the origin on the lens, the line of sight viewed from the point (s, t, 0) is also shown. Further, (a) also shows the rays on the image (Image) side (−z side) together with the scene (Scene) side (+ z side). In the perspective coordinate system of FIG. 2A, the line of sight viewed from the point (s, t, 0) is (s, t, 0) on the + z side as compared to the line of sight viewed from the origin O (0, 0, 0). ). Rays passing through the origins O (0, 0, 0) and (p, q, f) and rays passing through the points (s, t, 0) and (p + s, q + t, f) parallel to this have focal length −f. Since they coincide in the plane, rays passing through the points (s, t, 0) and (p + s, q + t, f) are (x, y) = (z / f) * (p, q) + ( s, t), and on the negative side, (x, y) = (z / f) * (p + s, q + t) + (s, t). In the perspective coordinate system and the orthogonal coordinate system, when z ≧ f, (X, Y) = (f / z) * (x, y), Z = (1−f / z), and z ≦ −f, X, Y) = (f / z) * (x, y), and Z = −f / z.

In the Cartesian coordinate system of FIG. 2B, the line of sight that looks at infinity when the viewpoint is at the origin O (0, 0, 0) and at the point (s, t, 0) matches at z = ∞. The line of sight of the point (p + s, q + t, f) from the viewpoint (s, t, 0) is a straight line

(X, Y) = (p, q) + (1-Z) * (s, t) (Equation 1)

It is represented by Further, the parallel luminous flux in the perspective coordinate system passes through a lens having _a radius r _a (assuming that the current point (s, t, 0) is on this circumference) from the origin O (0, 0, 0), and the focal length. Focus on the focal plane of -f. This parallel light beam is centered on a straight line extending in the Z direction (X, Y) = (p, q) in the orthogonal coordinate system, and the straight line of (Equation 1) is inside a cone rotated around this axis. expressed.

  Using a lens will contain information about the luminous flux. Then, there are a large number of surfaces having different z values in the imaging space, and when focusing on a certain surface, defocusing occurs on the other surface, but the blur information includes light flux information. There is a possibility that real space information included in the information can be extracted.

  FIG. 3 shows the relationship between light from infinity and a group of planes arranged in the Z direction in the orthogonal coordinate system. FIG. 3 is an MFI (multi-focus imaging) model. FIG. 3 combines both the + side (Scene side) and the − side (Image side). In the figure, a plurality of (N + 1 layers) XY planes arranged in the Z direction are represented by Z = 0, 1 / N, 2 / N,..., (N−i) / N,. ) / N, (N−1) / N, 1. In the perspective coordinate system, z = f, Nf / (N−1),..., Nf / i,..., Nf / 2, Nf, ∞ on the + side, and z = −∞, − on the − side. This corresponds to the positions of Nf, -Nf / 2, ..., -Nf / i, ..., -Nf / (N-1), -f. On the + side, the light beam from infinity can be seen as the shaded portion between the arrows, and on the-side, the light beam after passing through the lens can be seen as the shaded portion between the arrows.

A parallel light beam from infinity passes through a lens having a radius r _a (the point (s, t, 0) is on the circumference) centered on the origin O. As shown in FIG. 3 in the orthogonal coordinate system, this light beam has a radius r _a (point (point (Z)) with the line connecting the point (p, q) and infinity (Z = 1) as the central axis. p + s, q + t) is represented inside a cone having a circle on this circumference. After passing through the lens, this parallel light beam is condensed on the focal point of the plane (Z = 1) having a focal length −f, and then spreads gradually on the plane of Z = 0 (corresponding to z = −∞) on the bottom surface (Z = 0) and a circle with a radius r _a centered on the point (p, q) (the point (p + s, q + t) is on this circumference). That is, the luminous flux is expressed inside a cone having _a circle connecting the point (p, q) and the focal point (Z = 1) as the central axis and having a bottom surface (Z = 0) and a radius ra. Therefore, the XY plane group arranged in the Z direction intersects with a parallel light beam from infinity on a + side (Scene side) and has information on the parallel light beam in the circle, and also on the − side (Image side) ) Intersects with a parallel light beam after passing through the focal point (−f) in a circle, and a photographed image having these surfaces as imaging surfaces has information on the parallel light beam in this circle.

In this way, parallel light from infinity is collected at the focal point, but in the real space, perspective coordinate system, z = Nf, Nf / 2,..., Nf / i,. When there is an object (photographing target) on the plane, reflected light from the object is generated, and Z = (N−1) / N, (N−2) / N,. An image is formed on the XY plane of (N−i) / N,..., 2 / N, 1 / N. In these cases, the light flux from the reflected light from the object is Z = (N−1) / N, (N−2) / N,..., (Ni) / N,. It is represented by a cone extending on both sides of the +/-, 0 plane. Therefore, on the + side (real space side), each XY plane intersects with the reflected light from the object in a circle, and has information on the object (photographing target) in the circle. On the negative side (imaging space side), each XY plane intersects with the lens transmitted light and a circle, and a photographed image having these surfaces as the imaging surface has information on an object (photographing target) in this circle. . Here, the information (brightness) of the n-th XY plane in real space is f (X, Y, Z _n ) = f _n (X, Y), and is photographed with the focus on the n-th XY plane. An acquired image obtained on the XY plane of the nth layer in the imaging space is assumed to be g (X, Y, Z _n ) = g _n (X, Y). Note that Z _n = n / N.

In the MFI model, all rays through the lens are used, so that the light flux is continuous inside the cone and contains all the information in the cone. Thus, a visual image corresponding to an arbitrary viewpoint, aperture, and focusing can be generated by rendering based on N acquired images g _n (X, Y) having different focal points.

FIG. 4 shows the relationship between real space information (+ z side) (Scene) and photographed image (−z side) (Image) group via a three-dimensional blur filter. FIG. 4A schematically shows the relationship between real space information including a subject to be photographed, a photographed image, and a three-dimensional blur filter, and FIG. 4B schematically shows the concept of the three-dimensional blur filter. The integrated photographed image g (X, Y, Z) obtained by integrating the real space information f (X, Y, Z) and the photographed image group g _n (X, Y) is related by the filter h (X, Y, Z). Attached. That is, the integrated photographed image g (X, Y, Z) is obtained by applying the filter h (X, Y, Z) to the real space information f (X, Y, Z). The captured image group g (X, Y, Z _n ) = g _n (X, Y) is an image group captured by changing the focusing, that is, by changing the imaging surface, and the object on the focused plane is clear. As a simple image, an object on a plane that is out of focus is photographed as a blurred and unclear image. The captured image group _g n (X, Y) is obtained by integrating the integrated photographic image g (X, Y, Z) and said. The filter represents the degree of defocusing obtained on each XY imaging surface (1 = <n = <N) when the XY plane (1 = <n = <N) is photographed so as to be in focus. It is. Schematically, it is represented by a cone extending on both sides of the +/− side with the in-focus plane as the apex. That is, as shown in FIG. 4B, the closer to the focal point, the smaller the circle that intersects the XY plane and the lower the degree of blur, and the farther from the focal point, the larger the circle that intersects the XY plane and the greater the degree of blur.

In geometric optics, the blur matrix that associates multiple imaging planes with different focus points is

It is expressed. If this blur matrix is expressed as a filter for convolution,
It is expressed.
Expressing this in the frequency domain,
It is expressed.

FIG. 5 schematically shows the relationship between real space information (Scene) f (X, Y, Z), a photographed image (Image) group g (X, Y, Z _n ), and a filter h (X, Y, Z). . FIG. 5A shows an MFI model. An object (photographing target) Q having a three-dimensional shape (in this case, the shape changes in the X and Z directions and is uniform in the Y direction) is indicated by a bold line, and is focused on each XY plane aligned in the Z direction. The state of the light beam when In FIG. 5 (a), the focal point is indicated by the intersection of rays indicating the observation direction, and the rays are continuous in the light flux. The spread of the light beam represents the degree of blur of the photographed image group g (X, Y, Z _n ). It can be considered that a blank area where f (X, Y, Z) = 0 is blurred.

FIG. 5B shows a process of combining the real space information f (X, Y, Z) and the photographed image group g (X, Y, Z _n ) with a filter h (X, Y, Z). That is, a clear image can be obtained on the focused imaging surface, but an image including blur can be obtained on the unfocused imaging surface. The filter h (X, Y, Z) has such a conversion filter function.

Next, consider the nature of the three-dimensional blur filter h (X, Y, Z). Consider a captured image group g (X, Y, Z _n ) corresponding to real space information f (X, Y, Z). For example, when the real space f (X, Y, Z) is observed from one direction, it can be clearly observed on an imaging surface that is orthogonal to the line of sight and is in focus, and is blurred and observed on a surface that is not in focus. This is applied to each imaging surface where n = 1 to 5, for example. Such a filter is h (X, Y, Z).

This relationship also applies in Fourier space. The Fourier space corresponding to the real space information f (X, Y, Z) is F (u, v, w), and the integrated Fourier transform imaging is performed in the Fourier space corresponding to the captured image group g (X, Y, Z _n ). The image is G (u, v, w), and the filter in the Fourier space corresponding to the filter h (X, Y, Z) is H (u, v, w). In the Fourier space, the filter H (u, v, w) is expressed by (Equation 3), and a frequency region that can be satisfactorily observed is extracted. As described later, in the Fourier space, a specific frequency component is extracted on the basis of the integrated imaging space g (X, Y, Z) by being expressed by a frequency component, and an arbitrary calculation is performed by computer calculation. Viewpoint, focus and aperture images can be generated.

The Fourier transform image F (u, v, w) is a Fourier transform image of the real space information f (X, Y, Z). The integrated Fourier transform photographed image G (u, v, w) is obtained by subjecting the photographed image group g _n (X, Y) of each layer to integration processing by computer calculation or the like and Fourier transform.

  FIG. 6 shows an example of frequency characteristics of a three-dimensional blur filter. FIG. 6A shows an integrated Fourier transform image G (u, v, w) obtained by applying a three-dimensional blur filter H (u, v, w) to a Fourier transform image F (u, v, w) of real space information. ) Shows the process of extraction. The filter H (u, v, w) is expressed by (Equation 3). In the MFI model, this filter extracts a frequency component related to a plane group perpendicular to the light beam. FIG. 6B shows an example of the filter H (u, v, w) when the lens blur has a Gaussian distribution. At the origin O (0, 0, 0), it is extracted by a delta function, and on the uv plane, it is extracted because H (u, v, w) ≠ 0. However, on the w axis, H (u, v, Since w) = 0, it is not extracted. FIG. 6C shows an example of a filter H (u, v, w) in the case of a pinhole camera, and it is extracted only on the uv plane.

In FIG. 7, the origin O (0, 0, 0) (hereinafter abbreviated as (0, 0)) is shifted from (s, t, 0) (hereinafter abbreviated as (s, t)). It is a figure for demonstrating the case where it image | photographed with the virtual camera which has an iris of radius r from the position where it was. FIG. 7A obtains a visual image group a (X, Y, Z _n ) = a _n (X, Y) having different focusing in the imaging space from the real space image f (X, Y, Z). , Filter h _a (X, Y, Z; r, s, t) corresponds to the light beam passing through the lens passing through the inclined cone region as shown in FIG. Note that the selectively generated image a (X, Y, Z) is obtained by integrating these visual image groups. FIG. 7B obtains a selective Fourier transform image A (u, v, w) obtained by extracting a specific frequency component from the Fourier transform image F (u, v, w). The filter is a filter H _a (u, v, w; r, s, t) that extracts a frequency region related to the inclined plane group as shown in FIG. In this way, even when shooting with a virtual camera having an iris (aperture) of radius r from a position (s, t) deviated from the origin O (0, 0), a specific frequency component is extracted using a filter. Thus, the selective Fourier transform image A (u, v, w) can be obtained, and if the selective Fourier transform image A (u, v, w) is further subjected to Fourier inverse transform, the selectively generated image a (X, Y) is obtained. , Z) can be generated, and a visual image a _n (X, Y) = a (X, Y, Z _n ) corresponding to an arbitrary viewpoint, aperture, and focusing can be obtained by cutting this out with an arbitrary XY plane. It can be said that it can be generated.

Here, the three-dimensional blur filter when the virtual camera is used to capture the image from the viewpoint (s, t) with the iris r is expressed by (Expression 4), and is expressed by (Expression 5) in the frequency space.

FIG. 8 shows the relationship between a captured image G (u, v, w) acquired by shooting and a generated image (selective Fourier transform image) A (u, v, w) generated by computer calculation in Fourier space. Indicates. Using a Fourier transform image of real space information f (X, Y, Z), F (u, v, w), and an iris with _a radius ra, the captured image is G (u, v, w). ), And the filter at that time is generated by computer calculation on the assumption that the image was taken with a camera having an iris of radius r from a position (s, t) deviated from the origin (0, 0), H (u, v, w) If the selective Fourier transform image is A (u, v, w) and the filter at that time is H _a (u, v, w; r, s, t),

G (u, v, w) = H (u, v, w) F (u, v, w) (Formula 6)
A (u, v, w) = H _a (u, v, w; r, s, t) F (u, v, w) (Expression 7)

Is established.

Thereby, in the frequency domain, the relationship between the captured image G (u, v, w) and the selective Fourier transform image (a Fourier transform of the visual image image) A (u, v, w) is

A (u, v, w) = H _a (u, v, w; r, s, t) H (u, v, w) ⁻¹ G (u, v, w) (Equation 8)

It becomes. This means that a selectively generated image a (X, Y, Z) corresponding to an arbitrary viewpoint, aperture, and focusing can be generated from an image acquired by MFI.

FIG. 9 shows conditions when a visual image can be generated. This is that can be generated when information of the photographed image group g _{n (X,} Y) of the plurality of acquired visual images produced during information a _{n (X,} Y) is present. That is, H _a (u, v, w; r, s) is generated by the three-dimensional blur filter of the generated image in an area where the three-dimensional blur filter related to the acquired captured image is H (u, v, w) = 0. , T) ≠ 0, if there is an area where H _a (u, v, w; r, s, t) H (u, v, w) ⁻¹ is not determined, an image cannot be obtained. In addition, if there is a region where H _a (u, v, w; r, s, t) ≠ 0 exists only in _a region where H (u, v, w) ≠ 0, H _a ( u, v, w; r, s, t) H (u, v, w) -1 is Sadamari, it is possible to produce visual images _a n (X, Y).

FIG. 10 shows a configuration example of the imaging apparatus in the present embodiment.
In FIG. 10, Q is a subject to be photographed. Reference numeral 1 denotes an imaging device, which stores an arithmetic control device 2 that performs arithmetic processing and control processing, an imaging device as an image acquisition means 8 that acquires a captured image, an acquired captured image, and an image generated by an arithmetic process and an arithmetic result Image storage means 9, an acquired photographed image, an image display means 10 for displaying an image generated by a calculation process or a calculation result, and the like. The arithmetic and control unit 2 integrates and Fourier transforms the captured image group g _n (x, y) having a defocus structure to form an integrated Fourier transform captured image G (u, v, w). 3. Extracting frequency components suitable for any viewpoint, aperture, and focusing from the integrated Fourier transform photographed image G (u, v, w) using filter processing, and selectively Fourier transform image A (u, v, w) The visual component image of the subject Q corresponding to an arbitrary viewpoint, aperture, and focusing by performing inverse Fourier transform and cutting out the selective Fourier transform image A (u, v, w) forming the frequency component extraction means 5 forming w) Inverse Fourier transform / cutting means 6 for generating a _n (X, Y), control means 7 for controlling each means constituting the imaging apparatus 1 to perform the function as the imaging apparatus 1, etc. . The imaging device 8 is variably configured position of the imaging surface in response to the focus, different photographed images g _n of focusing _(x, y), i.e., capable of photographing a photographed image group having a defocusing structure is there.

FIG. 11 shows an example of a processing flow of the imaging method in the present embodiment.
First, a photographic image group g _n (x, y) having a defocused structure related to the photographic subject 1 is photographed and obtained by the imaging device 8 as an image acquisition means (image acquisition step: step S001). The imaging surfaces are adjusted so as to be equally spaced in the orthogonal coordinate system. Next, the captured image group g _n (x, y) having a defocused structure acquired by imaging is stored in the image storage unit 9 (acquired image storage step: step S002). The storage in the image storage means 9 can be performed at the stage when the calculation process of the calculation control device 2 and the calculation result are obtained. Next, these photographic image groups g _n (x, y) are integrated and coordinate-transformed by the Fourier transform / integration means 3 to obtain an integrated photographic image g (X, Y, Z), and then the integrated photographic image. g (X, Y, Z) is subjected to Fourier transform to obtain an integrated Fourier transform photographed image G (u, v, w) (Fourier transform / integration step: step S003).

Next, the frequency component extraction unit 5 extracts and selectively selects frequency components corresponding to an arbitrary viewpoint, aperture, and focusing from the integrated Fourier transform photographed image G (u, v, w) using filter processing. A Fourier transform image A (u, v, w) is formed (frequency component extraction step: step S004). Specifically, using (Equation 8), H _a (u, v, w; r, s, t) H (u, v, w) ⁻¹ is applied to the integrated Fourier transform photographed image G (u, v, w). To generate a selective Fourier transform image A (u, v, w). Here, it is not always necessary to obtain the Fourier transform image F (u, v, w) of real space information, and H _a (u, v, w; r, s, t) H (u, v, w) ^−1. Act as one operator. Note that the Fourier transform image F (u, v, w) of the real space information may be obtained, but its accuracy depends on the number of layers and the layer interval of the photographed image group g _n (X, Y). A certain amount of spread occurs.

Next, the selective Fourier transform image A (u, v, w) extracted by the inverse Fourier transform / cutout means 6 is subjected to inverse Fourier transform to obtain a selectively generated image a (X, Y, Z). then, selective generated image a (X, Y, Z) the coordinate transformation, an arbitrary viewpoint, iris, visual images a _{n (x,} y) on imaging target Q corresponding to focusing the cut produce (reverse Fourier transform / cutout step: Step S005). Then, visual generated on the image display unit 10 images _a n (x, y) is displayed (Image display step: S006).

Here, in the Fourier transform / integration step (step S003), first, the photographed image group g _n (x, y) is Fourier-transformed into a Fourier transform photographed image group G _n (u, v), which is then integrated. Further, an integrated Fourier transform photographed image G (u, v, w) may be obtained by coordinate transformation. Further, the inverse Fourier transform-cutting step (step S005), selective Fourier transform image A (u, v, w) from the Fourier transform image _A n (u, v) of the visual images to extract, it inverse Fourier transform and to coordinate transformation, the visual images a _{n (x,} y) may be generated. Also, either the coordinate conversion or the integration order may be first, and the coordinate conversion or the cut-out order may be first.

FIG. 12 shows a first example of a visual image image generated from a captured image group. The leftmost column shows a plurality (64) of MFI photographed image groups g _n (X, Y) photographed with iris r _a = 1.0 and (s, t) = (0, 0). The shape of the object to be photographed has a middle part on the back side like a three-sided mirror. Via arrow to the right of these captured images _g n (X, Y), indicating these image groups _g n (X, Y) visual images are generated on the basis of _a n (X, Y). Visual images _a n (X, Y) Iris r = 0.0 in the left column of (pinhole), the case of iris r = 0.2 and the right column, all (s, t) = This is indicated by the state (0, 0). In FIG. 12, an image focused on Z = 16, 28, 40 is extracted and shown. It can be seen that in the state of r = 0.2, the blur of the image is between iris r = 0 and r = 1.

FIG. 13 shows a second example of the visual image image generated from the photographed image group. The MFI photographed image group g _n (X, Y) in the leftmost column is the same as FIG. Via arrow to the right of these captured images _g n (X, Y), indicating these image groups _g n (X, Y) visual images are generated on the basis of _a n (X, Y). Images with different viewpoints of 5 rows × 5 columns could be obtained. For these visual image images, iris r = 0.0, and s and t indicate five stages of -0.5, -0.25, 0, +0.25, and +0.5, respectively. The left column is seen from the left side of the subject Q, the right column is seen from the right side of the subject Q, the upper row is seen from the upper side of the subject Q, and the lower row is seen from the lower side of the subject Q. I understand.

FIG. 14 shows a third example of the visual image image generated from the photographed image group. The MFI photographed image group g _n (X, Y) in the leftmost column is the same as FIG. Via arrow to the right of these captured images _g n (X, Y), indicating these image groups _g n (X, Y) visual images are generated on the basis of _a n (X, Y). Three rows of visual image images could be obtained. In FIG. 14, an image focused on Z = 16, 28, 40 is extracted and shown. For these visual image images, iris r = 0.2, (s, t) = (+ 0.5, 0.0) in the left column and (s, t) = (0. 0, -0.5), and the right column is an image at (s, t) = (-0.25, +0.25). It can be seen that each row is slightly blurred from the right side, lower side, and upper left side of the subject Q.

  As described above, it can be understood that a visual image corresponding to an arbitrary viewpoint, aperture, and focusing can be generated from a group of captured images having a defocus structure.

[Second Embodiment]
Next, a second embodiment will be described.
In the present embodiment, an example will be described in which an image corresponding to an arbitrary viewpoint, aperture, and focusing is generated from a group of images taken with different viewpoints by rendering using filter processing.

  FIG. 15 is a diagram for explaining an imaging method according to the present embodiment. This is an example in which wide-angle cameras are photographed so as to overlap in a horizontal row. Thereby, it is possible to acquire a group of captured images observed from different viewpoints. When these captured images having different viewpoints are compared, light rays emitted in various directions from a certain point to be imaged are included. This means that a shot image taken from a different viewpoint essentially includes out-of-focus information about a certain virtual lens. A group of images can be constructed. In this case, since it is preferable that an arbitrary point to be captured is included in a captured image as wide as possible, it is desirable to use a wide-angle camera, and it is preferable to capture from a viewpoint as wide as possible. As described above, it is possible to generate a visual image corresponding to an arbitrary viewpoint, aperture, and focusing based on a group of captured images having different viewpoints. For example, it is possible to generate a visual image image viewed from the viewpoint indicated by an arrow. Even if the wide-angle cameras are arranged in a two-dimensional array on the plane, a group of captured images having a defocused structure can be formed, and a visual image image corresponding to an arbitrary viewpoint, aperture, and focusing can be generated.

[Third Embodiment]
Next, a third embodiment will be described. Similarly to the second embodiment, this embodiment also generates an image corresponding to an arbitrary viewpoint, aperture, and focusing by rendering using a filter process from a group of images taken with different viewpoints. It is an example.

  FIG. 16 is a diagram for explaining an imaging method according to the third embodiment. An example in which a plurality of wide-angle cameras described in the second embodiment are arranged in an annular shape so as to surround a subject to be photographed is shown. Thereby, it is possible to acquire a large number of photographed images of the photographing object observed from different viewpoints. In addition, information from 360 degrees around the subject to be imaged can be obtained. Based on these photographed image groups having different viewpoints, it is possible to generate visual image images corresponding to the viewpoint, aperture, and focusing from around 360 degrees around the photographing object. For example, it is possible to generate a visual image image viewed from the viewpoint indicated by a triangle in the figure. In addition, cameras of various sizes may be mixed, and the camera may be arranged so as to surround the object to be photographed even if it is not in an annular shape. It may be arranged so as to surround it. Moreover, even if the range of visualization is less than 360 degrees, it is sufficient to surround only that portion in an arc shape.

[Fourth Embodiment]
In this embodiment, the first embodiment is combined with the second embodiment or the third embodiment. In other words, a visual image corresponding to an arbitrary viewpoint, aperture, and focusing is created by rendering using a filter process from a plurality of images taken with different focus and viewpoint regarding a subject to be photographed. .
For example, wide-angle cameras are photographed so as to overlap in a horizontal row, and each wide-angle camera is photographed while changing the focus while moving the imaging surface. As a result, a group of photographed images obtained by combining the first embodiment and the second embodiment is obtained, and based on these photographed image groups, a visual image corresponding to an arbitrary viewpoint, aperture, and focusing is obtained. An image can be generated.

[Fifth Embodiment]
In Non-Patent Document 2, it has been demonstrated that a plurality of images with different focusing can be obtained by performing rendering based on captured images captured using light rays from a plurality of directions. Therefore, if image groups with different focusing can be obtained using captured images from a plurality of viewpoints, a vision corresponding to an arbitrary viewpoint, aperture, and focusing can be obtained by the principle and method described in the first embodiment. An image can be generated.

[Sixth Embodiment]
In the above embodiments, the description has been made assuming monochromatic light, but a chromatic image of any viewpoint, aperture, and focus may be generated using a plurality of light beams having different colors. Of course, it is possible to adopt the same viewpoint, aperture, and focusing for light beams with different colors, but it is also possible to generate images with different viewpoints, apertures, and focal points for each color. In this case, a Fourier transform / integration step, a frequency component extraction step, and an inverse Fourier transform / cutout step are performed for each color. Thereby, the visual image image of a structure and atmosphere different from the past can be provided.

[Seventh Embodiment]
In the above embodiment, the description has been made assuming a still image, but it can also be applied to a moving image. For example, it is possible to obtain a dynamic image by generating a visual image image by temporally changing at least one of the viewpoint or the focusing. In addition, a video camera is used as the image acquisition unit 8 as an image pickup device, and a group of captured images taken by changing the viewpoint or focus in a time-series manner with respect to the dynamic behavior of the shooting target is stored in the image storage unit 9. In addition, a dynamic visual image corresponding to an arbitrary viewpoint, aperture, and focusing can be generated by providing moving image editing means and editing dynamic behavior.
In this case, since a dynamic visual image corresponding to an arbitrary viewpoint, aperture, and focusing can be generated with a small amount of information compared to a plenoptic camera, high-speed processing is possible. It is understood that it is suitable for.

  The present invention can also be realized as a program for causing a computer to execute the imaging method using the defocus structure in the above-described embodiment, and also as a computer-readable recording medium on which the program is recorded. The program may be used by being recorded in a ROM built in the computer, or may be recorded on a recording medium such as an FD, CD-ROM, built-in or external magnetic disk, read by the computer, and used via the Internet. May be downloaded to a computer and used.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made to the embodiments without departing from the spirit of the present invention. It is obvious.

For example, in the above embodiment, the filter processing has been described with respect to the examples specifically shown in (Expression 2) and (Expression 3), but is not limited thereto. For example, when changing the viewpoint, a fisheye lens, When a lenticular lens is used, the functions are different (for example, not limited to the Gaussian distribution but a cylindrical shape, a stepped cylindrical shape, or a modified version thereof). Moreover, although the above embodiment demonstrated the example which does not obtain | require the Fourier-transform image F (u, v, w) of real space information, real space is obtained from integrated Fourier-transform imaging | photography image G (u, v, w). Fourier transform image F (u, v, w) the determined information, which the inverse Fourier transform to the f (X, Y, Z) , and now synthesized visual images _a n (X, Y), the coordinate transformation to, visual images a _{n (x,} y) may be generated, it is also possible to modify these sequence. Further, as a filter of four or more dimensions related to defocusing, a filter for forming two-dimensional space coordinates in two-dimensional space coordinates when changing the viewpoint, and a case of forming three-dimensional space coordinates in three-dimensional space coordinates. These filters can be considered. Moreover, although the example which moves the position of an imaging surface was demonstrated as a method of changing focusing, when mounting an imaging | photography object on a stage like a microscope, you may move a stage. Further, the photographing apparatus is not limited to a normal camera or a wide-angle camera, but can be applied to an infrared camera, a telescope, a microscope (including an electron microscope), and the like. Further, the number of captured images of the image group, the focusing interval between the captured images, the camera arrangement when changing the viewpoint, and the like can be variously changed according to the situation.

  The present invention can be used in image processing technology using a computer.

It is a figure for demonstrating the relationship between a perspective coordinate and a rectangular coordinate. It is a figure which shows the relationship between a perspective coordinate at the time of replacing the pinhole of the camera of FIG. 1 with a lens, and a rectangular coordinate. In a rectangular coordinate system, it is a figure which shows the relationship between the light from infinity and the plane group arranged in a Z direction. It is a figure which shows the relationship between real space information and a picked-up image group through a three-dimensional blur filter. It is a figure which shows typically the relationship between real space information, a picked-up image group, and a three-dimensional blur filter. It is a figure which shows the example of a frequency characteristic of a three-dimensional blur filter. It is a figure for demonstrating the case where it image | photographs with the virtual camera which has an iris of radius r from the position which shifted | deviated from the origin (s, t). It is a figure which shows the relationship between the picked-up image acquired by imaging | photography in the Fourier space, and the production | generation image produced | generated by computer calculation. It is a figure which shows the conditions in case a visual image image is generable. It is a figure which shows the structural example of the imaging device in 1st Embodiment. It is a figure which shows the example of a processing flow of the imaging method in 1st Embodiment. The 1st example of the visual image image produced | generated from a picked-up image group is shown. The 2nd example of the visual image image produced | generated from a picked-up image group is shown. The 3rd example of the visual image image produced | generated from a picked-up image group is shown. It is a figure for demonstrating the imaging method in 2nd Embodiment. It is a figure for demonstrating the imaging method in 3rd Embodiment. It is a figure which shows the some image from which the focusing differs obtained by the conventional method. It is a figure for demonstrating a plenoptic camera.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging apparatus 2 Arithmetic control apparatus 3 Fourier transform and integration means 5 Frequency component extraction means 6 Inverse Fourier transform and extraction means 7 Control means 8 Image acquisition means (imaging apparatus)
9 Image storage means 10 Image display means A (u, v, w) Selective Fourier transform image (frequency space)
a (X, Y, Z) Selectively generated image (orthogonal coordinates)
a (X, Y, Z _n ) = a _n (X, Y) Visual image image group (orthogonal coordinates)
a _n (x, y) Visual image group (perspective coordinates)
F (u, v, w) Integrated Fourier transform image (frequency space)
f (X, Y, Z) Real space image (Cartesian coordinates)
f (x, y, z) Real space image (perspective coordinates)
G (u, v, w) Integrated Fourier transform image (frequency space)
g (X, Y, Z) Integrated photographed image (Cartesian coordinates)
g (X, Y, Z _n ) = g _n (X, Y) Photographed image group (orthogonal coordinates)
g _n (x, y) Photographed image group (perspective coordinates)
H (u, v, w) Three-dimensional blur filter (frequency space) during imaging
h (X, Y, Z) Three-dimensional blur filter during imaging (real space)
H _a (u, v, w; r, s, t) Three-dimensional blur filter (frequency space) at the time of visual image generation
h _a (X, Y, Z; r, s, t) Three-dimensional blur filter (real space) when generating a visual image
N layer number n layer number O (0,0) origin (p, q) ray direction Q imaging object f the focal length _{r a} captured during the iris
r Iris (aperture) for visual image generation
(S, t) Viewpoints u, v, w Coordinates X, Y, Z in Fourier transform system x, y, z perspective coordinates

Claims (5)

And about the imaging target A photographic images taken by changing the focus or viewpoint, the captured images having a defocusing structure defocusing between the captured images is associated with more than two dimensions blur filter, Fourier transform / integration means for integrating and Fourier transforming as a group of images to form an integrated Fourier transform photographed image;
The integrated Fourier transform photographed image is subjected to filter processing using a blur filter of three or more dimensions corresponding to an arbitrary viewpoint, aperture, and focusing, and frequency components corresponding to the arbitrary viewpoint, aperture, and focusing are extracted. A frequency component extracting means for forming a selective Fourier transform image;
An inverse Fourier transform / cutout unit that performs inverse Fourier transform and cutout of the selective Fourier transform image to generate a visual image image related to the photographing object corresponding to the arbitrary viewpoint, aperture, and focusing ;
The photographed image group includes a plurality of photographed images with different focusing at equal intervals in the optical axis direction in an orthogonal coordinate system representing an imaging space;
The viewpoint and stop in the filter processing include a viewpoint and stop different from the pinhole at the center of the lens, and the filter used for the filter processing corresponds to a light beam passing through the viewpoint and stop;
An imaging device using a defocused structure. The captured image group is obtained by converting real space coordinates (x, y, z) into orthogonal coordinates (X = fx / z, Y = fy / z, Z = 1-f / z, f is a focal length). A plurality of captured images with different focusing in the Z direction which is the optical axis direction in the space;
An imaging apparatus using the defocus structure according to claim 1. A method executed by a computer;
And about the imaging target A photographic images taken by changing the focus or perspective, acquires photographed images having a defocusing structure defocusing between the captured images is associated with more than two dimensions blur filter An image group acquisition step to perform;
A Fourier transform / integration step of integrating the captured image group acquired in the image group acquisition step as a group of image groups and Fourier transforming to form an integrated Fourier transform captured image;
The integrated Fourier transform photographed image is subjected to filter processing using a blur filter of three or more dimensions corresponding to an arbitrary viewpoint, aperture, and focusing, and frequency components corresponding to the arbitrary viewpoint, aperture, and focusing are extracted. A frequency component extraction step for forming a selective Fourier transform image;
An inverse Fourier transform / cutout step of generating a visual image image relating to the photographing object corresponding to the arbitrary viewpoint, aperture, and focusing by performing inverse Fourier transform and cutout of the selective Fourier transform image ;
The photographed image group includes a plurality of photographed images with different focusing at equal intervals in the optical axis direction in an orthogonal coordinate system representing an imaging space;
The viewpoint and stop in the filter processing include a viewpoint and stop different from the pinhole at the center of the lens, and the filter used for the filter processing corresponds to a light beam passing through the viewpoint and stop;
An imaging method using a defocused structure. The captured image group is obtained by converting real space coordinates (x, y, z) into orthogonal coordinates (X = fx / z, Y = fy / z, Z = 1-f / z, f is a focal length). A plurality of captured images with different focusing in the Z direction which is the optical axis direction in the space;
An imaging method using the defocus structure according to claim 3.   A program for causing a computer to execute the imaging method using the defocus structure according to claim 3 or 4.