KR20210065030A - Method and apparatus for reconstructing three-dimensional image via a diffraction grating - Google Patents

Abstract

The present invention provides a method for reconstructing a three-dimensional image using the diffraction grating, which comprises the steps of: extracting parallax images from a raw image of an object photographed using a diffraction grating; and reconstructing a three-dimensional image from an extracted parallax image array using a virtual pinholes model.

Description

Method and apparatus for reconstructing a three-dimensional image using a diffraction grating {METHOD AND APPARATUS FOR RECONSTRUCTING THREE-DIMENSIONAL IMAGE VIA A DIFFRACTION GRATING}

The present invention relates to a three-dimensional image reconstruction method (or a three-dimensional image reconstruction method), and more particularly, to a three-dimensional image reconstruction method using a diffraction grating.

Since a Parallax Image Array (PIA) for real three-dimensional (3-D objects) contains a lot of perspective information about 3-D objects, 3D It is one of the most effective forms for imaging (3-D imaging), processing (processing) and display (display).

A captured parallax image array (PIA) is used to display the real 3D space using a lens array and a series of depth-sliced images using a computational reconstruction process. images) are converted to Accordingly, the acquisition of a parallax image array (PIA) is an essential part of 3-D imaging of the field.

In general, using a camera array or a lens array is one of the methods for picking up a PIA for a 3-D scene. one

In addition, through those optical methods, capturing a Parallax Image Array (PIA) is actively discussed in the field of integral imaging and light field analysis. It is an active topic.

Parallax Image Array (PIA) captured using a camera array compared to Parallax Image Array (PIA) captured using a lens array , which provides the advantage of high-resolution parallax images.

Higher resolution images are reconstructed from computational and optical reconstruction methods.

The high-resolution parallax images play a crucial part in a specific function such as high depth resolution.

However, the camera array may be used in other applications, such as a lens array with a camera or a moving camera with an electro-mechanical control system. Compared to the system, it is a high-cost and complicated imaging system.

Also, it sometimes requires the camera calibration and post-processing before and after picking up a PIA.).

A moving camera can be an alternative to a camera array, but is limited to use in stationary 3-D scenes.

Lens arrays are relatively inexpensive optical systems, but because they consist of a large number of single lenses, they suffer from problems such as optical aberrations and barrel distortion.

Therefore, it deserves to develop an imaging system of picking up a PIA having advantages of low-cost and high-resolution for the 3D imaging industry. for the 3-D imaging industry).

Recently, a new method through a diffraction grating has been proposed to acquire a parallax image array (PIA). This method consists of a diffraction grating plate and a camera.

The structure of the PIA-acquiring system using a diffraction grating and a camera according to the above method is similar to that of a system using a lens array having a camera (The structure of the PIA-acquiring system using a diffraction grating with a camera is similar to that of a system using a lens array with a camera.).

In addition, the optical structure of the PIA-acquisition system using the diffraction grating is cheaper and less complex than the lens array-based system.

Above all, a thin diffraction grating does not suffer from the problems in a lens array-based system.

Also, the parallax images through the diffraction grating are high-resolution. Therefore, a diffraction grating-based imaging system may be one of the promising technologies in 3-D imaging.

However, research on 3-D imaging using a diffraction grating has been focused on optically acquiring PIA, and research on computational reconstruction using a diffraction grating has not yet been discussed.

An object of the present invention for solving the above problems is to provide a 3-D image reconstruction method and apparatus using a diffraction grating.

The above and other objects, advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

The three-dimensional image reconstruction method using the diffraction grating of the present invention for achieving the above object is a method performed by a computing device including a processor, and captures parallax images diffracted by the diffraction grating using an imaging lens. Step, picking up the parallax images captured by the imaging lens on a pickup plane, rearranging the parallax images picked up on the pickup plane on a virtual image plane, and reconstructing the parallax images grown on the virtual image plane on a reconstruction plane and reconstructing a three-dimensional image of the real object by back-projection to a reconstruction plane.

According to the present invention, by acquiring a parallax image array through a diffraction grating, it is possible to develop an inexpensive and less complex parallax image array acquisition system as compared to a method using a camera array or a lens array.

In addition, in the parallax image array generated according to the present invention, since the position of the virtual pinhole and the position information and size information of each element image are well defined, it is possible to reconstruct a 3D image using the known CIIR method.

In addition, by defining the geometrical relationship between the effective object area (EOA), the picked-up PI area, the virtual pinhole, and the minimum image area (MIA), a three-dimensional image reconstruction method using a diffraction grating, which has not been discussed previously, was proposed for the first time. It has technical significance.

1A is a diagram showing an optical structure of a PIA capturing system using a camera array according to the present invention;
1B is a view showing an optical structure of a PIA capturing system using a lens array according to the present invention;
1C is a diagram showing an optical structure of a PIA capturing system using a diffraction grating that can be applied to the present invention.
2 is a schematic diagram of a relationship between a pickup process and a reconstruction process performed in a diffraction grating imaging system according to an embodiment of the present invention;
3 is a two-dimensional geometrical relationship between a point object, parallax images of a point object on a parallax image plane (PI plane) and parallax images captured by an imaging lens according to an embodiment of the present invention; A drawing showing (2-D) geometric relationships.
4 is a geometrical relationship between the path of a chief ray coming from a point object and the path of virtual rays coming from a parallax image according to an embodiment of the present invention; A drawing showing the relationship.
5 is a geometrical relationship between a point object, a virtual pinhole (VP), a virtual image plane (VI plane) and I(x _1st , y _O , z _{O ) in an embodiment of the present invention;} ) showing the drawing.
6A illustrates an effective object area (EOA), a parallax image area, picked up PI regions, virtual pinholes and half of the minimum image area (MIA) in an embodiment of the present invention; ) A diagram showing the geometrical relationship between
6B is a diagram showing how a parallax image of an original object is converted into a parallax image array for reconstruction in an actual three-dimensional space in an embodiment of the present invention;
7 is a diagram illustrating a process of back-projection to a reconstruction plane using a parallax image array (PIA) according to an embodiment of the present invention.
8 is a flowchart illustrating a 3D image reconstruction method using a diffraction grating according to an embodiment of the present invention.

The aspect in which the present invention is implemented will be described with reference to each preferred embodiment below. It is obvious that the present invention is not limited to the embodiments disclosed below, but may be implemented in other various forms within the scope of the technical spirit of the present invention. The terminology used herein is also for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprise” and/or “comprising” means that the stated component, step, action and/or element is the presence of one or more other components, steps, actions and/or elements. or added.

The present invention provides a computational reconstruction method using a raw image directly obtained from a diffraction grating image.

The reconstruction method according to the present invention includes an optical analysis for localizing each parallax image from a raw image and a reverse projection method for reconstructing a volume from an extracted parallax image. (back-projection method) can be configured.

1A is a view showing an optical structure of a PIA capturing system using a camera array related to the present invention, and FIG. 1B is a PIA capturing system (PIA) using a lens array related to the present invention. It is a diagram showing the optical structure of capturing). And, FIG. 1C is a diagram illustrating an optical structure of a PIA capturing system using a diffraction grating that can be applied to the present invention.

An optical structure of diffraction grating imaging using a diffraction grating according to the present invention is different from an optical structure based on a camera array or a lens array, as shown in FIGS. 1A to 1C . .

As shown in FIG. 1A , in an optical structure using a camera array 11, parallax images picked up on a pickup plane 12 due to a separable structure of the cameras. extraction is not necessary.

As shown in FIG. 1B, in the optical structure using a lens array 13, a lens pattern (for segmentation) of parallax images picked up on a pickup plane 14 A detection technique of the lens pattern is required, which has been widely discussed in the known literature.

As shown in Fig. 1C, in the optical structure using the diffraction grating 15, since the diffraction grating 15 is transparent and there is no pattern to be detected, it is located on the pickup plane 16 by the diffraction grating 15. Localization of the picked up parallax images is not easy. Here, the localization of the parallax images means segmentation of the parallax images.

To overcome this problem, the present invention provides 3-D computational reconstruction based on theoretical analysis of geometrical information of parallax images in diffraction grating imaging. reconstruction) method is proposed.

2 is a schematic diagram of a relationship between a pickup process and a reconstruction process in a diffraction grating imaging system according to an embodiment of the present invention.

In FIG. 2 , 'PI' denotes a parallax image, 'VP' denotes a virtual pinhole, and 'EOA' denotes an effective object area. And 'MIA' represents a minimum image area.

Referring to FIG. 2 , the diffraction grating imaging system according to an embodiment of the present invention performs a pickup process and a reconstruction process.

The diffraction grating imaging system includes a diffraction grating 41 and an imaging lens 43 for performing a silver pickup process, and a parallax image picked up on a pickup plane through the diffraction grating 41 and the imaging lens 43 and a computing device that performs a computing operation to reconstruct the picked-up parallax images.

The computing device may be configured to include a processor, a memory, a storage medium, a communication unit, and a system bus connecting them, and executes an algorithm related to a pickup process and a reconfiguration process among them, and performs various operations according to the executed algorithm The subject may be a processor.

A processor may include one or more general purpose microprocessors, digital signal processors (DSPs), hardware cores, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processors (GPUs), or any thereof. It can be implemented by the combination of

The storage medium temporarily or permanently stores an algorithm executed by the processor and intermediate data (intermediate value) and/or result data (result value) obtained from various operation processes performed according to the execution of the algorithm, or the like, or a pickup process It may be to store various known algorithms for performing the and reconfiguration process, or to provide an execution space for the algorithm.

A storage medium may also be referred to as a computer-readable medium, which includes a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), and a non-volatile (NVRAM) random access memory), electrically erasable programmable read-only memory (EEPROM), FLASH memory, a hard disk, and the like.

In the analysis of the pickup process for 3-D objects using a diffraction grating, the wavelength of the light source, the diffraction grating 41 ), and system parameters such as the positions of the diffraction grating 41 and the imaging lens 43 .

The reconstruction process is based on back-projection. Here, virtual pinholes (VPs) are defined to project parallax images into a 3-D reconstruction space. The virtual pinhole VP represents a virtual camera.

In the diffraction grating imaging system according to an embodiment of the present invention, the maximum area in which 3D objects can exist without overlapping each other in the picked-up image is defined as the effective object area (EOA), and the effective object area in the acquired images Detecting (EOA) is essential for reconstruction in diffraction grating imaging.

Also in the reconstruction process, a minimum image area (MIA) which is the minimum field of view of each virtual pinhole is defined.

In addition, for computational reconstruction, mapping between the effective object area EOA and the minimum image area MIA is required, and a related equation will be described below.

Pickup process of 3D objects in diffraction grating imaging

The diffraction grating 41 diffracts rays emanating from the 3-D object. Observing an object through a diffraction grating 41 with an imaging lens 43 is a basic concept of the pickup process in diffraction grating imaging.

Diffracted light rays by the diffraction grating 41 are observed as perspective images of the 3-D object. When the object space is observed through the diffraction grating 41 , it appears that virtual images of the object are generated on the rear surface of the diffraction grating 41 .

The virtual images have parallaxes for the 3-D object and are stored as a parallax image array (PIA) by a capturing device, such as an imaging lens 43 . Here, the imaging depth and size of the virtual object are the same as those of the original 3-D object.

3 is a two-dimensional geometrical relationship between a point object, parallax images of a point object on a parallax image plane (PI plane) and parallax images captured by an imaging lens according to an embodiment of the present invention; (2-D) It is a diagram showing geometric relationships.

Referring to FIG. 3 , it is assumed that the diffraction grating 41 is at a distance d from the imaging lens 43 in the xy plane, and the point object P1 is (x _O , y _O , z _{O ).} Here, the point object will image the objects included in the differential image to the point, z coordinates of all the differential image (the z -coordinates) are the same as z _O in accordance with the diffraction grating imaging (diffraction grating imaging) and, z is _O Imaging It may be interpreted as a distance from the lens 43 to the parallax image plane (PI plane).

, The negative first order differential image (PI (x _{-1st, O} y, _O z)) and + 1st-order differential image (PI (x _{1st, O} y, _O z)) as shown in Figure 3 is a diffraction grating ( 41) ± 1 order of the point object (± 1 ^st order) by is generated by the diffraction (diffraction). The point object at (x _O , y _O , z _O ) is used as the zero-order parallax image.

The diffraction angle (θ) between first-order parallax images (PI(x _1st , y _O , z _O )) and zero-order parallax images (PI(x _O , y _O , z _O ^{)) is θ= sin -1} ( λ/a), where λ is the wavelength of the light source and a is the aperture width of the diffraction grating.

Considering the diffraction order and the position of the point object, the x-coordinate of the parallax images is expressed by Equation 1 below.

where m is -1, 0 and 1. The y coordinate (y _O ) of the parallax image is obtained by replacing _{x O} with y _{O in Equation 1 .} An imaging point I(x _mth , y _nth , z _O ) on the pickup plane 16 is expressed by Equation 2 below.

Here m and n are -1, 0, 1, and z _I represents the z-coordinate of the pickup plane.

The diffraction grating 41 generates a diffraction grating generates parallax images in the space where the z-coordinate of the object and parallax images are the same.

Although the ray arriving at I(x _1th , y _O , z _{O ) in the pickup plane appears to come from PI (x 1th} , y _O , z _O ), only the ray emanating from the point object is real (the ray reaching I(x _1th , y _O , z _O ) in the pick-up plane seems to come from PI(x _1th , y _O , z _O ), only the rays emanating from the point object are real).

Thus, the viewpoint of each parallax image is a virtual ray from the virtual point objects P2 and P3 passing through the optical center 20 of the imaging lens 43 . It can be described by the relationship between the rays 21 , 22 and the actual ray 23 from the point object P1 .

4 is a geometrical relationship between the path of a chief ray coming from a point object and the path of virtual rays coming from a virtual point object according to an embodiment of the present invention; It is a diagram showing geometric relationship.

Referring to FIG. 4 , the imaginary rays 21 and 22 to the optical center 20 of the imaging lens 43 in ^{±1 st order parallax images have a position function G (} It meets the diffraction grating 41 at a point defined by _{x mth} , y _nth , z _{O ).}

At point G(x _mth , y _nth , z _O ), the diffraction grating 41 changes the path of the actual ray 23 from the point object P1 to the optical center 20 of the imaging lens 43 . The point G(x _mth , y _nth , z _O ) is expressed by Equation 3 below.

where m and n are -1, 0 and 1.

Hereinafter, virtual pinholes for back-projection in diffraction grating imaging corresponding to a critical part of the reconstruction process according to an embodiment of the present invention will be described.

Virtual pinholes and mapping locations in parallax images

Virtual pinholes may be considered a camera array. The present invention provides a formula for the position of the virtual pinholes corresponding to the parallax images and parallax images and virtual pinholes to implement a three-dimensional image reconstruction method: A formula for mapping between VPs is provided.

As shown in Fig. 4, the ray radiating from the point object P2 at an angle of Ψ is a first-order parallax image (PI(x _1st , y _{O ) located at (x 1st} , y _O , z _{O ).} z _O ))). A ray incident from PI(x _1st , y _O , z _O ) through the imaging lens 43 to the imaging point I(x _1st , y _O , z _O ) is equal to the angle of Ψ.

The imaging point I(x _1st , y _O , z _O ) is regarded as a parallax image of angle Ψ with respect to the ray emitted from the point object P1. Assuming another point of depth _{z I} on the pickup plane, a line passing through the point object with the same parallax as _{image I(x 1st} , y _O , z _{O ) and point G(x 1st} , y _O , z _O ) meets the pickup plane.

5 is a geometrical relationship between a point object, a virtual pinhole (VP), a virtual image plane (VI plane), and I(x _1st , y _O , z _{O ) in an embodiment of the present invention;} ) is a diagram showing

5, assuming that the optical center of the virtual pinhole 51 located at a point on the x-axis is a position function VP (x _mth , y _nth , z ₀ ), Equations 1 and 3 Using , VP(x _mth , y _nth , z _O ) is calculated as in Equation 4 below.

where m and n are -1, 0, 1.

Equation 4 above means that when the orders of the corresponding parallax images are the same, the position of each virtual pinhole is a unique value regardless of the position of the object.

However, Equation 4 means that the position of each virtual pinhole increases in the x and y directions as the depth of the object increases, and the imaging lens 43 is the 0th virtual pinhole (the 0 ^th). can be considered as a virtual pinhole).

Hereinafter, a virtual image plane (VI plane) in which a virtual image exists will be described.

The virtual images are images obtained by rearranging the image I(x _mth , y _nth , z _O ) of the pickup plane on the virtual image plane (VI plane). The position function of the virtual image is given by VI(x _mth , y _nth , z _O ) = (x, y, z _I ). where m and n are -1, 0 and 1.

The x-coordinate of the virtual image is as shown in Equation 5 below.

_{If x O} and x _mth are replaced with y _{O and y mth} in Equation 5, the y-coordinate y _O of the virtual image is obtained.

The virtual image VI(x _mth , y _nth , z _O ) is used for the reconstruction method according to the present invention.

These images are shift versions of the picked up parallax image I(x _mth , y _nth , z _{O ) with a mapping factor Δx mapping .}

Using Equations 2 and 5, _{the mapping factor (Δx mapping} ) representing the amount of movement between the _{parallax image I(x mth} , y _nth , z _O ) and the virtual image (VI(x _mth , y _nth , z _O )) is below It can be calculated by Equation 6 of

Parallax Image Segmentation and Reconstruction Methods

To segment a parallax image, the image area is used to determine the field of view of each virtual pinhole.

Segmenting individual parallax images from the raw image captured by the diffraction grating 41 is different from the raw image picked up by the lens array or camera array, which is transparent to the diffraction grating (transparent). difficult due to diffraction grating).

Moreover, in diffraction grating imaging, parallax images are superimposed on each other when the size of an object exceeds a specific limit.

The overlapping problem of parallax images can be explained by the size of the object and the distance between the object and the diffraction grating.

In order to avoid the overlapping problem, an Effective Object Area (EOA), which is the maximum size of an object, needs to be defined.

For convenience, it is assumed that the center of the Effective Object Area (EOA) is aligned with the optical axis of the imaging lens 43 .

6A is a diagram of an effective object area (EOA), a parallax image area, picked up PI regions, virtual pinholes, and the Minimum Image Area (MIA) in an embodiment of the present invention; It is a diagram showing a geometrical relationship between half (half), and FIG. 6B is a diagram for showing how a parallax image of an original object in an actual three-dimensional space is converted into a parallax image array for reconstruction in an embodiment of the present invention to be.

First, referring to FIG. 6A , when the effective object area EOA is defined, the area of the parallax image is defined.

In FIG. 6A , a first order PI region and a -1st order PI region of the effective object area EOA are marked in red and green, respectively.

The boundary between the picked-up PI regions is determined using Equation (2). Accordingly, segmentation of the parallax images in a picked-up plane is possible.

After mapping the area of the picked up parallax image on the pickup plane to the minimum image area MIA of the virtual image plane VI plane using Equation 6, the radius of the minimum image area MIA (Half of MIA) is It is obtained from the distance from the optical axis 60 of the virtual pinhole VP to the edge of the picked-up parallax image region PI region mapped to the virtual image plane VI plane.

The size of the effective object area (EOA) is equal to the size of the ±1st PI area when the point object is placed on the optical axis. The magnitude Δx depends on the point x _OC on the optical axis. Changing the Δx primary PI (1 ^st order PI) center (x _OC1st) the size of the dmf standard primary PI region (1 ^st order PI region) is changed up to Δx, i.e. the effective object area (EOA) is the Equation 7 below.

where λ is the wavelength of the light source, and a is the aperture width of the diffraction grating.

It can be seen from Equation 7 that the effective object area EOA (or the size of the EOA _{) is proportional to the distance (|z 0 - d|) between the object and the diffraction grating.} Half of the minimum image area (MIA), Δr, is calculated by Equation (8) below through Equations (4), (5) and (7).

6B shows how a parallax image of a circle object is converted into a parallax image array for computational reconstruction in an actual 3-D space.

If the effective object area EOA for the circle object on the left side of Fig. 6b is obtained by Equation 7 described above, then, as shown in the middle of Fig. 6b, the raw parallax picked up by the imaging lens 43 It is possible to determine the area of the raw parallax images.

As shown on the right side of Fig. 6b, a parallax image array (PIA) is generated in the virtual plane for 3D reconstruction through the proposed mapping method based on the virtual pinhole and the minimum image area (MIA).

7 is a diagram illustrating a process of back-projection to a reconstruction plane using a parallax image array (PIA) according to an embodiment of the present invention.

Referring to FIG. 7 , each MIA with each parallax image is back-projected in 3D space passing through a virtual pinhole. The back-projection process can be described as a process in which the moved versions of each parallax image are superimposed on the reconstruction plane. Here, it is possible to accurately find each parallax image by Equation 6 described above.

Also, the minimum image area MIA may overlap each other as highlighted in blue and orange in FIG. 7 . The inverse projection of all minimal image areas (MIA) can provide depth information that accurately focuses the object image in the reconstruction plane. Here, the depth of the reconstruction plane is equal to the position of the object. Thus, the reconstruction method of the present invention via diffraction grating imaging consists of determining the minimum image area (MIA), finding the parallax image captured in the minimum image area (MIA), and inversely projecting the minimum image area (MIA). .

In the PIA generated by the method according to the present invention, since the location of the virtual pinhole and location information and size information of each element image are well defined, a three-dimensional image reconstruction (reconstruction) is possible by the known CIIR method.

8 is a flowchart illustrating a 3D image reconstruction method using a diffraction grating according to an embodiment of the present invention, and a subject performing each of the following steps may be a processor in the computing device.

Referring to FIG. 8 , first, in S610 , a process in which parallax images diffracted by the diffraction grating are captured by an imaging lens and stored in a memory in the computing device as a parallax image array is performed.

Here, the diffracted parallax images include a first parallax image (PI(x ₀ , y ₀ , z ₀ )) including a real object P1 and a first parallax image including virtual objects P2 and P3 of the real object. 2 parallax images PI(x _1st , y ₀ , z ₀ ), PI(x _-1st , y ₀ , z ₀ ), wherein the second parallax image is derived from the real object for the first parallax image It may be diffracted at a diffraction angle determined by the wavelength of the emitted light and the aperture width of the diffraction grating.

Next, in S620 , a process in which the parallax images captured by the imaging lens are picked up on a pickup plane is performed.

Next, in S630, a process of relocating (mapped) the parallax images picked up on the pickup plane to a virtual image plane is performed.

x_mapping)에 따라 상기 가상 이미지 평면(VI plane)에 매핑하는 것일 수 있다.Here, in the step S630, the disparity images picked up on the pickup plane are divided based on a geometric relationship between a virtual pinhole defined on the same plane as the imaging lens and the virtual image plane, and then the divided parallax The images are mapped to the mapping factor (

x _mapping ) may be mapped to the virtual image plane (VI plane).

The mapping factor may include, wherein the diffracted disparity images include a first disparity image including a real object and a second disparity image including a virtual object of the real object, and the imaging lens in a three-dimensional space expressible by an xyz axis. When is located in the xy plane having the z coordinate as 0, the pickup plane is located in the xy plane having the z coordinate as z _I , when the second parallax image is expressed as an imaging point on the pickup plane, the virtual image plane It may be the amount of movement of the imaging point in the x-axis direction.

The mapping factor may be calculated by Equation 6 above, where x ₀ is the x-coordinate of the imaging lens, x _mth is the x-coordinate of the diffracted parallax image, and z _o is the actual is the object's z-coordinate, and d is the distance from the imaging lens to the diffraction grating.

In addition, the step S630 is a step of mapping the picked up parallax images to the minimum image regions defined in the virtual image plane, respectively, Each of the minimum image areas may be determined by a viewing angle of each of the virtual pinholes defined on the same plane as the imaging lens. In this case, the picked up parallax images may be mapped to edges within the minimum image regions.

Subsequently, in S640 , the disparity images rearranged in the virtual image plane are back-projected onto a reconstruction plane, and a 3D image of the real object is reconstructed.

So far, the present invention has been looked at mainly in the embodiments. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in variously changed or modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in terms of illustrative rather than restrictive views. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within an equivalent scope should be construed as being included in the present invention.

Claims (7)

In a method of reconstructing a three-dimensional image performed by a processor,
capturing parallax images diffracted by the diffraction grating using an imaging lens;
picking up parallax images captured by the imaging lens on a pickup plane;
relocating the parallax images picked up on the pickup plane to a virtual image plane; and
Reconstructing a three-dimensional image of the real object by back-projecting the disparity images rearranged on the virtual image plane onto a reconstruction plane.
A three-dimensional image reconstruction method using a diffraction grating comprising a. In claim 1,
The diffracted parallax images are
A first parallax image including a real object and a second parallax image including a virtual object of the real object,
Wherein the second parallax image is diffracted with respect to the first parallax image at a diffraction angle determined by a wavelength of a light ray emitted from the real object and an aperture width of the diffraction grating. In claim 1,
Relocating the parallax images picked up on the pickup plane to a virtual image plane comprises:
After dividing the disparity images picked up on the pickup plane based on the geometric relationship between the virtual pinhole and the virtual image plane defined on the same plane as the imaging lens, the divided disparity images are mapped by a mapping factor (Δx _mapping). ) according to the step of mapping to the virtual image plane (VI plane),
The virtual pinhole is a three-dimensional image reconstruction method that is a virtual camera defined on the same plane as the imaging lens. In claim 3,
The diffracted parallax images include a first parallax image including a real object and a second parallax image including a virtual object of the real object, and the imaging lens sets the z coordinate to 0 in a three-dimensional space expressible on the xyz axis. When the pickup plane is located in the xy plane with z coordinates as z _I and the pickup plane is located in the xy plane with z I ,
The mapping factor (Δx _mapping ) is,
When the second parallax image is expressed as an imaging point on the pickup plane, the three-dimensional image reconstruction method is the amount of movement of the imaging point in the x-axis direction on the virtual image plane. In claim 3,
The diffracted parallax images include a first parallax image including a real object and a second parallax image including a virtual object of the real object. When the pickup plane is located in the xy plane with z coordinates as z _I and the pickup plane is located in the xy plane with z I ,
The mapping factor (Δx _mapping ) is,

is calculated by
where x ₀ is the x-coordinate of the imaging lens, x _mth is the x-coordinate of the diffracted parallax image, z _o is the z-coordinate of the real object, and d is the distance from the imaging lens to the diffraction grating 3D image reconstruction method. In claim 1,
Relocating the parallax images picked up on the pickup plane to a virtual image plane comprises:
mapping the picked up parallax images to minimum image regions defined in the virtual image plane, respectively;
Each of the minimum image areas is
It is determined by the viewing angle of each of the virtual pinholes defined on the same plane as the imaging lens,
wherein the virtual pinholes are virtual cameras. In claim 6,
wherein the picked up parallax images are mapped to edges within the minimum image regions.

Images