KR20210085543A - A digital hologram generation method using artificial intelligence and deep learning - Google Patents

Abstract

The present invention relates to a digital hologram generation method using artificial intelligence and deep learning, in which a deep neural network is trained to create a complex two-dimensional fringe (diffraction) pattern made by object points in the form of point clouds, and fringe patterns made for each object point are accumulated, thereby creating one sheet of hologram. The method includes: (a) sampling points of a three-dimensional object, creating a first fringe pattern created by a computer generated hologram (CGH) algorithm for the sampled points, and a second fringe pattern by a fringe pattern generator, and updating the fringe pattern generator with a probability value of the similarity of the first and second fringe patterns, thereby training the fringe pattern generator; and (b) creating third fringe patterns through the trained fringe pattern generator for all points of the three-dimensional object, multiplying each generated third fringe pattern by a brightness value, and accumulating all fringe patterns multiplied by the brightness value, thereby creating a digital hologram. According to the above method, the fringe pattern is created using a neural network, so that calculations is performed at very high speed while saving memory capacity.

Description

{ A digital hologram generation method using artificial intelligence and deep learning }

In particular, the present invention generates a hologram using deep learning technology, but trains a deep neural network to generate a complex two-dimensional fringe (diffraction) pattern made by object points in the form of a point cloud, and fringes made for each object point. It relates to a method of generating a digital hologram using artificial intelligence and deep learning technology, which generates a single hologram by accumulating patterns.

In general, a hologram is composed of fringe patterns generated as a result of interfering a light wave (reference wave) as a reference with a light wave (object wave) reflected from an object. A digital hologram (DH) is obtained by acquiring a pattern as digital data. Therefore, although a hologram is 2D data, it is 3D data including distance information to an object due to a phase difference between an object wave and a reference wave [Non-Patent Document 1]. When the reference wave is irradiated to this hologram, the original image of the object at the original position is reproduced. Therefore, although the holographic image has a viewing angle limitation, since it displays a three-dimensional image, it is considered as the ultimate 3D image [Non-Patent Document 2].

From the perspective of broad classification, a method for generating a traditional digital hologram can be divided into a method using optical imaging and a computer generated hologram (CGH) method using a computer. 1 shows a method using optical imaging. Countless studies on optical imaging and CGH methods have been conducted so far. Each of these methods will have different advantages and disadvantages.

In the case of the CGH method, the biggest disadvantage is that the operation processing speed is very slow. In order to improve this speed, methods such as improving an algorithm, using a GPU, making a field programmable gate array (FPGA), or making a dedicated chipset have been suggested.

In addition, as shown in FIG. 2 , a method for generating a hologram using a deep learning technology that has been rapidly developed in recent years has been proposed [Patent Document 1].

Korean Patent Publication No. 10-2018-0064884 (published on June 15, 2018)

S. A. Benton and V. M. Bove Jr., Holographic imaging, John Wiley and Sons Inc., 2008. Ki-Pyung Han, "Giga Korea," TTL Journal, Vol. 146, pp. 16-21, March 2013 (March 2013).

It is an object of the present invention to solve the above-described problems, and to generate a hologram using deep learning technology, but to generate a complex two-dimensional fringe (diffraction) pattern made by object points in the form of a point cloud. To provide a digital hologram creation method using artificial intelligence and deep learning technology that trains a deep neural network and generates a single hologram by accumulating fringe patterns made for each object point.

In order to achieve the above object, the present invention relates to a method of generating a digital hologram using artificial intelligence and deep learning technology, (a) sampling points of a three-dimensional object, and a computer generated hologram (CGH) algorithm for the sampled points By generating a first fringe pattern generated by and a second fringe pattern by a fringe pattern generator, and updating the fringe pattern generator with a probability value of the similarity of the first and second fringe patterns, the fringe pattern generator learning; and, (b) generating third fringe patterns through a fringe pattern generator learned for all points of the three-dimensional object, multiplying each generated third fringe pattern by a brightness value, and all fringe patterns multiplied by the brightness value and generating a digital hologram by accumulating the .

In addition, in the method of generating a digital hologram using artificial intelligence and deep learning technology, the present invention constructs a fringe pattern discriminator that probabilistically indicates how similar the generated fringe pattern is to the actual fringe pattern in step (a), , a fringe pattern discriminator is trained for the sampled points, and the fringe pattern generator is trained using the learned fringe pattern discriminator.

In addition, in the method of generating a digital hologram using artificial intelligence and deep learning technology, the present invention is characterized in that the fringe pattern generator is composed of a deep neural network.

In addition, the present invention is characterized in that in the method of generating a digital hologram using artificial intelligence and deep learning technology, the fringe pattern discriminator is composed of a deep neural network.

In addition, the present invention provides a method for generating a digital hologram using artificial intelligence and deep learning technology, wherein the fringe pattern generator is composed of a real part generator and an imaginary part generator, and each generator generates a real fringe pattern and an imaginary fringe pattern. characterized in that

In addition, in the method of generating a digital hologram using artificial intelligence and deep learning technology, the present invention is to learn by optimizing the _{parameter θ D of the fringe pattern discriminator using the following Equation 1 in the step (a).} characterized.

[Formula 1]

However, L _i is a loss function, ▽ _θD represents the del operator to the differentiation of a deep neural network using the stochastic gradient ascending technique by θ _D , and the Optimizer(x,parameters) is a parameter parameter so that the value of x is minimized. Represents a function that optimizes by changing

In addition, in the method of generating a digital hologram using artificial intelligence and deep learning technology, the present invention is _{characterized in that the loss function L i} is obtained as shown in Equation 2 below.

[Formula 2]

only,

는 프린지 패턴 생성기에 의한 프린지 패턴이고, L_i는 손실함수이고, D(

), D(f_i), D(

)는 각각

, f_i,

의 프린지 패턴 판별기(D)에 의한 값이고,

는

를

로 델 연산자 ▽에 의해 미분한 값을 나타냄.Here, ε is a preset weight, f _i is a fringe pattern by the CGH algorithm,

is the fringe pattern by the fringe pattern generator, L _i is the loss function, D(

), D(f _i ), D(

) is each

, f _i ,

is the value by the fringe pattern discriminator (D) of

is

to

Represents the value differentiated by the Rhodel operator ▽.

In addition, in the method of generating a digital hologram using artificial intelligence and deep learning technology, the present invention is characterized by optimizing the _{parameter θ G of the fringe pattern generator using Equation 3 below in the step (a).} do it with

[Equation 3]

However, ▽θ _G represents the del operator by θ _{G , and G(z i} | c _i ) represents the value of the fringe pattern generator of the noise image z _i when each sampling point c _{i is given.}

In addition, the present invention relates to a computer-readable recording medium in which a program for performing a method of generating a digital hologram using artificial intelligence and deep learning technology is recorded.

As described above, according to the digital hologram generation method using artificial intelligence and deep learning technology according to the present invention, by generating a fringe pattern using a neural network, it is possible to perform calculations at a very high speed while saving memory capacity. effect is obtained.

That is, compared to the lookup table method according to the related art, the memory capacity used can be very small. Specifically, in the case of the lookup table method, a memory capacity of several giga to several tera is required, whereas the method of the neural network method according to the present invention uses only several tens of mega to a maximum of several hundred mega.

In addition, the method according to the present invention has a very high operation speed compared to the prior art. That is, in the case of the software method, compared to the method using the formula, the method using the formula requires repeated calculations as much as the size of the hologram to generate the fringe pattern for one light source, whereas the method of the neural network method according to the present invention is one Since a single fringe pattern is output with one repetition, theoretically, the speed increases in proportion to the size of the hologram.

1 is an exemplary view showing a method of generating and restoring a digital hologram by an optical acquisition method according to the prior art.
2 is an exemplary view showing a digital hologram generation method using artificial intelligence according to the prior art.
3 is a diagram showing the configuration of an overall system for implementing the present invention.
4 is a diagram illustrating a process of generating a fringe pattern based on object points according to the method for generating a hologram of the present invention.
5 is a view showing a process of generating a fringe pattern (diffraction pattern for a single object pointer) using a deep neural network (deep learning model) according to the hologram generation method of the present invention.
6 is a flowchart illustrating a method of generating a digital hologram using artificial intelligence and deep learning technology according to an embodiment of the present invention.
7 is a view showing a process of training a deep neural network (deep learning model) for generating a diffraction pattern (diffraction pattern for a single object pointer) according to an embodiment of the present invention.
8 is a pseudo code illustrating a learning process of a deep neural network for generating a fringe pattern according to an embodiment of the present invention.
9 is a flowchart illustrating a hologram generation step according to an embodiment of the present invention.
10 is a pseudo code showing a hologram generation process using a fringe pattern generation deep heart network according to an embodiment of the present invention.
11 is an exemplary view of a fringe pattern generated using a deep neural network according to an experiment of the present invention, (a) a real part, (b) an imaginary part.
12 is an exemplary diagram of a hologram generated using a deep neural network according to an experiment of the present invention, and is an exemplary diagram of (a) a real part and (b) an imaginary part.
13 is an exemplary view of a hologram generation result using a deep neural network according to an experiment of the present invention, (a) a point cloud for an object, and (b) an exemplary view showing a hologram restoration result using the hologram of Fig. 8;

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific contents for carrying out the present invention will be described with reference to the drawings.

In addition, in demonstrating this invention, the same part is attached|subjected with the same code|symbol, and the repetition description is abbreviate|omitted.

First, examples of the configuration of the entire system for implementing the present invention will be described with reference to FIG. 3 .

As shown in FIG. 3 , the method for generating a digital hologram using artificial intelligence and deep learning technology according to the present invention can be implemented as a program system on a computer terminal 20 that receives three-dimensional data 10 and generates a hologram. have. That is, the digital hologram generating method may be configured as a program and installed in the computer terminal 20 to be executed. A program installed in the computer terminal 20 may operate as one program system 30 .

On the other hand, as another embodiment, the digital hologram generation method using artificial intelligence and deep learning technology may be implemented with a single electronic circuit such as an ASIC (application-specific semiconductor) in addition to being configured as a program and operating in a general-purpose computer. Alternatively, it may be developed as a dedicated computer terminal that exclusively processes only the generation of a holographic image. This will be referred to as a digital hologram generating device 30 . Other possible forms may also be implemented.

On the other hand, 3D data is composed of a point cloud representing a 3D object.

Next, the basic principle and process of a method for generating a digital hologram using a deep learning technique according to the present invention will be described with reference to FIGS. 4 to 5 .

A computer-generated hologram (CGH) is obtained by accumulating the fringe patterns for all light sources input from the coordinates of all hologram planes and then accumulating the patterns.

Equation 1 shows the CGH formula of the real part for generating a complete complex hologram.

[Equation 1]

Here, I(u,v) is the brightness at the coordinates (u,v) of the hologram plane, and A _j (x,y,z) is the brightness of the light source at the x,y,z coordinates. N is the number of light sources, λ is the wavelength of a reference wave used to generate the hologram, and p is the size of the pixels of the hologram and the light source.

The process of Equation 1 is separated as shown in Equations 2 and 3.

[Equation 2]

[Equation 3]

Accordingly, as shown in FIG. 4 , a fringe pattern is generated for each object point using Equation 2 .

Meanwhile, the fringe pattern is made in the form of a complex number (I _Re + iI _Im ), and in the complex number, the real part (I _Re ) and the imaginary part (I _Im ) are calculated separately. In Equation 2, the real part is calculated as a cosine, but the imaginary part is calculated as a sine function.

At this time, as shown in FIG. 5 , the method according to the present invention is generated using a deep neural network for Equation (2).

The generated fringe pattern is multiplied by a brightness value as shown in Equation 3, and cumulative addition is performed to generate a final hologram.

Next, a method of generating a digital hologram using artificial intelligence and deep learning technology according to an embodiment of the present invention will be described with reference to FIGS. 6 to 10 .

As shown in FIG. 6 , the method for generating a digital hologram according to the present invention includes the steps of learning a fringe pattern generator (S10), and generating a digital hologram from object data using the learned fringe pattern generator (20). ) is composed of

First, the step of learning the fringe pattern generator (S10) will be described.

7 shows the structure of a model for learning a deep neural network to generate a fringe pattern. The overall structure is based on the adversarial generative model.

As shown in FIG. 7 , when object point information is input, a real fringe pattern (hereinafter, a first fringe pattern) is generated using a computer generated hologram (CGH) algorithm. At the same time, a fake fringe pattern (hereinafter referred to as a second fringe pattern) is generated using a deep neural network-based fringe pattern generator. However, here, in Real Fringe, Real means real, and in Real Data, Real means real information.

The deep neural network-based fringe pattern discriminator compares these two results and outputs how similar the real and fake are as a probability value, and based on this probabilistic calculation result, the fringe pattern generator creates a fake that is more similar to the real one. update the internal parameters of

If this process is repeatedly performed, the real and fake become indistinguishably similar. That is, the fringe pattern generator can generate the same fringe pattern as the fringe pattern generated by CGH, and the deep neural network outputs the fringe pattern. It means you have been taught to do it.

A specific procedure for training using the fringe pattern generation training model is shown in the form of pseudocodes in FIG. 8 . Each term and method of FIG. 8 will be described in detail below.

c, c _i : As spatial coordinates of an object point in the form of an integer or real number, it is condition information.

z ₁ ,z ₂ ...,z _m : A noise image of the _{same size (H h} × W _h ) as the fringe pattern to be created.

f ₁ ,f ₂ ...,f _m : A Real Fringe Pattern _{of (H h} × W _h ) size generated using a computer-generated hologram algorithm.

G(z _i | c _{i ): Given the coordinate condition c i} of the object point, a fringe pattern is generated using the noise image z _{i as an input.} This is called a fringe pattern generator, and G _Re (z _i |c _i ) and G _Im (z _i |c _i ) are used to generate real holograms and imaginary holograms. It consists of two deep neural networks.

D(x) : represents the probability that _{x came from the noise p g (z).} That is, it probabilistically indicates how similar the generated fringe pattern is to the actual fringe pattern. It is called a fringe pattern discriminator, and two deep neural networks, _{D Re} (x) and D _Im (x), are used to discriminate real holograms and imaginary holograms. network) is composed of

: 프린지 패턴 생성기를 통해 만들어진 프린지 패턴이다.

: It is a fringe pattern created through the fringe pattern generator.

▽ _x : It is a Del operator for the differentiation of deep neural networks using stochastic gradient ascending with respect to x.

ε: means a random number between 0 and 1.

θ _G : A parameter of the fringe pattern generator composed of a deep neural network. These parameters include the weight, bias, and optimizer parameters of the neural network. The parameters of the optimizer are determined according to the type of optimizer. For example, when using Adam, α, β ₁ , and β ₂ are included.

θ _D : A parameter of the fringe pattern discriminator composed of a deep neural network. These parameters include the weight, bias, and optimizer parameters of the neural network. The parameters of the optimizer are determined according to the type of optimizer. For example, when using Adam, α, β ₁ , and β ₂ are included.

L: It is a loss function used for deep neural network training in deep learning. Currently, this document uses WGAN-GP (Wasserstein Generative Adversarial Net-) as an example, but other loss functions may be used.

m: number of mini-batches

k : number of training (training)

n: the number of training times of the discriminator per mini-batch

That is, points of a three-dimensional object are sampled with a sampling size m, and a fringe pattern discriminator and a fringe pattern generator composed of a neural network are trained (or learned) using the sampling point data.

First, for the sampling point c ₁ ,c ₂ ...,c _m , the noise image z ₁ ,z ₂ ...,z _m is sampled from the predetermined noise p _{g (z), and the} 1 Create a fringe pattern f ₁ ,f ₂ ...,f _m .

Next, for each sampling point c _i , a second fringe pattern is generated by a fringe pattern generator. In this case, the following equation is used.

[Equation 4]

)을 무작위 값 ε으로 가중치를 달리하여 더한다. 다음 수학식과 같다. _{Next, a fringe pattern (f i} ) generated through a computer-generated hologram (CGH) and a fringe pattern (

) is added with different weights as a random value ε. It is as the following equation.

[Equation 5]

Next, a loss function L _i is calculated using Equation (6).

[Equation 6]

Here, λ is a penalty coefficient and serves to adjust the size of the loss function. λ is a predetermined value.

) 등은 프린지 패턴 판별기에 의한 값으로서, 실제 프린지 패턴과 얼마나 유사한지 확률적으로 나타낸 것이다.Also, D(

) and the like are values obtained by the fringe pattern discriminator, which indicates how similar the actual fringe pattern is.

Also, |||||| ₂ is an L2 norm operation, and it is an operation which squares and adds each element inside, and takes the square root. That is, the normal operation is an operation that measures the length or size of a vector.

는

를

로 델 연산자(확률적 경사 상승 기법을 이용한 심층신경망의 미분으로의 델 연산자) ▽에 의해 미분한 값이다.

is

to

It is a value differentiated by Rhodel operator (Del operator to differentiation of deep neural network using stochastic gradient escalation technique) ▽.

Next, if the loss function is obtained as above for all sampling points, learning is performed by optimizing the fringe pattern discriminator as shown in the following equation.

[Equation 7]

θ _D represents the parameters of the fringe pattern discriminator. The parameters of the pattern discriminator are parameters of the neural network, and represent weights, biases, and the like. Also, Optimizer(x,parameters) represents a function that optimizes by changing the parameter parameters so that the value of x is minimized.

That is, Equation 7 indicates that the average value of the loss function L _{i of each sampling point c i} is differentiated by the del operator (▽), and the optimization is performed by obtaining the values of parameters optimizing the differentiated value.

In other words, the point with the smallest loss in the loss function is the point where the slope of the loss function is zero. To find the point where the slope of the loss function, which is a multidimensional function, is zero, the slope of the loss function is obtained by taking the del operator. The stochastic gradient descent method is to construct a loss function with the difference between the result obtained through forward propagation of the neural network and the correct answer, and then adjust the parameters constituting the neural network so that the gradient of the loss function becomes zero.

However, Equation 7 is an optimization process. After obtaining the slope by differentiating the loss function, the problem of how to change the parameters of the neural network so that the slope of the loss function becomes zero using this slope remains. Here, the process of changing the parameters of the neural network using the differential value of the loss function is optimization, and performing this is the optimizer.

Next, the previous learning process (or training process) is repeated as many times as the number of training k. That is, in the previous process, the fringe pattern discriminator was trained once using the sampling data. This learning process is repeated k times to learn the fringe pattern discriminator.

Next, for training of the fringe pattern generator, a point of an object and a noise image are sampled. And, as shown in the following equation, the result (or probability) generated by the generator G is determined by the discriminator D, and the parameters of the generator G are optimized by maximizing the discrimination result. In this case, the previously trained discriminator is used.

[Equation 8]

In summary, we use the method of training generator 1 after training discriminator k. The noise image, actual fringe pattern and condition are randomly extracted as many as m mini-batch sizes, and the model is propagated using this. After calculating the loss function using the forward propagation result and finding the derivative of the loss function, the parameters of the discriminator are updated through the backpropagation operation. Next, the model is forward propagated again using the randomly extracted noise image and conditions, and the parameters of the generator are updated while backpropagating using the results. Here, forward propagation is the process of making an output by putting an input into the neural network, and it is the same as inference or testing. Backpropagation is a process of differentiating the loss function, which is the difference between the forward propagation result and the correct answer, and changing the parameters of the neural network while moving the differentiation result from the output part to the input part of the neural network using an optimization algorithm.

Next, the step 20 of generating a digital hologram will be described.

A procedure for generating a digital hologram using the fringe pattern generator created through the fringe pattern generation training step S10 is shown in FIG. 9 .

As described above, the generator consists of two parts: a real part generator and an imaginary part generator. Each generator generates a real fringe pattern and an imaginary fringe pattern. The generated fringe pattern is multiplied by the brightness value, and cumulative addition is performed with the previously generated fringe pattern. When fringe patterns for all object points are accumulated and added, a final hologram is created.

A specific procedure for generating a hologram using the fringe pattern generation model is shown in the form of a pseudo code in FIG. 10 . Each term and method for FIG. 10 has been described in detail below.

condition(object points): The process of calling 3D coordinates from object point information

intensity(object points): The process of calling brightness information from object point information

A _i : Brightness value of object point

H: One digital hologram

That is, coordinate information and brightness information to be used as conditions for the deep neural network are called from the object point information. A single fringe pattern is generated through a generator using coordinate information, a brightness weight is assigned to the generated patterns, and then a hologram is generated by accumulating it.

In this case, the following equation is used.

[Equation 9]

Next, the effects of the present invention will be described through experiments.

11 shows a complex fringe pattern generated using a fringe pattern generator. The real number in Fig. 11(a) and the imaginary part fringe pattern in Fig. 11(b) are shown for 64 conditions, respectively.

Fig. 8 shows the hologram generated by cumulative addition of the fringe patterns generated through the fringe pattern generator. Fig. 12(a) is a real part, and Fig. 12(b) is an imaginary part.

The result of restoring the generated hologram is shown in FIG. 13 . 13(a) shows the point cloud information of an object made into an 8-bit image file. FIG. 13( b ) is a result of numerically restoring the hologram of FIG. 12 . As can be seen from the results, it was confirmed that a digital hologram very similar to that of CGH was generated with a difference of 1 to 5 dB.

In the above, the invention made by the present inventors has been described in detail according to the above embodiments, but the present invention is not limited to the above embodiments, and various modifications can be made without departing from the gist of the present invention.

10: three-dimensional data 20: computer terminal
30: program system

Claims (9)

In the method of generating a digital hologram using artificial intelligence and deep learning technology,
(a) sampling points of a three-dimensional object, and generating a first fringe pattern generated by a computer generated hologram (CGH) algorithm for the sampled points, and a second fringe pattern by a fringe pattern generator, and the first and learning the fringe pattern generator by updating the fringe pattern generator with a probability value of the similarity of the second fringe pattern. and;
(b) generating third fringe patterns through a fringe pattern generator learned for all points of the three-dimensional object, multiplying each generated third fringe pattern with a brightness value, and accumulating all fringe patterns multiplied by the brightness value A method of generating a digital hologram using artificial intelligence and deep learning technology, characterized in that it comprises the step of generating a digital hologram.
According to claim 1,
In step (a), a fringe pattern discriminator is constructed that probabilistically indicates how similar the generated fringe pattern is to the actual fringe pattern, and the fringe pattern discriminator is trained on the sampled points, and the learned fringe pattern discriminator is used to A method of generating a digital hologram using artificial intelligence and deep learning technology, characterized in that learning the fringe pattern generator.
According to claim 1,
The fringe pattern generator is a digital hologram generation method using artificial intelligence and deep learning technology, characterized in that it consists of a deep neural network.
3. The method of claim 2,
The fringe pattern discriminator is a method of generating a digital hologram using artificial intelligence and deep learning technology, characterized in that it consists of a deep neural network.
3. The method of claim 2,
The fringe pattern generator is composed of a real part generator and an imaginary part generator, and each generator generates a real fringe pattern and an imaginary fringe pattern. A method of generating a digital hologram using artificial intelligence and deep learning technology.
3. The method of claim 2,
In the step (a), using the following Equation 1, the method of generating a digital hologram using artificial intelligence and deep learning technology, characterized in that learning by optimizing the _{parameter θ D of the fringe pattern discriminator.}
[Formula 1]

However, L _i is a loss function, ▽ _θD represents the del operator to the differentiation of a deep neural network using the stochastic gradient ascending technique by θ _D , and the Optimizer(x,parameters) is a parameter parameter so that the value of x is minimized. Represents a function that optimizes by changing
7. The method of claim 6,
A method of generating a digital hologram using artificial intelligence and deep learning technology, characterized in that the loss function L _{i is obtained as shown in Equation 2 below.}
[Formula 2]

only,

Here, ε is a preset weight, f _i is a fringe pattern by the CGH algorithm,

is the fringe pattern by the fringe pattern generator, L _i is the loss function, D(

), D(f _i ), D(

) is each

, f _i ,

is the value by the fringe pattern discriminator (D) of

is

to

Represents the value differentiated by the Rhodel operator ▽.
7. The method of claim 6,
In the step (a), using the following Equation 3, the method of generating a digital hologram using artificial intelligence and deep learning technology, characterized in that learning by optimizing the _{parameter θ G of the fringe pattern generator.}
[Equation 3]

However, ▽θ _G represents the del operator by θ _{G , and G(z i} | c _i ) represents the value of the fringe pattern generator from the noise image z _i when each sampling point c _{i is given.}
A computer-readable recording medium in which a program for performing the method of generating a digital hologram using the artificial intelligence and deep learning technology of any one of claims 1 to 8 is recorded.

Images