KR102190773B1 - Apparatus and method for speckle removal in hologram using light field conversion and deep learning - Google Patents

Abstract

Various embodiments of the present invention relate to a method for removing a speckle in a hologram using light field conversion and deep learning and a device thereof. The method includes: converting the hologram to light field data; removing speckle noise from light field data; compensating resolution for the light field data; and synthesizing new holograms from the light field data.

Description

Hologram speckle removal method and device applying light field conversion technology and deep learning {APPARATUS AND METHOD FOR SPECKLE REMOVAL IN HOLOGRAM USING LIGHT FIELD CONVERSION AND DEEP LEARNING}

Various embodiments relate to a method and apparatus for removing hologram speckles to which light field transformation technology and deep learning are applied.

One of the inherent problems of digital holography is speckle noise. The random phase distribution on the object surface generates speckle noise during reconstruction, which degrades the reproduced holographic image. Accordingly, various methods have been proposed to reduce speckle noise. For example, there are spectrum averaging technology, spatial interleaving technology, and temporal averaging technology. In the case of the spectrum averaging technique, speckle noise is removed, but there is a problem that the holographic image is blurred. Meanwhile, in the case of the spatial interleaving technique and the temporal averaging technique, speckle noise is effectively removed, but there is a problem that a plurality of holographic images are required.

Various embodiments provide an electronic device capable of effectively removing speckle noise from a hologram and a method of operating the same.

Various embodiments provide an electronic device capable of suppressing a decrease in resolution of a hologram while removing speckle noise from a hologram, and a method of operating the same.

An operation method of an electronic device according to various embodiments includes an operation of converting a hologram into light field data, an operation of removing speckle noise from the light field data, an operation of compensating a resolution for the light field data, and the It may include an operation of synthesizing a new hologram from the light field data.

An electronic device according to various embodiments of the present disclosure includes a memory and a processor connected to the memory to operate, the processor converting a hologram into light field data, removing speckle noise from the light field data, The resolution may be compensated for the light field data, and a new hologram may be synthesized from the light field data.

According to various embodiments, the electronic device may convert a hologram into light field data, and may remove speckle noise from the light field data. In this case, the electronic device may compensate for the reduced resolution based on the light field data. Through this, the electronic device may synthesize a new hologram from light field data for which speckle noise is removed and resolution is compensated. Accordingly, the electronic device can maintain the resolution of the original hologram even for the new hologram. That is, the electronic device can suppress a decrease in the resolution of the hologram while removing speckle noise from the hologram.

1 is a diagram illustrating an electronic device according to various embodiments.
2 is a diagram illustrating a method of operating an electronic device according to various embodiments.
3 and 4 are diagrams for explaining a light field data conversion operation of FIG. 2.
5 and 6 are diagrams for describing a write field data processing operation of FIG. 2.
7 and 8 are diagrams for describing a new hologram synthesis operation of FIG. 2.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings.

Various embodiments provide an electronic device capable of effectively removing speckle noise from a hologram and a method of operating the same. Various embodiments provide an electronic device capable of suppressing a decrease in resolution of a hologram while removing speckle noise from a hologram, and a method of operating the same. According to various embodiments, instead of extracting a 3D model of an object, the electronic device may extract light field (LF) data, that is, various views of 3D objects without a depth map. According to various embodiments, the electronic device may convert a hologram into light field data and then process the light field data to remove speckle noise and improve resolution. According to various embodiments, the processed light field data may be finally used to synthesize a desired carrier and a new hologram so that it can be used for each spectrum interleaving technique or time averaging technique. According to various embodiments, the electronic device may remove speckle noise and improve resolution by performing deep learning based on a denoise convolution neural network (DnCNN) and a super resolution convolution neural network (SRCNN). According to various embodiments, the electronic device may synthesize a new hologram from the final light field data. To this end, the electronic device may use a non-hogel-based computer generated hologram (CGH) technology. The non-Hogel-based CGH technology makes it possible to use arbitrary carriers for hologram synthesis from light field data, thereby generating interleaved spectral or uncorrelated random phase distributions required for speckle reduction technology. The non-Hogel-based CGH technology can also be free from the trade-off between each resolution and the spatial resolution of the existing Hogel-based CGH technology.

1 is a diagram illustrating an electronic device 100 according to various embodiments.

Referring to FIG. 1, an electronic device 100 according to various embodiments may include an input module 110, an output module 120, a memory 130, and a processor 140.

The input module 110 may input commands or data to be used for at least one component of the electronic device 100. In this case, the input module 110 may include at least one of an input device through which a user can directly input commands or data into the electronic device 100 or a communication device through which commands or data can be received from an external device. have. For example, the communication device may include at least one of a wired communication device or a wireless communication device.

The output module 120 may output data to the outside of the electronic device 100. In this case, the output module 120 may include at least one of an output device capable of directly displaying or reproducing data or a communication device capable of transmitting data to an external device. For example, the communication device may include at least one of a wired communication device or a wireless communication device.

The memory 130 may store various types of data used by at least one component of the electronic device 100. For example, the memory 130 may include at least one of a volatile memory or a nonvolatile memory. The data may include input data or output data for a program or a command related thereto.

The processor 140 may execute a program in the memory 160 to control at least one component of the electronic device 100 and may perform data processing or operation. The processor 140 may convert the hologram into light field data. In this case, the processor 140 may extract a plurality of orthographic amplitude views as light field data. In addition, the processor 140 may process the write field data. The processor 140 may remove speckle noise from light field data and improve resolution of light field data based on a deep learning technology. The deep learning technology may include at least one of a denoise convolution neural network (DnCNN) or a super resolution convolution neural network (SRCNN). The processor 140 may remove speckle noise from the extracted orthographic amplitude view through DnCNN. The processor 140 may improve the resolution of the extracted orthographic amplitude view through the SRCNN. Through this, the processor 140 may synthesize a new hologram from the light field data. In this case, the processor 140 may synthesize a new hologram from the light field data using an arbitrary carrier. The processor 140 may use a non-Hogel based hologram synthesis technology from light field data.

2 is a diagram illustrating a method of operating an electronic device 100 according to various embodiments. 3 and 4 are diagrams for explaining a light field data conversion operation of FIG. 2. 5 and 6 are diagrams for describing a write field data processing operation of FIG. 2. 7 and 8 are diagrams for describing a new hologram synthesis operation of FIG. 2.

Referring to FIG. 2, in operation 210, the electronic device 100 may convert a hologram into light field data. The processor 140 may extract light field data from a given hologram. Lightfield data can consist of a collection of different views in perspective or orthographic projection geometry. Since it is required for a non-hogel-based computer generated hologram (CGH) technology used in an operation described below, an orthogonal projection geometry can be used. Since the views of the extracted light field data comply with the non-Hogel based CGH technology, they may be amplitude distributions rather than intensity distributions. Through this, the processor 140 may extract a plurality of orthographic amplitude views as shown in FIG. 3.

In this case, the processor 140 may extract each orthographic amplitude view as shown in FIG. 4. At this time, the orthogonal amplitude view U _fxo,fyo (x,y) of a specific angle (θ _xo , θ _yo ) may be obtained by bandpass filtering of the hologram as shown in [Equation 1] below.

Here, F[·] and F ^-1 [·] may represent Fourier and inverse Fourier transforms, respectively. M _fxo,fyo (f _x , f _y ) may represent a binary mask of 1 within a passband. The passband may have a bandwidth B _p , and the center of the passband may be (f _xo , f _yo ) = (sinθ _xo /λ, sinθ _yo /λ).

The bandwidth B _p of the bandpass filter may determine a resolution and angular selectivity of the extracted orthogonal amplitude view having a trade-off relationship. Increasing the bandwidth B _p improves the resolution of the extracted orthographic amplitude view, but may decrease the angular selectivity. In various embodiments, a narrow bandwidth B _p can be used to extract an orthographic amplitude view with high angular selectivity. Spatial resolution reduction may be compensated for using a super resolution convolution neural network (SRCNN) in an operation described later.

The electronic device 100 may process the light field data in operation 220. At this time, the extracted orthographic amplitude view may be contaminated by speckle noise. And, as described above, the extracted orthographic amplitude view may have a low resolution. The processor 140 may remove speckle noise from light field data and improve resolution of light field data based on a deep learning technology. The deep learning technology may include at least one of a denoise convolution neural network (DnCNN) or a super resolution convolution neural network (SRCNN). The processor 140 may remove speckle noise from the extracted orthographic amplitude view through DnCNN. The processor 140 may improve the resolution of the extracted orthographic amplitude view through the SRCNN.

DnCNN can be used to reduce speckle noise in the extracted orthographic amplitude view. DnCNN is a deep learning technique proposed to remove additive Gaussian noise from an image. DnCNN may be composed of three layers as shown in FIG. 5. The first layer may be composed of convolution (Conv) and rectified linear unit (ReLU) functions. 64 filters each having a size of 3Х3Х1 can be used to obtain 64 feature maps. The ReLU function can be used to provide nonlinearity. The second layer may consist of convolution (Conv), batch normalization (BN) and rectification linear unit (ReLU) functions. It can be composed of 64 filters each having a size of 3Х3Х64. The second layer can be repeated to isolate the noise. The degree of noise removal can be adjusted by the depth of the layer or the number of repetitions. Finally, the third layer consists of a convolution (Conv) function, which can be used to get the resulting image. The resulting image may be a residual image with only noise. By subtracting the residual image from the original image, a noise reduction image can be obtained.

The original DnCNN was designed for an additive Gaussian noise model as shown in [Equation 2] below.

Here, x denotes an image contaminated with noise, and y denotes a corresponding noise-free image. n _σ represents noise and can be assumed to have a Gaussian distribution with zero mean and standard deviation σ. Through the noise model of [Equation 2], a noise-contaminated image is generated from a noise-free image, and a pair of a noise-free image and a noise-contaminated image is used for training DnCNN. However, the noise model of [Equation 2] cannot be used as it is in various embodiments because speckles have different statistics. In various embodiments, while maintaining the original DnCNN design as much as possible, the noise model of [Equation 2] may be modified to approximate the speckle statistics.

As shown in [Equation 3] below, speckle A has the same amplitude |a| And it can be assumed that it is generated by averaging the complex numbers of arbitrary steps by N.

Here, the phase θ _n may be uniformly distributed within the range [0, 2π]. This may indicate that the joint probability density function of the real part Ar and the imaginary part Ai of the speckle A is as shown in [Equation 4] below.

The probability density function f _|A| of the speckled amplitude image Can be obtained as follows [Equation 5]. Also, the average of the speckle amplitude images m _|A| And standard deviation σ _|A| Can be obtained as follows [Equation 6] and [Equation 7]. The following [Equation 6] and [Equation 7] may mean that both the average and the standard deviation of the speckle amplitude image are proportional to the amplitude |a| of the original image. And, the probability density function f _|A| of the following [Equation 5] Can be approximated to a Gaussian distribution with the same mean and standard deviation. Through this, the speckle amplitude image x can be modeled as follows [Equation 8].

를 갖는 가우시안 노이즈일 수 있다.

를 정의함으로써, 상기 [수학식 8]은 하기 [수학식 9]와 같이 변경될 수 있다. Where n _σo denotes the noise, zero mean and standard deviation

It may be Gaussian noise with.

By defining [Equation 8] can be changed as shown in [Equation 9] below.

를 갖는 가우시안 분포를 가질 수 있다. 요약하면, 다양한 실시예들에 있어서, 상기 [수학식 9]와 같이 수정된 노이즈 모델은 DnCNN의 훈련 단계에서 상기 [수학식 2]를 대신하여 사용될 수 있다. Where n _σ'denotes noise, zero mean and standard deviation

It can have a Gaussian distribution with In summary, in various embodiments, the noise model modified as in [Equation 9] may be used in place of [Equation 2] in the training step of DnCNN.

SRCNN can be used for resolution compensation of the extracted orthographic amplitude view. As described above, the extracted orthographic amplitude view may have a lower resolution than the hologram itself. This reduction in resolution may be due to bandpass filtering used when extracting the orthogonal amplitude view from the hologram. For the bandwidth B _p of the bandpass filter and the bandwidth B _H of the hologram itself, the effective resolution of the extracted orthogonal amplitude view can be reduced by the ratio B _p /B _H from the original resolution of the hologram. In addition, as the speckle noise is removed using DnCNN, the resolution of the extracted orthographic amplitude view may be lowered. SRCNN is a deep learning technology that performs single-image super-resolution. As illustrated in FIG. 6, the SRCNN may be composed of a plurality of layers that perform patch extraction, representation, non-linear mapping, and reconstruction, respectively.

According to various embodiments, the processor 140 may apply DnCNN to remove speckle noise from the orthogonal amplitude view, and apply SRCNN to compensate the resolution of the orthogonal amplitude view. Through this, the processor 140 may obtain final light field data composed of a set of orthographic images.

The electronic device 100 may synthesize a new hologram from the light field data in operation 230. In this case, the electronic device 100 may use a desired carrier wave. The light field data may be composed of amplitude images without phase information. The phase distribution on the surface of the reconstructed 3D object of the hologram may be determined by a carrier wave used for hologram synthesis. In various embodiments, the carrier may be selected according to the requirements of the applied holographic speckle reduction technique as shown in FIG. 7. For example, for temporal averaging of the random phase technique, a plurality of holograms may be synthesized with other random phase carriers. As another example, for time multiplexing of the interleaved angular spectrum technique, a plurality of holograms may be synthesized with different sets of interleaved plane carriers. Accordingly, in various embodiments, the processor 140 may synthesize a new hologram from the light field data using an arbitrary carrier.

The processor 140 may use a non-Hogel based hologram synthesis technology from light field data. Many holographic synthesis techniques for light field data generate holographic stereograms without controlling the carrier, but non-Hogel-based CGH technology is a phase offset determined by the selected carrier, and is a continuous wavefront for each object point. It is possible to create a hologram with In addition, the non-Hogel-based technology can be free from the spatio-angular resolution tradeoff of the existing Hogel-based holographic stereogram technology. This feature makes it possible to synthesize a hologram with the same resolution as the light field data. The processor 140 may synthesize a hologram from light field data based on the non-Hogel based CGH technology as shown in FIG. 8. The processor 140 configures the four-dimensional (4D) light field data L(x, y, f _x , f _y ) in the orthogonal projection amplitude view U _fx,fy (x, y) (here, L(x, y, f _x , f _y ) = U _{fx, fy} (x, y)), non-Hogel-based CGH technology can synthesize a hologram H (x, y) as shown in [Equation 10] below.

Here, L(x, y, f _x , f _y ) may be a 2D Fourier transform for the f _x and f _y axes. W(x _c , y _c ) may be a complex field of a carrier for the hologram plane. Accordingly, in various embodiments, a carrier wave or W(x _c , y _c ) is selected according to an applied speckle removal hologram reconstruction technique, and may be applied to [Equation 10].

The electronic device 100 according to various embodiments includes a memory 130 and a processor 140 connected to the memory 130 to operate, and the processor 140 converts a hologram into write field data , It may be configured to remove speckle noise from the light field data, compensate the resolution for the light field data, and synthesize a new hologram from the light field data.

According to various embodiments, the processor 140 may be configured to extract a plurality of orthographic amplitude views from a hologram as light field data.

According to various embodiments, the processor 140 performs deep learning based on a denoise convolution neural network (DnCNN) to obtain a residual image related to speckle noise, and a residual image from an original image related to light field data. Can be configured to subtract.

According to various embodiments, DnCNN may be designed based on an additive Gaussian noise model such as [Equation 9].

According to various embodiments, the processor 140 may be configured to extract orthogonal amplitude views by using a bandpass filter having a predetermined bandwidth, and compensate for a resolution that decreases in response to the bandwidth.

According to various embodiments, the processor 140 may be configured to compensate for resolution by performing deep learning based on a super resolution convolution neural network (SRCNN).

According to various embodiments, the processor 140 is an apparatus configured to synthesize a new hologram using a phase offset determined by a carrier wave.

According to various embodiments, the processor 140 may synthesize a new hologram based on [Equation 10].

An operation method of the electronic device 100 according to various embodiments includes an operation of converting a hologram into light field data, an operation of removing speckle noise from light field data, an operation of compensating resolution for light field data, and It may include an operation of synthesizing a new hologram from the light field data.

According to various embodiments, converting the light field data may include extracting a plurality of orthographic amplitude views from a hologram.

According to various embodiments, the operation of removing speckle noise is an operation of acquiring a residual image related to speckle noise by performing deep learning based on a denoise convolution neural network (DnCNN), and an operation related to light field data. It may include an operation of subtracting the residual image from the original image.

According to various embodiments, DnCNN may be designed based on an additive Gaussian noise model such as [Equation 9].

According to various embodiments, the operation of extracting the orthogonal amplitude views includes extracting the orthogonal amplitude views using a bandpass filter having a predetermined bandwidth, and the operation of compensating the resolution compensates for a resolution that decreases in response to the bandwidth. can do.

According to various embodiments, the operation of compensating the resolution may compensate the resolution by performing deep learning based on a super resolution convolution neural network (SRCNN).

According to various embodiments, the operation of synthesizing a new hologram may synthesize a new hologram by using a phase offset determined by a carrier wave.

According to various embodiments, the operation of synthesizing a new hologram may synthesize a new hologram based on [Equation 10].

According to various embodiments, the electronic device 100 may convert a hologram into light field data and may remove speckle noise from the light field data. In this case, the electronic device 100 may compensate for the reduced resolution based on the light field data. Through this, the electronic device 100 may synthesize a new hologram from light field data for which speckle noise is removed and resolution is compensated. Accordingly, the electronic device 100 can maintain the resolution of the original hologram, even for the new hologram. That is, the electronic device 100 may suppress a decrease in the resolution of the hologram while removing speckle noise from the hologram.

Various embodiments of the present document and terms used therein are not intended to limit the technology described in this document to a specific embodiment, and should be understood to include various modifications, equivalents, and/or substitutes of the corresponding embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar elements. Singular expressions may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and/or B", "A, B or C" or "at least one of A, B and/or C" are all of the items listed together. It can include possible combinations. Expressions such as "first", "second", "first" or "second" can modify the corresponding elements regardless of their order or importance, and are only used to distinguish one element from another. The components are not limited. When it is mentioned that a certain (eg, first) component is “(functionally or communicatively) connected” or “connected” to another (eg, second) component, the certain component is It may be directly connected to the component, or may be connected through another component (eg, a third component).

The term "module" used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic blocks, parts, or circuits. A module may be an integrally configured component or a minimum unit or a part of one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software including one or more instructions stored in a storage medium that can be read by a machine (for example, the electronic device 100). For example, the processor of the device may invoke and execute at least one of the one or more instructions stored from the storage medium. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transient' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term refers to the case where data is semi-permanently stored in the storage medium. It does not distinguish between temporary storage cases.

According to various embodiments, each component (eg, a module or program) of the described components may include a singular number or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallel, repeatedly, or heuristically executed, or one or more of the operations may be executed in a different order, or omitted. , Or one or more other actions may be added.

Claims (15)

In the method of operating an electronic device,
Converting the hologram into light field data;
Removing speckle noise from the light field data;
Compensating the resolution for the light field data; And
And synthesizing a new hologram from the light field data,
The operation of removing the speckle noise,
Performing deep learning based on a denoise convolution neural network (DnCNN) to obtain a residual image related to the speckle noise; And
And subtracting the residual image from the original image related to the light field data,
The DnCNN is,
A method designed based on the additive Gaussian noise model as shown in the following equation.

Here, x represents an image contaminated with noise, y'represents an image without noise, n _{σ'represents} noise, and zero mean and standard deviation

Has a Gaussian distribution with
The method of claim 1,
The operation of converting to the light field data,
And extracting a plurality of orthographic amplitude views from the hologram.
delete delete The method of claim 2,
The operation of extracting the orthographic amplitude views,
Using a bandpass filter having a predetermined bandwidth, extracting the orthogonal amplitude views,
The operation of compensating for the resolution,
A method of compensating for a reduced resolution corresponding to the bandwidth.
The method of claim 1,
The operation of compensating for the resolution,
A method of compensating for the resolution by performing deep learning based on a super resolution convolution neural network (SRCNN).
delete delete In the electronic device,
Memory; And
A processor connected to the memory and operating,
The processor,
Convert the hologram to light field data,
Remove speckle noise from the light field data,
Compensating the resolution for the light field data,
It is configured to synthesize a new hologram from the light field data,
The processor,
By performing deep learning based on DnCNN (denoise convolution neural network), a residual image related to the speckle noise is obtained,
It is configured to subtract the residual image from the original image related to the light field data,
The DnCNN is,
A device designed based on the additive Gaussian noise model as shown in the following equation.

Here, x denotes an image contaminated with noise, y'denotes an image without noise, n _σ'denotes noise, and zero mean and standard deviation

Has a Gaussian distribution with
The method of claim 9,
The processor,
An apparatus configured to extract a plurality of orthographic amplitude views from the hologram as the lightfield data.
delete delete The method of claim 10,
The processor,
Using a bandpass filter having a predetermined bandwidth, extracting the orthogonal amplitude views,
An apparatus configured to compensate for a reduced resolution corresponding to the bandwidth.
The method of claim 9,
The processor,
An apparatus configured to compensate for the resolution by performing deep learning based on a super resolution convolution neural network (SRCNN).
delete

Images