KR20200072158A - Apparatus extracting three-dimensional location of object using optical scanning holography and convolutional neural network and method thereof - Google Patents

Abstract

The present invention relates to an apparatus extracting a 3D location of an object using an optical scanning holography and a convolutional neural network, and a method thereof. According to the present invention, in a method of extracting a 3D location in the apparatus for extracting the 3D location using the optical scanning holography and the convolutional neural network, provided is the method of extracting the 3D position of the object including the steps of: obtaining hologram data of an object using the optical scanning holography; inputting the obtained hologram data into a pre-learned CNN neural network to extract a depth position and a horizontal location of the object, and providing the 3D location of the object using the extracted depth and horizontal location. According to the present invention, it is possible to provide the 3D by extracting the depth and horizontal location of an object from the hologram data of the object using the optical scanning holography and the convolutional neural network.

Description

Apparatus extracting three-dimensional location of object using optical scanning holography and convolutional neural network and method thereof

The present invention relates to an apparatus and a method for extracting a 3D position of an object using optical scanning holography and a convolutional neural network, and more specifically, to extract a 3D position of an object using hologram data and a deep learning technique. The present invention relates to an apparatus and a method for extracting a 3D position of an object.

Hologram is one of the key technologies of the future in the field of the 4th Industrial Revolution, and is being applied to various fields. Currently, holograms are being recognized in our lives as they are grafted into virtual reality (VR), augmented reality (AR), and entertainment.

The hologram, which can enjoy a three-dimensional sense of space, can be applied to other fields such as medical and service beyond culture and art. However, unlike general image information that deals with 2D information, it is still a difficult step to use 3D information through depth and horizontal information of hologram information to other advanced industries.

The reason is that accurate location information needs to be grasped, and, unlike 2D information, a large amount of data possessed by 3D information must be effectively calculated. In order to solve these problems, a new technique that can efficiently calculate the depth and horizontal position of hologram information is required.

The background technology of the present invention is in Korean Patent Publication No. 2013-0081127 (published on July 16, 2013).

An object of the present invention is to provide an apparatus and a method for extracting a 3D location of an object that can extract a 3D location of an object using optical scanning holography and a convolutional neural network.

The present invention, in a three-dimensional position extraction method in a three-dimensional position extraction apparatus of an object using optical scanning holography and a convolutional neural network, obtaining the hologram data of the object using the optical scanning holography, And inputting the obtained hologram data into a pre-trained CNN neural network to extract the depth position and the horizontal position of the object, and providing the 3D position of the object using the extracted depth and horizontal position. Provides a method for extracting a 3D position of an object.

In addition, the three-dimensional position extraction method of the object, further comprising the step of learning the CNN neural network by using the feature information extracted from the hologram data of the learning object for each of the plurality of position information and the corresponding position information as learning data. , The location information may include a depth location and a horizontal location.

In addition, the hologram data may be one selected from a complex hologram obtained from the optical scanning holography, a real hologram corresponding to the real part of the complex hologram, and a deaxial hologram obtained using the complex hologram.

In addition, the CNN neural network may extract the feature information from all or part of an image area for the hologram data.

In addition, the partial region may mean a region within a set range based on a center of mass (COM) of the image or a region of a fringe pattern that is an interference fringe formed around an object in the image.

In addition, the CNN neural network may use the hologram phase information on the frequency domain obtained by Fourier transforming an entire region or a partial region of the hologram data as the feature information.

Also, the CNN neural network extracts only phase information within a set frequency radius from a frequency origin on the frequency domain and uses it as the feature information, and the set frequency may be a maximum frequency of an object to be photographed.

In addition, the present invention, in the apparatus for extracting a 3D position of an object using optical scanning holography and a convolutional neural network, an acquisition unit for acquiring hologram data of the object using the optical scanning holography, and the acquired hologram data 3 of an object including an extraction unit for extracting a depth position and a horizontal position of the object by inputting it into a pre-trained CNN neural network, and a provision unit for providing a 3D position of the object corresponding to the extracted depth and horizontal position. It provides a dimensional position extraction device.

In addition, the apparatus for extracting a 3D position of an object further includes a learning unit for learning the CNN neural network by using feature information extracted from hologram data of a learning object for a plurality of position information and corresponding position information as learning data. , The location information may include a depth location and a horizontal location.

According to the present invention, it is possible to provide a three-dimensional position by extracting the depth and horizontal position of the object from the hologram data of the object using optical scanning holography and a convolutional neural network.

1 is a view showing the configuration of an apparatus for extracting a 3D position of an object using optical scanning holography and a convolutional neural network according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a method for extracting a 3D position of an object using the apparatus of FIG. 1.
3 is a diagram illustrating a concept of using all or part of an image for hologram data for learning in an embodiment of the present invention.
4 is a diagram illustrating a fringe pattern observed around an object in hologram data.
5 is a diagram illustrating a result of Fourier transform of a predetermined hologram data.
FIG. 6 is a diagram briefly showing the principle of forming a feature map using a composite product between an image and a filter in a CNN neural network.
7 is a diagram illustrating a state in which a feature map is generated using various filters on the image of FIG. 6.
8 is a diagram illustrating a pooling operation performed in a CNN neural network.
9 is a diagram illustrating the structure of a CNN neural network.
10 is a diagram illustrating a principle of extracting the depth and horizontal position of an object using CNN analysis from hologram information according to an embodiment of the present invention.

Then, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains can easily practice.

1 is a view showing the configuration of an apparatus for extracting a 3D position of an object using optical scanning holography and a convolutional neural network according to an embodiment of the present invention.

As shown in FIG. 1, the apparatus for extracting a 3D position of an object using optical scanning holography and a convolutional neural network according to an embodiment of the present invention includes an acquiring unit 110, an extracting unit 120, and a provision unit 130, a learning unit 140.

First, the acquiring unit 110 acquires hologram data of an object using optical scanning holography (OSH). Optical scanning holography is a device for acquiring a hologram of an object based on optical scanning, and is known in various ways. The optical scanning holography may be included in the device of the present invention or may be operated in a wired or wireless connection with the device.

The following embodiment of the present invention assumes that the hologram data of the object is obtained by using the hologram acquisition apparatus of FIG. 1 shown in Korean Patent Application Publication No. 2013-0081127 by the present applicant mentioned above in the background art. However, the present invention is not necessarily limited to this, and can be selected and applied among known devices.

The configuration of the used hologram acquisition device is largely divided into an optical system and an electronic processing unit. The optical system modulates the two beams separated from the light source to form a scan beam by superimposing each other on a spherical wave and a plane wave using interference means. Then, an object (an object to be photographed) is scanned using the formed scan beam, and then the scan beam reflected from the object is condensed to focus on the light detection means, and an electrical signal proportional to the intensity of the focused light is generated through the light detection means. To the electronic processing unit.

When these electrical signals are stored according to each scanning position, the electrical signals become holograms of objects. At this time, in order to extract the complex hologram of the object having no paired image and background noise, the optical system scans the object by modulating the scan beam with time using a method of shifting the frequencies of two overlapping beams.

The electronic processing unit receives the electrical signal output from the light detection means, converts it into a digital signal, generates hologram data of the object from the converted digital signal, and stores it in a storage unit. Here, the electronic processing unit generates and stores a complex number hologram for the object through a method of synthesizing a digital signal using a complex number synthesis method and storing it according to each scanning position. Hereinafter, an embodiment of the present invention will be described on the assumption that hologram data generated through the above-described method and stored in the storage unit is used.

In addition, the hologram data used in the embodiment of the present invention may be a complex hologram obtained from optical scanning holography, a real hologram corresponding to a real part of the complex hologram, or decomposition synthesized using a complex hologram ( It may be an off-axis hologram.

In an embodiment of the present invention, the extraction unit 120 extracts the depth position and the horizontal position of the object by inputting the obtained hologram data into a pre-trained CNN neural network. Here, CNN means a well-known convolutional neural network.

Here, the depth position may mean a distance at which an object is relatively separated based on the position of the optical scanning holography (eg, a preset reference depth position z=0) for obtaining hologram data. Of course, if the object to be scanned moves back and forth, the depth position of the object may vary.

The horizontal position means the position on the two-dimensional plane at the corresponding depth. That is, the horizontal position may mean a two-dimensional position of the object with respect to a reference point (for example, x=0, y=0) of a preset x,y plane, and the horizontal position of the object when the object moves up, down, left, and right Can be different.

In addition, in an embodiment of the present invention, the pre-trained CNN neural network uses hologram data for a learning object acquired from various 3D location points already known as a target for a learning object and location information corresponding thereto as learning data. Thus, it may correspond to a neural network previously learned by the learning unit 140. Here, of course, the position information is information including the depth position and the horizontal position of the object.

The learning unit 140 pre-trains the CNN neural network by using the pre-obtained hologram data for the learning object for each of the plurality of location information and the corresponding location information as the training data. Here, the learning unit 140 learns feature information extracted from hologram data and location information corresponding thereto.

According to this, it is possible to quickly and easily obtain a three-dimensional position of the object by simply inputting hologram data for a given object into the learned CNN neural network. In the embodiment of the present invention, the type of the learning object used for the learning of the CNN neural network is not particularly limited, and it is needless to say that hologram data for various types of objects may be used for depth information learning.

The providing unit 130 provides a three-dimensional position of the object using the depth and horizontal position of the object extracted from the extraction unit 120. The providing unit 130 may confirm the three-dimensional position of the object by combining the depth on the z axis of the object and the horizontal position on the x,y plane.

As described above, an embodiment of the present invention can provide a hologram data of an object obtained from optical scanning holography by inputting it into a pre-trained CNN neural network and extracting the three-dimensional position of the object from the output result.

FIG. 2 is a diagram illustrating a method for extracting a 3D position of an object using the apparatus of FIG. 1.

First, the learning unit 140 pre-trains the CNN neural network by using feature information extracted from hologram data of a learning object for each location information and location information corresponding to the location information as the learning data (S210).

The CNN neural network extracts the features of the input data through the convolutional layer, and trains the neural network by using the extracted features and the corresponding result values as inputs and outputs, respectively.

The learning unit 140 uses feature information extracted from hologram data of a learning object and corresponding depth and horizontal positions as learning data. Accordingly, the feature of the hologram data is learned for each depth and horizontal position of the learning object.

3 is a diagram illustrating a concept of using all or part of an image for hologram data for learning in an embodiment of the present invention.

As shown in FIG. 3, in an embodiment of the present invention, the CNN neural network can learn by extracting feature information from all or part of an image area for hologram data. That is, the whole hologram data may be machine-learned or some specific area may be machine-learned. This is because the amount of data computation and data computation time can be reduced when the specific holographic data is processed in a specific area rather than the entire hologram data.

The first picture in FIG. 3 is a case where the entire area of the image is utilized, the second picture is a case where some areas that are characteristic of the object are used for learning, and the third picture is a center of mass (COM; Center) for the entire area. Of Mass).

Methods for obtaining the center of mass of an image are well known. In addition, the set range may be an arbitrary radius or an arbitrary rectangular area, and it is preferable that the specific radius is the size of a FZP (Fresnel zone plate) of optical scanning holography.

As described above, in an embodiment of the present invention, an area within a set range based on the center of mass of an image for hologram data may be used as a partial area, or an area of a fringe pattern that is an interference fringe formed around an object in an image You can also use

In particular, fringe patterns contain information about the depth and horizontal position of objects away from the optical scanning holography device. As such, the learning unit 140 may extract features of the fringe pattern portion, which is an interference fringe formed around the object, from the hologram data and use it for learning.

4 is a view illustrating a fringe pattern observed around an object in hologram data.

Referring to the red arrow in FIG. 4, the interval of the fringe pattern formed along the periphery of the object has a shape of a chirp function that gradually increases in the direction of the object. Here, a fringe interval or an increase or decrease in the interval or a change in length may vary depending on the depth and horizontal position of an object away from the optical scanning holography device.

As described above, by analyzing the characteristics of the fringe pattern, information about the depth and horizontal position of the object can be confirmed. In addition, by using these points, an embodiment of the present invention can learn location information by using a fringe pattern around an object on hologram data as feature information.

In addition, in an embodiment of the present invention, the CNN neural network may use holographic phase information on the frequency domain obtained by Fourier Transformation of an entire region or a partial region of the hologram data as feature information.

In this case, the learning unit 140 may machine-train the CNN neural network by using the phase of the hologram in the Fourier domain as the input of the CNN neural network and the depth position and the horizontal position of the object as outputs. Of course, the learning unit 140 may use the entire hologram phase in the Fourier domain obtained by Fourier transform as an input of a CNN neural network or a part of it.

5 is a diagram illustrating a result of Fourier transform of a predetermined hologram data. In FIG. 5, data on the horizontal axis, vertical axis, and height axis refer to spatial frequencies in the x-direction, spatial frequencies in the y-direction, and intensity, respectively.

In the right figure of FIG. 5, the center of the red circle means the maximum spatial frequency of the object, and hologram information and object information are mixed inside the red circle. A CNN neural network can be trained using a hologram phase feature within a frequency radius of a corresponding region.

As described above, the CNN neural network may use all phase information on the frequency domain, but only phase information within a set frequency radius may be extracted from the frequency origin on the frequency domain and used as feature information. Here, of course, the set frequency represents the maximum frequency of the object to be photographed.

The learning principle of the CNN neural network used in the embodiment of the present invention corresponds to the well-known, but briefly described as follows. The CNN neural network forms a feature map by applying a filter for feature extraction to the input data, and undergoes a pooling operation to speed up the computation of the extracted features.

FIG. 6 is a diagram briefly showing a principle of forming a feature map using a composite product between an image and a filter in a CNN neural network, and FIG. 7 is a diagram illustrating a state in which a feature map is generated using various filters for the image of FIG. 6 to be.

6(a) illustrates an input image and a filter applied thereto, and (b) shows a feature map formed by convolution by applying a corresponding filter to the input image. . Of course, this corresponds to a known technique.

For example, as shown in FIG. 6, a filter is applied to hologram data input to a CNN neural network to form a feature map for discriminating characteristics of data through a composite product between the input data and the filter. At this time, as shown in FIG. 7, various sizes and types of filters are used to classify the channels into which the corresponding filters are applied, and the received data can be further classified and analyzed. Features extracted by the above method go through a pooling operation to speed up the calculation.

8 is a diagram illustrating a pooling operation performed in a CNN neural network. 8 shows both max pooling using a maximum value and an average pooling method using an average value. It can be seen that the feature map shown on the left side of FIG. 8 has been reduced in size as a result of the pulling operation.

Such a pooling operation is a method of reducing the size of data by obtaining a maximum value or an average value for values in a specific area of a square matrix from a matrix structure of a feature map in which data is listed, and replacing it with a single value on the corresponding area.

9 is a diagram illustrating the structure of a CNN neural network. FIG. 9 shows a CNN neural network having a structure in which a feature map generation and a pooling operation are repeated, and a filter and pooling using a composite product can be repeatedly combined to construct a convolutional layer to extract features, and the extracted features are neural. It is learned with depth locations in the network.

After the learning is completed, the acquisition unit 110 acquires a hologram of the object using optical scanning holography (S220). Then, the extraction unit 120 derives the depth position and the horizontal position of the object by inputting the obtained hologram data into the pre-trained CNN neural network (S230), and the providing unit 130 uses the extracted depth and horizontal position Provide a three-dimensional position of the object (S240).

10 is a diagram illustrating a principle of extracting the depth and horizontal position of an object using CNN analysis from hologram information according to an embodiment of the present invention. 10 shows that when hologram data for an object is input to the CNN, the depth and horizontal position of the object are output.

According to the present invention as described above, by using the optical scanning holography and convolutional neural network, it is possible to provide a three-dimensional position by extracting the depth and horizontal position of the object from the hologram data of the object.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: 3D position extraction device of the object
110: acquisition unit 120: extraction unit
130: providing unit 140: learning unit

Claims (14)

In the 3D position extraction method in the 3D position extraction apparatus of an object using optical scanning holography and a convolutional neural network,
Acquiring hologram data of an object using the optical scanning holography;
Extracting the depth position and the horizontal position of the object by inputting the obtained hologram data into a pre-trained CNN neural network; And
And providing a three-dimensional position of the object using the extracted depth and horizontal position. The method according to claim 1,
Further comprising the step of learning the CNN neural network by using the feature information extracted from the hologram data of the learning object for each location information and the corresponding location information as the training data,
The location information is a method of extracting a 3D location of an object including a depth location and a horizontal location. The method according to claim 2,
The hologram data,
A method for extracting a 3D position of an object that is a selected one of a complex hologram obtained from the optical scanning holography, a real hologram corresponding to the real part of the complex hologram, and a deaxial hologram obtained using the complex hologram. The method according to claim 2,
The CNN neural network,
A method of extracting a 3D position of an object that extracts the feature information from all or part of an image area for the hologram data. The method according to claim 4,
Some of the areas,
A method for extracting a three-dimensional position of an object, which means an area within a set range based on a center of mass (COM) of the image or an area of a fringe pattern that is an interference fringe formed around the object in the image. The method according to claim 4,
The CNN neural network,
A method of extracting a three-dimensional position of an object using holographic phase information on a frequency domain obtained by Fourier transforming an entire region or a partial region of the image for the hologram data as the feature information. The method according to claim 6,
The CNN neural network,
Only the phase information within a set frequency radius is extracted from the frequency origin on the frequency domain and used as the feature information.
The set frequency is a method of extracting a 3D position of an object that is a maximum frequency of an object to be photographed. In the apparatus for extracting a 3D position of an object using optical scanning holography and a convolutional neural network,
An acquiring unit for acquiring hologram data of an object using the optical scanning holography;
An extraction unit that inputs the obtained hologram data into a pre-trained CNN neural network to extract a depth position and a horizontal position of the object; And
A device for extracting a 3D position of an object, including a provision unit for providing a 3D position of the object corresponding to the extracted depth and horizontal position. The method according to claim 8,
Further comprising a learning unit for learning the CNN neural network by using the feature information extracted from the hologram data of the learning object for each location information and the corresponding location information as the training data,
The position information is a device for extracting a 3D position of an object including a depth position and a horizontal position. The method according to claim 9,
The hologram data,
3D position extraction apparatus for an object that is a selected one of a complex hologram obtained from the optical scanning holography, a real hologram corresponding to the real part of the complex hologram, and a de-axis hologram obtained using the complex hologram. The method according to claim 9,
The CNN neural network,
An apparatus for extracting a 3D position of an object that extracts the feature information from all or part of an image area for the hologram data. The method according to claim 11,
Some of the areas,
3D position extraction device for an object that refers to a region within a set range based on a center of mass (COM) of the image or a fringe pattern that is an interference fringe formed around an object in the image. The method according to claim 11,
The CNN neural network,
An apparatus for extracting a 3D position of an object using holographic phase information on a frequency domain obtained by Fourier transforming an entire area or a partial area of the image for the hologram data as the feature information. The method according to claim 13,
The CNN neural network,
Only the phase information within a set frequency radius is extracted from the frequency origin on the frequency domain and used as the feature information.
The set frequency is the maximum frequency of the object to be photographed.

Images