KR102164395B1 - Apparatus for three-dimensional object recognition and tracking using optical scanning holography and convolutional neural network and method thereof - Google Patents

Abstract

The present invention relates to a 3D object recognition and tracking apparatus and method using optical scanning holography and a convolutional neural network. According to the present invention, the steps of obtaining hologram data of an object using the optical scanning holography, extracting the depth position of the object by inputting the hologram data into a pre-learned first CNN neural network, and the depth position Restoring a holographic image of the object using, recognizing the type of the object by inputting the holographic image to a pre-learned second CNN neural network, and feature points of the object extracted from the holographic image And extracting a two-dimensional position of the extracted feature points by comparing pre-stored feature points corresponding to the recognized type of object, and using the depth position and the two-dimensional position of the feature points It provides a three-dimensional object recognition and tracking method comprising the step of tracking the location.
According to the present invention, a depth position of an object is extracted from hologram data of an object using optical scanning holography and a convolutional neural network, and the type of object is recognized from a holographic image reconstructed from the depth position, By extracting the dimensional position, it is possible to finally acquire and track the 3D position of the object.

Description

TECHNICAL FIELD [Apparatus for three-dimensional object recognition and tracking using optical scanning holography and convolutional neural network and method thereof] using optical scanning holography and convolutional neural network

The present invention relates to a three-dimensional object recognition and tracking apparatus and method using optical scanning holography and a convolutional neural network, and more particularly, to recognize an object type and three-dimensional position tracking using holographic data and deep learning techniques. It relates to a 3D object recognition and tracking apparatus and method capable of performing.

Recently, deep learning technology has been grafted into various application fields with rapid development. In particular, in the field of image recognition, a variety of complex unstructured image data can be efficiently analyzed and processed using deep learning technology, which is a collection of advanced algorithms, and thus has been applied for various purposes.

However, since the image data used in general image recognition technology corresponds to 2D information that has no depth information and only light intensity information, it is difficult to recognize the depth position of the 3D space for the object only with this 2D image information. Follows.

Unlike 2D images, hologram data has depth information. However, in the case of hologram data having depth information, since the amount of data is larger than that of a general 2D image, a large amount of computation and time are required.

Therefore, in order to solve these problems, a new 3D recognition technology is required that combines hologram data having depth information on a 3D space and an existing deep learning technology.

The technology behind the present invention is in Korean Patent Application Publication No. 2013-0081127 (published on July 16, 2013).

An object of the present invention is to provide a 3D object recognition and tracking apparatus and method capable of recognizing an object and tracking a 3D position using optical scanning holography and a convolutional neural network.

The present invention provides a method for recognizing and tracking a 3D object in a 3D object recognition and tracking apparatus using optical scanning holography and a convolutional neural network, the method comprising: acquiring hologram data of an object by using the optical scanning holography; , Extracting the depth position of the object by inputting the hologram data into a pre-learned first CNN neural network, and restoring a holographic image of the object in two dimensions at the depth position using the hologram data and the depth position. And recognizing the type of the object by inputting the holographic image into a pre-learned second CNN neural network; and pre-stored corresponding to the object's feature points extracted from the holographic image and the recognized type. 3D object recognition comprising comparing the feature points with each other, extracting the two-dimensional position of the extracted feature points, and tracking the three-dimensional position of the object using the depth position and the two-dimensional position of the feature points And a tracking method.

In addition, the 3D object recognition and tracking method further includes the step of training the first CNN neural network using hologram data previously acquired for the learning object for each of a plurality of depth positions and depth positions corresponding thereto as training data. can do.

In addition, the first CNN neural network uses feature information extracted from hologram data of a learning object for each of a plurality of depth locations and a depth location corresponding thereto as training data, and the feature information is an interference fringe formed around the object. It may be a feature of the fringe pattern.

In addition, the 3D object recognition and tracking method may further include training the second CNN neural network by using a hologram image of a learning object and a type of the learning object corresponding thereto as training data.

In addition, the step of extracting the two-dimensional position of the feature points may include extracting feature points of the object by applying a SURF (Speed Up Robust Feature) algorithm to the hologram image, and matching the extracted feature points with the previously stored feature points. , It is possible to extract the two-dimensional position of the extracted feature points.

In addition, the present invention provides an apparatus for recognizing and tracking a 3D object using optical scanning holography and a convolutional neural network, an acquisition unit that acquires hologram data of an object using the optical scanning holography, and the hologram data. A first extractor for extracting the depth position of the object by inputting it into the learned first CNN neural network, and a restoration unit for restoring a holographic image of the object in two dimensions at the depth position using the hologram data and the depth position; , A recognition unit for recognizing the type of the object by inputting the holographic image to a pre-learned second CNN neural network, and a pre-stored corresponding to the feature points of the object extracted from the holographic image and the recognized type of object. 3 including a second extracting unit for comparing the feature points with each other to extract the two-dimensional position of the extracted feature points, and a tracking unit for tracking the three-dimensional position of the object using the depth position and the two-dimensional position of the feature points It provides a dimensional object recognition and tracking device.

In addition, the 3D object recognition and tracking apparatus is a first learning unit for training the first CNN neural network by using the hologram data previously acquired for the learning object for each depth position and the corresponding depth position as training data. It may contain more.

In addition, the 3D object recognition and tracking apparatus may further include a second learning unit for training the second CNN neural network by using a holographic image of a learning object and a type of the learning object corresponding thereto as training data. .

In addition, the second extracting unit extracts feature points of the object by applying a SURF (Speed Up Robust Feature) algorithm to the holographic image, and matches the extracted feature points with the previously stored feature points to match the extracted feature points. You can extract the two-dimensional position for

According to the present invention, a depth position of an object is extracted from hologram data of an object using optical scanning holography and a convolutional neural network, and the type of object is recognized from a holographic image restored at the corresponding depth position, It provides the advantage of extracting the 2D position and finally obtaining and tracking the 3D position of the object.

1 is a diagram showing a configuration of a 3D object recognition and tracking apparatus using optical scanning holography and a convolutional neural network according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a 3D object recognition and tracking method using the device of FIG. 1.
3 is a diagram illustrating a state in which a fringe pattern is observed around an object in hologram data.
4 is a diagram schematically showing the principle of forming a feature map using convolutions between an image and a filter in a CNN neural network.
5 is a diagram illustrating an example of generating a feature map using various filters for the image of FIG. 4.
6 is a diagram showing a pooling operation performed in a CNN neural network.
7 is a diagram illustrating the structure of a CNN neural network.
8 is a diagram for explaining a principle of recognizing an object using CNN analysis from a reconstructed holographic image in an embodiment of the present invention.
9 is a diagram schematically illustrating a concept of extracting a feature point using a SURF algorithm in an embodiment of the present invention.
10 is a diagram illustrating a concept of obtaining a two-dimensional position by applying a matching algorithm between feature points after feature point extraction in an embodiment of the present invention.

Then, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention.

1 is a diagram showing a configuration of a 3D object recognition and tracking apparatus using optical scanning holography and a convolutional neural network according to an embodiment of the present invention.

As shown in FIG. 1, the apparatus 100 for recognizing and tracking a 3D object using optical scanning holography and a convolutional neural network according to an embodiment of the present invention includes an acquisition unit 110, a first extraction unit 120, A restoration unit 130, a recognition unit 140, a second extraction unit 150, a tracking unit 160, a first learning unit 170, and a second learning unit 180 are included.

First, the acquisition unit 110 acquires hologram data of an object using Optical Scanning Holography (OSH). Optical scanning holography is a device that acquires a hologram of an object based on optical scanning, and has been known in various ways. Optical scanning holography may be included in the device of the present invention or may be operated by wired or wireless connection with the device.

In the following embodiments of the present invention, it is assumed that hologram data of an object is obtained by using the hologram acquisition device of FIG. 1 shown in Patent Publication No. 2013-0081127 by the present applicant mentioned in the background art. However, the present invention is not necessarily limited thereto, and may be selectively applied among known devices.

The configuration of the hologram acquisition device used is largely divided into an optical system and an electronic processing unit. The optical system modulates two beams separated from a light source to form a scan beam by overlapping a spherical wave and a plane wave generated by using an interference means. Then, after scanning the object (object to be photographed) using the formed scan beam, the scan beam reflected from the object is condensed and focused on the light detection means, and an electric signal proportional to the intensity of the focused light is generated through the light detection means. And transfer it to the electronic processing unit.

When these electrical signals are stored for each scanning position, the electrical signal becomes a hologram of the object. In this case, in order to extract a pair image and a complex hologram of an object without background noise, the optical system scans the object by modulating the scan beam over time by using a method of deviating the frequencies of two overlapping beams.

The electronic processing unit receives the electrical signal output from the optical detection means, converts it into a digital signal, generates hologram data of the object from the converted digital signal, and stores it in the storage unit. Here, the electronic processing unit generates and stores a complex hologram of an object through a method of synthesizing the digital signal using a complex number synthesis method and storing it according to each scanning position. Hereinafter, embodiments of the present invention will be described on the assumption that hologram data generated through the above-described method and stored in a storage unit are used.

The first extraction unit 120 extracts the depth position of the object by inputting the obtained hologram data to the first CNN neural network that has been learned. Here, CNN means a known convolutional neural network.

Here, the depth position may mean a distance from which an object is relatively distanced from the position of the optical scanning holography for obtaining holographic data (eg, a preset reference depth position z=0). Of course, when the object to be scanned moves back and forth, the depth position of the object may change over time.

And, in the embodiment of the present invention, the first CNN neural network that has been pre-trained means that the hologram data of the learning object acquired for each of various depth positions previously known to the learning object and the corresponding depth position are used as learning data. , May correspond to a neural network previously learned by the first learning unit 170.

That is, the first learning unit 170 trains the first CNN neural network in advance by using the hologram data previously acquired for the learning object for each of the plurality of depth positions and the corresponding depth position as training data. The first learning unit 170 learns feature information extracted from hologram data and a depth position corresponding thereto.

According to this, it is possible to quickly and easily acquire a depth position for a corresponding object by simply inputting hologram data for a predetermined object into the learned first CNN neural network.

It goes without saying that the type of learning object used for learning the first CNN neural network is not particularly limited, and hologram data for various types of objects can be used for learning depth information.

The holographic data is restored as a holographic image through the restoration unit 130. Various methods have been known in the art for a means for restoring holographic data obtained by optical scanning an object into an image.

In this case, the restoration unit 130 restores the holographic image of the object in 2D at the corresponding depth position by using the hologram data of the object and the extracted depth position. The restoration unit 130 provides a holographic image of an object formed in a two-dimensional shape at a corresponding depth position, and provides a two-dimensional holographic image reconstructed at a corresponding depth position through this. That is, the restoration unit 130 restores a holographic image in the form of a 2D image at a distance point separated by a corresponding depth position based on the device position of the present invention.

Thereafter, the recognition unit 140 recognizes the type of the object by inputting the reconstructed holographic image to the pre-learned second CNN neural network.

Here, the pre-trained second CNN neural network may correspond to a CNN neural network previously learned by the second learning unit 180 using a two-dimensional holographic image (image) of a learning object and a type of a learning object corresponding thereto. have. The second learning unit 180 learns feature information extracted from the reconstructed holographic image and the type of object corresponding thereto.

According to this, simply by inputting the reconstructed 2-dimensional holographic image of the object into the learned second CNN neural network, it is possible to distinguish what kind of object the object is, and output and provide the result. For example, by inputting an image of a reconstructed 2D holographic image into the second CNN neural network, it is possible to quickly and easily determine whether an object in the current reconstructed image is an enemy or a friendly unit, or a cat or a dog.

Here, the holographic image input to the second CNN neural network refers to an image of a 2-dimensional holographic image reconstructed at a corresponding depth position. That is, the second extraction unit 150 inputs an image corresponding to the restored holographic image to the second CNN neural network, and determines and provides the type of object corresponding thereto. In this case, the image may mean image data obtained by photographing a holographic image restored at a corresponding depth position by the apparatus of the present invention.

The second extraction unit 150 compares the feature points of the object extracted from the holographic image and the feature points previously stored in the DB corresponding to the recognized type of object, and the two-dimensional position (horizontal position) of the feature points extracted from the holographic image. Extract.

Specifically, the second extraction unit 150 extracts feature points of an object by applying a SURF (Speed Up Robust Feature) algorithm to the hologram image, and matches the extracted feature points with pre-stored feature points to determine the two-dimensional position of each feature point. Extract.

For example, when an object recognized from a holographic image is identified as a cat, the feature points extracted from the holographic image and the feature points set in the representative image of the cat object previously stored in the DB are compared with each other through the SURF algorithm, and the same feature points are compared. Using a matching method, a two-dimensional position of each feature point in a holographic image is extracted.

Here, the two-dimensional position of the feature point may mean a relative coordinate two-dimensionally away from a reference point (Origin point) based on the device of the present invention. For example, with respect to the 3D coordinates (x ₀ , y ₀ , z ₀ ) of the reference point, the 2D position of each feature point may correspond to (x _i , y _i ).

In this way, in the extraction of the two-dimensional position of each feature point based on feature point matching, information about the distance (depth location) of the object away from the reference point may be used in consideration of a photographing angle and a change in viewpoint.

The above-described SURF algorithm is a technique for searching for feature points that are invariant to environmental changes from several images in consideration of environmental changes such as an image ratio, scale, lighting, and viewpoint. Since this corresponds to a known algorithm, a detailed description will be omitted. In addition, the SIFT (Scale Invariant Feature Transform) algorithm similar to the SURF algorithm may be used for position detection based on such feature point matching.

In the implementation of the present invention, it is possible to check the horizontal position of the holographic reconstructed image by extracting the position of the matched feature point using the SURF algorithm. When the horizontal position and the depth position extracted from the previous processor are converted to the horizontal and depth positions in real space The position of the reconstructed image in the 3D space can be specified.

The tracking unit 160 tracks the three-dimensional position of the object using the depth position of the object and the two-dimensional position of feature points. For example, the depth position (z _i ) of the reconstructed image (object) with respect to the reference point and the 2D position (x _i ,y _i ) of each feature point in the reconstructed image are used to provide a position in the 3D space for the reconstructed image. can do. In addition, when the object to be photographed moves, the 3D position of the object can be acquired and tracked in real time.

As described above, the embodiment of the present invention recognizes the type of object in the holographic image restored through the second CNN neural network, and then, based on the feature points corresponding to the recognized object type, the two-dimensional position of each feature point of the object. It is possible to know the horizontal position of the reconstructed image by calculating. In addition, a three-dimensional position of an object may be provided by combining the two-dimensional position of the feature point and the depth position of the object.

FIG. 2 is a diagram illustrating a 3D object recognition and tracking method using the device of FIG. 1.

First, the first learning unit 170 and the second learning unit 180 train a first CNN neural network and a second CNN neural network, respectively (S210).

To this end, the first learning unit 170 constructs the hologram data previously acquired for the learning object for each of the plurality of depth positions and the corresponding depth position as learning data, and uses the learning data to pre-establish the first CNN neural network. To learn.

In addition, the second learning unit 180 constructs a holographic image of a learning object and a type of a learning object corresponding thereto as learning data, and trains the second CNN neural network in advance by using this.

Each CNN neural network extracts features of the input data through a convolutional layer, and trains the neural network by using the extracted features and the corresponding result values as inputs and outputs, respectively.

The first learning unit 170 uses the feature information extracted from the hologram data of the learning object and the depth position corresponding thereto as learning data. Accordingly, features of hologram data are learned for each depth position of the learning object. The second learning unit 180 uses feature information extracted from a holographic image reconstructed for a learning object and an object type corresponding thereto as learning data. Accordingly, characteristics of holographic images are learned for each type of learning object.

In an embodiment of the present invention, the first learning unit 170 may extract features of a fringe pattern portion, which is an interference pattern formed around an object, from holographic data and use it for learning.

3 is a diagram illustrating a state in which a fringe pattern is observed around an object in hologram data. Referring to the red arrow of FIG. 3, the spacing of the fringe pattern formed along the periphery of the object has a form of a chirp function that gradually increases in the direction of the object. Here, according to the depth of the object away from the optical scanning holographic apparatus, a fringe interval, an increase/decrease rate of the interval, a length change, and the like may vary.

In this way, by analyzing the characteristics of the fringe pattern, information on the depth position of the object can be confirmed. In addition, using this point, an embodiment of the present invention can learn depth information using a fringe pattern around an object on hologram data as feature information.

The learning principle of the CNN neural network corresponds to a known one, but a brief description of this is as follows. The CNN neural network forms a feature map by applying a filter for feature extraction to input data, and performs a pooling operation to increase the computational speed of the extracted features.

FIG. 4 is a schematic diagram showing the principle of forming a feature map by using convolutions between images and filters in a CNN neural network, and FIG. 5 is a diagram illustrating a feature map generation using various filters for the image of FIG. 4 to be.

4A illustrates an input image and a filter applied thereto, and (b) shows a feature map formed through convolution by applying the filter to the input image. . Of course, this corresponds to a known technique.

For the first CNN neural network, for example, a filter is applied to hologram data input to the CNN neural network as shown in FIG. 4 to form a feature map for discriminating characteristics of the data through convolution between the input data and the filter. In this case, as shown in FIG. 5, filters of various sizes and types are used to classify channels to which a corresponding filter is applied, and through this, input data can be classified and analyzed in more detail. Features extracted by the above method undergo a pooling operation to speed up the computation.

6 is a diagram showing a pooling operation performed in a CNN neural network. 6 shows a max pooling method using a maximum value and an average pooling method using an average value. It can be seen that the size of the feature map shown on the left side of FIG. 6 is reduced as a result of the pooling operation.

This pooling operation is a method of reducing the size of data by obtaining a maximum value or an average value of values within a specific area of a square matrix in a matrix structure of a feature map in which data is listed, and replacing it with one value in the corresponding area.

7 is a diagram illustrating the structure of a CNN neural network. 7 shows a CNN neural network in which a feature map generation task and a pooling task are repeated. A convolutional layer can be constructed by repeatedly combining a filter and pooling using convolution to extract features, and the extracted features are neural. It is learned along with the depth position in the network.

In the case of the second CNN neural network, the input/output data is different from that of the first CNN neural network, but the practical learning principle is the same, so a repetitive description is omitted.

After the learning is completed, the acquisition unit 110 acquires a hologram of the object using optical scanning holography (S220),

The first extraction unit 120 inputs the obtained hologram data into the first CNN neural network to derive the depth position of the object (S230), and the restoration unit 130 2D converts the holographic image of the object at the derived depth position. It is restored to the shape (S240).

Then, the recognition unit 140 recognizes the type of the object by inputting the restored holographic image to the second CNN neural network (S250).

8 is a diagram for explaining a principle of recognizing an object using CNN analysis from a reconstructed holographic image in an embodiment of the present invention. The example of FIG. 8 is a result of outputting'A' among possible output values A to D (eg, A = cat, B = dog, C = Puma, D = lion) when a holographic reconstructed image is input to CNN. Represents. In other words, it indicates that the object in the reconstructed image is a cat.

Thereafter, the second extraction unit 150 reads the feature points previously stored in the DB corresponding to the recognized object of the corresponding type and compares the feature points with the feature points of the object extracted from the hologram image. The location is extracted (S260).

9 is a diagram schematically illustrating a concept of extracting a feature point using a SURF algorithm in an embodiment of the present invention.

The SURF feature point detection algorithm replaces the brightness of the image consisting of points in the holographic reconstructed image with the area, simplifies the brightness of the area by applying a filter, and extracts the feature points by calculating singular points using a Hessian matrix. can do.

10 is a diagram illustrating a concept of obtaining a two-dimensional position by applying a matching algorithm between feature points after feature point extraction in an embodiment of the present invention.

The feature points extracted from the reconstructed image and the feature points for the representative image of the corresponding type are matched, and the matched location is extracted to check the horizontal position of the holographic reconstructed image. Here, when the corresponding horizontal position and the previously extracted depth position are converted into horizontal and depth positions in real space, the position of the holographic reconstructed image in the 3D space can be specified.

Accordingly, the tracking unit 160 checks the 3D position of the object using the depth position and the 2D position of the feature points and tracks it in real time (S270).

According to the present invention as described above, a depth position of an object is extracted from hologram data of an object using optical scanning holography and a convolutional neural network, and the type of object is recognized from a holographic image restored at the depth position. By extracting the 2D position of the feature points of, it provides an advantage of finally obtaining and tracking the 3D position of the object.

The present invention has been described with reference to the embodiments shown in the drawings, but these are only exemplary, and those of ordinary skill in the art will appreciate that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical scope of the present invention should be determined by the technical spirit of the appended claims.

100: 3D object recognition and tracking device
110: acquisition unit 120: first extraction unit
130: restoration unit 140: recognition unit
150: second extraction unit 160: tracking unit
170: first learning unit 180: second learning unit

Claims (10)

In a 3D object recognition and tracking method in a 3D object recognition and tracking apparatus using optical scanning holography and a convolutional neural network,
Obtaining holographic data of an object by using the optical scanning holography;
Extracting a depth position of the object by inputting the hologram data into a pre-learned first CNN neural network;
Restoring a holographic image of an object in two dimensions at the depth position using the hologram data and the depth position;
Recognizing the type of the object by inputting the holographic image to a pre-trained second CNN neural network;
Comparing feature points of the object extracted from the hologram image with feature points previously stored corresponding to the recognized type of object, and extracting a two-dimensional position of the extracted feature points; And
3D object recognition and tracking method comprising the step of tracking the 3D position of the object using the depth position and the 2D position of the feature points. The method according to claim 1,
Before the step of acquiring the hologram data, training the first CNN neural network by using the hologram data previously acquired for the learning object for each of the plurality of depth positions and the corresponding depth position as training data 3 Dimensional object recognition and tracking method. The method according to claim 1,
The first CNN neural network,
The feature information extracted from the hologram data of the learning object for each of the plurality of depth positions and the corresponding depth position are used as the learning data,
The feature information,
A 3D object recognition and tracking method that is a characteristic of a fringe pattern, which is an interference pattern formed around an object. The method according to claim 1,
Prior to the step of acquiring the hologram data, 3D object recognition and tracking further comprising the step of training the second CNN neural network using a holographic image of the learning object and the type of the learning object corresponding thereto as training data Way. The method according to claim 1,
Extracting the two-dimensional positions of the feature points,
3 extracting feature points of the object by applying a SURF (Speed Up Robust Feature) algorithm to the hologram image, matching the extracted feature points with the pre-stored feature points, and extracting a two-dimensional position of the extracted feature points Dimensional object recognition and tracking method. In a 3D object recognition and tracking device using optical scanning holography and a convolutional neural network,
An acquisition unit that acquires hologram data of an object by using the optical scanning holography;
A first extraction unit for extracting a depth position of the object by inputting the hologram data into a pre-learned first CNN neural network;
A reconstruction unit for restoring a holographic image of an object in two dimensions at the depth position by using the hologram data and the depth position;
A recognition unit for recognizing the type of the object by inputting the holographic image to a pre-trained second CNN neural network;
A second extractor configured to compare the feature points of the object extracted from the hologram image with the feature points previously stored corresponding to the recognized type of object, and extract a two-dimensional position of the extracted feature points; And
3D object recognition and tracking apparatus comprising a tracking unit for tracking the 3D position of the object using the depth position and the 2D position of the feature points. The method of claim 6,
3D object recognition and tracking apparatus further comprising a first learning unit for training the first CNN neural network by using the hologram data previously acquired for the learning object for each depth position and the corresponding depth position as training data. The method of claim 6,
The first CNN neural network,
The feature information extracted from the hologram data of the learning object for each of the plurality of depth positions and the corresponding depth position are used as the learning data,
The feature information,
A 3D object recognition and tracking device that is a characteristic of a fringe pattern, which is an interference pattern formed around an object. The method of claim 6,
3D object recognition and tracking apparatus further comprising a second learning unit for training the second CNN neural network using the hologram image of the learning object and the type of the learning object corresponding thereto as training data. The method of claim 6,
The second extraction unit,
3 extracting feature points of the object by applying a SURF (Speed Up Robust Feature) algorithm to the hologram image, matching the extracted feature points with the pre-stored feature points, and extracting a two-dimensional position of the extracted feature points Dimensional object recognition and tracking device.

Images